2024-04-22T08:04:59.701 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-22T08:04:59.701 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/teuthology-2024-04-21_22:00:02-rados-reef-distro-default-smithi/7667865 --owner scheduled_teuthology@teuthology --name teuthology-2024-04-21_22:00:02-rados-reef-distro-default-smithi --verbose --description rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/one workloads/pool-create-delete} -- /home/teuthworker/archive/teuthology-2024-04-21_22:00:02-rados-reef-distro-default-smithi/7667865/orig.config.yaml 2024-04-22T08:04:59.743 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-22T08:04:59.846 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-21_22:00:02-rados-reef-distro-default-smithi/7667865 branch: reef description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/one workloads/pool-create-delete} email: ceph-qa@ceph.com first_in_suite: false job_id: '7667865' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-04-21_22:00:02-rados-reef-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: reef ceph: conf: client: debug monc: 20 debug ms: 1 global: mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr1: false ms bind msgr2: true ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: mon ms inject internal delays: 0.002 ms inject socket failures: 2500 ms type: async mgr: debug mgr: 20 debug monc: 10 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 paxos service trim min: 5 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd memory target: 1610612736 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(POOL_APP_NOT_ENABLED\) - slow request - overall HEALTH_ sha1: 646b8c7d293d8ec31a40c69f7b3237b19c1be24d ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 646b8c7d293d8ec31a40c69f7b3237b19c1be24d thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: reef sha1: c3a345d31ea77c01771428ab80e1cf32c9b05875 owner: scheduled_teuthology@teuthology priority: 931 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 7353 sha1: 646b8c7d293d8ec31a40c69f7b3237b19c1be24d sleep_before_teardown: 0 subset: 3333/100000 suite: rados suite_branch: reef suite_path: /home/teuthworker/src/git.ceph.com_ceph_c3a345d31ea77c01771428ab80e1cf32c9b05875/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: c3a345d31ea77c01771428ab80e1cf32c9b05875 targets: smithi040.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPCbB9kJt2IkRaeY5ZmIgRyOPqr6BAiF7tecZKlWcpj8VtR4PzBUj1MStl+E6kL7uxljhbCijOSXZo4js8l61XM= smithi189.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIM87Hq2VE6K9aTqZUaXZf7MMMTbfy/r9Mue2robyIO43KX2BBm9xDGsc6DW4OnueKI73GgyIld9xWpRFr1Xk8I= tasks: - install: null - ceph: null - mon_thrash: revive_delay: 20 thrash_delay: 1 - exec: client.0: - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-21_22:00:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-22T08:04:59.846 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_c3a345d31ea77c01771428ab80e1cf32c9b05875/qa; will attempt to use it 2024-04-22T08:04:59.847 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_c3a345d31ea77c01771428ab80e1cf32c9b05875/qa/tasks 2024-04-22T08:04:59.847 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-22T08:04:59.849 INFO:teuthology.task.internal:Checking packages... 2024-04-22T08:04:59.874 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '646b8c7d293d8ec31a40c69f7b3237b19c1be24d' 2024-04-22T08:04:59.874 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-22T08:04:59.874 INFO:teuthology.packaging:ref: None 2024-04-22T08:04:59.874 INFO:teuthology.packaging:tag: None 2024-04-22T08:04:59.874 INFO:teuthology.packaging:branch: reef 2024-04-22T08:04:59.874 INFO:teuthology.packaging:sha1: 646b8c7d293d8ec31a40c69f7b3237b19c1be24d 2024-04-22T08:04:59.874 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-04-22T08:05:00.107 INFO:teuthology.task.internal:Found packages for ceph version 18.2.2-0 2024-04-22T08:05:00.109 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-22T08:05:00.116 INFO:teuthology.task.internal:no buildpackages task found 2024-04-22T08:05:00.116 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-22T08:05:00.156 INFO:teuthology.task.internal:Saving configuration 2024-04-22T08:05:00.172 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-22T08:05:00.215 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T08:05:00.249 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi040.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-21_22:00:02-rados-reef-distro-default-smithi/7667865', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 07:59:50.725407', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPCbB9kJt2IkRaeY5ZmIgRyOPqr6BAiF7tecZKlWcpj8VtR4PzBUj1MStl+E6kL7uxljhbCijOSXZo4js8l61XM='} 2024-04-22T08:05:00.282 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi189.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-21_22:00:02-rados-reef-distro-default-smithi/7667865', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 07:59:50.727005', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIM87Hq2VE6K9aTqZUaXZf7MMMTbfy/r9Mue2robyIO43KX2BBm9xDGsc6DW4OnueKI73GgyIld9xWpRFr1Xk8I='} 2024-04-22T08:05:00.282 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-22T08:05:00.288 INFO:teuthology.task.internal:roles: ubuntu@smithi040.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2024-04-22T08:05:00.288 INFO:teuthology.task.internal:roles: ubuntu@smithi189.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2024-04-22T08:05:00.288 INFO:teuthology.run_tasks:Running task console_log... 2024-04-22T08:05:00.387 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f758e83ae50>, signals=[15]) 2024-04-22T08:05:00.388 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-22T08:05:00.394 INFO:teuthology.task.internal:Opening connections... 2024-04-22T08:05:00.394 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi040.front.sepia.ceph.com 2024-04-22T08:05:00.396 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T08:05:00.496 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi189.front.sepia.ceph.com 2024-04-22T08:05:00.497 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T08:05:00.581 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-22T08:05:00.588 DEBUG:teuthology.orchestra.run.smithi040:> uname -m 2024-04-22T08:05:00.607 INFO:teuthology.orchestra.run.smithi040.stdout:x86_64 2024-04-22T08:05:00.607 DEBUG:teuthology.orchestra.run.smithi040:> cat /etc/os-release 2024-04-22T08:05:00.665 INFO:teuthology.orchestra.run.smithi040.stdout:NAME="CentOS Stream" 2024-04-22T08:05:00.666 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION="9" 2024-04-22T08:05:00.666 INFO:teuthology.orchestra.run.smithi040.stdout:ID="centos" 2024-04-22T08:05:00.666 INFO:teuthology.orchestra.run.smithi040.stdout:ID_LIKE="rhel fedora" 2024-04-22T08:05:00.666 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION_ID="9" 2024-04-22T08:05:00.666 INFO:teuthology.orchestra.run.smithi040.stdout:PLATFORM_ID="platform:el9" 2024-04-22T08:05:00.666 INFO:teuthology.orchestra.run.smithi040.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-22T08:05:00.666 INFO:teuthology.orchestra.run.smithi040.stdout:ANSI_COLOR="0;31" 2024-04-22T08:05:00.666 INFO:teuthology.orchestra.run.smithi040.stdout:LOGO="fedora-logo-icon" 2024-04-22T08:05:00.666 INFO:teuthology.orchestra.run.smithi040.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-22T08:05:00.666 INFO:teuthology.orchestra.run.smithi040.stdout:HOME_URL="https://centos.org/" 2024-04-22T08:05:00.666 INFO:teuthology.orchestra.run.smithi040.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-22T08:05:00.667 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-22T08:05:00.667 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-22T08:05:00.667 INFO:teuthology.lock.ops:Updating smithi040.front.sepia.ceph.com on lock server 2024-04-22T08:05:00.726 DEBUG:teuthology.orchestra.run.smithi189:> uname -m 2024-04-22T08:05:00.744 INFO:teuthology.orchestra.run.smithi189.stdout:x86_64 2024-04-22T08:05:00.745 DEBUG:teuthology.orchestra.run.smithi189:> cat /etc/os-release 2024-04-22T08:05:00.801 INFO:teuthology.orchestra.run.smithi189.stdout:NAME="CentOS Stream" 2024-04-22T08:05:00.801 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION="9" 2024-04-22T08:05:00.802 INFO:teuthology.orchestra.run.smithi189.stdout:ID="centos" 2024-04-22T08:05:00.802 INFO:teuthology.orchestra.run.smithi189.stdout:ID_LIKE="rhel fedora" 2024-04-22T08:05:00.802 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION_ID="9" 2024-04-22T08:05:00.802 INFO:teuthology.orchestra.run.smithi189.stdout:PLATFORM_ID="platform:el9" 2024-04-22T08:05:00.802 INFO:teuthology.orchestra.run.smithi189.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-22T08:05:00.802 INFO:teuthology.orchestra.run.smithi189.stdout:ANSI_COLOR="0;31" 2024-04-22T08:05:00.802 INFO:teuthology.orchestra.run.smithi189.stdout:LOGO="fedora-logo-icon" 2024-04-22T08:05:00.802 INFO:teuthology.orchestra.run.smithi189.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-22T08:05:00.802 INFO:teuthology.orchestra.run.smithi189.stdout:HOME_URL="https://centos.org/" 2024-04-22T08:05:00.802 INFO:teuthology.orchestra.run.smithi189.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-22T08:05:00.802 INFO:teuthology.orchestra.run.smithi189.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-22T08:05:00.802 INFO:teuthology.orchestra.run.smithi189.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-22T08:05:00.803 INFO:teuthology.lock.ops:Updating smithi189.front.sepia.ceph.com on lock server 2024-04-22T08:05:00.843 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-22T08:05:00.852 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-22T08:05:00.872 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-22T08:05:00.872 DEBUG:teuthology.orchestra.run.smithi040:> test '!' -e /home/ubuntu/cephtest 2024-04-22T08:05:00.875 DEBUG:teuthology.orchestra.run.smithi189:> test '!' -e /home/ubuntu/cephtest 2024-04-22T08:05:00.891 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-22T08:05:00.897 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-22T08:05:00.897 DEBUG:teuthology.orchestra.run.smithi040:> test -z $(ls -A /var/lib/ceph) 2024-04-22T08:05:00.933 DEBUG:teuthology.orchestra.run.smithi189:> test -z $(ls -A /var/lib/ceph) 2024-04-22T08:05:01.036 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-22T08:05:01.106 INFO:teuthology.run_tasks:Running task kernel... 2024-04-22T08:05:01.119 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-22T08:05:01.120 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-22T08:05:01.120 DEBUG:teuthology.orchestra.run.smithi040:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-22T08:05:01.120 DEBUG:teuthology.orchestra.run.smithi189:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-22T08:05:01.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T08:05:01.138 DEBUG:teuthology.orchestra.run.smithi189:> uname -r 2024-04-22T08:05:01.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T08:05:01.139 DEBUG:teuthology.orchestra.run.smithi040:> uname -r 2024-04-22T08:05:01.195 INFO:teuthology.orchestra.run.smithi189.stdout:5.14.0-437.el9.x86_64 2024-04-22T08:05:01.195 INFO:teuthology.task.kernel:Running kernel on smithi189: 5.14.0-437.el9.x86_64 2024-04-22T08:05:01.195 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum install -y kernel 2024-04-22T08:05:01.196 INFO:teuthology.orchestra.run.smithi040.stdout:5.14.0-437.el9.x86_64 2024-04-22T08:05:01.197 INFO:teuthology.task.kernel:Running kernel on smithi040: 5.14.0-437.el9.x86_64 2024-04-22T08:05:01.197 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum install -y kernel 2024-04-22T08:05:03.590 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-04-22T08:05:05.448 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - AppStream 24 MB/s | 19 MB 00:00 2024-04-22T08:05:08.615 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - BaseOS 1.4 MB/s | 8.0 MB 00:05 2024-04-22T08:05:10.715 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.2 MB 00:01 2024-04-22T08:05:12.625 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 16 kB 00:00 2024-04-22T08:05:14.534 INFO:teuthology.orchestra.run.smithi189.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-04-22T08:05:19.550 INFO:teuthology.orchestra.run.smithi189.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-04-22T08:05:21.406 INFO:teuthology.orchestra.run.smithi189.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-22T08:05:21.406 INFO:teuthology.orchestra.run.smithi189.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-22T08:05:21.487 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-22T08:05:21.488 INFO:teuthology.orchestra.run.smithi189.stdout:Nothing to do. 2024-04-22T08:05:21.488 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-22T08:05:21.598 DEBUG:teuthology.orchestra.run.smithi189:> echo no | sudo yum reinstall kernel || true 2024-04-22T08:05:22.137 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:00:03 ago on Mon 22 Apr 2024 08:05:19 AM UTC. 2024-04-22T08:05:22.257 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-22T08:05:22.279 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:05:22.279 INFO:teuthology.orchestra.run.smithi189.stdout: Package Architecture Version Repository Size 2024-04-22T08:05:22.279 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:05:22.279 INFO:teuthology.orchestra.run.smithi189.stdout:Reinstalling: 2024-04-22T08:05:22.280 INFO:teuthology.orchestra.run.smithi189.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T08:05:22.280 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:05:22.280 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-22T08:05:22.280 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:05:22.280 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:05:22.280 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 255 k 2024-04-22T08:05:22.280 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 0 2024-04-22T08:05:22.280 INFO:teuthology.orchestra.run.smithi189.stderr:Operation aborted. 2024-04-22T08:05:22.344 INFO:teuthology.orchestra.run.smithi189.stdout:Is this ok [y/N]: 2024-04-22T08:05:22.344 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum reinstall -y kernel || true 2024-04-22T08:05:22.841 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:00:03 ago on Mon 22 Apr 2024 08:05:19 AM UTC. 2024-04-22T08:05:22.958 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-22T08:05:22.959 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:05:22.959 INFO:teuthology.orchestra.run.smithi189.stdout: Package Architecture Version Repository Size 2024-04-22T08:05:22.959 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:05:22.959 INFO:teuthology.orchestra.run.smithi189.stdout:Reinstalling: 2024-04-22T08:05:22.960 INFO:teuthology.orchestra.run.smithi189.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T08:05:22.960 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:05:22.960 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-22T08:05:22.960 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:05:22.960 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:05:22.960 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 255 k 2024-04-22T08:05:22.960 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 0 2024-04-22T08:05:22.960 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-22T08:05:23.669 INFO:teuthology.orchestra.run.smithi189.stdout:kernel-5.14.0-437.el9.x86_64.rpm 572 kB/s | 255 kB 00:00 2024-04-22T08:05:23.669 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-22T08:05:23.670 INFO:teuthology.orchestra.run.smithi189.stdout:Total 359 kB/s | 255 kB 00:00 2024-04-22T08:05:23.691 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-22T08:05:23.880 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-22T08:05:23.880 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-22T08:05:23.885 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-22T08:05:23.885 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-22T08:05:24.039 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-22T08:05:24.112 INFO:teuthology.orchestra.run.smithi189.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T08:05:24.112 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T08:05:24.743 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T08:05:24.743 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T08:05:25.098 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:15 2024-04-22T08:05:25.454 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T08:05:25.454 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:05:25.454 INFO:teuthology.orchestra.run.smithi189.stdout:Reinstalled: 2024-04-22T08:05:25.454 INFO:teuthology.orchestra.run.smithi189.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-22T08:05:25.455 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:05:25.455 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-22T08:05:25.632 DEBUG:teuthology.orchestra.run.smithi189:> rpm -q kernel | sort -rV | head -n 1 2024-04-22T08:05:25.718 INFO:teuthology.orchestra.run.smithi189.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-22T08:05:25.719 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-22T08:05:25.719 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-22T08:05:25.719 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-22T08:05:25.719 DEBUG:teuthology.orchestra.run.smithi189:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-22T08:05:25.751 INFO:teuthology.orchestra.run.smithi189.stdout:ttyS1 2024-04-22T08:05:25.773 DEBUG:teuthology.parallel:result is None 2024-04-22T08:05:30.232 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - CRB 4.5 MB/s | 6.2 MB 00:01 2024-04-22T08:05:31.899 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-04-22T08:05:33.759 INFO:teuthology.orchestra.run.smithi040.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-04-22T08:05:39.009 INFO:teuthology.orchestra.run.smithi040.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-04-22T08:05:39.201 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:00:01 ago on Mon 22 Apr 2024 08:05:38 AM UTC. 2024-04-22T08:05:40.973 INFO:teuthology.orchestra.run.smithi040.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-22T08:05:40.973 INFO:teuthology.orchestra.run.smithi040.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-22T08:05:41.070 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-22T08:05:41.071 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2024-04-22T08:05:41.071 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-22T08:05:41.170 DEBUG:teuthology.orchestra.run.smithi040:> echo no | sudo yum reinstall kernel || true 2024-04-22T08:05:41.688 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:00:03 ago on Mon 22 Apr 2024 08:05:38 AM UTC. 2024-04-22T08:05:41.811 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-22T08:05:41.824 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:05:41.824 INFO:teuthology.orchestra.run.smithi040.stdout: Package Architecture Version Repository Size 2024-04-22T08:05:41.824 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:05:41.824 INFO:teuthology.orchestra.run.smithi040.stdout:Reinstalling: 2024-04-22T08:05:41.824 INFO:teuthology.orchestra.run.smithi040.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T08:05:41.825 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:05:41.825 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-04-22T08:05:41.825 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:05:41.825 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:05:41.825 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 255 k 2024-04-22T08:05:41.825 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 0 2024-04-22T08:05:41.825 INFO:teuthology.orchestra.run.smithi040.stderr:Operation aborted. 2024-04-22T08:05:41.884 INFO:teuthology.orchestra.run.smithi040.stdout:Is this ok [y/N]: 2024-04-22T08:05:41.885 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum reinstall -y kernel || true 2024-04-22T08:05:42.401 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:00:04 ago on Mon 22 Apr 2024 08:05:38 AM UTC. 2024-04-22T08:05:42.522 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-22T08:05:42.522 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:05:42.522 INFO:teuthology.orchestra.run.smithi040.stdout: Package Architecture Version Repository Size 2024-04-22T08:05:42.522 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:05:42.522 INFO:teuthology.orchestra.run.smithi040.stdout:Reinstalling: 2024-04-22T08:05:42.523 INFO:teuthology.orchestra.run.smithi040.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T08:05:42.523 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:05:42.523 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-04-22T08:05:42.523 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:05:42.523 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:05:42.523 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 255 k 2024-04-22T08:05:42.523 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 0 2024-04-22T08:05:42.524 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-04-22T08:05:42.811 INFO:teuthology.orchestra.run.smithi040.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.5 MB/s | 255 kB 00:00 2024-04-22T08:05:42.811 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-04-22T08:05:42.811 INFO:teuthology.orchestra.run.smithi040.stdout:Total 887 kB/s | 255 kB 00:00 2024-04-22T08:05:42.826 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-04-22T08:05:43.031 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-04-22T08:05:43.031 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-04-22T08:05:43.036 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-04-22T08:05:43.037 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-04-22T08:05:43.176 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-04-22T08:05:43.237 INFO:teuthology.orchestra.run.smithi040.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T08:05:43.237 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T08:05:43.909 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T08:05:43.909 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T08:05:44.609 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T08:05:44.609 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:05:44.609 INFO:teuthology.orchestra.run.smithi040.stdout:Reinstalled: 2024-04-22T08:05:44.609 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-22T08:05:44.609 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:05:44.609 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-22T08:05:44.801 DEBUG:teuthology.orchestra.run.smithi040:> rpm -q kernel | sort -rV | head -n 1 2024-04-22T08:05:44.849 INFO:teuthology.orchestra.run.smithi040.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-22T08:05:44.849 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-22T08:05:44.849 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-22T08:05:44.849 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-22T08:05:44.849 DEBUG:teuthology.orchestra.run.smithi040:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-22T08:05:44.879 INFO:teuthology.orchestra.run.smithi040.stdout:ttyS1 2024-04-22T08:05:44.900 DEBUG:teuthology.parallel:result is None 2024-04-22T08:05:44.900 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-22T08:05:44.908 INFO:teuthology.task.internal:Creating test directory... 2024-04-22T08:05:44.909 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-22T08:05:44.943 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-22T08:05:44.961 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-22T08:05:44.968 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-22T08:05:44.992 INFO:teuthology.task.internal:Creating archive directory... 2024-04-22T08:05:44.993 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-22T08:05:44.998 DEBUG:teuthology.orchestra.run.smithi189:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-22T08:05:45.052 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-22T08:05:45.060 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-22T08:05:45.060 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-22T08:05:45.063 DEBUG:teuthology.orchestra.run.smithi189:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-22T08:05:45.096 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T08:05:45.105 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T08:05:45.126 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T08:05:45.137 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T08:05:45.139 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-22T08:05:45.146 INFO:teuthology.task.internal:Configuring sudo... 2024-04-22T08:05:45.147 DEBUG:teuthology.orchestra.run.smithi040:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-22T08:05:45.150 DEBUG:teuthology.orchestra.run.smithi189:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-22T08:05:45.205 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-22T08:05:45.215 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-22T08:05:45.215 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-22T08:05:45.217 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-22T08:05:45.262 DEBUG:teuthology.orchestra.run.smithi040:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T08:05:45.322 DEBUG:teuthology.orchestra.run.smithi040:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T08:05:45.371 DEBUG:teuthology.orchestra.run.smithi040:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T08:05:45.462 DEBUG:teuthology.orchestra.run.smithi040:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T08:05:45.492 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-22T08:05:45.492 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-22T08:05:45.556 DEBUG:teuthology.orchestra.run.smithi189:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T08:05:45.604 DEBUG:teuthology.orchestra.run.smithi189:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T08:05:45.649 DEBUG:teuthology.orchestra.run.smithi189:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T08:05:45.742 DEBUG:teuthology.orchestra.run.smithi189:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T08:05:45.776 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-22T08:05:45.776 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-22T08:05:45.842 DEBUG:teuthology.orchestra.run.smithi040:> sudo service rsyslog restart 2024-04-22T08:05:45.844 DEBUG:teuthology.orchestra.run.smithi189:> sudo service rsyslog restart 2024-04-22T08:05:45.884 INFO:teuthology.orchestra.run.smithi040.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T08:05:45.922 INFO:teuthology.orchestra.run.smithi189.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T08:05:46.210 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-22T08:05:46.218 INFO:teuthology.task.internal:Starting timer... 2024-04-22T08:05:46.218 INFO:teuthology.run_tasks:Running task pcp... 2024-04-22T08:05:46.254 INFO:teuthology.run_tasks:Running task selinux... 2024-04-22T08:05:46.286 DEBUG:teuthology.orchestra.run.smithi040:> sudo service auditd rotate 2024-04-22T08:05:46.370 INFO:teuthology.orchestra.run.smithi040.stdout:Rotating logs: 2024-04-22T08:05:46.372 DEBUG:teuthology.orchestra.run.smithi189:> sudo service auditd rotate 2024-04-22T08:05:46.418 INFO:teuthology.orchestra.run.smithi189.stdout:Rotating logs: 2024-04-22T08:05:46.420 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-22T08:05:46.420 DEBUG:teuthology.orchestra.run.smithi040:> /usr/sbin/getenforce 2024-04-22T08:05:46.444 INFO:teuthology.orchestra.run.smithi040.stdout:Permissive 2024-04-22T08:05:46.444 DEBUG:teuthology.orchestra.run.smithi189:> /usr/sbin/getenforce 2024-04-22T08:05:46.477 INFO:teuthology.orchestra.run.smithi189.stdout:Permissive 2024-04-22T08:05:46.477 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi040.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi189.front.sepia.ceph.com': 'permissive'} 2024-04-22T08:05:46.477 DEBUG:teuthology.orchestra.run.smithi040:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-22T08:05:46.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T08:05:46.510 DEBUG:teuthology.orchestra.run.smithi189:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-22T08:05:46.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T08:05:46.544 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-22T08:05:46.544 DEBUG:teuthology.orchestra.run.smithi040:> sudo /usr/sbin/setenforce permissive 2024-04-22T08:05:46.583 DEBUG:teuthology.orchestra.run.smithi189:> sudo /usr/sbin/setenforce permissive 2024-04-22T08:05:46.609 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-22T08:05:46.619 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-22T08:05:46.637 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-22T08:05:46.638 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi040.front.sepia.ceph.com,smithi189.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-22T08:09:18.637 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi040.front.sepia.ceph.com'), Remote(name='ubuntu@smithi189.front.sepia.ceph.com')] 2024-04-22T08:09:18.638 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-04-22T08:09:18.639 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T08:09:18.728 DEBUG:teuthology.orchestra.run.smithi040:> true 2024-04-22T08:09:18.809 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-04-22T08:09:18.810 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-04-22T08:09:18.810 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T08:09:18.896 DEBUG:teuthology.orchestra.run.smithi189:> true 2024-04-22T08:09:18.977 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-04-22T08:09:18.978 INFO:teuthology.run_tasks:Running task clock... 2024-04-22T08:09:18.988 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-22T08:09:18.988 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-22T08:09:18.989 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T08:09:18.991 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-22T08:09:18.991 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T08:09:19.018 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-22T08:09:19.031 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-22T08:09:19.046 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-22T08:09:19.054 INFO:teuthology.orchestra.run.smithi040.stderr:sudo: ntpd: command not found 2024-04-22T08:09:19.059 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-22T08:09:19.065 INFO:teuthology.orchestra.run.smithi040.stdout:506 Cannot talk to daemon 2024-04-22T08:09:19.078 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-22T08:09:19.081 INFO:teuthology.orchestra.run.smithi189.stderr:sudo: ntpd: command not found 2024-04-22T08:09:19.091 INFO:teuthology.orchestra.run.smithi189.stdout:506 Cannot talk to daemon 2024-04-22T08:09:19.092 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-22T08:09:19.104 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-22T08:09:19.117 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-22T08:09:19.154 INFO:teuthology.orchestra.run.smithi040.stderr:bash: line 1: ntpq: command not found 2024-04-22T08:09:19.158 INFO:teuthology.orchestra.run.smithi040.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T08:09:19.158 INFO:teuthology.orchestra.run.smithi040.stdout:=============================================================================== 2024-04-22T08:09:19.158 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T08:09:19.158 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T08:09:19.158 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T08:09:19.158 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T08:09:19.175 INFO:teuthology.orchestra.run.smithi189.stderr:bash: line 1: ntpq: command not found 2024-04-22T08:09:19.179 INFO:teuthology.orchestra.run.smithi189.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T08:09:19.179 INFO:teuthology.orchestra.run.smithi189.stdout:=============================================================================== 2024-04-22T08:09:19.179 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T08:09:19.179 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T08:09:19.179 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T08:09:19.179 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T08:09:19.180 INFO:teuthology.run_tasks:Running task install... 2024-04-22T08:09:19.188 DEBUG:teuthology.task.install:project ceph 2024-04-22T08:09:19.188 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '646b8c7d293d8ec31a40c69f7b3237b19c1be24d'}} 2024-04-22T08:09:19.188 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '646b8c7d293d8ec31a40c69f7b3237b19c1be24d'} 2024-04-22T08:09:19.188 INFO:teuthology.task.install:Using flavor: default 2024-04-22T08:09:19.194 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-22T08:09:19.194 INFO:teuthology.task.install:extra packages: [] 2024-04-22T08:09:19.194 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '646b8c7d293d8ec31a40c69f7b3237b19c1be24d', 'tag': None, 'wait_for_package': False} 2024-04-22T08:09:19.195 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=646b8c7d293d8ec31a40c69f7b3237b19c1be24d 2024-04-22T08:09:19.197 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '646b8c7d293d8ec31a40c69f7b3237b19c1be24d', 'tag': None, 'wait_for_package': False} 2024-04-22T08:09:19.197 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=646b8c7d293d8ec31a40c69f7b3237b19c1be24d 2024-04-22T08:09:19.358 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/centos/9/flavors/default/ 2024-04-22T08:09:19.358 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1237.g646b8c7d 2024-04-22T08:09:19.454 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/centos/9/flavors/default/ 2024-04-22T08:09:19.454 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1237.g646b8c7d 2024-04-22T08:09:19.735 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-22T08:09:19.735 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-22T08:09:19.735 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-22T08:09:19.763 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-22T08:09:19.763 DEBUG:teuthology.orchestra.run.smithi040:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-22T08:09:19.774 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-22T08:09:19.774 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-22T08:09:19.774 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-22T08:09:19.800 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-22T08:09:19.800 DEBUG:teuthology.orchestra.run.smithi189:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-22T08:09:19.830 DEBUG:teuthology.orchestra.run.smithi040:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-22T08:09:19.866 DEBUG:teuthology.orchestra.run.smithi189:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-22T08:09:19.905 DEBUG:teuthology.orchestra.run.smithi040:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-22T08:09:19.944 DEBUG:teuthology.orchestra.run.smithi189:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-22T08:09:19.970 INFO:teuthology.orchestra.run.smithi040.stdout:check_obsoletes = 1 2024-04-22T08:09:19.971 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum clean all 2024-04-22T08:09:20.009 INFO:teuthology.orchestra.run.smithi189.stdout:check_obsoletes = 1 2024-04-22T08:09:20.010 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum clean all 2024-04-22T08:09:20.205 INFO:teuthology.orchestra.run.smithi040.stdout:48 files removed 2024-04-22T08:09:20.231 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-radosgw 2024-04-22T08:09:20.235 INFO:teuthology.orchestra.run.smithi189.stdout:48 files removed 2024-04-22T08:09:20.262 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-radosgw 2024-04-22T08:09:21.332 INFO:teuthology.orchestra.run.smithi040.stdout:ceph packages for x86_64 93 kB/s | 77 kB 00:00 2024-04-22T08:09:21.444 INFO:teuthology.orchestra.run.smithi189.stdout:ceph packages for x86_64 83 kB/s | 77 kB 00:00 2024-04-22T08:09:21.927 INFO:teuthology.orchestra.run.smithi040.stdout:ceph noarch packages 21 kB/s | 11 kB 00:00 2024-04-22T08:09:22.044 INFO:teuthology.orchestra.run.smithi189.stdout:ceph noarch packages 20 kB/s | 11 kB 00:00 2024-04-22T08:09:22.566 INFO:teuthology.orchestra.run.smithi040.stdout:ceph source packages 3.1 kB/s | 1.9 kB 00:00 2024-04-22T08:09:22.686 INFO:teuthology.orchestra.run.smithi189.stdout:ceph source packages 3.1 kB/s | 1.9 kB 00:00 2024-04-22T08:09:24.190 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - BaseOS 5.1 MB/s | 8.0 MB 00:01 2024-04-22T08:09:24.270 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - BaseOS 5.2 MB/s | 8.0 MB 00:01 2024-04-22T08:09:26.920 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - AppStream 12 MB/s | 19 MB 00:01 2024-04-22T08:09:26.979 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - AppStream 12 MB/s | 19 MB 00:01 2024-04-22T08:09:31.242 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - CRB 9.8 MB/s | 6.2 MB 00:00 2024-04-22T08:09:32.768 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - Extras packages 49 kB/s | 16 kB 00:00 2024-04-22T08:09:35.028 INFO:teuthology.orchestra.run.smithi040.stdout:Extra Packages for Enterprise Linux 9.8 MB/s | 21 MB 00:02 2024-04-22T08:09:38.937 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - CRB 757 kB/s | 6.2 MB 00:08 2024-04-22T08:09:40.245 INFO:teuthology.orchestra.run.smithi040.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-04-22T08:09:40.621 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 16 kB 00:00 2024-04-22T08:09:42.464 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-22T08:09:42.466 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:09:42.466 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2024-04-22T08:09:42.467 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:09:42.467 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-04-22T08:09:42.467 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-radosgw x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 7.7 M 2024-04-22T08:09:42.467 INFO:teuthology.orchestra.run.smithi040.stdout:Upgrading: 2024-04-22T08:09:42.467 INFO:teuthology.orchestra.run.smithi040.stdout: librados2 x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 3.3 M 2024-04-22T08:09:42.467 INFO:teuthology.orchestra.run.smithi040.stdout: librbd1 x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 3.0 M 2024-04-22T08:09:42.467 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2024-04-22T08:09:42.467 INFO:teuthology.orchestra.run.smithi040.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-22T08:09:42.467 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-base x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 5.1 M 2024-04-22T08:09:42.467 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-common x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 18 M 2024-04-22T08:09:42.467 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-selinux x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 25 k 2024-04-22T08:09:42.467 INFO:teuthology.orchestra.run.smithi040.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-04-22T08:09:42.467 INFO:teuthology.orchestra.run.smithi040.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-22T08:09:42.468 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-22T08:09:42.468 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-22T08:09:42.468 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs2 x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 709 k 2024-04-22T08:09:42.468 INFO:teuthology.orchestra.run.smithi040.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-22T08:09:42.468 INFO:teuthology.orchestra.run.smithi040.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-22T08:09:42.468 INFO:teuthology.orchestra.run.smithi040.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-22T08:09:42.468 INFO:teuthology.orchestra.run.smithi040.stdout: libradosstriper1 x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 474 k 2024-04-22T08:09:42.468 INFO:teuthology.orchestra.run.smithi040.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-22T08:09:42.468 INFO:teuthology.orchestra.run.smithi040.stdout: librgw2 x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 4.5 M 2024-04-22T08:09:42.468 INFO:teuthology.orchestra.run.smithi040.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-22T08:09:42.468 INFO:teuthology.orchestra.run.smithi040.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-22T08:09:42.468 INFO:teuthology.orchestra.run.smithi040.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-22T08:09:42.468 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-argparse x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 45 k 2024-04-22T08:09:42.469 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-common x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 129 k 2024-04-22T08:09:42.469 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cephfs x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 161 k 2024-04-22T08:09:42.469 INFO:teuthology.orchestra.run.smithi040.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-22T08:09:42.469 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rados x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 320 k 2024-04-22T08:09:42.469 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rbd x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 298 k 2024-04-22T08:09:42.469 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rgw x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 99 k 2024-04-22T08:09:42.469 INFO:teuthology.orchestra.run.smithi040.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-22T08:09:42.469 INFO:teuthology.orchestra.run.smithi040.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-22T08:09:42.469 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:09:42.469 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-04-22T08:09:42.470 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:09:42.470 INFO:teuthology.orchestra.run.smithi040.stdout:Install 28 Packages 2024-04-22T08:09:42.470 INFO:teuthology.orchestra.run.smithi040.stdout:Upgrade 2 Packages 2024-04-22T08:09:42.470 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:09:42.470 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 53 M 2024-04-22T08:09:42.470 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-04-22T08:09:43.705 INFO:teuthology.orchestra.run.smithi040.stdout:(1/30): ceph-base-18.2.2-1237.g646b8c7d.el8.x86 6.2 MB/s | 5.1 MB 00:00 2024-04-22T08:09:43.780 INFO:teuthology.orchestra.run.smithi040.stdout:(2/30): ceph-selinux-18.2.2-1237.g646b8c7d.el8. 336 kB/s | 25 kB 00:00 2024-04-22T08:09:43.938 INFO:teuthology.orchestra.run.smithi040.stdout:(3/30): libcephfs2-18.2.2-1237.g646b8c7d.el8.x8 4.4 MB/s | 709 kB 00:00 2024-04-22T08:09:44.038 INFO:teuthology.orchestra.run.smithi040.stdout:(4/30): libradosstriper1-18.2.2-1237.g646b8c7d. 4.6 MB/s | 474 kB 00:00 2024-04-22T08:09:44.231 INFO:teuthology.orchestra.run.smithi189.stdout:Extra Packages for Enterprise Linux 6.0 MB/s | 21 MB 00:03 2024-04-22T08:09:44.456 INFO:teuthology.orchestra.run.smithi040.stdout:(5/30): ceph-common-18.2.2-1237.g646b8c7d.el8.x 12 MB/s | 18 MB 00:01 2024-04-22T08:09:44.522 INFO:teuthology.orchestra.run.smithi040.stdout:(6/30): librgw2-18.2.2-1237.g646b8c7d.el8.x86_6 9.2 MB/s | 4.5 MB 00:00 2024-04-22T08:09:44.548 INFO:teuthology.orchestra.run.smithi040.stdout:(7/30): python3-ceph-argparse-18.2.2-1237.g646b 490 kB/s | 45 kB 00:00 2024-04-22T08:09:44.573 INFO:teuthology.orchestra.run.smithi040.stdout:(8/30): python3-ceph-common-18.2.2-1237.g646b8c 2.5 MB/s | 129 kB 00:00 2024-04-22T08:09:44.598 INFO:teuthology.orchestra.run.smithi040.stdout:(9/30): python3-cephfs-18.2.2-1237.g646b8c7d.el 3.1 MB/s | 161 kB 00:00 2024-04-22T08:09:44.632 INFO:teuthology.orchestra.run.smithi040.stdout:(10/30): python3-rados-18.2.2-1237.g646b8c7d.el 5.4 MB/s | 320 kB 00:00 2024-04-22T08:09:44.657 INFO:teuthology.orchestra.run.smithi040.stdout:(11/30): python3-rbd-18.2.2-1237.g646b8c7d.el8. 5.0 MB/s | 298 kB 00:00 2024-04-22T08:09:44.682 INFO:teuthology.orchestra.run.smithi040.stdout:(12/30): python3-rgw-18.2.2-1237.g646b8c7d.el8. 1.9 MB/s | 99 kB 00:00 2024-04-22T08:09:44.782 INFO:teuthology.orchestra.run.smithi040.stdout:(13/30): ceph-radosgw-18.2.2-1237.g646b8c7d.el8 4.1 MB/s | 7.7 MB 00:01 2024-04-22T08:09:44.941 INFO:teuthology.orchestra.run.smithi040.stdout:(14/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 286 kB/s | 45 kB 00:00 2024-04-22T08:09:44.975 INFO:teuthology.orchestra.run.smithi040.stdout:(15/30): boost-program-options-1.75.0-8.el9.x86 336 kB/s | 107 kB 00:00 2024-04-22T08:09:45.001 INFO:teuthology.orchestra.run.smithi040.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 503 kB/s | 160 kB 00:00 2024-04-22T08:09:45.059 INFO:teuthology.orchestra.run.smithi040.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 723 kB/s | 42 kB 00:00 2024-04-22T08:09:45.085 INFO:teuthology.orchestra.run.smithi040.stdout:(18/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-04-22T08:09:45.202 INFO:teuthology.orchestra.run.smithi040.stdout:(19/30): librdkafka-1.6.1-102.el9.x86_64.rpm 2.5 MB/s | 662 kB 00:00 2024-04-22T08:09:45.477 INFO:teuthology.orchestra.run.smithi040.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 265 kB/s | 111 kB 00:00 2024-04-22T08:09:45.569 INFO:teuthology.orchestra.run.smithi040.stdout:(21/30): libarrow-doc-9.0.0-10.el9.noarch.rpm 283 kB/s | 26 kB 00:00 2024-04-22T08:09:45.661 INFO:teuthology.orchestra.run.smithi040.stdout:(22/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 537 kB/s | 309 kB 00:00 2024-04-22T08:09:45.686 INFO:teuthology.orchestra.run.smithi040.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 415 kB/s | 49 kB 00:00 2024-04-22T08:09:45.761 INFO:teuthology.orchestra.run.smithi040.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 672 kB/s | 67 kB 00:00 2024-04-22T08:09:45.903 INFO:teuthology.orchestra.run.smithi040.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-04-22T08:09:45.970 INFO:teuthology.orchestra.run.smithi040.stdout:(26/30): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.9 MB/s | 838 kB 00:00 2024-04-22T08:09:46.145 INFO:teuthology.orchestra.run.smithi040.stdout:(27/30): libarrow-9.0.0-10.el9.x86_64.rpm 4.7 MB/s | 4.4 MB 00:00 2024-04-22T08:09:46.287 INFO:teuthology.orchestra.run.smithi040.stdout:(28/30): librados2-18.2.2-1237.g646b8c7d.el8.x8 10 MB/s | 3.3 MB 00:00 2024-04-22T08:09:46.321 INFO:teuthology.orchestra.run.smithi040.stdout:(29/30): thrift-0.15.0-2.el9.x86_64.rpm 3.8 MB/s | 1.6 MB 00:00 2024-04-22T08:09:46.454 INFO:teuthology.orchestra.run.smithi040.stdout:(30/30): librbd1-18.2.2-1237.g646b8c7d.el8.x86_ 9.7 MB/s | 3.0 MB 00:00 2024-04-22T08:09:46.456 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-04-22T08:09:46.457 INFO:teuthology.orchestra.run.smithi040.stdout:Total 13 MB/s | 53 MB 00:03 2024-04-22T08:09:46.531 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-04-22T08:09:46.572 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-04-22T08:09:46.572 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-04-22T08:09:47.014 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-04-22T08:09:47.015 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-04-22T08:09:48.439 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-04-22T08:09:48.522 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-04-22T08:09:48.551 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-04-22T08:09:48.601 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-04-22T08:09:48.987 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-04-22T08:09:49.050 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : librados2-2:18.2.2-1237.g646b8c7d.el8.x86_64 5/32 2024-04-22T08:09:49.143 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librados2-2:18.2.2-1237.g646b8c7d.el8.x86_64 5/32 2024-04-22T08:09:49.230 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rados-2:18.2.2-1237.g646b8c7d.el8.x86_64 6/32 2024-04-22T08:09:49.260 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephfs2-2:18.2.2-1237.g646b8c7d.el8.x86_64 7/32 2024-04-22T08:09:49.323 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libcephfs2-2:18.2.2-1237.g646b8c7d.el8.x86_64 7/32 2024-04-22T08:09:49.351 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-04-22T08:09:49.396 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-ceph-argparse-2:18.2.2-1237.g646b8c7d.el8. 9/32 2024-04-22T08:09:49.463 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cephfs-2:18.2.2-1237.g646b8c7d.el8.x86_64 10/32 2024-04-22T08:09:49.487 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libradosstriper1-2:18.2.2-1237.g646b8c7d.el8.x86_6 11/32 2024-04-22T08:09:49.514 INFO:teuthology.orchestra.run.smithi189.stdout:lab-extras 6.0 kB/s | 1.7 kB 00:00 2024-04-22T08:09:49.545 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1237.g646b8c7d.el8.x86_6 11/32 2024-04-22T08:09:49.654 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-04-22T08:09:49.686 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/32 2024-04-22T08:09:49.726 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-04-22T08:09:49.759 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-04-22T08:09:49.786 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/32 2024-04-22T08:09:49.822 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-04-22T08:09:49.855 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-04-22T08:09:50.170 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-04-22T08:09:50.210 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : librbd1-2:18.2.2-1237.g646b8c7d.el8.x86_64 20/32 2024-04-22T08:09:50.309 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librbd1-2:18.2.2-1237.g646b8c7d.el8.x86_64 20/32 2024-04-22T08:09:50.339 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rbd-2:18.2.2-1237.g646b8c7d.el8.x86_64 21/32 2024-04-22T08:09:50.375 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-04-22T08:09:50.798 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 23/32 2024-04-22T08:09:50.820 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librgw2-2:18.2.2-1237.g646b8c7d.el8.x86_64 24/32 2024-04-22T08:09:50.892 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librgw2-2:18.2.2-1237.g646b8c7d.el8.x86_64 24/32 2024-04-22T08:09:50.971 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rgw-2:18.2.2-1237.g646b8c7d.el8.x86_64 25/32 2024-04-22T08:09:51.006 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-ceph-common-2:18.2.2-1237.g646b8c7d.el8.x8 26/32 2024-04-22T08:09:51.663 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-22T08:09:51.806 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:09:51.806 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repository Size 2024-04-22T08:09:51.806 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:09:51.806 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-22T08:09:51.806 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-radosgw x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 7.7 M 2024-04-22T08:09:51.806 INFO:teuthology.orchestra.run.smithi189.stdout:Upgrading: 2024-04-22T08:09:51.806 INFO:teuthology.orchestra.run.smithi189.stdout: librados2 x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 3.3 M 2024-04-22T08:09:51.807 INFO:teuthology.orchestra.run.smithi189.stdout: librbd1 x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 3.0 M 2024-04-22T08:09:51.807 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-04-22T08:09:51.807 INFO:teuthology.orchestra.run.smithi189.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-22T08:09:51.807 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-base x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 5.1 M 2024-04-22T08:09:51.807 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-common x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 18 M 2024-04-22T08:09:51.807 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-selinux x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 25 k 2024-04-22T08:09:51.807 INFO:teuthology.orchestra.run.smithi189.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-04-22T08:09:51.807 INFO:teuthology.orchestra.run.smithi189.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-22T08:09:51.807 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-22T08:09:51.807 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-22T08:09:51.807 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs2 x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 709 k 2024-04-22T08:09:51.807 INFO:teuthology.orchestra.run.smithi189.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-22T08:09:51.807 INFO:teuthology.orchestra.run.smithi189.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-22T08:09:51.808 INFO:teuthology.orchestra.run.smithi189.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-22T08:09:51.808 INFO:teuthology.orchestra.run.smithi189.stdout: libradosstriper1 x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 474 k 2024-04-22T08:09:51.808 INFO:teuthology.orchestra.run.smithi189.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-22T08:09:51.808 INFO:teuthology.orchestra.run.smithi189.stdout: librgw2 x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 4.5 M 2024-04-22T08:09:51.808 INFO:teuthology.orchestra.run.smithi189.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-22T08:09:51.808 INFO:teuthology.orchestra.run.smithi189.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-22T08:09:51.808 INFO:teuthology.orchestra.run.smithi189.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-22T08:09:51.808 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-argparse x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 45 k 2024-04-22T08:09:51.808 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-common x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 129 k 2024-04-22T08:09:51.808 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cephfs x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 161 k 2024-04-22T08:09:51.808 INFO:teuthology.orchestra.run.smithi189.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-22T08:09:51.808 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rados x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 320 k 2024-04-22T08:09:51.808 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rbd x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 298 k 2024-04-22T08:09:51.808 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rgw x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 99 k 2024-04-22T08:09:51.809 INFO:teuthology.orchestra.run.smithi189.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-22T08:09:51.809 INFO:teuthology.orchestra.run.smithi189.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-22T08:09:51.809 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:09:51.809 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-22T08:09:51.809 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:09:51.809 INFO:teuthology.orchestra.run.smithi189.stdout:Install 28 Packages 2024-04-22T08:09:51.809 INFO:teuthology.orchestra.run.smithi189.stdout:Upgrade 2 Packages 2024-04-22T08:09:51.809 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:09:51.809 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 53 M 2024-04-22T08:09:51.809 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-22T08:09:53.198 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-common-2:18.2.2-1237.g646b8c7d.el8.x86_64 27/32 2024-04-22T08:09:53.229 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-common-2:18.2.2-1237.g646b8c7d.el8.x86_64 27/32 2024-04-22T08:09:53.573 INFO:teuthology.orchestra.run.smithi189.stdout:(1/30): ceph-common-18.2.2-1237.g646b8c7d.el8.x 13 MB/s | 18 MB 00:01 2024-04-22T08:09:53.640 INFO:teuthology.orchestra.run.smithi189.stdout:(2/30): ceph-base-18.2.2-1237.g646b8c7d.el8.x86 3.5 MB/s | 5.1 MB 00:01 2024-04-22T08:09:53.657 INFO:teuthology.orchestra.run.smithi189.stdout:(3/30): ceph-selinux-18.2.2-1237.g646b8c7d.el8. 301 kB/s | 25 kB 00:00 2024-04-22T08:09:53.716 INFO:teuthology.orchestra.run.smithi189.stdout:(4/30): libradosstriper1-18.2.2-1237.g646b8c7d. 7.9 MB/s | 474 kB 00:00 2024-04-22T08:09:53.766 INFO:teuthology.orchestra.run.smithi189.stdout:(5/30): libcephfs2-18.2.2-1237.g646b8c7d.el8.x8 5.5 MB/s | 709 kB 00:00 2024-04-22T08:09:53.780 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-common-2:18.2.2-1237.g646b8c7d.el8.x86_64 27/32 2024-04-22T08:09:53.816 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-base-2:18.2.2-1237.g646b8c7d.el8.x86_64 28/32 2024-04-22T08:09:53.858 INFO:teuthology.orchestra.run.smithi189.stdout:(6/30): python3-ceph-argparse-18.2.2-1237.g646b 492 kB/s | 45 kB 00:00 2024-04-22T08:09:53.912 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-base-2:18.2.2-1237.g646b8c7d.el8.x86_64 28/32 2024-04-22T08:09:53.912 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-22T08:09:53.912 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-22T08:09:53.912 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:09:53.942 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-selinux-2:18.2.2-1237.g646b8c7d.el8.x86_64 29/32 2024-04-22T08:09:53.950 INFO:teuthology.orchestra.run.smithi189.stdout:(7/30): python3-ceph-common-18.2.2-1237.g646b8c 1.4 MB/s | 129 kB 00:00 2024-04-22T08:09:54.017 INFO:teuthology.orchestra.run.smithi189.stdout:(8/30): librgw2-18.2.2-1237.g646b8c7d.el8.x86_6 15 MB/s | 4.5 MB 00:00 2024-04-22T08:09:54.042 INFO:teuthology.orchestra.run.smithi189.stdout:(9/30): python3-cephfs-18.2.2-1237.g646b8c7d.el 1.7 MB/s | 161 kB 00:00 2024-04-22T08:09:54.067 INFO:teuthology.orchestra.run.smithi189.stdout:(10/30): python3-rados-18.2.2-1237.g646b8c7d.el 6.2 MB/s | 320 kB 00:00 2024-04-22T08:09:54.159 INFO:teuthology.orchestra.run.smithi189.stdout:(11/30): ceph-radosgw-18.2.2-1237.g646b8c7d.el8 3.9 MB/s | 7.7 MB 00:01 2024-04-22T08:09:54.185 INFO:teuthology.orchestra.run.smithi189.stdout:(12/30): python3-rgw-18.2.2-1237.g646b8c7d.el8. 850 kB/s | 99 kB 00:00 2024-04-22T08:09:54.210 INFO:teuthology.orchestra.run.smithi189.stdout:(13/30): python3-rbd-18.2.2-1237.g646b8c7d.el8. 1.7 MB/s | 298 kB 00:00 2024-04-22T08:09:54.418 INFO:teuthology.orchestra.run.smithi189.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 413 kB/s | 107 kB 00:00 2024-04-22T08:09:54.444 INFO:teuthology.orchestra.run.smithi189.stdout:(15/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 194 kB/s | 45 kB 00:00 2024-04-22T08:09:54.469 INFO:teuthology.orchestra.run.smithi189.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 564 kB/s | 160 kB 00:00 2024-04-22T08:09:54.544 INFO:teuthology.orchestra.run.smithi189.stdout:(17/30): librdkafka-1.6.1-102.el9.x86_64.rpm 5.2 MB/s | 662 kB 00:00 2024-04-22T08:09:54.570 INFO:teuthology.orchestra.run.smithi189.stdout:(18/30): python3-prettytable-0.7.2-27.el9.noarc 422 kB/s | 42 kB 00:00 2024-04-22T08:09:54.595 INFO:teuthology.orchestra.run.smithi189.stdout:(19/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.9 MB/s | 292 kB 00:00 2024-04-22T08:09:54.687 INFO:teuthology.orchestra.run.smithi189.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 779 kB/s | 111 kB 00:00 2024-04-22T08:09:54.729 INFO:teuthology.orchestra.run.smithi189.stdout:(21/30): libarrow-doc-9.0.0-10.el9.noarch.rpm 625 kB/s | 26 kB 00:00 2024-04-22T08:09:54.754 INFO:teuthology.orchestra.run.smithi189.stdout:(22/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.6 MB/s | 309 kB 00:00 2024-04-22T08:09:54.779 INFO:teuthology.orchestra.run.smithi189.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 967 kB/s | 49 kB 00:00 2024-04-22T08:09:54.804 INFO:teuthology.orchestra.run.smithi189.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-04-22T08:09:54.863 INFO:teuthology.orchestra.run.smithi189.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-04-22T08:09:55.038 INFO:teuthology.orchestra.run.smithi189.stdout:(26/30): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.2 MB/s | 838 kB 00:00 2024-04-22T08:09:55.147 INFO:teuthology.orchestra.run.smithi189.stdout:(27/30): thrift-0.15.0-2.el9.x86_64.rpm 5.6 MB/s | 1.6 MB 00:00 2024-04-22T08:09:55.397 INFO:teuthology.orchestra.run.smithi189.stdout:(28/30): librbd1-18.2.2-1237.g646b8c7d.el8.x86_ 12 MB/s | 3.0 MB 00:00 2024-04-22T08:09:55.539 INFO:teuthology.orchestra.run.smithi189.stdout:(29/30): libarrow-9.0.0-10.el9.x86_64.rpm 4.7 MB/s | 4.4 MB 00:00 2024-04-22T08:09:55.614 INFO:teuthology.orchestra.run.smithi189.stdout:(30/30): librados2-18.2.2-1237.g646b8c7d.el8.x8 5.7 MB/s | 3.3 MB 00:00 2024-04-22T08:09:55.620 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-22T08:09:55.620 INFO:teuthology.orchestra.run.smithi189.stdout:Total 13 MB/s | 53 MB 00:03 2024-04-22T08:09:55.692 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-22T08:09:55.729 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-22T08:09:55.729 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-22T08:09:56.155 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-22T08:09:56.156 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-22T08:09:57.535 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-22T08:09:57.678 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-04-22T08:09:57.712 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-04-22T08:09:57.761 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-04-22T08:09:58.074 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-04-22T08:09:58.177 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : librados2-2:18.2.2-1237.g646b8c7d.el8.x86_64 5/32 2024-04-22T08:09:58.269 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librados2-2:18.2.2-1237.g646b8c7d.el8.x86_64 5/32 2024-04-22T08:09:58.353 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rados-2:18.2.2-1237.g646b8c7d.el8.x86_64 6/32 2024-04-22T08:09:58.387 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libcephfs2-2:18.2.2-1237.g646b8c7d.el8.x86_64 7/32 2024-04-22T08:09:58.450 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libcephfs2-2:18.2.2-1237.g646b8c7d.el8.x86_64 7/32 2024-04-22T08:09:58.478 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-04-22T08:09:58.521 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-ceph-argparse-2:18.2.2-1237.g646b8c7d.el8. 9/32 2024-04-22T08:09:58.588 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cephfs-2:18.2.2-1237.g646b8c7d.el8.x86_64 10/32 2024-04-22T08:09:58.614 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libradosstriper1-2:18.2.2-1237.g646b8c7d.el8.x86_6 11/32 2024-04-22T08:09:58.680 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1237.g646b8c7d.el8.x86_6 11/32 2024-04-22T08:09:58.787 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-04-22T08:09:58.821 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/32 2024-04-22T08:09:58.862 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-04-22T08:09:58.895 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-04-22T08:09:58.922 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/32 2024-04-22T08:09:58.958 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-04-22T08:09:58.991 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-04-22T08:09:59.289 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-04-22T08:09:59.337 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : librbd1-2:18.2.2-1237.g646b8c7d.el8.x86_64 20/32 2024-04-22T08:09:59.427 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librbd1-2:18.2.2-1237.g646b8c7d.el8.x86_64 20/32 2024-04-22T08:09:59.458 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rbd-2:18.2.2-1237.g646b8c7d.el8.x86_64 21/32 2024-04-22T08:09:59.494 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-04-22T08:09:59.897 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 23/32 2024-04-22T08:09:59.923 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librgw2-2:18.2.2-1237.g646b8c7d.el8.x86_64 24/32 2024-04-22T08:09:59.986 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librgw2-2:18.2.2-1237.g646b8c7d.el8.x86_64 24/32 2024-04-22T08:10:00.068 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rgw-2:18.2.2-1237.g646b8c7d.el8.x86_64 25/32 2024-04-22T08:10:00.094 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-ceph-common-2:18.2.2-1237.g646b8c7d.el8.x8 26/32 2024-04-22T08:10:02.157 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-common-2:18.2.2-1237.g646b8c7d.el8.x86_64 27/32 2024-04-22T08:10:02.265 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-common-2:18.2.2-1237.g646b8c7d.el8.x86_64 27/32 2024-04-22T08:10:02.791 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-common-2:18.2.2-1237.g646b8c7d.el8.x86_64 27/32 2024-04-22T08:10:02.836 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-base-2:18.2.2-1237.g646b8c7d.el8.x86_64 28/32 2024-04-22T08:10:02.926 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-base-2:18.2.2-1237.g646b8c7d.el8.x86_64 28/32 2024-04-22T08:10:02.927 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-22T08:10:02.927 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-22T08:10:02.927 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:10:02.954 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-selinux-2:18.2.2-1237.g646b8c7d.el8.x86_64 29/32 2024-04-22T08:10:03.898 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1237.g646b8c7d.el8.x86_64 29/32 2024-04-22T08:10:03.898 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /sys 2024-04-22T08:10:03.898 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /proc 2024-04-22T08:10:03.898 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /mnt 2024-04-22T08:10:03.898 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /var/tmp 2024-04-22T08:10:03.898 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /home 2024-04-22T08:10:03.898 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /root 2024-04-22T08:10:03.899 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /tmp 2024-04-22T08:10:03.899 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:10:04.666 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-radosgw-2:18.2.2-1237.g646b8c7d.el8.x86_64 30/32 2024-04-22T08:10:04.692 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1237.g646b8c7d.el8.x86_64 30/32 2024-04-22T08:10:04.692 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T08:10:04.692 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-22T08:10:04.692 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T08:10:04.692 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T08:10:04.692 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:10:04.694 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-22T08:10:04.778 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-22T08:10:04.778 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-22T08:10:05.935 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-22T08:10:05.935 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-base-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/32 2024-04-22T08:10:05.935 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-common-2:18.2.2-1237.g646b8c7d.el8.x86_64 2/32 2024-04-22T08:10:05.935 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-radosgw-2:18.2.2-1237.g646b8c7d.el8.x86_64 3/32 2024-04-22T08:10:05.935 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-selinux-2:18.2.2-1237.g646b8c7d.el8.x86_64 4/32 2024-04-22T08:10:05.935 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephfs2-2:18.2.2-1237.g646b8c7d.el8.x86_64 5/32 2024-04-22T08:10:05.935 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libradosstriper1-2:18.2.2-1237.g646b8c7d.el8.x86_6 6/32 2024-04-22T08:10:05.935 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librgw2-2:18.2.2-1237.g646b8c7d.el8.x86_64 7/32 2024-04-22T08:10:05.936 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1237.g646b8c7d.el8. 8/32 2024-04-22T08:10:05.936 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-ceph-common-2:18.2.2-1237.g646b8c7d.el8.x8 9/32 2024-04-22T08:10:05.936 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cephfs-2:18.2.2-1237.g646b8c7d.el8.x86_64 10/32 2024-04-22T08:10:05.936 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rados-2:18.2.2-1237.g646b8c7d.el8.x86_64 11/32 2024-04-22T08:10:05.936 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rbd-2:18.2.2-1237.g646b8c7d.el8.x86_64 12/32 2024-04-22T08:10:05.936 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rgw-2:18.2.2-1237.g646b8c7d.el8.x86_64 13/32 2024-04-22T08:10:05.936 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-04-22T08:10:05.936 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-04-22T08:10:05.936 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-04-22T08:10:05.936 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-04-22T08:10:05.936 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-04-22T08:10:05.936 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-04-22T08:10:05.936 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-04-22T08:10:05.936 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-04-22T08:10:05.937 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 22/32 2024-04-22T08:10:05.937 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 23/32 2024-04-22T08:10:05.937 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-04-22T08:10:05.937 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-04-22T08:10:05.937 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 26/32 2024-04-22T08:10:05.937 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-04-22T08:10:05.937 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-04-22T08:10:05.937 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados2-2:18.2.2-1237.g646b8c7d.el8.x86_64 29/32 2024-04-22T08:10:05.937 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-04-22T08:10:05.937 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librbd1-2:18.2.2-1237.g646b8c7d.el8.x86_64 31/32 2024-04-22T08:10:06.795 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-04-22T08:10:06.795 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:10:06.795 INFO:teuthology.orchestra.run.smithi040.stdout:Upgraded: 2024-04-22T08:10:06.795 INFO:teuthology.orchestra.run.smithi040.stdout: librados2-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:10:06.795 INFO:teuthology.orchestra.run.smithi040.stdout: librbd1-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:10:06.795 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-04-22T08:10:06.795 INFO:teuthology.orchestra.run.smithi040.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-22T08:10:06.795 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-base-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:10:06.795 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-common-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:10:06.795 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-radosgw-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:10:06.795 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-selinux-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:10:06.796 INFO:teuthology.orchestra.run.smithi040.stdout: fmt-8.1.1-5.el9.x86_64 2024-04-22T08:10:06.796 INFO:teuthology.orchestra.run.smithi040.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-22T08:10:06.796 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-22T08:10:06.796 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-22T08:10:06.796 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs2-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:10:06.796 INFO:teuthology.orchestra.run.smithi040.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-22T08:10:06.796 INFO:teuthology.orchestra.run.smithi040.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-22T08:10:06.796 INFO:teuthology.orchestra.run.smithi040.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-22T08:10:06.796 INFO:teuthology.orchestra.run.smithi040.stdout: libradosstriper1-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:10:06.796 INFO:teuthology.orchestra.run.smithi040.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-22T08:10:06.796 INFO:teuthology.orchestra.run.smithi040.stdout: librgw2-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:10:06.796 INFO:teuthology.orchestra.run.smithi040.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-22T08:10:06.796 INFO:teuthology.orchestra.run.smithi040.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-22T08:10:06.796 INFO:teuthology.orchestra.run.smithi040.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-22T08:10:06.797 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-argparse-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:10:06.797 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-common-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:10:06.797 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cephfs-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:10:06.797 INFO:teuthology.orchestra.run.smithi040.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-22T08:10:06.797 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rados-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:10:06.797 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rbd-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:10:06.797 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rgw-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:10:06.797 INFO:teuthology.orchestra.run.smithi040.stdout: re2-1:20211101-3.el9.x86_64 2024-04-22T08:10:06.797 INFO:teuthology.orchestra.run.smithi040.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-22T08:10:06.797 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:10:06.797 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-22T08:10:07.093 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-test 2024-04-22T08:10:07.636 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:00:27 ago on Mon 22 Apr 2024 08:09:40 AM UTC. 2024-04-22T08:10:07.761 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-22T08:10:07.762 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:10:07.762 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2024-04-22T08:10:07.762 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:10:07.763 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-04-22T08:10:07.763 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-test x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 40 M 2024-04-22T08:10:07.763 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2024-04-22T08:10:07.763 INFO:teuthology.orchestra.run.smithi040.stdout: libcephsqlite x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 166 k 2024-04-22T08:10:07.763 INFO:teuthology.orchestra.run.smithi040.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-22T08:10:07.763 INFO:teuthology.orchestra.run.smithi040.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-22T08:10:07.763 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:10:07.763 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-04-22T08:10:07.763 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:10:07.763 INFO:teuthology.orchestra.run.smithi040.stdout:Install 4 Packages 2024-04-22T08:10:07.763 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:10:07.764 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 40 M 2024-04-22T08:10:07.764 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 163 M 2024-04-22T08:10:07.764 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-04-22T08:10:08.207 INFO:teuthology.orchestra.run.smithi040.stdout:(1/4): socat-1.7.4.1-5.el9.x86_64.rpm 1.8 MB/s | 305 kB 00:00 2024-04-22T08:10:08.256 INFO:teuthology.orchestra.run.smithi040.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.3 MB/s | 64 kB 00:00 2024-04-22T08:10:08.315 INFO:teuthology.orchestra.run.smithi040.stdout:(3/4): libcephsqlite-18.2.2-1237.g646b8c7d.el8. 611 kB/s | 166 kB 00:00 2024-04-22T08:10:10.475 INFO:teuthology.orchestra.run.smithi040.stdout:(4/4): ceph-test-18.2.2-1237.g646b8c7d.el8.x86_ 16 MB/s | 40 MB 00:02 2024-04-22T08:10:10.477 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-04-22T08:10:10.477 INFO:teuthology.orchestra.run.smithi040.stdout:Total 15 MB/s | 40 MB 00:02 2024-04-22T08:10:10.504 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-04-22T08:10:10.526 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-04-22T08:10:10.527 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-04-22T08:10:10.859 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-04-22T08:10:10.859 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-04-22T08:10:11.348 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-04-22T08:10:11.434 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-04-22T08:10:11.474 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-04-22T08:10:11.502 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephsqlite-2:18.2.2-1237.g646b8c7d.el8.x86_64 3/4 2024-04-22T08:10:12.349 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1237.g646b8c7d.el8.x86_64 29/32 2024-04-22T08:10:12.349 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /sys 2024-04-22T08:10:12.349 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /proc 2024-04-22T08:10:12.349 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /mnt 2024-04-22T08:10:12.349 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /var/tmp 2024-04-22T08:10:12.350 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /home 2024-04-22T08:10:12.350 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /root 2024-04-22T08:10:12.350 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /tmp 2024-04-22T08:10:12.350 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:10:13.085 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-radosgw-2:18.2.2-1237.g646b8c7d.el8.x86_64 30/32 2024-04-22T08:10:13.109 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1237.g646b8c7d.el8.x86_64 30/32 2024-04-22T08:10:13.109 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T08:10:13.109 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-22T08:10:13.109 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T08:10:13.109 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T08:10:13.109 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:10:13.111 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-22T08:10:13.187 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-22T08:10:13.187 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-22T08:10:14.368 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-22T08:10:14.368 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-base-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/32 2024-04-22T08:10:14.368 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-common-2:18.2.2-1237.g646b8c7d.el8.x86_64 2/32 2024-04-22T08:10:14.368 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-radosgw-2:18.2.2-1237.g646b8c7d.el8.x86_64 3/32 2024-04-22T08:10:14.368 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-selinux-2:18.2.2-1237.g646b8c7d.el8.x86_64 4/32 2024-04-22T08:10:14.368 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libcephfs2-2:18.2.2-1237.g646b8c7d.el8.x86_64 5/32 2024-04-22T08:10:14.368 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libradosstriper1-2:18.2.2-1237.g646b8c7d.el8.x86_6 6/32 2024-04-22T08:10:14.368 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librgw2-2:18.2.2-1237.g646b8c7d.el8.x86_64 7/32 2024-04-22T08:10:14.369 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1237.g646b8c7d.el8. 8/32 2024-04-22T08:10:14.369 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-ceph-common-2:18.2.2-1237.g646b8c7d.el8.x8 9/32 2024-04-22T08:10:14.369 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cephfs-2:18.2.2-1237.g646b8c7d.el8.x86_64 10/32 2024-04-22T08:10:14.369 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rados-2:18.2.2-1237.g646b8c7d.el8.x86_64 11/32 2024-04-22T08:10:14.369 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rbd-2:18.2.2-1237.g646b8c7d.el8.x86_64 12/32 2024-04-22T08:10:14.369 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rgw-2:18.2.2-1237.g646b8c7d.el8.x86_64 13/32 2024-04-22T08:10:14.369 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-04-22T08:10:14.369 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-04-22T08:10:14.369 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-04-22T08:10:14.369 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-04-22T08:10:14.369 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-04-22T08:10:14.369 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-04-22T08:10:14.369 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-04-22T08:10:14.369 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-04-22T08:10:14.370 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 22/32 2024-04-22T08:10:14.370 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 23/32 2024-04-22T08:10:14.370 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-04-22T08:10:14.370 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-04-22T08:10:14.370 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 26/32 2024-04-22T08:10:14.370 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-04-22T08:10:14.370 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-04-22T08:10:14.370 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librados2-2:18.2.2-1237.g646b8c7d.el8.x86_64 29/32 2024-04-22T08:10:14.370 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-04-22T08:10:14.370 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librbd1-2:18.2.2-1237.g646b8c7d.el8.x86_64 31/32 2024-04-22T08:10:15.057 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-04-22T08:10:15.057 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:10:15.057 INFO:teuthology.orchestra.run.smithi189.stdout:Upgraded: 2024-04-22T08:10:15.057 INFO:teuthology.orchestra.run.smithi189.stdout: librados2-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:10:15.057 INFO:teuthology.orchestra.run.smithi189.stdout: librbd1-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:10:15.057 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-22T08:10:15.057 INFO:teuthology.orchestra.run.smithi189.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-22T08:10:15.057 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-base-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:10:15.057 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-common-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:10:15.057 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-radosgw-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:10:15.057 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-selinux-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:10:15.058 INFO:teuthology.orchestra.run.smithi189.stdout: fmt-8.1.1-5.el9.x86_64 2024-04-22T08:10:15.058 INFO:teuthology.orchestra.run.smithi189.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-22T08:10:15.058 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-22T08:10:15.058 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-22T08:10:15.058 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs2-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:10:15.058 INFO:teuthology.orchestra.run.smithi189.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-22T08:10:15.058 INFO:teuthology.orchestra.run.smithi189.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-22T08:10:15.058 INFO:teuthology.orchestra.run.smithi189.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-22T08:10:15.058 INFO:teuthology.orchestra.run.smithi189.stdout: libradosstriper1-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:10:15.058 INFO:teuthology.orchestra.run.smithi189.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-22T08:10:15.058 INFO:teuthology.orchestra.run.smithi189.stdout: librgw2-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:10:15.058 INFO:teuthology.orchestra.run.smithi189.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-22T08:10:15.058 INFO:teuthology.orchestra.run.smithi189.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-22T08:10:15.059 INFO:teuthology.orchestra.run.smithi189.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-22T08:10:15.059 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-argparse-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:10:15.059 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-common-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:10:15.059 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cephfs-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:10:15.059 INFO:teuthology.orchestra.run.smithi189.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-22T08:10:15.059 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rados-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:10:15.059 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rbd-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:10:15.059 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rgw-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:10:15.059 INFO:teuthology.orchestra.run.smithi189.stdout: re2-1:20211101-3.el9.x86_64 2024-04-22T08:10:15.059 INFO:teuthology.orchestra.run.smithi189.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-22T08:10:15.059 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:10:15.059 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-22T08:10:15.338 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-test 2024-04-22T08:10:15.389 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1237.g646b8c7d.el8.x86_64 3/4 2024-04-22T08:10:15.467 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-test-2:18.2.2-1237.g646b8c7d.el8.x86_64 4/4 2024-04-22T08:10:15.876 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:00:26 ago on Mon 22 Apr 2024 08:09:49 AM UTC. 2024-04-22T08:10:15.997 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-22T08:10:15.998 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:10:15.998 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repository Size 2024-04-22T08:10:15.998 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:10:15.998 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-22T08:10:15.998 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-test x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 40 M 2024-04-22T08:10:15.998 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-04-22T08:10:15.998 INFO:teuthology.orchestra.run.smithi189.stdout: libcephsqlite x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 166 k 2024-04-22T08:10:15.998 INFO:teuthology.orchestra.run.smithi189.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-22T08:10:15.998 INFO:teuthology.orchestra.run.smithi189.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-22T08:10:15.999 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:10:15.999 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-22T08:10:15.999 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:10:15.999 INFO:teuthology.orchestra.run.smithi189.stdout:Install 4 Packages 2024-04-22T08:10:15.999 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:10:15.999 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 40 M 2024-04-22T08:10:15.999 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 163 M 2024-04-22T08:10:16.000 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-22T08:10:16.369 INFO:teuthology.orchestra.run.smithi189.stdout:(1/4): socat-1.7.4.1-5.el9.x86_64.rpm 1.3 MB/s | 305 kB 00:00 2024-04-22T08:10:16.402 INFO:teuthology.orchestra.run.smithi189.stdout:(2/4): libcephsqlite-18.2.2-1237.g646b8c7d.el8. 649 kB/s | 166 kB 00:00 2024-04-22T08:10:16.428 INFO:teuthology.orchestra.run.smithi189.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-04-22T08:10:17.497 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-test-2:18.2.2-1237.g646b8c7d.el8.x86_64 4/4 2024-04-22T08:10:17.497 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-test-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/4 2024-04-22T08:10:17.497 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephsqlite-2:18.2.2-1237.g646b8c7d.el8.x86_64 2/4 2024-04-22T08:10:17.498 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-04-22T08:10:17.771 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-04-22T08:10:17.771 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:10:17.771 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-04-22T08:10:17.771 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-test-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:10:17.771 INFO:teuthology.orchestra.run.smithi040.stdout: libcephsqlite-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:10:17.771 INFO:teuthology.orchestra.run.smithi040.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-22T08:10:17.771 INFO:teuthology.orchestra.run.smithi040.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-22T08:10:17.771 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:10:17.771 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-22T08:10:18.020 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph 2024-04-22T08:10:18.182 INFO:teuthology.orchestra.run.smithi189.stdout:(4/4): ceph-test-18.2.2-1237.g646b8c7d.el8.x86_ 20 MB/s | 40 MB 00:02 2024-04-22T08:10:18.185 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-22T08:10:18.185 INFO:teuthology.orchestra.run.smithi189.stdout:Total 18 MB/s | 40 MB 00:02 2024-04-22T08:10:18.211 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-22T08:10:18.231 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-22T08:10:18.231 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-22T08:10:18.551 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-22T08:10:18.551 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-22T08:10:18.553 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:00:38 ago on Mon 22 Apr 2024 08:09:40 AM UTC. 2024-04-22T08:10:18.681 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-22T08:10:18.684 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:10:18.684 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2024-04-22T08:10:18.684 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:10:18.684 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-04-22T08:10:18.684 INFO:teuthology.orchestra.run.smithi040.stdout: ceph x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 6.5 k 2024-04-22T08:10:18.684 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2024-04-22T08:10:18.685 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mds x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 2.1 M 2024-04-22T08:10:18.685 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 1.5 M 2024-04-22T08:10:18.685 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1237.g646b8c7d.el8 ceph-noarch 246 k 2024-04-22T08:10:18.685 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mon x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 4.7 M 2024-04-22T08:10:18.685 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-osd x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 17 M 2024-04-22T08:10:18.685 INFO:teuthology.orchestra.run.smithi040.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-22T08:10:18.685 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-22T08:10:18.686 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-22T08:10:18.686 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-22T08:10:18.686 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-22T08:10:18.686 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-22T08:10:18.686 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-22T08:10:18.686 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-collections 2024-04-22T08:10:18.686 INFO:teuthology.orchestra.run.smithi040.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-04-22T08:10:18.686 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-22T08:10:18.686 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-22T08:10:18.686 INFO:teuthology.orchestra.run.smithi040.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-22T08:10:18.687 INFO:teuthology.orchestra.run.smithi040.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-22T08:10:18.687 INFO:teuthology.orchestra.run.smithi040.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-22T08:10:18.687 INFO:teuthology.orchestra.run.smithi040.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-22T08:10:18.687 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-22T08:10:18.687 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-22T08:10:18.687 INFO:teuthology.orchestra.run.smithi040.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-22T08:10:18.687 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-22T08:10:18.687 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-22T08:10:18.687 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-22T08:10:18.688 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-22T08:10:18.688 INFO:teuthology.orchestra.run.smithi040.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-22T08:10:18.688 INFO:teuthology.orchestra.run.smithi040.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-22T08:10:18.688 INFO:teuthology.orchestra.run.smithi040.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-22T08:10:18.688 INFO:teuthology.orchestra.run.smithi040.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-22T08:10:18.688 INFO:teuthology.orchestra.run.smithi040.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-22T08:10:18.688 INFO:teuthology.orchestra.run.smithi040.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-22T08:10:18.688 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:10:18.688 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-04-22T08:10:18.688 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:10:18.688 INFO:teuthology.orchestra.run.smithi040.stdout:Install 33 Packages 2024-04-22T08:10:18.688 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:10:18.689 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 29 M 2024-04-22T08:10:18.689 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 104 M 2024-04-22T08:10:18.689 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-04-22T08:10:19.020 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-22T08:10:19.091 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-04-22T08:10:19.131 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-04-22T08:10:19.159 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libcephsqlite-2:18.2.2-1237.g646b8c7d.el8.x86_64 3/4 2024-04-22T08:10:19.368 INFO:teuthology.orchestra.run.smithi040.stdout:(1/33): ceph-18.2.2-1237.g646b8c7d.el8.x86_64.r 34 kB/s | 6.5 kB 00:00 2024-04-22T08:10:19.576 INFO:teuthology.orchestra.run.smithi040.stdout:(2/33): ceph-mds-18.2.2-1237.g646b8c7d.el8.x86_ 5.2 MB/s | 2.1 MB 00:00 2024-04-22T08:10:19.619 INFO:teuthology.orchestra.run.smithi040.stdout:(3/33): ceph-mgr-18.2.2-1237.g646b8c7d.el8.x86_ 3.3 MB/s | 1.5 MB 00:00 2024-04-22T08:10:19.710 INFO:teuthology.orchestra.run.smithi040.stdout:(4/33): ceph-mgr-modules-core-18.2.2-1237.g646b 2.6 MB/s | 246 kB 00:00 2024-04-22T08:10:20.452 INFO:teuthology.orchestra.run.smithi040.stdout:(5/33): ceph-osd-18.2.2-1237.g646b8c7d.el8.x86_ 19 MB/s | 17 MB 00:00 2024-04-22T08:10:20.544 INFO:teuthology.orchestra.run.smithi040.stdout:(6/33): ceph-mon-18.2.2-1237.g646b8c7d.el8.x86_ 4.0 MB/s | 4.7 MB 00:01 2024-04-22T08:10:20.653 INFO:teuthology.orchestra.run.smithi040.stdout:(7/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 269 kB/s | 253 kB 00:00 2024-04-22T08:10:21.336 INFO:teuthology.orchestra.run.smithi040.stdout:(8/33): python3-ply-3.11-14.el9.noarch.rpm 134 kB/s | 106 kB 00:00 2024-04-22T08:10:21.428 INFO:teuthology.orchestra.run.smithi040.stdout:(9/33): python3-pycparser-2.20-6.el9.noarch.rpm 174 kB/s | 135 kB 00:00 2024-04-22T08:10:22.136 INFO:teuthology.orchestra.run.smithi040.stdout:(10/33): python3-requests-2.25.1-8.el9.noarch.r 156 kB/s | 125 kB 00:00 2024-04-22T08:10:22.312 INFO:teuthology.orchestra.run.smithi040.stdout:(11/33): python3-urllib3-1.26.5-5.el9.noarch.rp 243 kB/s | 215 kB 00:00 2024-04-22T08:10:22.337 INFO:teuthology.orchestra.run.smithi040.stdout:(12/33): python3-mako-1.1.4-6.el9.noarch.rpm 862 kB/s | 172 kB 00:00 2024-04-22T08:10:22.387 INFO:teuthology.orchestra.run.smithi040.stdout:(13/33): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-04-22T08:10:22.437 INFO:teuthology.orchestra.run.smithi040.stdout:(14/33): python3-markupsafe-1.1.1-12.el9.x86_64 277 kB/s | 35 kB 00:00 2024-04-22T08:10:22.463 INFO:teuthology.orchestra.run.smithi040.stdout:(15/33): python3-toml-0.10.2-6.el9.noarch.rpm 554 kB/s | 42 kB 00:00 2024-04-22T08:10:22.571 INFO:teuthology.orchestra.run.smithi040.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 325 kB/s | 43 kB 00:00 2024-04-22T08:10:22.613 INFO:teuthology.orchestra.run.smithi040.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-04-22T08:10:22.655 INFO:teuthology.orchestra.run.smithi040.stdout:(18/33): python3-cryptography-36.0.1-4.el9.x86_ 570 kB/s | 1.2 MB 00:02 2024-04-22T08:10:22.684 INFO:teuthology.orchestra.run.smithi040.stdout:(19/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 160 kB/s | 11 kB 00:00 2024-04-22T08:10:22.706 INFO:teuthology.orchestra.run.smithi040.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 355 kB/s | 18 kB 00:00 2024-04-22T08:10:22.731 INFO:teuthology.orchestra.run.smithi040.stdout:(21/33): python3-cherrypy-18.6.1-2.el9.noarch.r 2.2 MB/s | 358 kB 00:00 2024-04-22T08:10:22.756 INFO:teuthology.orchestra.run.smithi040.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 386 kB/s | 19 kB 00:00 2024-04-22T08:10:22.781 INFO:teuthology.orchestra.run.smithi040.stdout:(23/33): python3-jaraco-collections-3.0.0-8.el9 230 kB/s | 23 kB 00:00 2024-04-22T08:10:22.807 INFO:teuthology.orchestra.run.smithi040.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 260 kB/s | 20 kB 00:00 2024-04-22T08:10:22.832 INFO:teuthology.orchestra.run.smithi040.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 614 kB/s | 46 kB 00:00 2024-04-22T08:10:22.836 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1237.g646b8c7d.el8.x86_64 3/4 2024-04-22T08:10:22.857 INFO:teuthology.orchestra.run.smithi040.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-04-22T08:10:22.882 INFO:teuthology.orchestra.run.smithi040.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 328 kB/s | 16 kB 00:00 2024-04-22T08:10:22.906 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-test-2:18.2.2-1237.g646b8c7d.el8.x86_64 4/4 2024-04-22T08:10:22.909 INFO:teuthology.orchestra.run.smithi040.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-04-22T08:10:22.933 INFO:teuthology.orchestra.run.smithi040.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 712 kB/s | 36 kB 00:00 2024-04-22T08:10:22.958 INFO:teuthology.orchestra.run.smithi040.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-04-22T08:10:22.992 INFO:teuthology.orchestra.run.smithi040.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-04-22T08:10:23.018 INFO:teuthology.orchestra.run.smithi040.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 341 kB/s | 20 kB 00:00 2024-04-22T08:10:23.068 INFO:teuthology.orchestra.run.smithi040.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.1 MB/s | 427 kB 00:00 2024-04-22T08:10:23.070 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-04-22T08:10:23.071 INFO:teuthology.orchestra.run.smithi040.stdout:Total 6.6 MB/s | 29 MB 00:04 2024-04-22T08:10:23.192 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-04-22T08:10:23.234 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-04-22T08:10:23.234 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-04-22T08:10:23.533 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-04-22T08:10:23.534 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-04-22T08:10:24.700 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-04-22T08:10:24.762 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-04-22T08:10:24.795 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-04-22T08:10:24.828 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-04-22T08:10:24.926 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-04-22T08:10:25.007 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-test-2:18.2.2-1237.g646b8c7d.el8.x86_64 4/4 2024-04-22T08:10:25.007 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-test-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/4 2024-04-22T08:10:25.007 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libcephsqlite-2:18.2.2-1237.g646b8c7d.el8.x86_64 2/4 2024-04-22T08:10:25.008 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-04-22T08:10:25.027 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-04-22T08:10:25.082 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-04-22T08:10:25.116 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-04-22T08:10:25.145 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-04-22T08:10:25.180 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-04-22T08:10:25.213 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-04-22T08:10:25.254 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-04-22T08:10:25.293 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-04-22T08:10:25.293 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:10:25.293 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-22T08:10:25.293 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-test-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:10:25.293 INFO:teuthology.orchestra.run.smithi189.stdout: libcephsqlite-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:10:25.293 INFO:teuthology.orchestra.run.smithi189.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-22T08:10:25.293 INFO:teuthology.orchestra.run.smithi189.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-22T08:10:25.293 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:10:25.294 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-22T08:10:25.298 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-04-22T08:10:25.331 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-04-22T08:10:25.370 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-04-22T08:10:25.449 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-04-22T08:10:25.547 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph 2024-04-22T08:10:25.601 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-04-22T08:10:25.691 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-04-22T08:10:25.750 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-04-22T08:10:25.787 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-04-22T08:10:25.846 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-04-22T08:10:25.910 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-04-22T08:10:26.063 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:00:37 ago on Mon 22 Apr 2024 08:09:49 AM UTC. 2024-04-22T08:10:26.109 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-04-22T08:10:26.158 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-04-22T08:10:26.189 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-22T08:10:26.192 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:10:26.192 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repository Size 2024-04-22T08:10:26.192 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:10:26.192 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-22T08:10:26.192 INFO:teuthology.orchestra.run.smithi189.stdout: ceph x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 6.5 k 2024-04-22T08:10:26.192 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-04-22T08:10:26.192 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mds x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 2.1 M 2024-04-22T08:10:26.192 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 1.5 M 2024-04-22T08:10:26.192 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1237.g646b8c7d.el8 ceph-noarch 246 k 2024-04-22T08:10:26.192 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mon x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 4.7 M 2024-04-22T08:10:26.193 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-osd x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 17 M 2024-04-22T08:10:26.193 INFO:teuthology.orchestra.run.smithi189.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-22T08:10:26.193 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-22T08:10:26.193 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-22T08:10:26.193 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-22T08:10:26.193 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-22T08:10:26.193 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-22T08:10:26.193 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-22T08:10:26.193 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-collections 2024-04-22T08:10:26.193 INFO:teuthology.orchestra.run.smithi189.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-04-22T08:10:26.193 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-22T08:10:26.193 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-22T08:10:26.194 INFO:teuthology.orchestra.run.smithi189.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-22T08:10:26.194 INFO:teuthology.orchestra.run.smithi189.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-22T08:10:26.194 INFO:teuthology.orchestra.run.smithi189.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-22T08:10:26.194 INFO:teuthology.orchestra.run.smithi189.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-22T08:10:26.194 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-22T08:10:26.194 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-22T08:10:26.194 INFO:teuthology.orchestra.run.smithi189.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-22T08:10:26.194 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-22T08:10:26.194 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-22T08:10:26.194 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-22T08:10:26.194 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-22T08:10:26.195 INFO:teuthology.orchestra.run.smithi189.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-22T08:10:26.195 INFO:teuthology.orchestra.run.smithi189.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-22T08:10:26.195 INFO:teuthology.orchestra.run.smithi189.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-22T08:10:26.195 INFO:teuthology.orchestra.run.smithi189.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-22T08:10:26.195 INFO:teuthology.orchestra.run.smithi189.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-22T08:10:26.195 INFO:teuthology.orchestra.run.smithi189.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-22T08:10:26.195 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:10:26.195 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-22T08:10:26.195 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:10:26.195 INFO:teuthology.orchestra.run.smithi189.stdout:Install 33 Packages 2024-04-22T08:10:26.195 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:10:26.196 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 29 M 2024-04-22T08:10:26.196 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 104 M 2024-04-22T08:10:26.196 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-22T08:10:26.240 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-04-22T08:10:26.363 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-04-22T08:10:26.398 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-04-22T08:10:26.525 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-04-22T08:10:26.688 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1237.g646b8c7d.el8. 28/33 2024-04-22T08:10:26.716 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-2:18.2.2-1237.g646b8c7d.el8.x86_64 29/33 2024-04-22T08:10:26.742 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1237.g646b8c7d.el8.x86_64 29/33 2024-04-22T08:10:26.742 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T08:10:26.742 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-22T08:10:26.742 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T08:10:26.742 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T08:10:26.742 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:10:26.932 INFO:teuthology.orchestra.run.smithi189.stdout:(1/33): ceph-18.2.2-1237.g646b8c7d.el8.x86_64.r 40 kB/s | 6.5 kB 00:00 2024-04-22T08:10:27.100 INFO:teuthology.orchestra.run.smithi189.stdout:(2/33): ceph-mgr-18.2.2-1237.g646b8c7d.el8.x86_ 4.4 MB/s | 1.5 MB 00:00 2024-04-22T08:10:27.625 INFO:teuthology.orchestra.run.smithi189.stdout:(3/33): ceph-mds-18.2.2-1237.g646b8c7d.el8.x86_ 2.4 MB/s | 2.1 MB 00:00 2024-04-22T08:10:27.725 INFO:teuthology.orchestra.run.smithi189.stdout:(4/33): ceph-mgr-modules-core-18.2.2-1237.g646b 2.4 MB/s | 246 kB 00:00 2024-04-22T08:10:28.050 INFO:teuthology.orchestra.run.smithi189.stdout:(5/33): ceph-osd-18.2.2-1237.g646b8c7d.el8.x86_ 17 MB/s | 17 MB 00:00 2024-04-22T08:10:28.117 INFO:teuthology.orchestra.run.smithi189.stdout:(6/33): ceph-mon-18.2.2-1237.g646b8c7d.el8.x86_ 3.9 MB/s | 4.7 MB 00:01 2024-04-22T08:10:28.235 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-osd-2:18.2.2-1237.g646b8c7d.el8.x86_64 30/33 2024-04-22T08:10:28.264 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-osd-2:18.2.2-1237.g646b8c7d.el8.x86_64 30/33 2024-04-22T08:10:28.264 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T08:10:28.264 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-22T08:10:28.264 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T08:10:28.264 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T08:10:28.264 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:10:28.434 INFO:teuthology.orchestra.run.smithi189.stdout:(7/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 357 kB/s | 253 kB 00:00 2024-04-22T08:10:28.476 INFO:teuthology.orchestra.run.smithi189.stdout:(8/33): python3-ply-3.11-14.el9.noarch.rpm 345 kB/s | 106 kB 00:00 2024-04-22T08:10:28.626 INFO:teuthology.orchestra.run.smithi189.stdout:(9/33): python3-cryptography-36.0.1-4.el9.x86_6 2.1 MB/s | 1.2 MB 00:00 2024-04-22T08:10:28.735 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mon-2:18.2.2-1237.g646b8c7d.el8.x86_64 31/33 2024-04-22T08:10:28.759 INFO:teuthology.orchestra.run.smithi189.stdout:(10/33): python3-pycparser-2.20-6.el9.noarch.rp 415 kB/s | 135 kB 00:00 2024-04-22T08:10:28.761 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mon-2:18.2.2-1237.g646b8c7d.el8.x86_64 31/33 2024-04-22T08:10:28.761 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T08:10:28.761 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-22T08:10:28.761 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T08:10:28.762 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T08:10:28.762 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:10:28.826 INFO:teuthology.orchestra.run.smithi189.stdout:(11/33): python3-requests-2.25.1-8.el9.noarch.r 356 kB/s | 125 kB 00:00 2024-04-22T08:10:28.935 INFO:teuthology.orchestra.run.smithi189.stdout:(12/33): python3-urllib3-1.26.5-5.el9.noarch.rp 696 kB/s | 215 kB 00:00 2024-04-22T08:10:28.969 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mds-2:18.2.2-1237.g646b8c7d.el8.x86_64 32/33 2024-04-22T08:10:28.994 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mds-2:18.2.2-1237.g646b8c7d.el8.x86_64 32/33 2024-04-22T08:10:28.994 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T08:10:28.994 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-22T08:10:28.994 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T08:10:28.994 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T08:10:28.994 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:10:29.059 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-2:18.2.2-1237.g646b8c7d.el8.x86_64 33/33 2024-04-22T08:10:29.094 INFO:teuthology.orchestra.run.smithi189.stdout:(13/33): python3-markupsafe-1.1.1-12.el9.x86_64 130 kB/s | 35 kB 00:00 2024-04-22T08:10:29.261 INFO:teuthology.orchestra.run.smithi189.stdout:(14/33): python3-mako-1.1.4-6.el9.noarch.rpm 344 kB/s | 172 kB 00:00 2024-04-22T08:10:29.286 INFO:teuthology.orchestra.run.smithi189.stdout:(15/33): python3-pytz-2021.1-5.el9.noarch.rpm 145 kB/s | 51 kB 00:00 2024-04-22T08:10:29.311 INFO:teuthology.orchestra.run.smithi189.stdout:(16/33): python3-toml-0.10.2-6.el9.noarch.rpm 192 kB/s | 42 kB 00:00 2024-04-22T08:10:29.403 INFO:teuthology.orchestra.run.smithi189.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 306 kB/s | 43 kB 00:00 2024-04-22T08:10:29.445 INFO:teuthology.orchestra.run.smithi189.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 257 kB/s | 11 kB 00:00 2024-04-22T08:10:29.470 INFO:teuthology.orchestra.run.smithi189.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-04-22T08:10:29.495 INFO:teuthology.orchestra.run.smithi189.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 351 kB/s | 18 kB 00:00 2024-04-22T08:10:29.521 INFO:teuthology.orchestra.run.smithi189.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-04-22T08:10:29.546 INFO:teuthology.orchestra.run.smithi189.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 386 kB/s | 19 kB 00:00 2024-04-22T08:10:29.563 INFO:teuthology.orchestra.run.smithi189.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 469 kB/s | 20 kB 00:00 2024-04-22T08:10:29.588 INFO:teuthology.orchestra.run.smithi189.stdout:(24/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-04-22T08:10:29.605 INFO:teuthology.orchestra.run.smithi189.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 787 kB/s | 46 kB 00:00 2024-04-22T08:10:29.631 INFO:teuthology.orchestra.run.smithi189.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 1.1 MB/s | 79 kB 00:00 2024-04-22T08:10:29.647 INFO:teuthology.orchestra.run.smithi189.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 394 kB/s | 16 kB 00:00 2024-04-22T08:10:29.673 INFO:teuthology.orchestra.run.smithi189.stdout:(28/33): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-04-22T08:10:29.698 INFO:teuthology.orchestra.run.smithi189.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 711 kB/s | 36 kB 00:00 2024-04-22T08:10:29.715 INFO:teuthology.orchestra.run.smithi189.stdout:(30/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.0 MB/s | 90 kB 00:00 2024-04-22T08:10:29.740 INFO:teuthology.orchestra.run.smithi189.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 3.3 MB/s | 230 kB 00:00 2024-04-22T08:10:29.765 INFO:teuthology.orchestra.run.smithi189.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 397 kB/s | 20 kB 00:00 2024-04-22T08:10:29.816 INFO:teuthology.orchestra.run.smithi189.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.6 MB/s | 427 kB 00:00 2024-04-22T08:10:29.817 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-22T08:10:29.818 INFO:teuthology.orchestra.run.smithi189.stdout:Total 8.0 MB/s | 29 MB 00:03 2024-04-22T08:10:29.937 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-22T08:10:29.974 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-22T08:10:29.975 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-22T08:10:30.256 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-22T08:10:30.256 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-22T08:10:30.648 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-2:18.2.2-1237.g646b8c7d.el8.x86_64 33/33 2024-04-22T08:10:30.648 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/33 2024-04-22T08:10:30.648 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mds-2:18.2.2-1237.g646b8c7d.el8.x86_64 2/33 2024-04-22T08:10:30.648 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-2:18.2.2-1237.g646b8c7d.el8.x86_64 3/33 2024-04-22T08:10:30.648 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mon-2:18.2.2-1237.g646b8c7d.el8.x86_64 4/33 2024-04-22T08:10:30.648 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-osd-2:18.2.2-1237.g646b8c7d.el8.x86_64 5/33 2024-04-22T08:10:30.648 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1237.g646b8c7d.el8. 6/33 2024-04-22T08:10:30.649 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-04-22T08:10:30.649 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-04-22T08:10:30.649 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-04-22T08:10:30.649 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-04-22T08:10:30.649 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-04-22T08:10:30.649 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-04-22T08:10:30.649 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-04-22T08:10:30.649 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-04-22T08:10:30.649 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-04-22T08:10:30.649 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-04-22T08:10:30.649 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-04-22T08:10:30.649 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-04-22T08:10:30.649 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-04-22T08:10:30.649 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-04-22T08:10:30.650 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-04-22T08:10:30.650 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-04-22T08:10:30.650 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-04-22T08:10:30.650 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-04-22T08:10:30.650 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-04-22T08:10:30.651 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-04-22T08:10:30.651 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-04-22T08:10:30.651 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-04-22T08:10:30.651 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-04-22T08:10:30.651 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-04-22T08:10:30.651 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-04-22T08:10:30.651 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-04-22T08:10:31.422 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-22T08:10:31.485 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-04-22T08:10:31.517 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-04-22T08:10:31.550 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-04-22T08:10:31.578 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-04-22T08:10:31.578 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:10:31.579 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-04-22T08:10:31.579 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:10:31.579 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mds-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:10:31.579 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:10:31.579 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-modules-core-2:18.2.2-1237.g646b8c7d.el8.noarch 2024-04-22T08:10:31.579 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mon-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:10:31.579 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-osd-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:10:31.579 INFO:teuthology.orchestra.run.smithi040.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-22T08:10:31.579 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-22T08:10:31.579 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-22T08:10:31.579 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-22T08:10:31.579 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-22T08:10:31.579 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-22T08:10:31.580 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-22T08:10:31.580 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-22T08:10:31.580 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-22T08:10:31.580 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-22T08:10:31.580 INFO:teuthology.orchestra.run.smithi040.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-22T08:10:31.580 INFO:teuthology.orchestra.run.smithi040.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-22T08:10:31.580 INFO:teuthology.orchestra.run.smithi040.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-22T08:10:31.580 INFO:teuthology.orchestra.run.smithi040.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-22T08:10:31.580 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-22T08:10:31.580 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ply-3.11-14.el9.noarch 2024-04-22T08:10:31.580 INFO:teuthology.orchestra.run.smithi040.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-22T08:10:31.580 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-22T08:10:31.580 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-22T08:10:31.580 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-22T08:10:31.581 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-22T08:10:31.581 INFO:teuthology.orchestra.run.smithi040.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-22T08:10:31.581 INFO:teuthology.orchestra.run.smithi040.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-22T08:10:31.581 INFO:teuthology.orchestra.run.smithi040.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-22T08:10:31.581 INFO:teuthology.orchestra.run.smithi040.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-22T08:10:31.581 INFO:teuthology.orchestra.run.smithi040.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-22T08:10:31.581 INFO:teuthology.orchestra.run.smithi040.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-22T08:10:31.581 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:10:31.581 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-22T08:10:31.643 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-04-22T08:10:31.713 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-04-22T08:10:31.753 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-04-22T08:10:31.788 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-04-22T08:10:31.826 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-04-22T08:10:31.861 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-04-22T08:10:31.894 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-04-22T08:10:31.929 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-base 2024-04-22T08:10:31.934 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-04-22T08:10:31.970 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-04-22T08:10:32.004 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-04-22T08:10:32.042 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-04-22T08:10:32.116 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-04-22T08:10:32.257 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-04-22T08:10:32.342 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-04-22T08:10:32.395 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-04-22T08:10:32.433 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-04-22T08:10:32.464 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:00:52 ago on Mon 22 Apr 2024 08:09:40 AM UTC. 2024-04-22T08:10:32.483 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-04-22T08:10:32.538 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-04-22T08:10:32.546 INFO:teuthology.orchestra.run.smithi040.stdout:Package ceph-base-2:18.2.2-1237.g646b8c7d.el8.x86_64 is already installed. 2024-04-22T08:10:32.589 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-22T08:10:32.590 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2024-04-22T08:10:32.590 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-22T08:10:32.663 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install cephadm 2024-04-22T08:10:32.726 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-04-22T08:10:32.787 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-04-22T08:10:32.866 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-04-22T08:10:32.954 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-04-22T08:10:32.994 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-04-22T08:10:33.111 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-04-22T08:10:33.191 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:00:53 ago on Mon 22 Apr 2024 08:09:40 AM UTC. 2024-04-22T08:10:33.270 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1237.g646b8c7d.el8. 28/33 2024-04-22T08:10:33.296 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-2:18.2.2-1237.g646b8c7d.el8.x86_64 29/33 2024-04-22T08:10:33.316 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-22T08:10:33.317 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:10:33.317 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2024-04-22T08:10:33.317 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:10:33.317 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-04-22T08:10:33.317 INFO:teuthology.orchestra.run.smithi040.stdout: cephadm noarch 2:18.2.2-1237.g646b8c7d.el8 ceph-noarch 224 k 2024-04-22T08:10:33.317 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:10:33.317 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-04-22T08:10:33.318 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:10:33.318 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2024-04-22T08:10:33.318 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:10:33.318 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 224 k 2024-04-22T08:10:33.318 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 216 k 2024-04-22T08:10:33.318 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-04-22T08:10:33.321 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1237.g646b8c7d.el8.x86_64 29/33 2024-04-22T08:10:33.321 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T08:10:33.321 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-22T08:10:33.321 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T08:10:33.321 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T08:10:33.322 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:10:33.761 INFO:teuthology.orchestra.run.smithi040.stdout:cephadm-18.2.2-1237.g646b8c7d.el8.noarch.rpm 507 kB/s | 224 kB 00:00 2024-04-22T08:10:33.761 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-04-22T08:10:33.762 INFO:teuthology.orchestra.run.smithi040.stdout:Total 505 kB/s | 224 kB 00:00 2024-04-22T08:10:33.762 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-04-22T08:10:33.767 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-04-22T08:10:33.767 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-04-22T08:10:33.770 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-04-22T08:10:33.770 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-04-22T08:10:33.840 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-04-22T08:10:34.290 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: cephadm-2:18.2.2-1237.g646b8c7d.el8.noarch 1/1 2024-04-22T08:10:34.374 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : cephadm-2:18.2.2-1237.g646b8c7d.el8.noarch 1/1 2024-04-22T08:10:34.651 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: cephadm-2:18.2.2-1237.g646b8c7d.el8.noarch 1/1 2024-04-22T08:10:34.784 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-osd-2:18.2.2-1237.g646b8c7d.el8.x86_64 30/33 2024-04-22T08:10:34.816 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-osd-2:18.2.2-1237.g646b8c7d.el8.x86_64 30/33 2024-04-22T08:10:34.816 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T08:10:34.816 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-22T08:10:34.816 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T08:10:34.816 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T08:10:34.816 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:10:34.860 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : cephadm-2:18.2.2-1237.g646b8c7d.el8.noarch 1/1 2024-04-22T08:10:34.861 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:10:34.861 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-04-22T08:10:34.861 INFO:teuthology.orchestra.run.smithi040.stdout: cephadm-2:18.2.2-1237.g646b8c7d.el8.noarch 2024-04-22T08:10:34.861 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:10:34.861 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-22T08:10:35.107 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-immutable-object-cache 2024-04-22T08:10:35.260 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mon-2:18.2.2-1237.g646b8c7d.el8.x86_64 31/33 2024-04-22T08:10:35.284 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mon-2:18.2.2-1237.g646b8c7d.el8.x86_64 31/33 2024-04-22T08:10:35.284 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T08:10:35.284 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-22T08:10:35.284 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T08:10:35.284 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T08:10:35.284 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:10:35.485 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mds-2:18.2.2-1237.g646b8c7d.el8.x86_64 32/33 2024-04-22T08:10:35.509 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mds-2:18.2.2-1237.g646b8c7d.el8.x86_64 32/33 2024-04-22T08:10:35.509 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T08:10:35.509 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-22T08:10:35.509 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T08:10:35.509 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T08:10:35.509 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:10:35.572 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-2:18.2.2-1237.g646b8c7d.el8.x86_64 33/33 2024-04-22T08:10:35.642 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:00:55 ago on Mon 22 Apr 2024 08:09:40 AM UTC. 2024-04-22T08:10:35.771 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-22T08:10:35.772 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:10:35.772 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repo Size 2024-04-22T08:10:35.772 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:10:35.772 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-04-22T08:10:35.772 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 143 k 2024-04-22T08:10:35.772 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:10:35.772 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-04-22T08:10:35.773 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:10:35.773 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2024-04-22T08:10:35.773 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:10:35.773 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 143 k 2024-04-22T08:10:35.774 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 395 k 2024-04-22T08:10:35.774 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-04-22T08:10:36.749 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-immutable-object-cache-18.2.2-1237.g646b8c 146 kB/s | 143 kB 00:00 2024-04-22T08:10:36.750 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-04-22T08:10:36.750 INFO:teuthology.orchestra.run.smithi040.stdout:Total 146 kB/s | 143 kB 00:00 2024-04-22T08:10:36.750 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-04-22T08:10:36.760 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-04-22T08:10:36.760 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-04-22T08:10:36.825 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-04-22T08:10:36.825 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-04-22T08:10:36.977 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-04-22T08:10:37.025 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1237.g646b8c7d. 1/1 2024-04-22T08:10:37.051 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1237.g646b8c7d. 1/1 2024-04-22T08:10:37.051 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T08:10:37.051 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-22T08:10:37.051 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:10:37.168 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-2:18.2.2-1237.g646b8c7d.el8.x86_64 33/33 2024-04-22T08:10:37.169 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/33 2024-04-22T08:10:37.169 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mds-2:18.2.2-1237.g646b8c7d.el8.x86_64 2/33 2024-04-22T08:10:37.169 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-2:18.2.2-1237.g646b8c7d.el8.x86_64 3/33 2024-04-22T08:10:37.169 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mon-2:18.2.2-1237.g646b8c7d.el8.x86_64 4/33 2024-04-22T08:10:37.169 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-osd-2:18.2.2-1237.g646b8c7d.el8.x86_64 5/33 2024-04-22T08:10:37.169 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1237.g646b8c7d.el8. 6/33 2024-04-22T08:10:37.169 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-04-22T08:10:37.169 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-04-22T08:10:37.169 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-04-22T08:10:37.169 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-04-22T08:10:37.169 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-04-22T08:10:37.169 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-04-22T08:10:37.169 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-04-22T08:10:37.170 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-04-22T08:10:37.170 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-04-22T08:10:37.170 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-04-22T08:10:37.170 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-04-22T08:10:37.170 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-04-22T08:10:37.170 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-04-22T08:10:37.170 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-04-22T08:10:37.170 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-04-22T08:10:37.170 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-04-22T08:10:37.170 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-04-22T08:10:37.170 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-04-22T08:10:37.170 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-04-22T08:10:37.170 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-04-22T08:10:37.171 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-04-22T08:10:37.171 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-04-22T08:10:37.171 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-04-22T08:10:37.171 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-04-22T08:10:37.171 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-04-22T08:10:37.171 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-04-22T08:10:37.884 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1237.g646b8c7d. 1/1 2024-04-22T08:10:37.885 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:10:37.885 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-04-22T08:10:37.885 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-immutable-object-cache-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:10:37.885 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:10:37.885 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-22T08:10:38.019 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-04-22T08:10:38.019 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:10:38.019 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-22T08:10:38.019 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:10:38.019 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mds-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:10:38.019 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:10:38.019 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-modules-core-2:18.2.2-1237.g646b8c7d.el8.noarch 2024-04-22T08:10:38.020 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mon-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:10:38.020 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-osd-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:10:38.020 INFO:teuthology.orchestra.run.smithi189.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-22T08:10:38.020 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-22T08:10:38.020 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-22T08:10:38.020 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-22T08:10:38.020 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-22T08:10:38.020 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-22T08:10:38.020 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-22T08:10:38.020 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-22T08:10:38.020 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-22T08:10:38.020 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-22T08:10:38.020 INFO:teuthology.orchestra.run.smithi189.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-22T08:10:38.020 INFO:teuthology.orchestra.run.smithi189.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-22T08:10:38.021 INFO:teuthology.orchestra.run.smithi189.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-22T08:10:38.021 INFO:teuthology.orchestra.run.smithi189.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-22T08:10:38.021 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-22T08:10:38.021 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ply-3.11-14.el9.noarch 2024-04-22T08:10:38.021 INFO:teuthology.orchestra.run.smithi189.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-22T08:10:38.021 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-22T08:10:38.021 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-22T08:10:38.021 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-22T08:10:38.021 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-22T08:10:38.021 INFO:teuthology.orchestra.run.smithi189.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-22T08:10:38.021 INFO:teuthology.orchestra.run.smithi189.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-22T08:10:38.021 INFO:teuthology.orchestra.run.smithi189.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-22T08:10:38.021 INFO:teuthology.orchestra.run.smithi189.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-22T08:10:38.021 INFO:teuthology.orchestra.run.smithi189.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-22T08:10:38.022 INFO:teuthology.orchestra.run.smithi189.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-22T08:10:38.022 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:10:38.022 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-22T08:10:38.119 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-mgr 2024-04-22T08:10:38.387 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-base 2024-04-22T08:10:38.655 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:00:58 ago on Mon 22 Apr 2024 08:09:40 AM UTC. 2024-04-22T08:10:38.738 INFO:teuthology.orchestra.run.smithi040.stdout:Package ceph-mgr-2:18.2.2-1237.g646b8c7d.el8.x86_64 is already installed. 2024-04-22T08:10:38.783 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-22T08:10:38.785 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2024-04-22T08:10:38.785 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-22T08:10:38.862 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-mgr-dashboard 2024-04-22T08:10:38.904 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:00:49 ago on Mon 22 Apr 2024 08:09:49 AM UTC. 2024-04-22T08:10:38.984 INFO:teuthology.orchestra.run.smithi189.stdout:Package ceph-base-2:18.2.2-1237.g646b8c7d.el8.x86_64 is already installed. 2024-04-22T08:10:39.025 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-22T08:10:39.025 INFO:teuthology.orchestra.run.smithi189.stdout:Nothing to do. 2024-04-22T08:10:39.026 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-22T08:10:39.092 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install cephadm 2024-04-22T08:10:39.424 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:00:59 ago on Mon 22 Apr 2024 08:09:40 AM UTC. 2024-04-22T08:10:39.549 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-22T08:10:39.549 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:10:39.550 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2024-04-22T08:10:39.550 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:10:39.550 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-04-22T08:10:39.550 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1237.g646b8c7d.el8 ceph-noarch 3.5 M 2024-04-22T08:10:39.550 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2024-04-22T08:10:39.550 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1237.g646b8c7d.el8 ceph-noarch 24 k 2024-04-22T08:10:39.550 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1237.g646b8c7d.el8 ceph-noarch 15 k 2024-04-22T08:10:39.550 INFO:teuthology.orchestra.run.smithi040.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-22T08:10:39.550 INFO:teuthology.orchestra.run.smithi040.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-22T08:10:39.550 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:10:39.550 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-04-22T08:10:39.550 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:10:39.550 INFO:teuthology.orchestra.run.smithi040.stdout:Install 5 Packages 2024-04-22T08:10:39.551 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:10:39.551 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 3.8 M 2024-04-22T08:10:39.551 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 87 M 2024-04-22T08:10:39.551 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-04-22T08:10:39.606 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:00:50 ago on Mon 22 Apr 2024 08:09:49 AM UTC. 2024-04-22T08:10:39.726 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-22T08:10:39.726 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:10:39.727 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repository Size 2024-04-22T08:10:39.727 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:10:39.727 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-22T08:10:39.727 INFO:teuthology.orchestra.run.smithi189.stdout: cephadm noarch 2:18.2.2-1237.g646b8c7d.el8 ceph-noarch 224 k 2024-04-22T08:10:39.727 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:10:39.727 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-22T08:10:39.727 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:10:39.727 INFO:teuthology.orchestra.run.smithi189.stdout:Install 1 Package 2024-04-22T08:10:39.727 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:10:39.727 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 224 k 2024-04-22T08:10:39.727 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 216 k 2024-04-22T08:10:39.727 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-22T08:10:40.353 INFO:teuthology.orchestra.run.smithi040.stdout:(1/5): ceph-grafana-dashboards-18.2.2-1237.g646 54 kB/s | 24 kB 00:00 2024-04-22T08:10:40.437 INFO:teuthology.orchestra.run.smithi040.stdout:(2/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 370 kB/s | 31 kB 00:00 2024-04-22T08:10:40.462 INFO:teuthology.orchestra.run.smithi040.stdout:(3/5): ceph-prometheus-alerts-18.2.2-1237.g646b 27 kB/s | 15 kB 00:00 2024-04-22T08:10:40.529 INFO:teuthology.orchestra.run.smithi040.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 2.0 MB/s | 188 kB 00:00 2024-04-22T08:10:41.043 INFO:teuthology.orchestra.run.smithi189.stdout:cephadm-18.2.2-1237.g646b8c7d.el8.noarch.rpm 170 kB/s | 224 kB 00:01 2024-04-22T08:10:41.044 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-22T08:10:41.044 INFO:teuthology.orchestra.run.smithi189.stdout:Total 170 kB/s | 224 kB 00:01 2024-04-22T08:10:41.045 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-22T08:10:41.051 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-22T08:10:41.051 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-22T08:10:41.054 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-22T08:10:41.054 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-22T08:10:41.123 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-22T08:10:41.592 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: cephadm-2:18.2.2-1237.g646b8c7d.el8.noarch 1/1 2024-04-22T08:10:41.671 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : cephadm-2:18.2.2-1237.g646b8c7d.el8.noarch 1/1 2024-04-22T08:10:41.854 INFO:teuthology.orchestra.run.smithi040.stdout:(5/5): ceph-mgr-dashboard-18.2.2-1237.g646b8c7d 1.8 MB/s | 3.5 MB 00:01 2024-04-22T08:10:41.857 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-04-22T08:10:41.858 INFO:teuthology.orchestra.run.smithi040.stdout:Total 1.6 MB/s | 3.8 MB 00:02 2024-04-22T08:10:41.858 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-04-22T08:10:41.866 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-04-22T08:10:41.867 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-04-22T08:10:41.936 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-04-22T08:10:41.936 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-04-22T08:10:41.974 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: cephadm-2:18.2.2-1237.g646b8c7d.el8.noarch 1/1 2024-04-22T08:10:42.192 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-04-22T08:10:42.195 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : cephadm-2:18.2.2-1237.g646b8c7d.el8.noarch 1/1 2024-04-22T08:10:42.195 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:10:42.195 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-22T08:10:42.196 INFO:teuthology.orchestra.run.smithi189.stdout: cephadm-2:18.2.2-1237.g646b8c7d.el8.noarch 2024-04-22T08:10:42.196 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:10:42.196 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-22T08:10:42.279 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-04-22T08:10:42.309 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-04-22T08:10:42.338 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1237.g646b8c7d.el8.n 3/5 2024-04-22T08:10:42.412 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-immutable-object-cache 2024-04-22T08:10:42.933 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:00:53 ago on Mon 22 Apr 2024 08:09:49 AM UTC. 2024-04-22T08:10:43.058 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-22T08:10:43.059 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:10:43.059 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repo Size 2024-04-22T08:10:43.059 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:10:43.059 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-22T08:10:43.059 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 143 k 2024-04-22T08:10:43.059 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:10:43.060 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-22T08:10:43.060 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:10:43.060 INFO:teuthology.orchestra.run.smithi189.stdout:Install 1 Package 2024-04-22T08:10:43.060 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:10:43.060 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 143 k 2024-04-22T08:10:43.060 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 395 k 2024-04-22T08:10:43.060 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-22T08:10:43.282 INFO:teuthology.orchestra.run.smithi189.stdout:ceph-immutable-object-cache-18.2.2-1237.g646b8c 635 kB/s | 143 kB 00:00 2024-04-22T08:10:43.283 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-22T08:10:43.283 INFO:teuthology.orchestra.run.smithi189.stdout:Total 629 kB/s | 143 kB 00:00 2024-04-22T08:10:43.283 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-22T08:10:43.291 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-22T08:10:43.291 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-22T08:10:43.347 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-22T08:10:43.348 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-22T08:10:43.450 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1237.g646b8c7d.el8. 4/5 2024-04-22T08:10:43.483 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1237.g646b8c7d.el8.noarc 5/5 2024-04-22T08:10:43.483 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-22T08:10:43.567 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1237.g646b8c7d. 1/1 2024-04-22T08:10:43.591 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1237.g646b8c7d. 1/1 2024-04-22T08:10:43.592 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T08:10:43.592 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-22T08:10:43.592 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:10:44.480 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1237.g646b8c7d. 1/1 2024-04-22T08:10:44.480 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:10:44.480 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-22T08:10:44.480 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-immutable-object-cache-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:10:44.481 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:10:44.481 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-22T08:10:44.485 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1237.g646b8c7d.el8.noarc 5/5 2024-04-22T08:10:44.485 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1237.g646b8c7d.el8. 1/5 2024-04-22T08:10:44.485 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1237.g646b8c7d.el8.noarc 2/5 2024-04-22T08:10:44.485 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1237.g646b8c7d.el8.n 3/5 2024-04-22T08:10:44.485 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-04-22T08:10:44.699 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-mgr 2024-04-22T08:10:44.726 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-04-22T08:10:44.726 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:10:44.726 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-04-22T08:10:44.726 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-grafana-dashboards-2:18.2.2-1237.g646b8c7d.el8.noarch 2024-04-22T08:10:44.726 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-dashboard-2:18.2.2-1237.g646b8c7d.el8.noarch 2024-04-22T08:10:44.726 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-prometheus-alerts-2:18.2.2-1237.g646b8c7d.el8.noarch 2024-04-22T08:10:44.726 INFO:teuthology.orchestra.run.smithi040.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-22T08:10:44.726 INFO:teuthology.orchestra.run.smithi040.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-22T08:10:44.726 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:10:44.726 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-22T08:10:44.932 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-22T08:10:45.224 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:00:56 ago on Mon 22 Apr 2024 08:09:49 AM UTC. 2024-04-22T08:10:45.305 INFO:teuthology.orchestra.run.smithi189.stdout:Package ceph-mgr-2:18.2.2-1237.g646b8c7d.el8.x86_64 is already installed. 2024-04-22T08:10:45.348 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-22T08:10:45.349 INFO:teuthology.orchestra.run.smithi189.stdout:Nothing to do. 2024-04-22T08:10:45.349 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-22T08:10:45.416 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-mgr-dashboard 2024-04-22T08:10:45.469 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:05 ago on Mon 22 Apr 2024 08:09:40 AM UTC. 2024-04-22T08:10:45.596 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-22T08:10:45.597 INFO:teuthology.orchestra.run.smithi040.stdout:======================================================================================= 2024-04-22T08:10:45.598 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2024-04-22T08:10:45.598 INFO:teuthology.orchestra.run.smithi040.stdout:======================================================================================= 2024-04-22T08:10:45.598 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-04-22T08:10:45.598 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1237.g646b8c7d.el8 ceph-noarch 7.4 M 2024-04-22T08:10:45.598 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2024-04-22T08:10:45.598 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-22T08:10:45.598 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-22T08:10:45.598 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-22T08:10:45.598 INFO:teuthology.orchestra.run.smithi040.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-22T08:10:45.598 INFO:teuthology.orchestra.run.smithi040.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-22T08:10:45.598 INFO:teuthology.orchestra.run.smithi040.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-22T08:10:45.599 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-22T08:10:45.599 INFO:teuthology.orchestra.run.smithi040.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-22T08:10:45.599 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-22T08:10:45.599 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-22T08:10:45.599 INFO:teuthology.orchestra.run.smithi040.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-22T08:10:45.599 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:10:45.599 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-04-22T08:10:45.599 INFO:teuthology.orchestra.run.smithi040.stdout:======================================================================================= 2024-04-22T08:10:45.599 INFO:teuthology.orchestra.run.smithi040.stdout:Install 12 Packages 2024-04-22T08:10:45.599 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:10:45.600 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 38 M 2024-04-22T08:10:45.600 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 209 M 2024-04-22T08:10:45.600 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-04-22T08:10:45.929 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:00:56 ago on Mon 22 Apr 2024 08:09:49 AM UTC. 2024-04-22T08:10:46.050 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-22T08:10:46.050 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:10:46.050 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repository Size 2024-04-22T08:10:46.050 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:10:46.050 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-22T08:10:46.050 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1237.g646b8c7d.el8 ceph-noarch 3.5 M 2024-04-22T08:10:46.051 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-04-22T08:10:46.051 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1237.g646b8c7d.el8 ceph-noarch 24 k 2024-04-22T08:10:46.051 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1237.g646b8c7d.el8 ceph-noarch 15 k 2024-04-22T08:10:46.051 INFO:teuthology.orchestra.run.smithi189.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-22T08:10:46.051 INFO:teuthology.orchestra.run.smithi189.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-22T08:10:46.051 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:10:46.051 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-22T08:10:46.051 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:10:46.051 INFO:teuthology.orchestra.run.smithi189.stdout:Install 5 Packages 2024-04-22T08:10:46.051 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:10:46.052 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 3.8 M 2024-04-22T08:10:46.052 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 87 M 2024-04-22T08:10:46.052 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-22T08:10:46.269 INFO:teuthology.orchestra.run.smithi040.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 736 kB/s | 195 kB 00:00 2024-04-22T08:10:46.303 INFO:teuthology.orchestra.run.smithi040.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.6 MB/s | 803 kB 00:00 2024-04-22T08:10:46.387 INFO:teuthology.orchestra.run.smithi040.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 279 kB/s | 33 kB 00:00 2024-04-22T08:10:46.445 INFO:teuthology.orchestra.run.smithi040.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 312 kB/s | 18 kB 00:00 2024-04-22T08:10:46.486 INFO:teuthology.orchestra.run.smithi189.stdout:(1/5): ceph-grafana-dashboards-18.2.2-1237.g646 94 kB/s | 24 kB 00:00 2024-04-22T08:10:46.504 INFO:teuthology.orchestra.run.smithi040.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 602 kB/s | 35 kB 00:00 2024-04-22T08:10:46.578 INFO:teuthology.orchestra.run.smithi189.stdout:(2/5): ceph-prometheus-alerts-18.2.2-1237.g646b 44 kB/s | 15 kB 00:00 2024-04-22T08:10:46.603 INFO:teuthology.orchestra.run.smithi189.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 263 kB/s | 31 kB 00:00 2024-04-22T08:10:46.629 INFO:teuthology.orchestra.run.smithi040.stdout:(6/12): ceph-mgr-diskprediction-local-18.2.2-12 12 MB/s | 7.4 MB 00:00 2024-04-22T08:10:46.728 INFO:teuthology.orchestra.run.smithi189.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 1.2 MB/s | 188 kB 00:00 2024-04-22T08:10:46.821 INFO:teuthology.orchestra.run.smithi040.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.8 MB/s | 3.0 MB 00:00 2024-04-22T08:10:46.936 INFO:teuthology.orchestra.run.smithi040.stdout:(8/12): python3-devel-3.9.18-3.el9.x86_64.rpm 796 kB/s | 244 kB 00:00 2024-04-22T08:10:46.979 INFO:teuthology.orchestra.run.smithi189.stdout:(5/5): ceph-mgr-dashboard-18.2.2-1237.g646b8c7d 4.7 MB/s | 3.5 MB 00:00 2024-04-22T08:10:46.981 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-22T08:10:46.981 INFO:teuthology.orchestra.run.smithi189.stdout:Total 4.1 MB/s | 3.8 MB 00:00 2024-04-22T08:10:46.981 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-22T08:10:46.988 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-22T08:10:46.988 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-22T08:10:46.996 INFO:teuthology.orchestra.run.smithi040.stdout:(9/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 9.6 MB/s | 4.7 MB 00:00 2024-04-22T08:10:47.047 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-22T08:10:47.049 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-22T08:10:47.147 INFO:teuthology.orchestra.run.smithi040.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 16 MB/s | 5.1 MB 00:00 2024-04-22T08:10:47.172 INFO:teuthology.orchestra.run.smithi040.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 1.0 MB/s | 250 kB 00:00 2024-04-22T08:10:47.287 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-22T08:10:47.368 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-04-22T08:10:47.400 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-04-22T08:10:47.429 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1237.g646b8c7d.el8.n 3/5 2024-04-22T08:10:47.656 INFO:teuthology.orchestra.run.smithi040.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 25 MB/s | 16 MB 00:00 2024-04-22T08:10:47.658 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-04-22T08:10:47.658 INFO:teuthology.orchestra.run.smithi040.stdout:Total 18 MB/s | 38 MB 00:02 2024-04-22T08:10:48.019 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-04-22T08:10:48.047 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-04-22T08:10:48.047 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-04-22T08:10:48.480 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-04-22T08:10:48.480 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-04-22T08:10:48.486 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1237.g646b8c7d.el8. 4/5 2024-04-22T08:10:48.516 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1237.g646b8c7d.el8.noarc 5/5 2024-04-22T08:10:49.255 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-04-22T08:10:49.310 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-04-22T08:10:49.355 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-04-22T08:10:49.463 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-04-22T08:10:49.505 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-04-22T08:10:49.602 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1237.g646b8c7d.el8.noarc 5/5 2024-04-22T08:10:49.602 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1237.g646b8c7d.el8. 1/5 2024-04-22T08:10:49.602 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1237.g646b8c7d.el8.noarc 2/5 2024-04-22T08:10:49.602 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1237.g646b8c7d.el8.n 3/5 2024-04-22T08:10:49.606 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-04-22T08:10:49.693 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-22T08:10:49.721 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-22T08:10:49.801 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-22T08:10:49.839 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-04-22T08:10:49.839 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:10:49.839 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-22T08:10:49.839 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-grafana-dashboards-2:18.2.2-1237.g646b8c7d.el8.noarch 2024-04-22T08:10:49.839 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-dashboard-2:18.2.2-1237.g646b8c7d.el8.noarch 2024-04-22T08:10:49.839 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-prometheus-alerts-2:18.2.2-1237.g646b8c7d.el8.noarch 2024-04-22T08:10:49.839 INFO:teuthology.orchestra.run.smithi189.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-22T08:10:49.839 INFO:teuthology.orchestra.run.smithi189.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-22T08:10:49.839 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:10:49.840 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-22T08:10:50.039 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-22T08:10:50.491 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-22T08:10:50.553 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:01 ago on Mon 22 Apr 2024 08:09:49 AM UTC. 2024-04-22T08:10:50.614 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-22T08:10:50.676 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-22T08:10:50.677 INFO:teuthology.orchestra.run.smithi189.stdout:======================================================================================= 2024-04-22T08:10:50.677 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repository Size 2024-04-22T08:10:50.677 INFO:teuthology.orchestra.run.smithi189.stdout:======================================================================================= 2024-04-22T08:10:50.677 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-22T08:10:50.677 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1237.g646b8c7d.el8 ceph-noarch 7.4 M 2024-04-22T08:10:50.677 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-04-22T08:10:50.677 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-22T08:10:50.678 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-22T08:10:50.678 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-22T08:10:50.678 INFO:teuthology.orchestra.run.smithi189.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-22T08:10:50.678 INFO:teuthology.orchestra.run.smithi189.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-22T08:10:50.678 INFO:teuthology.orchestra.run.smithi189.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-22T08:10:50.678 INFO:teuthology.orchestra.run.smithi189.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-22T08:10:50.678 INFO:teuthology.orchestra.run.smithi189.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-22T08:10:50.678 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-22T08:10:50.678 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-22T08:10:50.678 INFO:teuthology.orchestra.run.smithi189.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-22T08:10:50.678 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:10:50.678 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-22T08:10:50.678 INFO:teuthology.orchestra.run.smithi189.stdout:======================================================================================= 2024-04-22T08:10:50.679 INFO:teuthology.orchestra.run.smithi189.stdout:Install 12 Packages 2024-04-22T08:10:50.679 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:10:50.679 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 38 M 2024-04-22T08:10:50.679 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 209 M 2024-04-22T08:10:50.679 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-22T08:10:51.327 INFO:teuthology.orchestra.run.smithi189.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 791 kB/s | 195 kB 00:00 2024-04-22T08:10:51.444 INFO:teuthology.orchestra.run.smithi189.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.2 MB/s | 803 kB 00:00 2024-04-22T08:10:51.836 INFO:teuthology.orchestra.run.smithi189.stdout:(3/12): ceph-mgr-diskprediction-local-18.2.2-12 9.8 MB/s | 7.4 MB 00:00 2024-04-22T08:10:51.927 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-22T08:10:52.037 INFO:teuthology.orchestra.run.smithi189.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 46 kB/s | 33 kB 00:00 2024-04-22T08:10:52.395 INFO:teuthology.orchestra.run.smithi189.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 33 kB/s | 18 kB 00:00 2024-04-22T08:10:52.695 INFO:teuthology.orchestra.run.smithi189.stdout:(6/12): openblas-0.3.21-2.el9.x86_64.rpm 53 kB/s | 35 kB 00:00 2024-04-22T08:10:53.014 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-22T08:10:53.045 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1237.g646b8 12/12 2024-04-22T08:10:53.638 INFO:teuthology.orchestra.run.smithi189.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 259 kB/s | 244 kB 00:00 2024-04-22T08:10:55.647 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1237.g646b8 12/12 2024-04-22T08:10:55.647 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1237.g646b8 1/12 2024-04-22T08:10:55.648 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-22T08:10:55.648 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-04-22T08:10:55.648 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-04-22T08:10:55.648 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-04-22T08:10:55.648 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-04-22T08:10:55.648 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-04-22T08:10:55.648 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-04-22T08:10:55.648 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-04-22T08:10:55.648 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-04-22T08:10:55.648 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-04-22T08:10:56.038 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-04-22T08:10:56.038 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:10:56.038 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-04-22T08:10:56.038 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1237.g646b8c7d.el8.noarch 2024-04-22T08:10:56.038 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-22T08:10:56.039 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-22T08:10:56.039 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-22T08:10:56.039 INFO:teuthology.orchestra.run.smithi040.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-22T08:10:56.039 INFO:teuthology.orchestra.run.smithi040.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-22T08:10:56.039 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-22T08:10:56.039 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-22T08:10:56.039 INFO:teuthology.orchestra.run.smithi040.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-22T08:10:56.039 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-22T08:10:56.039 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-22T08:10:56.039 INFO:teuthology.orchestra.run.smithi040.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-22T08:10:56.039 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:10:56.039 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-22T08:10:56.137 INFO:teuthology.orchestra.run.smithi189.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 652 kB/s | 3.0 MB 00:04 2024-04-22T08:10:56.461 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-mgr-rook 2024-04-22T08:10:56.990 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:16 ago on Mon 22 Apr 2024 08:09:40 AM UTC. 2024-04-22T08:10:57.046 INFO:teuthology.orchestra.run.smithi189.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 276 kB/s | 250 kB 00:00 2024-04-22T08:10:57.116 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-22T08:10:57.118 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:10:57.118 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2024-04-22T08:10:57.118 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:10:57.118 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-04-22T08:10:57.118 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-rook noarch 2:18.2.2-1237.g646b8c7d.el8 ceph-noarch 49 k 2024-04-22T08:10:57.118 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2024-04-22T08:10:57.118 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-22T08:10:57.118 INFO:teuthology.orchestra.run.smithi040.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-22T08:10:57.119 INFO:teuthology.orchestra.run.smithi040.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-22T08:10:57.119 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-22T08:10:57.119 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-22T08:10:57.119 INFO:teuthology.orchestra.run.smithi040.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-22T08:10:57.119 INFO:teuthology.orchestra.run.smithi040.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-22T08:10:57.119 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-22T08:10:57.119 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-22T08:10:57.120 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-22T08:10:57.120 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-22T08:10:57.120 INFO:teuthology.orchestra.run.smithi040.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-22T08:10:57.120 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:10:57.120 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-04-22T08:10:57.120 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:10:57.120 INFO:teuthology.orchestra.run.smithi040.stdout:Install 13 Packages 2024-04-22T08:10:57.121 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:10:57.122 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 2.2 M 2024-04-22T08:10:57.122 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 26 M 2024-04-22T08:10:57.122 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-04-22T08:10:57.737 INFO:teuthology.orchestra.run.smithi040.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 141 kB/s | 26 kB 00:00 2024-04-22T08:10:57.763 INFO:teuthology.orchestra.run.smithi040.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 93 kB/s | 19 kB 00:00 2024-04-22T08:10:57.863 INFO:teuthology.orchestra.run.smithi040.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.7 MB/s | 222 kB 00:00 2024-04-22T08:10:57.888 INFO:teuthology.orchestra.run.smithi040.stdout:(4/13): ceph-mgr-rook-18.2.2-1237.g646b8c7d.el8 148 kB/s | 49 kB 00:00 2024-04-22T08:10:57.914 INFO:teuthology.orchestra.run.smithi040.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.0 MB/s | 159 kB 00:00 2024-04-22T08:10:57.947 INFO:teuthology.orchestra.run.smithi040.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.2 MB/s | 279 kB 00:00 2024-04-22T08:10:57.973 INFO:teuthology.orchestra.run.smithi040.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 639 kB/s | 54 kB 00:00 2024-04-22T08:10:58.173 INFO:teuthology.orchestra.run.smithi040.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 124 kB/s | 32 kB 00:00 2024-04-22T08:10:58.198 INFO:teuthology.orchestra.run.smithi040.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 60 kB/s | 15 kB 00:00 2024-04-22T08:10:58.357 INFO:teuthology.orchestra.run.smithi040.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 373 kB/s | 59 kB 00:00 2024-04-22T08:10:58.442 INFO:teuthology.orchestra.run.smithi040.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 485 kB/s | 203 kB 00:00 2024-04-22T08:10:58.774 INFO:teuthology.orchestra.run.smithi040.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 215 kB/s | 90 kB 00:00 2024-04-22T08:10:58.804 INFO:teuthology.orchestra.run.smithi189.stdout:(10/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 757 kB/s | 4.7 MB 00:06 2024-04-22T08:10:59.191 INFO:teuthology.orchestra.run.smithi040.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.0 MB/s | 1.0 MB 00:01 2024-04-22T08:10:59.197 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-04-22T08:10:59.197 INFO:teuthology.orchestra.run.smithi040.stdout:Total 1.1 MB/s | 2.2 MB 00:02 2024-04-22T08:10:59.266 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-04-22T08:10:59.282 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-04-22T08:10:59.282 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-04-22T08:10:59.375 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-04-22T08:10:59.376 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-04-22T08:10:59.976 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-04-22T08:11:00.167 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-22T08:11:00.220 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-22T08:11:00.276 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-22T08:11:00.311 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-22T08:11:00.353 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-22T08:11:00.452 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-22T08:11:00.563 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-22T08:11:00.596 INFO:teuthology.orchestra.run.smithi189.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 750 kB/s | 5.1 MB 00:06 2024-04-22T08:11:00.613 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-22T08:11:01.394 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-22T08:11:01.443 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-22T08:11:01.477 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-22T08:11:01.526 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-22T08:11:01.545 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-rook-2:18.2.2-1237.g646b8c7d.el8.noarch 13/13 2024-04-22T08:11:02.264 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1237.g646b8c7d.el8.noarch 13/13 2024-04-22T08:11:02.264 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1237.g646b8c7d.el8.noarch 1/13 2024-04-22T08:11:02.264 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-22T08:11:02.264 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-22T08:11:02.264 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-22T08:11:02.264 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-22T08:11:02.264 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-22T08:11:02.264 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-22T08:11:02.265 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-22T08:11:02.265 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-22T08:11:02.265 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-22T08:11:02.265 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-22T08:11:02.265 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-22T08:11:02.566 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-22T08:11:02.566 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:11:02.566 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-04-22T08:11:02.566 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-rook-2:18.2.2-1237.g646b8c7d.el8.noarch 2024-04-22T08:11:02.566 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-22T08:11:02.566 INFO:teuthology.orchestra.run.smithi040.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-22T08:11:02.566 INFO:teuthology.orchestra.run.smithi040.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-22T08:11:02.566 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-22T08:11:02.566 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-22T08:11:02.566 INFO:teuthology.orchestra.run.smithi040.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-22T08:11:02.566 INFO:teuthology.orchestra.run.smithi040.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-22T08:11:02.566 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-22T08:11:02.567 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-22T08:11:02.567 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-22T08:11:02.567 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-22T08:11:02.567 INFO:teuthology.orchestra.run.smithi040.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-22T08:11:02.567 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:11:02.567 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-22T08:11:02.798 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-mgr-cephadm 2024-04-22T08:11:03.332 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:23 ago on Mon 22 Apr 2024 08:09:40 AM UTC. 2024-04-22T08:11:03.460 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-22T08:11:03.461 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:11:03.461 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2024-04-22T08:11:03.461 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:11:03.461 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-04-22T08:11:03.461 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1237.g646b8c7d.el8 ceph-noarch 138 k 2024-04-22T08:11:03.461 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2024-04-22T08:11:03.461 INFO:teuthology.orchestra.run.smithi040.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-22T08:11:03.461 INFO:teuthology.orchestra.run.smithi040.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-22T08:11:03.461 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-22T08:11:03.461 INFO:teuthology.orchestra.run.smithi040.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-22T08:11:03.462 INFO:teuthology.orchestra.run.smithi040.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-22T08:11:03.462 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:11:03.462 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-04-22T08:11:03.462 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:11:03.462 INFO:teuthology.orchestra.run.smithi040.stdout:Install 6 Packages 2024-04-22T08:11:03.462 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:11:03.462 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 7.0 M 2024-04-22T08:11:03.463 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 33 M 2024-04-22T08:11:03.463 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-04-22T08:11:04.302 INFO:teuthology.orchestra.run.smithi040.stdout:(1/6): ceph-mgr-cephadm-18.2.2-1237.g646b8c7d.e 363 kB/s | 138 kB 00:00 2024-04-22T08:11:04.460 INFO:teuthology.orchestra.run.smithi040.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 3.4 MB/s | 548 kB 00:00 2024-04-22T08:11:04.502 INFO:teuthology.orchestra.run.smithi040.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.4 MB/s | 58 kB 00:00 2024-04-22T08:11:04.544 INFO:teuthology.orchestra.run.smithi040.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 1.2 MB/s | 51 kB 00:00 2024-04-22T08:11:04.928 INFO:teuthology.orchestra.run.smithi040.stdout:(5/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 246 kB/s | 248 kB 00:01 2024-04-22T08:11:11.844 INFO:teuthology.orchestra.run.smithi040.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 771 kB/s | 6.0 MB 00:07 2024-04-22T08:11:11.845 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-04-22T08:11:11.846 INFO:teuthology.orchestra.run.smithi040.stdout:Total 853 kB/s | 7.0 MB 00:08 2024-04-22T08:11:11.920 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-04-22T08:11:11.934 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-04-22T08:11:11.934 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-04-22T08:11:12.037 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-04-22T08:11:12.037 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-04-22T08:11:12.348 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-04-22T08:11:12.528 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-22T08:11:12.580 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-22T08:11:13.139 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-22T08:11:13.229 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-22T08:11:13.297 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-22T08:11:13.324 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1237.g646b8c7d.el8.noarch 6/6 2024-04-22T08:11:14.106 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1237.g646b8c7d.el8.noarch 6/6 2024-04-22T08:11:14.106 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1237.g646b8c7d.el8.noarch 1/6 2024-04-22T08:11:14.106 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-22T08:11:14.106 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-22T08:11:14.106 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-22T08:11:14.106 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-22T08:11:14.347 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-22T08:11:14.347 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:11:14.347 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-04-22T08:11:14.347 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-cephadm-2:18.2.2-1237.g646b8c7d.el8.noarch 2024-04-22T08:11:14.347 INFO:teuthology.orchestra.run.smithi040.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-22T08:11:14.347 INFO:teuthology.orchestra.run.smithi040.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-22T08:11:14.348 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-22T08:11:14.348 INFO:teuthology.orchestra.run.smithi040.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-22T08:11:14.348 INFO:teuthology.orchestra.run.smithi040.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-22T08:11:14.348 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:11:14.348 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-22T08:11:14.557 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-fuse 2024-04-22T08:11:15.092 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:35 ago on Mon 22 Apr 2024 08:09:40 AM UTC. 2024-04-22T08:11:15.218 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-22T08:11:15.218 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:11:15.218 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2024-04-22T08:11:15.218 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:11:15.218 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-04-22T08:11:15.218 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-fuse x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 848 k 2024-04-22T08:11:15.218 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2024-04-22T08:11:15.219 INFO:teuthology.orchestra.run.smithi040.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-22T08:11:15.219 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:11:15.219 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-04-22T08:11:15.219 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:11:15.219 INFO:teuthology.orchestra.run.smithi040.stdout:Install 2 Packages 2024-04-22T08:11:15.219 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:11:15.219 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 928 k 2024-04-22T08:11:15.220 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 2.7 M 2024-04-22T08:11:15.220 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-04-22T08:11:15.496 INFO:teuthology.orchestra.run.smithi040.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 644 kB/s | 80 kB 00:00 2024-04-22T08:11:15.912 INFO:teuthology.orchestra.run.smithi040.stdout:(2/2): ceph-fuse-18.2.2-1237.g646b8c7d.el8.x86_ 1.5 MB/s | 848 kB 00:00 2024-04-22T08:11:15.913 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-04-22T08:11:15.913 INFO:teuthology.orchestra.run.smithi040.stdout:Total 1.3 MB/s | 928 kB 00:00 2024-04-22T08:11:15.925 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-04-22T08:11:15.949 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-04-22T08:11:15.949 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-04-22T08:11:16.031 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-04-22T08:11:16.032 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-04-22T08:11:16.221 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-04-22T08:11:16.365 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-22T08:11:16.430 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-fuse-2:18.2.2-1237.g646b8c7d.el8.x86_64 2/2 2024-04-22T08:11:17.041 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-fuse-2:18.2.2-1237.g646b8c7d.el8.x86_64 2/2 2024-04-22T08:11:17.041 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-fuse-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/2 2024-04-22T08:11:17.297 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-22T08:11:17.297 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:11:17.297 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-04-22T08:11:17.298 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-fuse-2:18.2.2-1237.g646b8c7d.el8.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-22T08:11:17.298 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:11:17.298 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-22T08:11:17.550 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-volume 2024-04-22T08:11:18.092 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:38 ago on Mon 22 Apr 2024 08:09:40 AM UTC. 2024-04-22T08:11:18.219 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-22T08:11:18.219 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:11:18.219 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2024-04-22T08:11:18.220 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:11:18.220 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-04-22T08:11:18.220 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-volume noarch 2:18.2.2-1237.g646b8c7d.el8 ceph-noarch 264 k 2024-04-22T08:11:18.220 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2024-04-22T08:11:18.220 INFO:teuthology.orchestra.run.smithi040.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-22T08:11:18.221 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:11:18.221 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-04-22T08:11:18.221 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:11:18.221 INFO:teuthology.orchestra.run.smithi040.stdout:Install 2 Packages 2024-04-22T08:11:18.221 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:11:18.222 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 341 k 2024-04-22T08:11:18.222 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 1.5 M 2024-04-22T08:11:18.222 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-04-22T08:11:18.393 INFO:teuthology.orchestra.run.smithi189.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 780 kB/s | 16 MB 00:21 2024-04-22T08:11:18.395 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-22T08:11:18.395 INFO:teuthology.orchestra.run.smithi189.stdout:Total 1.4 MB/s | 38 MB 00:27 2024-04-22T08:11:18.744 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-22T08:11:18.772 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-22T08:11:18.772 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-22T08:11:18.892 INFO:teuthology.orchestra.run.smithi040.stdout:(1/2): ceph-volume-18.2.2-1237.g646b8c7d.el8.no 842 kB/s | 264 kB 00:00 2024-04-22T08:11:18.917 INFO:teuthology.orchestra.run.smithi040.stdout:(2/2): python3-packaging-20.9-5.el9.noarch.rpm 228 kB/s | 77 kB 00:00 2024-04-22T08:11:18.918 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-04-22T08:11:18.918 INFO:teuthology.orchestra.run.smithi040.stdout:Total 488 kB/s | 341 kB 00:00 2024-04-22T08:11:18.930 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-04-22T08:11:18.940 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-04-22T08:11:18.940 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-04-22T08:11:18.967 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-04-22T08:11:18.967 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-04-22T08:11:19.115 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-04-22T08:11:19.177 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-22T08:11:19.178 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-22T08:11:19.301 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-22T08:11:19.336 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-volume-2:18.2.2-1237.g646b8c7d.el8.noarch 2/2 2024-04-22T08:11:19.353 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-volume-2:18.2.2-1237.g646b8c7d.el8.noarch 2/2 2024-04-22T08:11:19.353 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T08:11:19.353 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-22T08:11:19.353 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:11:19.936 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-22T08:11:20.006 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-04-22T08:11:20.037 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-volume-2:18.2.2-1237.g646b8c7d.el8.noarch 1/2 2024-04-22T08:11:20.050 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-04-22T08:11:20.157 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-04-22T08:11:20.193 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-04-22T08:11:20.323 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-22T08:11:20.324 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:11:20.324 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-04-22T08:11:20.324 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-volume-2:18.2.2-1237.g646b8c7d.el8.noarch 2024-04-22T08:11:20.324 INFO:teuthology.orchestra.run.smithi040.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-22T08:11:20.324 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:11:20.324 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-22T08:11:20.370 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-22T08:11:20.400 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-22T08:11:20.505 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-22T08:11:20.672 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install librados-devel 2024-04-22T08:11:21.144 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-22T08:11:21.217 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:41 ago on Mon 22 Apr 2024 08:09:40 AM UTC. 2024-04-22T08:11:21.255 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-22T08:11:21.341 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-22T08:11:21.342 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:11:21.342 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repo Size 2024-04-22T08:11:21.342 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:11:21.342 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-04-22T08:11:21.342 INFO:teuthology.orchestra.run.smithi040.stdout: librados-devel x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 127 k 2024-04-22T08:11:21.342 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:11:21.342 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-04-22T08:11:21.342 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:11:21.342 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2024-04-22T08:11:21.342 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:11:21.343 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 127 k 2024-04-22T08:11:21.343 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 456 k 2024-04-22T08:11:21.343 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-04-22T08:11:21.634 INFO:teuthology.orchestra.run.smithi040.stdout:librados-devel-18.2.2-1237.g646b8c7d.el8.x86_64 437 kB/s | 127 kB 00:00 2024-04-22T08:11:21.635 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-04-22T08:11:21.635 INFO:teuthology.orchestra.run.smithi040.stdout:Total 434 kB/s | 127 kB 00:00 2024-04-22T08:11:21.635 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-04-22T08:11:21.641 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-04-22T08:11:21.641 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-04-22T08:11:21.707 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-04-22T08:11:21.707 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-04-22T08:11:21.865 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-04-22T08:11:21.974 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librados-devel-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-22T08:11:22.361 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librados-devel-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-22T08:11:22.484 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-22T08:11:22.620 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados-devel-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-22T08:11:22.620 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:11:22.620 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-04-22T08:11:22.620 INFO:teuthology.orchestra.run.smithi040.stdout: librados-devel-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:11:22.620 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:11:22.620 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-22T08:11:22.894 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install libcephfs2 2024-04-22T08:11:23.430 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:43 ago on Mon 22 Apr 2024 08:09:40 AM UTC. 2024-04-22T08:11:23.511 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-22T08:11:23.512 INFO:teuthology.orchestra.run.smithi040.stdout:Package libcephfs2-2:18.2.2-1237.g646b8c7d.el8.x86_64 is already installed. 2024-04-22T08:11:23.541 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1237.g646b8 12/12 2024-04-22T08:11:23.555 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-22T08:11:23.557 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2024-04-22T08:11:23.557 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-22T08:11:23.641 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install libcephfs-devel 2024-04-22T08:11:24.175 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:44 ago on Mon 22 Apr 2024 08:09:40 AM UTC. 2024-04-22T08:11:24.301 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-22T08:11:24.302 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:11:24.302 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repo Size 2024-04-22T08:11:24.302 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:11:24.302 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-04-22T08:11:24.302 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs-devel x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 31 k 2024-04-22T08:11:24.302 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:11:24.302 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-04-22T08:11:24.302 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:11:24.302 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2024-04-22T08:11:24.302 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:11:24.302 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 31 k 2024-04-22T08:11:24.303 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 139 k 2024-04-22T08:11:24.303 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-04-22T08:11:24.704 INFO:teuthology.orchestra.run.smithi040.stdout:libcephfs-devel-18.2.2-1237.g646b8c7d.el8.x86_6 77 kB/s | 31 kB 00:00 2024-04-22T08:11:24.705 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-04-22T08:11:24.705 INFO:teuthology.orchestra.run.smithi040.stdout:Total 77 kB/s | 31 kB 00:00 2024-04-22T08:11:24.705 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-04-22T08:11:24.709 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-04-22T08:11:24.709 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-04-22T08:11:24.711 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-04-22T08:11:24.711 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-04-22T08:11:24.801 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-04-22T08:11:24.912 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephfs-devel-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-22T08:11:25.221 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libcephfs-devel-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-22T08:11:25.429 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephfs-devel-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-22T08:11:25.429 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:11:25.429 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-04-22T08:11:25.429 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs-devel-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:11:25.429 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:11:25.429 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-22T08:11:25.619 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install librados2 2024-04-22T08:11:26.047 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1237.g646b8 12/12 2024-04-22T08:11:26.048 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1237.g646b8 1/12 2024-04-22T08:11:26.048 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-22T08:11:26.048 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-04-22T08:11:26.048 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-04-22T08:11:26.048 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-04-22T08:11:26.048 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-04-22T08:11:26.048 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-04-22T08:11:26.048 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-04-22T08:11:26.048 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-04-22T08:11:26.048 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-04-22T08:11:26.048 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-04-22T08:11:26.152 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:46 ago on Mon 22 Apr 2024 08:09:40 AM UTC. 2024-04-22T08:11:26.234 INFO:teuthology.orchestra.run.smithi040.stdout:Package librados2-2:18.2.2-1237.g646b8c7d.el8.x86_64 is already installed. 2024-04-22T08:11:26.277 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-22T08:11:26.278 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2024-04-22T08:11:26.278 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-22T08:11:26.356 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install librbd1 2024-04-22T08:11:26.457 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-04-22T08:11:26.457 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:11:26.457 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-22T08:11:26.457 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1237.g646b8c7d.el8.noarch 2024-04-22T08:11:26.457 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-22T08:11:26.457 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-22T08:11:26.458 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-22T08:11:26.458 INFO:teuthology.orchestra.run.smithi189.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-22T08:11:26.458 INFO:teuthology.orchestra.run.smithi189.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-22T08:11:26.458 INFO:teuthology.orchestra.run.smithi189.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-22T08:11:26.458 INFO:teuthology.orchestra.run.smithi189.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-22T08:11:26.458 INFO:teuthology.orchestra.run.smithi189.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-22T08:11:26.458 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-22T08:11:26.458 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-22T08:11:26.458 INFO:teuthology.orchestra.run.smithi189.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-22T08:11:26.458 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:11:26.458 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-22T08:11:26.856 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-mgr-rook 2024-04-22T08:11:26.892 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:46 ago on Mon 22 Apr 2024 08:09:40 AM UTC. 2024-04-22T08:11:26.973 INFO:teuthology.orchestra.run.smithi040.stdout:Package librbd1-2:18.2.2-1237.g646b8c7d.el8.x86_64 is already installed. 2024-04-22T08:11:27.016 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-22T08:11:27.017 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2024-04-22T08:11:27.017 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-22T08:11:27.100 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install python3-rados 2024-04-22T08:11:27.373 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:38 ago on Mon 22 Apr 2024 08:09:49 AM UTC. 2024-04-22T08:11:27.496 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-22T08:11:27.497 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:11:27.497 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repository Size 2024-04-22T08:11:27.497 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:11:27.497 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-22T08:11:27.497 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-rook noarch 2:18.2.2-1237.g646b8c7d.el8 ceph-noarch 49 k 2024-04-22T08:11:27.497 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-04-22T08:11:27.497 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-22T08:11:27.498 INFO:teuthology.orchestra.run.smithi189.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-22T08:11:27.498 INFO:teuthology.orchestra.run.smithi189.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-22T08:11:27.498 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-22T08:11:27.498 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-22T08:11:27.498 INFO:teuthology.orchestra.run.smithi189.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-22T08:11:27.498 INFO:teuthology.orchestra.run.smithi189.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-22T08:11:27.498 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-22T08:11:27.498 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-22T08:11:27.498 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-22T08:11:27.498 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-22T08:11:27.498 INFO:teuthology.orchestra.run.smithi189.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-22T08:11:27.499 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:11:27.499 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-22T08:11:27.499 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:11:27.499 INFO:teuthology.orchestra.run.smithi189.stdout:Install 13 Packages 2024-04-22T08:11:27.499 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:11:27.499 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 2.2 M 2024-04-22T08:11:27.499 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 26 M 2024-04-22T08:11:27.500 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-22T08:11:27.639 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:47 ago on Mon 22 Apr 2024 08:09:40 AM UTC. 2024-04-22T08:11:27.721 INFO:teuthology.orchestra.run.smithi040.stdout:Package python3-rados-2:18.2.2-1237.g646b8c7d.el8.x86_64 is already installed. 2024-04-22T08:11:27.763 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-22T08:11:27.765 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2024-04-22T08:11:27.765 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-22T08:11:27.841 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install python3-rgw 2024-04-22T08:11:28.141 INFO:teuthology.orchestra.run.smithi189.stdout:(1/13): ceph-mgr-rook-18.2.2-1237.g646b8c7d.el8 252 kB/s | 49 kB 00:00 2024-04-22T08:11:28.208 INFO:teuthology.orchestra.run.smithi189.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 73 kB/s | 19 kB 00:00 2024-04-22T08:11:28.233 INFO:teuthology.orchestra.run.smithi189.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 89 kB/s | 26 kB 00:00 2024-04-22T08:11:28.389 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:48 ago on Mon 22 Apr 2024 08:09:40 AM UTC. 2024-04-22T08:11:28.471 INFO:teuthology.orchestra.run.smithi040.stdout:Package python3-rgw-2:18.2.2-1237.g646b8c7d.el8.x86_64 is already installed. 2024-04-22T08:11:28.515 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-22T08:11:28.516 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2024-04-22T08:11:28.517 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-22T08:11:28.558 INFO:teuthology.orchestra.run.smithi189.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 453 kB/s | 159 kB 00:00 2024-04-22T08:11:28.589 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install python3-cephfs 2024-04-22T08:11:28.659 INFO:teuthology.orchestra.run.smithi189.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 429 kB/s | 222 kB 00:00 2024-04-22T08:11:28.684 INFO:teuthology.orchestra.run.smithi189.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 428 kB/s | 54 kB 00:00 2024-04-22T08:11:28.709 INFO:teuthology.orchestra.run.smithi189.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 586 kB/s | 279 kB 00:00 2024-04-22T08:11:29.132 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:49 ago on Mon 22 Apr 2024 08:09:40 AM UTC. 2024-04-22T08:11:29.134 INFO:teuthology.orchestra.run.smithi189.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 68 kB/s | 32 kB 00:00 2024-04-22T08:11:29.160 INFO:teuthology.orchestra.run.smithi189.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 31 kB/s | 15 kB 00:00 2024-04-22T08:11:29.212 INFO:teuthology.orchestra.run.smithi040.stdout:Package python3-cephfs-2:18.2.2-1237.g646b8c7d.el8.x86_64 is already installed. 2024-04-22T08:11:29.255 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-22T08:11:29.256 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2024-04-22T08:11:29.256 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-22T08:11:29.333 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install python3-rbd 2024-04-22T08:11:29.419 INFO:teuthology.orchestra.run.smithi189.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 229 kB/s | 59 kB 00:00 2024-04-22T08:11:29.494 INFO:teuthology.orchestra.run.smithi189.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 258 kB/s | 203 kB 00:00 2024-04-22T08:11:29.610 INFO:teuthology.orchestra.run.smithi189.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 468 kB/s | 90 kB 00:00 2024-04-22T08:11:29.871 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:49 ago on Mon 22 Apr 2024 08:09:40 AM UTC. 2024-04-22T08:11:29.953 INFO:teuthology.orchestra.run.smithi040.stdout:Package python3-rbd-2:18.2.2-1237.g646b8c7d.el8.x86_64 is already installed. 2024-04-22T08:11:29.998 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-22T08:11:29.998 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2024-04-22T08:11:29.998 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-22T08:11:30.079 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install rbd-fuse 2024-04-22T08:11:30.436 INFO:teuthology.orchestra.run.smithi189.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 803 kB/s | 1.0 MB 00:01 2024-04-22T08:11:30.439 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-22T08:11:30.440 INFO:teuthology.orchestra.run.smithi189.stdout:Total 765 kB/s | 2.2 MB 00:02 2024-04-22T08:11:30.509 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-22T08:11:30.525 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-22T08:11:30.525 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-22T08:11:30.611 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-22T08:11:30.611 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-22T08:11:30.614 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:50 ago on Mon 22 Apr 2024 08:09:40 AM UTC. 2024-04-22T08:11:30.740 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-22T08:11:30.740 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:11:30.740 INFO:teuthology.orchestra.run.smithi040.stdout: Package Architecture Version Repository Size 2024-04-22T08:11:30.741 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:11:30.741 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-04-22T08:11:30.741 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-fuse x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 87 k 2024-04-22T08:11:30.741 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:11:30.741 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-04-22T08:11:30.741 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:11:30.741 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2024-04-22T08:11:30.741 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:11:30.741 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 87 k 2024-04-22T08:11:30.741 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 227 k 2024-04-22T08:11:30.741 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-04-22T08:11:31.074 INFO:teuthology.orchestra.run.smithi040.stdout:rbd-fuse-18.2.2-1237.g646b8c7d.el8.x86_64.rpm 260 kB/s | 87 kB 00:00 2024-04-22T08:11:31.075 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-04-22T08:11:31.075 INFO:teuthology.orchestra.run.smithi040.stdout:Total 259 kB/s | 87 kB 00:00 2024-04-22T08:11:31.075 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-04-22T08:11:31.083 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-04-22T08:11:31.083 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-04-22T08:11:31.149 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-04-22T08:11:31.150 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-04-22T08:11:31.213 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-22T08:11:31.298 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-04-22T08:11:31.376 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-22T08:11:31.397 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-fuse-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-22T08:11:31.429 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-22T08:11:31.485 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-22T08:11:31.522 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-22T08:11:31.563 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-22T08:11:31.657 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-22T08:11:31.733 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: rbd-fuse-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-22T08:11:31.767 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-22T08:11:31.815 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-22T08:11:31.969 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-fuse-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-22T08:11:31.970 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:11:31.970 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-04-22T08:11:31.970 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-fuse-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:11:31.970 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:11:31.970 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-22T08:11:32.200 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install rbd-mirror 2024-04-22T08:11:32.550 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-22T08:11:32.612 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-22T08:11:32.646 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-22T08:11:32.694 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-22T08:11:32.732 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-rook-2:18.2.2-1237.g646b8c7d.el8.noarch 13/13 2024-04-22T08:11:32.739 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:52 ago on Mon 22 Apr 2024 08:09:40 AM UTC. 2024-04-22T08:11:32.865 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-22T08:11:32.866 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:11:32.866 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repo Size 2024-04-22T08:11:32.866 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:11:32.866 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-04-22T08:11:32.867 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-mirror x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 3.0 M 2024-04-22T08:11:32.867 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:11:32.867 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-04-22T08:11:32.867 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:11:32.867 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2024-04-22T08:11:32.867 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:11:32.867 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 3.0 M 2024-04-22T08:11:32.867 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 12 M 2024-04-22T08:11:32.867 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-04-22T08:11:33.386 INFO:teuthology.orchestra.run.smithi040.stdout:rbd-mirror-18.2.2-1237.g646b8c7d.el8.x86_64.rpm 5.8 MB/s | 3.0 MB 00:00 2024-04-22T08:11:33.387 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-04-22T08:11:33.387 INFO:teuthology.orchestra.run.smithi040.stdout:Total 5.7 MB/s | 3.0 MB 00:00 2024-04-22T08:11:33.388 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-04-22T08:11:33.398 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-04-22T08:11:33.398 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-04-22T08:11:33.484 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1237.g646b8c7d.el8.noarch 13/13 2024-04-22T08:11:33.484 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1237.g646b8c7d.el8.noarch 1/13 2024-04-22T08:11:33.484 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-22T08:11:33.485 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-22T08:11:33.485 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-22T08:11:33.485 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-22T08:11:33.485 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-22T08:11:33.485 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-22T08:11:33.485 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-22T08:11:33.485 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-22T08:11:33.485 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-22T08:11:33.485 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-22T08:11:33.485 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-22T08:11:33.486 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-04-22T08:11:33.487 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-04-22T08:11:33.776 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-22T08:11:33.776 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:11:33.776 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-22T08:11:33.776 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-rook-2:18.2.2-1237.g646b8c7d.el8.noarch 2024-04-22T08:11:33.777 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-22T08:11:33.777 INFO:teuthology.orchestra.run.smithi189.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-22T08:11:33.777 INFO:teuthology.orchestra.run.smithi189.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-22T08:11:33.777 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-22T08:11:33.777 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-22T08:11:33.777 INFO:teuthology.orchestra.run.smithi189.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-22T08:11:33.777 INFO:teuthology.orchestra.run.smithi189.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-22T08:11:33.777 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-22T08:11:33.777 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-22T08:11:33.777 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-22T08:11:33.777 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-22T08:11:33.777 INFO:teuthology.orchestra.run.smithi189.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-22T08:11:33.777 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:11:33.777 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-22T08:11:33.932 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-04-22T08:11:33.995 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-mirror-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-22T08:11:34.000 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-mgr-cephadm 2024-04-22T08:11:34.021 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-22T08:11:34.021 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T08:11:34.021 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-22T08:11:34.022 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T08:11:34.022 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T08:11:34.022 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:11:34.531 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:45 ago on Mon 22 Apr 2024 08:09:49 AM UTC. 2024-04-22T08:11:34.653 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-22T08:11:34.653 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:11:34.654 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repository Size 2024-04-22T08:11:34.654 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:11:34.654 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-22T08:11:34.654 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1237.g646b8c7d.el8 ceph-noarch 138 k 2024-04-22T08:11:34.654 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-04-22T08:11:34.654 INFO:teuthology.orchestra.run.smithi189.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-22T08:11:34.654 INFO:teuthology.orchestra.run.smithi189.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-22T08:11:34.654 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-22T08:11:34.654 INFO:teuthology.orchestra.run.smithi189.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-22T08:11:34.654 INFO:teuthology.orchestra.run.smithi189.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-22T08:11:34.654 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:11:34.654 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-22T08:11:34.654 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:11:34.655 INFO:teuthology.orchestra.run.smithi189.stdout:Install 6 Packages 2024-04-22T08:11:34.655 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:11:34.655 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 7.0 M 2024-04-22T08:11:34.655 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 33 M 2024-04-22T08:11:34.655 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-22T08:11:35.012 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-mirror-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-22T08:11:35.012 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:11:35.012 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-04-22T08:11:35.012 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-mirror-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:11:35.013 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:11:35.013 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-22T08:11:35.249 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install rbd-nbd 2024-04-22T08:11:35.255 INFO:teuthology.orchestra.run.smithi189.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.3 MB/s | 248 kB 00:00 2024-04-22T08:11:35.389 INFO:teuthology.orchestra.run.smithi189.stdout:(2/6): ceph-mgr-cephadm-18.2.2-1237.g646b8c7d.e 437 kB/s | 138 kB 00:00 2024-04-22T08:11:35.697 INFO:teuthology.orchestra.run.smithi189.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 187 kB/s | 58 kB 00:00 2024-04-22T08:11:35.784 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:55 ago on Mon 22 Apr 2024 08:09:40 AM UTC. 2024-04-22T08:11:35.798 INFO:teuthology.orchestra.run.smithi189.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.0 MB/s | 548 kB 00:00 2024-04-22T08:11:35.864 INFO:teuthology.orchestra.run.smithi189.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 305 kB/s | 51 kB 00:00 2024-04-22T08:11:35.912 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-22T08:11:35.912 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:11:35.912 INFO:teuthology.orchestra.run.smithi040.stdout: Package Architecture Version Repository Size 2024-04-22T08:11:35.913 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:11:35.913 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-04-22T08:11:35.913 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-nbd x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 172 k 2024-04-22T08:11:35.913 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:11:35.913 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-04-22T08:11:35.913 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-22T08:11:35.913 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2024-04-22T08:11:35.913 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:11:35.913 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 172 k 2024-04-22T08:11:35.913 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 494 k 2024-04-22T08:11:35.913 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-04-22T08:11:36.140 INFO:teuthology.orchestra.run.smithi040.stdout:rbd-nbd-18.2.2-1237.g646b8c7d.el8.x86_64.rpm 760 kB/s | 172 kB 00:00 2024-04-22T08:11:36.140 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-04-22T08:11:36.140 INFO:teuthology.orchestra.run.smithi040.stdout:Total 755 kB/s | 172 kB 00:00 2024-04-22T08:11:36.141 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-04-22T08:11:36.149 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-04-22T08:11:36.149 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-04-22T08:11:36.215 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-04-22T08:11:36.215 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-04-22T08:11:36.375 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-04-22T08:11:36.480 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-nbd-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-22T08:11:36.598 INFO:teuthology.orchestra.run.smithi189.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 3.9 MB/s | 6.0 MB 00:01 2024-04-22T08:11:36.599 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-22T08:11:36.599 INFO:teuthology.orchestra.run.smithi189.stdout:Total 3.6 MB/s | 7.0 MB 00:01 2024-04-22T08:11:36.671 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-22T08:11:36.683 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-22T08:11:36.683 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-22T08:11:36.780 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-22T08:11:36.782 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-22T08:11:36.861 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: rbd-nbd-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-22T08:11:37.083 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-22T08:11:37.113 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-nbd-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-22T08:11:37.113 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:11:37.113 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-04-22T08:11:37.113 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-nbd-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:11:37.113 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:11:37.113 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-22T08:11:37.248 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-22T08:11:37.308 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-22T08:11:37.360 DEBUG:teuthology.parallel:result is None 2024-04-22T08:11:37.841 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-22T08:11:37.965 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-22T08:11:38.032 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-22T08:11:38.062 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1237.g646b8c7d.el8.noarch 6/6 2024-04-22T08:11:38.766 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1237.g646b8c7d.el8.noarch 6/6 2024-04-22T08:11:38.766 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1237.g646b8c7d.el8.noarch 1/6 2024-04-22T08:11:38.766 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-22T08:11:38.766 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-22T08:11:38.766 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-22T08:11:38.767 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-22T08:11:39.015 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-22T08:11:39.015 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:11:39.016 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-22T08:11:39.016 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-cephadm-2:18.2.2-1237.g646b8c7d.el8.noarch 2024-04-22T08:11:39.016 INFO:teuthology.orchestra.run.smithi189.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-22T08:11:39.016 INFO:teuthology.orchestra.run.smithi189.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-22T08:11:39.016 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-22T08:11:39.016 INFO:teuthology.orchestra.run.smithi189.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-22T08:11:39.016 INFO:teuthology.orchestra.run.smithi189.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-22T08:11:39.016 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:11:39.016 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-22T08:11:39.245 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-fuse 2024-04-22T08:11:39.763 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:50 ago on Mon 22 Apr 2024 08:09:49 AM UTC. 2024-04-22T08:11:39.883 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-22T08:11:39.884 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:11:39.884 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repository Size 2024-04-22T08:11:39.884 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:11:39.884 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-22T08:11:39.884 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-fuse x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 848 k 2024-04-22T08:11:39.884 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-04-22T08:11:39.885 INFO:teuthology.orchestra.run.smithi189.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-22T08:11:39.885 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:11:39.885 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-22T08:11:39.885 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:11:39.885 INFO:teuthology.orchestra.run.smithi189.stdout:Install 2 Packages 2024-04-22T08:11:39.885 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:11:39.886 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 928 k 2024-04-22T08:11:39.886 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 2.7 M 2024-04-22T08:11:39.886 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-22T08:11:40.302 INFO:teuthology.orchestra.run.smithi189.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 654 kB/s | 80 kB 00:00 2024-04-22T08:11:40.492 INFO:teuthology.orchestra.run.smithi189.stdout:(2/2): ceph-fuse-18.2.2-1237.g646b8c7d.el8.x86_ 2.6 MB/s | 848 kB 00:00 2024-04-22T08:11:40.493 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-22T08:11:40.493 INFO:teuthology.orchestra.run.smithi189.stdout:Total 1.5 MB/s | 928 kB 00:00 2024-04-22T08:11:40.504 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-22T08:11:40.528 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-22T08:11:40.528 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-22T08:11:40.602 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-22T08:11:40.603 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-22T08:11:40.804 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-22T08:11:40.949 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-22T08:11:41.005 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-fuse-2:18.2.2-1237.g646b8c7d.el8.x86_64 2/2 2024-04-22T08:11:41.615 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-fuse-2:18.2.2-1237.g646b8c7d.el8.x86_64 2/2 2024-04-22T08:11:41.615 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-fuse-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/2 2024-04-22T08:11:41.882 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-22T08:11:41.882 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:11:41.882 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-22T08:11:41.882 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-fuse-2:18.2.2-1237.g646b8c7d.el8.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-22T08:11:41.882 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:11:41.882 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-22T08:11:42.176 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-volume 2024-04-22T08:11:42.699 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:53 ago on Mon 22 Apr 2024 08:09:49 AM UTC. 2024-04-22T08:11:42.818 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-22T08:11:42.819 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:11:42.819 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repository Size 2024-04-22T08:11:42.819 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:11:42.819 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-22T08:11:42.819 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-volume noarch 2:18.2.2-1237.g646b8c7d.el8 ceph-noarch 264 k 2024-04-22T08:11:42.819 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-04-22T08:11:42.819 INFO:teuthology.orchestra.run.smithi189.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-22T08:11:42.819 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:11:42.819 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-22T08:11:42.819 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:11:42.819 INFO:teuthology.orchestra.run.smithi189.stdout:Install 2 Packages 2024-04-22T08:11:42.819 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:11:42.820 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 341 k 2024-04-22T08:11:42.820 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 1.5 M 2024-04-22T08:11:42.820 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-22T08:11:43.346 INFO:teuthology.orchestra.run.smithi189.stdout:(1/2): ceph-volume-18.2.2-1237.g646b8c7d.el8.no 672 kB/s | 264 kB 00:00 2024-04-22T08:11:43.638 INFO:teuthology.orchestra.run.smithi189.stdout:(2/2): python3-packaging-20.9-5.el9.noarch.rpm 113 kB/s | 77 kB 00:00 2024-04-22T08:11:43.639 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-22T08:11:43.639 INFO:teuthology.orchestra.run.smithi189.stdout:Total 416 kB/s | 341 kB 00:00 2024-04-22T08:11:43.651 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-22T08:11:43.662 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-22T08:11:43.662 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-22T08:11:43.689 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-22T08:11:43.689 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-22T08:11:43.830 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-22T08:11:44.004 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-22T08:11:44.041 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-volume-2:18.2.2-1237.g646b8c7d.el8.noarch 2/2 2024-04-22T08:11:44.056 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-volume-2:18.2.2-1237.g646b8c7d.el8.noarch 2/2 2024-04-22T08:11:44.056 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T08:11:44.056 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-22T08:11:44.057 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:11:44.751 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-volume-2:18.2.2-1237.g646b8c7d.el8.noarch 1/2 2024-04-22T08:11:45.014 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-22T08:11:45.015 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:11:45.015 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-22T08:11:45.015 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-volume-2:18.2.2-1237.g646b8c7d.el8.noarch 2024-04-22T08:11:45.015 INFO:teuthology.orchestra.run.smithi189.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-22T08:11:45.015 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:11:45.016 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-22T08:11:45.397 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install librados-devel 2024-04-22T08:11:45.919 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:56 ago on Mon 22 Apr 2024 08:09:49 AM UTC. 2024-04-22T08:11:46.043 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-22T08:11:46.044 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:11:46.044 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repo Size 2024-04-22T08:11:46.044 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:11:46.044 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-22T08:11:46.045 INFO:teuthology.orchestra.run.smithi189.stdout: librados-devel x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 127 k 2024-04-22T08:11:46.045 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:11:46.045 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-22T08:11:46.045 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:11:46.045 INFO:teuthology.orchestra.run.smithi189.stdout:Install 1 Package 2024-04-22T08:11:46.045 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:11:46.045 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 127 k 2024-04-22T08:11:46.045 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 456 k 2024-04-22T08:11:46.045 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-22T08:11:46.405 INFO:teuthology.orchestra.run.smithi189.stdout:librados-devel-18.2.2-1237.g646b8c7d.el8.x86_64 353 kB/s | 127 kB 00:00 2024-04-22T08:11:46.405 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-22T08:11:46.406 INFO:teuthology.orchestra.run.smithi189.stdout:Total 352 kB/s | 127 kB 00:00 2024-04-22T08:11:46.406 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-22T08:11:46.412 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-22T08:11:46.412 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-22T08:11:46.476 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-22T08:11:46.476 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-22T08:11:46.631 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-22T08:11:46.735 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librados-devel-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-22T08:11:47.138 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librados-devel-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-22T08:11:47.371 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librados-devel-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-22T08:11:47.371 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:11:47.371 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-22T08:11:47.371 INFO:teuthology.orchestra.run.smithi189.stdout: librados-devel-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:11:47.371 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:11:47.371 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-22T08:11:47.624 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install libcephfs2 2024-04-22T08:11:48.148 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:59 ago on Mon 22 Apr 2024 08:09:49 AM UTC. 2024-04-22T08:11:48.229 INFO:teuthology.orchestra.run.smithi189.stdout:Package libcephfs2-2:18.2.2-1237.g646b8c7d.el8.x86_64 is already installed. 2024-04-22T08:11:48.271 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-22T08:11:48.272 INFO:teuthology.orchestra.run.smithi189.stdout:Nothing to do. 2024-04-22T08:11:48.272 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-22T08:11:48.351 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install libcephfs-devel 2024-04-22T08:11:48.864 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:59 ago on Mon 22 Apr 2024 08:09:49 AM UTC. 2024-04-22T08:11:48.983 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-22T08:11:48.984 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:11:48.984 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repo Size 2024-04-22T08:11:48.984 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:11:48.984 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-22T08:11:48.984 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs-devel x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 31 k 2024-04-22T08:11:48.984 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:11:48.985 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-22T08:11:48.985 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:11:48.985 INFO:teuthology.orchestra.run.smithi189.stdout:Install 1 Package 2024-04-22T08:11:48.985 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:11:48.985 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 31 k 2024-04-22T08:11:48.985 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 139 k 2024-04-22T08:11:48.985 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-22T08:11:49.249 INFO:teuthology.orchestra.run.smithi189.stdout:libcephfs-devel-18.2.2-1237.g646b8c7d.el8.x86_6 117 kB/s | 31 kB 00:00 2024-04-22T08:11:49.250 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-22T08:11:49.250 INFO:teuthology.orchestra.run.smithi189.stdout:Total 117 kB/s | 31 kB 00:00 2024-04-22T08:11:49.250 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-22T08:11:49.254 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-22T08:11:49.254 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-22T08:11:49.256 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-22T08:11:49.256 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-22T08:11:49.342 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-22T08:11:49.462 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libcephfs-devel-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-22T08:11:49.786 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libcephfs-devel-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-22T08:11:49.989 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libcephfs-devel-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-22T08:11:49.989 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:11:49.989 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-22T08:11:49.989 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs-devel-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:11:49.989 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:11:49.989 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-22T08:11:50.165 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install librados2 2024-04-22T08:11:50.676 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:02:01 ago on Mon 22 Apr 2024 08:09:49 AM UTC. 2024-04-22T08:11:50.755 INFO:teuthology.orchestra.run.smithi189.stdout:Package librados2-2:18.2.2-1237.g646b8c7d.el8.x86_64 is already installed. 2024-04-22T08:11:50.795 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-22T08:11:50.796 INFO:teuthology.orchestra.run.smithi189.stdout:Nothing to do. 2024-04-22T08:11:50.796 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-22T08:11:50.876 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install librbd1 2024-04-22T08:11:51.391 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:02:02 ago on Mon 22 Apr 2024 08:09:49 AM UTC. 2024-04-22T08:11:51.470 INFO:teuthology.orchestra.run.smithi189.stdout:Package librbd1-2:18.2.2-1237.g646b8c7d.el8.x86_64 is already installed. 2024-04-22T08:11:51.510 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-22T08:11:51.511 INFO:teuthology.orchestra.run.smithi189.stdout:Nothing to do. 2024-04-22T08:11:51.511 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-22T08:11:51.593 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install python3-rados 2024-04-22T08:11:52.109 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:02:03 ago on Mon 22 Apr 2024 08:09:49 AM UTC. 2024-04-22T08:11:52.189 INFO:teuthology.orchestra.run.smithi189.stdout:Package python3-rados-2:18.2.2-1237.g646b8c7d.el8.x86_64 is already installed. 2024-04-22T08:11:52.229 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-22T08:11:52.230 INFO:teuthology.orchestra.run.smithi189.stdout:Nothing to do. 2024-04-22T08:11:52.230 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-22T08:11:52.317 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install python3-rgw 2024-04-22T08:11:52.841 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:02:03 ago on Mon 22 Apr 2024 08:09:49 AM UTC. 2024-04-22T08:11:52.921 INFO:teuthology.orchestra.run.smithi189.stdout:Package python3-rgw-2:18.2.2-1237.g646b8c7d.el8.x86_64 is already installed. 2024-04-22T08:11:52.961 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-22T08:11:52.962 INFO:teuthology.orchestra.run.smithi189.stdout:Nothing to do. 2024-04-22T08:11:52.962 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-22T08:11:53.050 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install python3-cephfs 2024-04-22T08:11:53.575 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:02:04 ago on Mon 22 Apr 2024 08:09:49 AM UTC. 2024-04-22T08:11:53.657 INFO:teuthology.orchestra.run.smithi189.stdout:Package python3-cephfs-2:18.2.2-1237.g646b8c7d.el8.x86_64 is already installed. 2024-04-22T08:11:53.700 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-22T08:11:53.701 INFO:teuthology.orchestra.run.smithi189.stdout:Nothing to do. 2024-04-22T08:11:53.701 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-22T08:11:53.786 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install python3-rbd 2024-04-22T08:11:54.325 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:02:05 ago on Mon 22 Apr 2024 08:09:49 AM UTC. 2024-04-22T08:11:54.405 INFO:teuthology.orchestra.run.smithi189.stdout:Package python3-rbd-2:18.2.2-1237.g646b8c7d.el8.x86_64 is already installed. 2024-04-22T08:11:54.446 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-22T08:11:54.447 INFO:teuthology.orchestra.run.smithi189.stdout:Nothing to do. 2024-04-22T08:11:54.447 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-22T08:11:54.523 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install rbd-fuse 2024-04-22T08:11:55.038 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:02:06 ago on Mon 22 Apr 2024 08:09:49 AM UTC. 2024-04-22T08:11:55.159 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-22T08:11:55.160 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:11:55.160 INFO:teuthology.orchestra.run.smithi189.stdout: Package Architecture Version Repository Size 2024-04-22T08:11:55.160 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:11:55.160 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-22T08:11:55.160 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-fuse x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 87 k 2024-04-22T08:11:55.161 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:11:55.161 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-22T08:11:55.161 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:11:55.161 INFO:teuthology.orchestra.run.smithi189.stdout:Install 1 Package 2024-04-22T08:11:55.161 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:11:55.161 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 87 k 2024-04-22T08:11:55.161 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 227 k 2024-04-22T08:11:55.161 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-22T08:11:55.370 INFO:teuthology.orchestra.run.smithi189.stdout:rbd-fuse-18.2.2-1237.g646b8c7d.el8.x86_64.rpm 415 kB/s | 87 kB 00:00 2024-04-22T08:11:55.371 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-22T08:11:55.371 INFO:teuthology.orchestra.run.smithi189.stdout:Total 412 kB/s | 87 kB 00:00 2024-04-22T08:11:55.371 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-22T08:11:55.378 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-22T08:11:55.378 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-22T08:11:55.442 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-22T08:11:55.442 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-22T08:11:55.580 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-22T08:11:55.674 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : rbd-fuse-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-22T08:11:56.036 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: rbd-fuse-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-22T08:11:56.279 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : rbd-fuse-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-22T08:11:56.279 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:11:56.279 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-22T08:11:56.279 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-fuse-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:11:56.279 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:11:56.279 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-22T08:11:56.555 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install rbd-mirror 2024-04-22T08:11:57.079 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:02:08 ago on Mon 22 Apr 2024 08:09:49 AM UTC. 2024-04-22T08:11:57.202 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-22T08:11:57.203 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:11:57.203 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repo Size 2024-04-22T08:11:57.203 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:11:57.203 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-22T08:11:57.203 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-mirror x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 3.0 M 2024-04-22T08:11:57.203 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:11:57.203 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-22T08:11:57.203 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:11:57.203 INFO:teuthology.orchestra.run.smithi189.stdout:Install 1 Package 2024-04-22T08:11:57.203 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:11:57.204 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 3.0 M 2024-04-22T08:11:57.204 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 12 M 2024-04-22T08:11:57.204 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-22T08:11:58.073 INFO:teuthology.orchestra.run.smithi189.stdout:rbd-mirror-18.2.2-1237.g646b8c7d.el8.x86_64.rpm 3.4 MB/s | 3.0 MB 00:00 2024-04-22T08:11:58.074 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-22T08:11:58.074 INFO:teuthology.orchestra.run.smithi189.stdout:Total 3.4 MB/s | 3.0 MB 00:00 2024-04-22T08:11:58.075 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-22T08:11:58.085 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-22T08:11:58.085 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-22T08:11:58.170 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-22T08:11:58.170 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-22T08:11:58.590 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-22T08:11:58.649 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : rbd-mirror-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-22T08:11:58.674 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-22T08:11:58.674 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T08:11:58.674 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-22T08:11:58.674 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T08:11:58.674 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T08:11:58.674 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:11:59.639 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : rbd-mirror-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-22T08:11:59.639 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:11:59.639 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-22T08:11:59.639 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-mirror-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:11:59.639 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:11:59.639 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-22T08:11:59.887 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install rbd-nbd 2024-04-22T08:12:00.410 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:02:11 ago on Mon 22 Apr 2024 08:09:49 AM UTC. 2024-04-22T08:12:00.531 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-22T08:12:00.532 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:12:00.532 INFO:teuthology.orchestra.run.smithi189.stdout: Package Architecture Version Repository Size 2024-04-22T08:12:00.532 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:12:00.532 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-22T08:12:00.532 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-nbd x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 172 k 2024-04-22T08:12:00.532 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:12:00.532 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-22T08:12:00.532 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-22T08:12:00.532 INFO:teuthology.orchestra.run.smithi189.stdout:Install 1 Package 2024-04-22T08:12:00.533 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:12:00.533 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 172 k 2024-04-22T08:12:00.533 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 494 k 2024-04-22T08:12:00.533 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-22T08:12:00.861 INFO:teuthology.orchestra.run.smithi189.stdout:rbd-nbd-18.2.2-1237.g646b8c7d.el8.x86_64.rpm 524 kB/s | 172 kB 00:00 2024-04-22T08:12:00.862 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-22T08:12:00.862 INFO:teuthology.orchestra.run.smithi189.stdout:Total 522 kB/s | 172 kB 00:00 2024-04-22T08:12:00.862 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-22T08:12:00.870 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-22T08:12:00.870 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-22T08:12:00.933 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-22T08:12:00.933 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-22T08:12:01.089 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-22T08:12:01.199 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : rbd-nbd-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-22T08:12:01.540 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: rbd-nbd-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-22T08:12:01.779 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : rbd-nbd-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-22T08:12:01.779 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:12:01.779 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-22T08:12:01.779 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-nbd-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-22T08:12:01.779 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:12:01.779 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-22T08:12:02.014 DEBUG:teuthology.parallel:result is None 2024-04-22T08:12:02.014 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=646b8c7d293d8ec31a40c69f7b3237b19c1be24d 2024-04-22T08:12:02.268 DEBUG:teuthology.orchestra.run.smithi040:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-22T08:12:02.294 INFO:teuthology.orchestra.run.smithi040.stdout:18.2.2-1237.g646b8c7d.el8 2024-04-22T08:12:02.294 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1237.g646b8c7d.el8 2024-04-22T08:12:02.294 INFO:teuthology.task.install:The correct ceph version 18.2.2-1237.g646b8c7d is installed. 2024-04-22T08:12:02.296 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=646b8c7d293d8ec31a40c69f7b3237b19c1be24d 2024-04-22T08:12:02.551 DEBUG:teuthology.orchestra.run.smithi189:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-22T08:12:02.574 INFO:teuthology.orchestra.run.smithi189.stdout:18.2.2-1237.g646b8c7d.el8 2024-04-22T08:12:02.574 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1237.g646b8c7d.el8 2024-04-22T08:12:02.574 INFO:teuthology.task.install:The correct ceph version 18.2.2-1237.g646b8c7d is installed. 2024-04-22T08:12:02.576 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-22T08:12:02.576 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-22T08:12:02.576 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-22T08:12:02.606 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-22T08:12:02.606 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-22T08:12:02.638 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-22T08:12:02.639 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-22T08:12:02.639 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/daemon-helper 2024-04-22T08:12:02.674 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-22T08:12:02.742 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-22T08:12:02.742 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/daemon-helper 2024-04-22T08:12:02.767 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-22T08:12:02.832 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-22T08:12:02.832 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-22T08:12:02.832 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-22T08:12:02.860 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-22T08:12:02.926 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-22T08:12:02.926 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-22T08:12:02.952 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-22T08:12:03.016 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-22T08:12:03.017 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-22T08:12:03.017 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/stdin-killer 2024-04-22T08:12:03.041 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-22T08:12:03.106 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-22T08:12:03.106 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/stdin-killer 2024-04-22T08:12:03.131 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-22T08:12:03.197 INFO:teuthology.run_tasks:Running task ceph... 2024-04-22T08:12:03.282 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-22T08:12:03.283 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 777 /var/log/ceph 2024-04-22T08:12:03.285 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 777 /var/log/ceph 2024-04-22T08:12:03.308 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-22T08:12:03.308 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-22T08:12:03.350 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-22T08:12:03.373 INFO:tasks.ceph:Creating extra log directories... 2024-04-22T08:12:03.373 DEBUG:teuthology.orchestra.run.smithi040:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-22T08:12:03.414 DEBUG:teuthology.orchestra.run.smithi189:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-22T08:12:03.449 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-22T08:12:03.450 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'slow request', 'overall HEALTH_'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-22T08:12:03.450 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-04-21_22:00:02-rados-reef-distro-default-smithi/7667865', 'branch': 'reef', 'description': 'rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/one workloads/pool-create-delete}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7667865', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-04-21_22:00:02-rados-reef-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'reef'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'slow request', 'overall HEALTH_'], 'sha1': '646b8c7d293d8ec31a40c69f7b3237b19c1be24d'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '646b8c7d293d8ec31a40c69f7b3237b19c1be24d'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'reef', 'sha1': 'c3a345d31ea77c01771428ab80e1cf32c9b05875'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 931, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 7353, 'sha1': '646b8c7d293d8ec31a40c69f7b3237b19c1be24d', 'sleep_before_teardown': 0, 'subset': '3333/100000', 'suite': 'rados', 'suite_branch': 'reef', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_c3a345d31ea77c01771428ab80e1cf32c9b05875/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'c3a345d31ea77c01771428ab80e1cf32c9b05875', 'targets': {'smithi040.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPCbB9kJt2IkRaeY5ZmIgRyOPqr6BAiF7tecZKlWcpj8VtR4PzBUj1MStl+E6kL7uxljhbCijOSXZo4js8l61XM=', 'smithi189.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIM87Hq2VE6K9aTqZUaXZf7MMMTbfy/r9Mue2robyIO43KX2BBm9xDGsc6DW4OnueKI73GgyIld9xWpRFr1Xk8I='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'revive_delay': 20, 'thrash_delay': 1}}, {'exec': {'client.0': ['ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-21_22:00:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-22T08:12:03.451 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-22T08:12:03.479 DEBUG:teuthology.orchestra.run.smithi189:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-22T08:12:03.493 DEBUG:teuthology.orchestra.run.smithi040:> sudo install -d -m0777 -- /var/run/ceph 2024-04-22T08:12:03.535 DEBUG:teuthology.orchestra.run.smithi189:> sudo install -d -m0777 -- /var/run/ceph 2024-04-22T08:12:03.558 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-22T08:12:03.558 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/scratch_devs of=/dev/stdout 2024-04-22T08:12:03.612 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-22T08:12:03.613 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_1 2024-04-22T08:12:03.667 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-22T08:12:03.667 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T08:12:03.667 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-22T08:12:03.667 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T08:12:03.667 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T08:12:03.668 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-04-22 08:11:34.267476863 +0000 2024-04-22T08:12:03.668 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-04-22 08:08:59.471855584 +0000 2024-04-22T08:12:03.668 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-04-22 08:08:59.471855584 +0000 2024-04-22T08:12:03.668 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-04-22 08:08:59.471855584 +0000 2024-04-22T08:12:03.668 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-22T08:12:03.730 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-04-22T08:12:03.730 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-04-22T08:12:03.730 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000244988 s, 2.1 MB/s 2024-04-22T08:12:03.731 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-22T08:12:03.787 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_2 2024-04-22T08:12:03.842 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-22T08:12:03.842 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T08:12:03.843 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-04-22T08:12:03.843 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T08:12:03.843 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T08:12:03.843 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-04-22 08:11:34.267476863 +0000 2024-04-22T08:12:03.843 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-04-22 08:08:59.824851027 +0000 2024-04-22T08:12:03.843 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-04-22 08:08:59.824851027 +0000 2024-04-22T08:12:03.843 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-04-22 08:08:59.824851027 +0000 2024-04-22T08:12:03.843 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-22T08:12:03.905 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-04-22T08:12:03.905 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-04-22T08:12:03.905 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000215233 s, 2.4 MB/s 2024-04-22T08:12:03.906 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-22T08:12:03.962 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_3 2024-04-22T08:12:04.017 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-22T08:12:04.017 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T08:12:04.017 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-04-22T08:12:04.017 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T08:12:04.017 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T08:12:04.018 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-04-22 08:11:34.268476850 +0000 2024-04-22T08:12:04.018 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-04-22 08:09:00.179846443 +0000 2024-04-22T08:12:04.018 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-04-22 08:09:00.179846443 +0000 2024-04-22T08:12:04.018 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-04-22 08:09:00.179846443 +0000 2024-04-22T08:12:04.018 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-22T08:12:04.080 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-04-22T08:12:04.080 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-04-22T08:12:04.080 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000209142 s, 2.4 MB/s 2024-04-22T08:12:04.081 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-22T08:12:04.137 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_4 2024-04-22T08:12:04.192 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-22T08:12:04.193 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T08:12:04.193 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-22T08:12:04.193 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T08:12:04.193 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T08:12:04.193 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-04-22 08:11:34.268476850 +0000 2024-04-22T08:12:04.193 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-04-22 08:09:00.528841937 +0000 2024-04-22T08:12:04.193 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-04-22 08:09:00.528841937 +0000 2024-04-22T08:12:04.193 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-04-22 08:09:00.528841937 +0000 2024-04-22T08:12:04.193 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-22T08:12:04.256 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-04-22T08:12:04.256 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-04-22T08:12:04.256 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000234707 s, 2.2 MB/s 2024-04-22T08:12:04.257 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-22T08:12:04.313 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-22T08:12:04.313 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-22T08:12:04.313 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/scratch_devs of=/dev/stdout 2024-04-22T08:12:04.330 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-22T08:12:04.330 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_1 2024-04-22T08:12:04.386 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-22T08:12:04.387 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T08:12:04.387 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-04-22T08:12:04.387 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T08:12:04.387 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T08:12:04.387 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-04-22 08:11:58.937066917 +0000 2024-04-22T08:12:04.387 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-04-22 08:08:59.396063752 +0000 2024-04-22T08:12:04.387 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-04-22 08:08:59.396063752 +0000 2024-04-22T08:12:04.387 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-04-22 08:08:59.396063752 +0000 2024-04-22T08:12:04.388 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-22T08:12:04.450 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-04-22T08:12:04.450 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-04-22T08:12:04.450 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000217847 s, 2.4 MB/s 2024-04-22T08:12:04.452 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-22T08:12:04.509 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_2 2024-04-22T08:12:04.565 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-22T08:12:04.565 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T08:12:04.565 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-22T08:12:04.565 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T08:12:04.566 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T08:12:04.566 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-04-22 08:11:58.937066917 +0000 2024-04-22T08:12:04.566 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-04-22 08:08:59.733057433 +0000 2024-04-22T08:12:04.566 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-04-22 08:08:59.733057433 +0000 2024-04-22T08:12:04.566 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-04-22 08:08:59.733057433 +0000 2024-04-22T08:12:04.566 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-22T08:12:04.629 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-04-22T08:12:04.629 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-04-22T08:12:04.629 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.00023496 s, 2.2 MB/s 2024-04-22T08:12:04.631 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-22T08:12:04.688 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_3 2024-04-22T08:12:04.743 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-22T08:12:04.744 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T08:12:04.744 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-22T08:12:04.744 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T08:12:04.744 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T08:12:04.744 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-04-22 08:11:58.938066899 +0000 2024-04-22T08:12:04.745 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-04-22 08:09:00.064051227 +0000 2024-04-22T08:12:04.745 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-04-22 08:09:00.064051227 +0000 2024-04-22T08:12:04.745 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-04-22 08:09:00.064051227 +0000 2024-04-22T08:12:04.745 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-22T08:12:04.806 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-04-22T08:12:04.806 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-04-22T08:12:04.806 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000230977 s, 2.2 MB/s 2024-04-22T08:12:04.807 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-22T08:12:04.863 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_4 2024-04-22T08:12:04.918 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-22T08:12:04.918 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T08:12:04.918 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-22T08:12:04.918 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T08:12:04.918 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T08:12:04.918 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-04-22 08:11:58.938066899 +0000 2024-04-22T08:12:04.918 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-04-22 08:09:00.397044984 +0000 2024-04-22T08:12:04.918 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-04-22 08:09:00.397044984 +0000 2024-04-22T08:12:04.919 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-04-22 08:09:00.397044984 +0000 2024-04-22T08:12:04.919 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-22T08:12:04.980 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-04-22T08:12:04.981 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-04-22T08:12:04.981 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000189808 s, 2.7 MB/s 2024-04-22T08:12:04.982 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-22T08:12:05.038 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-04-22T08:12:05.038 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi040.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi189.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-04-22T08:12:05.039 INFO:tasks.ceph:Generating config... 2024-04-22T08:12:05.040 INFO:tasks.ceph:[client] debug monc = 20 2024-04-22T08:12:05.040 INFO:tasks.ceph:[client] debug ms = 1 2024-04-22T08:12:05.041 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-04-22T08:12:05.041 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-04-22T08:12:05.041 INFO:tasks.ceph:[global] ms bind msgr1 = False 2024-04-22T08:12:05.041 INFO:tasks.ceph:[global] ms bind msgr2 = True 2024-04-22T08:12:05.041 INFO:tasks.ceph:[global] ms inject delay max = 1 2024-04-22T08:12:05.041 INFO:tasks.ceph:[global] ms inject delay probability = 0.005 2024-04-22T08:12:05.041 INFO:tasks.ceph:[global] ms inject delay type = mon 2024-04-22T08:12:05.041 INFO:tasks.ceph:[global] ms inject internal delays = 0.002 2024-04-22T08:12:05.041 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2024-04-22T08:12:05.041 INFO:tasks.ceph:[global] ms type = async 2024-04-22T08:12:05.042 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-22T08:12:05.042 INFO:tasks.ceph:[mgr] debug monc = 10 2024-04-22T08:12:05.043 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-22T08:12:05.043 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-22T08:12:05.043 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-22T08:12:05.043 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-22T08:12:05.043 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2024-04-22T08:12:05.043 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-04-22T08:12:05.043 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-04-22T08:12:05.043 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-04-22T08:12:05.043 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2024-04-22T08:12:05.043 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2024-04-22T08:12:05.043 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-04-22T08:12:05.044 INFO:tasks.ceph:[mon] paxos service trim min = 5 2024-04-22T08:12:05.044 INFO:tasks.ceph:[osd] bdev async discard = True 2024-04-22T08:12:05.044 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-04-22T08:12:05.044 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-22T08:12:05.044 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-22T08:12:05.044 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-04-22T08:12:05.044 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-22T08:12:05.044 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-22T08:12:05.044 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-22T08:12:05.044 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-22T08:12:05.044 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-22T08:12:05.044 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-22T08:12:05.044 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-22T08:12:05.044 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-22T08:12:05.044 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-04-22T08:12:05.045 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-04-22T08:12:05.045 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-22T08:12:05.045 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-04-22T08:12:05.045 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-04-22T08:12:05.045 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2024-04-22T08:12:05.045 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-22T08:12:05.045 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-04-22T08:12:05.045 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-04-22T08:12:05.045 INFO:tasks.ceph:Setting up mon.a... 2024-04-22T08:12:05.045 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-22T08:12:05.103 INFO:teuthology.orchestra.run.smithi040.stdout:creating /etc/ceph/ceph.keyring 2024-04-22T08:12:05.103 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-22T08:12:05.149 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-22T08:12:05.175 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.40'), ('mon.c', '[v2:172.21.15.40:3301,v1:172.21.15.40:6790]'), ('mon.b', '172.21.15.189')] 2024-04-22T08:12:05.175 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.40,[v2:172.21.15.40:3301,v1:172.21.15.40:6790],172.21.15.189', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug monc': 10}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-04-22T08:12:05.176 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-22T08:12:05.176 DEBUG:teuthology.orchestra.run.smithi040:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-22T08:12:05.231 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.40 --addv c '[v2:172.21.15.40:3301,v1:172.21.15.40:6790]' --add b 172.21.15.189 --print /home/ubuntu/cephtest/ceph.monmap 2024-04-22T08:12:05.319 INFO:teuthology.orchestra.run.smithi040.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-22T08:12:05.319 INFO:teuthology.orchestra.run.smithi040.stdout:monmaptool: generated fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:05.319 INFO:teuthology.orchestra.run.smithi040.stdout:setting min_mon_release = pacific 2024-04-22T08:12:05.319 INFO:teuthology.orchestra.run.smithi040.stdout:epoch 0 2024-04-22T08:12:05.319 INFO:teuthology.orchestra.run.smithi040.stdout:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:05.319 INFO:teuthology.orchestra.run.smithi040.stdout:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:05.319 INFO:teuthology.orchestra.run.smithi040.stdout:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:05.319 INFO:teuthology.orchestra.run.smithi040.stdout:min_mon_release 16 (pacific) 2024-04-22T08:12:05.319 INFO:teuthology.orchestra.run.smithi040.stdout:election_strategy: 1 2024-04-22T08:12:05.319 INFO:teuthology.orchestra.run.smithi040.stdout:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:05.319 INFO:teuthology.orchestra.run.smithi040.stdout:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:05.320 INFO:teuthology.orchestra.run.smithi040.stdout:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:05.320 INFO:teuthology.orchestra.run.smithi040.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-22T08:12:05.321 DEBUG:teuthology.orchestra.run.smithi040:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-22T08:12:05.336 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 987d842f-7205-47b9-86e0-2e2623f3c370... 2024-04-22T08:12:05.337 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-22T08:12:05.379 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-22T08:12:05.419 INFO:teuthology.orchestra.run.smithi040.stdout:[global] 2024-04-22T08:12:05.419 INFO:teuthology.orchestra.run.smithi040.stdout: chdir = "" 2024-04-22T08:12:05.419 INFO:teuthology.orchestra.run.smithi040.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-22T08:12:05.419 INFO:teuthology.orchestra.run.smithi040.stdout: auth supported = cephx 2024-04-22T08:12:05.419 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:12:05.419 INFO:teuthology.orchestra.run.smithi040.stdout: filestore xattr use omap = true 2024-04-22T08:12:05.419 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:12:05.420 INFO:teuthology.orchestra.run.smithi040.stdout: mon clock drift allowed = 1.000 2024-04-22T08:12:05.420 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:12:05.420 INFO:teuthology.orchestra.run.smithi040.stdout: osd crush chooseleaf type = 0 2024-04-22T08:12:05.420 INFO:teuthology.orchestra.run.smithi040.stdout: auth debug = true 2024-04-22T08:12:05.420 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:12:05.420 INFO:teuthology.orchestra.run.smithi040.stdout: ms die on old message = true 2024-04-22T08:12:05.420 INFO:teuthology.orchestra.run.smithi040.stdout: ms die on bug = true 2024-04-22T08:12:05.420 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:12:05.420 INFO:teuthology.orchestra.run.smithi040.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-22T08:12:05.420 INFO:teuthology.orchestra.run.smithi040.stdout: mon pg warn max object skew = 0 2024-04-22T08:12:05.420 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:12:05.420 INFO:teuthology.orchestra.run.smithi040.stdout: # disable pg_autoscaler by default for new pools 2024-04-22T08:12:05.420 INFO:teuthology.orchestra.run.smithi040.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-22T08:12:05.420 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:12:05.421 INFO:teuthology.orchestra.run.smithi040.stdout: osd pool default size = 2 2024-04-22T08:12:05.421 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:12:05.421 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd allow primary affinity = true 2024-04-22T08:12:05.421 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd allow pg remap = true 2024-04-22T08:12:05.421 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on legacy crush tunables = false 2024-04-22T08:12:05.421 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on crush straw calc version zero = false 2024-04-22T08:12:05.421 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on no sortbitwise = false 2024-04-22T08:12:05.421 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on osd down out interval zero = false 2024-04-22T08:12:05.421 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on too few osds = false 2024-04-22T08:12:05.421 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-22T08:12:05.421 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-22T08:12:05.421 INFO:teuthology.orchestra.run.smithi040.stdout: mon_allow_pool_size_one = true 2024-04-22T08:12:05.421 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:12:05.422 INFO:teuthology.orchestra.run.smithi040.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-22T08:12:05.422 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:12:05.422 INFO:teuthology.orchestra.run.smithi040.stdout: osd default data pool replay window = 5 2024-04-22T08:12:05.422 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:12:05.422 INFO:teuthology.orchestra.run.smithi040.stdout: mon allow pool delete = true 2024-04-22T08:12:05.422 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:12:05.422 INFO:teuthology.orchestra.run.smithi040.stdout: mon cluster log file level = debug 2024-04-22T08:12:05.422 INFO:teuthology.orchestra.run.smithi040.stdout: debug asserts on shutdown = true 2024-04-22T08:12:05.422 INFO:teuthology.orchestra.run.smithi040.stdout: mon health detail to clog = false 2024-04-22T08:12:05.422 INFO:teuthology.orchestra.run.smithi040.stdout: mon host = "172.21.15.40,[v2:172.21.15.40:3301,v1:172.21.15.40:6790],172.21.15.189" 2024-04-22T08:12:05.422 INFO:teuthology.orchestra.run.smithi040.stdout: mon client directed command retry = 5 2024-04-22T08:12:05.422 INFO:teuthology.orchestra.run.smithi040.stdout: mon election default strategy = 1 2024-04-22T08:12:05.422 INFO:teuthology.orchestra.run.smithi040.stdout: ms bind msgr1 = False 2024-04-22T08:12:05.423 INFO:teuthology.orchestra.run.smithi040.stdout: ms bind msgr2 = True 2024-04-22T08:12:05.423 INFO:teuthology.orchestra.run.smithi040.stdout: ms inject delay max = 1 2024-04-22T08:12:05.423 INFO:teuthology.orchestra.run.smithi040.stdout: ms inject delay probability = 0.005 2024-04-22T08:12:05.423 INFO:teuthology.orchestra.run.smithi040.stdout: ms inject delay type = mon 2024-04-22T08:12:05.423 INFO:teuthology.orchestra.run.smithi040.stdout: ms inject internal delays = 0.002 2024-04-22T08:12:05.423 INFO:teuthology.orchestra.run.smithi040.stdout: ms inject socket failures = 2500 2024-04-22T08:12:05.423 INFO:teuthology.orchestra.run.smithi040.stdout: ms type = async 2024-04-22T08:12:05.423 INFO:teuthology.orchestra.run.smithi040.stdout: fsid = 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:05.423 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:12:05.423 INFO:teuthology.orchestra.run.smithi040.stdout:[osd] 2024-04-22T08:12:05.423 INFO:teuthology.orchestra.run.smithi040.stdout: osd journal size = 100 2024-04-22T08:12:05.423 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:12:05.423 INFO:teuthology.orchestra.run.smithi040.stdout: osd scrub load threshold = 5.0 2024-04-22T08:12:05.424 INFO:teuthology.orchestra.run.smithi040.stdout: osd scrub max interval = 600 2024-04-22T08:12:05.424 INFO:teuthology.orchestra.run.smithi040.stdout: osd mclock profile = high_recovery_ops 2024-04-22T08:12:05.424 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:12:05.424 INFO:teuthology.orchestra.run.smithi040.stdout: osd recover clone overlap = true 2024-04-22T08:12:05.424 INFO:teuthology.orchestra.run.smithi040.stdout: osd recovery max chunk = 1048576 2024-04-22T08:12:05.424 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:12:05.424 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug shutdown = true 2024-04-22T08:12:05.424 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug op order = true 2024-04-22T08:12:05.424 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug verify stray on activate = true 2024-04-22T08:12:05.424 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:12:05.424 INFO:teuthology.orchestra.run.smithi040.stdout: osd open classes on start = true 2024-04-22T08:12:05.424 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug pg log writeout = true 2024-04-22T08:12:05.424 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:12:05.425 INFO:teuthology.orchestra.run.smithi040.stdout: osd deep scrub update digest min age = 30 2024-04-22T08:12:05.425 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:12:05.425 INFO:teuthology.orchestra.run.smithi040.stdout: osd map max advance = 10 2024-04-22T08:12:05.425 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:12:05.425 INFO:teuthology.orchestra.run.smithi040.stdout: journal zero on create = true 2024-04-22T08:12:05.425 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:12:05.425 INFO:teuthology.orchestra.run.smithi040.stdout: filestore ondisk finisher threads = 3 2024-04-22T08:12:05.425 INFO:teuthology.orchestra.run.smithi040.stdout: filestore apply finisher threads = 3 2024-04-22T08:12:05.425 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:12:05.425 INFO:teuthology.orchestra.run.smithi040.stdout: bdev debug aio = true 2024-04-22T08:12:05.425 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug misdirected ops = true 2024-04-22T08:12:05.425 INFO:teuthology.orchestra.run.smithi040.stdout: bdev async discard = True 2024-04-22T08:12:05.425 INFO:teuthology.orchestra.run.smithi040.stdout: bdev enable discard = True 2024-04-22T08:12:05.425 INFO:teuthology.orchestra.run.smithi040.stdout: bluestore block size = 96636764160 2024-04-22T08:12:05.426 INFO:teuthology.orchestra.run.smithi040.stdout: bluestore fsck on mount = True 2024-04-22T08:12:05.426 INFO:teuthology.orchestra.run.smithi040.stdout: bluestore zero block detection = True 2024-04-22T08:12:05.426 INFO:teuthology.orchestra.run.smithi040.stdout: debug bluefs = 20 2024-04-22T08:12:05.426 INFO:teuthology.orchestra.run.smithi040.stdout: debug bluestore = 20 2024-04-22T08:12:05.426 INFO:teuthology.orchestra.run.smithi040.stdout: debug ms = 1 2024-04-22T08:12:05.426 INFO:teuthology.orchestra.run.smithi040.stdout: debug osd = 20 2024-04-22T08:12:05.426 INFO:teuthology.orchestra.run.smithi040.stdout: debug rocksdb = 10 2024-04-22T08:12:05.426 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd backfillfull_ratio = 0.85 2024-04-22T08:12:05.426 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd full ratio = 0.9 2024-04-22T08:12:05.426 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd nearfull ratio = 0.8 2024-04-22T08:12:05.426 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug verify cached snaps = True 2024-04-22T08:12:05.426 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug verify missing on start = True 2024-04-22T08:12:05.426 INFO:teuthology.orchestra.run.smithi040.stdout: osd failsafe full ratio = 0.95 2024-04-22T08:12:05.427 INFO:teuthology.orchestra.run.smithi040.stdout: osd mclock override recovery settings = True 2024-04-22T08:12:05.427 INFO:teuthology.orchestra.run.smithi040.stdout: osd memory target = 1610612736 2024-04-22T08:12:05.427 INFO:teuthology.orchestra.run.smithi040.stdout: osd objectstore = bluestore 2024-04-22T08:12:05.427 INFO:teuthology.orchestra.run.smithi040.stdout: osd op queue = debug_random 2024-04-22T08:12:05.427 INFO:teuthology.orchestra.run.smithi040.stdout: osd op queue cut off = debug_random 2024-04-22T08:12:05.427 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:12:05.427 INFO:teuthology.orchestra.run.smithi040.stdout:[mgr] 2024-04-22T08:12:05.427 INFO:teuthology.orchestra.run.smithi040.stdout: debug ms = 1 2024-04-22T08:12:05.427 INFO:teuthology.orchestra.run.smithi040.stdout: debug mgr = 20 2024-04-22T08:12:05.427 INFO:teuthology.orchestra.run.smithi040.stdout: debug mon = 20 2024-04-22T08:12:05.427 INFO:teuthology.orchestra.run.smithi040.stdout: debug auth = 20 2024-04-22T08:12:05.427 INFO:teuthology.orchestra.run.smithi040.stdout: mon reweight min pgs per osd = 4 2024-04-22T08:12:05.427 INFO:teuthology.orchestra.run.smithi040.stdout: mon reweight min bytes per osd = 10 2024-04-22T08:12:05.427 INFO:teuthology.orchestra.run.smithi040.stdout: mgr/telemetry/nag = false 2024-04-22T08:12:05.428 INFO:teuthology.orchestra.run.smithi040.stdout: debug monc = 10 2024-04-22T08:12:05.428 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:12:05.428 INFO:teuthology.orchestra.run.smithi040.stdout:[mon] 2024-04-22T08:12:05.428 INFO:teuthology.orchestra.run.smithi040.stdout: debug ms = 1 2024-04-22T08:12:05.428 INFO:teuthology.orchestra.run.smithi040.stdout: debug mon = 20 2024-04-22T08:12:05.428 INFO:teuthology.orchestra.run.smithi040.stdout: debug paxos = 20 2024-04-22T08:12:05.428 INFO:teuthology.orchestra.run.smithi040.stdout: debug auth = 20 2024-04-22T08:12:05.428 INFO:teuthology.orchestra.run.smithi040.stdout: mon data avail warn = 5 2024-04-22T08:12:05.428 INFO:teuthology.orchestra.run.smithi040.stdout: mon mgr mkfs grace = 240 2024-04-22T08:12:05.428 INFO:teuthology.orchestra.run.smithi040.stdout: mon reweight min pgs per osd = 4 2024-04-22T08:12:05.428 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd reporter subtree level = osd 2024-04-22T08:12:05.428 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd prime pg temp = true 2024-04-22T08:12:05.428 INFO:teuthology.orchestra.run.smithi040.stdout: mon reweight min bytes per osd = 10 2024-04-22T08:12:05.428 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:12:05.429 INFO:teuthology.orchestra.run.smithi040.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-22T08:12:05.429 INFO:teuthology.orchestra.run.smithi040.stdout: auth mon ticket ttl = 660 # 11m 2024-04-22T08:12:05.429 INFO:teuthology.orchestra.run.smithi040.stdout: auth service ticket ttl = 240 # 4m 2024-04-22T08:12:05.429 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:12:05.429 INFO:teuthology.orchestra.run.smithi040.stdout: # don't complain about insecure global_id in the test suite 2024-04-22T08:12:05.429 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-22T08:12:05.429 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-22T08:12:05.429 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:12:05.429 INFO:teuthology.orchestra.run.smithi040.stdout: # 1m isn't quite enough 2024-04-22T08:12:05.429 INFO:teuthology.orchestra.run.smithi040.stdout: mon_down_mkfs_grace = 2m 2024-04-22T08:12:05.429 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:12:05.429 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_filestore_osds = false 2024-04-22T08:12:05.429 INFO:teuthology.orchestra.run.smithi040.stdout: mon min osdmap epochs = 25 2024-04-22T08:12:05.429 INFO:teuthology.orchestra.run.smithi040.stdout: mon osdmap full prune interval = 2 2024-04-22T08:12:05.430 INFO:teuthology.orchestra.run.smithi040.stdout: mon osdmap full prune min = 15 2024-04-22T08:12:05.430 INFO:teuthology.orchestra.run.smithi040.stdout: mon osdmap full prune txsize = 2 2024-04-22T08:12:05.430 INFO:teuthology.orchestra.run.smithi040.stdout: mon scrub inject crc mismatch = 0.01 2024-04-22T08:12:05.430 INFO:teuthology.orchestra.run.smithi040.stdout: mon scrub inject missing keys = 0.05 2024-04-22T08:12:05.430 INFO:teuthology.orchestra.run.smithi040.stdout: mon scrub interval = 300 2024-04-22T08:12:05.430 INFO:teuthology.orchestra.run.smithi040.stdout: paxos service trim min = 5 2024-04-22T08:12:05.430 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:12:05.430 INFO:teuthology.orchestra.run.smithi040.stdout:[client] 2024-04-22T08:12:05.430 INFO:teuthology.orchestra.run.smithi040.stdout: rgw cache enabled = true 2024-04-22T08:12:05.430 INFO:teuthology.orchestra.run.smithi040.stdout: rgw enable ops log = true 2024-04-22T08:12:05.430 INFO:teuthology.orchestra.run.smithi040.stdout: rgw enable usage log = true 2024-04-22T08:12:05.430 INFO:teuthology.orchestra.run.smithi040.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-22T08:12:05.430 INFO:teuthology.orchestra.run.smithi040.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-22T08:12:05.430 INFO:teuthology.orchestra.run.smithi040.stdout: debug monc = 20 2024-04-22T08:12:05.431 INFO:teuthology.orchestra.run.smithi040.stdout: debug ms = 1 2024-04-22T08:12:05.431 INFO:teuthology.orchestra.run.smithi040.stdout:[mon.a] 2024-04-22T08:12:05.431 INFO:teuthology.orchestra.run.smithi040.stdout:[mon.c] 2024-04-22T08:12:05.431 INFO:teuthology.orchestra.run.smithi040.stdout:[mon.b] 2024-04-22T08:12:05.432 INFO:teuthology.orchestra.run.smithi189.stdout:[global] 2024-04-22T08:12:05.432 INFO:teuthology.orchestra.run.smithi189.stdout: chdir = "" 2024-04-22T08:12:05.433 INFO:teuthology.orchestra.run.smithi189.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-22T08:12:05.433 INFO:teuthology.orchestra.run.smithi189.stdout: auth supported = cephx 2024-04-22T08:12:05.433 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:12:05.433 INFO:teuthology.orchestra.run.smithi189.stdout: filestore xattr use omap = true 2024-04-22T08:12:05.433 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:12:05.433 INFO:teuthology.orchestra.run.smithi189.stdout: mon clock drift allowed = 1.000 2024-04-22T08:12:05.433 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:12:05.433 INFO:teuthology.orchestra.run.smithi189.stdout: osd crush chooseleaf type = 0 2024-04-22T08:12:05.433 INFO:teuthology.orchestra.run.smithi189.stdout: auth debug = true 2024-04-22T08:12:05.433 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:12:05.433 INFO:teuthology.orchestra.run.smithi189.stdout: ms die on old message = true 2024-04-22T08:12:05.433 INFO:teuthology.orchestra.run.smithi189.stdout: ms die on bug = true 2024-04-22T08:12:05.433 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:12:05.434 INFO:teuthology.orchestra.run.smithi189.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-22T08:12:05.434 INFO:teuthology.orchestra.run.smithi189.stdout: mon pg warn max object skew = 0 2024-04-22T08:12:05.434 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:12:05.434 INFO:teuthology.orchestra.run.smithi189.stdout: # disable pg_autoscaler by default for new pools 2024-04-22T08:12:05.434 INFO:teuthology.orchestra.run.smithi189.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-22T08:12:05.434 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:12:05.434 INFO:teuthology.orchestra.run.smithi189.stdout: osd pool default size = 2 2024-04-22T08:12:05.434 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:12:05.434 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd allow primary affinity = true 2024-04-22T08:12:05.434 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd allow pg remap = true 2024-04-22T08:12:05.434 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on legacy crush tunables = false 2024-04-22T08:12:05.434 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on crush straw calc version zero = false 2024-04-22T08:12:05.434 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on no sortbitwise = false 2024-04-22T08:12:05.434 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on osd down out interval zero = false 2024-04-22T08:12:05.435 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on too few osds = false 2024-04-22T08:12:05.435 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-22T08:12:05.435 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-22T08:12:05.435 INFO:teuthology.orchestra.run.smithi189.stdout: mon_allow_pool_size_one = true 2024-04-22T08:12:05.435 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:12:05.435 INFO:teuthology.orchestra.run.smithi189.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-22T08:12:05.435 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:12:05.435 INFO:teuthology.orchestra.run.smithi189.stdout: osd default data pool replay window = 5 2024-04-22T08:12:05.435 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:12:05.435 INFO:teuthology.orchestra.run.smithi189.stdout: mon allow pool delete = true 2024-04-22T08:12:05.435 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:12:05.435 INFO:teuthology.orchestra.run.smithi189.stdout: mon cluster log file level = debug 2024-04-22T08:12:05.435 INFO:teuthology.orchestra.run.smithi189.stdout: debug asserts on shutdown = true 2024-04-22T08:12:05.435 INFO:teuthology.orchestra.run.smithi189.stdout: mon health detail to clog = false 2024-04-22T08:12:05.436 INFO:teuthology.orchestra.run.smithi189.stdout: mon host = "172.21.15.40,[v2:172.21.15.40:3301,v1:172.21.15.40:6790],172.21.15.189" 2024-04-22T08:12:05.436 INFO:teuthology.orchestra.run.smithi189.stdout: mon client directed command retry = 5 2024-04-22T08:12:05.436 INFO:teuthology.orchestra.run.smithi189.stdout: mon election default strategy = 1 2024-04-22T08:12:05.436 INFO:teuthology.orchestra.run.smithi189.stdout: ms bind msgr1 = False 2024-04-22T08:12:05.436 INFO:teuthology.orchestra.run.smithi189.stdout: ms bind msgr2 = True 2024-04-22T08:12:05.436 INFO:teuthology.orchestra.run.smithi189.stdout: ms inject delay max = 1 2024-04-22T08:12:05.436 INFO:teuthology.orchestra.run.smithi189.stdout: ms inject delay probability = 0.005 2024-04-22T08:12:05.436 INFO:teuthology.orchestra.run.smithi189.stdout: ms inject delay type = mon 2024-04-22T08:12:05.436 INFO:teuthology.orchestra.run.smithi189.stdout: ms inject internal delays = 0.002 2024-04-22T08:12:05.436 INFO:teuthology.orchestra.run.smithi189.stdout: ms inject socket failures = 2500 2024-04-22T08:12:05.436 INFO:teuthology.orchestra.run.smithi189.stdout: ms type = async 2024-04-22T08:12:05.436 INFO:teuthology.orchestra.run.smithi189.stdout: fsid = 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:05.436 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:12:05.437 INFO:teuthology.orchestra.run.smithi189.stdout:[osd] 2024-04-22T08:12:05.437 INFO:teuthology.orchestra.run.smithi189.stdout: osd journal size = 100 2024-04-22T08:12:05.437 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:12:05.437 INFO:teuthology.orchestra.run.smithi189.stdout: osd scrub load threshold = 5.0 2024-04-22T08:12:05.437 INFO:teuthology.orchestra.run.smithi189.stdout: osd scrub max interval = 600 2024-04-22T08:12:05.437 INFO:teuthology.orchestra.run.smithi189.stdout: osd mclock profile = high_recovery_ops 2024-04-22T08:12:05.437 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:12:05.437 INFO:teuthology.orchestra.run.smithi189.stdout: osd recover clone overlap = true 2024-04-22T08:12:05.437 INFO:teuthology.orchestra.run.smithi189.stdout: osd recovery max chunk = 1048576 2024-04-22T08:12:05.437 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:12:05.437 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug shutdown = true 2024-04-22T08:12:05.437 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug op order = true 2024-04-22T08:12:05.437 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug verify stray on activate = true 2024-04-22T08:12:05.437 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:12:05.437 INFO:teuthology.orchestra.run.smithi189.stdout: osd open classes on start = true 2024-04-22T08:12:05.438 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug pg log writeout = true 2024-04-22T08:12:05.438 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:12:05.438 INFO:teuthology.orchestra.run.smithi189.stdout: osd deep scrub update digest min age = 30 2024-04-22T08:12:05.438 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:12:05.438 INFO:teuthology.orchestra.run.smithi189.stdout: osd map max advance = 10 2024-04-22T08:12:05.438 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:12:05.438 INFO:teuthology.orchestra.run.smithi189.stdout: journal zero on create = true 2024-04-22T08:12:05.438 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:12:05.438 INFO:teuthology.orchestra.run.smithi189.stdout: filestore ondisk finisher threads = 3 2024-04-22T08:12:05.438 INFO:teuthology.orchestra.run.smithi189.stdout: filestore apply finisher threads = 3 2024-04-22T08:12:05.438 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:12:05.438 INFO:teuthology.orchestra.run.smithi189.stdout: bdev debug aio = true 2024-04-22T08:12:05.439 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug misdirected ops = true 2024-04-22T08:12:05.439 INFO:teuthology.orchestra.run.smithi189.stdout: bdev async discard = True 2024-04-22T08:12:05.439 INFO:teuthology.orchestra.run.smithi189.stdout: bdev enable discard = True 2024-04-22T08:12:05.439 INFO:teuthology.orchestra.run.smithi189.stdout: bluestore block size = 96636764160 2024-04-22T08:12:05.439 INFO:teuthology.orchestra.run.smithi189.stdout: bluestore fsck on mount = True 2024-04-22T08:12:05.439 INFO:teuthology.orchestra.run.smithi189.stdout: bluestore zero block detection = True 2024-04-22T08:12:05.439 INFO:teuthology.orchestra.run.smithi189.stdout: debug bluefs = 20 2024-04-22T08:12:05.439 INFO:teuthology.orchestra.run.smithi189.stdout: debug bluestore = 20 2024-04-22T08:12:05.439 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2024-04-22T08:12:05.439 INFO:teuthology.orchestra.run.smithi189.stdout: debug osd = 20 2024-04-22T08:12:05.439 INFO:teuthology.orchestra.run.smithi189.stdout: debug rocksdb = 10 2024-04-22T08:12:05.439 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd backfillfull_ratio = 0.85 2024-04-22T08:12:05.439 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd full ratio = 0.9 2024-04-22T08:12:05.439 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd nearfull ratio = 0.8 2024-04-22T08:12:05.440 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug verify cached snaps = True 2024-04-22T08:12:05.440 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug verify missing on start = True 2024-04-22T08:12:05.440 INFO:teuthology.orchestra.run.smithi189.stdout: osd failsafe full ratio = 0.95 2024-04-22T08:12:05.440 INFO:teuthology.orchestra.run.smithi189.stdout: osd mclock override recovery settings = True 2024-04-22T08:12:05.440 INFO:teuthology.orchestra.run.smithi189.stdout: osd memory target = 1610612736 2024-04-22T08:12:05.440 INFO:teuthology.orchestra.run.smithi189.stdout: osd objectstore = bluestore 2024-04-22T08:12:05.440 INFO:teuthology.orchestra.run.smithi189.stdout: osd op queue = debug_random 2024-04-22T08:12:05.440 INFO:teuthology.orchestra.run.smithi189.stdout: osd op queue cut off = debug_random 2024-04-22T08:12:05.440 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:12:05.440 INFO:teuthology.orchestra.run.smithi189.stdout:[mgr] 2024-04-22T08:12:05.440 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2024-04-22T08:12:05.440 INFO:teuthology.orchestra.run.smithi189.stdout: debug mgr = 20 2024-04-22T08:12:05.440 INFO:teuthology.orchestra.run.smithi189.stdout: debug mon = 20 2024-04-22T08:12:05.440 INFO:teuthology.orchestra.run.smithi189.stdout: debug auth = 20 2024-04-22T08:12:05.441 INFO:teuthology.orchestra.run.smithi189.stdout: mon reweight min pgs per osd = 4 2024-04-22T08:12:05.441 INFO:teuthology.orchestra.run.smithi189.stdout: mon reweight min bytes per osd = 10 2024-04-22T08:12:05.441 INFO:teuthology.orchestra.run.smithi189.stdout: mgr/telemetry/nag = false 2024-04-22T08:12:05.441 INFO:teuthology.orchestra.run.smithi189.stdout: debug monc = 10 2024-04-22T08:12:05.441 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:12:05.441 INFO:teuthology.orchestra.run.smithi189.stdout:[mon] 2024-04-22T08:12:05.441 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2024-04-22T08:12:05.441 INFO:teuthology.orchestra.run.smithi189.stdout: debug mon = 20 2024-04-22T08:12:05.441 INFO:teuthology.orchestra.run.smithi189.stdout: debug paxos = 20 2024-04-22T08:12:05.441 INFO:teuthology.orchestra.run.smithi189.stdout: debug auth = 20 2024-04-22T08:12:05.441 INFO:teuthology.orchestra.run.smithi189.stdout: mon data avail warn = 5 2024-04-22T08:12:05.441 INFO:teuthology.orchestra.run.smithi189.stdout: mon mgr mkfs grace = 240 2024-04-22T08:12:05.441 INFO:teuthology.orchestra.run.smithi189.stdout: mon reweight min pgs per osd = 4 2024-04-22T08:12:05.441 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd reporter subtree level = osd 2024-04-22T08:12:05.441 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd prime pg temp = true 2024-04-22T08:12:05.442 INFO:teuthology.orchestra.run.smithi189.stdout: mon reweight min bytes per osd = 10 2024-04-22T08:12:05.442 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:12:05.442 INFO:teuthology.orchestra.run.smithi189.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-22T08:12:05.442 INFO:teuthology.orchestra.run.smithi189.stdout: auth mon ticket ttl = 660 # 11m 2024-04-22T08:12:05.442 INFO:teuthology.orchestra.run.smithi189.stdout: auth service ticket ttl = 240 # 4m 2024-04-22T08:12:05.442 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:12:05.442 INFO:teuthology.orchestra.run.smithi189.stdout: # don't complain about insecure global_id in the test suite 2024-04-22T08:12:05.442 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-22T08:12:05.442 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-22T08:12:05.442 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:12:05.442 INFO:teuthology.orchestra.run.smithi189.stdout: # 1m isn't quite enough 2024-04-22T08:12:05.442 INFO:teuthology.orchestra.run.smithi189.stdout: mon_down_mkfs_grace = 2m 2024-04-22T08:12:05.442 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:12:05.442 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_filestore_osds = false 2024-04-22T08:12:05.443 INFO:teuthology.orchestra.run.smithi189.stdout: mon min osdmap epochs = 25 2024-04-22T08:12:05.443 INFO:teuthology.orchestra.run.smithi189.stdout: mon osdmap full prune interval = 2 2024-04-22T08:12:05.443 INFO:teuthology.orchestra.run.smithi189.stdout: mon osdmap full prune min = 15 2024-04-22T08:12:05.443 INFO:teuthology.orchestra.run.smithi189.stdout: mon osdmap full prune txsize = 2 2024-04-22T08:12:05.443 INFO:teuthology.orchestra.run.smithi189.stdout: mon scrub inject crc mismatch = 0.01 2024-04-22T08:12:05.443 INFO:teuthology.orchestra.run.smithi189.stdout: mon scrub inject missing keys = 0.05 2024-04-22T08:12:05.443 INFO:teuthology.orchestra.run.smithi189.stdout: mon scrub interval = 300 2024-04-22T08:12:05.443 INFO:teuthology.orchestra.run.smithi189.stdout: paxos service trim min = 5 2024-04-22T08:12:05.443 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-22T08:12:05.443 INFO:teuthology.orchestra.run.smithi189.stdout:[client] 2024-04-22T08:12:05.443 INFO:teuthology.orchestra.run.smithi189.stdout: rgw cache enabled = true 2024-04-22T08:12:05.443 INFO:teuthology.orchestra.run.smithi189.stdout: rgw enable ops log = true 2024-04-22T08:12:05.443 INFO:teuthology.orchestra.run.smithi189.stdout: rgw enable usage log = true 2024-04-22T08:12:05.443 INFO:teuthology.orchestra.run.smithi189.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-22T08:12:05.443 INFO:teuthology.orchestra.run.smithi189.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-22T08:12:05.443 INFO:teuthology.orchestra.run.smithi189.stdout: debug monc = 20 2024-04-22T08:12:05.444 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2024-04-22T08:12:05.444 INFO:teuthology.orchestra.run.smithi189.stdout:[mon.a] 2024-04-22T08:12:05.444 INFO:teuthology.orchestra.run.smithi189.stdout:[mon.c] 2024-04-22T08:12:05.444 INFO:teuthology.orchestra.run.smithi189.stdout:[mon.b] 2024-04-22T08:12:05.444 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-22T08:12:05.444 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-22T08:12:05.525 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-22T08:12:05.525 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-22T08:12:05.525 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-22T08:12:05.540 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-22T08:12:05.540 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-22T08:12:05.595 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi040.front.sepia.ceph.com 2024-04-22T08:12:05.595 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-22T08:12:05.595 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-22T08:12:05.596 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-22T08:12:05.669 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-22T08:12:05.669 DEBUG:teuthology.orchestra.run.smithi040:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-22T08:12:05.725 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi189.front.sepia.ceph.com 2024-04-22T08:12:05.725 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-22T08:12:05.725 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-22T08:12:05.725 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-22T08:12:05.759 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-22T08:12:05.759 DEBUG:teuthology.orchestra.run.smithi189:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-22T08:12:05.813 INFO:tasks.ceph:Setting up mon nodes... 2024-04-22T08:12:05.813 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-22T08:12:05.814 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-22T08:12:05.908 INFO:teuthology.orchestra.run.smithi189.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-22T08:12:05.910 INFO:tasks.ceph:Setting up mds nodes... 2024-04-22T08:12:05.910 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-22T08:12:05.911 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-22T08:12:05.949 INFO:teuthology.orchestra.run.smithi189.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-22T08:12:05.960 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-22T08:12:05.961 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi040.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi189.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-04-22T08:12:05.961 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-22T08:12:05.985 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-22T08:12:05.985 INFO:tasks.ceph:role: osd.0 2024-04-22T08:12:05.985 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi040.front.sepia.ceph.com 2024-04-22T08:12:05.985 DEBUG:teuthology.orchestra.run.smithi040:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-22T08:12:06.057 INFO:teuthology.orchestra.run.smithi040.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-22T08:12:06.057 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-22T08:12:06.057 INFO:teuthology.orchestra.run.smithi040.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-22T08:12:06.057 INFO:teuthology.orchestra.run.smithi040.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-22T08:12:06.057 INFO:teuthology.orchestra.run.smithi040.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-22T08:12:06.057 INFO:teuthology.orchestra.run.smithi040.stdout: = sunit=0 swidth=0 blks 2024-04-22T08:12:06.057 INFO:teuthology.orchestra.run.smithi040.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-22T08:12:06.058 INFO:teuthology.orchestra.run.smithi040.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-22T08:12:06.058 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-22T08:12:06.058 INFO:teuthology.orchestra.run.smithi040.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-22T08:12:06.565 INFO:teuthology.orchestra.run.smithi040.stdout:Discarding blocks...Done. 2024-04-22T08:12:06.567 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi040.front.sepia.ceph.com -o noatime 2024-04-22T08:12:06.568 DEBUG:teuthology.orchestra.run.smithi040:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-22T08:12:06.668 DEBUG:teuthology.orchestra.run.smithi040:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-22T08:12:06.699 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-22T08:12:06.764 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-22T08:12:06.764 INFO:tasks.ceph:role: osd.1 2024-04-22T08:12:06.765 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi040.front.sepia.ceph.com 2024-04-22T08:12:06.765 DEBUG:teuthology.orchestra.run.smithi040:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-22T08:12:06.837 INFO:teuthology.orchestra.run.smithi040.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-22T08:12:06.837 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-22T08:12:06.837 INFO:teuthology.orchestra.run.smithi040.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-22T08:12:06.837 INFO:teuthology.orchestra.run.smithi040.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-22T08:12:06.837 INFO:teuthology.orchestra.run.smithi040.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-22T08:12:06.837 INFO:teuthology.orchestra.run.smithi040.stdout: = sunit=0 swidth=0 blks 2024-04-22T08:12:06.837 INFO:teuthology.orchestra.run.smithi040.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-22T08:12:06.837 INFO:teuthology.orchestra.run.smithi040.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-22T08:12:06.838 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-22T08:12:06.838 INFO:teuthology.orchestra.run.smithi040.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-22T08:12:07.298 INFO:teuthology.orchestra.run.smithi040.stdout:Discarding blocks...Done. 2024-04-22T08:12:07.300 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi040.front.sepia.ceph.com -o noatime 2024-04-22T08:12:07.300 DEBUG:teuthology.orchestra.run.smithi040:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-22T08:12:07.392 DEBUG:teuthology.orchestra.run.smithi040:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-22T08:12:07.422 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-22T08:12:07.488 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-22T08:12:07.488 INFO:tasks.ceph:role: osd.2 2024-04-22T08:12:07.488 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi040.front.sepia.ceph.com 2024-04-22T08:12:07.488 DEBUG:teuthology.orchestra.run.smithi040:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-22T08:12:07.561 INFO:teuthology.orchestra.run.smithi040.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-22T08:12:07.561 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-22T08:12:07.561 INFO:teuthology.orchestra.run.smithi040.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-22T08:12:07.561 INFO:teuthology.orchestra.run.smithi040.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-22T08:12:07.561 INFO:teuthology.orchestra.run.smithi040.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-22T08:12:07.561 INFO:teuthology.orchestra.run.smithi040.stdout: = sunit=0 swidth=0 blks 2024-04-22T08:12:07.561 INFO:teuthology.orchestra.run.smithi040.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-22T08:12:07.561 INFO:teuthology.orchestra.run.smithi040.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-22T08:12:07.561 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-22T08:12:07.561 INFO:teuthology.orchestra.run.smithi040.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-22T08:12:08.047 INFO:teuthology.orchestra.run.smithi040.stdout:Discarding blocks...Done. 2024-04-22T08:12:08.049 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi040.front.sepia.ceph.com -o noatime 2024-04-22T08:12:08.049 DEBUG:teuthology.orchestra.run.smithi040:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-22T08:12:08.202 DEBUG:teuthology.orchestra.run.smithi040:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-22T08:12:08.235 DEBUG:teuthology.orchestra.run.smithi040:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-22T08:12:08.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:08.327+0000 7f0378bb8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-22T08:12:08.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:08.328+0000 7f0378bb8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-22T08:12:08.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:08.328+0000 7f0378bb8640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-22T08:12:08.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:08.328+0000 7f0378bb8640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-22T08:12:08.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:08.328+0000 7f0378bb8640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-22T08:12:08.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:08.336+0000 7f0378bb8640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-22T08:12:11.187 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-22T08:12:11.216 DEBUG:teuthology.orchestra.run.smithi040:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-22T08:12:11.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:11.305+0000 7fa8ccc63640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-22T08:12:11.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:11.305+0000 7fa8ccc63640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-22T08:12:11.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:11.305+0000 7fa8ccc63640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-22T08:12:11.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:11.305+0000 7fa8ccc63640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-22T08:12:11.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:11.305+0000 7fa8ccc63640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-22T08:12:11.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:11.306+0000 7fa8ccc63640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-22T08:12:14.156 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-22T08:12:14.184 DEBUG:teuthology.orchestra.run.smithi040:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-22T08:12:14.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:14.270+0000 7f651e00c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-22T08:12:14.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:14.271+0000 7f651e00c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-22T08:12:14.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:14.271+0000 7f651e00c640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-22T08:12:14.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:14.271+0000 7f651e00c640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-22T08:12:14.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:14.271+0000 7f651e00c640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-22T08:12:14.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:14.271+0000 7f651e00c640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-22T08:12:17.122 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-22T08:12:17.151 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-22T08:12:17.180 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-04-22T08:12:17.180 INFO:tasks.ceph:role: osd.3 2024-04-22T08:12:17.180 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi189.front.sepia.ceph.com 2024-04-22T08:12:17.181 DEBUG:teuthology.orchestra.run.smithi189:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-22T08:12:17.254 INFO:teuthology.orchestra.run.smithi189.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-22T08:12:17.255 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-22T08:12:17.255 INFO:teuthology.orchestra.run.smithi189.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-22T08:12:17.255 INFO:teuthology.orchestra.run.smithi189.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-22T08:12:17.255 INFO:teuthology.orchestra.run.smithi189.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-22T08:12:17.255 INFO:teuthology.orchestra.run.smithi189.stdout: = sunit=0 swidth=0 blks 2024-04-22T08:12:17.255 INFO:teuthology.orchestra.run.smithi189.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-22T08:12:17.255 INFO:teuthology.orchestra.run.smithi189.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-22T08:12:17.255 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-22T08:12:17.255 INFO:teuthology.orchestra.run.smithi189.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-22T08:12:17.767 INFO:teuthology.orchestra.run.smithi189.stdout:Discarding blocks...Done. 2024-04-22T08:12:17.770 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi189.front.sepia.ceph.com -o noatime 2024-04-22T08:12:17.770 DEBUG:teuthology.orchestra.run.smithi189:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-04-22T08:12:17.922 DEBUG:teuthology.orchestra.run.smithi189:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-22T08:12:17.957 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-22T08:12:18.025 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-04-22T08:12:18.025 INFO:tasks.ceph:role: osd.4 2024-04-22T08:12:18.025 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi189.front.sepia.ceph.com 2024-04-22T08:12:18.025 DEBUG:teuthology.orchestra.run.smithi189:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-22T08:12:18.098 INFO:teuthology.orchestra.run.smithi189.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-22T08:12:18.098 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-22T08:12:18.098 INFO:teuthology.orchestra.run.smithi189.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-22T08:12:18.098 INFO:teuthology.orchestra.run.smithi189.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-22T08:12:18.098 INFO:teuthology.orchestra.run.smithi189.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-22T08:12:18.098 INFO:teuthology.orchestra.run.smithi189.stdout: = sunit=0 swidth=0 blks 2024-04-22T08:12:18.098 INFO:teuthology.orchestra.run.smithi189.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-22T08:12:18.098 INFO:teuthology.orchestra.run.smithi189.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-22T08:12:18.098 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-22T08:12:18.098 INFO:teuthology.orchestra.run.smithi189.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-22T08:12:18.543 INFO:teuthology.orchestra.run.smithi189.stdout:Discarding blocks...Done. 2024-04-22T08:12:18.545 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi189.front.sepia.ceph.com -o noatime 2024-04-22T08:12:18.545 DEBUG:teuthology.orchestra.run.smithi189:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-04-22T08:12:18.635 DEBUG:teuthology.orchestra.run.smithi189:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-22T08:12:18.662 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-22T08:12:18.726 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-04-22T08:12:18.726 INFO:tasks.ceph:role: osd.5 2024-04-22T08:12:18.726 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi189.front.sepia.ceph.com 2024-04-22T08:12:18.726 DEBUG:teuthology.orchestra.run.smithi189:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-22T08:12:18.797 INFO:teuthology.orchestra.run.smithi189.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-22T08:12:18.797 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-22T08:12:18.797 INFO:teuthology.orchestra.run.smithi189.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-22T08:12:18.797 INFO:teuthology.orchestra.run.smithi189.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-22T08:12:18.797 INFO:teuthology.orchestra.run.smithi189.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-22T08:12:18.798 INFO:teuthology.orchestra.run.smithi189.stdout: = sunit=0 swidth=0 blks 2024-04-22T08:12:18.798 INFO:teuthology.orchestra.run.smithi189.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-22T08:12:18.798 INFO:teuthology.orchestra.run.smithi189.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-22T08:12:18.798 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-22T08:12:18.798 INFO:teuthology.orchestra.run.smithi189.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-22T08:12:19.284 INFO:teuthology.orchestra.run.smithi189.stdout:Discarding blocks...Done. 2024-04-22T08:12:19.287 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi189.front.sepia.ceph.com -o noatime 2024-04-22T08:12:19.287 DEBUG:teuthology.orchestra.run.smithi189:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-04-22T08:12:19.435 DEBUG:teuthology.orchestra.run.smithi189:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-22T08:12:19.465 DEBUG:teuthology.orchestra.run.smithi189:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-22T08:12:19.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:19.562+0000 7f9a566ae640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-22T08:12:19.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:19.564+0000 7f9a566ae640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-22T08:12:19.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:19.564+0000 7f9a566ae640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-22T08:12:19.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:19.564+0000 7f9a566ae640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-22T08:12:19.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:19.564+0000 7f9a566ae640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-22T08:12:19.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:19.570+0000 7f9a566ae640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-22T08:12:22.415 DEBUG:teuthology.orchestra.run.smithi189:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-22T08:12:22.443 DEBUG:teuthology.orchestra.run.smithi189:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-22T08:12:22.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:22.532+0000 7fd3697b8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-22T08:12:22.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:22.532+0000 7fd3697b8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-22T08:12:22.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:22.532+0000 7fd3697b8640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-22T08:12:22.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:22.532+0000 7fd3697b8640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-22T08:12:22.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:22.532+0000 7fd3697b8640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-22T08:12:22.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:22.533+0000 7fd3697b8640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-22T08:12:25.372 DEBUG:teuthology.orchestra.run.smithi189:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-22T08:12:25.401 DEBUG:teuthology.orchestra.run.smithi189:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-22T08:12:25.489 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:25.489+0000 7f571a018640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-22T08:12:25.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:25.490+0000 7f571a018640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-22T08:12:25.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:25.490+0000 7f571a018640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-22T08:12:25.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:25.490+0000 7f571a018640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-22T08:12:25.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:25.490+0000 7f571a018640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-22T08:12:25.490 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:25.490+0000 7f571a018640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-22T08:12:28.345 DEBUG:teuthology.orchestra.run.smithi189:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-22T08:12:28.373 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-22T08:12:28.374 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-22T08:12:28.374 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-22T08:12:28.404 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-22T08:12:28.404 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-22T08:12:28.473 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-22T08:12:28.473 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-22T08:12:28.542 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-22T08:12:28.543 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-22T08:12:28.571 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-22T08:12:28.571 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-22T08:12:28.638 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-22T08:12:28.638 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-22T08:12:28.707 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-22T08:12:28.707 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-22T08:12:28.772 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-22T08:12:28.772 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-22T08:12:28.828 INFO:tasks.ceph:Adding keys to all mons... 2024-04-22T08:12:28.829 DEBUG:teuthology.orchestra.run.smithi040:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-22T08:12:28.831 DEBUG:teuthology.orchestra.run.smithi189:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-22T08:12:28.872 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.0] 2024-04-22T08:12:28.872 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDYGyZm44qXExAA4xAOHTXYRkA70LJ+Rex0jQ== 2024-04-22T08:12:28.872 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.1] 2024-04-22T08:12:28.872 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDbGyZmR2o9EhAAgOBKOwQ269lMrxEi7d1O+g== 2024-04-22T08:12:28.872 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.2] 2024-04-22T08:12:28.872 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDeGyZmOTU/EBAA2wWXQvQEpmO6fTd6Xyd0iw== 2024-04-22T08:12:28.872 INFO:teuthology.orchestra.run.smithi040.stdout:[mgr.x] 2024-04-22T08:12:28.873 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDVGyZmClkzNhAAPYYyaifm1KN28ioVC+q7Eg== 2024-04-22T08:12:28.873 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.3] 2024-04-22T08:12:28.873 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDjGyZmHquQIRAA9syS7+Er0o9ag78VyU2RnQ== 2024-04-22T08:12:28.873 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.4] 2024-04-22T08:12:28.873 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDmGyZmeaLJHxAAltKCTmKUMRW7FE+296981Q== 2024-04-22T08:12:28.873 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.5] 2024-04-22T08:12:28.873 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDpGyZmpgk/HRAANWAx5TySvep6I7zqEwQiNQ== 2024-04-22T08:12:28.873 INFO:teuthology.orchestra.run.smithi040.stdout:[client.0] 2024-04-22T08:12:28.873 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDVGyZmLOOoOBAA4dCLEd1Sc2p8KLj/u7Da7g== 2024-04-22T08:12:28.892 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.0] 2024-04-22T08:12:28.892 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQDYGyZm44qXExAA4xAOHTXYRkA70LJ+Rex0jQ== 2024-04-22T08:12:28.892 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.1] 2024-04-22T08:12:28.892 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQDbGyZmR2o9EhAAgOBKOwQ269lMrxEi7d1O+g== 2024-04-22T08:12:28.892 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.2] 2024-04-22T08:12:28.892 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQDeGyZmOTU/EBAA2wWXQvQEpmO6fTd6Xyd0iw== 2024-04-22T08:12:28.892 INFO:teuthology.orchestra.run.smithi189.stdout:[mgr.x] 2024-04-22T08:12:28.892 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQDVGyZmClkzNhAAPYYyaifm1KN28ioVC+q7Eg== 2024-04-22T08:12:28.892 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.3] 2024-04-22T08:12:28.892 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQDjGyZmHquQIRAA9syS7+Er0o9ag78VyU2RnQ== 2024-04-22T08:12:28.893 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.4] 2024-04-22T08:12:28.893 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQDmGyZmeaLJHxAAltKCTmKUMRW7FE+296981Q== 2024-04-22T08:12:28.893 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.5] 2024-04-22T08:12:28.893 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQDpGyZmpgk/HRAANWAx5TySvep6I7zqEwQiNQ== 2024-04-22T08:12:28.893 INFO:teuthology.orchestra.run.smithi189.stdout:[client.0] 2024-04-22T08:12:28.893 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQDVGyZmLOOoOBAA4dCLEd1Sc2p8KLj/u7Da7g== 2024-04-22T08:12:28.894 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T08:12:28.918 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T08:12:28.983 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T08:12:28.986 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T08:12:29.035 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T08:12:29.038 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T08:12:29.086 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-22T08:12:29.089 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-22T08:12:29.138 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T08:12:29.141 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T08:12:29.189 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T08:12:29.192 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T08:12:29.243 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T08:12:29.245 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T08:12:29.295 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-22T08:12:29.297 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-22T08:12:29.349 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-22T08:12:29.350 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-22T08:12:29.374 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-22T08:12:29.468 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-22T08:12:29.492 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-22T08:12:29.556 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-22T08:12:29.651 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-22T08:12:29.675 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-22T08:12:29.699 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-22T08:12:29.793 DEBUG:teuthology.orchestra.run.smithi189:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-22T08:12:29.816 DEBUG:teuthology.orchestra.run.smithi040:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-22T08:12:29.818 DEBUG:teuthology.orchestra.run.smithi189:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-22T08:12:29.872 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-22T08:12:29.872 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-22T08:12:29.872 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-22T08:12:29.874 INFO:tasks.ceph.mon.a:Started 2024-04-22T08:12:29.874 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-22T08:12:29.874 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-22T08:12:29.876 INFO:tasks.ceph.mon.c:Started 2024-04-22T08:12:29.876 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-22T08:12:29.876 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-22T08:12:29.914 INFO:tasks.ceph.mon.b:Started 2024-04-22T08:12:29.914 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-22T08:12:29.915 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-22T08:12:29.915 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-22T08:12:29.917 INFO:tasks.ceph.mgr.x:Started 2024-04-22T08:12:29.917 INFO:tasks.ceph:Setting crush tunables to default 2024-04-22T08:12:29.917 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --cluster ceph osd crush tunables default 2024-04-22T08:12:30.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.018+0000 7f5de343f640 10 monclient: get_monmap_and_config 2024-04-22T08:12:30.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.018+0000 7f5de343f640 10 monclient: build_initial_monmap 2024-04-22T08:12:30.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.018+0000 7f5de343f640 10 monclient: monmap: 2024-04-22T08:12:30.025 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:30.025 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:30.025 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:30.019873+0000 2024-04-22T08:12:30.025 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:30.019873+0000 2024-04-22T08:12:30.025 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:30.025 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:30.025 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:12:30.025 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:12:30.025 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:12:30.025 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:12:30.025 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:12:30.026 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:30.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.018+0000 7f5de343f640 1 Processor -- start 2024-04-22T08:12:30.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.018+0000 7f5de343f640 1 -- start start 2024-04-22T08:12:30.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.018+0000 7f5de343f640 10 monclient: init 2024-04-22T08:12:30.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.019+0000 7f5de343f640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:30.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.019+0000 7f5de343f640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-04-22T08:12:30.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.019+0000 7f5de343f640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ddc09ee10 0x7f5ddc1362c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:30.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.019+0000 7f5de343f640 10 monclient(hunting): picked mon.noname-b con 0x7f5ddc09ee10 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:30.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.019+0000 7f5de343f640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5ddc0a2d80 0x7f5ddc0a31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:30.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.019+0000 7f5de343f640 10 monclient(hunting): picked mon.noname-a con 0x7f5ddc0a2d80 addr v2:172.21.15.40:3300/0 2024-04-22T08:12:30.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.019+0000 7f5de343f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5ddc0a3720 addr v1:172.21.15.40:6789/0 2024-04-22T08:12:30.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.019+0000 7f5de343f640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ddc0a4260 con 0x7f5ddc0a3720 2024-04-22T08:12:30.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.019+0000 7f5de343f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:30.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.019+0000 7f5de343f640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ddc09fc90 con 0x7f5ddc0a2d80 2024-04-22T08:12:30.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.019+0000 7f5de343f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:30.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.019+0000 7f5de343f640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ddc0a3b90 con 0x7f5ddc09ee10 2024-04-22T08:12:30.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.019+0000 7f5de343f640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:30.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.019+0000 7f5de343f640 10 monclient(hunting): authenticate will time out at 824.014161s 2024-04-22T08:12:30.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.019+0000 7f5de09da640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5ddc0a2d80 0x7f5ddc0a31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:30.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.019+0000 7f5de11db640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ddc09ee10 0x7f5ddc1362c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:30.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.019+0000 7f5de19dc640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f5ddc0a3720 0x7f5ddc132b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:55444/0 (socket says 172.21.15.40:55444) 2024-04-22T08:12:30.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.019+0000 7f5de11db640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ddc09ee10 0x7f5ddc1362c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:34694/0 (socket says 172.21.15.40:34694) 2024-04-22T08:12:30.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.019+0000 7f5de19dc640 1 -- 172.21.15.40:0/1093751737 learned_addr learned my addr 172.21.15.40:0/1093751737 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:12:30.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.019+0000 7f5de09da640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5ddc0a2d80 0x7f5ddc0a31e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57682/0 (socket says 172.21.15.40:57682) 2024-04-22T08:12:30.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.019+0000 7f5de11db640 10 monclient(hunting): get_auth_request con 0x7f5ddc09ee10 auth_method 0 2024-04-22T08:12:30.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.019+0000 7f5de11db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:30.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.019+0000 7f5de11db640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:30.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.019+0000 7f5de11db640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:30.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.019+0000 7f5de09da640 10 monclient(hunting): get_auth_request con 0x7f5ddc0a2d80 auth_method 0 2024-04-22T08:12:30.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.019+0000 7f5de09da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:30.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.019+0000 7f5de09da640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:30.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.019+0000 7f5de09da640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:30.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.019+0000 7f5de11db640 1 -- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ddc09ee10 msgr2=0x7f5ddc1362c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:12:30.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.019+0000 7f5de11db640 1 -- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ddc09ee10 msgr2=0x7f5ddc1362c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:12:30.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.019+0000 7f5de11db640 1 --2- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ddc09ee10 0x7f5ddc1362c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:12:30.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.019+0000 7f5de09da640 1 -- 172.21.15.40:0/1093751737 >> v2:172.21.15.40:3300/0 conn(0x7f5ddc0a2d80 msgr2=0x7f5ddc0a31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:12:30.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.019+0000 7f5de11db640 1 --2- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ddc09ee10 0x7f5ddc1362c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:12:30.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.019+0000 7f5de09da640 1 -- 172.21.15.40:0/1093751737 >> v2:172.21.15.40:3300/0 conn(0x7f5ddc0a2d80 msgr2=0x7f5ddc0a31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:12:30.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.019+0000 7f5de09da640 1 --2- 172.21.15.40:0/1093751737 >> v2:172.21.15.40:3300/0 conn(0x7f5ddc0a2d80 0x7f5ddc0a31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:12:30.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.019+0000 7f5de09da640 1 --2- 172.21.15.40:0/1093751737 >> v2:172.21.15.40:3300/0 conn(0x7f5ddc0a2d80 0x7f5ddc0a31e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:12:30.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.021+0000 7f5de19dc640 1 -- 172.21.15.40:0/1093751737 >> v1:172.21.15.40:6789/0 conn(0x7f5ddc0a3720 legacy=0x7f5ddc132b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:30.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.220+0000 7f5de11db640 1 --2- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ddc09ee10 0x7f5ddc1362c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:30.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.220+0000 7f5de09da640 1 --2- 172.21.15.40:0/1093751737 >> v2:172.21.15.40:3300/0 conn(0x7f5ddc0a2d80 0x7f5ddc0a31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:30.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.220+0000 7f5de11db640 10 monclient(hunting): get_auth_request con 0x7f5ddc09ee10 auth_method 0 2024-04-22T08:12:30.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.220+0000 7f5de11db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:30.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.220+0000 7f5de11db640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:30.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.220+0000 7f5de11db640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:12:30.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.220+0000 7f5de09da640 10 monclient(hunting): get_auth_request con 0x7f5ddc0a2d80 auth_method 0 2024-04-22T08:12:30.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.220+0000 7f5de09da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:30.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.220+0000 7f5de09da640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:30.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.220+0000 7f5de09da640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:12:30.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.220+0000 7f5de09da640 1 -- 172.21.15.40:0/1093751737 >> v2:172.21.15.40:3300/0 conn(0x7f5ddc0a2d80 msgr2=0x7f5ddc0a31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:12:30.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.220+0000 7f5de09da640 1 -- 172.21.15.40:0/1093751737 >> v2:172.21.15.40:3300/0 conn(0x7f5ddc0a2d80 msgr2=0x7f5ddc0a31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:12:30.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.220+0000 7f5de09da640 1 --2- 172.21.15.40:0/1093751737 >> v2:172.21.15.40:3300/0 conn(0x7f5ddc0a2d80 0x7f5ddc0a31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:12:30.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.220+0000 7f5de11db640 1 -- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ddc09ee10 msgr2=0x7f5ddc1362c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:12:30.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.220+0000 7f5de11db640 1 -- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ddc09ee10 msgr2=0x7f5ddc1362c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:12:30.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.220+0000 7f5de11db640 1 --2- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ddc09ee10 0x7f5ddc1362c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:12:30.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.220+0000 7f5de09da640 1 --2- 172.21.15.40:0/1093751737 >> v2:172.21.15.40:3300/0 conn(0x7f5ddc0a2d80 0x7f5ddc0a31e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:12:30.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.220+0000 7f5de11db640 1 --2- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ddc09ee10 0x7f5ddc1362c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:12:30.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.621+0000 7f5de09da640 1 --2- 172.21.15.40:0/1093751737 >> v2:172.21.15.40:3300/0 conn(0x7f5ddc0a2d80 0x7f5ddc0a31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:30.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.621+0000 7f5de11db640 1 --2- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ddc09ee10 0x7f5ddc1362c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:30.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.621+0000 7f5de11db640 10 monclient(hunting): get_auth_request con 0x7f5ddc09ee10 auth_method 0 2024-04-22T08:12:30.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.621+0000 7f5de11db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:30.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.621+0000 7f5de11db640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:30.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.621+0000 7f5de11db640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:12:30.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.621+0000 7f5de09da640 10 monclient(hunting): get_auth_request con 0x7f5ddc0a2d80 auth_method 0 2024-04-22T08:12:30.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.621+0000 7f5de09da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:30.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.621+0000 7f5de09da640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:30.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.621+0000 7f5de09da640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:12:30.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.621+0000 7f5de09da640 1 -- 172.21.15.40:0/1093751737 >> v2:172.21.15.40:3300/0 conn(0x7f5ddc0a2d80 msgr2=0x7f5ddc0a31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:12:30.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.621+0000 7f5de09da640 1 -- 172.21.15.40:0/1093751737 >> v2:172.21.15.40:3300/0 conn(0x7f5ddc0a2d80 msgr2=0x7f5ddc0a31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:12:30.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.621+0000 7f5de11db640 1 -- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ddc09ee10 msgr2=0x7f5ddc1362c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:12:30.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.621+0000 7f5de09da640 1 --2- 172.21.15.40:0/1093751737 >> v2:172.21.15.40:3300/0 conn(0x7f5ddc0a2d80 0x7f5ddc0a31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:12:30.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.621+0000 7f5de11db640 1 -- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ddc09ee10 msgr2=0x7f5ddc1362c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:12:30.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.621+0000 7f5de11db640 1 --2- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ddc09ee10 0x7f5ddc1362c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:12:30.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.621+0000 7f5de09da640 1 --2- 172.21.15.40:0/1093751737 >> v2:172.21.15.40:3300/0 conn(0x7f5ddc0a2d80 0x7f5ddc0a31e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:12:30.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:30.621+0000 7f5de11db640 1 --2- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ddc09ee10 0x7f5ddc1362c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:12:31.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:31.423+0000 7f5de11db640 1 --2- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ddc09ee10 0x7f5ddc1362c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:31.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:31.423+0000 7f5de09da640 1 --2- 172.21.15.40:0/1093751737 >> v2:172.21.15.40:3300/0 conn(0x7f5ddc0a2d80 0x7f5ddc0a31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:31.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:31.423+0000 7f5de11db640 10 monclient(hunting): get_auth_request con 0x7f5ddc09ee10 auth_method 0 2024-04-22T08:12:31.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:31.423+0000 7f5de11db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:31.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:31.423+0000 7f5de11db640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:31.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:31.423+0000 7f5de11db640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:12:31.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:31.423+0000 7f5de09da640 10 monclient(hunting): get_auth_request con 0x7f5ddc0a2d80 auth_method 0 2024-04-22T08:12:31.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:31.423+0000 7f5de09da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:31.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:31.423+0000 7f5de09da640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:31.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:31.423+0000 7f5de09da640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:12:31.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:31.423+0000 7f5de11db640 1 -- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ddc09ee10 msgr2=0x7f5ddc1362c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:12:31.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:31.423+0000 7f5de09da640 1 -- 172.21.15.40:0/1093751737 >> v2:172.21.15.40:3300/0 conn(0x7f5ddc0a2d80 msgr2=0x7f5ddc0a31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:12:31.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:31.423+0000 7f5de11db640 1 -- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ddc09ee10 msgr2=0x7f5ddc1362c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:12:31.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:31.423+0000 7f5de09da640 1 -- 172.21.15.40:0/1093751737 >> v2:172.21.15.40:3300/0 conn(0x7f5ddc0a2d80 msgr2=0x7f5ddc0a31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:12:31.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:31.423+0000 7f5de11db640 1 --2- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ddc09ee10 0x7f5ddc1362c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:12:31.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:31.423+0000 7f5de09da640 1 --2- 172.21.15.40:0/1093751737 >> v2:172.21.15.40:3300/0 conn(0x7f5ddc0a2d80 0x7f5ddc0a31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:12:31.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:31.423+0000 7f5de11db640 1 --2- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ddc09ee10 0x7f5ddc1362c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T08:12:31.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:31.423+0000 7f5de09da640 1 --2- 172.21.15.40:0/1093751737 >> v2:172.21.15.40:3300/0 conn(0x7f5ddc0a2d80 0x7f5ddc0a31e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T08:12:33.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:33.020+0000 7f5dd2ffd640 10 monclient(hunting): tick 2024-04-22T08:12:33.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:33.020+0000 7f5dd2ffd640 10 monclient(hunting): _check_auth_tickets 2024-04-22T08:12:33.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:33.020+0000 7f5dd2ffd640 1 monclient(hunting): continuing hunt 2024-04-22T08:12:33.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:33.020+0000 7f5dd2ffd640 10 monclient(hunting): _reopen_session rank -1 2024-04-22T08:12:33.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:33.020+0000 7f5dd2ffd640 1 -- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ddc09ee10 msgr2=0x7f5ddc1362c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:12:33.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:33.020+0000 7f5dd2ffd640 1 --2- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ddc09ee10 0x7f5ddc1362c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:33.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:33.020+0000 7f5dd2ffd640 1 -- 172.21.15.40:0/1093751737 >> v2:172.21.15.40:3300/0 conn(0x7f5ddc0a2d80 msgr2=0x7f5ddc0a31e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:12:33.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:33.020+0000 7f5dd2ffd640 1 --2- 172.21.15.40:0/1093751737 >> v2:172.21.15.40:3300/0 conn(0x7f5ddc0a2d80 0x7f5ddc0a31e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:33.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:33.020+0000 7f5dd2ffd640 1 -- 172.21.15.40:0/1093751737 >> v1:172.21.15.40:6789/0 conn(0x7f5ddc0a3720 legacy=0x7f5ddc132b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:33.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:33.020+0000 7f5dd2ffd640 10 monclient: _add_conns ranks=[1,4] 2024-04-22T08:12:33.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:33.020+0000 7f5dd2ffd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5dc0003760 addr v1:172.21.15.189:6789/0 2024-04-22T08:12:33.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:33.020+0000 7f5dd2ffd640 1 --2- 172.21.15.40:0/1093751737 >> v2:172.21.15.189:3300/0 conn(0x7f5dc0007490 0x7f5dc0009950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:33.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:33.020+0000 7f5dd2ffd640 10 monclient(hunting): picked mon.noname-c con 0x7f5dc0007490 addr v2:172.21.15.189:3300/0 2024-04-22T08:12:33.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:33.020+0000 7f5dd2ffd640 1 -- 172.21.15.40:0/1093751737 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ddc136980 con 0x7f5dc0003760 2024-04-22T08:12:33.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:33.020+0000 7f5dd2ffd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:33.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:33.020+0000 7f5dd2ffd640 1 -- 172.21.15.40:0/1093751737 --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ddc09fc90 con 0x7f5dc0007490 2024-04-22T08:12:33.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:33.020+0000 7f5dd2ffd640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:33.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:33.020+0000 7f5de09da640 1 --2- 172.21.15.40:0/1093751737 >> v2:172.21.15.189:3300/0 conn(0x7f5dc0007490 0x7f5dc0009950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:33.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:33.020+0000 7f5de09da640 10 monclient(hunting): get_auth_request con 0x7f5dc0007490 auth_method 0 2024-04-22T08:12:33.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:33.020+0000 7f5de09da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:33.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:33.020+0000 7f5de09da640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:33.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:33.020+0000 7f5de09da640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:33.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:33.021+0000 7f5de09da640 1 -- 172.21.15.40:0/1093751737 >> v2:172.21.15.189:3300/0 conn(0x7f5dc0007490 msgr2=0x7f5dc0009950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:12:33.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:33.021+0000 7f5de09da640 1 -- 172.21.15.40:0/1093751737 >> v2:172.21.15.189:3300/0 conn(0x7f5dc0007490 msgr2=0x7f5dc0009950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:12:33.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:33.021+0000 7f5de09da640 1 --2- 172.21.15.40:0/1093751737 >> v2:172.21.15.189:3300/0 conn(0x7f5dc0007490 0x7f5dc0009950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:12:33.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:33.021+0000 7f5de09da640 1 --2- 172.21.15.40:0/1093751737 >> v2:172.21.15.189:3300/0 conn(0x7f5dc0007490 0x7f5dc0009950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:12:33.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:33.023+0000 7f5de11db640 1 -- 172.21.15.40:0/1093751737 >> v1:172.21.15.189:6789/0 conn(0x7f5dc0003760 legacy=0x7f5dc0005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:33.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:33.222+0000 7f5de09da640 1 --2- 172.21.15.40:0/1093751737 >> v2:172.21.15.189:3300/0 conn(0x7f5dc0007490 0x7f5dc0009950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:33.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:33.222+0000 7f5de09da640 10 monclient(hunting): get_auth_request con 0x7f5dc0007490 auth_method 0 2024-04-22T08:12:33.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:33.222+0000 7f5de09da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:33.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:33.222+0000 7f5de09da640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:33.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:33.222+0000 7f5de09da640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:12:33.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:33.222+0000 7f5de09da640 1 -- 172.21.15.40:0/1093751737 >> v2:172.21.15.189:3300/0 conn(0x7f5dc0007490 msgr2=0x7f5dc0009950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:12:33.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:33.222+0000 7f5de09da640 1 -- 172.21.15.40:0/1093751737 >> v2:172.21.15.189:3300/0 conn(0x7f5dc0007490 msgr2=0x7f5dc0009950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:12:33.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:33.222+0000 7f5de09da640 1 --2- 172.21.15.40:0/1093751737 >> v2:172.21.15.189:3300/0 conn(0x7f5dc0007490 0x7f5dc0009950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:12:33.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:33.222+0000 7f5de09da640 1 --2- 172.21.15.40:0/1093751737 >> v2:172.21.15.189:3300/0 conn(0x7f5dc0007490 0x7f5dc0009950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:12:33.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:33.623+0000 7f5de09da640 1 --2- 172.21.15.40:0/1093751737 >> v2:172.21.15.189:3300/0 conn(0x7f5dc0007490 0x7f5dc0009950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:33.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:33.623+0000 7f5de09da640 10 monclient(hunting): get_auth_request con 0x7f5dc0007490 auth_method 0 2024-04-22T08:12:33.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:33.623+0000 7f5de09da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:33.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:33.623+0000 7f5de09da640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:33.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:33.623+0000 7f5de09da640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:12:33.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:33.623+0000 7f5de09da640 1 -- 172.21.15.40:0/1093751737 >> v2:172.21.15.189:3300/0 conn(0x7f5dc0007490 msgr2=0x7f5dc0009950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:12:33.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:33.623+0000 7f5de09da640 1 -- 172.21.15.40:0/1093751737 >> v2:172.21.15.189:3300/0 conn(0x7f5dc0007490 msgr2=0x7f5dc0009950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:12:33.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:33.623+0000 7f5de09da640 1 --2- 172.21.15.40:0/1093751737 >> v2:172.21.15.189:3300/0 conn(0x7f5dc0007490 0x7f5dc0009950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:12:33.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:33.623+0000 7f5de09da640 1 --2- 172.21.15.40:0/1093751737 >> v2:172.21.15.189:3300/0 conn(0x7f5dc0007490 0x7f5dc0009950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:12:34.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:34.425+0000 7f5de09da640 1 --2- 172.21.15.40:0/1093751737 >> v2:172.21.15.189:3300/0 conn(0x7f5dc0007490 0x7f5dc0009950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:34.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:34.425+0000 7f5de09da640 10 monclient(hunting): get_auth_request con 0x7f5dc0007490 auth_method 0 2024-04-22T08:12:34.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:34.425+0000 7f5de09da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:34.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:34.425+0000 7f5de09da640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:34.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:34.425+0000 7f5de09da640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:12:34.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:34.425+0000 7f5de09da640 1 -- 172.21.15.40:0/1093751737 >> v2:172.21.15.189:3300/0 conn(0x7f5dc0007490 msgr2=0x7f5dc0009950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:12:34.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:34.425+0000 7f5de09da640 1 -- 172.21.15.40:0/1093751737 >> v2:172.21.15.189:3300/0 conn(0x7f5dc0007490 msgr2=0x7f5dc0009950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:12:34.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:34.425+0000 7f5de09da640 1 --2- 172.21.15.40:0/1093751737 >> v2:172.21.15.189:3300/0 conn(0x7f5dc0007490 0x7f5dc0009950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:12:34.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:34.425+0000 7f5de09da640 1 --2- 172.21.15.40:0/1093751737 >> v2:172.21.15.189:3300/0 conn(0x7f5dc0007490 0x7f5dc0009950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T08:12:36.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.020+0000 7f5dd2ffd640 10 monclient(hunting): tick 2024-04-22T08:12:36.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.020+0000 7f5dd2ffd640 10 monclient(hunting): _check_auth_tickets 2024-04-22T08:12:36.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.020+0000 7f5dd2ffd640 1 monclient(hunting): continuing hunt 2024-04-22T08:12:36.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.020+0000 7f5dd2ffd640 10 monclient(hunting): _reopen_session rank -1 2024-04-22T08:12:36.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.020+0000 7f5dd2ffd640 1 -- 172.21.15.40:0/1093751737 >> v2:172.21.15.189:3300/0 conn(0x7f5dc0007490 msgr2=0x7f5dc0009950 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:12:36.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.020+0000 7f5dd2ffd640 1 --2- 172.21.15.40:0/1093751737 >> v2:172.21.15.189:3300/0 conn(0x7f5dc0007490 0x7f5dc0009950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:36.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.020+0000 7f5dd2ffd640 1 -- 172.21.15.40:0/1093751737 >> v1:172.21.15.189:6789/0 conn(0x7f5dc0003760 legacy=0x7f5dc0005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:36.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.020+0000 7f5dd2ffd640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-04-22T08:12:36.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.020+0000 7f5de19dc640 1 -- 172.21.15.40:0/1093751737 reap_dead start 2024-04-22T08:12:36.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.020+0000 7f5dd2ffd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5ddc0a3720 addr v1:172.21.15.40:6789/0 2024-04-22T08:12:36.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.020+0000 7f5dd2ffd640 1 --2- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5dc000a890 0x7f5dc0009950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:36.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.020+0000 7f5dd2ffd640 10 monclient(hunting): picked mon.noname-b con 0x7f5dc000a890 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:36.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.020+0000 7f5dd2ffd640 1 --2- 172.21.15.40:0/1093751737 >> v2:172.21.15.189:3300/0 conn(0x7f5dc0006390 0x7f5dc0005be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:36.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.020+0000 7f5dd2ffd640 10 monclient(hunting): picked mon.noname-c con 0x7f5dc0006390 addr v2:172.21.15.189:3300/0 2024-04-22T08:12:36.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.020+0000 7f5dd2ffd640 1 -- 172.21.15.40:0/1093751737 --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dc000b0c0 con 0x7f5ddc0a3720 2024-04-22T08:12:36.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.020+0000 7f5dd2ffd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:36.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.020+0000 7f5dd2ffd640 1 -- 172.21.15.40:0/1093751737 --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ddc09fc90 con 0x7f5dc0006390 2024-04-22T08:12:36.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.020+0000 7f5dd2ffd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:36.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.020+0000 7f5dd2ffd640 1 -- 172.21.15.40:0/1093751737 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ddc0a3b90 con 0x7f5dc000a890 2024-04-22T08:12:36.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.020+0000 7f5dd2ffd640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:36.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.020+0000 7f5de09da640 1 --2- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5dc000a890 0x7f5dc0009950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:36.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.020+0000 7f5de09da640 10 monclient(hunting): get_auth_request con 0x7f5dc000a890 auth_method 0 2024-04-22T08:12:36.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.021+0000 7f5de09da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:36.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.021+0000 7f5de09da640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:36.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.021+0000 7f5de09da640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:36.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.021+0000 7f5de19dc640 1 --2- 172.21.15.40:0/1093751737 >> v2:172.21.15.189:3300/0 conn(0x7f5dc0006390 0x7f5dc0005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:36.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.021+0000 7f5de09da640 1 -- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5dc000a890 msgr2=0x7f5dc0009950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:12:36.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.021+0000 7f5de09da640 1 -- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5dc000a890 msgr2=0x7f5dc0009950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:12:36.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.021+0000 7f5de09da640 1 --2- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5dc000a890 0x7f5dc0009950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:12:36.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.021+0000 7f5de09da640 1 --2- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5dc000a890 0x7f5dc0009950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:12:36.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.021+0000 7f5de19dc640 10 monclient(hunting): get_auth_request con 0x7f5dc0006390 auth_method 0 2024-04-22T08:12:36.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.021+0000 7f5de19dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:36.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.021+0000 7f5de19dc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:36.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.021+0000 7f5de19dc640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:36.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.021+0000 7f5de19dc640 1 -- 172.21.15.40:0/1093751737 >> v2:172.21.15.189:3300/0 conn(0x7f5dc0006390 msgr2=0x7f5dc0005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-22T08:12:36.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.021+0000 7f5de19dc640 1 -- 172.21.15.40:0/1093751737 >> v2:172.21.15.189:3300/0 conn(0x7f5dc0006390 msgr2=0x7f5dc0005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:12:36.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.021+0000 7f5de19dc640 1 --2- 172.21.15.40:0/1093751737 >> v2:172.21.15.189:3300/0 conn(0x7f5dc0006390 0x7f5dc0005be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:12:36.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.021+0000 7f5de19dc640 1 --2- 172.21.15.40:0/1093751737 >> v2:172.21.15.189:3300/0 conn(0x7f5dc0006390 0x7f5dc0005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:12:36.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.023+0000 7f5de11db640 1 -- 172.21.15.40:0/1093751737 >> v1:172.21.15.40:6789/0 conn(0x7f5ddc0a3720 legacy=0x7f5dc000a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:36.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.031+0000 7f5dd3fff640 1 -- 172.21.15.40:0/1093751737 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 255852622 0 0) 0x7f5dc000b0c0 con 0x7f5ddc0a3720 2024-04-22T08:12:36.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.031+0000 7f5dd3fff640 10 monclient(hunting): _negotiate 2024-04-22T08:12:36.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.031+0000 7f5dd3fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:36.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.031+0000 7f5dd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:36.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.031+0000 7f5dd3fff640 10 monclient(hunting): my global_id is 4098 2024-04-22T08:12:36.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.031+0000 7f5dd3fff640 1 -- 172.21.15.40:0/1093751737 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dc8003520 con 0x7f5ddc0a3720 2024-04-22T08:12:36.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.031+0000 7f5dd3fff640 1 -- 172.21.15.40:0/1093751737 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3332673293 0 0) 0x7f5dc8003520 con 0x7f5ddc0a3720 2024-04-22T08:12:36.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.031+0000 7f5dd3fff640 1 -- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5dc000a890 msgr2=0x7f5dc0009950 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:12:36.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.031+0000 7f5dd3fff640 1 --2- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5dc000a890 0x7f5dc0009950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:36.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.031+0000 7f5dd3fff640 1 -- 172.21.15.40:0/1093751737 >> v2:172.21.15.189:3300/0 conn(0x7f5dc0006390 msgr2=0x7f5dc0005be0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:12:36.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.031+0000 7f5dd3fff640 1 --2- 172.21.15.40:0/1093751737 >> v2:172.21.15.189:3300/0 conn(0x7f5dc0006390 0x7f5dc0005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:36.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.031+0000 7f5dd3fff640 10 monclient: _finish_hunting 0 2024-04-22T08:12:36.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.031+0000 7f5dd3fff640 1 monclient: found mon.noname-a-legacy 2024-04-22T08:12:36.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.031+0000 7f5dd3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:36.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.031+0000 7f5dd3fff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.40:6789/0 2024-04-22T08:12:36.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.031+0000 7f5dd3fff640 1 -- 172.21.15.40:0/1093751737 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc000ade0 con 0x7f5ddc0a3720 2024-04-22T08:12:36.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.031+0000 7f5dd3fff640 10 monclient: _finish_auth 0 2024-04-22T08:12:36.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.031+0000 7f5dd3fff640 10 monclient: _check_auth_tickets 2024-04-22T08:12:36.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.031+0000 7f5dd3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:36.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.031+0000 7f5dd3fff640 1 -- 172.21.15.40:0/1093751737 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1087006497 0 0) 0x7f5dd800d680 con 0x7f5ddc0a3720 2024-04-22T08:12:36.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.031+0000 7f5de343f640 5 monclient: authenticate success, global_id 4098 2024-04-22T08:12:36.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.031+0000 7f5de343f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:12:36.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.031+0000 7f5dd3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:36.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5dd3fff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-22T08:12:36.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5dd3fff640 10 monclient: dump: 2024-04-22T08:12:36.041 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:36.041 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:36.041 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:36.041 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:36.041 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:36.041 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:36.041 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:36.042 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:36.042 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:36.042 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:36.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5dd3fff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] but i'm connected to v1:172.21.15.40:6789/0, reconnecting 2024-04-22T08:12:36.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5dd3fff640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:36.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5dd3fff640 1 -- 172.21.15.40:0/1093751737 >> v1:172.21.15.40:6789/0 conn(0x7f5ddc0a3720 legacy=0x7f5dc000a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:36.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5dd3fff640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-22T08:12:36.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5dd3fff640 1 --2- 172.21.15.40:0/1093751737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dc8003970 0x7f5dc8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:36.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5dd3fff640 10 monclient(hunting): picked mon.b con 0x7f5dc8003970 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:36.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5dd3fff640 1 --2- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dc80047c0 0x7f5dc8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:36.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5dd3fff640 10 monclient(hunting): picked mon.a con 0x7f5dc80047c0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:36.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5dd3fff640 1 --2- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5dc000a890 0x7f5dc8005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:36.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5dd3fff640 10 monclient(hunting): picked mon.c con 0x7f5dc000a890 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:36.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5dd3fff640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:36.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5dd3fff640 1 -- 172.21.15.40:0/1093751737 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ddc09fc90 con 0x7f5dc80047c0 2024-04-22T08:12:36.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5dd3fff640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:36.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5dd3fff640 1 -- 172.21.15.40:0/1093751737 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ddc0a3b90 con 0x7f5dc8003970 2024-04-22T08:12:36.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5dd3fff640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:36.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5dd3fff640 1 -- 172.21.15.40:0/1093751737 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc800bb60 con 0x7f5dc000a890 2024-04-22T08:12:36.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5dd3fff640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:36.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5dd3fff640 10 monclient(hunting): _finish_auth 0 2024-04-22T08:12:36.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5de343f640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-04-22T08:12:36.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5de19dc640 1 --2- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5dc000a890 0x7f5dc8005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:36.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5de09da640 1 --2- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dc80047c0 0x7f5dc8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:36.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5de19dc640 10 monclient(hunting): get_auth_request con 0x7f5dc000a890 auth_method 0 2024-04-22T08:12:36.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5de19dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:36.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5de19dc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:36.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5de19dc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:12:36.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5de09da640 10 monclient(hunting): get_auth_request con 0x7f5dc80047c0 auth_method 0 2024-04-22T08:12:36.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5de09da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:36.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5de09da640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:36.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5de09da640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:12:36.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5de19dc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:36.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5de19dc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:36.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5de11db640 1 --2- 172.21.15.40:0/1093751737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dc8003970 0x7f5dc8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:36.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5de19dc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-22T08:12:36.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5de09da640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:36.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5de09da640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:36.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5de09da640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-22T08:12:36.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5de11db640 10 monclient(hunting): get_auth_request con 0x7f5dc8003970 auth_method 0 2024-04-22T08:12:36.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5de11db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:36.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5de11db640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:36.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5de11db640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:12:36.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5de09da640 10 monclient(hunting): handle_auth_done global_id 4098 payload 293 2024-04-22T08:12:36.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5de09da640 1 -- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5dc000a890 msgr2=0x7f5dc8005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:36.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5de09da640 1 --2- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5dc000a890 0x7f5dc8005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:36.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5de09da640 1 -- 172.21.15.40:0/1093751737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dc8003970 msgr2=0x7f5dc8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:36.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5de09da640 1 --2- 172.21.15.40:0/1093751737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dc8003970 0x7f5dc8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:36.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5de09da640 10 monclient: _finish_hunting 0 2024-04-22T08:12:36.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5de09da640 1 monclient: found mon.a 2024-04-22T08:12:36.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5de09da640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:36.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5de09da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:36.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5de09da640 1 -- 172.21.15.40:0/1093751737 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc8003520 con 0x7f5dc80047c0 2024-04-22T08:12:36.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5de09da640 10 monclient: _finish_auth 0 2024-04-22T08:12:36.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5de09da640 10 monclient: _check_auth_tickets 2024-04-22T08:12:36.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5de09da640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:36.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5de19dc640 1 --2- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5dc000a890 0x7f5dc8005230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:12:36.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.032+0000 7f5de11db640 1 --2- 172.21.15.40:0/1093751737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dc8003970 0x7f5dc8003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T08:12:36.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.033+0000 7f5de09da640 1 -- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dc80047c0 msgr2=0x7f5dc8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-22T08:12:36.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.033+0000 7f5de09da640 1 -- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dc80047c0 msgr2=0x7f5dc8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:12:36.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.033+0000 7f5de09da640 1 --2- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dc80047c0 0x7f5dc8004c40 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0x7f5dcc017cc0 tx=0x7f5dc80057f0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:12:36.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.033+0000 7f5de09da640 1 --2- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dc80047c0 0x7f5dc8004c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:12:36.162 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-04-22T08:12:36.162+0000 7f0813e38040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T08:12:36.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5de09da640 1 --2- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dc80047c0 0x7f5dc8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:36.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5de09da640 10 monclient: get_auth_request con 0x7f5dc80047c0 auth_method 0 2024-04-22T08:12:36.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5de09da640 0 --2- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dc80047c0 0x7f5dc8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-04-22T08:12:36.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5de09da640 1 --2- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dc80047c0 0x7f5dc8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:36.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5de19dc640 1 -- 172.21.15.40:0/1093751737 reap_dead start 2024-04-22T08:12:36.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5dd3fff640 10 monclient: ms_handle_reset current mon [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:36.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5dd3fff640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:36.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5dd3fff640 1 -- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dc80047c0 msgr2=0x7f5dc8004c40 unknown :-1 s=STATE_CLOSED l=0).mark_down 2024-04-22T08:12:36.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5dd3fff640 1 --2- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dc80047c0 0x7f5dc8004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:36.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5dd3fff640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-22T08:12:36.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5dd3fff640 1 --2- 172.21.15.40:0/1093751737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dc8005230 0x7f5dc8003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:36.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5dd3fff640 10 monclient(hunting): picked mon.b con 0x7f5dc8005230 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:36.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5dd3fff640 1 --2- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dc8005ae0 0x7f5dc8005ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:36.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5dd3fff640 10 monclient(hunting): picked mon.a con 0x7f5dc8005ae0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:36.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5dd3fff640 1 --2- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5dc8006630 0x7f5dc8006ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:36.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5dd3fff640 10 monclient(hunting): picked mon.c con 0x7f5dc8006630 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:36.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5dd3fff640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:36.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5dd3fff640 1 -- 172.21.15.40:0/1093751737 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc8007420 con 0x7f5dc8005ae0 2024-04-22T08:12:36.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5dd3fff640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:36.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5dd3fff640 1 -- 172.21.15.40:0/1093751737 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc8007590 con 0x7f5dc8005230 2024-04-22T08:12:36.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5dd3fff640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:36.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5dd3fff640 1 -- 172.21.15.40:0/1093751737 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc8007750 con 0x7f5dc8006630 2024-04-22T08:12:36.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5dd3fff640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:36.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5dd3fff640 10 monclient(hunting): ms_handle_reset hunted mon [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:36.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5de09da640 1 --2- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dc8005ae0 0x7f5dc8005ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:36.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5de09da640 10 monclient(hunting): get_auth_request con 0x7f5dc8005ae0 auth_method 0 2024-04-22T08:12:36.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5de09da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:36.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5de09da640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:36.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5de09da640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:12:36.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5de19dc640 1 --2- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5dc8006630 0x7f5dc8006ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:36.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5de19dc640 10 monclient(hunting): get_auth_request con 0x7f5dc8006630 auth_method 0 2024-04-22T08:12:36.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5de19dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:36.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5de19dc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:36.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5de19dc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:12:36.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5de11db640 1 --2- 172.21.15.40:0/1093751737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dc8005230 0x7f5dc8003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:36.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5de09da640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:36.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5de09da640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:36.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5de09da640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-22T08:12:36.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5de19dc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:36.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5de19dc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:36.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5de19dc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-22T08:12:36.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5de11db640 10 monclient(hunting): get_auth_request con 0x7f5dc8005230 auth_method 0 2024-04-22T08:12:36.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5de11db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:36.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5de11db640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:36.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5de11db640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:12:36.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5de09da640 10 monclient(hunting): handle_auth_done global_id 4098 payload 293 2024-04-22T08:12:36.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5de09da640 1 -- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5dc8006630 msgr2=0x7f5dc8006ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:36.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5de09da640 1 --2- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5dc8006630 0x7f5dc8006ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:36.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.234+0000 7f5de09da640 1 -- 172.21.15.40:0/1093751737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dc8005230 msgr2=0x7f5dc8003d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:36.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.235+0000 7f5de09da640 1 --2- 172.21.15.40:0/1093751737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dc8005230 0x7f5dc8003d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:36.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.235+0000 7f5de09da640 10 monclient: _finish_hunting 0 2024-04-22T08:12:36.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.235+0000 7f5de09da640 1 monclient: found mon.a 2024-04-22T08:12:36.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.235+0000 7f5de09da640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:36.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.235+0000 7f5de09da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:36.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.235+0000 7f5de09da640 1 -- 172.21.15.40:0/1093751737 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=2+}) v3 -- 0x7f5dc000b0c0 con 0x7f5dc8005ae0 2024-04-22T08:12:36.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.235+0000 7f5de09da640 10 monclient: _finish_auth 0 2024-04-22T08:12:36.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.235+0000 7f5de09da640 10 monclient: _check_auth_tickets 2024-04-22T08:12:36.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.235+0000 7f5de09da640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:36.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.235+0000 7f5de11db640 1 --2- 172.21.15.40:0/1093751737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dc8005230 0x7f5dc8003d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T08:12:36.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.237+0000 7f5de09da640 1 -- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dc8005ae0 msgr2=0x7f5dc8005ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:36.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.237+0000 7f5de09da640 1 --2- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dc8005ae0 0x7f5dc8005ec0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc017cc0 tx=0x7f5dcc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0d1c016b1f070b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:36.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.237+0000 7f5dd3fff640 1 -- 172.21.15.40:0/1093751737 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dcc004030 con 0x7f5dc8005ae0 2024-04-22T08:12:36.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.237+0000 7f5dd3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:36.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.237+0000 7f5dd3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:12:36.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.237+0000 7f5dd3fff640 10 monclient: dump: 2024-04-22T08:12:36.248 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:36.248 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:36.248 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:36.248 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:36.248 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:36.248 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:36.248 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:36.248 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:36.248 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:36.248 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:36.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.237+0000 7f5dd3fff640 1 -- 172.21.15.40:0/1093751737 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dcc004190 con 0x7f5dc8005ae0 2024-04-22T08:12:36.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.237+0000 7f5de343f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:12:36.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.237+0000 7f5dd3fff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:36.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.237+0000 7f5de343f640 10 monclient: get_monmap_and_config success 2024-04-22T08:12:36.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.237+0000 7f5de343f640 4 set_mon_vals no callback set 2024-04-22T08:12:36.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.238+0000 7f5de343f640 10 monclient: shutdown 2024-04-22T08:12:36.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.238+0000 7f5de343f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:36.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.238+0000 7f5de343f640 1 -- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dc8005ae0 msgr2=0x7f5dc8005ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:36.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.238+0000 7f5de343f640 1 --2- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dc8005ae0 0x7f5dc8005ec0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc017cc0 tx=0x7f5dcc008040 comp rx=0 tx=0).stop 2024-04-22T08:12:36.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.238+0000 7f5de343f640 1 -- 172.21.15.40:0/1093751737 shutdown_connections 2024-04-22T08:12:36.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.238+0000 7f5de343f640 1 --2- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5dc8006630 0x7f5dc8006ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:36.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.238+0000 7f5de343f640 1 --2- 172.21.15.40:0/1093751737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dc8005ae0 0x7f5dc8005ec0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:36.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.238+0000 7f5de343f640 1 --2- 172.21.15.40:0/1093751737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dc8005230 0x7f5dc8003d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:36.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.238+0000 7f5de343f640 1 -- 172.21.15.40:0/1093751737 >> 172.21.15.40:0/1093751737 conn(0x7f5ddc0a5060 msgr2=0x7f5ddc0a5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:36.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.238+0000 7f5de343f640 1 -- 172.21.15.40:0/1093751737 shutdown_connections 2024-04-22T08:12:36.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.238+0000 7f5de343f640 1 -- 172.21.15.40:0/1093751737 wait complete. 2024-04-22T08:12:36.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.239+0000 7f5de343f640 10 monclient: build_initial_monmap 2024-04-22T08:12:36.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.239+0000 7f5de343f640 10 monclient: monmap: 2024-04-22T08:12:36.250 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:36.250 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:12:36.250 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:12:36.250 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:12:36.250 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:36.250 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:36.250 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:12:36.250 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:12:36.250 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:12:36.250 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:36.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.240+0000 7f5de343f640 1 Processor -- start 2024-04-22T08:12:36.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.240+0000 7f5de343f640 1 -- start start 2024-04-22T08:12:36.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.240+0000 7f5de343f640 10 monclient: init 2024-04-22T08:12:36.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.240+0000 7f5de343f640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:36.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.240+0000 7f5de343f640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-22T08:12:36.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.240+0000 7f5de343f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dc8005230 0x7f5ddc1ca760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:36.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.240+0000 7f5de343f640 10 monclient(hunting): picked mon.noname-a con 0x7f5dc8005230 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:36.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.240+0000 7f5de343f640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5dc8005ae0 0x7f5ddc134d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:36.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.240+0000 7f5de343f640 10 monclient(hunting): picked mon.noname-c con 0x7f5dc8005ae0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:36.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.240+0000 7f5de343f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dc8006630 0x7f5ddc1352c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:36.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.240+0000 7f5de343f640 10 monclient(hunting): picked mon.noname-b con 0x7f5dc8006630 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:36.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.240+0000 7f5de343f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:36.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.240+0000 7f5de343f640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ddc120d90 con 0x7f5dc8005230 2024-04-22T08:12:36.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.240+0000 7f5de343f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:36.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.240+0000 7f5de343f640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ddc1caca0 con 0x7f5dc8006630 2024-04-22T08:12:36.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.240+0000 7f5de343f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:36.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.240+0000 7f5de343f640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ddc122180 con 0x7f5dc8005ae0 2024-04-22T08:12:36.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.240+0000 7f5de343f640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:36.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.240+0000 7f5de343f640 10 monclient(hunting): authenticate will time out at 830.234453s 2024-04-22T08:12:36.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.240+0000 7f5de11db640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dc8005230 0x7f5ddc1ca760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:36.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.240+0000 7f5de09da640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5dc8005ae0 0x7f5ddc134d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:36.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.240+0000 7f5de11db640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dc8005230 0x7f5ddc1ca760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57738/0 (socket says 172.21.15.40:57738) 2024-04-22T08:12:36.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.240+0000 7f5de09da640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5dc8005ae0 0x7f5ddc134d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:34782/0 (socket says 172.21.15.40:34782) 2024-04-22T08:12:36.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.240+0000 7f5de11db640 1 -- 172.21.15.40:0/1021798192 learned_addr learned my addr 172.21.15.40:0/1021798192 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:36.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.240+0000 7f5de11db640 10 monclient(hunting): get_auth_request con 0x7f5dc8005230 auth_method 0 2024-04-22T08:12:36.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.240+0000 7f5de11db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:36.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.240+0000 7f5de11db640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:36.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.240+0000 7f5de11db640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:36.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.240+0000 7f5de09da640 10 monclient(hunting): get_auth_request con 0x7f5dc8005ae0 auth_method 0 2024-04-22T08:12:36.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.240+0000 7f5de09da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:36.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.240+0000 7f5de09da640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:36.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.240+0000 7f5de09da640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:36.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.240+0000 7f5de11db640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:36.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.240+0000 7f5de11db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:36.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.240+0000 7f5de11db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:36.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.240+0000 7f5de09da640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:36.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.240+0000 7f5de09da640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:36.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.241+0000 7f5de09da640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:36.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.241+0000 7f5de19dc640 1 --2- 172.21.15.40:0/1021798192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dc8006630 0x7f5ddc1352c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:36.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.241+0000 7f5de19dc640 10 monclient(hunting): get_auth_request con 0x7f5dc8006630 auth_method 0 2024-04-22T08:12:36.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.241+0000 7f5de19dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:36.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.241+0000 7f5de19dc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:36.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.241+0000 7f5de19dc640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:36.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.241+0000 7f5de19dc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:36.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.241+0000 7f5de19dc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:36.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.241+0000 7f5de19dc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:36.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.241+0000 7f5de11db640 10 monclient(hunting): handle_auth_done global_id 4104 payload 824 2024-04-22T08:12:36.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.241+0000 7f5de11db640 1 -- 172.21.15.40:0/1021798192 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5dc8005ae0 msgr2=0x7f5ddc134d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:36.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.241+0000 7f5de11db640 1 --2- 172.21.15.40:0/1021798192 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5dc8005ae0 0x7f5ddc134d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:36.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.241+0000 7f5de11db640 1 -- 172.21.15.40:0/1021798192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dc8006630 msgr2=0x7f5ddc1352c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:36.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.241+0000 7f5de11db640 1 --2- 172.21.15.40:0/1021798192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dc8006630 0x7f5ddc1352c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:36.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.241+0000 7f5de11db640 10 monclient: _finish_hunting 0 2024-04-22T08:12:36.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.241+0000 7f5de11db640 1 monclient: found mon.noname-a 2024-04-22T08:12:36.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.241+0000 7f5de11db640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:36.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.241+0000 7f5de11db640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:12:36.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.241+0000 7f5de11db640 1 -- 172.21.15.40:0/1021798192 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ddc122770 con 0x7f5dc8005230 2024-04-22T08:12:36.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.243+0000 7f5de11db640 1 -- 172.21.15.40:0/1021798192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dc8005230 msgr2=0x7f5ddc1ca760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:36.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.243+0000 7f5de11db640 1 --2- 172.21.15.40:0/1021798192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dc8005230 0x7f5ddc1ca760 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5dd8017a00 tx=0x7f5dd8017e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:36.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.243+0000 7f5dd1ffb640 1 -- 172.21.15.40:0/1021798192 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd8015d00 con 0x7f5dc8005230 2024-04-22T08:12:36.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.243+0000 7f5dd1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:36.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.243+0000 7f5dd1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:12:36.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.243+0000 7f5dd1ffb640 10 monclient: dump: 2024-04-22T08:12:36.255 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:36.255 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:36.255 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:36.255 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:36.255 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:36.255 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:36.255 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:36.255 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:36.255 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:36.256 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:36.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.243+0000 7f5dd1ffb640 10 monclient: _finish_auth 0 2024-04-22T08:12:36.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.243+0000 7f5dd1ffb640 10 monclient: _check_auth_tickets 2024-04-22T08:12:36.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.243+0000 7f5dd1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:36.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.243+0000 7f5dd1ffb640 1 -- 172.21.15.40:0/1021798192 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dd801c070 con 0x7f5dc8005230 2024-04-22T08:12:36.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.243+0000 7f5de343f640 5 monclient: authenticate success, global_id 4104 2024-04-22T08:12:36.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.243+0000 7f5de343f640 10 monclient: _renew_subs 2024-04-22T08:12:36.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.243+0000 7f5de343f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:36.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.243+0000 7f5de343f640 1 -- 172.21.15.40:0/1021798192 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ddc0712e0 con 0x7f5dc8005230 2024-04-22T08:12:36.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.243+0000 7f5dd1ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:36.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.243+0000 7f5dd1ffb640 1 -- 172.21.15.40:0/1021798192 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd8023680 con 0x7f5dc8005230 2024-04-22T08:12:36.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.243+0000 7f5dd1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:36.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.243+0000 7f5dd1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:12:36.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.243+0000 7f5dd1ffb640 10 monclient: dump: 2024-04-22T08:12:36.257 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:36.257 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:36.257 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:36.257 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:36.257 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:36.257 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:36.257 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:36.257 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:36.257 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:36.257 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:36.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.243+0000 7f5dd37fe640 4 set_mon_vals no callback set 2024-04-22T08:12:36.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.244+0000 7f5de343f640 10 monclient: _renew_subs 2024-04-22T08:12:36.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.244+0000 7f5de343f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:36.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.244+0000 7f5de343f640 1 -- 172.21.15.40:0/1021798192 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ddc121f50 con 0x7f5dc8005230 2024-04-22T08:12:36.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.244+0000 7f5dd1ffb640 1 -- 172.21.15.40:0/1021798192 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f5dd8005990 con 0x7f5dc8005230 2024-04-22T08:12:36.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.244+0000 7f5dd1ffb640 1 -- 172.21.15.40:0/1021798192 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f5dd801e070 con 0x7f5dc8005230 2024-04-22T08:12:36.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.245+0000 7f5de343f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:36.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.245+0000 7f5de343f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:36.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.245+0000 7f5de343f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:36.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.245+0000 7f5de343f640 1 -- 172.21.15.40:0/1021798192 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ddc04e9b0 con 0x7f5dc8005230 2024-04-22T08:12:36.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.248+0000 7f5dd1ffb640 1 -- 172.21.15.40:0/1021798192 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (secure 0 0 0) 0x7f5dd8015e60 con 0x7f5dc8005230 2024-04-22T08:12:36.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.248+0000 7f5dd1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:36.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.248+0000 7f5dd1ffb640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:12:36.314 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-04-22T08:12:36.314+0000 7f0813e38040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T08:12:36.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.316+0000 7f5de343f640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-04-22T08:12:36.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.316+0000 7f5de343f640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-04-22T08:12:36.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.316+0000 7f5de343f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:36.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.316+0000 7f5de343f640 1 -- 172.21.15.40:0/1021798192 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f5ddc04e9b0 con 0x7f5dc8005230 2024-04-22T08:12:36.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.318+0000 7f5dd1ffb640 1 -- 172.21.15.40:0/1021798192 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f5dd8018450 con 0x7f5dc8005230 2024-04-22T08:12:36.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.319+0000 7f5dd1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-04-22T08:12:36.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.319+0000 7f5dd1ffb640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-04-22T08:12:36.318 INFO:teuthology.orchestra.run.smithi040.stderr:adjusted tunables profile to default 2024-04-22T08:12:36.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.319+0000 7f5de343f640 10 monclient: shutdown 2024-04-22T08:12:36.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.319+0000 7f5de343f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:36.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.319+0000 7f5de343f640 1 -- 172.21.15.40:0/1021798192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dc8005230 msgr2=0x7f5ddc1ca760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:36.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.319+0000 7f5de343f640 1 --2- 172.21.15.40:0/1021798192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dc8005230 0x7f5ddc1ca760 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5dd8017a00 tx=0x7f5dd8017e00 comp rx=0 tx=0).stop 2024-04-22T08:12:36.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.319+0000 7f5de343f640 1 -- 172.21.15.40:0/1021798192 shutdown_connections 2024-04-22T08:12:36.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.319+0000 7f5de343f640 1 --2- 172.21.15.40:0/1021798192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dc8006630 0x7f5ddc1352c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:36.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.320+0000 7f5de343f640 1 --2- 172.21.15.40:0/1021798192 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5dc8005ae0 0x7f5ddc134d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:36.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.320+0000 7f5de343f640 1 --2- 172.21.15.40:0/1021798192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dc8005230 0x7f5ddc1ca760 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:36.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.320+0000 7f5de343f640 1 -- 172.21.15.40:0/1021798192 >> 172.21.15.40:0/1021798192 conn(0x7f5ddc0a5060 msgr2=0x7f5ddc0a15b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:36.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.320+0000 7f5de343f640 1 -- 172.21.15.40:0/1021798192 shutdown_connections 2024-04-22T08:12:36.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:36.320+0000 7f5de343f640 1 -- 172.21.15.40:0/1021798192 wait complete. 2024-04-22T08:12:36.331 INFO:tasks.ceph:check_enable_crimson: False 2024-04-22T08:12:36.331 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-22T08:12:36.331 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-22T08:12:36.331 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-22T08:12:36.357 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-22T08:12:36.358 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-22T08:12:36.367 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-04-22T08:12:36.367+0000 7f0813e38040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T08:12:36.422 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-22T08:12:36.422 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-22T08:12:36.487 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-22T08:12:36.487 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-22T08:12:36.511 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-22T08:12:36.511 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-22T08:12:36.575 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-22T08:12:36.575 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-22T08:12:36.610 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-04-22T08:12:36.610+0000 7f0813e38040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T08:12:36.633 DEBUG:teuthology.orchestra.run.smithi189:> sudo ceph --cluster ceph osd new b9a0210c-5881-4ca0-8318-a4a987109405 0 2024-04-22T08:12:36.657 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-04-22T08:12:36.657+0000 7f0813e38040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T08:12:36.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.753+0000 7fda08329640 10 monclient: get_monmap_and_config 2024-04-22T08:12:36.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.753+0000 7fda08329640 10 monclient: build_initial_monmap 2024-04-22T08:12:36.753 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.753+0000 7fda08329640 10 monclient: monmap: 2024-04-22T08:12:36.753 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-04-22T08:12:36.753 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:36.753 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-04-22T08:12:36.754186+0000 2024-04-22T08:12:36.754 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-04-22T08:12:36.754186+0000 2024-04-22T08:12:36.754 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:36.754 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-04-22T08:12:36.754 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:12:36.754 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:12:36.754 INFO:teuthology.orchestra.run.smithi189.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:12:36.754 INFO:teuthology.orchestra.run.smithi189.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:12:36.754 INFO:teuthology.orchestra.run.smithi189.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:12:36.754 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-04-22T08:12:36.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.753+0000 7fda08329640 1 Processor -- start 2024-04-22T08:12:36.754 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.753+0000 7fda08329640 1 -- start start 2024-04-22T08:12:36.755 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.753+0000 7fda08329640 10 monclient: init 2024-04-22T08:12:36.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.754+0000 7fda08329640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:36.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.754+0000 7fda08329640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-04-22T08:12:36.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.754+0000 7fda08329640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fda0009ee10 0x7fda001362c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:36.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.754+0000 7fda08329640 10 monclient(hunting): picked mon.noname-a con 0x7fda0009ee10 addr v2:172.21.15.40:3300/0 2024-04-22T08:12:36.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.754+0000 7fda08329640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fda000a2d80 0x7fda000a3210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:36.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.754+0000 7fda08329640 10 monclient(hunting): picked mon.noname-c con 0x7fda000a2d80 addr v2:172.21.15.189:3300/0 2024-04-22T08:12:36.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.754+0000 7fda08329640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fda000a3750 addr v1:172.21.15.40:6789/0 2024-04-22T08:12:36.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.754+0000 7fda08329640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda000a4260 con 0x7fda000a3750 2024-04-22T08:12:36.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.754+0000 7fda08329640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:36.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.754+0000 7fda08329640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda0009fc90 con 0x7fda0009ee10 2024-04-22T08:12:36.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.754+0000 7fda08329640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:36.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.754+0000 7fda08329640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda000a3bc0 con 0x7fda000a2d80 2024-04-22T08:12:36.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.754+0000 7fda08329640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:36.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.754+0000 7fda08329640 10 monclient(hunting): authenticate will time out at 810.047000s 2024-04-22T08:12:36.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.754+0000 7fda058c4640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fda000a2d80 0x7fda000a3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:36.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.754+0000 7fda058c4640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fda000a2d80 0x7fda000a3210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48548/0 (socket says 172.21.15.189:48548) 2024-04-22T08:12:36.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.754+0000 7fda058c4640 1 -- 172.21.15.189:0/2281188462 learned_addr learned my addr 172.21.15.189:0/2281188462 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-22T08:12:36.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.754+0000 7fda058c4640 10 monclient(hunting): get_auth_request con 0x7fda000a2d80 auth_method 0 2024-04-22T08:12:36.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.754+0000 7fda058c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:36.758 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.754+0000 7fda058c4640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:36.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.754+0000 7fda058c4640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:36.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.754+0000 7fda058c4640 1 -- 172.21.15.189:0/2281188462 >> v2:172.21.15.189:3300/0 conn(0x7fda000a2d80 msgr2=0x7fda000a3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:12:36.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.754+0000 7fda058c4640 1 -- 172.21.15.189:0/2281188462 >> v2:172.21.15.189:3300/0 conn(0x7fda000a2d80 msgr2=0x7fda000a3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:12:36.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.754+0000 7fda058c4640 1 --2- 172.21.15.189:0/2281188462 >> v2:172.21.15.189:3300/0 conn(0x7fda000a2d80 0x7fda000a3210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:12:36.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.754+0000 7fda058c4640 1 --2- 172.21.15.189:0/2281188462 >> v2:172.21.15.189:3300/0 conn(0x7fda000a2d80 0x7fda000a3210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:12:36.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.754+0000 7fda060c5640 1 --2- 172.21.15.189:0/2281188462 >> v2:172.21.15.40:3300/0 conn(0x7fda0009ee10 0x7fda001362c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:36.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.754+0000 7fda060c5640 10 monclient(hunting): get_auth_request con 0x7fda0009ee10 auth_method 0 2024-04-22T08:12:36.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.754+0000 7fda060c5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:36.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.754+0000 7fda060c5640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:36.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.754+0000 7fda060c5640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:36.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.754+0000 7fda060c5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:36.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.754+0000 7fda060c5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:36.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.754+0000 7fda060c5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:36.759 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.756+0000 7fda068c6640 1 -- 172.21.15.189:0/2281188462 >> v1:172.21.15.40:6789/0 conn(0x7fda000a3750 legacy=0x7fda00132b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:36.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.757+0000 7fda060c5640 10 monclient(hunting): handle_auth_done global_id 4107 payload 274 2024-04-22T08:12:36.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.757+0000 7fda060c5640 1 -- 172.21.15.189:0/2281188462 >> v2:172.21.15.189:3300/0 conn(0x7fda000a2d80 msgr2=0x7fda000a3210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:12:36.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.757+0000 7fda060c5640 1 --2- 172.21.15.189:0/2281188462 >> v2:172.21.15.189:3300/0 conn(0x7fda000a2d80 0x7fda000a3210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:36.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.757+0000 7fda060c5640 1 -- 172.21.15.189:0/2281188462 >> v1:172.21.15.40:6789/0 conn(0x7fda000a3750 legacy=0x7fda00132b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:36.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.757+0000 7fda060c5640 10 monclient: _finish_hunting 0 2024-04-22T08:12:36.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.757+0000 7fda060c5640 1 monclient: found mon.noname-a 2024-04-22T08:12:36.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.757+0000 7fda060c5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:36.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.757+0000 7fda060c5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:12:36.760 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.757+0000 7fda060c5640 1 -- 172.21.15.189:0/2281188462 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda00136980 con 0x7fda0009ee10 2024-04-22T08:12:36.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.760+0000 7fda060c5640 1 -- 172.21.15.189:0/2281188462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda0009ee10 msgr2=0x7fda001362c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:36.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.760+0000 7fda060c5640 1 --2- 172.21.15.189:0/2281188462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda0009ee10 0x7fda001362c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fda000a4c70 tx=0x7fd9f802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0fd94166c1714be server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:36.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.760+0000 7fda050c3640 1 -- 172.21.15.189:0/2281188462 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f802cdd0 con 0x7fda0009ee10 2024-04-22T08:12:36.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.760+0000 7fda050c3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:36.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.760+0000 7fda050c3640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:12:36.766 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.760+0000 7fda050c3640 10 monclient: dump: 2024-04-22T08:12:36.766 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-04-22T08:12:36.766 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:36.766 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:36.767 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:36.767 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:36.767 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-04-22T08:12:36.767 INFO:teuthology.orchestra.run.smithi189.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:36.767 INFO:teuthology.orchestra.run.smithi189.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:36.767 INFO:teuthology.orchestra.run.smithi189.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:36.767 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-04-22T08:12:36.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.760+0000 7fda050c3640 10 monclient: _finish_auth 0 2024-04-22T08:12:36.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.760+0000 7fda050c3640 10 monclient: _check_auth_tickets 2024-04-22T08:12:36.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.760+0000 7fda050c3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:36.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.760+0000 7fda050c3640 1 -- 172.21.15.189:0/2281188462 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9f8034440 con 0x7fda0009ee10 2024-04-22T08:12:36.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.760+0000 7fda08329640 5 monclient: authenticate success, global_id 4107 2024-04-22T08:12:36.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.760+0000 7fda08329640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:12:36.767 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.760+0000 7fda050c3640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:36.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.760+0000 7fda08329640 10 monclient: get_monmap_and_config success 2024-04-22T08:12:36.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.760+0000 7fda050c3640 1 -- 172.21.15.189:0/2281188462 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f80347d0 con 0x7fda0009ee10 2024-04-22T08:12:36.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.760+0000 7fda050c3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:36.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.760+0000 7fda08329640 4 set_mon_vals no callback set 2024-04-22T08:12:36.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.760+0000 7fda050c3640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:12:36.768 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.760+0000 7fda050c3640 10 monclient: dump: 2024-04-22T08:12:36.768 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-04-22T08:12:36.768 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:36.768 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:36.768 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:36.768 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:36.768 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-04-22T08:12:36.768 INFO:teuthology.orchestra.run.smithi189.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:36.768 INFO:teuthology.orchestra.run.smithi189.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:36.769 INFO:teuthology.orchestra.run.smithi189.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:36.769 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-04-22T08:12:36.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.761+0000 7fda08329640 10 monclient: shutdown 2024-04-22T08:12:36.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.761+0000 7fda08329640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:36.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.761+0000 7fda08329640 1 -- 172.21.15.189:0/2281188462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda0009ee10 msgr2=0x7fda001362c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:36.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.761+0000 7fda08329640 1 --2- 172.21.15.189:0/2281188462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda0009ee10 0x7fda001362c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fda000a4c70 tx=0x7fd9f802be10 comp rx=0 tx=0).stop 2024-04-22T08:12:36.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.761+0000 7fda08329640 1 -- 172.21.15.189:0/2281188462 shutdown_connections 2024-04-22T08:12:36.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.761+0000 7fda08329640 1 --2- 172.21.15.189:0/2281188462 >> v2:172.21.15.189:3300/0 conn(0x7fda000a2d80 0x7fda000a3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:36.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.761+0000 7fda08329640 1 --2- 172.21.15.189:0/2281188462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda0009ee10 0x7fda001362c0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:36.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.761+0000 7fda08329640 1 -- 172.21.15.189:0/2281188462 >> 172.21.15.189:0/2281188462 conn(0x7fda000a5060 msgr2=0x7fda000a5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:36.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.761+0000 7fda08329640 1 -- 172.21.15.189:0/2281188462 shutdown_connections 2024-04-22T08:12:36.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.761+0000 7fda08329640 1 -- 172.21.15.189:0/2281188462 wait complete. 2024-04-22T08:12:36.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.761+0000 7fda08329640 10 monclient: build_initial_monmap 2024-04-22T08:12:36.769 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.761+0000 7fda08329640 10 monclient: monmap: 2024-04-22T08:12:36.770 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-04-22T08:12:36.770 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:12:36.770 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 0.000000 2024-04-22T08:12:36.770 INFO:teuthology.orchestra.run.smithi189.stderr:created 0.000000 2024-04-22T08:12:36.770 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:36.770 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-04-22T08:12:36.770 INFO:teuthology.orchestra.run.smithi189.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:12:36.770 INFO:teuthology.orchestra.run.smithi189.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:12:36.770 INFO:teuthology.orchestra.run.smithi189.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:12:36.770 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-04-22T08:12:36.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda08329640 1 Processor -- start 2024-04-22T08:12:36.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda08329640 1 -- start start 2024-04-22T08:12:36.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda08329640 10 monclient: init 2024-04-22T08:12:36.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda08329640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:36.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda08329640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-22T08:12:36.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda08329640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fda0009ee10 0x7fda0011e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:36.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda08329640 10 monclient(hunting): picked mon.noname-c con 0x7fda0009ee10 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:36.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda08329640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda000a2d80 0x7fda0011ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:36.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda08329640 10 monclient(hunting): picked mon.noname-b con 0x7fda000a2d80 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:36.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda08329640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda000a3750 0x7fda0011f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:36.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda08329640 10 monclient(hunting): picked mon.noname-a con 0x7fda000a3750 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:36.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda08329640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:36.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda08329640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda00120020 con 0x7fda000a3750 2024-04-22T08:12:36.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda08329640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:36.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda08329640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda000a2830 con 0x7fda000a2d80 2024-04-22T08:12:36.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda08329640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:36.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda08329640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda000a6580 con 0x7fda0009ee10 2024-04-22T08:12:36.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda08329640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:36.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda08329640 10 monclient(hunting): authenticate will time out at 810.055078s 2024-04-22T08:12:36.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda058c4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda000a2d80 0x7fda0011ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:36.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda058c4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda000a2d80 0x7fda0011ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48554/0 (socket says 172.21.15.189:48554) 2024-04-22T08:12:36.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda058c4640 1 -- 172.21.15.189:0/3232915883 learned_addr learned my addr 172.21.15.189:0/3232915883 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-22T08:12:36.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda058c4640 10 monclient(hunting): get_auth_request con 0x7fda000a2d80 auth_method 0 2024-04-22T08:12:36.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda058c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:36.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda058c4640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:36.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda058c4640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:36.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda058c4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:36.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda058c4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:36.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda068c6640 1 --2- 172.21.15.189:0/3232915883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda000a3750 0x7fda0011f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:36.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda058c4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:36.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda060c5640 1 --2- 172.21.15.189:0/3232915883 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fda0009ee10 0x7fda0011e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:36.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda068c6640 10 monclient(hunting): get_auth_request con 0x7fda000a3750 auth_method 0 2024-04-22T08:12:36.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda068c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:36.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda068c6640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:36.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda068c6640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:36.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda060c5640 10 monclient(hunting): get_auth_request con 0x7fda0009ee10 auth_method 0 2024-04-22T08:12:36.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda060c5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:36.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda060c5640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:36.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda060c5640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:36.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda068c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:36.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda068c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:36.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda068c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:36.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda060c5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:36.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda060c5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:36.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda060c5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:36.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda058c4640 10 monclient(hunting): handle_auth_done global_id 4108 payload 824 2024-04-22T08:12:36.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda058c4640 1 -- 172.21.15.189:0/3232915883 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fda0009ee10 msgr2=0x7fda0011e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:36.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda058c4640 1 --2- 172.21.15.189:0/3232915883 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fda0009ee10 0x7fda0011e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:36.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda058c4640 1 -- 172.21.15.189:0/3232915883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda000a3750 msgr2=0x7fda0011f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:36.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda058c4640 1 --2- 172.21.15.189:0/3232915883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda000a3750 0x7fda0011f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:36.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda058c4640 10 monclient: _finish_hunting 0 2024-04-22T08:12:36.777 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda058c4640 1 monclient: found mon.noname-b 2024-04-22T08:12:36.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda058c4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:36.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda058c4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.189:3300/0 2024-04-22T08:12:36.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.762+0000 7fda058c4640 1 -- 172.21.15.189:0/3232915883 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9f8009a70 con 0x7fda000a2d80 2024-04-22T08:12:36.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.765+0000 7fda058c4640 1 -- 172.21.15.189:0/3232915883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda000a2d80 msgr2=0x7fda0011ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:36.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.765+0000 7fda058c4640 1 --2- 172.21.15.189:0/3232915883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda000a2d80 0x7fda0011ef10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd9f000e810 tx=0x7fd9f000ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:36.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.765+0000 7fd9f6ffd640 1 -- 172.21.15.189:0/3232915883 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f0004020 con 0x7fda000a2d80 2024-04-22T08:12:36.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.765+0000 7fd9f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:36.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.765+0000 7fd9f6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:12:36.778 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.765+0000 7fd9f6ffd640 10 monclient: dump: 2024-04-22T08:12:36.778 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-04-22T08:12:36.778 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:36.778 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:36.778 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:36.778 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:36.779 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-04-22T08:12:36.779 INFO:teuthology.orchestra.run.smithi189.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:36.779 INFO:teuthology.orchestra.run.smithi189.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:36.779 INFO:teuthology.orchestra.run.smithi189.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:36.779 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-04-22T08:12:36.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.765+0000 7fd9f6ffd640 10 monclient: _finish_auth 0 2024-04-22T08:12:36.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.765+0000 7fd9f6ffd640 10 monclient: _check_auth_tickets 2024-04-22T08:12:36.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.765+0000 7fd9f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:36.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.765+0000 7fd9f6ffd640 1 -- 172.21.15.189:0/3232915883 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9f0004180 con 0x7fda000a2d80 2024-04-22T08:12:36.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.765+0000 7fda08329640 5 monclient: authenticate success, global_id 4108 2024-04-22T08:12:36.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.765+0000 7fda08329640 10 monclient: _renew_subs 2024-04-22T08:12:36.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.765+0000 7fda08329640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.189:3300/0 2024-04-22T08:12:36.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.765+0000 7fda08329640 1 -- 172.21.15.189:0/3232915883 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda00121680 con 0x7fda000a2d80 2024-04-22T08:12:36.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.765+0000 7fd9f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:36.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.765+0000 7fd9f6ffd640 1 -- 172.21.15.189:0/3232915883 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f000c8e0 con 0x7fda000a2d80 2024-04-22T08:12:36.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.765+0000 7fd9f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:36.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.765+0000 7fd9f6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:12:36.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.765+0000 7fd9f6ffd640 10 monclient: dump: 2024-04-22T08:12:36.780 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-04-22T08:12:36.780 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:36.780 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:36.780 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:36.780 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:36.780 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-04-22T08:12:36.780 INFO:teuthology.orchestra.run.smithi189.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:36.780 INFO:teuthology.orchestra.run.smithi189.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:36.780 INFO:teuthology.orchestra.run.smithi189.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:36.780 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-04-22T08:12:36.780 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.765+0000 7fda048c2640 4 set_mon_vals no callback set 2024-04-22T08:12:36.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.765+0000 7fda08329640 10 monclient: _renew_subs 2024-04-22T08:12:36.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.765+0000 7fda08329640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.189:3300/0 2024-04-22T08:12:36.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.765+0000 7fda08329640 1 -- 172.21.15.189:0/3232915883 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda001208b0 con 0x7fda000a2d80 2024-04-22T08:12:36.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.765+0000 7fda058c4640 1 --2- 172.21.15.189:0/3232915883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda000a2d80 0x7fda0011ef10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd9f000e810 tx=0x7fd9f000ecd0 comp rx=0 tx=0).queue_received will delay after 1.71377e+09 on 0x7fd9f0013410 osd_map(2..2 src has 1..2) v4 2024-04-22T08:12:36.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.765+0000 7fd9f6ffd640 1 -- 172.21.15.189:0/3232915883 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7fd9f0003610 con 0x7fda000a2d80 2024-04-22T08:12:36.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.766+0000 7fda08329640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:36.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.766+0000 7fda08329640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:36.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.766+0000 7fda08329640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.189:3300/0 2024-04-22T08:12:36.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.766+0000 7fda08329640 1 -- 172.21.15.189:0/3232915883 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda001219b0 con 0x7fda000a2d80 2024-04-22T08:12:36.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:36.768+0000 7fd9f6ffd640 1 -- 172.21.15.189:0/3232915883 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd9f0013410 con 0x7fda000a2d80 2024-04-22T08:12:37.354 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-04-22T08:12:37.354+0000 7f0813e38040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T08:12:37.440 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-04-22T08:12:37.440+0000 7f0813e38040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T08:12:37.482 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-04-22T08:12:37.482+0000 7f0813e38040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T08:12:37.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.593+0000 7fd9f6ffd640 1 -- 172.21.15.189:0/3232915883 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (secure 0 0 0) 0x7fd9f0020060 con 0x7fda000a2d80 2024-04-22T08:12:37.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.593+0000 7fd9f6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:37.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.593+0000 7fd9f6ffd640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:12:37.645 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-04-22T08:12:37.645+0000 7f0813e38040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T08:12:37.664 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.665+0000 7fda08329640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b9a0210c-5881-4ca0-8318-a4a987109405", "id": 0}] 2024-04-22T08:12:37.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.665+0000 7fda08329640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b9a0210c-5881-4ca0-8318-a4a987109405", "id": 0}] 2024-04-22T08:12:37.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.665+0000 7fda08329640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.189:3300/0 2024-04-22T08:12:37.665 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.665+0000 7fda08329640 1 -- 172.21.15.189:0/3232915883 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b9a0210c-5881-4ca0-8318-a4a987109405", "id": 0} v 0) v1 -- 0x7fda0004e9b0 con 0x7fda000a2d80 2024-04-22T08:12:37.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.678+0000 7fd9f6ffd640 1 -- 172.21.15.189:0/3232915883 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b9a0210c-5881-4ca0-8318-a4a987109405", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fd9f002a0b0 con 0x7fda000a2d80 2024-04-22T08:12:37.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.678+0000 7fd9f6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b9a0210c-5881-4ca0-8318-a4a987109405", "id": 0}] 2024-04-22T08:12:37.678 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.678+0000 7fd9f6ffd640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:12:37.678 INFO:teuthology.orchestra.run.smithi189.stdout:0 2024-04-22T08:12:37.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.679+0000 7fda08329640 10 monclient: shutdown 2024-04-22T08:12:37.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.679+0000 7fda08329640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:37.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.679+0000 7fda08329640 1 -- 172.21.15.189:0/3232915883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda000a2d80 msgr2=0x7fda0011ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:37.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.679+0000 7fda08329640 1 --2- 172.21.15.189:0/3232915883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda000a2d80 0x7fda0011ef10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd9f000e810 tx=0x7fd9f000ecd0 comp rx=0 tx=0).stop 2024-04-22T08:12:37.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.679+0000 7fda08329640 1 -- 172.21.15.189:0/3232915883 shutdown_connections 2024-04-22T08:12:37.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.679+0000 7fda08329640 1 --2- 172.21.15.189:0/3232915883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda000a2d80 0x7fda0011ef10 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:37.679 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.679+0000 7fda08329640 1 --2- 172.21.15.189:0/3232915883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda000a3750 0x7fda0011f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:37.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.679+0000 7fda08329640 1 --2- 172.21.15.189:0/3232915883 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fda0009ee10 0x7fda0011e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:37.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.679+0000 7fda08329640 1 -- 172.21.15.189:0/3232915883 >> 172.21.15.189:0/3232915883 conn(0x7fda000a5060 msgr2=0x7fda00132480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:37.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.679+0000 7fda08329640 1 -- 172.21.15.189:0/3232915883 shutdown_connections 2024-04-22T08:12:37.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.679+0000 7fda08329640 1 -- 172.21.15.189:0/3232915883 wait complete. 2024-04-22T08:12:37.690 DEBUG:teuthology.orchestra.run.smithi189:> sudo ceph --cluster ceph osd new 11370cc8-09a6-4c35-b92c-4d926c6c18d9 1 2024-04-22T08:12:37.729 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-04-22T08:12:37.729+0000 7f0813e38040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T08:12:37.784 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-04-22T08:12:37.785+0000 7f0813e38040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T08:12:37.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.785+0000 7f1c10dca640 10 monclient: get_monmap_and_config 2024-04-22T08:12:37.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.785+0000 7f1c10dca640 10 monclient: build_initial_monmap 2024-04-22T08:12:37.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.785+0000 7f1c10dca640 10 monclient: monmap: 2024-04-22T08:12:37.785 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-04-22T08:12:37.785 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:37.785 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-04-22T08:12:37.786689+0000 2024-04-22T08:12:37.785 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-04-22T08:12:37.786689+0000 2024-04-22T08:12:37.786 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:37.786 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-04-22T08:12:37.786 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:12:37.786 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:12:37.786 INFO:teuthology.orchestra.run.smithi189.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:12:37.786 INFO:teuthology.orchestra.run.smithi189.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:12:37.786 INFO:teuthology.orchestra.run.smithi189.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:12:37.786 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-04-22T08:12:37.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.786+0000 7f1c10dca640 1 Processor -- start 2024-04-22T08:12:37.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.786+0000 7f1c10dca640 1 -- start start 2024-04-22T08:12:37.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.786+0000 7f1c10dca640 10 monclient: init 2024-04-22T08:12:37.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.786+0000 7f1c10dca640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:37.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.786+0000 7f1c10dca640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-04-22T08:12:37.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.786+0000 7f1c10dca640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1c0c09ee10 0x7f1c0c1362c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:37.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.786+0000 7f1c10dca640 10 monclient(hunting): picked mon.noname-b con 0x7f1c0c09ee10 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:37.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.786+0000 7f1c10dca640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1c0c0a2d80 0x7f1c0c0a31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:37.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.786+0000 7f1c10dca640 10 monclient(hunting): picked mon.noname-c con 0x7f1c0c0a2d80 addr v2:172.21.15.189:3300/0 2024-04-22T08:12:37.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.786+0000 7f1c10dca640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1c0c0a3720 0x7f1c0c132b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:37.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.786+0000 7f1c10dca640 10 monclient(hunting): picked mon.noname-a con 0x7f1c0c0a3720 addr v2:172.21.15.40:3300/0 2024-04-22T08:12:37.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.786+0000 7f1c10dca640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:37.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.786+0000 7f1c10dca640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c0c09fc90 con 0x7f1c0c0a3720 2024-04-22T08:12:37.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.786+0000 7f1c10dca640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:37.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.786+0000 7f1c10dca640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c0c0a4260 con 0x7f1c0c0a2d80 2024-04-22T08:12:37.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.786+0000 7f1c10dca640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:37.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.786+0000 7f1c10dca640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0c0a3b90 con 0x7f1c0c09ee10 2024-04-22T08:12:37.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.786+0000 7f1c10dca640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:37.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.786+0000 7f1c10dca640 10 monclient(hunting): authenticate will time out at 811.079507s 2024-04-22T08:12:37.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.787+0000 7f1c09d9b640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1c0c0a2d80 0x7f1c0c0a31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:37.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.787+0000 7f1c09d9b640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1c0c0a2d80 0x7f1c0c0a31e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48562/0 (socket says 172.21.15.189:48562) 2024-04-22T08:12:37.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.787+0000 7f1c09d9b640 1 -- 172.21.15.189:0/3381042234 learned_addr learned my addr 172.21.15.189:0/3381042234 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-22T08:12:37.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.787+0000 7f1c09d9b640 10 monclient(hunting): get_auth_request con 0x7f1c0c0a2d80 auth_method 0 2024-04-22T08:12:37.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.787+0000 7f1c09d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:37.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.787+0000 7f1c09d9b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:37.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.787+0000 7f1c09d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:37.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.787+0000 7f1c09d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:37.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.787+0000 7f1c09d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:37.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.787+0000 7f1c09d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:37.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.787+0000 7f1c09d9b640 10 monclient(hunting): handle_auth_done global_id 4111 payload 274 2024-04-22T08:12:37.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.787+0000 7f1c0a59c640 1 --2- 172.21.15.189:0/3381042234 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1c0c09ee10 0x7f1c0c1362c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:37.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.787+0000 7f1c09d9b640 1 -- 172.21.15.189:0/3381042234 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1c0c09ee10 msgr2=0x7f1c0c1362c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:37.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.787+0000 7f1c0ad9d640 1 --2- 172.21.15.189:0/3381042234 >> v2:172.21.15.40:3300/0 conn(0x7f1c0c0a3720 0x7f1c0c132b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:37.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.787+0000 7f1c09d9b640 1 --2- 172.21.15.189:0/3381042234 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1c0c09ee10 0x7f1c0c1362c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:37.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.787+0000 7f1c09d9b640 1 -- 172.21.15.189:0/3381042234 >> v2:172.21.15.40:3300/0 conn(0x7f1c0c0a3720 msgr2=0x7f1c0c132b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:37.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.787+0000 7f1c09d9b640 1 --2- 172.21.15.189:0/3381042234 >> v2:172.21.15.40:3300/0 conn(0x7f1c0c0a3720 0x7f1c0c132b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:37.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.787+0000 7f1c09d9b640 10 monclient: _finish_hunting 0 2024-04-22T08:12:37.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.787+0000 7f1c09d9b640 1 monclient: found mon.noname-c 2024-04-22T08:12:37.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.787+0000 7f1c09d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:37.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.787+0000 7f1c09d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.189:3300/0 2024-04-22T08:12:37.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.787+0000 7f1c09d9b640 1 -- 172.21.15.189:0/3381042234 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c0c136980 con 0x7f1c0c0a2d80 2024-04-22T08:12:37.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.787+0000 7f1c0a59c640 10 monclient: get_auth_request con 0x7f1c0c09ee10 auth_method 0 2024-04-22T08:12:37.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.787+0000 7f1c0a59c640 1 --2- 172.21.15.189:0/3381042234 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1c0c09ee10 0x7f1c0c1362c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T08:12:37.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.790+0000 7f1c09d9b640 1 -- 172.21.15.189:0/3381042234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c0c0a2d80 msgr2=0x7f1c0c0a31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:37.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.790+0000 7f1c09d9b640 1 --2- 172.21.15.189:0/3381042234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c0c0a2d80 0x7f1c0c0a31e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c136b70 tx=0x7f1c0002b7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93723e5b14f0fa17 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:37.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.790+0000 7f1c0959a640 1 -- 172.21.15.189:0/3381042234 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c00031a30 con 0x7f1c0c0a2d80 2024-04-22T08:12:37.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.790+0000 7f1c0959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:37.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.790+0000 7f1c0959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:12:37.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.790+0000 7f1c0959a640 10 monclient: dump: 2024-04-22T08:12:37.803 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-04-22T08:12:37.803 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:37.803 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:37.803 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:37.803 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:37.803 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-04-22T08:12:37.803 INFO:teuthology.orchestra.run.smithi189.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:37.803 INFO:teuthology.orchestra.run.smithi189.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:37.803 INFO:teuthology.orchestra.run.smithi189.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:37.803 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-04-22T08:12:37.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.790+0000 7f1c0959a640 10 monclient: _finish_auth 0 2024-04-22T08:12:37.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.790+0000 7f1c0959a640 10 monclient: _check_auth_tickets 2024-04-22T08:12:37.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.790+0000 7f1c0959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:37.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.790+0000 7f1c0959a640 1 -- 172.21.15.189:0/3381042234 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c00031b90 con 0x7f1c0c0a2d80 2024-04-22T08:12:37.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.790+0000 7f1c10dca640 5 monclient: authenticate success, global_id 4111 2024-04-22T08:12:37.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.790+0000 7f1c10dca640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:12:37.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.790+0000 7f1c0959a640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:37.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.790+0000 7f1c10dca640 10 monclient: get_monmap_and_config success 2024-04-22T08:12:37.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.790+0000 7f1c0959a640 1 -- 172.21.15.189:0/3381042234 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c00033430 con 0x7f1c0c0a2d80 2024-04-22T08:12:37.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.790+0000 7f1c0959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:37.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.790+0000 7f1c10dca640 4 set_mon_vals no callback set 2024-04-22T08:12:37.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.790+0000 7f1c0959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:12:37.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.790+0000 7f1c0959a640 10 monclient: dump: 2024-04-22T08:12:37.805 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-04-22T08:12:37.805 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:37.805 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:37.805 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:37.805 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:37.805 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-04-22T08:12:37.805 INFO:teuthology.orchestra.run.smithi189.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:37.805 INFO:teuthology.orchestra.run.smithi189.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:37.805 INFO:teuthology.orchestra.run.smithi189.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:37.805 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-04-22T08:12:37.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.790+0000 7f1c10dca640 10 monclient: shutdown 2024-04-22T08:12:37.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.790+0000 7f1c10dca640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:37.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.790+0000 7f1c10dca640 1 -- 172.21.15.189:0/3381042234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c0c0a2d80 msgr2=0x7f1c0c0a31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:37.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.790+0000 7f1c10dca640 1 --2- 172.21.15.189:0/3381042234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c0c0a2d80 0x7f1c0c0a31e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c136b70 tx=0x7f1c0002b7e0 comp rx=0 tx=0).stop 2024-04-22T08:12:37.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.791+0000 7f1c10dca640 1 -- 172.21.15.189:0/3381042234 shutdown_connections 2024-04-22T08:12:37.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.791+0000 7f1c10dca640 1 --2- 172.21.15.189:0/3381042234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c0c0a2d80 0x7f1c0c0a31e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:37.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.791+0000 7f1c10dca640 1 --2- 172.21.15.189:0/3381042234 >> v2:172.21.15.40:3300/0 conn(0x7f1c0c0a3720 0x7f1c0c132b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:37.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.791+0000 7f1c10dca640 1 --2- 172.21.15.189:0/3381042234 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1c0c09ee10 0x7f1c0c1362c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:37.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.791+0000 7f1c10dca640 1 -- 172.21.15.189:0/3381042234 >> 172.21.15.189:0/3381042234 conn(0x7f1c0c0a5060 msgr2=0x7f1c0c0a5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:37.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.791+0000 7f1c10dca640 1 -- 172.21.15.189:0/3381042234 shutdown_connections 2024-04-22T08:12:37.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.791+0000 7f1c10dca640 1 -- 172.21.15.189:0/3381042234 wait complete. 2024-04-22T08:12:37.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.791+0000 7f1c10dca640 10 monclient: build_initial_monmap 2024-04-22T08:12:37.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.791+0000 7f1c10dca640 10 monclient: monmap: 2024-04-22T08:12:37.806 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-04-22T08:12:37.806 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:12:37.806 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 0.000000 2024-04-22T08:12:37.807 INFO:teuthology.orchestra.run.smithi189.stderr:created 0.000000 2024-04-22T08:12:37.807 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:37.807 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-04-22T08:12:37.807 INFO:teuthology.orchestra.run.smithi189.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:12:37.807 INFO:teuthology.orchestra.run.smithi189.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:12:37.807 INFO:teuthology.orchestra.run.smithi189.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:12:37.807 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-04-22T08:12:37.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.792+0000 7f1c10dca640 1 Processor -- start 2024-04-22T08:12:37.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.792+0000 7f1c10dca640 1 -- start start 2024-04-22T08:12:37.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.792+0000 7f1c10dca640 10 monclient: init 2024-04-22T08:12:37.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.792+0000 7f1c10dca640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:37.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.792+0000 7f1c10dca640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:12:37.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.792+0000 7f1c10dca640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c0c09ee10 0x7f1c0c11f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:37.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.792+0000 7f1c10dca640 10 monclient(hunting): picked mon.noname-a con 0x7f1c0c09ee10 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:37.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.792+0000 7f1c10dca640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c0c0a2d80 0x7f1c0c11f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:37.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.792+0000 7f1c10dca640 10 monclient(hunting): picked mon.noname-b con 0x7f1c0c0a2d80 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:37.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.792+0000 7f1c10dca640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1c0c0a3720 0x7f1c0c11fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:37.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.792+0000 7f1c10dca640 10 monclient(hunting): picked mon.noname-c con 0x7f1c0c0a3720 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:37.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.792+0000 7f1c10dca640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:37.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.792+0000 7f1c10dca640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0c1215c0 con 0x7f1c0c09ee10 2024-04-22T08:12:37.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.792+0000 7f1c10dca640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:37.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.792+0000 7f1c10dca640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0c1354a0 con 0x7f1c0c0a2d80 2024-04-22T08:12:37.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.792+0000 7f1c10dca640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:37.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.792+0000 7f1c10dca640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0c0a2830 con 0x7f1c0c0a3720 2024-04-22T08:12:37.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.792+0000 7f1c10dca640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:37.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.792+0000 7f1c10dca640 10 monclient(hunting): authenticate will time out at 811.085247s 2024-04-22T08:12:37.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.792+0000 7f1c09d9b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c0c0a2d80 0x7f1c0c11f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:37.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.792+0000 7f1c09d9b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c0c0a2d80 0x7f1c0c11f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48576/0 (socket says 172.21.15.189:48576) 2024-04-22T08:12:37.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.792+0000 7f1c09d9b640 1 -- 172.21.15.189:0/1093758657 learned_addr learned my addr 172.21.15.189:0/1093758657 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-22T08:12:37.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.792+0000 7f1c09d9b640 10 monclient(hunting): get_auth_request con 0x7f1c0c0a2d80 auth_method 0 2024-04-22T08:12:37.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.792+0000 7f1c09d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:37.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.792+0000 7f1c09d9b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:37.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.792+0000 7f1c09d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:37.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.792+0000 7f1c09d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:37.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.792+0000 7f1c09d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:37.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.792+0000 7f1c09d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:37.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.792+0000 7f1c0ad9d640 1 --2- 172.21.15.189:0/1093758657 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1c0c0a3720 0x7f1c0c11fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:37.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.792+0000 7f1c0a59c640 1 --2- 172.21.15.189:0/1093758657 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c0c09ee10 0x7f1c0c11f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:37.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.792+0000 7f1c0ad9d640 10 monclient(hunting): get_auth_request con 0x7f1c0c0a3720 auth_method 0 2024-04-22T08:12:37.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.792+0000 7f1c0ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:37.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.792+0000 7f1c0ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:37.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.792+0000 7f1c0ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:37.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.792+0000 7f1c0a59c640 10 monclient(hunting): get_auth_request con 0x7f1c0c09ee10 auth_method 0 2024-04-22T08:12:37.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.792+0000 7f1c0a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:37.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.792+0000 7f1c0a59c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:37.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.792+0000 7f1c0a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:37.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.792+0000 7f1c0ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:37.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.792+0000 7f1c0ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:37.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.792+0000 7f1c0ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:37.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.793+0000 7f1c0a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:37.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.793+0000 7f1c0a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:37.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.793+0000 7f1c0a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:37.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.793+0000 7f1c09d9b640 10 monclient(hunting): handle_auth_done global_id 4114 payload 824 2024-04-22T08:12:37.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.793+0000 7f1c09d9b640 1 -- 172.21.15.189:0/1093758657 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1c0c0a3720 msgr2=0x7f1c0c11fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:37.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.793+0000 7f1c09d9b640 1 --2- 172.21.15.189:0/1093758657 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1c0c0a3720 0x7f1c0c11fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:37.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.793+0000 7f1c09d9b640 1 -- 172.21.15.189:0/1093758657 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c0c09ee10 msgr2=0x7f1c0c11f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:37.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.793+0000 7f1c09d9b640 1 --2- 172.21.15.189:0/1093758657 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c0c09ee10 0x7f1c0c11f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:37.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.793+0000 7f1c09d9b640 10 monclient: _finish_hunting 0 2024-04-22T08:12:37.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.793+0000 7f1c09d9b640 1 monclient: found mon.noname-b 2024-04-22T08:12:37.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.793+0000 7f1c09d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:37.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.793+0000 7f1c09d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.189:3300/0 2024-04-22T08:12:37.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.793+0000 7f1c09d9b640 1 -- 172.21.15.189:0/1093758657 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c000094d0 con 0x7f1c0c0a2d80 2024-04-22T08:12:37.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.795+0000 7f1c09d9b640 1 -- 172.21.15.189:0/1093758657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c0c0a2d80 msgr2=0x7f1c0c11f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:37.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.795+0000 7f1c09d9b640 1 --2- 172.21.15.189:0/1093758657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c0c0a2d80 0x7f1c0c11f5a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1c000090b0 tx=0x7f1c000046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:37.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.795+0000 7f1bfaffd640 1 -- 172.21.15.189:0/1093758657 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c00031f00 con 0x7f1c0c0a2d80 2024-04-22T08:12:37.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.795+0000 7f1bfaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:37.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.795+0000 7f1bfaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:12:37.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.795+0000 7f1bfaffd640 10 monclient: dump: 2024-04-22T08:12:37.811 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-04-22T08:12:37.811 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:37.811 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:37.811 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:37.812 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:37.812 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-04-22T08:12:37.812 INFO:teuthology.orchestra.run.smithi189.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:37.812 INFO:teuthology.orchestra.run.smithi189.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:37.812 INFO:teuthology.orchestra.run.smithi189.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:37.812 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-04-22T08:12:37.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.795+0000 7f1bfaffd640 10 monclient: _finish_auth 0 2024-04-22T08:12:37.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.795+0000 7f1bfaffd640 10 monclient: _check_auth_tickets 2024-04-22T08:12:37.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.795+0000 7f1bfaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:37.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.795+0000 7f1bfaffd640 1 -- 172.21.15.189:0/1093758657 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c0003c400 con 0x7f1c0c0a2d80 2024-04-22T08:12:37.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.795+0000 7f1c10dca640 5 monclient: authenticate success, global_id 4114 2024-04-22T08:12:37.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.795+0000 7f1bfaffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:37.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.795+0000 7f1bfaffd640 1 -- 172.21.15.189:0/1093758657 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c0003c7b0 con 0x7f1c0c0a2d80 2024-04-22T08:12:37.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.795+0000 7f1c10dca640 10 monclient: _renew_subs 2024-04-22T08:12:37.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.795+0000 7f1c10dca640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.189:3300/0 2024-04-22T08:12:37.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.795+0000 7f1c10dca640 1 -- 172.21.15.189:0/1093758657 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c0c122d50 con 0x7f1c0c0a2d80 2024-04-22T08:12:37.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.795+0000 7f1bfaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:37.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.795+0000 7f1bfaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:12:37.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.795+0000 7f1bfaffd640 10 monclient: dump: 2024-04-22T08:12:37.813 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-04-22T08:12:37.813 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:37.813 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:37.813 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:37.813 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:37.813 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-04-22T08:12:37.813 INFO:teuthology.orchestra.run.smithi189.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:37.813 INFO:teuthology.orchestra.run.smithi189.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:37.813 INFO:teuthology.orchestra.run.smithi189.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:37.813 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-04-22T08:12:37.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.795+0000 7f1c08d99640 4 set_mon_vals no callback set 2024-04-22T08:12:37.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.795+0000 7f1c10dca640 10 monclient: _renew_subs 2024-04-22T08:12:37.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.795+0000 7f1c10dca640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.189:3300/0 2024-04-22T08:12:37.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.795+0000 7f1c10dca640 1 -- 172.21.15.189:0/1093758657 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c0c1209f0 con 0x7f1c0c0a2d80 2024-04-22T08:12:37.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.795+0000 7f1bfaffd640 1 -- 172.21.15.189:0/1093758657 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f1c0003a040 con 0x7f1c0c0a2d80 2024-04-22T08:12:37.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.795+0000 7f1bfaffd640 1 -- 172.21.15.189:0/1093758657 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f1c00048c10 con 0x7f1c0c0a2d80 2024-04-22T08:12:37.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.796+0000 7f1c10dca640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:37.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.796+0000 7f1c10dca640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:37.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.796+0000 7f1c10dca640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.189:3300/0 2024-04-22T08:12:37.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.796+0000 7f1c10dca640 1 -- 172.21.15.189:0/1093758657 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c0c04e9b0 con 0x7f1c0c0a2d80 2024-04-22T08:12:37.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.798+0000 7f1bfaffd640 1 -- 172.21.15.189:0/1093758657 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (secure 0 0 0) 0x7f1c00067dc0 con 0x7f1c0c0a2d80 2024-04-22T08:12:37.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.798+0000 7f1bfaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:37.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.798+0000 7f1bfaffd640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:12:37.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.869+0000 7f1c10dca640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "11370cc8-09a6-4c35-b92c-4d926c6c18d9", "id": 1}] 2024-04-22T08:12:37.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.869+0000 7f1c10dca640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "11370cc8-09a6-4c35-b92c-4d926c6c18d9", "id": 1}] 2024-04-22T08:12:37.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.869+0000 7f1c10dca640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.189:3300/0 2024-04-22T08:12:37.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.869+0000 7f1c10dca640 1 -- 172.21.15.189:0/1093758657 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd new", "uuid": "11370cc8-09a6-4c35-b92c-4d926c6c18d9", "id": 1} v 0) v1 -- 0x7f1c0c06a030 con 0x7f1c0c0a2d80 2024-04-22T08:12:37.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.877+0000 7f1bfaffd640 1 -- 172.21.15.189:0/1093758657 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "11370cc8-09a6-4c35-b92c-4d926c6c18d9", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f1c00056070 con 0x7f1c0c0a2d80 2024-04-22T08:12:37.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.877+0000 7f1bfaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "11370cc8-09a6-4c35-b92c-4d926c6c18d9", "id": 1}] 2024-04-22T08:12:37.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.877+0000 7f1bfaffd640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:12:37.877 INFO:teuthology.orchestra.run.smithi189.stdout:1 2024-04-22T08:12:37.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.878+0000 7f1c10dca640 10 monclient: shutdown 2024-04-22T08:12:37.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.878+0000 7f1c10dca640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:37.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.878+0000 7f1c10dca640 1 -- 172.21.15.189:0/1093758657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c0c0a2d80 msgr2=0x7f1c0c11f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:37.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.878+0000 7f1c10dca640 1 --2- 172.21.15.189:0/1093758657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c0c0a2d80 0x7f1c0c11f5a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1c000090b0 tx=0x7f1c000046c0 comp rx=0 tx=0).stop 2024-04-22T08:12:37.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.878+0000 7f1c10dca640 1 -- 172.21.15.189:0/1093758657 shutdown_connections 2024-04-22T08:12:37.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.878+0000 7f1c10dca640 1 --2- 172.21.15.189:0/1093758657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c0c0a2d80 0x7f1c0c11f5a0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:37.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.878+0000 7f1c10dca640 1 --2- 172.21.15.189:0/1093758657 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1c0c0a3720 0x7f1c0c11fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:37.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.878+0000 7f1c10dca640 1 --2- 172.21.15.189:0/1093758657 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c0c09ee10 0x7f1c0c11f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:37.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.878+0000 7f1c10dca640 1 -- 172.21.15.189:0/1093758657 >> 172.21.15.189:0/1093758657 conn(0x7f1c0c0a5060 msgr2=0x7f1c0c135790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:37.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.878+0000 7f1c10dca640 1 -- 172.21.15.189:0/1093758657 shutdown_connections 2024-04-22T08:12:37.879 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.878+0000 7f1c10dca640 1 -- 172.21.15.189:0/1093758657 wait complete. 2024-04-22T08:12:37.888 DEBUG:teuthology.orchestra.run.smithi189:> sudo ceph --cluster ceph osd new d03db34e-f630-4a3e-86fb-7234bee0d0f0 2 2024-04-22T08:12:37.902 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-04-22T08:12:37.902+0000 7f0813e38040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T08:12:37.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.997+0000 7f1a5e5b9640 10 monclient: get_monmap_and_config 2024-04-22T08:12:37.999 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.997+0000 7f1a5e5b9640 10 monclient: build_initial_monmap 2024-04-22T08:12:38.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.997+0000 7f1a5e5b9640 10 monclient: monmap: 2024-04-22T08:12:38.000 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-04-22T08:12:38.000 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:38.000 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-04-22T08:12:37.998595+0000 2024-04-22T08:12:38.000 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-04-22T08:12:37.998595+0000 2024-04-22T08:12:38.000 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:38.000 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-04-22T08:12:38.000 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:12:38.000 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:12:38.000 INFO:teuthology.orchestra.run.smithi189.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:12:38.000 INFO:teuthology.orchestra.run.smithi189.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:12:38.000 INFO:teuthology.orchestra.run.smithi189.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:12:38.000 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-04-22T08:12:38.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.998+0000 7f1a5e5b9640 1 Processor -- start 2024-04-22T08:12:38.000 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.998+0000 7f1a5e5b9640 1 -- start start 2024-04-22T08:12:38.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.998+0000 7f1a5e5b9640 10 monclient: init 2024-04-22T08:12:38.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.998+0000 7f1a5e5b9640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:38.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.998+0000 7f1a5e5b9640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-04-22T08:12:38.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.998+0000 7f1a5e5b9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1a5809ee10 addr v1:172.21.15.189:6789/0 2024-04-22T08:12:38.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.998+0000 7f1a5e5b9640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1a581362c0 0x7f1a580a3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:38.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.998+0000 7f1a5e5b9640 10 monclient(hunting): picked mon.noname-a con 0x7f1a581362c0 addr v2:172.21.15.40:3300/0 2024-04-22T08:12:38.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.998+0000 7f1a5e5b9640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1a5813b1e0 0x7f1a5813d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:38.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.998+0000 7f1a5e5b9640 10 monclient(hunting): picked mon.noname-c con 0x7f1a5813b1e0 addr v2:172.21.15.189:3300/0 2024-04-22T08:12:38.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.998+0000 7f1a5e5b9640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a58136750 con 0x7f1a5809ee10 2024-04-22T08:12:38.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.998+0000 7f1a5e5b9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:38.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.998+0000 7f1a5e5b9640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a5809fc90 con 0x7f1a581362c0 2024-04-22T08:12:38.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.998+0000 7f1a5e5b9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:38.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.998+0000 7f1a5e5b9640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a580a4260 con 0x7f1a5813b1e0 2024-04-22T08:12:38.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.998+0000 7f1a5e5b9640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:38.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.998+0000 7f1a5e5b9640 10 monclient(hunting): authenticate will time out at 811.291433s 2024-04-22T08:12:38.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.998+0000 7f1a5cb56640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1a5813b1e0 0x7f1a5813d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:38.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.998+0000 7f1a57fff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f1a5809ee10 0x7f1a580a2d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57920/0 (socket says 172.21.15.189:57920) 2024-04-22T08:12:38.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.998+0000 7f1a57fff640 1 -- 172.21.15.189:0/3288758418 learned_addr learned my addr 172.21.15.189:0/3288758418 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-22T08:12:38.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.999+0000 7f1a5cb56640 10 monclient(hunting): get_auth_request con 0x7f1a5813b1e0 auth_method 0 2024-04-22T08:12:38.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.999+0000 7f1a5cb56640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:38.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.999+0000 7f1a5cb56640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:38.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.999+0000 7f1a5cb56640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:38.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.999+0000 7f1a5cb56640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:38.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.999+0000 7f1a5cb56640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:38.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.999+0000 7f1a5cb56640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:38.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.999+0000 7f1a5cb56640 10 monclient(hunting): handle_auth_done global_id 4117 payload 274 2024-04-22T08:12:38.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.999+0000 7f1a577fe640 1 --2- 172.21.15.189:0/3288758418 >> v2:172.21.15.40:3300/0 conn(0x7f1a581362c0 0x7f1a580a3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:38.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.999+0000 7f1a5cb56640 1 -- 172.21.15.189:0/3288758418 >> v2:172.21.15.40:3300/0 conn(0x7f1a581362c0 msgr2=0x7f1a580a3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:38.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.999+0000 7f1a5cb56640 1 --2- 172.21.15.189:0/3288758418 >> v2:172.21.15.40:3300/0 conn(0x7f1a581362c0 0x7f1a580a3490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:38.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.999+0000 7f1a5cb56640 1 -- 172.21.15.189:0/3288758418 >> v1:172.21.15.189:6789/0 conn(0x7f1a5809ee10 legacy=0x7f1a580a2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:38.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.999+0000 7f1a5cb56640 10 monclient: _finish_hunting 0 2024-04-22T08:12:38.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.999+0000 7f1a5cb56640 1 monclient: found mon.noname-c 2024-04-22T08:12:38.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.999+0000 7f1a5cb56640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:38.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.999+0000 7f1a5cb56640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.189:3300/0 2024-04-22T08:12:38.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:37.999+0000 7f1a5cb56640 1 -- 172.21.15.189:0/3288758418 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a58136980 con 0x7f1a5813b1e0 2024-04-22T08:12:38.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.002+0000 7f1a5cb56640 1 -- 172.21.15.189:0/3288758418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a5813b1e0 msgr2=0x7f1a5813d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:38.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.002+0000 7f1a5cb56640 1 --2- 172.21.15.189:0/3288758418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a5813b1e0 0x7f1a5813d5d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1a5809fdd0 tx=0x7f1a5002bd00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0d49a16a85174b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:38.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.002+0000 7f1a56ffd640 1 -- 172.21.15.189:0/3288758418 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a5002cda0 con 0x7f1a5813b1e0 2024-04-22T08:12:38.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.002+0000 7f1a56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:38.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.002+0000 7f1a56ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:12:38.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.002+0000 7f1a56ffd640 10 monclient: dump: 2024-04-22T08:12:38.009 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-04-22T08:12:38.009 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:38.009 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:38.009 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:38.009 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:38.009 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-04-22T08:12:38.009 INFO:teuthology.orchestra.run.smithi189.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:38.009 INFO:teuthology.orchestra.run.smithi189.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:38.009 INFO:teuthology.orchestra.run.smithi189.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:38.009 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-04-22T08:12:38.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.002+0000 7f1a56ffd640 10 monclient: _finish_auth 0 2024-04-22T08:12:38.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.002+0000 7f1a56ffd640 10 monclient: _check_auth_tickets 2024-04-22T08:12:38.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.002+0000 7f1a56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:38.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.002+0000 7f1a56ffd640 1 -- 172.21.15.189:0/3288758418 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a50034440 con 0x7f1a5813b1e0 2024-04-22T08:12:38.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.002+0000 7f1a5e5b9640 5 monclient: authenticate success, global_id 4117 2024-04-22T08:12:38.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.002+0000 7f1a5e5b9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:12:38.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.002+0000 7f1a56ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:38.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.002+0000 7f1a5e5b9640 10 monclient: get_monmap_and_config success 2024-04-22T08:12:38.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.002+0000 7f1a56ffd640 1 -- 172.21.15.189:0/3288758418 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a500347d0 con 0x7f1a5813b1e0 2024-04-22T08:12:38.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.002+0000 7f1a56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:38.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.002+0000 7f1a5e5b9640 4 set_mon_vals no callback set 2024-04-22T08:12:38.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.002+0000 7f1a56ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:12:38.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.002+0000 7f1a56ffd640 10 monclient: dump: 2024-04-22T08:12:38.010 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-04-22T08:12:38.010 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:38.010 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:38.010 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:38.010 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:38.011 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-04-22T08:12:38.011 INFO:teuthology.orchestra.run.smithi189.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:38.011 INFO:teuthology.orchestra.run.smithi189.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:38.011 INFO:teuthology.orchestra.run.smithi189.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:38.011 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-04-22T08:12:38.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.002+0000 7f1a5e5b9640 10 monclient: shutdown 2024-04-22T08:12:38.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.002+0000 7f1a5e5b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:38.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.002+0000 7f1a5e5b9640 1 -- 172.21.15.189:0/3288758418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a5813b1e0 msgr2=0x7f1a5813d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:38.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.002+0000 7f1a5e5b9640 1 --2- 172.21.15.189:0/3288758418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a5813b1e0 0x7f1a5813d5d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1a5809fdd0 tx=0x7f1a5002bd00 comp rx=0 tx=0).stop 2024-04-22T08:12:38.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.002+0000 7f1a5e5b9640 1 -- 172.21.15.189:0/3288758418 shutdown_connections 2024-04-22T08:12:38.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.002+0000 7f1a5e5b9640 1 --2- 172.21.15.189:0/3288758418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a5813b1e0 0x7f1a5813d5d0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:38.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.002+0000 7f1a5e5b9640 1 --2- 172.21.15.189:0/3288758418 >> v2:172.21.15.40:3300/0 conn(0x7f1a581362c0 0x7f1a580a3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:38.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.002+0000 7f1a5e5b9640 1 -- 172.21.15.189:0/3288758418 >> 172.21.15.189:0/3288758418 conn(0x7f1a580a5060 msgr2=0x7f1a580a5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:38.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.002+0000 7f1a5e5b9640 1 -- 172.21.15.189:0/3288758418 shutdown_connections 2024-04-22T08:12:38.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.002+0000 7f1a5e5b9640 1 -- 172.21.15.189:0/3288758418 wait complete. 2024-04-22T08:12:38.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.003+0000 7f1a5e5b9640 10 monclient: build_initial_monmap 2024-04-22T08:12:38.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.003+0000 7f1a5e5b9640 10 monclient: monmap: 2024-04-22T08:12:38.012 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-04-22T08:12:38.012 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:12:38.012 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 0.000000 2024-04-22T08:12:38.012 INFO:teuthology.orchestra.run.smithi189.stderr:created 0.000000 2024-04-22T08:12:38.012 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:38.012 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-04-22T08:12:38.012 INFO:teuthology.orchestra.run.smithi189.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:12:38.013 INFO:teuthology.orchestra.run.smithi189.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:12:38.013 INFO:teuthology.orchestra.run.smithi189.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:12:38.013 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-04-22T08:12:38.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.003+0000 7f1a5e5b9640 1 Processor -- start 2024-04-22T08:12:38.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a5e5b9640 1 -- start start 2024-04-22T08:12:38.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a5e5b9640 10 monclient: init 2024-04-22T08:12:38.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a5e5b9640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:38.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a5e5b9640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-22T08:12:38.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a5e5b9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a5809ee10 0x7f1a581cac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:38.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a5e5b9640 10 monclient(hunting): picked mon.noname-a con 0x7f1a5809ee10 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:38.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a5e5b9640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a581362c0 0x7f1a581cb170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:38.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a5e5b9640 10 monclient(hunting): picked mon.noname-c con 0x7f1a581362c0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:38.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a5e5b9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a5813b1e0 0x7f1a5806ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:38.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a5e5b9640 10 monclient(hunting): picked mon.noname-b con 0x7f1a5813b1e0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:38.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a5e5b9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:38.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a5e5b9640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a580711e0 con 0x7f1a5809ee10 2024-04-22T08:12:38.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a5e5b9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:38.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a5e5b9640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5811ebb0 con 0x7f1a5813b1e0 2024-04-22T08:12:38.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a5e5b9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:38.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a5e5b9640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a580a2830 con 0x7f1a581362c0 2024-04-22T08:12:38.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a5e5b9640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:38.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a5e5b9640 10 monclient(hunting): authenticate will time out at 811.297004s 2024-04-22T08:12:38.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a5cb56640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a5813b1e0 0x7f1a5806ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:38.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a5cb56640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a5813b1e0 0x7f1a5806ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48590/0 (socket says 172.21.15.189:48590) 2024-04-22T08:12:38.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a5cb56640 1 -- 172.21.15.189:0/1691576739 learned_addr learned my addr 172.21.15.189:0/1691576739 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-22T08:12:38.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a5cb56640 10 monclient(hunting): get_auth_request con 0x7f1a5813b1e0 auth_method 0 2024-04-22T08:12:38.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a5cb56640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:38.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a5cb56640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:38.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a5cb56640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:38.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a5cb56640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:38.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a5cb56640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:38.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a5cb56640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:38.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a57fff640 1 --2- 172.21.15.189:0/1691576739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a5809ee10 0x7f1a581cac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:38.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a577fe640 1 --2- 172.21.15.189:0/1691576739 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a581362c0 0x7f1a581cb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:38.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a577fe640 10 monclient(hunting): get_auth_request con 0x7f1a581362c0 auth_method 0 2024-04-22T08:12:38.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:38.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a577fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:38.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a577fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:38.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a57fff640 10 monclient(hunting): get_auth_request con 0x7f1a5809ee10 auth_method 0 2024-04-22T08:12:38.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a57fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:38.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a57fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:38.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a57fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:38.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a5cb56640 10 monclient(hunting): handle_auth_done global_id 4120 payload 824 2024-04-22T08:12:38.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a5cb56640 1 -- 172.21.15.189:0/1691576739 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a581362c0 msgr2=0x7f1a581cb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:38.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a5cb56640 1 --2- 172.21.15.189:0/1691576739 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a581362c0 0x7f1a581cb170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:38.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a5cb56640 1 -- 172.21.15.189:0/1691576739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a5809ee10 msgr2=0x7f1a581cac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:38.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a5cb56640 1 --2- 172.21.15.189:0/1691576739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a5809ee10 0x7f1a581cac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:38.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a5cb56640 10 monclient: _finish_hunting 0 2024-04-22T08:12:38.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a5cb56640 1 monclient: found mon.noname-b 2024-04-22T08:12:38.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a5cb56640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:38.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a5cb56640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.189:3300/0 2024-04-22T08:12:38.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a5cb56640 1 -- 172.21.15.189:0/1691576739 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a50009a70 con 0x7f1a5813b1e0 2024-04-22T08:12:38.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a577fe640 1 --2- 172.21.15.189:0/1691576739 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a581362c0 0x7f1a581cb170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T08:12:38.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.004+0000 7f1a57fff640 1 --2- 172.21.15.189:0/1691576739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a5809ee10 0x7f1a581cac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T08:12:38.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.007+0000 7f1a5cb56640 1 -- 172.21.15.189:0/1691576739 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a5813b1e0 msgr2=0x7f1a5806ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:38.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.007+0000 7f1a5cb56640 1 --2- 172.21.15.189:0/1691576739 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a5813b1e0 0x7f1a5806ea70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1a50037040 tx=0x7f1a50004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:38.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.007+0000 7f1a54ff9640 1 -- 172.21.15.189:0/1691576739 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a50040650 con 0x7f1a5813b1e0 2024-04-22T08:12:38.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.007+0000 7f1a54ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:38.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.007+0000 7f1a54ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:12:38.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.007+0000 7f1a54ff9640 10 monclient: dump: 2024-04-22T08:12:38.017 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-04-22T08:12:38.017 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:38.017 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:38.017 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:38.017 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:38.017 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-04-22T08:12:38.017 INFO:teuthology.orchestra.run.smithi189.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:38.017 INFO:teuthology.orchestra.run.smithi189.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:38.017 INFO:teuthology.orchestra.run.smithi189.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:38.018 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-04-22T08:12:38.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.007+0000 7f1a54ff9640 10 monclient: _finish_auth 0 2024-04-22T08:12:38.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.007+0000 7f1a54ff9640 10 monclient: _check_auth_tickets 2024-04-22T08:12:38.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.007+0000 7f1a54ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:38.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.007+0000 7f1a5e5b9640 5 monclient: authenticate success, global_id 4120 2024-04-22T08:12:38.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.007+0000 7f1a54ff9640 1 -- 172.21.15.189:0/1691576739 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a500407b0 con 0x7f1a5813b1e0 2024-04-22T08:12:38.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.007+0000 7f1a5e5b9640 10 monclient: _renew_subs 2024-04-22T08:12:38.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.007+0000 7f1a5e5b9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.189:3300/0 2024-04-22T08:12:38.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.007+0000 7f1a5e5b9640 1 -- 172.21.15.189:0/1691576739 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a580a6640 con 0x7f1a5813b1e0 2024-04-22T08:12:38.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.007+0000 7f1a54ff9640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:38.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.007+0000 7f1a54ff9640 1 -- 172.21.15.189:0/1691576739 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a50040b40 con 0x7f1a5813b1e0 2024-04-22T08:12:38.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.007+0000 7f1a54ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:38.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.007+0000 7f1a54ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:12:38.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.007+0000 7f1a54ff9640 10 monclient: dump: 2024-04-22T08:12:38.019 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-04-22T08:12:38.019 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:38.019 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:38.019 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:38.019 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:38.019 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-04-22T08:12:38.019 INFO:teuthology.orchestra.run.smithi189.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:38.019 INFO:teuthology.orchestra.run.smithi189.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:38.019 INFO:teuthology.orchestra.run.smithi189.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:38.019 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-04-22T08:12:38.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.007+0000 7f1a567fc640 4 set_mon_vals no callback set 2024-04-22T08:12:38.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.007+0000 7f1a5e5b9640 10 monclient: _renew_subs 2024-04-22T08:12:38.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.007+0000 7f1a5e5b9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.189:3300/0 2024-04-22T08:12:38.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.007+0000 7f1a5e5b9640 1 -- 172.21.15.189:0/1691576739 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a5811f450 con 0x7f1a5813b1e0 2024-04-22T08:12:38.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.007+0000 7f1a54ff9640 1 -- 172.21.15.189:0/1691576739 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f1a50049400 con 0x7f1a5813b1e0 2024-04-22T08:12:38.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.007+0000 7f1a54ff9640 1 -- 172.21.15.189:0/1691576739 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f1a5003e820 con 0x7f1a5813b1e0 2024-04-22T08:12:38.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.008+0000 7f1a5e5b9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:38.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.008+0000 7f1a5e5b9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:38.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.008+0000 7f1a5e5b9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.189:3300/0 2024-04-22T08:12:38.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.008+0000 7f1a5e5b9640 1 -- 172.21.15.189:0/1691576739 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a58070690 con 0x7f1a5813b1e0 2024-04-22T08:12:38.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.011+0000 7f1a54ff9640 1 -- 172.21.15.189:0/1691576739 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (secure 0 0 0) 0x7f1a50067dc0 con 0x7f1a5813b1e0 2024-04-22T08:12:38.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.011+0000 7f1a54ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:38.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.011+0000 7f1a54ff9640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:12:38.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.084+0000 7f1a5e5b9640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d03db34e-f630-4a3e-86fb-7234bee0d0f0", "id": 2}] 2024-04-22T08:12:38.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.084+0000 7f1a5e5b9640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d03db34e-f630-4a3e-86fb-7234bee0d0f0", "id": 2}] 2024-04-22T08:12:38.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.084+0000 7f1a5e5b9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.189:3300/0 2024-04-22T08:12:38.084 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.084+0000 7f1a5e5b9640 1 -- 172.21.15.189:0/1691576739 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d03db34e-f630-4a3e-86fb-7234bee0d0f0", "id": 2} v 0) v1 -- 0x7f1a5811ef20 con 0x7f1a5813b1e0 2024-04-22T08:12:38.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.092+0000 7f1a5cb56640 1 --2- 172.21.15.189:0/1691576739 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a5813b1e0 0x7f1a5806ea70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1a50037040 tx=0x7f1a50004a40 comp rx=0 tx=0).queue_received will delay after 1.71377e+09 on 0x7f1a50056030 mon_command_ack([{"prefix": "osd new", "uuid": "d03db34e-f630-4a3e-86fb-7234bee0d0f0", "id": 2}]=0 v5) v1 2024-04-22T08:12:38.292 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-04-22T08:12:38.292+0000 7f0813e38040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T08:12:38.543 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-04-22T08:12:38.543+0000 7f0813e38040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T08:12:38.594 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-04-22T08:12:38.594+0000 7f0813e38040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T08:12:38.707 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-04-22T08:12:38.707+0000 7f0813e38040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T08:12:38.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.920+0000 7f1a54ff9640 1 -- 172.21.15.189:0/1691576739 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d03db34e-f630-4a3e-86fb-7234bee0d0f0", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f1a50056030 con 0x7f1a5813b1e0 2024-04-22T08:12:38.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.920+0000 7f1a54ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d03db34e-f630-4a3e-86fb-7234bee0d0f0", "id": 2}] 2024-04-22T08:12:38.920 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.920+0000 7f1a54ff9640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:12:38.921 INFO:teuthology.orchestra.run.smithi189.stdout:2 2024-04-22T08:12:38.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.921+0000 7f1a5e5b9640 10 monclient: shutdown 2024-04-22T08:12:38.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.921+0000 7f1a5e5b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:38.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.921+0000 7f1a5e5b9640 1 -- 172.21.15.189:0/1691576739 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a5813b1e0 msgr2=0x7f1a5806ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:38.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.921+0000 7f1a5e5b9640 1 --2- 172.21.15.189:0/1691576739 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a5813b1e0 0x7f1a5806ea70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1a50037040 tx=0x7f1a50004a40 comp rx=0 tx=0).stop 2024-04-22T08:12:38.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.921+0000 7f1a5e5b9640 1 -- 172.21.15.189:0/1691576739 shutdown_connections 2024-04-22T08:12:38.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.921+0000 7f1a5e5b9640 1 --2- 172.21.15.189:0/1691576739 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a5813b1e0 0x7f1a5806ea70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:38.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.921+0000 7f1a5e5b9640 1 --2- 172.21.15.189:0/1691576739 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a581362c0 0x7f1a581cb170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:38.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.921+0000 7f1a5e5b9640 1 --2- 172.21.15.189:0/1691576739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a5809ee10 0x7f1a581cac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:38.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.921+0000 7f1a5e5b9640 1 -- 172.21.15.189:0/1691576739 >> 172.21.15.189:0/1691576739 conn(0x7f1a580a5060 msgr2=0x7f1a580a0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:38.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.922+0000 7f1a5e5b9640 1 -- 172.21.15.189:0/1691576739 shutdown_connections 2024-04-22T08:12:38.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:38.922+0000 7f1a5e5b9640 1 -- 172.21.15.189:0/1691576739 wait complete. 2024-04-22T08:12:38.932 DEBUG:teuthology.orchestra.run.smithi189:> sudo ceph --cluster ceph osd new d94b2052-5847-4955-b1d4-b7e627572811 3 2024-04-22T08:12:38.981 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-04-22T08:12:38.981+0000 7f0813e38040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T08:12:39.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.028+0000 7f6f64ca1640 10 monclient: get_monmap_and_config 2024-04-22T08:12:39.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.028+0000 7f6f64ca1640 10 monclient: build_initial_monmap 2024-04-22T08:12:39.028 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.028+0000 7f6f64ca1640 10 monclient: monmap: 2024-04-22T08:12:39.028 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-04-22T08:12:39.028 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:39.028 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-04-22T08:12:39.029266+0000 2024-04-22T08:12:39.028 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-04-22T08:12:39.029266+0000 2024-04-22T08:12:39.028 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:39.028 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-04-22T08:12:39.028 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:12:39.028 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:12:39.028 INFO:teuthology.orchestra.run.smithi189.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:12:39.029 INFO:teuthology.orchestra.run.smithi189.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:12:39.029 INFO:teuthology.orchestra.run.smithi189.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:12:39.029 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-04-22T08:12:39.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.028+0000 7f6f64ca1640 1 Processor -- start 2024-04-22T08:12:39.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.028+0000 7f6f64ca1640 1 -- start start 2024-04-22T08:12:39.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.028+0000 7f6f64ca1640 10 monclient: init 2024-04-22T08:12:39.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.029+0000 7f6f64ca1640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:39.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.029+0000 7f6f64ca1640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-04-22T08:12:39.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.029+0000 7f6f64ca1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6f6009ee10 addr v1:172.21.15.189:6789/0 2024-04-22T08:12:39.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.029+0000 7f6f64ca1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6f601362c0 addr v1:172.21.15.40:6789/0 2024-04-22T08:12:39.031 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.029+0000 7f6f64ca1640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6f6013c1f0 0x7f6f6013e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:39.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.029+0000 7f6f64ca1640 10 monclient(hunting): picked mon.noname-b con 0x7f6f6013c1f0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:39.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.029+0000 7f6f64ca1640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f60136750 con 0x7f6f601362c0 2024-04-22T08:12:39.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.029+0000 7f6f64ca1640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f600a4260 con 0x7f6f6009ee10 2024-04-22T08:12:39.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.029+0000 7f6f64ca1640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:39.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.029+0000 7f6f64ca1640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6009fc90 con 0x7f6f6013c1f0 2024-04-22T08:12:39.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.029+0000 7f6f64ca1640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:39.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.029+0000 7f6f64ca1640 10 monclient(hunting): authenticate will time out at 812.322143s 2024-04-22T08:12:39.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.029+0000 7f6f5e59c640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f6f6009ee10 0x7f6f600a2d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57924/0 (socket says 172.21.15.189:57924) 2024-04-22T08:12:39.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.029+0000 7f6f5e59c640 1 -- 172.21.15.189:0/3790644515 learned_addr learned my addr 172.21.15.189:0/3790644515 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-22T08:12:39.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.029+0000 7f6f5ed9d640 1 --2- 172.21.15.189:0/3790644515 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6f6013c1f0 0x7f6f6013e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:39.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.029+0000 7f6f5ed9d640 10 monclient(hunting): get_auth_request con 0x7f6f6013c1f0 auth_method 0 2024-04-22T08:12:39.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.029+0000 7f6f5ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:39.032 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.029+0000 7f6f5ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:39.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.029+0000 7f6f5ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:39.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.029+0000 7f6f5ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:39.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.029+0000 7f6f5ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:39.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.029+0000 7f6f5ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:39.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.030+0000 7f6f5ed9d640 10 monclient(hunting): handle_auth_done global_id 4115 payload 274 2024-04-22T08:12:39.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.030+0000 7f6f5ed9d640 1 -- 172.21.15.189:0/3790644515 >> v1:172.21.15.189:6789/0 conn(0x7f6f6009ee10 legacy=0x7f6f600a2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:39.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.030+0000 7f6f5ed9d640 1 -- 172.21.15.189:0/3790644515 >> v1:172.21.15.40:6789/0 conn(0x7f6f601362c0 legacy=0x7f6f600a3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:39.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.030+0000 7f6f5ed9d640 10 monclient: _finish_hunting 0 2024-04-22T08:12:39.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.030+0000 7f6f5ed9d640 1 monclient: found mon.noname-b 2024-04-22T08:12:39.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.030+0000 7f6f5ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:39.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.030+0000 7f6f5ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:12:39.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.030+0000 7f6f5ed9d640 1 -- 172.21.15.189:0/3790644515 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f600a3ba0 con 0x7f6f6013c1f0 2024-04-22T08:12:39.034 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-04-22T08:12:39.031+0000 7f0813e38040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T08:12:39.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.032+0000 7f6f5ed9d640 1 -- 172.21.15.189:0/3790644515 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6f6013c1f0 msgr2=0x7f6f6013e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:39.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.032+0000 7f6f5ed9d640 1 --2- 172.21.15.189:0/3790644515 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6f6013c1f0 0x7f6f6013e5e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6f58009dc0 tx=0x7f6f5802bce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fad4dd956eef6124 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:39.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.033+0000 7f6f5d59a640 1 -- 172.21.15.189:0/3790644515 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f5802cd80 con 0x7f6f6013c1f0 2024-04-22T08:12:39.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.033+0000 7f6f5d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:39.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.033+0000 7f6f5d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:12:39.039 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.033+0000 7f6f5d59a640 10 monclient: dump: 2024-04-22T08:12:39.039 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-04-22T08:12:39.039 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:39.039 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:39.040 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:39.040 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:39.040 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-04-22T08:12:39.040 INFO:teuthology.orchestra.run.smithi189.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:39.040 INFO:teuthology.orchestra.run.smithi189.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:39.040 INFO:teuthology.orchestra.run.smithi189.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:39.040 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-04-22T08:12:39.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.033+0000 7f6f5d59a640 10 monclient: _finish_auth 0 2024-04-22T08:12:39.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.033+0000 7f6f5d59a640 10 monclient: _check_auth_tickets 2024-04-22T08:12:39.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.033+0000 7f6f5d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:39.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.033+0000 7f6f5d59a640 1 -- 172.21.15.189:0/3790644515 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f58034440 con 0x7f6f6013c1f0 2024-04-22T08:12:39.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.033+0000 7f6f64ca1640 5 monclient: authenticate success, global_id 4115 2024-04-22T08:12:39.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.033+0000 7f6f64ca1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:12:39.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.033+0000 7f6f5d59a640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:39.040 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.033+0000 7f6f64ca1640 10 monclient: get_monmap_and_config success 2024-04-22T08:12:39.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.033+0000 7f6f5d59a640 1 -- 172.21.15.189:0/3790644515 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f580347d0 con 0x7f6f6013c1f0 2024-04-22T08:12:39.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.033+0000 7f6f64ca1640 4 set_mon_vals no callback set 2024-04-22T08:12:39.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.033+0000 7f6f5d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:39.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.033+0000 7f6f5d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:12:39.041 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.033+0000 7f6f5d59a640 10 monclient: dump: 2024-04-22T08:12:39.041 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-04-22T08:12:39.041 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:39.041 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:39.041 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:39.041 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:39.041 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-04-22T08:12:39.041 INFO:teuthology.orchestra.run.smithi189.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:39.041 INFO:teuthology.orchestra.run.smithi189.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:39.041 INFO:teuthology.orchestra.run.smithi189.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:39.042 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-04-22T08:12:39.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.033+0000 7f6f64ca1640 10 monclient: shutdown 2024-04-22T08:12:39.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.033+0000 7f6f64ca1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:39.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.033+0000 7f6f64ca1640 1 -- 172.21.15.189:0/3790644515 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6f6013c1f0 msgr2=0x7f6f6013e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:39.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.033+0000 7f6f64ca1640 1 --2- 172.21.15.189:0/3790644515 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6f6013c1f0 0x7f6f6013e5e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6f58009dc0 tx=0x7f6f5802bce0 comp rx=0 tx=0).stop 2024-04-22T08:12:39.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.033+0000 7f6f64ca1640 1 -- 172.21.15.189:0/3790644515 shutdown_connections 2024-04-22T08:12:39.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.033+0000 7f6f64ca1640 1 --2- 172.21.15.189:0/3790644515 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6f6013c1f0 0x7f6f6013e5e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:39.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.033+0000 7f6f64ca1640 1 -- 172.21.15.189:0/3790644515 >> 172.21.15.189:0/3790644515 conn(0x7f6f600a5060 msgr2=0x7f6f600a5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:39.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.034+0000 7f6f64ca1640 1 -- 172.21.15.189:0/3790644515 shutdown_connections 2024-04-22T08:12:39.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.034+0000 7f6f64ca1640 1 -- 172.21.15.189:0/3790644515 wait complete. 2024-04-22T08:12:39.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.034+0000 7f6f64ca1640 10 monclient: build_initial_monmap 2024-04-22T08:12:39.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.034+0000 7f6f64ca1640 10 monclient: monmap: 2024-04-22T08:12:39.043 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-04-22T08:12:39.043 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:12:39.043 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 0.000000 2024-04-22T08:12:39.043 INFO:teuthology.orchestra.run.smithi189.stderr:created 0.000000 2024-04-22T08:12:39.043 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:39.043 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-04-22T08:12:39.043 INFO:teuthology.orchestra.run.smithi189.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:12:39.043 INFO:teuthology.orchestra.run.smithi189.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:12:39.043 INFO:teuthology.orchestra.run.smithi189.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:12:39.043 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-04-22T08:12:39.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f64ca1640 1 Processor -- start 2024-04-22T08:12:39.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f64ca1640 1 -- start start 2024-04-22T08:12:39.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f64ca1640 10 monclient: init 2024-04-22T08:12:39.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f64ca1640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:39.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f64ca1640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-22T08:12:39.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f64ca1640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6f6009ee10 0x7f6f601cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:39.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f64ca1640 10 monclient(hunting): picked mon.noname-c con 0x7f6f6009ee10 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:39.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f64ca1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6f601362c0 0x7f6f601cb340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:39.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f64ca1640 10 monclient(hunting): picked mon.noname-a con 0x7f6f601362c0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:39.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f64ca1640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f6013c1f0 0x7f6f6006dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:39.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f64ca1640 10 monclient(hunting): picked mon.noname-b con 0x7f6f6013c1f0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:39.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f64ca1640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:39.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f64ca1640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f60071400 con 0x7f6f601362c0 2024-04-22T08:12:39.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f64ca1640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:39.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f64ca1640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f600a65f0 con 0x7f6f6013c1f0 2024-04-22T08:12:39.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f64ca1640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:39.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f64ca1640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f60071650 con 0x7f6f6009ee10 2024-04-22T08:12:39.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f64ca1640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:39.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f64ca1640 10 monclient(hunting): authenticate will time out at 812.328093s 2024-04-22T08:12:39.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f5ed9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f6013c1f0 0x7f6f6006dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:39.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f5ed9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f6013c1f0 0x7f6f6006dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48594/0 (socket says 172.21.15.189:48594) 2024-04-22T08:12:39.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f5ed9d640 1 -- 172.21.15.189:0/2230307557 learned_addr learned my addr 172.21.15.189:0/2230307557 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-22T08:12:39.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f5ed9d640 10 monclient(hunting): get_auth_request con 0x7f6f6013c1f0 auth_method 0 2024-04-22T08:12:39.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f5ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:39.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f5ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:39.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f5ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:39.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f5ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:39.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f5ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:39.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f5dd9b640 1 --2- 172.21.15.189:0/2230307557 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6f601362c0 0x7f6f601cb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:39.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f5e59c640 1 --2- 172.21.15.189:0/2230307557 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6f6009ee10 0x7f6f601cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:39.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f5ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:39.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f5e59c640 10 monclient(hunting): get_auth_request con 0x7f6f6009ee10 auth_method 0 2024-04-22T08:12:39.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f5e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:39.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f5e59c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:39.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f5e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:39.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f5dd9b640 10 monclient(hunting): get_auth_request con 0x7f6f601362c0 auth_method 0 2024-04-22T08:12:39.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f5dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:39.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f5dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:39.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f5dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:39.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f5e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:39.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f5e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:39.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f5e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:39.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f5dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:39.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f5dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:39.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f5dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:39.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f5ed9d640 10 monclient(hunting): handle_auth_done global_id 4123 payload 824 2024-04-22T08:12:39.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f5ed9d640 1 -- 172.21.15.189:0/2230307557 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6f6009ee10 msgr2=0x7f6f601cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:39.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f5ed9d640 1 --2- 172.21.15.189:0/2230307557 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6f6009ee10 0x7f6f601cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:39.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f5ed9d640 1 -- 172.21.15.189:0/2230307557 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6f601362c0 msgr2=0x7f6f601cb340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:39.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f5ed9d640 1 --2- 172.21.15.189:0/2230307557 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6f601362c0 0x7f6f601cb340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:39.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f5ed9d640 10 monclient: _finish_hunting 0 2024-04-22T08:12:39.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f5ed9d640 1 monclient: found mon.noname-b 2024-04-22T08:12:39.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f5ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:39.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.035+0000 7f6f5ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.189:3300/0 2024-04-22T08:12:39.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.036+0000 7f6f5ed9d640 1 -- 172.21.15.189:0/2230307557 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f58009a70 con 0x7f6f6013c1f0 2024-04-22T08:12:39.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.038+0000 7f6f5ed9d640 1 -- 172.21.15.189:0/2230307557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f6013c1f0 msgr2=0x7f6f6006dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:39.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.038+0000 7f6f5ed9d640 1 --2- 172.21.15.189:0/2230307557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f6013c1f0 0x7f6f6006dae0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6f58009eb0 tx=0x7f6f58004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:39.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.038+0000 7f6f56ffd640 1 -- 172.21.15.189:0/2230307557 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f5803f420 con 0x7f6f6013c1f0 2024-04-22T08:12:39.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.038+0000 7f6f56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:39.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.038+0000 7f6f56ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:12:39.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.038+0000 7f6f56ffd640 10 monclient: dump: 2024-04-22T08:12:39.048 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-04-22T08:12:39.048 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:39.048 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:39.048 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:39.048 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:39.048 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-04-22T08:12:39.048 INFO:teuthology.orchestra.run.smithi189.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:39.048 INFO:teuthology.orchestra.run.smithi189.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:39.048 INFO:teuthology.orchestra.run.smithi189.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:39.048 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-04-22T08:12:39.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.038+0000 7f6f56ffd640 10 monclient: _finish_auth 0 2024-04-22T08:12:39.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.038+0000 7f6f56ffd640 10 monclient: _check_auth_tickets 2024-04-22T08:12:39.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.038+0000 7f6f56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:39.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.038+0000 7f6f56ffd640 1 -- 172.21.15.189:0/2230307557 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f5803f580 con 0x7f6f6013c1f0 2024-04-22T08:12:39.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.038+0000 7f6f56ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:39.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.038+0000 7f6f56ffd640 1 -- 172.21.15.189:0/2230307557 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f5803f910 con 0x7f6f6013c1f0 2024-04-22T08:12:39.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.038+0000 7f6f64ca1640 5 monclient: authenticate success, global_id 4123 2024-04-22T08:12:39.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.038+0000 7f6f64ca1640 10 monclient: _renew_subs 2024-04-22T08:12:39.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.038+0000 7f6f64ca1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.189:3300/0 2024-04-22T08:12:39.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.038+0000 7f6f64ca1640 1 -- 172.21.15.189:0/2230307557 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f60070130 con 0x7f6f6013c1f0 2024-04-22T08:12:39.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.038+0000 7f6f5cd99640 4 set_mon_vals no callback set 2024-04-22T08:12:39.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.038+0000 7f6f56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:39.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.038+0000 7f6f56ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:12:39.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.038+0000 7f6f56ffd640 10 monclient: dump: 2024-04-22T08:12:39.049 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-04-22T08:12:39.050 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:39.050 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:39.050 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:39.050 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:39.050 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-04-22T08:12:39.050 INFO:teuthology.orchestra.run.smithi189.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:39.050 INFO:teuthology.orchestra.run.smithi189.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:39.050 INFO:teuthology.orchestra.run.smithi189.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:39.050 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-04-22T08:12:39.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.038+0000 7f6f64ca1640 10 monclient: _renew_subs 2024-04-22T08:12:39.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.038+0000 7f6f64ca1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.189:3300/0 2024-04-22T08:12:39.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.038+0000 7f6f64ca1640 1 -- 172.21.15.189:0/2230307557 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f60071930 con 0x7f6f6013c1f0 2024-04-22T08:12:39.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.038+0000 7f6f56ffd640 1 -- 172.21.15.189:0/2230307557 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f6f58048410 con 0x7f6f6013c1f0 2024-04-22T08:12:39.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.038+0000 7f6f56ffd640 1 -- 172.21.15.189:0/2230307557 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f6f58048b50 con 0x7f6f6013c1f0 2024-04-22T08:12:39.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.039+0000 7f6f64ca1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:39.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.039+0000 7f6f64ca1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:39.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.039+0000 7f6f64ca1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.189:3300/0 2024-04-22T08:12:39.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.039+0000 7f6f64ca1640 1 -- 172.21.15.189:0/2230307557 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f60071790 con 0x7f6f6013c1f0 2024-04-22T08:12:39.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.043+0000 7f6f56ffd640 1 -- 172.21.15.189:0/2230307557 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (secure 0 0 0) 0x7f6f58067dc0 con 0x7f6f6013c1f0 2024-04-22T08:12:39.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.043+0000 7f6f56ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:39.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.043+0000 7f6f56ffd640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:12:39.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.115+0000 7f6f64ca1640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d94b2052-5847-4955-b1d4-b7e627572811", "id": 3}] 2024-04-22T08:12:39.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.115+0000 7f6f64ca1640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d94b2052-5847-4955-b1d4-b7e627572811", "id": 3}] 2024-04-22T08:12:39.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.115+0000 7f6f64ca1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.189:3300/0 2024-04-22T08:12:39.115 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.115+0000 7f6f64ca1640 1 -- 172.21.15.189:0/2230307557 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d94b2052-5847-4955-b1d4-b7e627572811", "id": 3} v 0) v1 -- 0x7f6f6004e9b0 con 0x7f6f6013c1f0 2024-04-22T08:12:39.190 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-04-22T08:12:39.191+0000 7f0813e38040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T08:12:39.323 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-04-22T08:12:39.324+0000 7f0813e38040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T08:12:39.372 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-04-22T08:12:39.372+0000 7f0813e38040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T08:12:39.440 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-04-22T08:12:39.440+0000 7f0813e38040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T08:12:39.484 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-04-22T08:12:39.484+0000 7f0813e38040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T08:12:39.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.928+0000 7f6f56ffd640 1 -- 172.21.15.189:0/2230307557 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d94b2052-5847-4955-b1d4-b7e627572811", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f6f58056070 con 0x7f6f6013c1f0 2024-04-22T08:12:39.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.928+0000 7f6f56ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d94b2052-5847-4955-b1d4-b7e627572811", "id": 3}] 2024-04-22T08:12:39.928 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.928+0000 7f6f56ffd640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:12:39.928 INFO:teuthology.orchestra.run.smithi189.stdout:3 2024-04-22T08:12:39.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.929+0000 7f6f64ca1640 10 monclient: shutdown 2024-04-22T08:12:39.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.929+0000 7f6f64ca1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:39.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.929+0000 7f6f64ca1640 1 -- 172.21.15.189:0/2230307557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f6013c1f0 msgr2=0x7f6f6006dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:39.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.929+0000 7f6f64ca1640 1 --2- 172.21.15.189:0/2230307557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f6013c1f0 0x7f6f6006dae0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6f58009eb0 tx=0x7f6f58004a40 comp rx=0 tx=0).stop 2024-04-22T08:12:39.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.929+0000 7f6f64ca1640 1 -- 172.21.15.189:0/2230307557 shutdown_connections 2024-04-22T08:12:39.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.929+0000 7f6f64ca1640 1 --2- 172.21.15.189:0/2230307557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f6013c1f0 0x7f6f6006dae0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:39.929 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.929+0000 7f6f64ca1640 1 --2- 172.21.15.189:0/2230307557 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6f601362c0 0x7f6f601cb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:39.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.929+0000 7f6f64ca1640 1 --2- 172.21.15.189:0/2230307557 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6f6009ee10 0x7f6f601cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:39.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.929+0000 7f6f64ca1640 1 -- 172.21.15.189:0/2230307557 >> 172.21.15.189:0/2230307557 conn(0x7f6f600a5060 msgr2=0x7f6f600a0770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:39.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.929+0000 7f6f64ca1640 1 -- 172.21.15.189:0/2230307557 shutdown_connections 2024-04-22T08:12:39.930 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:39.929+0000 7f6f64ca1640 1 -- 172.21.15.189:0/2230307557 wait complete. 2024-04-22T08:12:39.940 DEBUG:teuthology.orchestra.run.smithi189:> sudo ceph --cluster ceph osd new cb626202-7411-4299-8d20-988bcc9103f3 4 2024-04-22T08:12:39.945 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-04-22T08:12:39.946+0000 7f0813e38040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T08:12:40.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.037+0000 7f5cd43f5640 10 monclient: get_monmap_and_config 2024-04-22T08:12:40.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.037+0000 7f5cd43f5640 10 monclient: build_initial_monmap 2024-04-22T08:12:40.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.037+0000 7f5cd43f5640 10 monclient: monmap: 2024-04-22T08:12:40.037 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-04-22T08:12:40.037 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:40.037 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-04-22T08:12:40.038134+0000 2024-04-22T08:12:40.037 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-04-22T08:12:40.038134+0000 2024-04-22T08:12:40.037 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:40.038 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-04-22T08:12:40.038 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:12:40.038 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:12:40.038 INFO:teuthology.orchestra.run.smithi189.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:12:40.038 INFO:teuthology.orchestra.run.smithi189.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:12:40.038 INFO:teuthology.orchestra.run.smithi189.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:12:40.038 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-04-22T08:12:40.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.037+0000 7f5cd43f5640 1 Processor -- start 2024-04-22T08:12:40.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.037+0000 7f5cd43f5640 1 -- start start 2024-04-22T08:12:40.038 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.037+0000 7f5cd43f5640 10 monclient: init 2024-04-22T08:12:40.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.038+0000 7f5cd43f5640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:40.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.038+0000 7f5cd43f5640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-04-22T08:12:40.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.038+0000 7f5cd43f5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5ccc136200 addr v1:172.21.15.40:6789/0 2024-04-22T08:12:40.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.038+0000 7f5cd43f5640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5ccc0a0370 0x7f5ccc1365e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:40.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.038+0000 7f5cd43f5640 10 monclient(hunting): picked mon.noname-a con 0x7f5ccc0a0370 addr v2:172.21.15.40:3300/0 2024-04-22T08:12:40.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.038+0000 7f5cd43f5640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5ccc09e510 0x7f5ccc09e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:40.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.038+0000 7f5cd43f5640 10 monclient(hunting): picked mon.noname-c con 0x7f5ccc09e510 addr v2:172.21.15.189:3300/0 2024-04-22T08:12:40.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.038+0000 7f5cd43f5640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ccc0a4bc0 con 0x7f5ccc136200 2024-04-22T08:12:40.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.038+0000 7f5cd43f5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:40.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.038+0000 7f5cd43f5640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ccc0a3580 con 0x7f5ccc0a0370 2024-04-22T08:12:40.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.038+0000 7f5cd43f5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:40.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.038+0000 7f5cd43f5640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ccc136b20 con 0x7f5ccc09e510 2024-04-22T08:12:40.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.038+0000 7f5cd43f5640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:40.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.038+0000 7f5cd43f5640 10 monclient(hunting): authenticate will time out at 813.330926s 2024-04-22T08:12:40.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.038+0000 7f5cd2992640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5ccc09e510 0x7f5ccc09e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:40.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.038+0000 7f5cd2992640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5ccc09e510 0x7f5ccc09e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40508/0 (socket says 172.21.15.189:40508) 2024-04-22T08:12:40.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.038+0000 7f5cd2992640 1 -- 172.21.15.189:0/522381527 learned_addr learned my addr 172.21.15.189:0/522381527 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-22T08:12:40.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.038+0000 7f5cd2992640 10 monclient(hunting): get_auth_request con 0x7f5ccc09e510 auth_method 0 2024-04-22T08:12:40.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.038+0000 7f5cd2992640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:40.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.038+0000 7f5cd2992640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:40.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.038+0000 7f5cd2992640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:40.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.038+0000 7f5cd2992640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:40.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.038+0000 7f5cd2992640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:40.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.038+0000 7f5cd2992640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:40.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.038+0000 7f5cd1990640 1 --2- 172.21.15.189:0/522381527 >> v2:172.21.15.40:3300/0 conn(0x7f5ccc0a0370 0x7f5ccc1365e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:40.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.038+0000 7f5cd1990640 10 monclient(hunting): get_auth_request con 0x7f5ccc0a0370 auth_method 0 2024-04-22T08:12:40.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.038+0000 7f5cd1990640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:40.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.038+0000 7f5cd1990640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:40.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.038+0000 7f5cd1990640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:40.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.038+0000 7f5cd2992640 10 monclient(hunting): handle_auth_done global_id 4126 payload 274 2024-04-22T08:12:40.049 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.038+0000 7f5cd2992640 1 -- 172.21.15.189:0/522381527 >> v2:172.21.15.40:3300/0 conn(0x7f5ccc0a0370 msgr2=0x7f5ccc1365e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:40.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.038+0000 7f5cd2992640 1 --2- 172.21.15.189:0/522381527 >> v2:172.21.15.40:3300/0 conn(0x7f5ccc0a0370 0x7f5ccc1365e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:40.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.038+0000 7f5cd2992640 1 -- 172.21.15.189:0/522381527 >> v1:172.21.15.40:6789/0 conn(0x7f5ccc136200 legacy=0x7f5ccc09de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:40.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.038+0000 7f5cd2992640 10 monclient: _finish_hunting 0 2024-04-22T08:12:40.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.038+0000 7f5cd2992640 1 monclient: found mon.noname-c 2024-04-22T08:12:40.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.038+0000 7f5cd2992640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:40.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.038+0000 7f5cd2992640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.189:3300/0 2024-04-22T08:12:40.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.038+0000 7f5cd2992640 1 -- 172.21.15.189:0/522381527 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ccc09f990 con 0x7f5ccc09e510 2024-04-22T08:12:40.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.041+0000 7f5cd2992640 1 -- 172.21.15.189:0/522381527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ccc09e510 msgr2=0x7f5ccc09e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:40.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.041+0000 7f5cd2992640 1 --2- 172.21.15.189:0/522381527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ccc09e510 0x7f5ccc09e980 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4009e50 tx=0x7f5cc402bdc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0515a8698d9c06d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:40.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.041+0000 7f5cd118f640 1 -- 172.21.15.189:0/522381527 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc402cb70 con 0x7f5ccc09e510 2024-04-22T08:12:40.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.041+0000 7f5cd118f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:40.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.041+0000 7f5cd118f640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:12:40.050 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.041+0000 7f5cd118f640 10 monclient: dump: 2024-04-22T08:12:40.050 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-04-22T08:12:40.051 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:40.051 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:40.051 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:40.051 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:40.051 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-04-22T08:12:40.051 INFO:teuthology.orchestra.run.smithi189.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:40.051 INFO:teuthology.orchestra.run.smithi189.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:40.051 INFO:teuthology.orchestra.run.smithi189.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:40.051 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-04-22T08:12:40.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.041+0000 7f5cd118f640 10 monclient: _finish_auth 0 2024-04-22T08:12:40.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.041+0000 7f5cd118f640 10 monclient: _check_auth_tickets 2024-04-22T08:12:40.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.041+0000 7f5cd118f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:40.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.041+0000 7f5cd118f640 1 -- 172.21.15.189:0/522381527 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cc402ccd0 con 0x7f5ccc09e510 2024-04-22T08:12:40.051 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.041+0000 7f5cd43f5640 5 monclient: authenticate success, global_id 4126 2024-04-22T08:12:40.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.041+0000 7f5cd43f5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:12:40.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.041+0000 7f5cd118f640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:40.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.041+0000 7f5cd43f5640 10 monclient: get_monmap_and_config success 2024-04-22T08:12:40.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.041+0000 7f5cd118f640 1 -- 172.21.15.189:0/522381527 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc4034610 con 0x7f5ccc09e510 2024-04-22T08:12:40.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.041+0000 7f5cd118f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:40.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.041+0000 7f5cd43f5640 4 set_mon_vals no callback set 2024-04-22T08:12:40.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.041+0000 7f5cd118f640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:12:40.052 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.041+0000 7f5cd118f640 10 monclient: dump: 2024-04-22T08:12:40.052 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-04-22T08:12:40.052 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:40.052 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:40.052 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:40.052 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:40.052 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-04-22T08:12:40.053 INFO:teuthology.orchestra.run.smithi189.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:40.053 INFO:teuthology.orchestra.run.smithi189.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:40.053 INFO:teuthology.orchestra.run.smithi189.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:40.053 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-04-22T08:12:40.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.042+0000 7f5cd43f5640 10 monclient: shutdown 2024-04-22T08:12:40.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.042+0000 7f5cd43f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:40.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.042+0000 7f5cd43f5640 1 -- 172.21.15.189:0/522381527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ccc09e510 msgr2=0x7f5ccc09e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:40.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.042+0000 7f5cd43f5640 1 --2- 172.21.15.189:0/522381527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ccc09e510 0x7f5ccc09e980 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4009e50 tx=0x7f5cc402bdc0 comp rx=0 tx=0).stop 2024-04-22T08:12:40.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.042+0000 7f5cd43f5640 1 -- 172.21.15.189:0/522381527 shutdown_connections 2024-04-22T08:12:40.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.042+0000 7f5cd43f5640 1 --2- 172.21.15.189:0/522381527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ccc09e510 0x7f5ccc09e980 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:40.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.042+0000 7f5cd43f5640 1 --2- 172.21.15.189:0/522381527 >> v2:172.21.15.40:3300/0 conn(0x7f5ccc0a0370 0x7f5ccc1365e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:40.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.042+0000 7f5cd43f5640 1 -- 172.21.15.189:0/522381527 >> 172.21.15.189:0/522381527 conn(0x7f5ccc0a85d0 msgr2=0x7f5ccc0a89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:40.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.042+0000 7f5cd43f5640 1 -- 172.21.15.189:0/522381527 shutdown_connections 2024-04-22T08:12:40.053 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.042+0000 7f5cd43f5640 1 -- 172.21.15.189:0/522381527 wait complete. 2024-04-22T08:12:40.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.043+0000 7f5cd43f5640 10 monclient: build_initial_monmap 2024-04-22T08:12:40.054 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.043+0000 7f5cd43f5640 10 monclient: monmap: 2024-04-22T08:12:40.054 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-04-22T08:12:40.054 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:12:40.054 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 0.000000 2024-04-22T08:12:40.054 INFO:teuthology.orchestra.run.smithi189.stderr:created 0.000000 2024-04-22T08:12:40.054 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:40.054 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-04-22T08:12:40.054 INFO:teuthology.orchestra.run.smithi189.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:12:40.054 INFO:teuthology.orchestra.run.smithi189.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:12:40.054 INFO:teuthology.orchestra.run.smithi189.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:12:40.054 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-04-22T08:12:40.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.043+0000 7f5cd43f5640 1 Processor -- start 2024-04-22T08:12:40.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.043+0000 7f5cd43f5640 1 -- start start 2024-04-22T08:12:40.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.043+0000 7f5cd43f5640 10 monclient: init 2024-04-22T08:12:40.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.043+0000 7f5cd43f5640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:40.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.043+0000 7f5cd43f5640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-22T08:12:40.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.043+0000 7f5cd43f5640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ccc09e510 0x7f5ccc1cacd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:40.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.043+0000 7f5cd43f5640 10 monclient(hunting): picked mon.noname-c con 0x7f5ccc09e510 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:40.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.043+0000 7f5cd43f5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ccc0a0370 0x7f5ccc1cb210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:40.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.043+0000 7f5cd43f5640 10 monclient(hunting): picked mon.noname-b con 0x7f5ccc0a0370 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:40.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.043+0000 7f5cd43f5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ccc136200 0x7f5ccc1222e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:40.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.043+0000 7f5cd43f5640 10 monclient(hunting): picked mon.noname-a con 0x7f5ccc136200 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:40.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.043+0000 7f5cd43f5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:40.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.043+0000 7f5cd43f5640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ccc125850 con 0x7f5ccc136200 2024-04-22T08:12:40.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.043+0000 7f5cd43f5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:40.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.043+0000 7f5cd43f5640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ccc124960 con 0x7f5ccc0a0370 2024-04-22T08:12:40.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.043+0000 7f5cd43f5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:40.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.043+0000 7f5cd43f5640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ccc0a1d60 con 0x7f5ccc09e510 2024-04-22T08:12:40.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.043+0000 7f5cd43f5640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:40.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.043+0000 7f5cd43f5640 10 monclient(hunting): authenticate will time out at 813.336516s 2024-04-22T08:12:40.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.044+0000 7f5cd1990640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ccc0a0370 0x7f5ccc1cb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:40.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.044+0000 7f5cd1990640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ccc0a0370 0x7f5ccc1cb210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40520/0 (socket says 172.21.15.189:40520) 2024-04-22T08:12:40.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.044+0000 7f5cd1990640 1 -- 172.21.15.189:0/3138580061 learned_addr learned my addr 172.21.15.189:0/3138580061 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-22T08:12:40.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.044+0000 7f5cd1990640 10 monclient(hunting): get_auth_request con 0x7f5ccc0a0370 auth_method 0 2024-04-22T08:12:40.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.044+0000 7f5cd1990640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:40.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.044+0000 7f5cd1990640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:40.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.044+0000 7f5cd1990640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:40.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.044+0000 7f5cd2191640 1 --2- 172.21.15.189:0/3138580061 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ccc09e510 0x7f5ccc1cacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:40.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.044+0000 7f5cd2191640 10 monclient(hunting): get_auth_request con 0x7f5ccc09e510 auth_method 0 2024-04-22T08:12:40.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.044+0000 7f5cd2191640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:40.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.044+0000 7f5cd2191640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:40.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.044+0000 7f5cd2191640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:40.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.044+0000 7f5cd1990640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:40.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.044+0000 7f5cd1990640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:40.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.044+0000 7f5cd2992640 1 --2- 172.21.15.189:0/3138580061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ccc136200 0x7f5ccc1222e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:40.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.044+0000 7f5cd1990640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:40.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.044+0000 7f5cd2992640 10 monclient(hunting): get_auth_request con 0x7f5ccc136200 auth_method 0 2024-04-22T08:12:40.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.044+0000 7f5cd2992640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:40.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.044+0000 7f5cd2992640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:40.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.044+0000 7f5cd2992640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:40.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.044+0000 7f5cd2191640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:40.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.044+0000 7f5cd2191640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:40.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.044+0000 7f5cd2191640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:40.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.044+0000 7f5cd2992640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:40.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.044+0000 7f5cd2992640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:40.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.044+0000 7f5cd2992640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:40.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.044+0000 7f5cd1990640 10 monclient(hunting): handle_auth_done global_id 4129 payload 824 2024-04-22T08:12:40.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.044+0000 7f5cd1990640 1 -- 172.21.15.189:0/3138580061 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ccc09e510 msgr2=0x7f5ccc1cacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:40.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.044+0000 7f5cd1990640 1 --2- 172.21.15.189:0/3138580061 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ccc09e510 0x7f5ccc1cacd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:40.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.044+0000 7f5cd1990640 1 -- 172.21.15.189:0/3138580061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ccc136200 msgr2=0x7f5ccc1222e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:40.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.044+0000 7f5cd1990640 1 --2- 172.21.15.189:0/3138580061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ccc136200 0x7f5ccc1222e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:40.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.044+0000 7f5cd1990640 10 monclient: _finish_hunting 0 2024-04-22T08:12:40.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.044+0000 7f5cd1990640 1 monclient: found mon.noname-b 2024-04-22T08:12:40.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.044+0000 7f5cd1990640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:40.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.044+0000 7f5cd1990640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.189:3300/0 2024-04-22T08:12:40.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.044+0000 7f5cd1990640 1 -- 172.21.15.189:0/3138580061 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cc4009b30 con 0x7f5ccc0a0370 2024-04-22T08:12:40.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.046+0000 7f5cd1990640 1 -- 172.21.15.189:0/3138580061 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ccc0a0370 msgr2=0x7f5ccc1cb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:40.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.046+0000 7f5cd1990640 1 --2- 172.21.15.189:0/3138580061 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ccc0a0370 0x7f5ccc1cb210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc00d800 tx=0x7f5cbc00dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:40.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.046+0000 7f5cc2ffd640 1 -- 172.21.15.189:0/3138580061 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc00b8d0 con 0x7f5ccc0a0370 2024-04-22T08:12:40.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.046+0000 7f5cc2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:40.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.046+0000 7f5cc2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:12:40.059 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.046+0000 7f5cc2ffd640 10 monclient: dump: 2024-04-22T08:12:40.059 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-04-22T08:12:40.059 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:40.059 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:40.059 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:40.059 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:40.059 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-04-22T08:12:40.059 INFO:teuthology.orchestra.run.smithi189.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:40.060 INFO:teuthology.orchestra.run.smithi189.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:40.060 INFO:teuthology.orchestra.run.smithi189.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:40.060 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-04-22T08:12:40.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.046+0000 7f5cc2ffd640 10 monclient: _finish_auth 0 2024-04-22T08:12:40.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.046+0000 7f5cc2ffd640 10 monclient: _check_auth_tickets 2024-04-22T08:12:40.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.046+0000 7f5cc2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:40.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.046+0000 7f5cc2ffd640 1 -- 172.21.15.189:0/3138580061 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cbc00ba30 con 0x7f5ccc0a0370 2024-04-22T08:12:40.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.046+0000 7f5cd43f5640 5 monclient: authenticate success, global_id 4129 2024-04-22T08:12:40.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.046+0000 7f5cd43f5640 10 monclient: _renew_subs 2024-04-22T08:12:40.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.046+0000 7f5cd43f5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.189:3300/0 2024-04-22T08:12:40.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.046+0000 7f5cd43f5640 1 -- 172.21.15.189:0/3138580061 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ccc071640 con 0x7f5ccc0a0370 2024-04-22T08:12:40.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.046+0000 7f5cc2ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:40.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.046+0000 7f5cc2ffd640 1 -- 172.21.15.189:0/3138580061 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc00bdc0 con 0x7f5ccc0a0370 2024-04-22T08:12:40.060 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.046+0000 7f5cc2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:40.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.046+0000 7f5cc2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:12:40.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.046+0000 7f5cd098e640 4 set_mon_vals no callback set 2024-04-22T08:12:40.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.046+0000 7f5cc2ffd640 10 monclient: dump: 2024-04-22T08:12:40.061 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-04-22T08:12:40.061 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:40.061 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:40.061 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:40.061 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:40.061 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-04-22T08:12:40.061 INFO:teuthology.orchestra.run.smithi189.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:40.061 INFO:teuthology.orchestra.run.smithi189.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:40.061 INFO:teuthology.orchestra.run.smithi189.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:40.061 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-04-22T08:12:40.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.046+0000 7f5cd43f5640 10 monclient: _renew_subs 2024-04-22T08:12:40.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.046+0000 7f5cd43f5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.189:3300/0 2024-04-22T08:12:40.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.046+0000 7f5cd43f5640 1 -- 172.21.15.189:0/3138580061 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ccc123d20 con 0x7f5ccc0a0370 2024-04-22T08:12:40.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.046+0000 7f5cc2ffd640 1 -- 172.21.15.189:0/3138580061 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f5cbc01e760 con 0x7f5ccc0a0370 2024-04-22T08:12:40.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.047+0000 7f5cc2ffd640 1 -- 172.21.15.189:0/3138580061 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f5cbc014d00 con 0x7f5ccc0a0370 2024-04-22T08:12:40.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.047+0000 7f5cd43f5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:40.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.047+0000 7f5cd43f5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:40.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.047+0000 7f5cd43f5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.189:3300/0 2024-04-22T08:12:40.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.047+0000 7f5cd43f5640 1 -- 172.21.15.189:0/3138580061 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ccc04e9b0 con 0x7f5ccc0a0370 2024-04-22T08:12:40.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.049+0000 7f5cc2ffd640 1 -- 172.21.15.189:0/3138580061 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (secure 0 0 0) 0x7f5cbc0147f0 con 0x7f5ccc0a0370 2024-04-22T08:12:40.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.049+0000 7f5cc2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:40.063 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.049+0000 7f5cc2ffd640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:12:40.067 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-04-22T08:12:40.068+0000 7f0813e38040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T08:12:40.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.073+0000 7f5cc2ffd640 1 -- 172.21.15.189:0/3138580061 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 2) v1 ==== 44528+0+0 (secure 0 0 0) 0x7f5cbc014990 con 0x7f5ccc0a0370 2024-04-22T08:12:40.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.128+0000 7f5cd43f5640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "cb626202-7411-4299-8d20-988bcc9103f3", "id": 4}] 2024-04-22T08:12:40.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.128+0000 7f5cd43f5640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "cb626202-7411-4299-8d20-988bcc9103f3", "id": 4}] 2024-04-22T08:12:40.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.128+0000 7f5cd43f5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.189:3300/0 2024-04-22T08:12:40.128 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.128+0000 7f5cd43f5640 1 -- 172.21.15.189:0/3138580061 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd new", "uuid": "cb626202-7411-4299-8d20-988bcc9103f3", "id": 4} v 0) v1 -- 0x7f5ccc06a230 con 0x7f5ccc0a0370 2024-04-22T08:12:40.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.134+0000 7f5cc2ffd640 1 -- 172.21.15.189:0/3138580061 <== mon.1 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cb626202-7411-4299-8d20-988bcc9103f3", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5cbc0273d0 con 0x7f5ccc0a0370 2024-04-22T08:12:40.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.134+0000 7f5cc2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "cb626202-7411-4299-8d20-988bcc9103f3", "id": 4}] 2024-04-22T08:12:40.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.134+0000 7f5cc2ffd640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:12:40.134 INFO:teuthology.orchestra.run.smithi189.stdout:4 2024-04-22T08:12:40.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.135+0000 7f5cd43f5640 10 monclient: shutdown 2024-04-22T08:12:40.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.135+0000 7f5cd43f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:40.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.135+0000 7f5cd43f5640 1 -- 172.21.15.189:0/3138580061 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ccc0a0370 msgr2=0x7f5ccc1cb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:40.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.135+0000 7f5cd43f5640 1 --2- 172.21.15.189:0/3138580061 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ccc0a0370 0x7f5ccc1cb210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc00d800 tx=0x7f5cbc00dcc0 comp rx=0 tx=0).stop 2024-04-22T08:12:40.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.135+0000 7f5cd43f5640 1 -- 172.21.15.189:0/3138580061 shutdown_connections 2024-04-22T08:12:40.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.135+0000 7f5cd43f5640 1 --2- 172.21.15.189:0/3138580061 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ccc0a0370 0x7f5ccc1cb210 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:40.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.135+0000 7f5cd43f5640 1 --2- 172.21.15.189:0/3138580061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ccc136200 0x7f5ccc1222e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:40.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.135+0000 7f5cd43f5640 1 --2- 172.21.15.189:0/3138580061 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ccc09e510 0x7f5ccc1cacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:40.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.135+0000 7f5cd43f5640 1 -- 172.21.15.189:0/3138580061 >> 172.21.15.189:0/3138580061 conn(0x7f5ccc0a85d0 msgr2=0x7f5ccc06ef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:40.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.135+0000 7f5cd43f5640 1 -- 172.21.15.189:0/3138580061 shutdown_connections 2024-04-22T08:12:40.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.135+0000 7f5cd43f5640 1 -- 172.21.15.189:0/3138580061 wait complete. 2024-04-22T08:12:40.145 DEBUG:teuthology.orchestra.run.smithi189:> sudo ceph --cluster ceph osd new abbac246-54d6-47d7-a41c-a68df0b80ddb 5 2024-04-22T08:12:40.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.279+0000 7f6827db8640 10 monclient: get_monmap_and_config 2024-04-22T08:12:40.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.279+0000 7f6827db8640 10 monclient: build_initial_monmap 2024-04-22T08:12:40.280 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.279+0000 7f6827db8640 10 monclient: monmap: 2024-04-22T08:12:40.280 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-04-22T08:12:40.280 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:40.280 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-04-22T08:12:40.280550+0000 2024-04-22T08:12:40.280 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-04-22T08:12:40.280550+0000 2024-04-22T08:12:40.280 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:40.280 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-04-22T08:12:40.280 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:12:40.281 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:12:40.281 INFO:teuthology.orchestra.run.smithi189.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:12:40.281 INFO:teuthology.orchestra.run.smithi189.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:12:40.281 INFO:teuthology.orchestra.run.smithi189.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:12:40.281 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-04-22T08:12:40.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.280+0000 7f6827db8640 1 Processor -- start 2024-04-22T08:12:40.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.280+0000 7f6827db8640 1 -- start start 2024-04-22T08:12:40.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.280+0000 7f6827db8640 10 monclient: init 2024-04-22T08:12:40.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.280+0000 7f6827db8640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:40.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.280+0000 7f6827db8640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-04-22T08:12:40.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.280+0000 7f6827db8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6820098cd0 addr v1:172.21.15.189:6789/0 2024-04-22T08:12:40.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.280+0000 7f6827db8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f68200997c0 addr v1:172.21.15.40:6789/0 2024-04-22T08:12:40.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.280+0000 7f6827db8640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f68201392f0 0x7f682013b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:40.286 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.280+0000 7f6827db8640 10 monclient(hunting): picked mon.noname-a con 0x7f68201392f0 addr v2:172.21.15.40:3300/0 2024-04-22T08:12:40.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.280+0000 7f6827db8640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f682009e110 con 0x7f68200997c0 2024-04-22T08:12:40.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.280+0000 7f6827db8640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6820071c30 con 0x7f6820098cd0 2024-04-22T08:12:40.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.280+0000 7f6827db8640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:40.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.280+0000 7f6827db8640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f682009a360 con 0x7f68201392f0 2024-04-22T08:12:40.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.280+0000 7f6827db8640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:40.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.280+0000 7f6827db8640 10 monclient(hunting): authenticate will time out at 813.573392s 2024-04-22T08:12:40.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.280+0000 7f6825b54640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f6820098cd0 0x7f68200990b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33118/0 (socket says 172.21.15.189:33118) 2024-04-22T08:12:40.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.280+0000 7f6825b54640 1 -- 172.21.15.189:0/4047902487 learned_addr learned my addr 172.21.15.189:0/4047902487 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-04-22T08:12:40.287 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.281+0000 7f6826355640 1 --2- 172.21.15.189:0/4047902487 >> v2:172.21.15.40:3300/0 conn(0x7f68201392f0 0x7f682013b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:40.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.281+0000 7f6826355640 10 monclient(hunting): get_auth_request con 0x7f68201392f0 auth_method 0 2024-04-22T08:12:40.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.281+0000 7f6826355640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:40.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.281+0000 7f6826355640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:40.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.281+0000 7f6826355640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:40.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.281+0000 7f6826355640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:40.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.281+0000 7f6826355640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:40.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.281+0000 7f6826355640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:40.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.281+0000 7f6826355640 10 monclient(hunting): handle_auth_done global_id 4146 payload 274 2024-04-22T08:12:40.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.281+0000 7f6826355640 1 -- 172.21.15.189:0/4047902487 >> v1:172.21.15.189:6789/0 conn(0x7f6820098cd0 legacy=0x7f68200990b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:40.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.281+0000 7f6826355640 1 -- 172.21.15.189:0/4047902487 >> v1:172.21.15.40:6789/0 conn(0x7f68200997c0 legacy=0x7f6820099c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:40.288 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.281+0000 7f6826355640 10 monclient: _finish_hunting 0 2024-04-22T08:12:40.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.281+0000 7f6826355640 1 monclient: found mon.noname-a 2024-04-22T08:12:40.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.281+0000 7f6826355640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:40.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.281+0000 7f6826355640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:12:40.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.281+0000 7f6826355640 1 -- 172.21.15.189:0/4047902487 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68200a0950 con 0x7f68201392f0 2024-04-22T08:12:40.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.284+0000 7f6826355640 1 -- 172.21.15.189:0/4047902487 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68201392f0 msgr2=0x7f682013b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:40.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.284+0000 7f6826355640 1 --2- 172.21.15.189:0/4047902487 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68201392f0 0x7f682013b6e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6818009d90 tx=0x7f681802bd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bd698e4dc47cc97 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:40.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.285+0000 7f6824b52640 1 -- 172.21.15.189:0/4047902487 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681802cda0 con 0x7f68201392f0 2024-04-22T08:12:40.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.285+0000 7f6824b52640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:40.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.285+0000 7f6824b52640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:12:40.289 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.285+0000 7f6824b52640 10 monclient: dump: 2024-04-22T08:12:40.290 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-04-22T08:12:40.290 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:40.290 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:40.290 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:40.290 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:40.290 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-04-22T08:12:40.290 INFO:teuthology.orchestra.run.smithi189.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:40.290 INFO:teuthology.orchestra.run.smithi189.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:40.290 INFO:teuthology.orchestra.run.smithi189.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:40.290 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-04-22T08:12:40.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.285+0000 7f6824b52640 10 monclient: _finish_auth 0 2024-04-22T08:12:40.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.285+0000 7f6824b52640 10 monclient: _check_auth_tickets 2024-04-22T08:12:40.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.285+0000 7f6824b52640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:40.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.285+0000 7f6824b52640 1 -- 172.21.15.189:0/4047902487 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6818034440 con 0x7f68201392f0 2024-04-22T08:12:40.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.285+0000 7f6824b52640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:40.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.285+0000 7f6824b52640 1 -- 172.21.15.189:0/4047902487 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68180347d0 con 0x7f68201392f0 2024-04-22T08:12:40.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.285+0000 7f6827db8640 5 monclient: authenticate success, global_id 4146 2024-04-22T08:12:40.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.285+0000 7f6827db8640 10 monclient: get_monmap_and_config success 2024-04-22T08:12:40.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.285+0000 7f6827db8640 4 set_mon_vals no callback set 2024-04-22T08:12:40.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.285+0000 7f6824b52640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:40.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.285+0000 7f6824b52640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:12:40.291 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.285+0000 7f6824b52640 10 monclient: dump: 2024-04-22T08:12:40.291 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-04-22T08:12:40.292 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:40.292 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:40.292 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:40.292 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:40.292 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-04-22T08:12:40.292 INFO:teuthology.orchestra.run.smithi189.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:40.292 INFO:teuthology.orchestra.run.smithi189.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:40.292 INFO:teuthology.orchestra.run.smithi189.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:40.292 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-04-22T08:12:40.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.285+0000 7f6827db8640 10 monclient: shutdown 2024-04-22T08:12:40.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.285+0000 7f6827db8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:40.292 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.285+0000 7f6827db8640 1 -- 172.21.15.189:0/4047902487 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68201392f0 msgr2=0x7f682013b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:40.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.285+0000 7f6827db8640 1 --2- 172.21.15.189:0/4047902487 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68201392f0 0x7f682013b6e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6818009d90 tx=0x7f681802bd00 comp rx=0 tx=0).stop 2024-04-22T08:12:40.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.285+0000 7f6827db8640 1 -- 172.21.15.189:0/4047902487 shutdown_connections 2024-04-22T08:12:40.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.285+0000 7f6827db8640 1 --2- 172.21.15.189:0/4047902487 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68201392f0 0x7f682013b6e0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:40.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.285+0000 7f6827db8640 1 -- 172.21.15.189:0/4047902487 >> 172.21.15.189:0/4047902487 conn(0x7f68200a1280 msgr2=0x7f682009eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:40.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.285+0000 7f6827db8640 1 -- 172.21.15.189:0/4047902487 shutdown_connections 2024-04-22T08:12:40.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.285+0000 7f6827db8640 1 -- 172.21.15.189:0/4047902487 wait complete. 2024-04-22T08:12:40.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.286+0000 7f6827db8640 10 monclient: build_initial_monmap 2024-04-22T08:12:40.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.286+0000 7f6827db8640 10 monclient: monmap: 2024-04-22T08:12:40.293 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-04-22T08:12:40.293 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:12:40.293 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 0.000000 2024-04-22T08:12:40.293 INFO:teuthology.orchestra.run.smithi189.stderr:created 0.000000 2024-04-22T08:12:40.294 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:40.294 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-04-22T08:12:40.294 INFO:teuthology.orchestra.run.smithi189.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:12:40.294 INFO:teuthology.orchestra.run.smithi189.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:12:40.294 INFO:teuthology.orchestra.run.smithi189.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:12:40.294 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-04-22T08:12:40.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.286+0000 7f6827db8640 1 Processor -- start 2024-04-22T08:12:40.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.286+0000 7f6827db8640 1 -- start start 2024-04-22T08:12:40.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.286+0000 7f6827db8640 10 monclient: init 2024-04-22T08:12:40.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.286+0000 7f6827db8640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:40.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6827db8640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-22T08:12:40.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6827db8640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6820098cd0 0x7f6820134e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:40.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6827db8640 10 monclient(hunting): picked mon.noname-b con 0x7f6820098cd0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:40.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6827db8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68200997c0 0x7f682006ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:40.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6827db8640 10 monclient(hunting): picked mon.noname-a con 0x7f68200997c0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:40.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6827db8640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f68201392f0 0x7f682006f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:40.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6827db8640 10 monclient(hunting): picked mon.noname-c con 0x7f68201392f0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:40.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6827db8640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:40.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6827db8640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68201349e0 con 0x7f68200997c0 2024-04-22T08:12:40.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6827db8640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:40.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6827db8640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6820134420 con 0x7f6820098cd0 2024-04-22T08:12:40.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6827db8640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:40.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6827db8640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6820134140 con 0x7f68201392f0 2024-04-22T08:12:40.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6827db8640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:40.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6827db8640 10 monclient(hunting): authenticate will time out at 813.579655s 2024-04-22T08:12:40.299 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6825b54640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6820098cd0 0x7f6820134e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:40.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6825b54640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6820098cd0 0x7f6820134e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40562/0 (socket says 172.21.15.189:40562) 2024-04-22T08:12:40.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6825b54640 1 -- 172.21.15.189:0/3840910736 learned_addr learned my addr 172.21.15.189:0/3840910736 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-22T08:12:40.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6825b54640 10 monclient(hunting): get_auth_request con 0x7f6820098cd0 auth_method 0 2024-04-22T08:12:40.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6825b54640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:40.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6825b54640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:40.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6825b54640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:40.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6825b54640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:40.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6825b54640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:40.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6825b54640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:40.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6825353640 1 --2- 172.21.15.189:0/3840910736 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68200997c0 0x7f682006ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:40.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6825353640 10 monclient(hunting): get_auth_request con 0x7f68200997c0 auth_method 0 2024-04-22T08:12:40.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6825353640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:40.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6825353640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:40.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6825353640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:40.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6826355640 1 --2- 172.21.15.189:0/3840910736 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f68201392f0 0x7f682006f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:40.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6826355640 10 monclient(hunting): get_auth_request con 0x7f68201392f0 auth_method 0 2024-04-22T08:12:40.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6826355640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:40.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6826355640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:40.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6826355640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:40.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6825353640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:40.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6825353640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:40.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6825353640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:40.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6825b54640 10 monclient(hunting): handle_auth_done global_id 4147 payload 824 2024-04-22T08:12:40.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6825b54640 1 -- 172.21.15.189:0/3840910736 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f68201392f0 msgr2=0x7f682006f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:40.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6825b54640 1 --2- 172.21.15.189:0/3840910736 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f68201392f0 0x7f682006f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:40.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6825b54640 1 -- 172.21.15.189:0/3840910736 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68200997c0 msgr2=0x7f682006ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:40.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6825b54640 1 --2- 172.21.15.189:0/3840910736 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68200997c0 0x7f682006ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:40.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6825b54640 10 monclient: _finish_hunting 0 2024-04-22T08:12:40.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6825b54640 1 monclient: found mon.noname-b 2024-04-22T08:12:40.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6825b54640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:40.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6825b54640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.189:3300/0 2024-04-22T08:12:40.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.287+0000 7f6825b54640 1 -- 172.21.15.189:0/3840910736 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6818009a70 con 0x7f6820098cd0 2024-04-22T08:12:40.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.289+0000 7f6825b54640 1 -- 172.21.15.189:0/3840910736 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6820098cd0 msgr2=0x7f6820134e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:40.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.289+0000 7f6825b54640 1 --2- 172.21.15.189:0/3840910736 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6820098cd0 0x7f6820134e00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f681c00c460 tx=0x7f681c00c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:40.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.289+0000 7f68167fc640 1 -- 172.21.15.189:0/3840910736 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681c007070 con 0x7f6820098cd0 2024-04-22T08:12:40.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.289+0000 7f68167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:40.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.289+0000 7f68167fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:12:40.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.289+0000 7f68167fc640 10 monclient: dump: 2024-04-22T08:12:40.302 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-04-22T08:12:40.302 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:40.302 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:40.302 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:40.302 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:40.302 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-04-22T08:12:40.303 INFO:teuthology.orchestra.run.smithi189.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:40.303 INFO:teuthology.orchestra.run.smithi189.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:40.303 INFO:teuthology.orchestra.run.smithi189.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:40.303 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-04-22T08:12:40.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.289+0000 7f68167fc640 10 monclient: _finish_auth 0 2024-04-22T08:12:40.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.289+0000 7f68167fc640 10 monclient: _check_auth_tickets 2024-04-22T08:12:40.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.289+0000 7f68167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:40.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.289+0000 7f6827db8640 5 monclient: authenticate success, global_id 4147 2024-04-22T08:12:40.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.289+0000 7f68167fc640 1 -- 172.21.15.189:0/3840910736 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f681c0071f0 con 0x7f6820098cd0 2024-04-22T08:12:40.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.289+0000 7f6827db8640 10 monclient: _renew_subs 2024-04-22T08:12:40.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.289+0000 7f6827db8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.189:3300/0 2024-04-22T08:12:40.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.289+0000 7f6827db8640 1 -- 172.21.15.189:0/3840910736 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68200706d0 con 0x7f6820098cd0 2024-04-22T08:12:40.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.289+0000 7f68167fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:40.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.289+0000 7f68167fc640 1 -- 172.21.15.189:0/3840910736 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681c0075a0 con 0x7f6820098cd0 2024-04-22T08:12:40.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.289+0000 7f68167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:40.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.289+0000 7f68167fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:12:40.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.289+0000 7f68167fc640 10 monclient: dump: 2024-04-22T08:12:40.304 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-04-22T08:12:40.304 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:40.304 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:40.304 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:40.304 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:40.304 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-04-22T08:12:40.304 INFO:teuthology.orchestra.run.smithi189.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:40.304 INFO:teuthology.orchestra.run.smithi189.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:40.304 INFO:teuthology.orchestra.run.smithi189.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:40.304 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-04-22T08:12:40.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.289+0000 7f6827db8640 10 monclient: _renew_subs 2024-04-22T08:12:40.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.289+0000 7f6827db8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.189:3300/0 2024-04-22T08:12:40.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.289+0000 7f6817fff640 4 set_mon_vals no callback set 2024-04-22T08:12:40.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.289+0000 7f6827db8640 1 -- 172.21.15.189:0/3840910736 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6820071730 con 0x7f6820098cd0 2024-04-22T08:12:40.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.290+0000 7f68167fc640 1 -- 172.21.15.189:0/3840910736 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 2) v1 ==== 44528+0+0 (secure 0 0 0) 0x7f681c01b400 con 0x7f6820098cd0 2024-04-22T08:12:40.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.290+0000 7f6827db8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:40.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.290+0000 7f6827db8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:40.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.290+0000 7f6827db8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.189:3300/0 2024-04-22T08:12:40.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.290+0000 7f6827db8640 1 -- 172.21.15.189:0/3840910736 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f682004e9b0 con 0x7f6820098cd0 2024-04-22T08:12:40.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.290+0000 7f68167fc640 1 -- 172.21.15.189:0/3840910736 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f681c0490b0 con 0x7f6820098cd0 2024-04-22T08:12:40.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.292+0000 7f68167fc640 1 -- 172.21.15.189:0/3840910736 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (secure 0 0 0) 0x7f681c01be60 con 0x7f6820098cd0 2024-04-22T08:12:40.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.292+0000 7f68167fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:40.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.292+0000 7f68167fc640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:12:40.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.362+0000 7f6827db8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "abbac246-54d6-47d7-a41c-a68df0b80ddb", "id": 5}] 2024-04-22T08:12:40.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.362+0000 7f6827db8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "abbac246-54d6-47d7-a41c-a68df0b80ddb", "id": 5}] 2024-04-22T08:12:40.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.362+0000 7f6827db8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.189:3300/0 2024-04-22T08:12:40.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.362+0000 7f6827db8640 1 -- 172.21.15.189:0/3840910736 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd new", "uuid": "abbac246-54d6-47d7-a41c-a68df0b80ddb", "id": 5} v 0) v1 -- 0x7f6820065bd0 con 0x7f6820098cd0 2024-04-22T08:12:40.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.366+0000 7f68167fc640 1 -- 172.21.15.189:0/3840910736 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "abbac246-54d6-47d7-a41c-a68df0b80ddb", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f681c01a360 con 0x7f6820098cd0 2024-04-22T08:12:40.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.366+0000 7f68167fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "abbac246-54d6-47d7-a41c-a68df0b80ddb", "id": 5}] 2024-04-22T08:12:40.366 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.366+0000 7f68167fc640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:12:40.366 INFO:teuthology.orchestra.run.smithi189.stdout:5 2024-04-22T08:12:40.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.367+0000 7f6827db8640 10 monclient: shutdown 2024-04-22T08:12:40.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.367+0000 7f6827db8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:40.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.367+0000 7f6827db8640 1 -- 172.21.15.189:0/3840910736 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6820098cd0 msgr2=0x7f6820134e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:40.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.367+0000 7f6827db8640 1 --2- 172.21.15.189:0/3840910736 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6820098cd0 0x7f6820134e00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f681c00c460 tx=0x7f681c00c920 comp rx=0 tx=0).stop 2024-04-22T08:12:40.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.367+0000 7f6827db8640 1 -- 172.21.15.189:0/3840910736 shutdown_connections 2024-04-22T08:12:40.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.367+0000 7f6827db8640 1 --2- 172.21.15.189:0/3840910736 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f68201392f0 0x7f682006f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:40.367 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.367+0000 7f6827db8640 1 --2- 172.21.15.189:0/3840910736 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68200997c0 0x7f682006ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:40.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.367+0000 7f6827db8640 1 --2- 172.21.15.189:0/3840910736 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6820098cd0 0x7f6820134e00 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:40.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.367+0000 7f6827db8640 1 -- 172.21.15.189:0/3840910736 >> 172.21.15.189:0/3840910736 conn(0x7f68200a1280 msgr2=0x7f68200a2050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:40.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.367+0000 7f6827db8640 1 -- 172.21.15.189:0/3840910736 shutdown_connections 2024-04-22T08:12:40.368 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-22T08:12:40.367+0000 7f6827db8640 1 -- 172.21.15.189:0/3840910736 wait complete. 2024-04-22T08:12:40.377 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-22T08:12:40.377 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-22T08:12:40.379 INFO:tasks.ceph.osd.0:Started 2024-04-22T08:12:40.380 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-22T08:12:40.380 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-22T08:12:40.382 INFO:tasks.ceph.osd.1:Started 2024-04-22T08:12:40.382 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-22T08:12:40.382 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-22T08:12:40.384 INFO:tasks.ceph.osd.2:Started 2024-04-22T08:12:40.384 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-22T08:12:40.384 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-22T08:12:40.386 INFO:tasks.ceph.osd.3:Started 2024-04-22T08:12:40.387 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-22T08:12:40.387 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-22T08:12:40.389 INFO:tasks.ceph.osd.4:Started 2024-04-22T08:12:40.389 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-22T08:12:40.389 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-22T08:12:40.391 INFO:tasks.ceph.osd.5:Started 2024-04-22T08:12:40.391 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T08:12:40.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.506+0000 7f5b2cd34640 10 monclient: get_monmap_and_config 2024-04-22T08:12:40.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.506+0000 7f5b2cd34640 10 monclient: build_initial_monmap 2024-04-22T08:12:40.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.506+0000 7f5b2cd34640 10 monclient: monmap: 2024-04-22T08:12:40.507 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:40.507 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:40.507 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:40.507640+0000 2024-04-22T08:12:40.507 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:40.507640+0000 2024-04-22T08:12:40.507 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:40.507 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:40.507 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:12:40.507 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:12:40.507 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:12:40.507 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:12:40.508 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:12:40.508 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:40.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.507+0000 7f5b2cd34640 1 Processor -- start 2024-04-22T08:12:40.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.507+0000 7f5b2cd34640 1 -- start start 2024-04-22T08:12:40.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.507+0000 7f5b2cd34640 10 monclient: init 2024-04-22T08:12:40.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.508+0000 7f5b2cd34640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:40.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.508+0000 7f5b2cd34640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-04-22T08:12:40.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.508+0000 7f5b2cd34640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5b2809de00 addr v1:172.21.15.40:6789/0 2024-04-22T08:12:40.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.508+0000 7f5b2cd34640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5b280a4330 0x7f5b2813b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:40.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.508+0000 7f5b2cd34640 10 monclient(hunting): picked mon.noname-a con 0x7f5b280a4330 addr v2:172.21.15.40:3300/0 2024-04-22T08:12:40.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.508+0000 7f5b2cd34640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5b280a0240 addr v1:172.21.15.189:6789/0 2024-04-22T08:12:40.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.508+0000 7f5b2cd34640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b280a3410 con 0x7f5b2809de00 2024-04-22T08:12:40.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.508+0000 7f5b2cd34640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b280a4b00 con 0x7f5b280a0240 2024-04-22T08:12:40.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.508+0000 7f5b2cd34640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:40.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.508+0000 7f5b2cd34640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b280a3f50 con 0x7f5b280a4330 2024-04-22T08:12:40.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.508+0000 7f5b2cd34640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:40.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.508+0000 7f5b2cd34640 10 monclient(hunting): authenticate will time out at 834.502008s 2024-04-22T08:12:40.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.508+0000 7f5b25d9b640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5b280a4330 0x7f5b2813b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:40.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.508+0000 7f5b2659c640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f5b2809de00 0x7f5b2809e1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:36778/0 (socket says 172.21.15.40:36778) 2024-04-22T08:12:40.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.508+0000 7f5b2659c640 1 -- 172.21.15.40:0/2378791970 learned_addr learned my addr 172.21.15.40:0/2378791970 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:12:40.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.508+0000 7f5b25d9b640 10 monclient(hunting): get_auth_request con 0x7f5b280a4330 auth_method 0 2024-04-22T08:12:40.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.508+0000 7f5b25d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:40.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.508+0000 7f5b25d9b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:40.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.508+0000 7f5b25d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:40.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.508+0000 7f5b25d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:40.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.508+0000 7f5b25d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:40.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.508+0000 7f5b25d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:40.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.508+0000 7f5b25d9b640 10 monclient(hunting): handle_auth_done global_id 4164 payload 274 2024-04-22T08:12:40.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.508+0000 7f5b25d9b640 1 -- 172.21.15.40:0/2378791970 >> v1:172.21.15.189:6789/0 conn(0x7f5b280a0240 legacy=0x7f5b2813d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:40.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.508+0000 7f5b25d9b640 1 -- 172.21.15.40:0/2378791970 >> v1:172.21.15.40:6789/0 conn(0x7f5b2809de00 legacy=0x7f5b2809e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:40.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.508+0000 7f5b25d9b640 10 monclient: _finish_hunting 0 2024-04-22T08:12:40.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.508+0000 7f5b25d9b640 1 monclient: found mon.noname-a 2024-04-22T08:12:40.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.508+0000 7f5b25d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:40.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.508+0000 7f5b25d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:12:40.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.508+0000 7f5b25d9b640 1 -- 172.21.15.40:0/2378791970 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b280a4090 con 0x7f5b280a4330 2024-04-22T08:12:40.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.511+0000 7f5b25d9b640 1 -- 172.21.15.40:0/2378791970 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b280a4330 msgr2=0x7f5b2813b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:40.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.511+0000 7f5b25d9b640 1 --2- 172.21.15.40:0/2378791970 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b280a4330 0x7f5b2813b070 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5b180097a0 tx=0x7f5b1802b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b73d5cc55e38aabd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:40.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.511+0000 7f5b2559a640 1 -- 172.21.15.40:0/2378791970 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b18030a00 con 0x7f5b280a4330 2024-04-22T08:12:40.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.511+0000 7f5b2559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:40.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.511+0000 7f5b2559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:12:40.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.511+0000 7f5b2559a640 10 monclient: dump: 2024-04-22T08:12:40.517 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:40.517 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:40.517 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:40.517 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:40.517 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:40.517 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:40.517 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:40.517 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:40.517 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:40.518 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:40.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.511+0000 7f5b2559a640 10 monclient: _finish_auth 0 2024-04-22T08:12:40.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.511+0000 7f5b2559a640 10 monclient: _check_auth_tickets 2024-04-22T08:12:40.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.511+0000 7f5b2559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:40.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.511+0000 7f5b2559a640 1 -- 172.21.15.40:0/2378791970 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b1802e050 con 0x7f5b280a4330 2024-04-22T08:12:40.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.511+0000 7f5b2cd34640 5 monclient: authenticate success, global_id 4164 2024-04-22T08:12:40.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.511+0000 7f5b2cd34640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:12:40.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.511+0000 7f5b2559a640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:40.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.511+0000 7f5b2cd34640 10 monclient: get_monmap_and_config success 2024-04-22T08:12:40.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.511+0000 7f5b2559a640 1 -- 172.21.15.40:0/2378791970 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b18030d80 con 0x7f5b280a4330 2024-04-22T08:12:40.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.511+0000 7f5b2cd34640 4 set_mon_vals no callback set 2024-04-22T08:12:40.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.511+0000 7f5b2559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:40.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.511+0000 7f5b2559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:12:40.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.511+0000 7f5b2559a640 10 monclient: dump: 2024-04-22T08:12:40.519 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:40.519 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:40.519 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:40.519 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:40.519 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:40.519 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:40.519 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:40.519 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:40.520 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:40.520 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:40.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.512+0000 7f5b2cd34640 10 monclient: shutdown 2024-04-22T08:12:40.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.512+0000 7f5b2cd34640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:40.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.512+0000 7f5b2cd34640 1 -- 172.21.15.40:0/2378791970 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b280a4330 msgr2=0x7f5b2813b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:40.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.512+0000 7f5b2cd34640 1 --2- 172.21.15.40:0/2378791970 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b280a4330 0x7f5b2813b070 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5b180097a0 tx=0x7f5b1802b710 comp rx=0 tx=0).stop 2024-04-22T08:12:40.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.512+0000 7f5b2cd34640 1 -- 172.21.15.40:0/2378791970 shutdown_connections 2024-04-22T08:12:40.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.512+0000 7f5b2cd34640 1 --2- 172.21.15.40:0/2378791970 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b280a4330 0x7f5b2813b070 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:40.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.512+0000 7f5b2cd34640 1 -- 172.21.15.40:0/2378791970 >> 172.21.15.40:0/2378791970 conn(0x7f5b280a83f0 msgr2=0x7f5b280a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:40.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.512+0000 7f5b2cd34640 1 -- 172.21.15.40:0/2378791970 shutdown_connections 2024-04-22T08:12:40.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.512+0000 7f5b2cd34640 1 -- 172.21.15.40:0/2378791970 wait complete. 2024-04-22T08:12:40.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.513+0000 7f5b2cd34640 10 monclient: build_initial_monmap 2024-04-22T08:12:40.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.513+0000 7f5b2cd34640 10 monclient: monmap: 2024-04-22T08:12:40.521 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:40.521 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:12:40.521 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:12:40.521 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:12:40.521 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:40.521 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:40.521 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:12:40.521 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:12:40.521 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:12:40.521 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:40.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.513+0000 7f5b2cd34640 1 Processor -- start 2024-04-22T08:12:40.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.513+0000 7f5b2cd34640 1 -- start start 2024-04-22T08:12:40.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.513+0000 7f5b2cd34640 10 monclient: init 2024-04-22T08:12:40.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.513+0000 7f5b2cd34640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:40.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.513+0000 7f5b2cd34640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-22T08:12:40.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.513+0000 7f5b2cd34640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b2809de00 0x7f5b281322a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:40.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.513+0000 7f5b2cd34640 10 monclient(hunting): picked mon.noname-b con 0x7f5b2809de00 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:40.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.513+0000 7f5b2cd34640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b280a0240 0x7f5b281327e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:40.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.513+0000 7f5b2cd34640 10 monclient(hunting): picked mon.noname-c con 0x7f5b280a0240 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:40.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.513+0000 7f5b2cd34640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b28132d20 0x7f5b281cac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:40.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.513+0000 7f5b2cd34640 10 monclient(hunting): picked mon.noname-a con 0x7f5b28132d20 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:40.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.513+0000 7f5b2cd34640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:40.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.513+0000 7f5b2cd34640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b281352f0 con 0x7f5b28132d20 2024-04-22T08:12:40.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.513+0000 7f5b2cd34640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:40.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.513+0000 7f5b2cd34640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b28131040 con 0x7f5b2809de00 2024-04-22T08:12:40.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.513+0000 7f5b2cd34640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:40.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.513+0000 7f5b2cd34640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b28134a70 con 0x7f5b280a0240 2024-04-22T08:12:40.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.513+0000 7f5b2cd34640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:40.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.513+0000 7f5b2cd34640 10 monclient(hunting): authenticate will time out at 834.507742s 2024-04-22T08:12:40.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.514+0000 7f5b25d9b640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b280a0240 0x7f5b281327e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:40.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.514+0000 7f5b26d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b28132d20 0x7f5b281cac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:40.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.514+0000 7f5b25d9b640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b280a0240 0x7f5b281327e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:50554/0 (socket says 172.21.15.40:50554) 2024-04-22T08:12:40.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.514+0000 7f5b26d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b28132d20 0x7f5b281cac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57666/0 (socket says 172.21.15.40:57666) 2024-04-22T08:12:40.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.514+0000 7f5b25d9b640 1 -- 172.21.15.40:0/1369172352 learned_addr learned my addr 172.21.15.40:0/1369172352 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:40.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.514+0000 7f5b25d9b640 10 monclient(hunting): get_auth_request con 0x7f5b280a0240 auth_method 0 2024-04-22T08:12:40.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.514+0000 7f5b25d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:40.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.514+0000 7f5b25d9b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:40.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.514+0000 7f5b25d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:40.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.514+0000 7f5b26d9d640 10 monclient(hunting): get_auth_request con 0x7f5b28132d20 auth_method 0 2024-04-22T08:12:40.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.514+0000 7f5b26d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:40.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.514+0000 7f5b26d9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:40.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.514+0000 7f5b26d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:40.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.514+0000 7f5b2659c640 1 --2- 172.21.15.40:0/1369172352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b2809de00 0x7f5b281322a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:40.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.514+0000 7f5b2659c640 10 monclient(hunting): get_auth_request con 0x7f5b2809de00 auth_method 0 2024-04-22T08:12:40.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.514+0000 7f5b2659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:40.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.514+0000 7f5b2659c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:40.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.514+0000 7f5b2659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:40.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.514+0000 7f5b26d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:40.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.514+0000 7f5b26d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:40.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.514+0000 7f5b26d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:40.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.514+0000 7f5b25d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:40.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.514+0000 7f5b25d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:40.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.514+0000 7f5b25d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:40.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.514+0000 7f5b2659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:40.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.514+0000 7f5b2659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:40.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.514+0000 7f5b2659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:40.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.514+0000 7f5b26d9d640 10 monclient(hunting): handle_auth_done global_id 4167 payload 824 2024-04-22T08:12:40.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.514+0000 7f5b26d9d640 1 -- 172.21.15.40:0/1369172352 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b280a0240 msgr2=0x7f5b281327e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:40.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.514+0000 7f5b26d9d640 1 --2- 172.21.15.40:0/1369172352 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b280a0240 0x7f5b281327e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:40.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.514+0000 7f5b26d9d640 1 -- 172.21.15.40:0/1369172352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b2809de00 msgr2=0x7f5b281322a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:40.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.514+0000 7f5b26d9d640 1 --2- 172.21.15.40:0/1369172352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b2809de00 0x7f5b281322a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:40.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.514+0000 7f5b26d9d640 10 monclient: _finish_hunting 0 2024-04-22T08:12:40.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.514+0000 7f5b26d9d640 1 monclient: found mon.noname-a 2024-04-22T08:12:40.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.514+0000 7f5b26d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:40.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.514+0000 7f5b26d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:12:40.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.514+0000 7f5b26d9d640 1 -- 172.21.15.40:0/1369172352 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b18009400 con 0x7f5b28132d20 2024-04-22T08:12:40.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.516+0000 7f5b26d9d640 1 -- 172.21.15.40:0/1369172352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b28132d20 msgr2=0x7f5b281cac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:40.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.516+0000 7f5b26d9d640 1 --2- 172.21.15.40:0/1369172352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b28132d20 0x7f5b281cac10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5b2000c780 tx=0x7f5b2000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:40.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.516+0000 7f5b1effd640 1 -- 172.21.15.40:0/1369172352 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b2000ac60 con 0x7f5b28132d20 2024-04-22T08:12:40.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.516+0000 7f5b1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:40.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.516+0000 7f5b1effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:12:40.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.516+0000 7f5b1effd640 10 monclient: dump: 2024-04-22T08:12:40.531 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:40.531 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:40.531 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:40.531 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:40.531 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:40.531 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:40.531 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:40.531 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:40.532 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:40.532 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:40.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.516+0000 7f5b1effd640 10 monclient: _finish_auth 0 2024-04-22T08:12:40.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.516+0000 7f5b1effd640 10 monclient: _check_auth_tickets 2024-04-22T08:12:40.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.516+0000 7f5b1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:40.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.516+0000 7f5b1effd640 1 -- 172.21.15.40:0/1369172352 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b2000adc0 con 0x7f5b28132d20 2024-04-22T08:12:40.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.516+0000 7f5b1effd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:40.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.516+0000 7f5b1effd640 1 -- 172.21.15.40:0/1369172352 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b20012660 con 0x7f5b28132d20 2024-04-22T08:12:40.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.516+0000 7f5b2cd34640 5 monclient: authenticate success, global_id 4167 2024-04-22T08:12:40.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.517+0000 7f5b2cd34640 10 monclient: _renew_subs 2024-04-22T08:12:40.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.517+0000 7f5b2cd34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:40.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.517+0000 7f5b2cd34640 1 -- 172.21.15.40:0/1369172352 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b281357d0 con 0x7f5b28132d20 2024-04-22T08:12:40.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.516+0000 7f5b24d99640 4 set_mon_vals no callback set 2024-04-22T08:12:40.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.517+0000 7f5b1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:40.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.517+0000 7f5b1effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:12:40.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.517+0000 7f5b1effd640 10 monclient: dump: 2024-04-22T08:12:40.533 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:40.533 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:40.533 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:40.533 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:40.533 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:40.533 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:40.533 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:40.533 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:40.533 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:40.534 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:40.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.517+0000 7f5b2cd34640 10 monclient: _renew_subs 2024-04-22T08:12:40.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.517+0000 7f5b2cd34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:40.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.517+0000 7f5b2cd34640 1 -- 172.21.15.40:0/1369172352 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b28133fd0 con 0x7f5b28132d20 2024-04-22T08:12:40.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.517+0000 7f5b2cd34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:40.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.517+0000 7f5b2cd34640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:40.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.518+0000 7f5b2cd34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:40.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.518+0000 7f5b2cd34640 1 -- 172.21.15.40:0/1369172352 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b2804e9b0 con 0x7f5b28132d20 2024-04-22T08:12:40.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.518+0000 7f5b1effd640 1 -- 172.21.15.40:0/1369172352 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 2) v1 ==== 44528+0+0 (secure 0 0 0) 0x7f5b20012880 con 0x7f5b28132d20 2024-04-22T08:12:40.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.518+0000 7f5b1effd640 1 -- 172.21.15.40:0/1369172352 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f5b200496a0 con 0x7f5b28132d20 2024-04-22T08:12:40.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.520+0000 7f5b1effd640 1 -- 172.21.15.40:0/1369172352 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (secure 0 0 0) 0x7f5b2000ce30 con 0x7f5b28132d20 2024-04-22T08:12:40.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.520+0000 7f5b1effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:40.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.520+0000 7f5b1effd640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:12:40.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.581+0000 7f5b2cd34640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-22T08:12:40.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.581+0000 7f5b2cd34640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-22T08:12:40.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.581+0000 7f5b2cd34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:40.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.581+0000 7f5b2cd34640 1 -- 172.21.15.40:0/1369172352 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b2806a270 con 0x7f5b28132d20 2024-04-22T08:12:40.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.582+0000 7f5b1effd640 1 -- 172.21.15.40:0/1369172352 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) v1 ==== 74+0+5604 (secure 0 0 0) 0x7f5b20016030 con 0x7f5b28132d20 2024-04-22T08:12:40.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.582+0000 7f5b1effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-22T08:12:40.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.582+0000 7f5b1effd640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:12:40.582 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:12:40.583 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":8,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","created":"2024-04-22T08:12:36.003043+0000","modified":"2024-04-22T08:12:40.364359+0000","last_up_change":"0.000000","last_in_change":"2024-04-22T08:12:40.364359+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9a0210c-5881-4ca0-8318-a4a987109405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"11370cc8-09a6-4c35-b92c-4d926c6c18d9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d03db34e-f630-4a3e-86fb-7234bee0d0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d94b2052-5847-4955-b1d4-b7e627572811","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb626202-7411-4299-8d20-988bcc9103f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"abbac246-54d6-47d7-a41c-a68df0b80ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T08:12:40.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.582+0000 7f5b2cd34640 10 monclient: shutdown 2024-04-22T08:12:40.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.582+0000 7f5b2cd34640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:40.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.582+0000 7f5b2cd34640 1 -- 172.21.15.40:0/1369172352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b28132d20 msgr2=0x7f5b281cac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:40.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.582+0000 7f5b2cd34640 1 --2- 172.21.15.40:0/1369172352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b28132d20 0x7f5b281cac10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5b2000c780 tx=0x7f5b2000cc40 comp rx=0 tx=0).stop 2024-04-22T08:12:40.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.582+0000 7f5b2cd34640 1 -- 172.21.15.40:0/1369172352 shutdown_connections 2024-04-22T08:12:40.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.582+0000 7f5b2cd34640 1 --2- 172.21.15.40:0/1369172352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b28132d20 0x7f5b281cac10 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:40.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.582+0000 7f5b2cd34640 1 --2- 172.21.15.40:0/1369172352 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b280a0240 0x7f5b281327e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:40.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.582+0000 7f5b2cd34640 1 --2- 172.21.15.40:0/1369172352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b2809de00 0x7f5b281322a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:40.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.582+0000 7f5b2cd34640 1 -- 172.21.15.40:0/1369172352 >> 172.21.15.40:0/1369172352 conn(0x7f5b280a83f0 msgr2=0x7f5b281364b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:40.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.582+0000 7f5b2cd34640 1 -- 172.21.15.40:0/1369172352 shutdown_connections 2024-04-22T08:12:40.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:40.582+0000 7f5b2cd34640 1 -- 172.21.15.40:0/1369172352 wait complete. 2024-04-22T08:12:40.592 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-22T08:12:40.593 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-22T08:12:40.987 INFO:tasks.ceph.osd.4.smithi189.stderr:2024-04-22T08:12:40.987+0000 7fb1d0121640 -1 Falling back to public interface 2024-04-22T08:12:40.991 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-04-22T08:12:40.991+0000 7f58e120f640 -1 Falling back to public interface 2024-04-22T08:12:40.991 INFO:tasks.ceph.osd.3.smithi189.stderr:2024-04-22T08:12:40.991+0000 7f2523fb2640 -1 Falling back to public interface 2024-04-22T08:12:40.991 INFO:tasks.ceph.osd.5.smithi189.stderr:2024-04-22T08:12:40.991+0000 7f99c700d640 -1 Falling back to public interface 2024-04-22T08:12:40.996 INFO:tasks.ceph.osd.0.smithi040.stderr:2024-04-22T08:12:40.996+0000 7f1640c18640 -1 Falling back to public interface 2024-04-22T08:12:40.999 INFO:tasks.ceph.osd.2.smithi040.stderr:2024-04-22T08:12:40.999+0000 7f16db218640 -1 Falling back to public interface 2024-04-22T08:12:41.196 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-22T08:12:41.294 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.294+0000 7fec77358640 10 monclient: get_monmap_and_config 2024-04-22T08:12:41.294 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.294+0000 7fec77358640 10 monclient: build_initial_monmap 2024-04-22T08:12:41.294 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.294+0000 7fec77358640 10 monclient: monmap: 2024-04-22T08:12:41.294 INFO:teuthology.misc.health.smithi040.stderr:epoch 0 2024-04-22T08:12:41.294 INFO:teuthology.misc.health.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:41.294 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2024-04-22T08:12:41.295000+0000 2024-04-22T08:12:41.295 INFO:teuthology.misc.health.smithi040.stderr:created 2024-04-22T08:12:41.295000+0000 2024-04-22T08:12:41.295 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:41.295 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:41.295 INFO:teuthology.misc.health.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:12:41.295 INFO:teuthology.misc.health.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:12:41.295 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:12:41.295 INFO:teuthology.misc.health.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:12:41.295 INFO:teuthology.misc.health.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:12:41.295 INFO:teuthology.misc.health.smithi040.stderr: 2024-04-22T08:12:41.295 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.294+0000 7fec77358640 1 Processor -- start 2024-04-22T08:12:41.295 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec77358640 1 -- start start 2024-04-22T08:12:41.295 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec77358640 10 monclient: init 2024-04-22T08:12:41.298 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec77358640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:41.298 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec77358640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-04-22T08:12:41.298 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec77358640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fec700a1b70 0x7fec700a1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:41.298 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec77358640 10 monclient(hunting): picked mon.noname-c con 0x7fec700a1b70 addr v2:172.21.15.189:3300/0 2024-04-22T08:12:41.298 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec77358640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fec700a3300 0x7fec70135e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:41.299 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec77358640 10 monclient(hunting): picked mon.noname-a con 0x7fec700a3300 addr v2:172.21.15.40:3300/0 2024-04-22T08:12:41.299 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec77358640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec700a3790 0x7fec70136340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:41.299 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec77358640 10 monclient(hunting): picked mon.noname-b con 0x7fec700a3790 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:41.299 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec77358640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:41.299 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec77358640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec70072410 con 0x7fec700a3300 2024-04-22T08:12:41.299 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec77358640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:41.299 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec77358640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec7009fe10 con 0x7fec700a1b70 2024-04-22T08:12:41.299 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec77358640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:41.299 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec77358640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec700a0080 con 0x7fec700a3790 2024-04-22T08:12:41.299 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec77358640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:41.299 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec77358640 10 monclient(hunting): authenticate will time out at 835.289275s 2024-04-22T08:12:41.300 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec758f5640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec700a3790 0x7fec70136340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:41.300 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec748f3640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fec700a3300 0x7fec70135e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:41.300 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec748f3640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fec700a3300 0x7fec70135e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57676/0 (socket says 172.21.15.40:57676) 2024-04-22T08:12:41.300 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec758f5640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec700a3790 0x7fec70136340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:50564/0 (socket says 172.21.15.40:50564) 2024-04-22T08:12:41.300 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec748f3640 1 -- 172.21.15.40:0/2474311946 learned_addr learned my addr 172.21.15.40:0/2474311946 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:41.300 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec748f3640 10 monclient(hunting): get_auth_request con 0x7fec700a3300 auth_method 0 2024-04-22T08:12:41.300 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec748f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:41.300 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec748f3640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:41.300 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec748f3640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:41.300 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec758f5640 10 monclient(hunting): get_auth_request con 0x7fec700a3790 auth_method 0 2024-04-22T08:12:41.300 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec758f5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:41.300 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec758f5640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:41.300 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec758f5640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:41.300 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec748f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:41.301 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec748f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:41.301 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec748f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:41.301 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec758f5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:41.301 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec758f5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:41.301 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec758f5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:41.301 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec750f4640 1 --2- 172.21.15.40:0/2474311946 >> v2:172.21.15.189:3300/0 conn(0x7fec700a1b70 0x7fec700a1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:41.301 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec750f4640 10 monclient(hunting): get_auth_request con 0x7fec700a1b70 auth_method 0 2024-04-22T08:12:41.301 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec750f4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:41.301 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec750f4640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:41.301 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec750f4640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:41.301 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec748f3640 10 monclient(hunting): handle_auth_done global_id 4170 payload 274 2024-04-22T08:12:41.301 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec748f3640 1 -- 172.21.15.40:0/2474311946 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec700a3790 msgr2=0x7fec70136340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:41.301 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec748f3640 1 --2- 172.21.15.40:0/2474311946 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec700a3790 0x7fec70136340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:41.301 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec748f3640 1 -- 172.21.15.40:0/2474311946 >> v2:172.21.15.189:3300/0 conn(0x7fec700a1b70 msgr2=0x7fec700a1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:41.302 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec748f3640 1 --2- 172.21.15.40:0/2474311946 >> v2:172.21.15.189:3300/0 conn(0x7fec700a1b70 0x7fec700a1f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:41.302 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec748f3640 10 monclient: _finish_hunting 0 2024-04-22T08:12:41.302 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec748f3640 1 monclient: found mon.noname-a 2024-04-22T08:12:41.302 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec748f3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:41.302 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec748f3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:12:41.302 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.295+0000 7fec748f3640 1 -- 172.21.15.40:0/2474311946 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec700a0360 con 0x7fec700a3300 2024-04-22T08:12:41.302 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.298+0000 7fec748f3640 1 -- 172.21.15.40:0/2474311946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec700a3300 msgr2=0x7fec70135e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:41.302 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.298+0000 7fec748f3640 1 --2- 172.21.15.40:0/2474311946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec700a3300 0x7fec70135e00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fec700746b0 tx=0x7fec6002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e35c2aa4e27a3c66 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:41.302 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.298+0000 7fec67fff640 1 -- 172.21.15.40:0/2474311946 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec60031960 con 0x7fec700a3300 2024-04-22T08:12:41.302 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.298+0000 7fec67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:41.302 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.298+0000 7fec67fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:12:41.302 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.298+0000 7fec67fff640 10 monclient: dump: 2024-04-22T08:12:41.302 INFO:teuthology.misc.health.smithi040.stderr:epoch 1 2024-04-22T08:12:41.303 INFO:teuthology.misc.health.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:41.303 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:41.303 INFO:teuthology.misc.health.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:41.303 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:41.303 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:41.303 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:41.303 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:41.303 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:41.303 INFO:teuthology.misc.health.smithi040.stderr: 2024-04-22T08:12:41.303 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.298+0000 7fec67fff640 10 monclient: _finish_auth 0 2024-04-22T08:12:41.303 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.298+0000 7fec67fff640 10 monclient: _check_auth_tickets 2024-04-22T08:12:41.303 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.298+0000 7fec67fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:41.303 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.298+0000 7fec67fff640 1 -- 172.21.15.40:0/2474311946 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec60031ac0 con 0x7fec700a3300 2024-04-22T08:12:41.303 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.299+0000 7fec77358640 5 monclient: authenticate success, global_id 4170 2024-04-22T08:12:41.304 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.299+0000 7fec77358640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:12:41.304 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.299+0000 7fec67fff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:41.304 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.299+0000 7fec77358640 10 monclient: get_monmap_and_config success 2024-04-22T08:12:41.304 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.299+0000 7fec67fff640 1 -- 172.21.15.40:0/2474311946 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec60031e50 con 0x7fec700a3300 2024-04-22T08:12:41.304 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.299+0000 7fec77358640 4 set_mon_vals no callback set 2024-04-22T08:12:41.304 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.299+0000 7fec67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:41.304 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.299+0000 7fec67fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:12:41.304 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.299+0000 7fec67fff640 10 monclient: dump: 2024-04-22T08:12:41.304 INFO:teuthology.misc.health.smithi040.stderr:epoch 1 2024-04-22T08:12:41.304 INFO:teuthology.misc.health.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:41.304 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:41.304 INFO:teuthology.misc.health.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:41.304 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:41.304 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:41.305 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:41.305 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:41.305 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:41.305 INFO:teuthology.misc.health.smithi040.stderr: 2024-04-22T08:12:41.311 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.299+0000 7fec77358640 10 monclient: shutdown 2024-04-22T08:12:41.311 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.299+0000 7fec77358640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:41.311 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.299+0000 7fec77358640 1 -- 172.21.15.40:0/2474311946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec700a3300 msgr2=0x7fec70135e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:41.311 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.299+0000 7fec77358640 1 --2- 172.21.15.40:0/2474311946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec700a3300 0x7fec70135e00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fec700746b0 tx=0x7fec6002b7f0 comp rx=0 tx=0).stop 2024-04-22T08:12:41.311 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.299+0000 7fec77358640 1 -- 172.21.15.40:0/2474311946 shutdown_connections 2024-04-22T08:12:41.311 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.299+0000 7fec77358640 1 --2- 172.21.15.40:0/2474311946 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec700a3790 0x7fec70136340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:41.311 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.299+0000 7fec77358640 1 --2- 172.21.15.40:0/2474311946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec700a3300 0x7fec70135e00 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:41.311 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.299+0000 7fec77358640 1 --2- 172.21.15.40:0/2474311946 >> v2:172.21.15.189:3300/0 conn(0x7fec700a1b70 0x7fec700a1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:41.312 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.299+0000 7fec77358640 1 -- 172.21.15.40:0/2474311946 >> 172.21.15.40:0/2474311946 conn(0x7fec700a4f60 msgr2=0x7fec700a5370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:41.312 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.299+0000 7fec77358640 1 -- 172.21.15.40:0/2474311946 shutdown_connections 2024-04-22T08:12:41.312 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.299+0000 7fec77358640 1 -- 172.21.15.40:0/2474311946 wait complete. 2024-04-22T08:12:41.312 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.300+0000 7fec77358640 10 monclient: build_initial_monmap 2024-04-22T08:12:41.312 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.300+0000 7fec77358640 10 monclient: monmap: 2024-04-22T08:12:41.312 INFO:teuthology.misc.health.smithi040.stderr:epoch 0 2024-04-22T08:12:41.312 INFO:teuthology.misc.health.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:12:41.312 INFO:teuthology.misc.health.smithi040.stderr:last_changed 0.000000 2024-04-22T08:12:41.312 INFO:teuthology.misc.health.smithi040.stderr:created 0.000000 2024-04-22T08:12:41.313 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:41.313 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:41.313 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:12:41.313 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:12:41.313 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:12:41.313 INFO:teuthology.misc.health.smithi040.stderr: 2024-04-22T08:12:41.313 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.300+0000 7fec77358640 1 Processor -- start 2024-04-22T08:12:41.313 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.300+0000 7fec77358640 1 -- start start 2024-04-22T08:12:41.313 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.300+0000 7fec77358640 10 monclient: init 2024-04-22T08:12:41.313 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec77358640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:41.314 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec77358640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-22T08:12:41.314 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec77358640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec700a1b70 0x7fec701caf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:41.314 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec77358640 10 monclient(hunting): picked mon.noname-b con 0x7fec700a1b70 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:41.314 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec77358640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec700a3300 0x7fec701cb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:41.314 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec77358640 10 monclient(hunting): picked mon.noname-a con 0x7fec700a3300 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:41.314 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec77358640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec700a3790 0x7fec70121280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:41.314 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec77358640 10 monclient(hunting): picked mon.noname-c con 0x7fec700a3790 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:41.314 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec77358640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:41.314 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec77358640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec70125100 con 0x7fec700a3300 2024-04-22T08:12:41.314 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec77358640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:41.315 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec77358640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec70122da0 con 0x7fec700a1b70 2024-04-22T08:12:41.315 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec77358640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:41.315 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec77358640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec70123970 con 0x7fec700a3790 2024-04-22T08:12:41.315 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec77358640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:41.315 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec77358640 10 monclient(hunting): authenticate will time out at 835.294911s 2024-04-22T08:12:41.315 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec748f3640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec700a3300 0x7fec701cb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:41.315 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec748f3640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec700a3300 0x7fec701cb4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57684/0 (socket says 172.21.15.40:57684) 2024-04-22T08:12:41.315 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec748f3640 1 -- 172.21.15.40:0/3800498467 learned_addr learned my addr 172.21.15.40:0/3800498467 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:41.315 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec748f3640 10 monclient(hunting): get_auth_request con 0x7fec700a3300 auth_method 0 2024-04-22T08:12:41.315 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec748f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:41.316 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec748f3640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:41.316 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec748f3640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:41.316 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec758f5640 1 --2- 172.21.15.40:0/3800498467 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec700a3790 0x7fec70121280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:41.316 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec758f5640 10 monclient(hunting): get_auth_request con 0x7fec700a3790 auth_method 0 2024-04-22T08:12:41.316 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec758f5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:41.316 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec758f5640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:41.316 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec758f5640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:41.316 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec748f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:41.316 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec748f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:41.317 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec750f4640 1 --2- 172.21.15.40:0/3800498467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec700a1b70 0x7fec701caf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:41.317 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec748f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:41.317 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec750f4640 10 monclient(hunting): get_auth_request con 0x7fec700a1b70 auth_method 0 2024-04-22T08:12:41.317 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec750f4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:41.317 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec750f4640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:41.317 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec750f4640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:41.317 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec758f5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:41.317 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec758f5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:41.317 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec758f5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:41.318 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec750f4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:41.318 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec750f4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:41.318 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec750f4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:41.318 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec758f5640 10 monclient(hunting): handle_auth_done global_id 4163 payload 824 2024-04-22T08:12:41.318 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec758f5640 1 -- 172.21.15.40:0/3800498467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec700a1b70 msgr2=0x7fec701caf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:41.318 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec758f5640 1 --2- 172.21.15.40:0/3800498467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec700a1b70 0x7fec701caf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:41.318 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec758f5640 1 -- 172.21.15.40:0/3800498467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec700a3300 msgr2=0x7fec701cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:41.318 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec758f5640 1 --2- 172.21.15.40:0/3800498467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec700a3300 0x7fec701cb4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:41.318 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec758f5640 10 monclient: _finish_hunting 0 2024-04-22T08:12:41.318 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec758f5640 1 monclient: found mon.noname-c 2024-04-22T08:12:41.318 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec758f5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:41.318 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec758f5640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:12:41.319 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec758f5640 1 -- 172.21.15.40:0/3800498467 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec600094d0 con 0x7fec700a3790 2024-04-22T08:12:41.319 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.301+0000 7fec748f3640 1 --2- 172.21.15.40:0/3800498467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec700a3300 0x7fec701cb4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:12:41.319 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.303+0000 7fec758f5640 1 -- 172.21.15.40:0/3800498467 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec700a3790 msgr2=0x7fec70121280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:41.319 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.303+0000 7fec758f5640 1 --2- 172.21.15.40:0/3800498467 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec700a3790 0x7fec70121280 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fec6800efc0 tx=0x7fec6800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:41.319 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.304+0000 7fec65ffb640 1 -- 172.21.15.40:0/3800498467 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec68009c00 con 0x7fec700a3790 2024-04-22T08:12:41.319 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.304+0000 7fec65ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:41.319 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.304+0000 7fec65ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:12:41.319 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.304+0000 7fec65ffb640 10 monclient: dump: 2024-04-22T08:12:41.319 INFO:teuthology.misc.health.smithi040.stderr:epoch 1 2024-04-22T08:12:41.319 INFO:teuthology.misc.health.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:41.319 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:41.319 INFO:teuthology.misc.health.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:41.319 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:41.319 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:41.319 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:41.320 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:41.320 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:41.320 INFO:teuthology.misc.health.smithi040.stderr: 2024-04-22T08:12:41.320 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.304+0000 7fec65ffb640 10 monclient: _finish_auth 0 2024-04-22T08:12:41.320 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.304+0000 7fec65ffb640 10 monclient: _check_auth_tickets 2024-04-22T08:12:41.320 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.304+0000 7fec65ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:41.320 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.304+0000 7fec77358640 5 monclient: authenticate success, global_id 4163 2024-04-22T08:12:41.320 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.304+0000 7fec77358640 10 monclient: _renew_subs 2024-04-22T08:12:41.320 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.304+0000 7fec65ffb640 1 -- 172.21.15.40:0/3800498467 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec6800f040 con 0x7fec700a3790 2024-04-22T08:12:41.320 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.304+0000 7fec77358640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:41.320 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.304+0000 7fec77358640 1 -- 172.21.15.40:0/3800498467 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec70135d10 con 0x7fec700a3790 2024-04-22T08:12:41.320 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.304+0000 7fec65ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:41.320 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.304+0000 7fec65ffb640 1 -- 172.21.15.40:0/3800498467 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec680148e0 con 0x7fec700a3790 2024-04-22T08:12:41.320 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.304+0000 7fec65ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:41.321 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.304+0000 7fec65ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:12:41.321 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.304+0000 7fec65ffb640 10 monclient: dump: 2024-04-22T08:12:41.321 INFO:teuthology.misc.health.smithi040.stderr:epoch 1 2024-04-22T08:12:41.321 INFO:teuthology.misc.health.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:41.321 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:41.321 INFO:teuthology.misc.health.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:41.321 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:41.321 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:41.321 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:41.321 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:41.321 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:41.321 INFO:teuthology.misc.health.smithi040.stderr: 2024-04-22T08:12:41.321 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.304+0000 7fec677fe640 4 set_mon_vals no callback set 2024-04-22T08:12:41.321 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.304+0000 7fec77358640 10 monclient: _renew_subs 2024-04-22T08:12:41.321 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.304+0000 7fec77358640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:41.322 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.304+0000 7fec77358640 1 -- 172.21.15.40:0/3800498467 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec70124860 con 0x7fec700a3790 2024-04-22T08:12:41.322 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.305+0000 7fec65ffb640 1 -- 172.21.15.40:0/3800498467 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 2) v1 ==== 44528+0+0 (secure 0 0 0) 0x7fec68014a40 con 0x7fec700a3790 2024-04-22T08:12:41.322 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.305+0000 7fec77358640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:41.322 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.305+0000 7fec77358640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:41.322 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.305+0000 7fec77358640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:41.322 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.305+0000 7fec77358640 1 -- 172.21.15.40:0/3800498467 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec70124a80 con 0x7fec700a3790 2024-04-22T08:12:41.322 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.305+0000 7fec65ffb640 1 -- 172.21.15.40:0/3800498467 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7fec6804a370 con 0x7fec700a3790 2024-04-22T08:12:41.322 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.307+0000 7fec65ffb640 1 -- 172.21.15.40:0/3800498467 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (secure 0 0 0) 0x7fec68018080 con 0x7fec700a3790 2024-04-22T08:12:41.322 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.307+0000 7fec65ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:41.322 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.307+0000 7fec65ffb640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:12:41.368 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.368+0000 7fec77358640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-22T08:12:41.368 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.368+0000 7fec77358640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-22T08:12:41.368 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.368+0000 7fec77358640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:41.368 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.368+0000 7fec77358640 1 -- 172.21.15.40:0/3800498467 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec70069650 con 0x7fec700a3790 2024-04-22T08:12:41.724 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.725+0000 7fec65ffb640 1 -- 172.21.15.40:0/3800498467 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mgrmap(e 3) v1 ==== 44620+0+0 (secure 0 0 0) 0x7fec68014cf0 con 0x7fec700a3790 2024-04-22T08:12:41.725 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.725+0000 7fec65ffb640 1 --2- 172.21.15.40:0/3800498467 >> v2:172.21.15.189:6800/30168 conn(0x7fec54038b00 0x7fec5403aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:41.725 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.726+0000 7fec750f4640 1 --2- 172.21.15.40:0/3800498467 >> v2:172.21.15.189:6800/30168 conn(0x7fec54038b00 0x7fec5403aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:41.725 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.726+0000 7fec750f4640 10 monclient: get_auth_request con 0x7fec54038b00 auth_method 0 2024-04-22T08:12:41.726 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.726+0000 7fec65ffb640 1 -- 172.21.15.40:0/3800498467 <== mon.2 v2:172.21.15.40:3301/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) v1 ==== 74+0+5604 (secure 0 0 0) 0x7fec68007b60 con 0x7fec700a3790 2024-04-22T08:12:41.726 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.726+0000 7fec65ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-22T08:12:41.726 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.726+0000 7fec65ffb640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:12:41.726 INFO:teuthology.misc.health.smithi040.stdout: 2024-04-22T08:12:41.727 INFO:teuthology.misc.health.smithi040.stdout:{"epoch":8,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","created":"2024-04-22T08:12:36.003043+0000","modified":"2024-04-22T08:12:40.364359+0000","last_up_change":"0.000000","last_in_change":"2024-04-22T08:12:40.364359+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9a0210c-5881-4ca0-8318-a4a987109405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"11370cc8-09a6-4c35-b92c-4d926c6c18d9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d03db34e-f630-4a3e-86fb-7234bee0d0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d94b2052-5847-4955-b1d4-b7e627572811","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb626202-7411-4299-8d20-988bcc9103f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"abbac246-54d6-47d7-a41c-a68df0b80ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T08:12:41.728 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.727+0000 7fec77358640 1 -- 172.21.15.40:0/3800498467 >> v2:172.21.15.189:6800/30168 conn(0x7fec54038b00 msgr2=0x7fec5403aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:41.728 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.727+0000 7fec77358640 1 --2- 172.21.15.40:0/3800498467 >> v2:172.21.15.189:6800/30168 conn(0x7fec54038b00 0x7fec5403aef0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7fec6c006fd0 tx=0x7fec6c008040 comp rx=0 tx=0).stop 2024-04-22T08:12:41.728 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.727+0000 7fec77358640 10 monclient: shutdown 2024-04-22T08:12:41.728 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.727+0000 7fec77358640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:41.728 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.728+0000 7fec77358640 1 -- 172.21.15.40:0/3800498467 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec700a3790 msgr2=0x7fec70121280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:41.728 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.728+0000 7fec77358640 1 --2- 172.21.15.40:0/3800498467 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec700a3790 0x7fec70121280 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fec6800efc0 tx=0x7fec6800c490 comp rx=0 tx=0).stop 2024-04-22T08:12:41.728 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.728+0000 7fec77358640 1 -- 172.21.15.40:0/3800498467 shutdown_connections 2024-04-22T08:12:41.728 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.728+0000 7fec77358640 1 --2- 172.21.15.40:0/3800498467 >> v2:172.21.15.189:6800/30168 conn(0x7fec54038b00 0x7fec5403aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:41.729 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.728+0000 7fec77358640 1 --2- 172.21.15.40:0/3800498467 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec700a3790 0x7fec70121280 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:41.729 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.728+0000 7fec77358640 1 --2- 172.21.15.40:0/3800498467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec700a3300 0x7fec701cb4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:41.729 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.728+0000 7fec77358640 1 --2- 172.21.15.40:0/3800498467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec700a1b70 0x7fec701caf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:41.729 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.728+0000 7fec77358640 1 -- 172.21.15.40:0/3800498467 >> 172.21.15.40:0/3800498467 conn(0x7fec700a4f60 msgr2=0x7fec700a6360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:41.729 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.728+0000 7fec77358640 1 -- 172.21.15.40:0/3800498467 shutdown_connections 2024-04-22T08:12:41.729 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:41.728+0000 7fec77358640 1 -- 172.21.15.40:0/3800498467 wait complete. 2024-04-22T08:12:41.731 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-04-22T08:12:41.731+0000 7f07b4b85640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-22T08:12:41.733 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-04-22T08:12:41.734+0000 7f07b4b85640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-22T08:12:41.737 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-04-22T08:12:44.435 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-04-22T08:12:44.436+0000 7f58e120f640 -1 osd.1 0 log_to_monitors true 2024-04-22T08:12:44.444 INFO:tasks.ceph.osd.5.smithi189.stderr:2024-04-22T08:12:44.444+0000 7f99c700d640 -1 osd.5 0 log_to_monitors true 2024-04-22T08:12:44.489 INFO:tasks.ceph.osd.4.smithi189.stderr:2024-04-22T08:12:44.489+0000 7fb1d0121640 -1 osd.4 0 log_to_monitors true 2024-04-22T08:12:44.489 INFO:tasks.ceph.osd.3.smithi189.stderr:2024-04-22T08:12:44.489+0000 7f2523fb2640 -1 osd.3 0 log_to_monitors true 2024-04-22T08:12:44.490 INFO:tasks.ceph.osd.2.smithi040.stderr:2024-04-22T08:12:44.491+0000 7f16db218640 -1 osd.2 0 log_to_monitors true 2024-04-22T08:12:44.503 INFO:tasks.ceph.osd.0.smithi040.stderr:2024-04-22T08:12:44.504+0000 7f1640c18640 -1 osd.0 0 log_to_monitors true 2024-04-22T08:12:46.014 INFO:tasks.ceph.osd.0.smithi040.stderr:2024-04-22T08:12:46.014+0000 7f163be50640 -1 osd.0 0 waiting for initial osdmap 2024-04-22T08:12:46.014 INFO:tasks.ceph.osd.2.smithi040.stderr:2024-04-22T08:12:46.014+0000 7f16d642c640 -1 osd.2 0 waiting for initial osdmap 2024-04-22T08:12:46.018 INFO:tasks.ceph.osd.0.smithi040.stderr:2024-04-22T08:12:46.019+0000 7f163848b640 -1 osd.0 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T08:12:46.018 INFO:tasks.ceph.osd.2.smithi040.stderr:2024-04-22T08:12:46.019+0000 7f16d2a67640 -1 osd.2 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T08:12:46.020 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-04-22T08:12:46.021+0000 7f58dc2b4640 -1 osd.1 0 waiting for initial osdmap 2024-04-22T08:12:46.021 INFO:tasks.ceph.osd.5.smithi189.stderr:2024-04-22T08:12:46.021+0000 7f99c214c640 -1 osd.5 0 waiting for initial osdmap 2024-04-22T08:12:46.021 INFO:tasks.ceph.osd.3.smithi189.stderr:2024-04-22T08:12:46.021+0000 7f251f71c640 -1 osd.3 0 waiting for initial osdmap 2024-04-22T08:12:46.024 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-04-22T08:12:46.024+0000 7f58d88ef640 -1 osd.1 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T08:12:46.031 INFO:tasks.ceph.osd.5.smithi189.stderr:2024-04-22T08:12:46.032+0000 7f99be787640 -1 osd.5 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T08:12:46.035 INFO:tasks.ceph.osd.3.smithi189.stderr:2024-04-22T08:12:46.035+0000 7f251bd57640 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T08:12:46.816 INFO:tasks.ceph.osd.4.smithi189.stderr:2024-04-22T08:12:46.816+0000 7fb1cb88b640 -1 osd.4 0 waiting for initial osdmap 2024-04-22T08:12:46.821 INFO:tasks.ceph.osd.4.smithi189.stderr:2024-04-22T08:12:46.821+0000 7fb1c3ebe640 -1 osd.4 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T08:12:48.343 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-22T08:12:48.445 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.445+0000 7fa6d3bb9640 10 monclient: get_monmap_and_config 2024-04-22T08:12:48.445 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.445+0000 7fa6d3bb9640 10 monclient: build_initial_monmap 2024-04-22T08:12:48.445 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.445+0000 7fa6d3bb9640 10 monclient: monmap: 2024-04-22T08:12:48.445 INFO:teuthology.misc.health.smithi040.stderr:epoch 0 2024-04-22T08:12:48.446 INFO:teuthology.misc.health.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:48.446 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2024-04-22T08:12:48.446640+0000 2024-04-22T08:12:48.446 INFO:teuthology.misc.health.smithi040.stderr:created 2024-04-22T08:12:48.446640+0000 2024-04-22T08:12:48.446 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:48.446 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:48.446 INFO:teuthology.misc.health.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:12:48.446 INFO:teuthology.misc.health.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:12:48.446 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:12:48.446 INFO:teuthology.misc.health.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:12:48.446 INFO:teuthology.misc.health.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:12:48.446 INFO:teuthology.misc.health.smithi040.stderr: 2024-04-22T08:12:48.448 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.446+0000 7fa6d3bb9640 1 Processor -- start 2024-04-22T08:12:48.448 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.446+0000 7fa6d3bb9640 1 -- start start 2024-04-22T08:12:48.448 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.446+0000 7fa6d3bb9640 10 monclient: init 2024-04-22T08:12:48.449 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.447+0000 7fa6d3bb9640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:48.449 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.447+0000 7fa6d3bb9640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-04-22T08:12:48.449 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.447+0000 7fa6d3bb9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa6cc0a1b70 addr v1:172.21.15.189:6789/0 2024-04-22T08:12:48.449 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.447+0000 7fa6d3bb9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa6cc0a3300 addr v1:172.21.15.40:6789/0 2024-04-22T08:12:48.449 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.447+0000 7fa6d3bb9640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa6cc0a3790 0x7fa6cc13e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:48.449 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.447+0000 7fa6d3bb9640 10 monclient(hunting): picked mon.noname-b con 0x7fa6cc0a3790 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:48.449 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.447+0000 7fa6d3bb9640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6cc0a2660 con 0x7fa6cc0a3300 2024-04-22T08:12:48.449 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.447+0000 7fa6d3bb9640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6cc09fe10 con 0x7fa6cc0a1b70 2024-04-22T08:12:48.449 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.447+0000 7fa6d3bb9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:48.449 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.447+0000 7fa6d3bb9640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6cc072410 con 0x7fa6cc0a3790 2024-04-22T08:12:48.449 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.447+0000 7fa6d3bb9640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:48.449 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.447+0000 7fa6d3bb9640 10 monclient(hunting): authenticate will time out at 842.441159s 2024-04-22T08:12:48.449 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.447+0000 7fa6d1154640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa6cc0a3300 0x7fa6cc13b120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:36780/0 (socket says 172.21.15.40:36780) 2024-04-22T08:12:48.450 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.447+0000 7fa6d2156640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa6cc0a3790 0x7fa6cc13e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:48.450 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.447+0000 7fa6d1154640 1 -- 172.21.15.40:0/1380524011 learned_addr learned my addr 172.21.15.40:0/1380524011 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:12:48.450 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.447+0000 7fa6d2156640 10 monclient(hunting): get_auth_request con 0x7fa6cc0a3790 auth_method 0 2024-04-22T08:12:48.450 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.447+0000 7fa6d2156640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:48.450 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.447+0000 7fa6d2156640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:48.450 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.447+0000 7fa6d2156640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:48.450 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.447+0000 7fa6d2156640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:48.450 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.447+0000 7fa6d2156640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:48.450 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.447+0000 7fa6d2156640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:48.451 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.447+0000 7fa6d2156640 10 monclient(hunting): handle_auth_done global_id 4184 payload 274 2024-04-22T08:12:48.451 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.447+0000 7fa6d2156640 1 -- 172.21.15.40:0/1380524011 >> v1:172.21.15.189:6789/0 conn(0x7fa6cc0a1b70 legacy=0x7fa6cc0a1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:48.451 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.447+0000 7fa6d2156640 1 -- 172.21.15.40:0/1380524011 >> v1:172.21.15.40:6789/0 conn(0x7fa6cc0a3300 legacy=0x7fa6cc13b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:48.451 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.447+0000 7fa6d2156640 10 monclient: _finish_hunting 0 2024-04-22T08:12:48.451 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.447+0000 7fa6d2156640 1 monclient: found mon.noname-b 2024-04-22T08:12:48.451 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.447+0000 7fa6d2156640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:48.451 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.447+0000 7fa6d2156640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:12:48.451 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.447+0000 7fa6d2156640 1 -- 172.21.15.40:0/1380524011 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6cc0a0080 con 0x7fa6cc0a3790 2024-04-22T08:12:48.457 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.450+0000 7fa6d2156640 1 -- 172.21.15.40:0/1380524011 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa6cc0a3790 msgr2=0x7fa6cc13e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:48.457 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.450+0000 7fa6d2156640 1 --2- 172.21.15.40:0/1380524011 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa6cc0a3790 0x7fa6cc13e850 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4009dc0 tx=0x7fa6c402bce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=876a874e280a5cfe server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:48.457 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.450+0000 7fa6d0953640 1 -- 172.21.15.40:0/1380524011 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c402cd80 con 0x7fa6cc0a3790 2024-04-22T08:12:48.458 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.450+0000 7fa6d0953640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:48.458 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.450+0000 7fa6d0953640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:12:48.458 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.450+0000 7fa6d0953640 10 monclient: dump: 2024-04-22T08:12:48.458 INFO:teuthology.misc.health.smithi040.stderr:epoch 1 2024-04-22T08:12:48.458 INFO:teuthology.misc.health.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:48.458 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:48.458 INFO:teuthology.misc.health.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:48.458 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:48.458 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:48.458 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:48.458 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:48.458 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:48.458 INFO:teuthology.misc.health.smithi040.stderr: 2024-04-22T08:12:48.459 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.450+0000 7fa6d0953640 10 monclient: _finish_auth 0 2024-04-22T08:12:48.459 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.450+0000 7fa6d0953640 10 monclient: _check_auth_tickets 2024-04-22T08:12:48.459 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.450+0000 7fa6d0953640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:48.459 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.450+0000 7fa6d3bb9640 5 monclient: authenticate success, global_id 4184 2024-04-22T08:12:48.459 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.450+0000 7fa6d3bb9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:12:48.459 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.450+0000 7fa6d0953640 1 -- 172.21.15.40:0/1380524011 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6c4034440 con 0x7fa6cc0a3790 2024-04-22T08:12:48.459 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.450+0000 7fa6d0953640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:48.459 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.450+0000 7fa6d3bb9640 10 monclient: get_monmap_and_config success 2024-04-22T08:12:48.459 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.450+0000 7fa6d3bb9640 4 set_mon_vals no callback set 2024-04-22T08:12:48.459 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.450+0000 7fa6d0953640 1 -- 172.21.15.40:0/1380524011 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c40347d0 con 0x7fa6cc0a3790 2024-04-22T08:12:48.459 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.450+0000 7fa6d0953640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:48.459 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.450+0000 7fa6d0953640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:12:48.459 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.450+0000 7fa6d0953640 10 monclient: dump: 2024-04-22T08:12:48.459 INFO:teuthology.misc.health.smithi040.stderr:epoch 1 2024-04-22T08:12:48.459 INFO:teuthology.misc.health.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:48.460 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:48.460 INFO:teuthology.misc.health.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:48.460 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:48.460 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:48.460 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:48.460 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:48.460 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:48.460 INFO:teuthology.misc.health.smithi040.stderr: 2024-04-22T08:12:48.460 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.451+0000 7fa6d3bb9640 10 monclient: shutdown 2024-04-22T08:12:48.461 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.451+0000 7fa6d3bb9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:48.461 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.451+0000 7fa6d3bb9640 1 -- 172.21.15.40:0/1380524011 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa6cc0a3790 msgr2=0x7fa6cc13e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:48.461 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.451+0000 7fa6d3bb9640 1 --2- 172.21.15.40:0/1380524011 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa6cc0a3790 0x7fa6cc13e850 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4009dc0 tx=0x7fa6c402bce0 comp rx=0 tx=0).stop 2024-04-22T08:12:48.461 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.451+0000 7fa6d3bb9640 1 -- 172.21.15.40:0/1380524011 shutdown_connections 2024-04-22T08:12:48.461 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.451+0000 7fa6d3bb9640 1 --2- 172.21.15.40:0/1380524011 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa6cc0a3790 0x7fa6cc13e850 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:48.461 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.451+0000 7fa6d3bb9640 1 -- 172.21.15.40:0/1380524011 >> 172.21.15.40:0/1380524011 conn(0x7fa6cc0a4f60 msgr2=0x7fa6cc0a5370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:48.461 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.451+0000 7fa6d3bb9640 1 -- 172.21.15.40:0/1380524011 shutdown_connections 2024-04-22T08:12:48.461 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.451+0000 7fa6d3bb9640 1 -- 172.21.15.40:0/1380524011 wait complete. 2024-04-22T08:12:48.461 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.452+0000 7fa6d3bb9640 10 monclient: build_initial_monmap 2024-04-22T08:12:48.461 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.452+0000 7fa6d3bb9640 10 monclient: monmap: 2024-04-22T08:12:48.461 INFO:teuthology.misc.health.smithi040.stderr:epoch 0 2024-04-22T08:12:48.461 INFO:teuthology.misc.health.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:12:48.461 INFO:teuthology.misc.health.smithi040.stderr:last_changed 0.000000 2024-04-22T08:12:48.461 INFO:teuthology.misc.health.smithi040.stderr:created 0.000000 2024-04-22T08:12:48.462 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:48.462 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:48.462 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:12:48.462 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:12:48.462 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:12:48.462 INFO:teuthology.misc.health.smithi040.stderr: 2024-04-22T08:12:48.462 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.452+0000 7fa6d3bb9640 1 Processor -- start 2024-04-22T08:12:48.462 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.452+0000 7fa6d3bb9640 1 -- start start 2024-04-22T08:12:48.462 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.452+0000 7fa6d3bb9640 10 monclient: init 2024-04-22T08:12:48.462 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.453+0000 7fa6d3bb9640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:48.462 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.453+0000 7fa6d3bb9640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-22T08:12:48.462 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.453+0000 7fa6d3bb9640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa6cc0a1b70 0x7fa6cc13aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:48.462 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.453+0000 7fa6d3bb9640 10 monclient(hunting): picked mon.noname-c con 0x7fa6cc0a1b70 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:48.462 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.453+0000 7fa6d3bb9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6cc0a3300 0x7fa6cc13b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:48.462 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.453+0000 7fa6d3bb9640 10 monclient(hunting): picked mon.noname-b con 0x7fa6cc0a3300 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:48.463 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.453+0000 7fa6d3bb9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6cc0a3790 0x7fa6cc13b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:48.463 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.453+0000 7fa6d3bb9640 10 monclient(hunting): picked mon.noname-a con 0x7fa6cc0a3790 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:48.463 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.453+0000 7fa6d3bb9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:48.464 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.453+0000 7fa6d3bb9640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6cc13cee0 con 0x7fa6cc0a3790 2024-04-22T08:12:48.464 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.453+0000 7fa6d3bb9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:48.464 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.453+0000 7fa6d3bb9640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6cc136c90 con 0x7fa6cc0a3300 2024-04-22T08:12:48.464 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.453+0000 7fa6d3bb9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:48.464 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.453+0000 7fa6d3bb9640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6cc0a2810 con 0x7fa6cc0a1b70 2024-04-22T08:12:48.464 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.453+0000 7fa6d3bb9640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:48.465 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.453+0000 7fa6d3bb9640 10 monclient(hunting): authenticate will time out at 842.446942s 2024-04-22T08:12:48.465 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.453+0000 7fa6d1955640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa6cc0a1b70 0x7fa6cc13aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:48.465 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.453+0000 7fa6d2156640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6cc0a3790 0x7fa6cc13b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:48.465 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.453+0000 7fa6d1955640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa6cc0a1b70 0x7fa6cc13aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:50622/0 (socket says 172.21.15.40:50622) 2024-04-22T08:12:48.465 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.453+0000 7fa6d1955640 1 -- 172.21.15.40:0/2845488191 learned_addr learned my addr 172.21.15.40:0/2845488191 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:48.465 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.453+0000 7fa6d1955640 10 monclient(hunting): get_auth_request con 0x7fa6cc0a1b70 auth_method 0 2024-04-22T08:12:48.465 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.453+0000 7fa6d1955640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:48.465 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.453+0000 7fa6d1955640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:48.465 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.453+0000 7fa6d1955640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:48.465 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.453+0000 7fa6d2156640 10 monclient(hunting): get_auth_request con 0x7fa6cc0a3790 auth_method 0 2024-04-22T08:12:48.465 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.453+0000 7fa6d2156640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:48.465 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.453+0000 7fa6d2156640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:48.465 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.453+0000 7fa6d2156640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:48.466 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.453+0000 7fa6d1955640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:48.466 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.453+0000 7fa6d1955640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:48.466 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.453+0000 7fa6d1955640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:48.466 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.453+0000 7fa6d2156640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:48.466 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.453+0000 7fa6d2156640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:48.466 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.453+0000 7fa6d2156640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:48.466 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.453+0000 7fa6d1154640 1 --2- 172.21.15.40:0/2845488191 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6cc0a3300 0x7fa6cc13b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:48.466 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.453+0000 7fa6d1955640 10 monclient(hunting): handle_auth_done global_id 4187 payload 824 2024-04-22T08:12:48.466 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.453+0000 7fa6d1955640 1 -- 172.21.15.40:0/2845488191 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6cc0a3300 msgr2=0x7fa6cc13b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:48.466 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.453+0000 7fa6d1955640 1 --2- 172.21.15.40:0/2845488191 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6cc0a3300 0x7fa6cc13b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:48.467 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.453+0000 7fa6d1955640 1 -- 172.21.15.40:0/2845488191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6cc0a3790 msgr2=0x7fa6cc13b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:48.467 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.453+0000 7fa6d1955640 1 --2- 172.21.15.40:0/2845488191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6cc0a3790 0x7fa6cc13b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:48.467 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.453+0000 7fa6d1955640 10 monclient: _finish_hunting 0 2024-04-22T08:12:48.467 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.453+0000 7fa6d1955640 1 monclient: found mon.noname-c 2024-04-22T08:12:48.467 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.453+0000 7fa6d1955640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:48.467 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.453+0000 7fa6d1955640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:12:48.467 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.453+0000 7fa6d1955640 1 -- 172.21.15.40:0/2845488191 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c4009a70 con 0x7fa6cc0a1b70 2024-04-22T08:12:48.467 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.455+0000 7fa6d1955640 1 -- 172.21.15.40:0/2845488191 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa6cc0a1b70 msgr2=0x7fa6cc13aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:48.467 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.455+0000 7fa6d1955640 1 --2- 172.21.15.40:0/2845488191 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa6cc0a1b70 0x7fa6cc13aaf0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa6c800c430 tx=0x7fa6c800c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:48.467 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.455+0000 7fa6c27fc640 1 -- 172.21.15.40:0/2845488191 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c80119a0 con 0x7fa6cc0a1b70 2024-04-22T08:12:48.467 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.456+0000 7fa6c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:48.467 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.456+0000 7fa6c27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:12:48.467 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.456+0000 7fa6c27fc640 10 monclient: dump: 2024-04-22T08:12:48.467 INFO:teuthology.misc.health.smithi040.stderr:epoch 1 2024-04-22T08:12:48.468 INFO:teuthology.misc.health.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:48.468 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:48.468 INFO:teuthology.misc.health.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:48.468 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:48.468 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:48.468 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:48.468 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:48.468 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:48.468 INFO:teuthology.misc.health.smithi040.stderr: 2024-04-22T08:12:48.468 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.456+0000 7fa6c27fc640 10 monclient: _finish_auth 0 2024-04-22T08:12:48.468 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.456+0000 7fa6c27fc640 10 monclient: _check_auth_tickets 2024-04-22T08:12:48.468 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.456+0000 7fa6c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:48.468 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.456+0000 7fa6c27fc640 1 -- 172.21.15.40:0/2845488191 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6c8011b00 con 0x7fa6cc0a1b70 2024-04-22T08:12:48.468 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.456+0000 7fa6d3bb9640 5 monclient: authenticate success, global_id 4187 2024-04-22T08:12:48.469 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.456+0000 7fa6d3bb9640 10 monclient: _renew_subs 2024-04-22T08:12:48.469 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.456+0000 7fa6d3bb9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:48.469 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.456+0000 7fa6d3bb9640 1 -- 172.21.15.40:0/2845488191 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6cc11ef30 con 0x7fa6cc0a1b70 2024-04-22T08:12:48.469 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.456+0000 7fa6c27fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:48.469 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.456+0000 7fa6c27fc640 1 -- 172.21.15.40:0/2845488191 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c8012420 con 0x7fa6cc0a1b70 2024-04-22T08:12:48.469 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.456+0000 7fa6c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:48.469 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.456+0000 7fa6c27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:12:48.469 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.456+0000 7fa6c27fc640 10 monclient: dump: 2024-04-22T08:12:48.469 INFO:teuthology.misc.health.smithi040.stderr:epoch 1 2024-04-22T08:12:48.469 INFO:teuthology.misc.health.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:48.469 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:48.469 INFO:teuthology.misc.health.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:48.469 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:48.469 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:48.469 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:48.470 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:48.470 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:48.470 INFO:teuthology.misc.health.smithi040.stderr: 2024-04-22T08:12:48.470 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.456+0000 7fa6c3fff640 4 set_mon_vals no callback set 2024-04-22T08:12:48.470 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.456+0000 7fa6d3bb9640 10 monclient: _renew_subs 2024-04-22T08:12:48.470 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.456+0000 7fa6d3bb9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:48.470 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.456+0000 7fa6d3bb9640 1 -- 172.21.15.40:0/2845488191 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6cc13cc60 con 0x7fa6cc0a1b70 2024-04-22T08:12:48.470 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.456+0000 7fa6c27fc640 1 -- 172.21.15.40:0/2845488191 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 4) v1 ==== 44726+0+0 (secure 0 0 0) 0x7fa6c8012640 con 0x7fa6cc0a1b70 2024-04-22T08:12:48.470 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.457+0000 7fa6c27fc640 1 --2- 172.21.15.40:0/2845488191 >> v2:172.21.15.189:6800/30168 conn(0x7fa6b0037790 0x7fa6b0039c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:48.470 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.457+0000 7fa6d3bb9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:48.470 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.457+0000 7fa6d3bb9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:48.470 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.457+0000 7fa6c27fc640 1 -- 172.21.15.40:0/2845488191 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(11..11 src has 1..11) v4 ==== 2767+0+0 (secure 0 0 0) 0x7fa6c8049d40 con 0x7fa6cc0a1b70 2024-04-22T08:12:48.470 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.457+0000 7fa6d3bb9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:48.470 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.457+0000 7fa6d3bb9640 1 -- 172.21.15.40:0/2845488191 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6cc069650 con 0x7fa6cc0a1b70 2024-04-22T08:12:48.471 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.457+0000 7fa6d1154640 1 --2- 172.21.15.40:0/2845488191 >> v2:172.21.15.189:6800/30168 conn(0x7fa6b0037790 0x7fa6b0039c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:48.471 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.457+0000 7fa6d1154640 10 monclient: get_auth_request con 0x7fa6b0037790 auth_method 0 2024-04-22T08:12:48.471 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.457+0000 7fa6d1154640 1 -- 172.21.15.40:0/2845488191 >> v2:172.21.15.189:6800/30168 conn(0x7fa6b0037790 msgr2=0x7fa6b0039c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-22T08:12:48.471 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.457+0000 7fa6d1154640 1 -- 172.21.15.40:0/2845488191 >> v2:172.21.15.189:6800/30168 conn(0x7fa6b0037790 msgr2=0x7fa6b0039c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:12:48.471 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.457+0000 7fa6d1154640 1 --2- 172.21.15.40:0/2845488191 >> v2:172.21.15.189:6800/30168 conn(0x7fa6b0037790 0x7fa6b0039c50 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc136140 tx=0x7fa6bc009040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:12:48.471 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.457+0000 7fa6d1154640 1 --2- 172.21.15.40:0/2845488191 >> v2:172.21.15.189:6800/30168 conn(0x7fa6b0037790 0x7fa6b0039c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:12:48.471 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.462+0000 7fa6c27fc640 1 -- 172.21.15.40:0/2845488191 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7fa6c801c460 con 0x7fa6cc0a1b70 2024-04-22T08:12:48.471 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.462+0000 7fa6c27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:48.472 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.462+0000 7fa6c27fc640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:12:48.644 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.645+0000 7fa6d3bb9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-22T08:12:48.644 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.645+0000 7fa6d3bb9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-22T08:12:48.644 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.645+0000 7fa6d3bb9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:48.644 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.645+0000 7fa6d3bb9640 1 -- 172.21.15.40:0/2845488191 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa6cc0008d0 con 0x7fa6cc0a1b70 2024-04-22T08:12:48.645 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.645+0000 7fa6c27fc640 1 -- 172.21.15.40:0/2845488191 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+6816 (secure 0 0 0) 0x7fa6c8010750 con 0x7fa6cc0a1b70 2024-04-22T08:12:48.645 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.645+0000 7fa6c27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-22T08:12:48.645 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.645+0000 7fa6c27fc640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:12:48.645 INFO:teuthology.misc.health.smithi040.stdout: 2024-04-22T08:12:48.645 INFO:teuthology.misc.health.smithi040.stdout:{"epoch":11,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","created":"2024-04-22T08:12:36.003043+0000","modified":"2024-04-22T08:12:47.016043+0000","last_up_change":"2024-04-22T08:12:47.016043+0000","last_in_change":"2024-04-22T08:12:40.364359+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b9a0210c-5881-4ca0-8318-a4a987109405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":30669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6805","nonce":30669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6807","nonce":30669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":30669}]},"public_addr":"172.21.15.40:6804/30669","cluster_addr":"172.21.15.40:6805/30669","heartbeat_back_addr":"172.21.15.40:6807/30669","heartbeat_front_addr":"172.21.15.40:6806/30669","state":["exists","up"]},{"osd":1,"uuid":"11370cc8-09a6-4c35-b92c-4d926c6c18d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":30670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6801","nonce":30670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6803","nonce":30670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":30670}]},"public_addr":"172.21.15.40:6800/30670","cluster_addr":"172.21.15.40:6801/30670","heartbeat_back_addr":"172.21.15.40:6803/30670","heartbeat_front_addr":"172.21.15.40:6802/30670","state":["exists","up"]},{"osd":2,"uuid":"d03db34e-f630-4a3e-86fb-7234bee0d0f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":30671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":30671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6811","nonce":30671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":30671}]},"public_addr":"172.21.15.40:6808/30671","cluster_addr":"172.21.15.40:6809/30671","heartbeat_back_addr":"172.21.15.40:6811/30671","heartbeat_front_addr":"172.21.15.40:6810/30671","state":["exists","up"]},{"osd":3,"uuid":"d94b2052-5847-4955-b1d4-b7e627572811","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6805","nonce":30673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6807","nonce":30673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6811","nonce":30673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6809","nonce":30673}]},"public_addr":"172.21.15.189:6805/30673","cluster_addr":"172.21.15.189:6807/30673","heartbeat_back_addr":"172.21.15.189:6811/30673","heartbeat_front_addr":"172.21.15.189:6809/30673","state":["exists","up"]},{"osd":4,"uuid":"cb626202-7411-4299-8d20-988bcc9103f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6801","nonce":30672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":30672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":30672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6803","nonce":30672}]},"public_addr":"172.21.15.189:6801/30672","cluster_addr":"172.21.15.189:6802/30672","heartbeat_back_addr":"172.21.15.189:6804/30672","heartbeat_front_addr":"172.21.15.189:6803/30672","state":["exists","up"]},{"osd":5,"uuid":"abbac246-54d6-47d7-a41c-a68df0b80ddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":30674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":30674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":30674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":30674}]},"public_addr":"172.21.15.189:6806/30674","cluster_addr":"172.21.15.189:6808/30674","heartbeat_back_addr":"172.21.15.189:6812/30674","heartbeat_front_addr":"172.21.15.189:6810/30674","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T08:12:48.650 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.647+0000 7fa6d3bb9640 1 -- 172.21.15.40:0/2845488191 >> v2:172.21.15.189:6800/30168 conn(0x7fa6b0037790 msgr2=0x7fa6b0039c50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:12:48.650 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.647+0000 7fa6d3bb9640 1 --2- 172.21.15.40:0/2845488191 >> v2:172.21.15.189:6800/30168 conn(0x7fa6b0037790 0x7fa6b0039c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:48.651 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.647+0000 7fa6d3bb9640 10 monclient: shutdown 2024-04-22T08:12:48.651 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.647+0000 7fa6d3bb9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:48.651 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.647+0000 7fa6d3bb9640 1 -- 172.21.15.40:0/2845488191 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa6cc0a1b70 msgr2=0x7fa6cc13aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:48.651 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.647+0000 7fa6d3bb9640 1 --2- 172.21.15.40:0/2845488191 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa6cc0a1b70 0x7fa6cc13aaf0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa6c800c430 tx=0x7fa6c800c8f0 comp rx=0 tx=0).stop 2024-04-22T08:12:48.651 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.647+0000 7fa6d3bb9640 1 -- 172.21.15.40:0/2845488191 shutdown_connections 2024-04-22T08:12:48.651 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.647+0000 7fa6d3bb9640 1 --2- 172.21.15.40:0/2845488191 >> v2:172.21.15.189:6800/30168 conn(0x7fa6b0037790 0x7fa6b0039c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:48.651 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.647+0000 7fa6d3bb9640 1 --2- 172.21.15.40:0/2845488191 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6cc0a3300 0x7fa6cc13b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:48.651 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.647+0000 7fa6d3bb9640 1 --2- 172.21.15.40:0/2845488191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6cc0a3790 0x7fa6cc13b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:48.651 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.647+0000 7fa6d3bb9640 1 --2- 172.21.15.40:0/2845488191 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa6cc0a1b70 0x7fa6cc13aaf0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:48.651 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.647+0000 7fa6d3bb9640 1 -- 172.21.15.40:0/2845488191 >> 172.21.15.40:0/2845488191 conn(0x7fa6cc0a4f60 msgr2=0x7fa6cc06f760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:48.651 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.647+0000 7fa6d3bb9640 1 -- 172.21.15.40:0/2845488191 shutdown_connections 2024-04-22T08:12:48.651 INFO:teuthology.misc.health.smithi040.stderr:2024-04-22T08:12:48.648+0000 7fa6d3bb9640 1 -- 172.21.15.40:0/2845488191 wait complete. 2024-04-22T08:12:48.657 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-04-22T08:12:48.657 INFO:tasks.ceph:Creating RBD pool 2024-04-22T08:12:48.657 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-22T08:12:48.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.756+0000 7f4ed5fb4640 10 monclient: get_monmap_and_config 2024-04-22T08:12:48.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.756+0000 7f4ed5fb4640 10 monclient: build_initial_monmap 2024-04-22T08:12:48.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.756+0000 7f4ed5fb4640 10 monclient: monmap: 2024-04-22T08:12:48.756 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:48.756 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:48.756 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:48.756956+0000 2024-04-22T08:12:48.756 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:48.756956+0000 2024-04-22T08:12:48.756 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:48.756 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:48.756 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:12:48.756 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:12:48.756 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:12:48.756 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:12:48.756 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:12:48.757 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:48.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.756+0000 7f4ed5fb4640 1 Processor -- start 2024-04-22T08:12:48.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.756+0000 7f4ed5fb4640 1 -- start start 2024-04-22T08:12:48.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.756+0000 7f4ed5fb4640 10 monclient: init 2024-04-22T08:12:48.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.757+0000 7f4ed5fb4640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:48.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.757+0000 7f4ed5fb4640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-04-22T08:12:48.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.757+0000 7f4ed5fb4640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4ed00a19a0 0x7f4ed00999f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:48.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.757+0000 7f4ed5fb4640 10 monclient(hunting): picked mon.noname-c con 0x7f4ed00a19a0 addr v2:172.21.15.189:3300/0 2024-04-22T08:12:48.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.757+0000 7f4ed5fb4640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4ed009dc10 0x7f4ed00a1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:48.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.757+0000 7f4ed5fb4640 10 monclient(hunting): picked mon.noname-a con 0x7f4ed009dc10 addr v2:172.21.15.40:3300/0 2024-04-22T08:12:48.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.757+0000 7f4ed5fb4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4ed00a22c0 addr v1:172.21.15.40:6789/0 2024-04-22T08:12:48.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.757+0000 7f4ed5fb4640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed00a0400 con 0x7f4ed00a22c0 2024-04-22T08:12:48.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.757+0000 7f4ed5fb4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:48.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.757+0000 7f4ed5fb4640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ed009f0c0 con 0x7f4ed009dc10 2024-04-22T08:12:48.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.757+0000 7f4ed5fb4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:48.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.757+0000 7f4ed5fb4640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ed009f3a0 con 0x7f4ed00a19a0 2024-04-22T08:12:48.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.757+0000 7f4ed5fb4640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:48.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.757+0000 7f4ed5fb4640 10 monclient(hunting): authenticate will time out at 842.751160s 2024-04-22T08:12:48.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.757+0000 7f4eceffd640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4ed009dc10 0x7f4ed00a1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:48.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.757+0000 7f4ecffff640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f4ed00a22c0 0x7f4ed0135b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:36790/0 (socket says 172.21.15.40:36790) 2024-04-22T08:12:48.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.757+0000 7f4eceffd640 10 monclient(hunting): get_auth_request con 0x7f4ed009dc10 auth_method 0 2024-04-22T08:12:48.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.757+0000 7f4ecffff640 1 -- 172.21.15.40:0/136376834 learned_addr learned my addr 172.21.15.40:0/136376834 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:12:48.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.757+0000 7f4eceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:48.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.757+0000 7f4eceffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:48.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.757+0000 7f4eceffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:48.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.757+0000 7f4eceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:48.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.757+0000 7f4eceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:48.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.757+0000 7f4eceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:48.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.757+0000 7f4eceffd640 10 monclient(hunting): handle_auth_done global_id 4197 payload 274 2024-04-22T08:12:48.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.757+0000 7f4eceffd640 1 -- 172.21.15.40:0/136376834 >> v2:172.21.15.189:3300/0 conn(0x7f4ed00a19a0 msgr2=0x7f4ed00999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:48.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.757+0000 7f4eceffd640 1 --2- 172.21.15.40:0/136376834 >> v2:172.21.15.189:3300/0 conn(0x7f4ed00a19a0 0x7f4ed00999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:48.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.757+0000 7f4eceffd640 1 -- 172.21.15.40:0/136376834 >> v1:172.21.15.40:6789/0 conn(0x7f4ed00a22c0 legacy=0x7f4ed0135b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:48.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.757+0000 7f4eceffd640 10 monclient: _finish_hunting 0 2024-04-22T08:12:48.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.757+0000 7f4eceffd640 1 monclient: found mon.noname-a 2024-04-22T08:12:48.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.757+0000 7f4eceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:48.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.757+0000 7f4eceffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:12:48.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.757+0000 7f4eceffd640 1 -- 172.21.15.40:0/136376834 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed00a2750 con 0x7f4ed009dc10 2024-04-22T08:12:48.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.760+0000 7f4eceffd640 1 -- 172.21.15.40:0/136376834 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ed009dc10 msgr2=0x7f4ed00a1d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:48.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.760+0000 7f4eceffd640 1 --2- 172.21.15.40:0/136376834 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ed009dc10 0x7f4ed00a1d80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4ec40097a0 tx=0x7f4ec402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ccaefaa0b4fa11a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:48.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.761+0000 7f4ece7fc640 1 -- 172.21.15.40:0/136376834 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec402f050 con 0x7f4ed009dc10 2024-04-22T08:12:48.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.761+0000 7f4ece7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:48.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.761+0000 7f4ece7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:12:48.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.761+0000 7f4ece7fc640 10 monclient: dump: 2024-04-22T08:12:48.769 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:48.769 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:48.769 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:48.769 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:48.769 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:48.769 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:48.769 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:48.769 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:48.769 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:48.769 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:48.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.761+0000 7f4ece7fc640 10 monclient: _finish_auth 0 2024-04-22T08:12:48.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.761+0000 7f4ece7fc640 10 monclient: _check_auth_tickets 2024-04-22T08:12:48.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.761+0000 7f4ece7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:48.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.761+0000 7f4ece7fc640 1 -- 172.21.15.40:0/136376834 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ec40319f0 con 0x7f4ed009dc10 2024-04-22T08:12:48.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.761+0000 7f4ed5fb4640 5 monclient: authenticate success, global_id 4197 2024-04-22T08:12:48.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.761+0000 7f4ed5fb4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:12:48.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.761+0000 7f4ece7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:48.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.761+0000 7f4ed5fb4640 10 monclient: get_monmap_and_config success 2024-04-22T08:12:48.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.761+0000 7f4ece7fc640 1 -- 172.21.15.40:0/136376834 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec4031d80 con 0x7f4ed009dc10 2024-04-22T08:12:48.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.761+0000 7f4ed5fb4640 4 set_mon_vals no callback set 2024-04-22T08:12:48.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.761+0000 7f4ece7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:48.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.761+0000 7f4ece7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:12:48.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.761+0000 7f4ece7fc640 10 monclient: dump: 2024-04-22T08:12:48.770 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:48.771 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:48.771 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:48.771 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:48.771 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:48.771 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:48.771 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:48.771 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:48.771 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:48.771 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:48.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.761+0000 7f4ed5fb4640 10 monclient: shutdown 2024-04-22T08:12:48.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.761+0000 7f4ed5fb4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:48.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.761+0000 7f4ed5fb4640 1 -- 172.21.15.40:0/136376834 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ed009dc10 msgr2=0x7f4ed00a1d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:48.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.761+0000 7f4ed5fb4640 1 --2- 172.21.15.40:0/136376834 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ed009dc10 0x7f4ed00a1d80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4ec40097a0 tx=0x7f4ec402b7a0 comp rx=0 tx=0).stop 2024-04-22T08:12:48.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.761+0000 7f4ed5fb4640 1 -- 172.21.15.40:0/136376834 shutdown_connections 2024-04-22T08:12:48.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.761+0000 7f4ed5fb4640 1 --2- 172.21.15.40:0/136376834 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ed009dc10 0x7f4ed00a1d80 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:48.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.761+0000 7f4ed5fb4640 1 --2- 172.21.15.40:0/136376834 >> v2:172.21.15.189:3300/0 conn(0x7f4ed00a19a0 0x7f4ed00999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:48.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.761+0000 7f4ed5fb4640 1 -- 172.21.15.40:0/136376834 >> 172.21.15.40:0/136376834 conn(0x7f4ed00a65c0 msgr2=0x7f4ed00a2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:48.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.761+0000 7f4ed5fb4640 1 -- 172.21.15.40:0/136376834 shutdown_connections 2024-04-22T08:12:48.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.761+0000 7f4ed5fb4640 1 -- 172.21.15.40:0/136376834 wait complete. 2024-04-22T08:12:48.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.762+0000 7f4ed5fb4640 10 monclient: build_initial_monmap 2024-04-22T08:12:48.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.762+0000 7f4ed5fb4640 10 monclient: monmap: 2024-04-22T08:12:48.772 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:48.772 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:12:48.772 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:12:48.772 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:12:48.772 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:48.773 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:48.773 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:12:48.773 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:12:48.773 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:12:48.773 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:48.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.762+0000 7f4ed5fb4640 1 Processor -- start 2024-04-22T08:12:48.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.762+0000 7f4ed5fb4640 1 -- start start 2024-04-22T08:12:48.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.762+0000 7f4ed5fb4640 10 monclient: init 2024-04-22T08:12:48.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4ed5fb4640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:48.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4ed5fb4640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:12:48.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4ed5fb4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ed009dc10 0x7f4ed01caca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:48.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4ed5fb4640 10 monclient(hunting): picked mon.noname-a con 0x7f4ed009dc10 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:48.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4ed5fb4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ed00a19a0 0x7f4ed01cb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:48.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4ed5fb4640 10 monclient(hunting): picked mon.noname-b con 0x7f4ed00a19a0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:48.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4ed5fb4640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4ed00a22c0 0x7f4ed0122270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:48.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4ed5fb4640 10 monclient(hunting): picked mon.noname-c con 0x7f4ed00a22c0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:48.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4ed5fb4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:48.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4ed5fb4640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed01249f0 con 0x7f4ed009dc10 2024-04-22T08:12:48.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4ed5fb4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:48.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4ed5fb4640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed01243a0 con 0x7f4ed00a19a0 2024-04-22T08:12:48.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4ed5fb4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:48.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4ed5fb4640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed0072050 con 0x7f4ed00a22c0 2024-04-22T08:12:48.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4ed5fb4640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:48.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4ed5fb4640 10 monclient(hunting): authenticate will time out at 842.757030s 2024-04-22T08:12:48.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4ecf7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ed009dc10 0x7f4ed01caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:48.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4ecffff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4ed00a22c0 0x7f4ed0122270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:48.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4ecf7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ed009dc10 0x7f4ed01caca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57716/0 (socket says 172.21.15.40:57716) 2024-04-22T08:12:48.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4ecf7fe640 1 -- 172.21.15.40:0/2230065877 learned_addr learned my addr 172.21.15.40:0/2230065877 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:48.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4ecf7fe640 10 monclient(hunting): get_auth_request con 0x7f4ed009dc10 auth_method 0 2024-04-22T08:12:48.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4ecf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:48.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4ecf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:48.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4ecf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:48.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4ecffff640 10 monclient(hunting): get_auth_request con 0x7f4ed00a22c0 auth_method 0 2024-04-22T08:12:48.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4ecffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:48.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4ecffff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:48.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4ecffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:48.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4ecf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:48.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4ecf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:48.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4ecf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:48.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4ecffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:48.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4ecffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:48.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4ecffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:48.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4eceffd640 1 --2- 172.21.15.40:0/2230065877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ed00a19a0 0x7f4ed01cb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:48.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4eceffd640 10 monclient(hunting): get_auth_request con 0x7f4ed00a19a0 auth_method 0 2024-04-22T08:12:48.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4eceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:48.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4eceffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:48.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4eceffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:48.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4ecf7fe640 10 monclient(hunting): handle_auth_done global_id 4200 payload 824 2024-04-22T08:12:48.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4ecf7fe640 1 -- 172.21.15.40:0/2230065877 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4ed00a22c0 msgr2=0x7f4ed0122270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:48.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4ecf7fe640 1 --2- 172.21.15.40:0/2230065877 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4ed00a22c0 0x7f4ed0122270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:48.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4ecf7fe640 1 -- 172.21.15.40:0/2230065877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ed00a19a0 msgr2=0x7f4ed01cb1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:48.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4ecf7fe640 1 --2- 172.21.15.40:0/2230065877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ed00a19a0 0x7f4ed01cb1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:48.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4ecf7fe640 10 monclient: _finish_hunting 0 2024-04-22T08:12:48.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4ecf7fe640 1 monclient: found mon.noname-a 2024-04-22T08:12:48.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4ecf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:48.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4ecf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:12:48.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4ecf7fe640 1 -- 172.21.15.40:0/2230065877 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ec4009400 con 0x7f4ed009dc10 2024-04-22T08:12:48.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.763+0000 7f4ecffff640 1 --2- 172.21.15.40:0/2230065877 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4ed00a22c0 0x7f4ed0122270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:12:48.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.766+0000 7f4ecf7fe640 1 -- 172.21.15.40:0/2230065877 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ed009dc10 msgr2=0x7f4ed01caca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:48.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.766+0000 7f4ecf7fe640 1 --2- 172.21.15.40:0/2230065877 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ed009dc10 0x7f4ed01caca0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4ec000b4a0 tx=0x7f4ec000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:48.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.766+0000 7f4eb7fff640 1 -- 172.21.15.40:0/2230065877 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec0010820 con 0x7f4ed009dc10 2024-04-22T08:12:48.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.766+0000 7f4eb7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:48.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.766+0000 7f4eb7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:12:48.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.766+0000 7f4eb7fff640 10 monclient: dump: 2024-04-22T08:12:48.777 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:48.777 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:48.777 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:48.777 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:48.778 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:48.778 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:48.778 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:48.778 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:48.778 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:48.778 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:48.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.766+0000 7f4eb7fff640 10 monclient: _finish_auth 0 2024-04-22T08:12:48.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.766+0000 7f4eb7fff640 10 monclient: _check_auth_tickets 2024-04-22T08:12:48.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.766+0000 7f4eb7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:48.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.766+0000 7f4eb7fff640 1 -- 172.21.15.40:0/2230065877 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ec0010980 con 0x7f4ed009dc10 2024-04-22T08:12:48.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.766+0000 7f4ed5fb4640 5 monclient: authenticate success, global_id 4200 2024-04-22T08:12:48.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.766+0000 7f4ed5fb4640 10 monclient: _renew_subs 2024-04-22T08:12:48.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.766+0000 7f4ed5fb4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:48.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.766+0000 7f4ed5fb4640 1 -- 172.21.15.40:0/2230065877 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ed009ce70 con 0x7f4ed009dc10 2024-04-22T08:12:48.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.766+0000 7f4eb7fff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:48.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.766+0000 7f4eb7fff640 1 -- 172.21.15.40:0/2230065877 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec0010d30 con 0x7f4ed009dc10 2024-04-22T08:12:48.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.766+0000 7f4eb7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:48.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.766+0000 7f4eb7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:12:48.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.766+0000 7f4eb7fff640 10 monclient: dump: 2024-04-22T08:12:48.779 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:48.779 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:48.779 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:48.779 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:48.780 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:48.780 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:48.780 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:48.780 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:48.780 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:48.780 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:48.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.766+0000 7f4ed5fb4640 10 monclient: _renew_subs 2024-04-22T08:12:48.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.766+0000 7f4ed5fb4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:48.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.766+0000 7f4ed5fb4640 1 -- 172.21.15.40:0/2230065877 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ed01261f0 con 0x7f4ed009dc10 2024-04-22T08:12:48.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.766+0000 7f4ecdffb640 4 set_mon_vals no callback set 2024-04-22T08:12:48.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.767+0000 7f4eb7fff640 1 -- 172.21.15.40:0/2230065877 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 4) v1 ==== 44726+0+0 (secure 0 0 0) 0x7f4ec0017360 con 0x7f4ed009dc10 2024-04-22T08:12:48.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.767+0000 7f4ed5fb4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:48.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.767+0000 7f4ed5fb4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:48.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.767+0000 7f4ed5fb4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:48.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.767+0000 7f4ed5fb4640 1 -- 172.21.15.40:0/2230065877 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ed0125b00 con 0x7f4ed009dc10 2024-04-22T08:12:48.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.767+0000 7f4eb7fff640 1 --2- 172.21.15.40:0/2230065877 >> v2:172.21.15.189:6800/30168 conn(0x7f4ebc037740 0x7f4ebc039c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:48.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.767+0000 7f4eb7fff640 1 -- 172.21.15.40:0/2230065877 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(11..11 src has 1..11) v4 ==== 2767+0+0 (secure 0 0 0) 0x7f4ec0049b90 con 0x7f4ed009dc10 2024-04-22T08:12:48.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.767+0000 7f4eceffd640 1 --2- 172.21.15.40:0/2230065877 >> v2:172.21.15.189:6800/30168 conn(0x7f4ebc037740 0x7f4ebc039c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:48.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.767+0000 7f4eceffd640 10 monclient: get_auth_request con 0x7f4ebc037740 auth_method 0 2024-04-22T08:12:48.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.770+0000 7f4eceffd640 1 --2- 172.21.15.40:0/2230065877 >> v2:172.21.15.189:6800/30168 conn(0x7f4ebc037740 0x7f4ebc039c00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4ed009fe30 tx=0x7f4ec4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:48.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.772+0000 7f4eb7fff640 1 -- 172.21.15.40:0/2230065877 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f4ec000d300 con 0x7f4ed009dc10 2024-04-22T08:12:48.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.772+0000 7f4eb7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:48.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.772+0000 7f4eb7fff640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:12:48.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.955+0000 7f4ed5fb4640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-04-22T08:12:48.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.955+0000 7f4ed5fb4640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-04-22T08:12:48.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.955+0000 7f4ed5fb4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:48.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:48.955+0000 7f4ed5fb4640 1 -- 172.21.15.40:0/2230065877 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f4ed0125950 con 0x7f4ed009dc10 2024-04-22T08:12:49.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.827+0000 7f4eb7fff640 1 -- 172.21.15.40:0/2230065877 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v13) v1 ==== 109+0+0 (secure 0 0 0) 0x7f4ec0014020 con 0x7f4ed009dc10 2024-04-22T08:12:49.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.827+0000 7f4eb7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-04-22T08:12:49.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.827+0000 7f4eb7fff640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-04-22T08:12:49.827 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'rbd' created 2024-04-22T08:12:49.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.830+0000 7f4ed5fb4640 1 -- 172.21.15.40:0/2230065877 >> v2:172.21.15.189:6800/30168 conn(0x7f4ebc037740 msgr2=0x7f4ebc039c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:49.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.830+0000 7f4ed5fb4640 1 --2- 172.21.15.40:0/2230065877 >> v2:172.21.15.189:6800/30168 conn(0x7f4ebc037740 0x7f4ebc039c00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4ed009fe30 tx=0x7f4ec4008030 comp rx=0 tx=0).stop 2024-04-22T08:12:49.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.830+0000 7f4ed5fb4640 10 monclient: shutdown 2024-04-22T08:12:49.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.830+0000 7f4ed5fb4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:49.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.830+0000 7f4ed5fb4640 1 -- 172.21.15.40:0/2230065877 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ed009dc10 msgr2=0x7f4ed01caca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:49.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.830+0000 7f4ed5fb4640 1 --2- 172.21.15.40:0/2230065877 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ed009dc10 0x7f4ed01caca0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4ec000b4a0 tx=0x7f4ec000b960 comp rx=0 tx=0).stop 2024-04-22T08:12:49.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.831+0000 7f4ed5fb4640 1 -- 172.21.15.40:0/2230065877 shutdown_connections 2024-04-22T08:12:49.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.831+0000 7f4ed5fb4640 1 --2- 172.21.15.40:0/2230065877 >> v2:172.21.15.189:6800/30168 conn(0x7f4ebc037740 0x7f4ebc039c00 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:49.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.831+0000 7f4ed5fb4640 1 --2- 172.21.15.40:0/2230065877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ed00a19a0 0x7f4ed01cb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:49.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.831+0000 7f4ed5fb4640 1 --2- 172.21.15.40:0/2230065877 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4ed00a22c0 0x7f4ed0122270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:49.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.831+0000 7f4ed5fb4640 1 --2- 172.21.15.40:0/2230065877 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ed009dc10 0x7f4ed01caca0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:49.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.831+0000 7f4ed5fb4640 1 -- 172.21.15.40:0/2230065877 >> 172.21.15.40:0/2230065877 conn(0x7f4ed00a65c0 msgr2=0x7f4ed009a470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:49.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.831+0000 7f4ed5fb4640 1 -- 172.21.15.40:0/2230065877 shutdown_connections 2024-04-22T08:12:49.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.831+0000 7f4ed5fb4640 1 -- 172.21.15.40:0/2230065877 wait complete. 2024-04-22T08:12:49.843 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-22T08:12:49.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.950+0000 7f4c47044640 10 monclient: get_monmap_and_config 2024-04-22T08:12:49.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.950+0000 7f4c47044640 10 monclient: build_initial_monmap 2024-04-22T08:12:49.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.950+0000 7f4c47044640 10 monclient: monmap: 2024-04-22T08:12:49.950 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:49.950 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:49.950 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:49.950942+0000 2024-04-22T08:12:49.950 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:49.950942+0000 2024-04-22T08:12:49.950 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:49.950 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:49.950 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:12:49.950 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:12:49.950 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:12:49.950 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:12:49.951 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:12:49.951 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:49.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.950+0000 7f4c47044640 1 Processor -- start 2024-04-22T08:12:49.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.950+0000 7f4c47044640 1 -- start start 2024-04-22T08:12:49.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.950+0000 7f4c47044640 10 monclient: init 2024-04-22T08:12:49.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c47044640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:49.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c47044640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-04-22T08:12:49.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c47044640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4c40135f70 0x7f4c400a0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:49.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c47044640 10 monclient(hunting): picked mon.noname-b con 0x7f4c40135f70 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:49.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c47044640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4c4009f070 0x7f4c40136350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:49.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c47044640 10 monclient(hunting): picked mon.noname-c con 0x7f4c4009f070 addr v2:172.21.15.189:3300/0 2024-04-22T08:12:49.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c47044640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4c400a2df0 0x7f4c400a3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:49.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c47044640 10 monclient(hunting): picked mon.noname-a con 0x7f4c400a2df0 addr v2:172.21.15.40:3300/0 2024-04-22T08:12:49.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c47044640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:49.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c47044640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c4009fa90 con 0x7f4c400a2df0 2024-04-22T08:12:49.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c47044640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:49.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c47044640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c4009fd70 con 0x7f4c4009f070 2024-04-22T08:12:49.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c47044640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:49.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c47044640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c400a43a0 con 0x7f4c40135f70 2024-04-22T08:12:49.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c47044640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:49.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c47044640 10 monclient(hunting): authenticate will time out at 843.945182s 2024-04-22T08:12:49.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c455e1640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4c400a2df0 0x7f4c400a3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:49.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c44de0640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4c40135f70 0x7f4c400a0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:49.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c455e1640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4c400a2df0 0x7f4c400a3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57734/0 (socket says 172.21.15.40:57734) 2024-04-22T08:12:49.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c44de0640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4c40135f70 0x7f4c400a0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49564/0 (socket says 172.21.15.40:49564) 2024-04-22T08:12:49.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c455e1640 1 -- 172.21.15.40:0/1673145670 learned_addr learned my addr 172.21.15.40:0/1673145670 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:49.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c455e1640 10 monclient(hunting): get_auth_request con 0x7f4c400a2df0 auth_method 0 2024-04-22T08:12:49.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c455e1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:49.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c455e1640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:49.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c455e1640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:49.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c44de0640 10 monclient(hunting): get_auth_request con 0x7f4c40135f70 auth_method 0 2024-04-22T08:12:49.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c44de0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:49.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c44de0640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:49.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c44de0640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:49.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c44de0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:49.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c44de0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:49.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c44de0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:49.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c3ffff640 1 --2- 172.21.15.40:0/1673145670 >> v2:172.21.15.189:3300/0 conn(0x7f4c4009f070 0x7f4c40136350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:49.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c455e1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:49.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c455e1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:49.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c455e1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:49.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c3ffff640 10 monclient(hunting): get_auth_request con 0x7f4c4009f070 auth_method 0 2024-04-22T08:12:49.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:49.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c3ffff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:49.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:49.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:49.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:49.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:49.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c44de0640 10 monclient(hunting): handle_auth_done global_id 4193 payload 274 2024-04-22T08:12:49.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c44de0640 1 -- 172.21.15.40:0/1673145670 >> v2:172.21.15.189:3300/0 conn(0x7f4c4009f070 msgr2=0x7f4c40136350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:49.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c44de0640 1 --2- 172.21.15.40:0/1673145670 >> v2:172.21.15.189:3300/0 conn(0x7f4c4009f070 0x7f4c40136350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:49.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c44de0640 1 -- 172.21.15.40:0/1673145670 >> v2:172.21.15.40:3300/0 conn(0x7f4c400a2df0 msgr2=0x7f4c400a3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:49.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c44de0640 1 --2- 172.21.15.40:0/1673145670 >> v2:172.21.15.40:3300/0 conn(0x7f4c400a2df0 0x7f4c400a3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:49.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c44de0640 10 monclient: _finish_hunting 0 2024-04-22T08:12:49.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c44de0640 1 monclient: found mon.noname-b 2024-04-22T08:12:49.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c44de0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:49.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c44de0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:12:49.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c44de0640 1 -- 172.21.15.40:0/1673145670 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c400a5c80 con 0x7f4c40135f70 2024-04-22T08:12:49.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.951+0000 7f4c455e1640 1 --2- 172.21.15.40:0/1673145670 >> v2:172.21.15.40:3300/0 conn(0x7f4c400a2df0 0x7f4c400a3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:12:49.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.954+0000 7f4c44de0640 1 -- 172.21.15.40:0/1673145670 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4c40135f70 msgr2=0x7f4c400a0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:49.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.954+0000 7f4c44de0640 1 --2- 172.21.15.40:0/1673145670 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4c40135f70 0x7f4c400a0470 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4c40066d90 tx=0x7f4c2802b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb16cb2f5e936b0a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:49.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.955+0000 7f4c3f7fe640 1 -- 172.21.15.40:0/1673145670 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c28030920 con 0x7f4c40135f70 2024-04-22T08:12:49.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.955+0000 7f4c3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:49.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.955+0000 7f4c3f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:12:49.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.955+0000 7f4c3f7fe640 10 monclient: dump: 2024-04-22T08:12:49.964 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:49.964 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:49.964 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:49.964 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:49.964 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:49.964 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:49.964 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:49.964 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:49.964 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:49.964 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:49.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.955+0000 7f4c3f7fe640 10 monclient: _finish_auth 0 2024-04-22T08:12:49.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.955+0000 7f4c3f7fe640 10 monclient: _check_auth_tickets 2024-04-22T08:12:49.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.955+0000 7f4c3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:49.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.955+0000 7f4c47044640 5 monclient: authenticate success, global_id 4193 2024-04-22T08:12:49.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.955+0000 7f4c3f7fe640 1 -- 172.21.15.40:0/1673145670 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c2802e050 con 0x7f4c40135f70 2024-04-22T08:12:49.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.955+0000 7f4c3f7fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:49.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.955+0000 7f4c47044640 10 monclient: get_monmap_and_config success 2024-04-22T08:12:49.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.955+0000 7f4c3f7fe640 1 -- 172.21.15.40:0/1673145670 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c28030ca0 con 0x7f4c40135f70 2024-04-22T08:12:49.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.955+0000 7f4c3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:49.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.955+0000 7f4c47044640 4 set_mon_vals no callback set 2024-04-22T08:12:49.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.955+0000 7f4c3f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:12:49.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.955+0000 7f4c3f7fe640 10 monclient: dump: 2024-04-22T08:12:49.965 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:49.965 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:49.965 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:49.966 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:49.966 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:49.966 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:49.966 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:49.966 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:49.966 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:49.966 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:49.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.955+0000 7f4c47044640 10 monclient: shutdown 2024-04-22T08:12:49.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.955+0000 7f4c47044640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:49.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.955+0000 7f4c47044640 1 -- 172.21.15.40:0/1673145670 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4c40135f70 msgr2=0x7f4c400a0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:49.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.955+0000 7f4c47044640 1 --2- 172.21.15.40:0/1673145670 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4c40135f70 0x7f4c400a0470 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4c40066d90 tx=0x7f4c2802b710 comp rx=0 tx=0).stop 2024-04-22T08:12:49.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.955+0000 7f4c47044640 1 -- 172.21.15.40:0/1673145670 shutdown_connections 2024-04-22T08:12:49.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.955+0000 7f4c47044640 1 --2- 172.21.15.40:0/1673145670 >> v2:172.21.15.189:3300/0 conn(0x7f4c4009f070 0x7f4c40136350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:49.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.955+0000 7f4c47044640 1 --2- 172.21.15.40:0/1673145670 >> v2:172.21.15.40:3300/0 conn(0x7f4c400a2df0 0x7f4c400a3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:49.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.955+0000 7f4c47044640 1 --2- 172.21.15.40:0/1673145670 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4c40135f70 0x7f4c400a0470 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:49.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.955+0000 7f4c47044640 1 -- 172.21.15.40:0/1673145670 >> 172.21.15.40:0/1673145670 conn(0x7f4c400a4ef0 msgr2=0x7f4c400a5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:49.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.955+0000 7f4c47044640 1 -- 172.21.15.40:0/1673145670 shutdown_connections 2024-04-22T08:12:49.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.955+0000 7f4c47044640 1 -- 172.21.15.40:0/1673145670 wait complete. 2024-04-22T08:12:49.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.956+0000 7f4c47044640 10 monclient: build_initial_monmap 2024-04-22T08:12:49.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.956+0000 7f4c47044640 10 monclient: monmap: 2024-04-22T08:12:49.967 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:49.967 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:12:49.967 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:12:49.967 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:12:49.967 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:49.967 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:49.967 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:12:49.968 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:12:49.968 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:12:49.968 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:49.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.956+0000 7f4c47044640 1 Processor -- start 2024-04-22T08:12:49.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.956+0000 7f4c47044640 1 -- start start 2024-04-22T08:12:49.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.956+0000 7f4c47044640 10 monclient: init 2024-04-22T08:12:49.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c47044640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:49.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c47044640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-22T08:12:49.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c47044640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4c4009f070 0x7f4c401caba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:49.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c47044640 10 monclient(hunting): picked mon.noname-c con 0x7f4c4009f070 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:49.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c47044640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c400a2df0 0x7f4c401cb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:49.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c47044640 10 monclient(hunting): picked mon.noname-a con 0x7f4c400a2df0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:49.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c47044640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c40135f70 0x7f4c40122230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:49.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c47044640 10 monclient(hunting): picked mon.noname-b con 0x7f4c40135f70 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:49.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c47044640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:49.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c47044640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c401258f0 con 0x7f4c400a2df0 2024-04-22T08:12:49.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c47044640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:49.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c47044640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c400719d0 con 0x7f4c40135f70 2024-04-22T08:12:49.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c47044640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:49.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c47044640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c40124a70 con 0x7f4c4009f070 2024-04-22T08:12:49.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c47044640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:49.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c47044640 10 monclient(hunting): authenticate will time out at 843.951262s 2024-04-22T08:12:49.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c44de0640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4c4009f070 0x7f4c401caba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:49.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c3ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c400a2df0 0x7f4c401cb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:49.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c44de0640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4c4009f070 0x7f4c401caba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49578/0 (socket says 172.21.15.40:49578) 2024-04-22T08:12:49.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c3ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c400a2df0 0x7f4c401cb0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57746/0 (socket says 172.21.15.40:57746) 2024-04-22T08:12:49.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c44de0640 1 -- 172.21.15.40:0/3801770801 learned_addr learned my addr 172.21.15.40:0/3801770801 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:49.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c44de0640 10 monclient(hunting): get_auth_request con 0x7f4c4009f070 auth_method 0 2024-04-22T08:12:49.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c44de0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:49.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c44de0640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:49.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c44de0640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:49.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c3ffff640 10 monclient(hunting): get_auth_request con 0x7f4c400a2df0 auth_method 0 2024-04-22T08:12:49.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:49.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c3ffff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:49.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:49.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c44de0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:49.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c44de0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:49.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c44de0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:49.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:49.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:49.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:49.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c455e1640 1 --2- 172.21.15.40:0/3801770801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c40135f70 0x7f4c40122230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:49.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c455e1640 10 monclient(hunting): get_auth_request con 0x7f4c40135f70 auth_method 0 2024-04-22T08:12:49.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c455e1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:49.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c455e1640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:49.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c455e1640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:49.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c44de0640 10 monclient(hunting): handle_auth_done global_id 4196 payload 824 2024-04-22T08:12:49.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c44de0640 1 -- 172.21.15.40:0/3801770801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c40135f70 msgr2=0x7f4c40122230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:49.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c44de0640 1 --2- 172.21.15.40:0/3801770801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c40135f70 0x7f4c40122230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:49.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c44de0640 1 -- 172.21.15.40:0/3801770801 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c400a2df0 msgr2=0x7f4c401cb0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:49.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c44de0640 1 --2- 172.21.15.40:0/3801770801 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c400a2df0 0x7f4c401cb0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:49.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c44de0640 10 monclient: _finish_hunting 0 2024-04-22T08:12:49.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c44de0640 1 monclient: found mon.noname-c 2024-04-22T08:12:49.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c44de0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:49.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c44de0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:12:49.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c44de0640 1 -- 172.21.15.40:0/3801770801 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c38009b30 con 0x7f4c4009f070 2024-04-22T08:12:49.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.957+0000 7f4c3ffff640 1 --2- 172.21.15.40:0/3801770801 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c400a2df0 0x7f4c401cb0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:12:49.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.960+0000 7f4c44de0640 1 -- 172.21.15.40:0/3801770801 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4c4009f070 msgr2=0x7f4c401caba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:49.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.960+0000 7f4c44de0640 1 --2- 172.21.15.40:0/3801770801 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4c4009f070 0x7f4c401caba0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4c2802bd20 tx=0x7f4c280046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:49.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.960+0000 7f4c3d7fa640 1 -- 172.21.15.40:0/3801770801 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c2803ca30 con 0x7f4c4009f070 2024-04-22T08:12:49.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.960+0000 7f4c3d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:49.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.960+0000 7f4c3d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:12:49.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.960+0000 7f4c3d7fa640 10 monclient: dump: 2024-04-22T08:12:49.972 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:49.972 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:49.972 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:49.972 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:49.972 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:49.973 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:49.973 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:49.973 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:49.973 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:49.973 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:49.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.960+0000 7f4c3d7fa640 10 monclient: _finish_auth 0 2024-04-22T08:12:49.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.960+0000 7f4c3d7fa640 10 monclient: _check_auth_tickets 2024-04-22T08:12:49.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.960+0000 7f4c3d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:49.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.960+0000 7f4c47044640 5 monclient: authenticate success, global_id 4196 2024-04-22T08:12:49.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.960+0000 7f4c3d7fa640 1 -- 172.21.15.40:0/3801770801 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c2803cbb0 con 0x7f4c4009f070 2024-04-22T08:12:49.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.960+0000 7f4c3d7fa640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:49.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.960+0000 7f4c3d7fa640 1 -- 172.21.15.40:0/3801770801 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c28032400 con 0x7f4c4009f070 2024-04-22T08:12:49.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.960+0000 7f4c47044640 10 monclient: _renew_subs 2024-04-22T08:12:49.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.960+0000 7f4c47044640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:49.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.960+0000 7f4c47044640 1 -- 172.21.15.40:0/3801770801 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c28009400 con 0x7f4c4009f070 2024-04-22T08:12:49.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.960+0000 7f4c3effd640 4 set_mon_vals no callback set 2024-04-22T08:12:49.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.960+0000 7f4c3d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:49.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.960+0000 7f4c3d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:12:49.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.960+0000 7f4c3d7fa640 10 monclient: dump: 2024-04-22T08:12:49.974 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:49.974 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:49.974 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:49.975 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:49.975 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:49.975 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:49.975 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:49.975 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:49.975 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:49.975 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:49.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.960+0000 7f4c47044640 10 monclient: _renew_subs 2024-04-22T08:12:49.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.960+0000 7f4c47044640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:49.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.960+0000 7f4c47044640 1 -- 172.21.15.40:0/3801770801 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c40135a10 con 0x7f4c4009f070 2024-04-22T08:12:49.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.961+0000 7f4c47044640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:49.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.961+0000 7f4c47044640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:49.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.961+0000 7f4c47044640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:49.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.961+0000 7f4c3d7fa640 1 -- 172.21.15.40:0/3801770801 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 4) v1 ==== 44726+0+0 (secure 0 0 0) 0x7f4c28039070 con 0x7f4c4009f070 2024-04-22T08:12:49.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.961+0000 7f4c47044640 1 -- 172.21.15.40:0/3801770801 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c40066650 con 0x7f4c4009f070 2024-04-22T08:12:49.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.961+0000 7f4c3d7fa640 1 --2- 172.21.15.40:0/3801770801 >> v2:172.21.15.189:6800/30168 conn(0x7f4c20037740 0x7f4c20039c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:49.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.961+0000 7f4c3d7fa640 1 -- 172.21.15.40:0/3801770801 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 3484+0+0 (secure 0 0 0) 0x7f4c2806bc20 con 0x7f4c4009f070 2024-04-22T08:12:49.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.961+0000 7f4c3ffff640 1 --2- 172.21.15.40:0/3801770801 >> v2:172.21.15.189:6800/30168 conn(0x7f4c20037740 0x7f4c20039c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:49.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.961+0000 7f4c3ffff640 10 monclient: get_auth_request con 0x7f4c20037740 auth_method 0 2024-04-22T08:12:49.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.964+0000 7f4c3ffff640 1 --2- 172.21.15.40:0/3801770801 >> v2:172.21.15.189:6800/30168 conn(0x7f4c20037740 0x7f4c20039c00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4c401241b0 tx=0x7f4c30009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:49.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.966+0000 7f4c3d7fa640 1 -- 172.21.15.40:0/3801770801 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f4c280325a0 con 0x7f4c4009f070 2024-04-22T08:12:49.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.966+0000 7f4c3d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:49.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:49.966+0000 7f4c3d7fa640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:12:50.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.146+0000 7f4c47044640 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-04-22T08:12:50.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.146+0000 7f4c47044640 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-04-22T08:12:50.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.146+0000 7f4c47044640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:50.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.146+0000 7f4c47044640 1 -- 172.21.15.40:0/3801770801 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f4c4004e9b0 con 0x7f4c4009f070 2024-04-22T08:12:50.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.834+0000 7f4c3d7fa640 1 -- 172.21.15.40:0/3801770801 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v14) v1 ==== 173+0+0 (secure 0 0 0) 0x7f4c28037030 con 0x7f4c4009f070 2024-04-22T08:12:50.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.835+0000 7f4c3d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-04-22T08:12:50.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.835+0000 7f4c3d7fa640 10 monclient: _finish_command 2 = system:0 enabled application 'rbd' on pool 'rbd' 2024-04-22T08:12:50.835 INFO:teuthology.orchestra.run.smithi040.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-22T08:12:50.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.838+0000 7f4c47044640 1 -- 172.21.15.40:0/3801770801 >> v2:172.21.15.189:6800/30168 conn(0x7f4c20037740 msgr2=0x7f4c20039c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:50.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.838+0000 7f4c47044640 1 --2- 172.21.15.40:0/3801770801 >> v2:172.21.15.189:6800/30168 conn(0x7f4c20037740 0x7f4c20039c00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4c401241b0 tx=0x7f4c30009190 comp rx=0 tx=0).stop 2024-04-22T08:12:50.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.838+0000 7f4c47044640 10 monclient: shutdown 2024-04-22T08:12:50.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.838+0000 7f4c47044640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:50.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.838+0000 7f4c47044640 1 -- 172.21.15.40:0/3801770801 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4c4009f070 msgr2=0x7f4c401caba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:50.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.838+0000 7f4c47044640 1 --2- 172.21.15.40:0/3801770801 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4c4009f070 0x7f4c401caba0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4c2802bd20 tx=0x7f4c280046c0 comp rx=0 tx=0).stop 2024-04-22T08:12:50.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.838+0000 7f4c47044640 1 -- 172.21.15.40:0/3801770801 shutdown_connections 2024-04-22T08:12:50.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.838+0000 7f4c47044640 1 --2- 172.21.15.40:0/3801770801 >> v2:172.21.15.189:6800/30168 conn(0x7f4c20037740 0x7f4c20039c00 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:50.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.838+0000 7f4c47044640 1 --2- 172.21.15.40:0/3801770801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c40135f70 0x7f4c40122230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:50.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.838+0000 7f4c47044640 1 --2- 172.21.15.40:0/3801770801 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c400a2df0 0x7f4c401cb0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:50.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.838+0000 7f4c47044640 1 --2- 172.21.15.40:0/3801770801 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4c4009f070 0x7f4c401caba0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:50.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.838+0000 7f4c47044640 1 -- 172.21.15.40:0/3801770801 >> 172.21.15.40:0/3801770801 conn(0x7f4c400a4ef0 msgr2=0x7f4c4006eea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:50.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.838+0000 7f4c47044640 1 -- 172.21.15.40:0/3801770801 shutdown_connections 2024-04-22T08:12:50.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.838+0000 7f4c47044640 1 -- 172.21.15.40:0/3801770801 wait complete. 2024-04-22T08:12:50.851 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-22T08:12:50.852 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-22T08:12:50.852 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-22T08:12:50.852 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-22T08:12:50.853 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-22T08:12:50.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.958+0000 7ff79644a640 10 monclient: get_monmap_and_config 2024-04-22T08:12:50.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.958+0000 7ff79644a640 10 monclient: build_initial_monmap 2024-04-22T08:12:50.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.958+0000 7ff79644a640 10 monclient: monmap: 2024-04-22T08:12:50.961 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:50.961 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:50.961 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:50.959025+0000 2024-04-22T08:12:50.961 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:50.959025+0000 2024-04-22T08:12:50.961 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:50.962 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:50.962 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:12:50.962 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:12:50.962 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:12:50.962 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:12:50.962 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:12:50.962 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:50.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.958+0000 7ff79644a640 1 Processor -- start 2024-04-22T08:12:50.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.958+0000 7ff79644a640 1 -- start start 2024-04-22T08:12:50.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.959+0000 7ff79644a640 10 monclient: init 2024-04-22T08:12:50.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.959+0000 7ff79644a640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:50.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.959+0000 7ff79644a640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-04-22T08:12:50.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.959+0000 7ff79644a640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff79009de00 0x7ff79009e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:50.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.959+0000 7ff79644a640 10 monclient(hunting): picked mon.noname-b con 0x7ff79009de00 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:50.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.959+0000 7ff79644a640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff7900a4330 0x7ff79009e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:50.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.959+0000 7ff79644a640 10 monclient(hunting): picked mon.noname-c con 0x7ff7900a4330 addr v2:172.21.15.189:3300/0 2024-04-22T08:12:50.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.959+0000 7ff79644a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff7900a0240 addr v1:172.21.15.40:6789/0 2024-04-22T08:12:50.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.959+0000 7ff79644a640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff79009ec60 con 0x7ff7900a0240 2024-04-22T08:12:50.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.959+0000 7ff79644a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:50.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.959+0000 7ff79644a640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff79009fb40 con 0x7ff7900a4330 2024-04-22T08:12:50.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.959+0000 7ff78ffff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff79009de00 0x7ff79009e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:50.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.959+0000 7ff79644a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:50.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.959+0000 7ff79644a640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7900a3410 con 0x7ff79009de00 2024-04-22T08:12:50.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.959+0000 7ff79644a640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:50.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.959+0000 7ff79644a640 10 monclient(hunting): authenticate will time out at 844.953487s 2024-04-22T08:12:50.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.959+0000 7ff78ffff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff79009de00 0x7ff79009e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49582/0 (socket says 172.21.15.40:49582) 2024-04-22T08:12:50.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.959+0000 7ff78ffff640 1 -- 172.21.15.40:0/3653611468 learned_addr learned my addr 172.21.15.40:0/3653611468 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:50.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.959+0000 7ff78ffff640 10 monclient(hunting): get_auth_request con 0x7ff79009de00 auth_method 0 2024-04-22T08:12:50.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.959+0000 7ff78ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:50.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.959+0000 7ff78ffff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:50.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.959+0000 7ff78ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:50.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.959+0000 7ff78ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:50.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.959+0000 7ff78ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:50.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.959+0000 7ff78ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:50.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.959+0000 7ff78f7fe640 1 --2- 172.21.15.40:0/3653611468 >> v2:172.21.15.189:3300/0 conn(0x7ff7900a4330 0x7ff79009e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:50.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.959+0000 7ff78f7fe640 10 monclient(hunting): get_auth_request con 0x7ff7900a4330 auth_method 0 2024-04-22T08:12:50.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.959+0000 7ff78f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:50.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.959+0000 7ff78f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:50.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.959+0000 7ff78f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:50.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.959+0000 7ff78ffff640 10 monclient(hunting): handle_auth_done global_id 4199 payload 274 2024-04-22T08:12:50.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.959+0000 7ff78ffff640 1 -- 172.21.15.40:0/3653611468 >> v2:172.21.15.189:3300/0 conn(0x7ff7900a4330 msgr2=0x7ff79009e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:50.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.959+0000 7ff78ffff640 1 --2- 172.21.15.40:0/3653611468 >> v2:172.21.15.189:3300/0 conn(0x7ff7900a4330 0x7ff79009e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:50.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.960+0000 7ff78ffff640 1 -- 172.21.15.40:0/3653611468 >> v1:172.21.15.40:6789/0 conn(0x7ff7900a0240 legacy=0x7ff790135bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:50.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.960+0000 7ff78ffff640 10 monclient: _finish_hunting 0 2024-04-22T08:12:50.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.960+0000 7ff78ffff640 1 monclient: found mon.noname-b 2024-04-22T08:12:50.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.960+0000 7ff78ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:50.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.960+0000 7ff78ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:12:50.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.960+0000 7ff78ffff640 1 -- 172.21.15.40:0/3653611468 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7900a4b00 con 0x7ff79009de00 2024-04-22T08:12:50.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.962+0000 7ff78ffff640 1 -- 172.21.15.40:0/3653611468 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff79009de00 msgr2=0x7ff79009e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:50.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.962+0000 7ff78ffff640 1 --2- 172.21.15.40:0/3653611468 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff79009de00 0x7ff79009e1e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff778009870 tx=0x7ff77802b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=308c9ea57dd54170 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:50.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.963+0000 7ff78effd640 1 -- 172.21.15.40:0/3653611468 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff778031950 con 0x7ff79009de00 2024-04-22T08:12:50.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.963+0000 7ff78effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:50.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.963+0000 7ff78effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:12:50.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.963+0000 7ff78effd640 10 monclient: dump: 2024-04-22T08:12:50.973 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:50.973 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:50.973 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:50.974 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:50.974 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:50.974 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:50.974 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:50.974 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:50.974 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:50.974 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:50.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.963+0000 7ff78effd640 10 monclient: _finish_auth 0 2024-04-22T08:12:50.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.963+0000 7ff78effd640 10 monclient: _check_auth_tickets 2024-04-22T08:12:50.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.963+0000 7ff78effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:50.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.963+0000 7ff78effd640 1 -- 172.21.15.40:0/3653611468 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff778031ab0 con 0x7ff79009de00 2024-04-22T08:12:50.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.963+0000 7ff78effd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:50.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.963+0000 7ff78effd640 1 -- 172.21.15.40:0/3653611468 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff778031e40 con 0x7ff79009de00 2024-04-22T08:12:50.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.963+0000 7ff79644a640 5 monclient: authenticate success, global_id 4199 2024-04-22T08:12:50.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.963+0000 7ff79644a640 10 monclient: get_monmap_and_config success 2024-04-22T08:12:50.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.963+0000 7ff79644a640 4 set_mon_vals no callback set 2024-04-22T08:12:50.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.963+0000 7ff78effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:50.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.963+0000 7ff78effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:12:50.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.963+0000 7ff78effd640 10 monclient: dump: 2024-04-22T08:12:50.975 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:50.975 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:50.975 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:50.975 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:50.975 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:50.975 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:50.975 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:50.975 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:50.976 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:50.976 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:50.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.963+0000 7ff79644a640 10 monclient: shutdown 2024-04-22T08:12:50.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.963+0000 7ff79644a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:50.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.963+0000 7ff79644a640 1 -- 172.21.15.40:0/3653611468 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff79009de00 msgr2=0x7ff79009e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:50.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.963+0000 7ff79644a640 1 --2- 172.21.15.40:0/3653611468 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff79009de00 0x7ff79009e1e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff778009870 tx=0x7ff77802b7e0 comp rx=0 tx=0).stop 2024-04-22T08:12:50.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.963+0000 7ff79644a640 1 -- 172.21.15.40:0/3653611468 shutdown_connections 2024-04-22T08:12:50.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.963+0000 7ff79644a640 1 --2- 172.21.15.40:0/3653611468 >> v2:172.21.15.189:3300/0 conn(0x7ff7900a4330 0x7ff79009e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:50.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.963+0000 7ff79644a640 1 --2- 172.21.15.40:0/3653611468 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff79009de00 0x7ff79009e1e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:50.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.963+0000 7ff79644a640 1 -- 172.21.15.40:0/3653611468 >> 172.21.15.40:0/3653611468 conn(0x7ff7900a83f0 msgr2=0x7ff7900a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:50.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.963+0000 7ff79644a640 1 -- 172.21.15.40:0/3653611468 shutdown_connections 2024-04-22T08:12:50.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.963+0000 7ff79644a640 1 -- 172.21.15.40:0/3653611468 wait complete. 2024-04-22T08:12:50.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.964+0000 7ff79644a640 10 monclient: build_initial_monmap 2024-04-22T08:12:50.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.964+0000 7ff79644a640 10 monclient: monmap: 2024-04-22T08:12:50.977 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:50.977 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:12:50.977 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:12:50.977 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:12:50.977 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:50.977 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:50.977 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:12:50.977 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:12:50.977 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:12:50.977 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:50.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.964+0000 7ff79644a640 1 Processor -- start 2024-04-22T08:12:50.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.964+0000 7ff79644a640 1 -- start start 2024-04-22T08:12:50.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.964+0000 7ff79644a640 10 monclient: init 2024-04-22T08:12:50.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff79644a640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:50.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff79644a640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-22T08:12:50.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff79644a640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff79009de00 0x7ff790123550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:50.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff79644a640 10 monclient(hunting): picked mon.noname-c con 0x7ff79009de00 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:50.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff79644a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7900a0240 0x7ff7901caef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:50.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff79644a640 10 monclient(hunting): picked mon.noname-b con 0x7ff7900a0240 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:50.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff79644a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7901cb430 0x7ff79011f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:50.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff79644a640 10 monclient(hunting): picked mon.noname-a con 0x7ff7901cb430 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:50.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff79644a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:50.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff79644a640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff790122c00 con 0x7ff7901cb430 2024-04-22T08:12:50.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff79644a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:50.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff79644a640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7900a1c30 con 0x7ff7900a0240 2024-04-22T08:12:50.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff79644a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:50.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff79644a640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7900a6e50 con 0x7ff79009de00 2024-04-22T08:12:50.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff79644a640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:50.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff79644a640 10 monclient(hunting): authenticate will time out at 844.959118s 2024-04-22T08:12:50.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff78ffff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff79009de00 0x7ff790123550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:50.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff78ffff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff79009de00 0x7ff790123550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49590/0 (socket says 172.21.15.40:49590) 2024-04-22T08:12:50.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff78ffff640 1 -- 172.21.15.40:0/1686802019 learned_addr learned my addr 172.21.15.40:0/1686802019 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:50.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff78ffff640 10 monclient(hunting): get_auth_request con 0x7ff79009de00 auth_method 0 2024-04-22T08:12:50.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff78ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:50.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff78ffff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:50.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff78ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:50.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff7949e7640 1 --2- 172.21.15.40:0/1686802019 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7901cb430 0x7ff79011f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:50.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff7949e7640 10 monclient(hunting): get_auth_request con 0x7ff7901cb430 auth_method 0 2024-04-22T08:12:50.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff7949e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:50.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff7949e7640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:50.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff7949e7640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:50.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff78ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:50.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff78ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:50.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff78ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:50.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff78f7fe640 1 --2- 172.21.15.40:0/1686802019 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7900a0240 0x7ff7901caef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:50.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff78f7fe640 10 monclient(hunting): get_auth_request con 0x7ff7900a0240 auth_method 0 2024-04-22T08:12:50.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff78f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:50.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff78f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:50.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff78f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:50.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff7949e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:50.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff7949e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:50.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff7949e7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:50.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff78f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:50.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff78f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:50.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff78f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:50.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff78ffff640 10 monclient(hunting): handle_auth_done global_id 4202 payload 824 2024-04-22T08:12:50.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff78ffff640 1 -- 172.21.15.40:0/1686802019 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7900a0240 msgr2=0x7ff7901caef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:50.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff78ffff640 1 --2- 172.21.15.40:0/1686802019 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7900a0240 0x7ff7901caef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:50.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff78ffff640 1 -- 172.21.15.40:0/1686802019 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7901cb430 msgr2=0x7ff79011f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:50.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff78ffff640 1 --2- 172.21.15.40:0/1686802019 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7901cb430 0x7ff79011f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:50.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff78ffff640 10 monclient: _finish_hunting 0 2024-04-22T08:12:50.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff78ffff640 1 monclient: found mon.noname-c 2024-04-22T08:12:50.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff78ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:50.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff78ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:12:50.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.965+0000 7ff78ffff640 1 -- 172.21.15.40:0/1686802019 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7780094d0 con 0x7ff79009de00 2024-04-22T08:12:50.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.968+0000 7ff78ffff640 1 -- 172.21.15.40:0/1686802019 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff79009de00 msgr2=0x7ff790123550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:50.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.968+0000 7ff78ffff640 1 --2- 172.21.15.40:0/1686802019 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff79009de00 0x7ff790123550 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff7780090b0 tx=0x7ff7780046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:50.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.968+0000 7ff78cff9640 1 -- 172.21.15.40:0/1686802019 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff778033d30 con 0x7ff79009de00 2024-04-22T08:12:50.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.968+0000 7ff78cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:50.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.968+0000 7ff78cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:12:50.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.968+0000 7ff78cff9640 10 monclient: dump: 2024-04-22T08:12:50.982 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:50.982 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:50.982 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:50.982 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:50.982 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:50.982 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:50.982 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:50.982 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:50.982 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:50.982 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:50.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.968+0000 7ff78cff9640 10 monclient: _finish_auth 0 2024-04-22T08:12:50.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.968+0000 7ff78cff9640 10 monclient: _check_auth_tickets 2024-04-22T08:12:50.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.968+0000 7ff78cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:50.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.968+0000 7ff78cff9640 1 -- 172.21.15.40:0/1686802019 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff778033e90 con 0x7ff79009de00 2024-04-22T08:12:50.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.968+0000 7ff79644a640 5 monclient: authenticate success, global_id 4202 2024-04-22T08:12:50.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.968+0000 7ff79644a640 10 monclient: _renew_subs 2024-04-22T08:12:50.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.968+0000 7ff79644a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:50.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.968+0000 7ff79644a640 1 -- 172.21.15.40:0/1686802019 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff790123130 con 0x7ff79009de00 2024-04-22T08:12:50.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.968+0000 7ff78cff9640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:50.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.968+0000 7ff78cff9640 1 -- 172.21.15.40:0/1686802019 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff77803c620 con 0x7ff79009de00 2024-04-22T08:12:50.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.968+0000 7ff78cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:50.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.968+0000 7ff78cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:12:50.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.968+0000 7ff78cff9640 10 monclient: dump: 2024-04-22T08:12:50.983 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:50.983 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:50.983 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:50.983 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:50.983 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:50.983 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:50.984 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:50.984 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:50.984 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:50.984 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:50.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.968+0000 7ff78e7fc640 4 set_mon_vals no callback set 2024-04-22T08:12:50.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.968+0000 7ff79644a640 10 monclient: _renew_subs 2024-04-22T08:12:50.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.968+0000 7ff79644a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:50.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.968+0000 7ff79644a640 1 -- 172.21.15.40:0/1686802019 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff790120d60 con 0x7ff79009de00 2024-04-22T08:12:50.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.969+0000 7ff78cff9640 1 -- 172.21.15.40:0/1686802019 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 4) v1 ==== 44726+0+0 (secure 0 0 0) 0x7ff77803a040 con 0x7ff79009de00 2024-04-22T08:12:50.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.969+0000 7ff79644a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:50.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.969+0000 7ff79644a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:50.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.969+0000 7ff79644a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:50.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.969+0000 7ff79644a640 1 -- 172.21.15.40:0/1686802019 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff79004e9b0 con 0x7ff79009de00 2024-04-22T08:12:50.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.969+0000 7ff78cff9640 1 --2- 172.21.15.40:0/1686802019 >> v2:172.21.15.189:6800/30168 conn(0x7ff770037790 0x7ff770039c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:50.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.969+0000 7ff78cff9640 1 -- 172.21.15.40:0/1686802019 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 3506+0+0 (secure 0 0 0) 0x7ff77806bbf0 con 0x7ff79009de00 2024-04-22T08:12:50.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.969+0000 7ff78f7fe640 1 --2- 172.21.15.40:0/1686802019 >> v2:172.21.15.189:6800/30168 conn(0x7ff770037790 0x7ff770039c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:50.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.969+0000 7ff78f7fe640 10 monclient: get_auth_request con 0x7ff770037790 auth_method 0 2024-04-22T08:12:50.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.972+0000 7ff78f7fe640 1 --2- 172.21.15.40:0/1686802019 >> v2:172.21.15.189:6800/30168 conn(0x7ff770037790 0x7ff770039c50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff7901210e0 tx=0x7ff780008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:50.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.974+0000 7ff78cff9640 1 -- 172.21.15.40:0/1686802019 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7ff778046b90 con 0x7ff79009de00 2024-04-22T08:12:50.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.974+0000 7ff78cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:50.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:50.974+0000 7ff78cff9640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:12:51.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.202+0000 7ff79644a640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-04-22T08:12:51.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.202+0000 7ff79644a640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-04-22T08:12:51.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.202+0000 7ff79644a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:51.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.202+0000 7ff79644a640 1 -- 172.21.15.40:0/1686802019 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7ff79006a270 con 0x7ff79009de00 2024-04-22T08:12:51.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.204+0000 7ff78cff9640 1 -- 172.21.15.40:0/1686802019 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v4) v1 ==== 74+0+85983 (secure 0 0 0) 0x7ff77802f030 con 0x7ff79009de00 2024-04-22T08:12:51.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.204+0000 7ff78cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-04-22T08:12:51.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.204+0000 7ff78cff9640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:12:51.205 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:12:51.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.206+0000 7ff79644a640 1 -- 172.21.15.40:0/1686802019 >> v2:172.21.15.189:6800/30168 conn(0x7ff770037790 msgr2=0x7ff770039c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:51.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.206+0000 7ff79644a640 1 --2- 172.21.15.40:0/1686802019 >> v2:172.21.15.189:6800/30168 conn(0x7ff770037790 0x7ff770039c50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff7901210e0 tx=0x7ff780008040 comp rx=0 tx=0).stop 2024-04-22T08:12:51.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.206+0000 7ff79644a640 10 monclient: shutdown 2024-04-22T08:12:51.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.206+0000 7ff79644a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:51.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.206+0000 7ff79644a640 1 -- 172.21.15.40:0/1686802019 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff79009de00 msgr2=0x7ff790123550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:51.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.206+0000 7ff79644a640 1 --2- 172.21.15.40:0/1686802019 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff79009de00 0x7ff790123550 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff7780090b0 tx=0x7ff7780046c0 comp rx=0 tx=0).stop 2024-04-22T08:12:51.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.206+0000 7ff79644a640 1 -- 172.21.15.40:0/1686802019 shutdown_connections 2024-04-22T08:12:51.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.206+0000 7ff79644a640 1 --2- 172.21.15.40:0/1686802019 >> v2:172.21.15.189:6800/30168 conn(0x7ff770037790 0x7ff770039c50 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:51.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.206+0000 7ff79644a640 1 --2- 172.21.15.40:0/1686802019 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7900a0240 0x7ff7901caef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:51.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.206+0000 7ff79644a640 1 --2- 172.21.15.40:0/1686802019 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7901cb430 0x7ff79011f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:51.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.206+0000 7ff79644a640 1 --2- 172.21.15.40:0/1686802019 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff79009de00 0x7ff790123550 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:51.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.206+0000 7ff79644a640 1 -- 172.21.15.40:0/1686802019 >> 172.21.15.40:0/1686802019 conn(0x7ff7900a83f0 msgr2=0x7ff7900a64f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:51.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.206+0000 7ff79644a640 1 -- 172.21.15.40:0/1686802019 shutdown_connections 2024-04-22T08:12:51.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.206+0000 7ff79644a640 1 -- 172.21.15.40:0/1686802019 wait complete. 2024-04-22T08:12:51.216 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":4,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":30168}]},"active_addr":"172.21.15.189:6800/30168","active_change":"2024-04-22T08:12:40.071326+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.189:0","nonce":4179360702}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.189:0","nonce":1011116873}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.189:0","nonce":3446907120}]}]} 2024-04-22T08:12:51.219 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-22T08:12:51.219 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-22T08:12:51.220 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T08:12:51.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.318+0000 7ff4bfcbd640 10 monclient: get_monmap_and_config 2024-04-22T08:12:51.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.318+0000 7ff4bfcbd640 10 monclient: build_initial_monmap 2024-04-22T08:12:51.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.318+0000 7ff4bfcbd640 10 monclient: monmap: 2024-04-22T08:12:51.318 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:51.318 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:51.318 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:51.319301+0000 2024-04-22T08:12:51.318 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:51.319301+0000 2024-04-22T08:12:51.319 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:51.319 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:51.319 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:12:51.319 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:12:51.319 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:12:51.319 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:12:51.319 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:12:51.319 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:51.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.319+0000 7ff4bfcbd640 1 Processor -- start 2024-04-22T08:12:51.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.319+0000 7ff4bfcbd640 1 -- start start 2024-04-22T08:12:51.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.319+0000 7ff4bfcbd640 10 monclient: init 2024-04-22T08:12:51.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.319+0000 7ff4bfcbd640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:51.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.319+0000 7ff4bfcbd640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-04-22T08:12:51.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.319+0000 7ff4bfcbd640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff4b80a1970 0x7ff4b81360d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:51.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.319+0000 7ff4bfcbd640 10 monclient(hunting): picked mon.noname-a con 0x7ff4b80a1970 addr v2:172.21.15.40:3300/0 2024-04-22T08:12:51.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.319+0000 7ff4bfcbd640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff4b80a1d50 0x7ff4b80a2df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:51.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.319+0000 7ff4bfcbd640 10 monclient(hunting): picked mon.noname-c con 0x7ff4b80a1d50 addr v2:172.21.15.189:3300/0 2024-04-22T08:12:51.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.319+0000 7ff4bfcbd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff4b80a2180 0x7ff4b80a3330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:51.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.319+0000 7ff4bfcbd640 10 monclient(hunting): picked mon.noname-b con 0x7ff4b80a2180 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:51.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.319+0000 7ff4bfcbd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:51.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.319+0000 7ff4bfcbd640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4b80a4110 con 0x7ff4b80a1970 2024-04-22T08:12:51.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.319+0000 7ff4bfcbd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:51.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.319+0000 7ff4bfcbd640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4b809fe80 con 0x7ff4b80a1d50 2024-04-22T08:12:51.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.319+0000 7ff4bfcbd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:51.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.319+0000 7ff4bfcbd640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b80a0160 con 0x7ff4b80a2180 2024-04-22T08:12:51.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.319+0000 7ff4bfcbd640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:51.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.319+0000 7ff4bfcbd640 10 monclient(hunting): authenticate will time out at 845.313559s 2024-04-22T08:12:51.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.319+0000 7ff4bda59640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff4b80a1970 0x7ff4b81360d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:51.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.319+0000 7ff4be25a640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff4b80a2180 0x7ff4b80a3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:51.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.319+0000 7ff4bda59640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff4b80a1970 0x7ff4b81360d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57768/0 (socket says 172.21.15.40:57768) 2024-04-22T08:12:51.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.319+0000 7ff4bda59640 1 -- 172.21.15.40:0/1869110064 learned_addr learned my addr 172.21.15.40:0/1869110064 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:51.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.319+0000 7ff4be25a640 10 monclient(hunting): get_auth_request con 0x7ff4b80a2180 auth_method 0 2024-04-22T08:12:51.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.319+0000 7ff4be25a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:51.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.319+0000 7ff4be25a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:51.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.319+0000 7ff4be25a640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:51.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.319+0000 7ff4bda59640 10 monclient(hunting): get_auth_request con 0x7ff4b80a1970 auth_method 0 2024-04-22T08:12:51.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.319+0000 7ff4bda59640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:51.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.319+0000 7ff4bda59640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:51.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.319+0000 7ff4bda59640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:51.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.320+0000 7ff4be25a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:51.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.320+0000 7ff4be25a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:51.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.320+0000 7ff4be25a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:51.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.320+0000 7ff4bda59640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:51.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.320+0000 7ff4bda59640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:51.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.320+0000 7ff4bda59640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:51.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.320+0000 7ff4bd258640 1 --2- 172.21.15.40:0/1869110064 >> v2:172.21.15.189:3300/0 conn(0x7ff4b80a1d50 0x7ff4b80a2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:51.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.320+0000 7ff4bd258640 10 monclient(hunting): get_auth_request con 0x7ff4b80a1d50 auth_method 0 2024-04-22T08:12:51.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.320+0000 7ff4bd258640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:51.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.320+0000 7ff4bd258640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:51.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.320+0000 7ff4bd258640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:51.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.320+0000 7ff4bda59640 10 monclient(hunting): handle_auth_done global_id 4212 payload 274 2024-04-22T08:12:51.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.320+0000 7ff4bda59640 1 -- 172.21.15.40:0/1869110064 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff4b80a2180 msgr2=0x7ff4b80a3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:51.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.320+0000 7ff4bda59640 1 --2- 172.21.15.40:0/1869110064 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff4b80a2180 0x7ff4b80a3330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:51.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.320+0000 7ff4bda59640 1 -- 172.21.15.40:0/1869110064 >> v2:172.21.15.189:3300/0 conn(0x7ff4b80a1d50 msgr2=0x7ff4b80a2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:51.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.320+0000 7ff4bda59640 1 --2- 172.21.15.40:0/1869110064 >> v2:172.21.15.189:3300/0 conn(0x7ff4b80a1d50 0x7ff4b80a2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:51.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.320+0000 7ff4bda59640 10 monclient: _finish_hunting 0 2024-04-22T08:12:51.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.320+0000 7ff4bda59640 1 monclient: found mon.noname-a 2024-04-22T08:12:51.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.320+0000 7ff4bda59640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:51.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.320+0000 7ff4bda59640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:12:51.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.320+0000 7ff4bda59640 1 -- 172.21.15.40:0/1869110064 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4b80a0440 con 0x7ff4b80a1970 2024-04-22T08:12:51.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.320+0000 7ff4be25a640 1 --2- 172.21.15.40:0/1869110064 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff4b80a2180 0x7ff4b80a3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:12:51.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.323+0000 7ff4bda59640 1 -- 172.21.15.40:0/1869110064 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4b80a1970 msgr2=0x7ff4b81360d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:51.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.323+0000 7ff4bda59640 1 --2- 172.21.15.40:0/1869110064 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4b80a1970 0x7ff4b81360d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff4b80a94b0 tx=0x7ff4b402b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=563870d0ce0cee74 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:51.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.323+0000 7ff4bca57640 1 -- 172.21.15.40:0/1869110064 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b4031950 con 0x7ff4b80a1970 2024-04-22T08:12:51.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.323+0000 7ff4bca57640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:51.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.323+0000 7ff4bca57640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:12:51.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.323+0000 7ff4bca57640 10 monclient: dump: 2024-04-22T08:12:51.333 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:51.333 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:51.333 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:51.333 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:51.333 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:51.333 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:51.333 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:51.333 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:51.333 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:51.333 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:51.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.323+0000 7ff4bca57640 10 monclient: _finish_auth 0 2024-04-22T08:12:51.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.323+0000 7ff4bca57640 10 monclient: _check_auth_tickets 2024-04-22T08:12:51.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.323+0000 7ff4bca57640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:51.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.323+0000 7ff4bca57640 1 -- 172.21.15.40:0/1869110064 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4b4031ab0 con 0x7ff4b80a1970 2024-04-22T08:12:51.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.323+0000 7ff4bfcbd640 5 monclient: authenticate success, global_id 4212 2024-04-22T08:12:51.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.323+0000 7ff4bfcbd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:12:51.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.323+0000 7ff4bca57640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:51.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.323+0000 7ff4bfcbd640 10 monclient: get_monmap_and_config success 2024-04-22T08:12:51.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.323+0000 7ff4bca57640 1 -- 172.21.15.40:0/1869110064 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b4031e40 con 0x7ff4b80a1970 2024-04-22T08:12:51.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.323+0000 7ff4bfcbd640 4 set_mon_vals no callback set 2024-04-22T08:12:51.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.323+0000 7ff4bca57640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:51.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.323+0000 7ff4bca57640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:12:51.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.323+0000 7ff4bca57640 10 monclient: dump: 2024-04-22T08:12:51.334 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:51.335 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:51.335 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:51.335 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:51.335 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:51.335 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:51.335 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:51.335 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:51.335 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:51.335 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:51.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.323+0000 7ff4bfcbd640 10 monclient: shutdown 2024-04-22T08:12:51.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.323+0000 7ff4bfcbd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:51.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.323+0000 7ff4bfcbd640 1 -- 172.21.15.40:0/1869110064 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4b80a1970 msgr2=0x7ff4b81360d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:51.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.323+0000 7ff4bfcbd640 1 --2- 172.21.15.40:0/1869110064 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4b80a1970 0x7ff4b81360d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff4b80a94b0 tx=0x7ff4b402b7e0 comp rx=0 tx=0).stop 2024-04-22T08:12:51.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.323+0000 7ff4bfcbd640 1 -- 172.21.15.40:0/1869110064 shutdown_connections 2024-04-22T08:12:51.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.323+0000 7ff4bfcbd640 1 --2- 172.21.15.40:0/1869110064 >> v2:172.21.15.189:3300/0 conn(0x7ff4b80a1d50 0x7ff4b80a2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:51.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.323+0000 7ff4bfcbd640 1 --2- 172.21.15.40:0/1869110064 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff4b80a2180 0x7ff4b80a3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:51.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.323+0000 7ff4bfcbd640 1 --2- 172.21.15.40:0/1869110064 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4b80a1970 0x7ff4b81360d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:51.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.323+0000 7ff4bfcbd640 1 -- 172.21.15.40:0/1869110064 >> 172.21.15.40:0/1869110064 conn(0x7ff4b80a4ef0 msgr2=0x7ff4b80a5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:51.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.323+0000 7ff4bfcbd640 1 -- 172.21.15.40:0/1869110064 shutdown_connections 2024-04-22T08:12:51.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.323+0000 7ff4bfcbd640 1 -- 172.21.15.40:0/1869110064 wait complete. 2024-04-22T08:12:51.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.324+0000 7ff4bfcbd640 10 monclient: build_initial_monmap 2024-04-22T08:12:51.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.324+0000 7ff4bfcbd640 10 monclient: monmap: 2024-04-22T08:12:51.336 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:51.337 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:12:51.337 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:12:51.337 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:12:51.337 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:51.337 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:51.337 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:12:51.337 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:12:51.338 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:12:51.338 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:51.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.324+0000 7ff4bfcbd640 1 Processor -- start 2024-04-22T08:12:51.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.324+0000 7ff4bfcbd640 1 -- start start 2024-04-22T08:12:51.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.324+0000 7ff4bfcbd640 10 monclient: init 2024-04-22T08:12:51.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4bfcbd640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:51.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4bfcbd640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-22T08:12:51.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4bfcbd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4b80a1970 0x7ff4b81cb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:51.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4bfcbd640 10 monclient(hunting): picked mon.noname-a con 0x7ff4b80a1970 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:51.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4bfcbd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff4b80a1d50 0x7ff4b811ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:51.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4bfcbd640 10 monclient(hunting): picked mon.noname-c con 0x7ff4b80a1d50 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:51.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4bfcbd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4b80a2180 0x7ff4b811ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:51.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4bfcbd640 10 monclient(hunting): picked mon.noname-b con 0x7ff4b80a2180 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:51.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4bfcbd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:51.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4bfcbd640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b81350b0 con 0x7ff4b80a1970 2024-04-22T08:12:51.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4bfcbd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:51.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4bfcbd640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b8122360 con 0x7ff4b80a2180 2024-04-22T08:12:51.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4bfcbd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:51.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4bfcbd640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b8120b60 con 0x7ff4b80a1d50 2024-04-22T08:12:51.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4bfcbd640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:51.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4bfcbd640 10 monclient(hunting): authenticate will time out at 845.319083s 2024-04-22T08:12:51.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4bda59640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4b80a1970 0x7ff4b81cb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:51.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4bd258640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff4b80a1d50 0x7ff4b811ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:51.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4bda59640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4b80a1970 0x7ff4b81cb0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57776/0 (socket says 172.21.15.40:57776) 2024-04-22T08:12:51.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4bd258640 10 monclient(hunting): get_auth_request con 0x7ff4b80a1d50 auth_method 0 2024-04-22T08:12:51.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4bda59640 1 -- 172.21.15.40:0/1273602216 learned_addr learned my addr 172.21.15.40:0/1273602216 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:51.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4bd258640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:51.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4bd258640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:51.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4bd258640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:51.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4bda59640 10 monclient(hunting): get_auth_request con 0x7ff4b80a1970 auth_method 0 2024-04-22T08:12:51.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4bda59640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:51.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4bda59640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:51.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4bda59640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:51.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4bd258640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:51.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4bd258640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:51.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4bd258640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:51.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4bda59640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:51.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4bda59640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:51.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4bda59640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:51.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4be25a640 1 --2- 172.21.15.40:0/1273602216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4b80a2180 0x7ff4b811ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:51.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4be25a640 10 monclient(hunting): get_auth_request con 0x7ff4b80a2180 auth_method 0 2024-04-22T08:12:51.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4be25a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:51.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4be25a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:51.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4be25a640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:51.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4be25a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:51.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4be25a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:51.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4be25a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:51.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4bd258640 10 monclient(hunting): handle_auth_done global_id 4208 payload 824 2024-04-22T08:12:51.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4bd258640 1 -- 172.21.15.40:0/1273602216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4b80a2180 msgr2=0x7ff4b811ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:51.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4bd258640 1 --2- 172.21.15.40:0/1273602216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4b80a2180 0x7ff4b811ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:51.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4bd258640 1 -- 172.21.15.40:0/1273602216 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4b80a1970 msgr2=0x7ff4b81cb0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:51.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4bd258640 1 --2- 172.21.15.40:0/1273602216 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4b80a1970 0x7ff4b81cb0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:51.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4bd258640 10 monclient: _finish_hunting 0 2024-04-22T08:12:51.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4bd258640 1 monclient: found mon.noname-c 2024-04-22T08:12:51.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4bd258640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:51.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4bd258640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:12:51.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4bd258640 1 -- 172.21.15.40:0/1273602216 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4b0009a70 con 0x7ff4b80a1d50 2024-04-22T08:12:51.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.325+0000 7ff4bda59640 1 --2- 172.21.15.40:0/1273602216 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4b80a1970 0x7ff4b81cb0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:12:51.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.328+0000 7ff4bd258640 1 -- 172.21.15.40:0/1273602216 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff4b80a1d50 msgr2=0x7ff4b811ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:51.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.328+0000 7ff4bd258640 1 --2- 172.21.15.40:0/1273602216 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff4b80a1d50 0x7ff4b811ea40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff4a800d5f0 tx=0x7ff4a800d9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:51.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.328+0000 7ff4ae7fc640 1 -- 172.21.15.40:0/1273602216 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a8012800 con 0x7ff4b80a1d50 2024-04-22T08:12:51.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.328+0000 7ff4ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:51.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.328+0000 7ff4ae7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:12:51.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.328+0000 7ff4ae7fc640 10 monclient: dump: 2024-04-22T08:12:51.342 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:51.342 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:51.342 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:51.342 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:51.342 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:51.343 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:51.343 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:51.343 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:51.343 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:51.343 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:51.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.328+0000 7ff4ae7fc640 10 monclient: _finish_auth 0 2024-04-22T08:12:51.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.328+0000 7ff4ae7fc640 10 monclient: _check_auth_tickets 2024-04-22T08:12:51.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.328+0000 7ff4ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:51.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.328+0000 7ff4ae7fc640 1 -- 172.21.15.40:0/1273602216 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4a8012960 con 0x7ff4b80a1d50 2024-04-22T08:12:51.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.328+0000 7ff4bfcbd640 5 monclient: authenticate success, global_id 4208 2024-04-22T08:12:51.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.328+0000 7ff4bfcbd640 10 monclient: _renew_subs 2024-04-22T08:12:51.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.328+0000 7ff4bfcbd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:51.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.328+0000 7ff4bfcbd640 1 -- 172.21.15.40:0/1273602216 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4b40094d0 con 0x7ff4b80a1d50 2024-04-22T08:12:51.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.328+0000 7ff4ae7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:51.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.328+0000 7ff4ae7fc640 1 -- 172.21.15.40:0/1273602216 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a8012cf0 con 0x7ff4b80a1d50 2024-04-22T08:12:51.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.328+0000 7ff4ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:51.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.328+0000 7ff4ae7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:12:51.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.328+0000 7ff4ae7fc640 10 monclient: dump: 2024-04-22T08:12:51.344 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:51.344 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:51.344 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:51.344 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:51.344 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:51.344 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:51.344 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:51.344 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:51.344 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:51.344 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:51.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.328+0000 7ff4affff640 4 set_mon_vals no callback set 2024-04-22T08:12:51.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.328+0000 7ff4bfcbd640 10 monclient: _renew_subs 2024-04-22T08:12:51.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.328+0000 7ff4bfcbd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:51.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.328+0000 7ff4bfcbd640 1 -- 172.21.15.40:0/1273602216 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4b8122080 con 0x7ff4b80a1d50 2024-04-22T08:12:51.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.329+0000 7ff4ae7fc640 1 -- 172.21.15.40:0/1273602216 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 4) v1 ==== 44726+0+0 (secure 0 0 0) 0x7ff4a801e460 con 0x7ff4b80a1d50 2024-04-22T08:12:51.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.329+0000 7ff4bfcbd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:51.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.329+0000 7ff4bfcbd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:51.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.329+0000 7ff4bfcbd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:51.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.329+0000 7ff4ae7fc640 1 --2- 172.21.15.40:0/1273602216 >> v2:172.21.15.189:6800/30168 conn(0x7ff49c037740 0x7ff49c039c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:51.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.329+0000 7ff4bfcbd640 1 -- 172.21.15.40:0/1273602216 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4b8121c50 con 0x7ff4b80a1d50 2024-04-22T08:12:51.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.329+0000 7ff4ae7fc640 1 -- 172.21.15.40:0/1273602216 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 3506+0+0 (secure 0 0 0) 0x7ff4a804a580 con 0x7ff4b80a1d50 2024-04-22T08:12:51.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.329+0000 7ff4bda59640 1 --2- 172.21.15.40:0/1273602216 >> v2:172.21.15.189:6800/30168 conn(0x7ff49c037740 0x7ff49c039c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:51.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.329+0000 7ff4bda59640 10 monclient: get_auth_request con 0x7ff49c037740 auth_method 0 2024-04-22T08:12:51.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.331+0000 7ff4bda59640 1 --2- 172.21.15.40:0/1273602216 >> v2:172.21.15.189:6800/30168 conn(0x7ff49c037740 0x7ff49c039c00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff4b402b820 tx=0x7ff4b4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:51.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.333+0000 7ff4ae7fc640 1 -- 172.21.15.40:0/1273602216 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7ff4a8011930 con 0x7ff4b80a1d50 2024-04-22T08:12:51.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.333+0000 7ff4ae7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:51.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.333+0000 7ff4ae7fc640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:12:51.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.530+0000 7ff4bfcbd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-22T08:12:51.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.530+0000 7ff4bfcbd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-22T08:12:51.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.530+0000 7ff4bfcbd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:51.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.530+0000 7ff4bfcbd640 1 -- 172.21.15.40:0/1273602216 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4b804e9b0 con 0x7ff4b80a1d50 2024-04-22T08:12:51.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.531+0000 7ff4ae7fc640 1 -- 172.21.15.40:0/1273602216 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+10524 (secure 0 0 0) 0x7ff4a8037090 con 0x7ff4b80a1d50 2024-04-22T08:12:51.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.531+0000 7ff4ae7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-22T08:12:51.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.531+0000 7ff4ae7fc640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:12:51.531 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:12:51.531 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":14,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","created":"2024-04-22T08:12:36.003043+0000","modified":"2024-04-22T08:12:50.824327+0000","last_up_change":"2024-04-22T08:12:47.016043+0000","last_in_change":"2024-04-22T08:12:40.364359+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T08:12:48.855046+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-22T08:12:48.986472+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b9a0210c-5881-4ca0-8318-a4a987109405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":30669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6805","nonce":30669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6807","nonce":30669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":30669}]},"public_addr":"172.21.15.40:6804/30669","cluster_addr":"172.21.15.40:6805/30669","heartbeat_back_addr":"172.21.15.40:6807/30669","heartbeat_front_addr":"172.21.15.40:6806/30669","state":["exists","up"]},{"osd":1,"uuid":"11370cc8-09a6-4c35-b92c-4d926c6c18d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":30670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6801","nonce":30670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6803","nonce":30670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":30670}]},"public_addr":"172.21.15.40:6800/30670","cluster_addr":"172.21.15.40:6801/30670","heartbeat_back_addr":"172.21.15.40:6803/30670","heartbeat_front_addr":"172.21.15.40:6802/30670","state":["exists","up"]},{"osd":2,"uuid":"d03db34e-f630-4a3e-86fb-7234bee0d0f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":30671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":30671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6811","nonce":30671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":30671}]},"public_addr":"172.21.15.40:6808/30671","cluster_addr":"172.21.15.40:6809/30671","heartbeat_back_addr":"172.21.15.40:6811/30671","heartbeat_front_addr":"172.21.15.40:6810/30671","state":["exists","up"]},{"osd":3,"uuid":"d94b2052-5847-4955-b1d4-b7e627572811","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6805","nonce":30673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6807","nonce":30673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6811","nonce":30673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6809","nonce":30673}]},"public_addr":"172.21.15.189:6805/30673","cluster_addr":"172.21.15.189:6807/30673","heartbeat_back_addr":"172.21.15.189:6811/30673","heartbeat_front_addr":"172.21.15.189:6809/30673","state":["exists","up"]},{"osd":4,"uuid":"cb626202-7411-4299-8d20-988bcc9103f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6801","nonce":30672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":30672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":30672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6803","nonce":30672}]},"public_addr":"172.21.15.189:6801/30672","cluster_addr":"172.21.15.189:6802/30672","heartbeat_back_addr":"172.21.15.189:6804/30672","heartbeat_front_addr":"172.21.15.189:6803/30672","state":["exists","up"]},{"osd":5,"uuid":"abbac246-54d6-47d7-a41c-a68df0b80ddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":30674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":30674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":30674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":30674}]},"public_addr":"172.21.15.189:6806/30674","cluster_addr":"172.21.15.189:6808/30674","heartbeat_back_addr":"172.21.15.189:6812/30674","heartbeat_front_addr":"172.21.15.189:6810/30674","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T08:12:45.488082+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T08:12:45.467994+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T08:12:45.503580+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T08:12:45.488169+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T08:12:45.465787+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T08:12:45.470729+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T08:12:51.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.533+0000 7ff4bfcbd640 1 -- 172.21.15.40:0/1273602216 >> v2:172.21.15.189:6800/30168 conn(0x7ff49c037740 msgr2=0x7ff49c039c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:51.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.533+0000 7ff4bfcbd640 1 --2- 172.21.15.40:0/1273602216 >> v2:172.21.15.189:6800/30168 conn(0x7ff49c037740 0x7ff49c039c00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff4b402b820 tx=0x7ff4b4008030 comp rx=0 tx=0).stop 2024-04-22T08:12:51.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.533+0000 7ff4bfcbd640 10 monclient: shutdown 2024-04-22T08:12:51.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.533+0000 7ff4bfcbd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:51.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.533+0000 7ff4bfcbd640 1 -- 172.21.15.40:0/1273602216 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff4b80a1d50 msgr2=0x7ff4b811ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:51.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.533+0000 7ff4bfcbd640 1 --2- 172.21.15.40:0/1273602216 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff4b80a1d50 0x7ff4b811ea40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff4a800d5f0 tx=0x7ff4a800d9f0 comp rx=0 tx=0).stop 2024-04-22T08:12:51.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.533+0000 7ff4bfcbd640 1 -- 172.21.15.40:0/1273602216 shutdown_connections 2024-04-22T08:12:51.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.533+0000 7ff4bfcbd640 1 --2- 172.21.15.40:0/1273602216 >> v2:172.21.15.189:6800/30168 conn(0x7ff49c037740 0x7ff49c039c00 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:51.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.533+0000 7ff4bfcbd640 1 --2- 172.21.15.40:0/1273602216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4b80a2180 0x7ff4b811ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:51.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.533+0000 7ff4bfcbd640 1 --2- 172.21.15.40:0/1273602216 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff4b80a1d50 0x7ff4b811ea40 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:51.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.533+0000 7ff4bfcbd640 1 --2- 172.21.15.40:0/1273602216 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4b80a1970 0x7ff4b81cb0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:51.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.533+0000 7ff4bfcbd640 1 -- 172.21.15.40:0/1273602216 >> 172.21.15.40:0/1273602216 conn(0x7ff4b80a4ef0 msgr2=0x7ff4b81352c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:51.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.533+0000 7ff4bfcbd640 1 -- 172.21.15.40:0/1273602216 shutdown_connections 2024-04-22T08:12:51.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.533+0000 7ff4bfcbd640 1 -- 172.21.15.40:0/1273602216 wait complete. 2024-04-22T08:12:51.543 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-22T08:12:51.543 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T08:12:51.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.644+0000 7fddf748f640 10 monclient: get_monmap_and_config 2024-04-22T08:12:51.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.644+0000 7fddf748f640 10 monclient: build_initial_monmap 2024-04-22T08:12:51.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.644+0000 7fddf748f640 10 monclient: monmap: 2024-04-22T08:12:51.644 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:51.644 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:51.644 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:51.645064+0000 2024-04-22T08:12:51.644 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:51.645064+0000 2024-04-22T08:12:51.644 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:51.644 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:51.644 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:12:51.644 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:12:51.645 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:12:51.645 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:12:51.645 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:12:51.645 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:51.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.644+0000 7fddf748f640 1 Processor -- start 2024-04-22T08:12:51.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.645+0000 7fddf748f640 1 -- start start 2024-04-22T08:12:51.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.645+0000 7fddf748f640 10 monclient: init 2024-04-22T08:12:51.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.645+0000 7fddf748f640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:51.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.645+0000 7fddf748f640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-04-22T08:12:51.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.645+0000 7fddf748f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fddf009ee10 addr v1:172.21.15.189:6789/0 2024-04-22T08:12:51.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.645+0000 7fddf748f640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fddf01361e0 0x7fddf013b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:51.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.645+0000 7fddf748f640 10 monclient(hunting): picked mon.noname-b con 0x7fddf01361e0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:51.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.645+0000 7fddf748f640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fddf013b5a0 0x7fddf013d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:51.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.645+0000 7fddf748f640 10 monclient(hunting): picked mon.noname-c con 0x7fddf013b5a0 addr v2:172.21.15.189:3300/0 2024-04-22T08:12:51.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.645+0000 7fddf748f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddf0136c80 con 0x7fddf009ee10 2024-04-22T08:12:51.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.645+0000 7fddf748f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:51.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.645+0000 7fddf748f640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddf00a3f40 con 0x7fddf013b5a0 2024-04-22T08:12:51.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.645+0000 7fddf748f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:51.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.645+0000 7fddf748f640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddf009f1f0 con 0x7fddf01361e0 2024-04-22T08:12:51.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.645+0000 7fddf748f640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:51.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.645+0000 7fddf748f640 10 monclient(hunting): authenticate will time out at 845.639338s 2024-04-22T08:12:51.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.645+0000 7fddf4a2a640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fddf01361e0 0x7fddf013b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:51.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.645+0000 7fddf4a2a640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fddf01361e0 0x7fddf013b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49612/0 (socket says 172.21.15.40:49612) 2024-04-22T08:12:51.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.645+0000 7fddf4a2a640 1 -- 172.21.15.40:0/2768463108 learned_addr learned my addr 172.21.15.40:0/2768463108 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:51.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.645+0000 7fddf4a2a640 10 monclient(hunting): get_auth_request con 0x7fddf01361e0 auth_method 0 2024-04-22T08:12:51.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.645+0000 7fddf4a2a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:51.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.645+0000 7fddf4a2a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:51.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.645+0000 7fddf4a2a640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:51.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.645+0000 7fddf522b640 1 --1- 172.21.15.40:0/2768463108 >> v1:172.21.15.189:6789/0 conn(0x7fddf009ee10 0x7fddf00a19c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.40:43566/0 (socket says 172.21.15.40:43566) 2024-04-22T08:12:51.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.645+0000 7fddf5a2c640 1 --2- 172.21.15.40:0/2768463108 >> v2:172.21.15.189:3300/0 conn(0x7fddf013b5a0 0x7fddf013d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:51.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.645+0000 7fddf5a2c640 10 monclient(hunting): get_auth_request con 0x7fddf013b5a0 auth_method 0 2024-04-22T08:12:51.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.645+0000 7fddf5a2c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:51.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.645+0000 7fddf5a2c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:51.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.645+0000 7fddf5a2c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:51.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.645+0000 7fddf4a2a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:51.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.645+0000 7fddf4a2a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:51.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.645+0000 7fddf4a2a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:51.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.645+0000 7fddf5a2c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:51.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.645+0000 7fddf5a2c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:51.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.645+0000 7fddf5a2c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:51.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.645+0000 7fddf4a2a640 10 monclient(hunting): handle_auth_done global_id 4217 payload 274 2024-04-22T08:12:51.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.646+0000 7fddf4a2a640 1 -- 172.21.15.40:0/2768463108 >> v2:172.21.15.189:3300/0 conn(0x7fddf013b5a0 msgr2=0x7fddf013d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:51.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.646+0000 7fddf4a2a640 1 --2- 172.21.15.40:0/2768463108 >> v2:172.21.15.189:3300/0 conn(0x7fddf013b5a0 0x7fddf013d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:51.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.646+0000 7fddf4a2a640 1 -- 172.21.15.40:0/2768463108 >> v1:172.21.15.189:6789/0 conn(0x7fddf009ee10 legacy=0x7fddf00a19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:51.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.646+0000 7fddf4a2a640 10 monclient: _finish_hunting 0 2024-04-22T08:12:51.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.646+0000 7fddf4a2a640 1 monclient: found mon.noname-b 2024-04-22T08:12:51.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.646+0000 7fddf4a2a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:51.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.646+0000 7fddf4a2a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:12:51.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.646+0000 7fddf4a2a640 1 -- 172.21.15.40:0/2768463108 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf00a20d0 con 0x7fddf01361e0 2024-04-22T08:12:51.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.648+0000 7fddf4a2a640 1 -- 172.21.15.40:0/2768463108 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fddf01361e0 msgr2=0x7fddf013b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:51.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.649+0000 7fddf4a2a640 1 --2- 172.21.15.40:0/2768463108 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fddf01361e0 0x7fddf013b060 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fdde0009870 tx=0x7fdde002b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b4c7723a5135111 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:51.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.649+0000 7fdde7fff640 1 -- 172.21.15.40:0/2768463108 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde0031950 con 0x7fddf01361e0 2024-04-22T08:12:51.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.649+0000 7fdde7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:51.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.649+0000 7fdde7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:12:51.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.649+0000 7fdde7fff640 10 monclient: dump: 2024-04-22T08:12:51.659 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:51.659 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:51.659 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:51.660 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:51.660 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:51.660 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:51.660 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:51.660 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:51.660 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:51.660 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:51.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.649+0000 7fdde7fff640 10 monclient: _finish_auth 0 2024-04-22T08:12:51.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.649+0000 7fdde7fff640 10 monclient: _check_auth_tickets 2024-04-22T08:12:51.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.649+0000 7fdde7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:51.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.649+0000 7fdde7fff640 1 -- 172.21.15.40:0/2768463108 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdde0031ab0 con 0x7fddf01361e0 2024-04-22T08:12:51.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.649+0000 7fdde7fff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:51.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.649+0000 7fdde7fff640 1 -- 172.21.15.40:0/2768463108 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde0031e40 con 0x7fddf01361e0 2024-04-22T08:12:51.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.649+0000 7fddf748f640 5 monclient: authenticate success, global_id 4217 2024-04-22T08:12:51.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.649+0000 7fddf748f640 10 monclient: get_monmap_and_config success 2024-04-22T08:12:51.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.649+0000 7fddf748f640 4 set_mon_vals no callback set 2024-04-22T08:12:51.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.649+0000 7fdde7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:51.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.649+0000 7fdde7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:12:51.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.649+0000 7fdde7fff640 10 monclient: dump: 2024-04-22T08:12:51.661 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:51.661 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:51.662 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:51.662 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:51.662 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:51.662 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:51.662 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:51.662 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:51.662 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:51.662 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:51.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.649+0000 7fddf748f640 10 monclient: shutdown 2024-04-22T08:12:51.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.649+0000 7fddf748f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:51.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.649+0000 7fddf748f640 1 -- 172.21.15.40:0/2768463108 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fddf01361e0 msgr2=0x7fddf013b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:51.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.649+0000 7fddf748f640 1 --2- 172.21.15.40:0/2768463108 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fddf01361e0 0x7fddf013b060 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fdde0009870 tx=0x7fdde002b7e0 comp rx=0 tx=0).stop 2024-04-22T08:12:51.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.649+0000 7fddf748f640 1 -- 172.21.15.40:0/2768463108 shutdown_connections 2024-04-22T08:12:51.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.649+0000 7fddf748f640 1 --2- 172.21.15.40:0/2768463108 >> v2:172.21.15.189:3300/0 conn(0x7fddf013b5a0 0x7fddf013d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:51.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.649+0000 7fddf748f640 1 --2- 172.21.15.40:0/2768463108 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fddf01361e0 0x7fddf013b060 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:51.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.649+0000 7fddf748f640 1 -- 172.21.15.40:0/2768463108 >> 172.21.15.40:0/2768463108 conn(0x7fddf00a4e80 msgr2=0x7fddf00a5290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:51.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.650+0000 7fddf748f640 1 -- 172.21.15.40:0/2768463108 shutdown_connections 2024-04-22T08:12:51.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.650+0000 7fddf748f640 1 -- 172.21.15.40:0/2768463108 wait complete. 2024-04-22T08:12:51.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.650+0000 7fddf748f640 10 monclient: build_initial_monmap 2024-04-22T08:12:51.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.650+0000 7fddf748f640 10 monclient: monmap: 2024-04-22T08:12:51.663 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:51.663 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:12:51.663 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:12:51.664 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:12:51.664 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:51.664 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:51.664 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:12:51.664 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:12:51.664 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:12:51.664 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:51.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.651+0000 7fddf748f640 1 Processor -- start 2024-04-22T08:12:51.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.651+0000 7fddf748f640 1 -- start start 2024-04-22T08:12:51.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.651+0000 7fddf748f640 10 monclient: init 2024-04-22T08:12:51.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.651+0000 7fddf748f640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:51.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.651+0000 7fddf748f640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-22T08:12:51.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.651+0000 7fddf748f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddf009ee10 0x7fddf0131be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:51.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.651+0000 7fddf748f640 10 monclient(hunting): picked mon.noname-b con 0x7fddf009ee10 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:51.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.651+0000 7fddf748f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fddf01361e0 0x7fddf0132120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:51.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.651+0000 7fddf748f640 10 monclient(hunting): picked mon.noname-a con 0x7fddf01361e0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:51.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.651+0000 7fddf748f640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fddf013b5a0 0x7fddf006db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:51.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.651+0000 7fddf748f640 10 monclient(hunting): picked mon.noname-c con 0x7fddf013b5a0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:51.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.651+0000 7fddf748f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:51.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.651+0000 7fddf748f640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf01cb7c0 con 0x7fddf01361e0 2024-04-22T08:12:51.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.651+0000 7fddf748f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:51.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.651+0000 7fddf748f640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf00a27c0 con 0x7fddf009ee10 2024-04-22T08:12:51.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.651+0000 7fddf748f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:51.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.651+0000 7fddf748f640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddf0133b20 con 0x7fddf013b5a0 2024-04-22T08:12:51.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.651+0000 7fddf748f640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:51.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.651+0000 7fddf748f640 10 monclient(hunting): authenticate will time out at 845.645427s 2024-04-22T08:12:51.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.651+0000 7fddf5a2c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fddf013b5a0 0x7fddf006db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:51.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.651+0000 7fddf4a2a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fddf01361e0 0x7fddf0132120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:51.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.651+0000 7fddf522b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddf009ee10 0x7fddf0131be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:51.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.651+0000 7fddf4a2a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fddf01361e0 0x7fddf0132120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57792/0 (socket says 172.21.15.40:57792) 2024-04-22T08:12:51.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.651+0000 7fddf522b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddf009ee10 0x7fddf0131be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.40:42626/0 (socket says 172.21.15.40:42626) 2024-04-22T08:12:51.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.651+0000 7fddf4a2a640 1 -- 172.21.15.40:0/2516359369 learned_addr learned my addr 172.21.15.40:0/2516359369 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:51.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.651+0000 7fddf5a2c640 1 -- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fddf013b5a0 msgr2=0x7fddf006db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-22T08:12:51.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.651+0000 7fddf4a2a640 10 monclient(hunting): get_auth_request con 0x7fddf01361e0 auth_method 0 2024-04-22T08:12:51.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.651+0000 7fddf5a2c640 1 -- 172.21.15.40:0/2516359369 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fddf013b5a0 msgr2=0x7fddf006db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:12:51.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.651+0000 7fddf4a2a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:51.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.651+0000 7fddf4a2a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:51.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.651+0000 7fddf4a2a640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:51.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.651+0000 7fddf5a2c640 1 --2- 172.21.15.40:0/2516359369 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fddf013b5a0 0x7fddf006db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:12:51.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.651+0000 7fddf522b640 10 monclient(hunting): get_auth_request con 0x7fddf009ee10 auth_method 0 2024-04-22T08:12:51.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.651+0000 7fddf5a2c640 1 --2- 172.21.15.40:0/2516359369 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fddf013b5a0 0x7fddf006db40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:12:51.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.651+0000 7fddf522b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:51.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.651+0000 7fddf522b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:51.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.651+0000 7fddf522b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:51.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.651+0000 7fddf4a2a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:51.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.651+0000 7fddf4a2a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:51.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.651+0000 7fddf4a2a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:51.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.651+0000 7fddf522b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:51.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.651+0000 7fddf522b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:51.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.652+0000 7fddf522b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:51.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.652+0000 7fddf4a2a640 10 monclient(hunting): handle_auth_done global_id 4224 payload 824 2024-04-22T08:12:51.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.652+0000 7fddf4a2a640 1 -- 172.21.15.40:0/2516359369 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fddf013b5a0 msgr2=0x7fddf006db40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:12:51.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.652+0000 7fddf4a2a640 1 --2- 172.21.15.40:0/2516359369 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fddf013b5a0 0x7fddf006db40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:51.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.652+0000 7fddf4a2a640 1 -- 172.21.15.40:0/2516359369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddf009ee10 msgr2=0x7fddf0131be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:51.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.652+0000 7fddf4a2a640 1 --2- 172.21.15.40:0/2516359369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddf009ee10 0x7fddf0131be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:51.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.652+0000 7fddf4a2a640 10 monclient: _finish_hunting 0 2024-04-22T08:12:51.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.652+0000 7fddf4a2a640 1 monclient: found mon.noname-a 2024-04-22T08:12:51.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.652+0000 7fddf4a2a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:51.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.652+0000 7fddf4a2a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:12:51.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.652+0000 7fddf4a2a640 1 -- 172.21.15.40:0/2516359369 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde00094d0 con 0x7fddf01361e0 2024-04-22T08:12:51.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.654+0000 7fddf4a2a640 1 -- 172.21.15.40:0/2516359369 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fddf01361e0 msgr2=0x7fddf0132120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:51.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.654+0000 7fddf4a2a640 1 --2- 172.21.15.40:0/2516359369 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fddf01361e0 0x7fddf0132120 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdde00090b0 tx=0x7fdde00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:51.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.654+0000 7fdde5ffb640 1 -- 172.21.15.40:0/2516359369 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde0033c30 con 0x7fddf01361e0 2024-04-22T08:12:51.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.654+0000 7fdde5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:51.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.654+0000 7fdde5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:12:51.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.654+0000 7fdde5ffb640 10 monclient: dump: 2024-04-22T08:12:51.669 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:51.669 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:51.669 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:51.669 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:51.669 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:51.669 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:51.669 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:51.669 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:51.669 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:51.669 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:51.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.654+0000 7fdde5ffb640 10 monclient: _finish_auth 0 2024-04-22T08:12:51.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.654+0000 7fdde5ffb640 10 monclient: _check_auth_tickets 2024-04-22T08:12:51.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.654+0000 7fdde5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:51.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.654+0000 7fddf748f640 5 monclient: authenticate success, global_id 4224 2024-04-22T08:12:51.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.654+0000 7fdde5ffb640 1 -- 172.21.15.40:0/2516359369 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdde0033db0 con 0x7fddf01361e0 2024-04-22T08:12:51.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.654+0000 7fdde5ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:51.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.654+0000 7fdde5ffb640 1 -- 172.21.15.40:0/2516359369 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde003d6b0 con 0x7fddf01361e0 2024-04-22T08:12:51.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.654+0000 7fddf748f640 10 monclient: _renew_subs 2024-04-22T08:12:51.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.654+0000 7fddf748f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:51.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.654+0000 7fddf748f640 1 -- 172.21.15.40:0/2516359369 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddf0071220 con 0x7fddf01361e0 2024-04-22T08:12:51.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.654+0000 7fdde5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:51.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.654+0000 7fdde77fe640 4 set_mon_vals no callback set 2024-04-22T08:12:51.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.654+0000 7fdde5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:12:51.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.654+0000 7fdde5ffb640 10 monclient: dump: 2024-04-22T08:12:51.671 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:51.671 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:51.671 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:51.671 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:51.671 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:51.671 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:51.671 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:51.671 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:51.672 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:51.672 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:51.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.655+0000 7fddf748f640 10 monclient: _renew_subs 2024-04-22T08:12:51.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.655+0000 7fddf748f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:51.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.655+0000 7fddf748f640 1 -- 172.21.15.40:0/2516359369 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddf006f6f0 con 0x7fddf01361e0 2024-04-22T08:12:51.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.656+0000 7fdde5ffb640 1 -- 172.21.15.40:0/2516359369 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 4) v1 ==== 44726+0+0 (secure 0 0 0) 0x7fdde003a040 con 0x7fddf01361e0 2024-04-22T08:12:51.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.656+0000 7fddf748f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:51.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.656+0000 7fddf748f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:51.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.656+0000 7fddf748f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:51.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.656+0000 7fddf748f640 1 -- 172.21.15.40:0/2516359369 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddf00a23c0 con 0x7fddf01361e0 2024-04-22T08:12:51.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.656+0000 7fdde5ffb640 1 --2- 172.21.15.40:0/2516359369 >> v2:172.21.15.189:6800/30168 conn(0x7fddd4037790 0x7fddd4039c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:51.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.656+0000 7fdde5ffb640 1 -- 172.21.15.40:0/2516359369 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 3506+0+0 (secure 0 0 0) 0x7fdde006bfb0 con 0x7fddf01361e0 2024-04-22T08:12:51.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.656+0000 7fddf522b640 1 --2- 172.21.15.40:0/2516359369 >> v2:172.21.15.189:6800/30168 conn(0x7fddd4037790 0x7fddd4039c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:51.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.656+0000 7fddf522b640 10 monclient: get_auth_request con 0x7fddd4037790 auth_method 0 2024-04-22T08:12:51.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.659+0000 7fddf522b640 1 --2- 172.21.15.40:0/2516359369 >> v2:172.21.15.189:6800/30168 conn(0x7fddd4037790 0x7fddd4039c50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fddec005fd0 tx=0x7fddec007e60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:51.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.661+0000 7fdde5ffb640 1 -- 172.21.15.40:0/2516359369 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7fddf00a23c0 con 0x7fddf01361e0 2024-04-22T08:12:51.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.661+0000 7fdde5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:51.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.661+0000 7fdde5ffb640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:12:51.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.846+0000 7fddf748f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-22T08:12:51.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.846+0000 7fddf748f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-22T08:12:51.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.846+0000 7fddf748f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:51.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.846+0000 7fddf748f640 1 -- 172.21.15.40:0/2516359369 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fddf00a23c0 con 0x7fddf01361e0 2024-04-22T08:12:51.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.847+0000 7fdde5ffb640 1 -- 172.21.15.40:0/2516359369 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+10524 (secure 0 0 0) 0x7fddf00a23c0 con 0x7fddf01361e0 2024-04-22T08:12:51.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.847+0000 7fdde5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-22T08:12:51.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.847+0000 7fdde5ffb640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:12:51.847 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:12:51.848 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":14,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","created":"2024-04-22T08:12:36.003043+0000","modified":"2024-04-22T08:12:50.824327+0000","last_up_change":"2024-04-22T08:12:47.016043+0000","last_in_change":"2024-04-22T08:12:40.364359+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T08:12:48.855046+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-22T08:12:48.986472+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b9a0210c-5881-4ca0-8318-a4a987109405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":30669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6805","nonce":30669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6807","nonce":30669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":30669}]},"public_addr":"172.21.15.40:6804/30669","cluster_addr":"172.21.15.40:6805/30669","heartbeat_back_addr":"172.21.15.40:6807/30669","heartbeat_front_addr":"172.21.15.40:6806/30669","state":["exists","up"]},{"osd":1,"uuid":"11370cc8-09a6-4c35-b92c-4d926c6c18d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":30670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6801","nonce":30670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6803","nonce":30670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":30670}]},"public_addr":"172.21.15.40:6800/30670","cluster_addr":"172.21.15.40:6801/30670","heartbeat_back_addr":"172.21.15.40:6803/30670","heartbeat_front_addr":"172.21.15.40:6802/30670","state":["exists","up"]},{"osd":2,"uuid":"d03db34e-f630-4a3e-86fb-7234bee0d0f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":30671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":30671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6811","nonce":30671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":30671}]},"public_addr":"172.21.15.40:6808/30671","cluster_addr":"172.21.15.40:6809/30671","heartbeat_back_addr":"172.21.15.40:6811/30671","heartbeat_front_addr":"172.21.15.40:6810/30671","state":["exists","up"]},{"osd":3,"uuid":"d94b2052-5847-4955-b1d4-b7e627572811","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6805","nonce":30673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6807","nonce":30673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6811","nonce":30673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6809","nonce":30673}]},"public_addr":"172.21.15.189:6805/30673","cluster_addr":"172.21.15.189:6807/30673","heartbeat_back_addr":"172.21.15.189:6811/30673","heartbeat_front_addr":"172.21.15.189:6809/30673","state":["exists","up"]},{"osd":4,"uuid":"cb626202-7411-4299-8d20-988bcc9103f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6801","nonce":30672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":30672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":30672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6803","nonce":30672}]},"public_addr":"172.21.15.189:6801/30672","cluster_addr":"172.21.15.189:6802/30672","heartbeat_back_addr":"172.21.15.189:6804/30672","heartbeat_front_addr":"172.21.15.189:6803/30672","state":["exists","up"]},{"osd":5,"uuid":"abbac246-54d6-47d7-a41c-a68df0b80ddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":30674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":30674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":30674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":30674}]},"public_addr":"172.21.15.189:6806/30674","cluster_addr":"172.21.15.189:6808/30674","heartbeat_back_addr":"172.21.15.189:6812/30674","heartbeat_front_addr":"172.21.15.189:6810/30674","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T08:12:45.488082+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T08:12:45.467994+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T08:12:45.503580+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T08:12:45.488169+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T08:12:45.465787+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T08:12:45.470729+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T08:12:51.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.849+0000 7fddf748f640 1 -- 172.21.15.40:0/2516359369 >> v2:172.21.15.189:6800/30168 conn(0x7fddd4037790 msgr2=0x7fddd4039c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:51.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.849+0000 7fddf748f640 1 --2- 172.21.15.40:0/2516359369 >> v2:172.21.15.189:6800/30168 conn(0x7fddd4037790 0x7fddd4039c50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fddec005fd0 tx=0x7fddec007e60 comp rx=0 tx=0).stop 2024-04-22T08:12:51.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.849+0000 7fddf748f640 10 monclient: shutdown 2024-04-22T08:12:51.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.849+0000 7fddf748f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:51.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.849+0000 7fddf748f640 1 -- 172.21.15.40:0/2516359369 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fddf01361e0 msgr2=0x7fddf0132120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:51.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.849+0000 7fddf748f640 1 --2- 172.21.15.40:0/2516359369 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fddf01361e0 0x7fddf0132120 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdde00090b0 tx=0x7fdde00046c0 comp rx=0 tx=0).stop 2024-04-22T08:12:51.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.849+0000 7fddf748f640 1 -- 172.21.15.40:0/2516359369 shutdown_connections 2024-04-22T08:12:51.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.849+0000 7fddf748f640 1 --2- 172.21.15.40:0/2516359369 >> v2:172.21.15.189:6800/30168 conn(0x7fddd4037790 0x7fddd4039c50 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:51.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.849+0000 7fddf748f640 1 --2- 172.21.15.40:0/2516359369 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fddf013b5a0 0x7fddf006db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:51.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.849+0000 7fddf748f640 1 --2- 172.21.15.40:0/2516359369 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fddf01361e0 0x7fddf0132120 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:51.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.849+0000 7fddf748f640 1 --2- 172.21.15.40:0/2516359369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddf009ee10 0x7fddf0131be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:51.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.849+0000 7fddf748f640 1 -- 172.21.15.40:0/2516359369 >> 172.21.15.40:0/2516359369 conn(0x7fddf00a4e80 msgr2=0x7fddf0132d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:51.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.849+0000 7fddf748f640 1 -- 172.21.15.40:0/2516359369 shutdown_connections 2024-04-22T08:12:51.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.849+0000 7fddf748f640 1 -- 172.21.15.40:0/2516359369 wait complete. 2024-04-22T08:12:51.860 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-22T08:12:51.861 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-22T08:12:51.861 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-22T08:12:51.861 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-22T08:12:51.862 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-22T08:12:51.862 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-22T08:12:51.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.976+0000 7fad3d3d0640 10 monclient: get_monmap_and_config 2024-04-22T08:12:51.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.976+0000 7fad3d3d0640 10 monclient: build_initial_monmap 2024-04-22T08:12:51.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.976+0000 7fad3d3d0640 10 monclient: monmap: 2024-04-22T08:12:51.977 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:51.977 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:51.977 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:51.977724+0000 2024-04-22T08:12:51.977 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:51.977724+0000 2024-04-22T08:12:51.977 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:51.977 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:51.977 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:12:51.977 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:12:51.977 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:12:51.977 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:12:51.977 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:12:51.977 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:51.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.977+0000 7f43133a5640 10 monclient: get_monmap_and_config 2024-04-22T08:12:51.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.977+0000 7f43133a5640 10 monclient: build_initial_monmap 2024-04-22T08:12:51.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.977+0000 7f43133a5640 10 monclient: monmap: 2024-04-22T08:12:51.978 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:51.978 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:51.978 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:51.978019+0000 2024-04-22T08:12:51.979 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:51.978019+0000 2024-04-22T08:12:51.979 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:51.979 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:51.979 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:12:51.979 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:12:51.979 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:12:51.979 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:12:51.979 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:12:51.979 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:51.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.977+0000 7fad3d3d0640 1 Processor -- start 2024-04-22T08:12:51.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.977+0000 7fad3d3d0640 1 -- start start 2024-04-22T08:12:51.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.977+0000 7fad3d3d0640 10 monclient: init 2024-04-22T08:12:51.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.977+0000 7f43133a5640 1 Processor -- start 2024-04-22T08:12:51.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.978+0000 7f43133a5640 1 -- start start 2024-04-22T08:12:51.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.978+0000 7f43133a5640 10 monclient: init 2024-04-22T08:12:51.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.978+0000 7fad3d3d0640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:51.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.978+0000 7fad3d3d0640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-04-22T08:12:51.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.978+0000 7fad36ffd640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fad380a1c60 0x7fad380a2040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:38868/0 (socket says 172.21.15.40:38868) 2024-04-22T08:12:51.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.978+0000 7f43133a5640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:51.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.978+0000 7f43133a5640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-04-22T08:12:51.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.978+0000 7f43133a5640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f430c0a1ba0 0x7f430c0a1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:51.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.978+0000 7f43133a5640 10 monclient(hunting): picked mon.noname-b con 0x7f430c0a1ba0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:51.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.978+0000 7f43133a5640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f430c136230 0x7f430c0a2d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:51.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.978+0000 7f43133a5640 10 monclient(hunting): picked mon.noname-a con 0x7f430c136230 addr v2:172.21.15.40:3300/0 2024-04-22T08:12:51.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.978+0000 7f43133a5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f430c136670 addr v1:172.21.15.189:6789/0 2024-04-22T08:12:51.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.978+0000 7f43133a5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f430c09fe80 con 0x7f430c136670 2024-04-22T08:12:51.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.978+0000 7f43133a5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:51.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.978+0000 7f43133a5640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f430c0722a0 con 0x7f430c136230 2024-04-22T08:12:51.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.978+0000 7f43133a5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:51.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.978+0000 7f43133a5640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f430c0a4290 con 0x7f430c0a1ba0 2024-04-22T08:12:51.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.978+0000 7f43133a5640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:51.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.978+0000 7fad36ffd640 1 -- 172.21.15.40:0/778573479 learned_addr learned my addr 172.21.15.40:0/778573479 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:12:51.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.978+0000 7f43133a5640 10 monclient(hunting): authenticate will time out at 845.972578s 2024-04-22T08:12:51.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.979+0000 7f4310940640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f430c136230 0x7f430c0a2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:51.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.979+0000 7f4311141640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f430c0a1ba0 0x7f430c0a1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:51.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.979+0000 7f4311141640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f430c0a1ba0 0x7f430c0a1f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49636/0 (socket says 172.21.15.40:49636) 2024-04-22T08:12:51.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.978+0000 7fad3d3d0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fad380a1c60 addr v1:172.21.15.40:6789/0 2024-04-22T08:12:51.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.978+0000 7fad3d3d0640 1 --2- 172.21.15.40:0/778573479 >> v2:172.21.15.189:3300/0 conn(0x7fad380a31d0 0x7fad3813aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:51.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.978+0000 7fad3d3d0640 10 monclient(hunting): picked mon.noname-c con 0x7fad380a31d0 addr v2:172.21.15.189:3300/0 2024-04-22T08:12:51.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.978+0000 7fad3d3d0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fad3813b410 addr v1:172.21.15.189:6789/0 2024-04-22T08:12:51.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.978+0000 7fad3d3d0640 1 -- 172.21.15.40:0/778573479 --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad380a41f0 con 0x7fad380a1c60 2024-04-22T08:12:51.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.979+0000 7f4311141640 1 -- 172.21.15.40:0/881331257 learned_addr learned my addr 172.21.15.40:0/881331257 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:51.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.979+0000 7f4311141640 10 monclient(hunting): get_auth_request con 0x7f430c0a1ba0 auth_method 0 2024-04-22T08:12:51.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.979+0000 7f4311141640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:51.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.979+0000 7f4311141640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:51.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.978+0000 7fad3d3d0640 1 -- 172.21.15.40:0/778573479 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad3809fe90 con 0x7fad3813b410 2024-04-22T08:12:51.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.978+0000 7fad3d3d0640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:51.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.978+0000 7fad3d3d0640 1 -- 172.21.15.40:0/778573479 --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad380a3660 con 0x7fad380a31d0 2024-04-22T08:12:51.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.979+0000 7fad3d3d0640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:51.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.979+0000 7fad3d3d0640 10 monclient(hunting): authenticate will time out at 845.972779s 2024-04-22T08:12:51.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.979+0000 7f4311141640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:51.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.979+0000 7f4310940640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f430c136230 0x7f430c0a2d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57804/0 (socket says 172.21.15.40:57804) 2024-04-22T08:12:51.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.979+0000 7f4310940640 10 monclient(hunting): get_auth_request con 0x7f430c136230 auth_method 0 2024-04-22T08:12:51.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.979+0000 7f4310940640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:51.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.979+0000 7f4310940640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:51.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.979+0000 7f4310940640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:51.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.979+0000 7fad367fc640 1 --2- 172.21.15.40:0/778573479 >> v2:172.21.15.189:3300/0 conn(0x7fad380a31d0 0x7fad3813aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:51.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.979+0000 7fad367fc640 10 monclient(hunting): get_auth_request con 0x7fad380a31d0 auth_method 0 2024-04-22T08:12:51.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.979+0000 7fad367fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:51.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.979+0000 7fad367fc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:51.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.979+0000 7fad367fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:51.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.979+0000 7f4311141640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:51.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.979+0000 7f4311141640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:51.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.979+0000 7f4311141640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:51.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.979+0000 7f4310940640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:51.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.979+0000 7fad367fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:51.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.979+0000 7f4310940640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:51.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.979+0000 7fad367fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:51.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.979+0000 7fad367fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:51.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.979+0000 7f4310940640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:51.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.979+0000 7fad367fc640 10 monclient(hunting): handle_auth_done global_id 4213 payload 274 2024-04-22T08:12:51.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.979+0000 7fad367fc640 1 -- 172.21.15.40:0/778573479 >> v1:172.21.15.189:6789/0 conn(0x7fad3813b410 legacy=0x7fad3813d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:51.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.979+0000 7fad367fc640 1 -- 172.21.15.40:0/778573479 >> v1:172.21.15.40:6789/0 conn(0x7fad380a1c60 legacy=0x7fad380a2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:51.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.979+0000 7f4311141640 10 monclient(hunting): handle_auth_done global_id 4220 payload 274 2024-04-22T08:12:51.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.979+0000 7fad367fc640 10 monclient: _finish_hunting 0 2024-04-22T08:12:51.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.979+0000 7fad367fc640 1 monclient: found mon.noname-c 2024-04-22T08:12:51.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.979+0000 7fad367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:51.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.979+0000 7fad367fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.189:3300/0 2024-04-22T08:12:51.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.979+0000 7fad367fc640 1 -- 172.21.15.40:0/778573479 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad380a2750 con 0x7fad380a31d0 2024-04-22T08:12:51.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.979+0000 7f4311141640 1 -- 172.21.15.40:0/881331257 >> v2:172.21.15.40:3300/0 conn(0x7f430c136230 msgr2=0x7f430c0a2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:51.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.979+0000 7f4311141640 1 --2- 172.21.15.40:0/881331257 >> v2:172.21.15.40:3300/0 conn(0x7f430c136230 0x7f430c0a2d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:51.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.979+0000 7f4311141640 1 -- 172.21.15.40:0/881331257 >> v1:172.21.15.189:6789/0 conn(0x7f430c136670 legacy=0x7f430c0a32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:51.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.979+0000 7f4310940640 1 --2- 172.21.15.40:0/881331257 >> v2:172.21.15.40:3300/0 conn(0x7f430c136230 0x7f430c0a2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T08:12:51.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.979+0000 7f4311141640 10 monclient: _finish_hunting 0 2024-04-22T08:12:51.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.979+0000 7f4311141640 1 monclient: found mon.noname-b 2024-04-22T08:12:51.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.979+0000 7f4311141640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:51.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.979+0000 7f4311141640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:12:51.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.979+0000 7f4311141640 1 -- 172.21.15.40:0/881331257 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f430c0a0160 con 0x7f430c0a1ba0 2024-04-22T08:12:51.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.982+0000 7f4311141640 1 -- 172.21.15.40:0/881331257 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f430c0a1ba0 msgr2=0x7f430c0a1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:51.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.982+0000 7fad367fc640 1 -- 172.21.15.40:0/778573479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad380a31d0 msgr2=0x7fad3813aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:51.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.982+0000 7fad367fc640 1 --2- 172.21.15.40:0/778573479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad380a31d0 0x7fad3813aed0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fad380a37a0 tx=0x7fad2c02b6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fef85f1a4c79e4d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:51.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.982+0000 7f4311141640 1 --2- 172.21.15.40:0/881331257 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f430c0a1ba0 0x7f430c0a1f80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4308009870 tx=0x7f430802b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad2d9ac7656592ff server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:51.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.983+0000 7f4303fff640 1 -- 172.21.15.40:0/881331257 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43080319d0 con 0x7f430c0a1ba0 2024-04-22T08:12:51.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.983+0000 7f4303fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:51.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.983+0000 7f4303fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:12:51.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.983+0000 7f4303fff640 10 monclient: dump: 2024-04-22T08:12:51.994 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:51.994 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:51.994 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:51.994 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:51.994 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:51.995 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:51.995 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:51.995 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:51.995 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:51.995 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:51.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.983+0000 7f4303fff640 10 monclient: _finish_auth 0 2024-04-22T08:12:51.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.983+0000 7f4303fff640 10 monclient: _check_auth_tickets 2024-04-22T08:12:51.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.983+0000 7f4303fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:51.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.983+0000 7f43133a5640 5 monclient: authenticate success, global_id 4220 2024-04-22T08:12:51.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.983+0000 7f43133a5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:12:51.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.983+0000 7fad35ffb640 1 -- 172.21.15.40:0/778573479 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2c030a00 con 0x7fad380a31d0 2024-04-22T08:12:51.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.983+0000 7fad35ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:51.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.983+0000 7fad35ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:12:51.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.983+0000 7f4303fff640 1 -- 172.21.15.40:0/881331257 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4308031b30 con 0x7f430c0a1ba0 2024-04-22T08:12:51.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.983+0000 7f4303fff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:51.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.983+0000 7fad35ffb640 10 monclient: dump: 2024-04-22T08:12:51.997 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:51.997 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:51.997 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:51.997 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:51.997 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:51.997 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:51.997 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:51.997 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:51.997 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:51.997 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:51.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.983+0000 7fad35ffb640 10 monclient: _finish_auth 0 2024-04-22T08:12:51.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.983+0000 7f4303fff640 1 -- 172.21.15.40:0/881331257 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4308032450 con 0x7f430c0a1ba0 2024-04-22T08:12:51.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.983+0000 7f4303fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:51.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.983+0000 7f4303fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:12:51.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.983+0000 7f4303fff640 10 monclient: dump: 2024-04-22T08:12:51.998 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:51.998 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:51.998 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:51.998 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:51.998 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:51.998 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:51.998 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:51.999 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:51.999 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:51.999 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:51.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.983+0000 7fad35ffb640 10 monclient: _check_auth_tickets 2024-04-22T08:12:51.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.983+0000 7fad35ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:52.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.983+0000 7fad3d3d0640 5 monclient: authenticate success, global_id 4213 2024-04-22T08:12:52.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.983+0000 7fad3d3d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:12:52.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.983+0000 7fad35ffb640 1 -- 172.21.15.40:0/778573479 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad2c02e050 con 0x7fad380a31d0 2024-04-22T08:12:52.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.983+0000 7fad35ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:52.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.983+0000 7fad35ffb640 1 -- 172.21.15.40:0/778573479 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2c030d80 con 0x7fad380a31d0 2024-04-22T08:12:52.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.983+0000 7fad35ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:52.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.983+0000 7fad35ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:12:52.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.983+0000 7fad35ffb640 10 monclient: dump: 2024-04-22T08:12:52.000 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:52.000 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.000 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.000 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.000 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:52.000 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.001 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:52.001 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:52.001 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:52.001 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.983+0000 7fad3d3d0640 10 monclient: get_monmap_and_config success 2024-04-22T08:12:52.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.983+0000 7fad3d3d0640 4 set_mon_vals no callback set 2024-04-22T08:12:52.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.983+0000 7f43133a5640 10 monclient: get_monmap_and_config success 2024-04-22T08:12:52.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.983+0000 7f43133a5640 4 set_mon_vals no callback set 2024-04-22T08:12:52.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.983+0000 7f43133a5640 10 monclient: shutdown 2024-04-22T08:12:52.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.983+0000 7fad3d3d0640 10 monclient: shutdown 2024-04-22T08:12:52.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.983+0000 7f43133a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:52.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.983+0000 7f43133a5640 1 -- 172.21.15.40:0/881331257 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f430c0a1ba0 msgr2=0x7f430c0a1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.983+0000 7f43133a5640 1 --2- 172.21.15.40:0/881331257 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f430c0a1ba0 0x7f430c0a1f80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4308009870 tx=0x7f430802b860 comp rx=0 tx=0).stop 2024-04-22T08:12:52.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.984+0000 7f43133a5640 1 -- 172.21.15.40:0/881331257 shutdown_connections 2024-04-22T08:12:52.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.983+0000 7fad3d3d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:52.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.983+0000 7fad3d3d0640 1 -- 172.21.15.40:0/778573479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad380a31d0 msgr2=0x7fad3813aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.983+0000 7fad3d3d0640 1 --2- 172.21.15.40:0/778573479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad380a31d0 0x7fad3813aed0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fad380a37a0 tx=0x7fad2c02b6f0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.984+0000 7f43133a5640 1 --2- 172.21.15.40:0/881331257 >> v2:172.21.15.40:3300/0 conn(0x7f430c136230 0x7f430c0a2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.984+0000 7f43133a5640 1 --2- 172.21.15.40:0/881331257 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f430c0a1ba0 0x7f430c0a1f80 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.984+0000 7f43133a5640 1 -- 172.21.15.40:0/881331257 >> 172.21.15.40:0/881331257 conn(0x7f430c0a4e80 msgr2=0x7f430c0a5290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:52.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.984+0000 7fad3d3d0640 1 -- 172.21.15.40:0/778573479 shutdown_connections 2024-04-22T08:12:52.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.984+0000 7fad3d3d0640 1 --2- 172.21.15.40:0/778573479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad380a31d0 0x7fad3813aed0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.984+0000 7fad3d3d0640 1 -- 172.21.15.40:0/778573479 >> 172.21.15.40:0/778573479 conn(0x7fad380a89a0 msgr2=0x7fad380a4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:52.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.984+0000 7f43133a5640 1 -- 172.21.15.40:0/881331257 shutdown_connections 2024-04-22T08:12:52.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.984+0000 7fad3d3d0640 1 -- 172.21.15.40:0/778573479 shutdown_connections 2024-04-22T08:12:52.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.984+0000 7fad3d3d0640 1 -- 172.21.15.40:0/778573479 wait complete. 2024-04-22T08:12:52.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.984+0000 7f43133a5640 1 -- 172.21.15.40:0/881331257 wait complete. 2024-04-22T08:12:52.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.985+0000 7fad3d3d0640 10 monclient: build_initial_monmap 2024-04-22T08:12:52.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.985+0000 7f43133a5640 10 monclient: build_initial_monmap 2024-04-22T08:12:52.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.985+0000 7f43133a5640 10 monclient: monmap: 2024-04-22T08:12:52.006 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:52.006 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:12:52.006 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:12:52.006 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:12:52.006 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:52.006 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.006 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:12:52.006 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:12:52.006 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:12:52.006 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.985+0000 7fad3d3d0640 10 monclient: monmap: 2024-04-22T08:12:52.007 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:52.007 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:12:52.007 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:12:52.007 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:12:52.007 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:52.007 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.007 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:12:52.007 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:12:52.007 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:12:52.007 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.985+0000 7f43133a5640 1 Processor -- start 2024-04-22T08:12:52.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.985+0000 7fad3d3d0640 1 Processor -- start 2024-04-22T08:12:52.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.985+0000 7f43133a5640 1 -- start start 2024-04-22T08:12:52.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.985+0000 7f43133a5640 10 monclient: init 2024-04-22T08:12:52.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.985+0000 7fad3d3d0640 1 -- start start 2024-04-22T08:12:52.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.985+0000 7fad3d3d0640 10 monclient: init 2024-04-22T08:12:52.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7fad3d3d0640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:52.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7fad3d3d0640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-22T08:12:52.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7fad3d3d0640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad380a1c60 0x7fad38130dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7fad3d3d0640 10 monclient(hunting): picked mon.noname-b con 0x7fad380a1c60 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:52.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7f43133a5640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:52.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7f43133a5640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-22T08:12:52.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7f43133a5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f430c0a1ba0 0x7f430c1cac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7f43133a5640 10 monclient(hunting): picked mon.noname-b con 0x7f430c0a1ba0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:52.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7f43133a5640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f430c136230 0x7f430c1cb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7f43133a5640 10 monclient(hunting): picked mon.noname-c con 0x7f430c136230 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:52.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7f43133a5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f430c136670 0x7f430c122050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7f43133a5640 10 monclient(hunting): picked mon.noname-a con 0x7f430c136670 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:52.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7f43133a5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7f43133a5640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f430c125760 con 0x7f430c136670 2024-04-22T08:12:52.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7f43133a5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7f43133a5640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f430c124870 con 0x7f430c0a1ba0 2024-04-22T08:12:52.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7f43133a5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7f43133a5640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f430c0a2710 con 0x7f430c136230 2024-04-22T08:12:52.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7f43133a5640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:52.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7f43133a5640 10 monclient(hunting): authenticate will time out at 845.980280s 2024-04-22T08:12:52.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7fad3d3d0640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fad380a31d0 0x7fad38131310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7fad3d3d0640 10 monclient(hunting): picked mon.noname-c con 0x7fad380a31d0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:52.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7fad3d3d0640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad3813b410 0x7fad381ca680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7fad3d3d0640 10 monclient(hunting): picked mon.noname-a con 0x7fad3813b410 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:52.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7fad3d3d0640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7fad3d3d0640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad380a4790 con 0x7fad3813b410 2024-04-22T08:12:52.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7fad3d3d0640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7fad3d3d0640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad3813add0 con 0x7fad380a1c60 2024-04-22T08:12:52.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7fad3d3d0640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7fad3d3d0640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad380702b0 con 0x7fad380a31d0 2024-04-22T08:12:52.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7fad3d3d0640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:52.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7fad3d3d0640 10 monclient(hunting): authenticate will time out at 845.980300s 2024-04-22T08:12:52.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7fad367fc640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fad380a31d0 0x7fad38131310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7fad36ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad380a1c60 0x7fad38130dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7fad367fc640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fad380a31d0 0x7fad38131310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49646/0 (socket says 172.21.15.40:49646) 2024-04-22T08:12:52.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7fad367fc640 1 -- 172.21.15.40:0/1972830962 learned_addr learned my addr 172.21.15.40:0/1972830962 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:52.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7fad367fc640 10 monclient(hunting): get_auth_request con 0x7fad380a31d0 auth_method 0 2024-04-22T08:12:52.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7f4311141640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f430c0a1ba0 0x7f430c1cac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7f4311141640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f430c0a1ba0 0x7f430c1cac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.40:42644/0 (socket says 172.21.15.40:42644) 2024-04-22T08:12:52.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7fad367fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:52.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7f4311141640 1 -- 172.21.15.40:0/182044375 learned_addr learned my addr 172.21.15.40:0/182044375 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:52.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.987+0000 7f4311141640 10 monclient(hunting): get_auth_request con 0x7f430c0a1ba0 auth_method 0 2024-04-22T08:12:52.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.987+0000 7f4311141640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:52.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.987+0000 7f4311141640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:52.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.987+0000 7f4311141640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:52.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.987+0000 7f4310940640 1 --2- 172.21.15.40:0/182044375 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f430c136230 0x7f430c1cb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.987+0000 7f4311141640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:52.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.987+0000 7f4311141640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:52.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.987+0000 7f4311141640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:52.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.987+0000 7f4311942640 1 --2- 172.21.15.40:0/182044375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f430c136670 0x7f430c122050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7fad367fc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:52.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.987+0000 7f4311942640 10 monclient(hunting): get_auth_request con 0x7f430c136670 auth_method 0 2024-04-22T08:12:52.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.987+0000 7f4311942640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:52.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.987+0000 7f4311942640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:52.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.987+0000 7f4311942640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:52.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.987+0000 7f4310940640 10 monclient(hunting): get_auth_request con 0x7f430c136230 auth_method 0 2024-04-22T08:12:52.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.987+0000 7f4310940640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:52.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.987+0000 7f4310940640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:52.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.987+0000 7f4310940640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:52.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.987+0000 7f4311942640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:52.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.987+0000 7f4311942640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:52.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.987+0000 7f4311942640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:52.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7fad367fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:52.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7fad36ffd640 10 monclient(hunting): get_auth_request con 0x7fad380a1c60 auth_method 0 2024-04-22T08:12:52.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.987+0000 7f4310940640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:52.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.987+0000 7f4310940640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:52.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.987+0000 7f4310940640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:52.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7fad36ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:52.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7fad36ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:52.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7fad36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:52.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7fad377fe640 1 --2- 172.21.15.40:0/1972830962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad3813b410 0x7fad381ca680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7fad377fe640 10 monclient(hunting): get_auth_request con 0x7fad3813b410 auth_method 0 2024-04-22T08:12:52.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7fad377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:52.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7fad377fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:52.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7fad377fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:52.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7fad367fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:52.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.986+0000 7fad367fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:52.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.987+0000 7fad367fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:52.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.987+0000 7fad377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:52.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.987+0000 7fad377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:52.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.987+0000 7fad377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:52.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.987+0000 7f4310940640 10 monclient(hunting): handle_auth_done global_id 4226 payload 824 2024-04-22T08:12:52.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.987+0000 7f4310940640 1 -- 172.21.15.40:0/182044375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f430c0a1ba0 msgr2=0x7f430c1cac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.987+0000 7f4310940640 1 --2- 172.21.15.40:0/182044375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f430c0a1ba0 0x7f430c1cac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.987+0000 7f4310940640 1 -- 172.21.15.40:0/182044375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f430c136670 msgr2=0x7f430c122050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.988+0000 7f4310940640 1 --2- 172.21.15.40:0/182044375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f430c136670 0x7f430c122050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.988+0000 7f4310940640 10 monclient: _finish_hunting 0 2024-04-22T08:12:52.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.988+0000 7f4310940640 1 monclient: found mon.noname-c 2024-04-22T08:12:52.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.988+0000 7f4310940640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:52.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.988+0000 7f4310940640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:12:52.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.988+0000 7f4310940640 1 -- 172.21.15.40:0/182044375 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43080094d0 con 0x7f430c136230 2024-04-22T08:12:52.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.988+0000 7f4311942640 1 --2- 172.21.15.40:0/182044375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f430c136670 0x7f430c122050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:12:52.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.988+0000 7fad377fe640 10 monclient(hunting): handle_auth_done global_id 4230 payload 824 2024-04-22T08:12:52.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.988+0000 7fad377fe640 1 -- 172.21.15.40:0/1972830962 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fad380a31d0 msgr2=0x7fad38131310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.988+0000 7fad377fe640 1 --2- 172.21.15.40:0/1972830962 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fad380a31d0 0x7fad38131310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.988+0000 7fad377fe640 1 -- 172.21.15.40:0/1972830962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad380a1c60 msgr2=0x7fad38130dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.988+0000 7fad377fe640 1 --2- 172.21.15.40:0/1972830962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad380a1c60 0x7fad38130dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.988+0000 7fad377fe640 10 monclient: _finish_hunting 0 2024-04-22T08:12:52.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.988+0000 7fad377fe640 1 monclient: found mon.noname-a 2024-04-22T08:12:52.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.988+0000 7fad377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:52.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.988+0000 7fad377fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:12:52.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.988+0000 7fad377fe640 1 -- 172.21.15.40:0/1972830962 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad2c009400 con 0x7fad3813b410 2024-04-22T08:12:52.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.988+0000 7fad367fc640 1 --2- 172.21.15.40:0/1972830962 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fad380a31d0 0x7fad38131310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:12:52.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.988+0000 7fad36ffd640 1 --2- 172.21.15.40:0/1972830962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad380a1c60 0x7fad38130dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T08:12:52.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.990+0000 7f4310940640 1 -- 172.21.15.40:0/182044375 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f430c136230 msgr2=0x7f430c1cb1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:52.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.990+0000 7f4310940640 1 --2- 172.21.15.40:0/182044375 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f430c136230 0x7f430c1cb1a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f42fc00e810 tx=0x7f42fc00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:52.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.990+0000 7fad377fe640 1 -- 172.21.15.40:0/1972830962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad3813b410 msgr2=0x7fad381ca680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:52.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.990+0000 7fad377fe640 1 --2- 172.21.15.40:0/1972830962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad3813b410 0x7fad381ca680 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fad3000c750 tx=0x7fad3000cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:52.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.991+0000 7f4301ffb640 1 -- 172.21.15.40:0/182044375 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42fc00cae0 con 0x7f430c136230 2024-04-22T08:12:52.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.991+0000 7f4301ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:52.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.991+0000 7f4301ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:12:52.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.991+0000 7f4301ffb640 10 monclient: dump: 2024-04-22T08:12:52.023 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:52.023 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.023 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.023 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.023 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:52.023 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.023 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:52.024 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:52.024 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:52.024 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.991+0000 7f4301ffb640 10 monclient: _finish_auth 0 2024-04-22T08:12:52.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.991+0000 7f4301ffb640 10 monclient: _check_auth_tickets 2024-04-22T08:12:52.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.991+0000 7f4301ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:52.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.991+0000 7f4301ffb640 1 -- 172.21.15.40:0/182044375 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42fc00cc40 con 0x7f430c136230 2024-04-22T08:12:52.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.991+0000 7f4301ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:52.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.991+0000 7f4301ffb640 1 -- 172.21.15.40:0/182044375 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42fc013410 con 0x7f430c136230 2024-04-22T08:12:52.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.991+0000 7f4301ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:52.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.991+0000 7f4301ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:12:52.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.991+0000 7f4301ffb640 10 monclient: dump: 2024-04-22T08:12:52.024 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:52.024 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.024 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.025 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.025 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:52.025 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.025 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:52.025 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:52.025 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:52.025 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.991+0000 7f43037fe640 4 set_mon_vals no callback set 2024-04-22T08:12:52.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.991+0000 7fad1f7fe640 1 -- 172.21.15.40:0/1972830962 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3000aa60 con 0x7fad3813b410 2024-04-22T08:12:52.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.991+0000 7fad1f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:52.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.991+0000 7fad1f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:12:52.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.991+0000 7fad1f7fe640 10 monclient: dump: 2024-04-22T08:12:52.027 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:52.027 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.027 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.027 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.027 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:52.028 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.028 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:52.028 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:52.028 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:52.028 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.991+0000 7fad1f7fe640 10 monclient: _finish_auth 0 2024-04-22T08:12:52.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.991+0000 7fad1f7fe640 10 monclient: _check_auth_tickets 2024-04-22T08:12:52.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.991+0000 7fad1f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:52.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.991+0000 7fad1f7fe640 1 -- 172.21.15.40:0/1972830962 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad3000abc0 con 0x7fad3813b410 2024-04-22T08:12:52.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.991+0000 7fad1f7fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:52.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.991+0000 7fad1f7fe640 1 -- 172.21.15.40:0/1972830962 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad30012420 con 0x7fad3813b410 2024-04-22T08:12:52.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.991+0000 7fad1f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:52.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.991+0000 7fad1f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:12:52.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.991+0000 7fad1f7fe640 10 monclient: dump: 2024-04-22T08:12:52.028 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:52.029 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.029 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.029 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.029 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:52.029 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.029 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:52.029 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:52.029 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:52.029 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.991+0000 7fad357fa640 4 set_mon_vals no callback set 2024-04-22T08:12:52.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.991+0000 7fad3d3d0640 5 monclient: authenticate success, global_id 4230 2024-04-22T08:12:52.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.991+0000 7fad3d3d0640 10 monclient: _renew_subs 2024-04-22T08:12:52.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.991+0000 7fad3d3d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:52.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.992+0000 7fad3d3d0640 1 -- 172.21.15.40:0/1972830962 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad38071230 con 0x7fad3813b410 2024-04-22T08:12:52.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.992+0000 7f43133a5640 5 monclient: authenticate success, global_id 4226 2024-04-22T08:12:52.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.992+0000 7f43133a5640 10 monclient: _renew_subs 2024-04-22T08:12:52.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.992+0000 7f43133a5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:52.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.992+0000 7f43133a5640 1 -- 172.21.15.40:0/182044375 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f430c0716c0 con 0x7f430c136230 2024-04-22T08:12:52.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.992+0000 7fad3d3d0640 10 monclient: _renew_subs 2024-04-22T08:12:52.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.992+0000 7fad3d3d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:52.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.992+0000 7fad3d3d0640 1 -- 172.21.15.40:0/1972830962 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad3806fa10 con 0x7fad3813b410 2024-04-22T08:12:52.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.992+0000 7f43133a5640 10 monclient: _renew_subs 2024-04-22T08:12:52.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.992+0000 7f43133a5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:52.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.992+0000 7f43133a5640 1 -- 172.21.15.40:0/182044375 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f430c123ca0 con 0x7f430c136230 2024-04-22T08:12:52.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.993+0000 7f43133a5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:52.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.993+0000 7f43133a5640 1 -- 172.21.15.40:0/182044375 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f430c0a24c0 con 0x7f430c136230 2024-04-22T08:12:52.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.994+0000 7fad3d3d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:52.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.994+0000 7fad3d3d0640 1 -- 172.21.15.40:0/1972830962 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fad380a0230 con 0x7fad3813b410 2024-04-22T08:12:52.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.993+0000 7f4301ffb640 1 -- 172.21.15.40:0/182044375 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 4) v1 ==== 44726+0+0 (secure 0 0 0) 0x7f42fc013570 con 0x7f430c136230 2024-04-22T08:12:52.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.994+0000 7f4301ffb640 1 --2- 172.21.15.40:0/182044375 >> v2:172.21.15.189:6800/30168 conn(0x7f42f0037790 0x7f42f0039c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.994+0000 7f4301ffb640 1 -- 172.21.15.40:0/182044375 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f42fc04a030 con 0x7f430c136230 2024-04-22T08:12:52.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.994+0000 7f4301ffb640 1 --2- 172.21.15.40:0/182044375 >> v2:172.21.15.40:6804/30669 conn(0x7f42f003d2c0 0x7f42f003f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.994+0000 7f4301ffb640 1 -- 172.21.15.40:0/182044375 --> v2:172.21.15.40:6804/30669 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f42f003fd50 con 0x7f42f003d2c0 2024-04-22T08:12:52.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.994+0000 7f4301ffb640 1 -- 172.21.15.40:0/182044375 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_get_version_reply(handle=1 version=14) v2 ==== 24+0+0 (secure 0 0 0) 0x7f42fc04c570 con 0x7f430c136230 2024-04-22T08:12:52.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.994+0000 7f4301ffb640 10 monclient: handle_get_version_reply finishing 1 version 14 2024-04-22T08:12:52.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.994+0000 7f4311141640 1 --2- 172.21.15.40:0/182044375 >> v2:172.21.15.189:6800/30168 conn(0x7f42f0037790 0x7f42f0039c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.994+0000 7f4311141640 10 monclient: get_auth_request con 0x7f42f0037790 auth_method 0 2024-04-22T08:12:52.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.994+0000 7f4311942640 1 --2- 172.21.15.40:0/182044375 >> v2:172.21.15.40:6804/30669 conn(0x7f42f003d2c0 0x7f42f003f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.995+0000 7f4311942640 10 monclient: get_auth_request con 0x7f42f003d2c0 auth_method 0 2024-04-22T08:12:52.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.994+0000 7fad1f7fe640 1 -- 172.21.15.40:0/1972830962 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 4) v1 ==== 44726+0+0 (secure 0 0 0) 0x7fad30012640 con 0x7fad3813b410 2024-04-22T08:12:52.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.995+0000 7fad1f7fe640 1 --2- 172.21.15.40:0/1972830962 >> v2:172.21.15.189:6800/30168 conn(0x7fad24037790 0x7fad24039c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.995+0000 7fad1f7fe640 1 -- 172.21.15.40:0/1972830962 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 3506+0+0 (secure 0 0 0) 0x7fad3004a570 con 0x7fad3813b410 2024-04-22T08:12:52.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.995+0000 7fad1f7fe640 1 --2- 172.21.15.40:0/1972830962 >> v2:172.21.15.189:6805/30673 conn(0x7fad2403d2c0 0x7fad2403f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.995+0000 7fad1f7fe640 1 -- 172.21.15.40:0/1972830962 --> v2:172.21.15.189:6805/30673 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fad2403fd50 con 0x7fad2403d2c0 2024-04-22T08:12:52.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.995+0000 7fad1f7fe640 1 -- 172.21.15.40:0/1972830962 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=14) v2 ==== 24+0+0 (secure 0 0 0) 0x7fad3004cab0 con 0x7fad3813b410 2024-04-22T08:12:52.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.995+0000 7fad1f7fe640 10 monclient: handle_get_version_reply finishing 1 version 14 2024-04-22T08:12:52.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.995+0000 7fad36ffd640 1 --2- 172.21.15.40:0/1972830962 >> v2:172.21.15.189:6800/30168 conn(0x7fad24037790 0x7fad24039c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.995+0000 7fad36ffd640 10 monclient: get_auth_request con 0x7fad24037790 auth_method 0 2024-04-22T08:12:52.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.995+0000 7fad367fc640 1 --2- 172.21.15.40:0/1972830962 >> v2:172.21.15.189:6805/30673 conn(0x7fad2403d2c0 0x7fad2403f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.995+0000 7fad367fc640 10 monclient: get_auth_request con 0x7fad2403d2c0 auth_method 0 2024-04-22T08:12:52.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.997+0000 7f4311141640 1 --2- 172.21.15.40:0/182044375 >> v2:172.21.15.189:6800/30168 conn(0x7f42f0037790 0x7f42f0039c50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4308009870 tx=0x7f430803d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:52.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.998+0000 7fad36ffd640 1 --2- 172.21.15.40:0/1972830962 >> v2:172.21.15.189:6800/30168 conn(0x7fad24037790 0x7fad24039c50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fad28005160 tx=0x7fad28009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:52.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:51.999+0000 7f4311942640 1 --2- 172.21.15.40:0/182044375 >> v2:172.21.15.40:6804/30669 conn(0x7f42f003d2c0 0x7f42f003f6e0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:52.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.000+0000 7fad367fc640 1 --2- 172.21.15.40:0/1972830962 >> v2:172.21.15.189:6805/30673 conn(0x7fad2403d2c0 0x7fad2403f6e0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:52.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.000+0000 7f4301ffb640 1 -- 172.21.15.40:0/182044375 <== osd.0 v2:172.21.15.40:6804/30669 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f42f003fd50 con 0x7f42f003d2c0 2024-04-22T08:12:52.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.001+0000 7fad1f7fe640 1 -- 172.21.15.40:0/1972830962 <== osd.3 v2:172.21.15.189:6805/30673 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fad2403fd50 con 0x7fad2403d2c0 2024-04-22T08:12:52.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.018+0000 7f3a5ee99640 10 monclient: get_monmap_and_config 2024-04-22T08:12:52.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.018+0000 7f3a5ee99640 10 monclient: build_initial_monmap 2024-04-22T08:12:52.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.018+0000 7f3a5ee99640 10 monclient: monmap: 2024-04-22T08:12:52.040 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:52.040 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.040 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:52.019153+0000 2024-04-22T08:12:52.040 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:52.019153+0000 2024-04-22T08:12:52.040 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:52.040 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.040 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:12:52.040 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:12:52.040 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:12:52.040 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:12:52.040 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:12:52.040 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.019+0000 7f3a5ee99640 1 Processor -- start 2024-04-22T08:12:52.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.019+0000 7f3a5ee99640 1 -- start start 2024-04-22T08:12:52.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.019+0000 7f3a5ee99640 10 monclient: init 2024-04-22T08:12:52.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.019+0000 7f3a5ee99640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:52.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.019+0000 7f3a5ee99640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-04-22T08:12:52.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.019+0000 7f3a5ee99640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3a580a1ba0 0x7f3a580a1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.019+0000 7f3a5ee99640 10 monclient(hunting): picked mon.noname-c con 0x7f3a580a1ba0 addr v2:172.21.15.189:3300/0 2024-04-22T08:12:52.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.019+0000 7f3a5ee99640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3a58136230 addr v1:172.21.15.40:6789/0 2024-04-22T08:12:52.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.020+0000 7f3a5ee99640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3a58136670 0x7f3a580a3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.020+0000 7f3a5ee99640 10 monclient(hunting): picked mon.noname-a con 0x7f3a58136670 addr v2:172.21.15.40:3300/0 2024-04-22T08:12:52.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.020+0000 7f3a5ee99640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a5809fe80 con 0x7f3a58136230 2024-04-22T08:12:52.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.020+0000 7f3a5ee99640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.020+0000 7f3a5ee99640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a580722a0 con 0x7f3a58136670 2024-04-22T08:12:52.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.020+0000 7f3a5ee99640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.020+0000 7f3a5ee99640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a580a4290 con 0x7f3a580a1ba0 2024-04-22T08:12:52.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.020+0000 7f3a5ee99640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:52.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.020+0000 7f3a5ee99640 10 monclient(hunting): authenticate will time out at 846.013961s 2024-04-22T08:12:52.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.020+0000 7f3a57fff640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f3a58136230 0x7f3a580a2d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:38878/0 (socket says 172.21.15.40:38878) 2024-04-22T08:12:52.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.020+0000 7f3a5d436640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3a58136670 0x7f3a580a3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.020+0000 7f3a57fff640 1 -- 172.21.15.40:0/2177688790 learned_addr learned my addr 172.21.15.40:0/2177688790 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:12:52.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.020+0000 7f3a5d436640 10 monclient(hunting): get_auth_request con 0x7f3a58136670 auth_method 0 2024-04-22T08:12:52.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.020+0000 7f3a5d436640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:52.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.020+0000 7f3a5d436640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:52.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.020+0000 7f3a5d436640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:52.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.020+0000 7f3a5cc35640 1 --2- 172.21.15.40:0/2177688790 >> v2:172.21.15.189:3300/0 conn(0x7f3a580a1ba0 0x7f3a580a1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.020+0000 7f3a5cc35640 10 monclient(hunting): get_auth_request con 0x7f3a580a1ba0 auth_method 0 2024-04-22T08:12:52.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.020+0000 7f3a5cc35640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:52.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.020+0000 7f3a5cc35640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:52.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.020+0000 7f3a5cc35640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:52.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.020+0000 7f3a5cc35640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:52.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.020+0000 7f3a5cc35640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:52.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.020+0000 7f3a5cc35640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:52.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.020+0000 7f3a5cc35640 10 monclient(hunting): handle_auth_done global_id 4222 payload 274 2024-04-22T08:12:52.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.020+0000 7f3a5cc35640 1 -- 172.21.15.40:0/2177688790 >> v2:172.21.15.40:3300/0 conn(0x7f3a58136670 msgr2=0x7f3a580a3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:52.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.020+0000 7f3a5cc35640 1 --2- 172.21.15.40:0/2177688790 >> v2:172.21.15.40:3300/0 conn(0x7f3a58136670 0x7f3a580a3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.020+0000 7f3a5cc35640 1 -- 172.21.15.40:0/2177688790 >> v1:172.21.15.40:6789/0 conn(0x7f3a58136230 legacy=0x7f3a580a2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:52.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.020+0000 7f3a5cc35640 10 monclient: _finish_hunting 0 2024-04-22T08:12:52.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.020+0000 7f3a5cc35640 1 monclient: found mon.noname-c 2024-04-22T08:12:52.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.020+0000 7f3a5cc35640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:52.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.020+0000 7f3a5cc35640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.189:3300/0 2024-04-22T08:12:52.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.020+0000 7f3a5cc35640 1 -- 172.21.15.40:0/2177688790 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a580a0160 con 0x7f3a580a1ba0 2024-04-22T08:12:52.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.022+0000 7fd9d3210640 10 monclient: get_monmap_and_config 2024-04-22T08:12:52.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.022+0000 7fd9d3210640 10 monclient: build_initial_monmap 2024-04-22T08:12:52.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.022+0000 7fd9d3210640 10 monclient: monmap: 2024-04-22T08:12:52.047 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:52.047 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.047 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:52.023241+0000 2024-04-22T08:12:52.047 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:52.023241+0000 2024-04-22T08:12:52.047 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:52.047 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.047 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:12:52.047 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:12:52.047 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:12:52.047 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:12:52.048 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:12:52.048 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.023+0000 7fd9d3210640 1 Processor -- start 2024-04-22T08:12:52.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.023+0000 7fd9d3210640 1 -- start start 2024-04-22T08:12:52.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.023+0000 7fd9d3210640 10 monclient: init 2024-04-22T08:12:52.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.023+0000 7fd9d3210640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:52.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.023+0000 7fd9d3210640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-04-22T08:12:52.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.023+0000 7fd9d3210640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd9cc0a1ba0 0x7fd9cc0a1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.023+0000 7fd9d3210640 10 monclient(hunting): picked mon.noname-a con 0x7fd9cc0a1ba0 addr v2:172.21.15.40:3300/0 2024-04-22T08:12:52.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.023+0000 7fd9d3210640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd9cc136230 addr v1:172.21.15.189:6789/0 2024-04-22T08:12:52.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.023+0000 7fd9d3210640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd9cc136670 0x7fd9cc0a3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.023+0000 7fd9d3210640 10 monclient(hunting): picked mon.noname-c con 0x7fd9cc136670 addr v2:172.21.15.189:3300/0 2024-04-22T08:12:52.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.023+0000 7fd9d3210640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9cc09fe80 con 0x7fd9cc136230 2024-04-22T08:12:52.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.023+0000 7fd9d3210640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.023+0000 7fd9d3210640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9cc0722a0 con 0x7fd9cc0a1ba0 2024-04-22T08:12:52.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.023+0000 7fd9d3210640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.023+0000 7fd9d3210640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9cc0a4290 con 0x7fd9cc136670 2024-04-22T08:12:52.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.023+0000 7fd9d3210640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:52.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.023+0000 7fd9d3210640 10 monclient(hunting): authenticate will time out at 846.017640s 2024-04-22T08:12:52.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.023+0000 7fd9d0fac640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd9cc0a1ba0 0x7fd9cc0a1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.024+0000 7fd9d0fac640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd9cc0a1ba0 0x7fd9cc0a1f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57838/0 (socket says 172.21.15.40:57838) 2024-04-22T08:12:52.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.024+0000 7fd9d0fac640 1 -- 172.21.15.40:0/857299001 learned_addr learned my addr 172.21.15.40:0/857299001 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:52.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.024+0000 7fd9d0fac640 10 monclient(hunting): get_auth_request con 0x7fd9cc0a1ba0 auth_method 0 2024-04-22T08:12:52.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.024+0000 7fd9d0fac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:52.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.024+0000 7fd9d0fac640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:52.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.024+0000 7fd9d0fac640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:52.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.024+0000 7fd9d0fac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:52.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.024+0000 7fd9d0fac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:52.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.024+0000 7fd9d0fac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:52.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.024+0000 7fd9d0fac640 10 monclient(hunting): handle_auth_done global_id 4239 payload 274 2024-04-22T08:12:52.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.024+0000 7f3a5cc35640 1 -- 172.21.15.40:0/2177688790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a580a1ba0 msgr2=0x7f3a580a1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:52.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.024+0000 7f3a5cc35640 1 --2- 172.21.15.40:0/2177688790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a580a1ba0 0x7f3a580a1f80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3a580a5c50 tx=0x7f3a4c02b7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42ac6b1c00a79515 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:52.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.024+0000 7fd9d0fac640 1 -- 172.21.15.40:0/857299001 >> v2:172.21.15.189:3300/0 conn(0x7fd9cc136670 msgr2=0x7fd9cc0a3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:52.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.024+0000 7fd9d0fac640 1 --2- 172.21.15.40:0/857299001 >> v2:172.21.15.189:3300/0 conn(0x7fd9cc136670 0x7fd9cc0a3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.024+0000 7fd9d0fac640 1 -- 172.21.15.40:0/857299001 >> v1:172.21.15.189:6789/0 conn(0x7fd9cc136230 legacy=0x7fd9cc0a2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:52.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.024+0000 7fd9d0fac640 10 monclient: _finish_hunting 0 2024-04-22T08:12:52.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.024+0000 7fd9d0fac640 1 monclient: found mon.noname-a 2024-04-22T08:12:52.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.024+0000 7fd9d0fac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:52.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.024+0000 7fd9d0fac640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:12:52.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.024+0000 7fd9d0fac640 1 -- 172.21.15.40:0/857299001 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9cc0a0160 con 0x7fd9cc0a1ba0 2024-04-22T08:12:52.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.024+0000 7f3a577fe640 1 -- 172.21.15.40:0/2177688790 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a4c031920 con 0x7f3a580a1ba0 2024-04-22T08:12:52.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.024+0000 7f3a577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:52.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.024+0000 7f3a577fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:12:52.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.024+0000 7f3a577fe640 10 monclient: dump: 2024-04-22T08:12:52.055 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:52.055 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.055 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.055 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.055 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:52.055 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.055 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:52.055 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:52.055 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:52.055 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.024+0000 7f3a577fe640 10 monclient: _finish_auth 0 2024-04-22T08:12:52.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.024+0000 7f3a577fe640 10 monclient: _check_auth_tickets 2024-04-22T08:12:52.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.024+0000 7f3a577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:52.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.024+0000 7f3a577fe640 1 -- 172.21.15.40:0/2177688790 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a4c02f050 con 0x7f3a580a1ba0 2024-04-22T08:12:52.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.024+0000 7f3a5ee99640 5 monclient: authenticate success, global_id 4222 2024-04-22T08:12:52.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.024+0000 7f3a5ee99640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:12:52.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.024+0000 7f3a577fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:52.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.024+0000 7f3a577fe640 1 -- 172.21.15.40:0/2177688790 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a4c031ca0 con 0x7f3a580a1ba0 2024-04-22T08:12:52.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.024+0000 7f3a5ee99640 10 monclient: get_monmap_and_config success 2024-04-22T08:12:52.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.024+0000 7f3a5ee99640 4 set_mon_vals no callback set 2024-04-22T08:12:52.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.025+0000 7f3a577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:52.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.025+0000 7f3a577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:12:52.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.025+0000 7f3a577fe640 10 monclient: dump: 2024-04-22T08:12:52.056 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:52.056 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.057 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.057 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.057 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:52.057 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.057 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:52.057 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:52.057 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:52.057 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.025+0000 7f3a5ee99640 10 monclient: shutdown 2024-04-22T08:12:52.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.025+0000 7f3a5ee99640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:52.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.025+0000 7f3a5ee99640 1 -- 172.21.15.40:0/2177688790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a580a1ba0 msgr2=0x7f3a580a1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.025+0000 7f3a5ee99640 1 --2- 172.21.15.40:0/2177688790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a580a1ba0 0x7f3a580a1f80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3a580a5c50 tx=0x7f3a4c02b7b0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.025+0000 7f3a5ee99640 1 -- 172.21.15.40:0/2177688790 shutdown_connections 2024-04-22T08:12:52.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.025+0000 7f3a5ee99640 1 --2- 172.21.15.40:0/2177688790 >> v2:172.21.15.40:3300/0 conn(0x7f3a58136670 0x7f3a580a3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.025+0000 7f3a5ee99640 1 --2- 172.21.15.40:0/2177688790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a580a1ba0 0x7f3a580a1f80 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.025+0000 7f3a5ee99640 1 -- 172.21.15.40:0/2177688790 >> 172.21.15.40:0/2177688790 conn(0x7f3a580a4e80 msgr2=0x7f3a580a5290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:52.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.025+0000 7f3a5ee99640 1 -- 172.21.15.40:0/2177688790 shutdown_connections 2024-04-22T08:12:52.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.025+0000 7f3a5ee99640 1 -- 172.21.15.40:0/2177688790 wait complete. 2024-04-22T08:12:52.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.026+0000 7f3a5ee99640 10 monclient: build_initial_monmap 2024-04-22T08:12:52.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.026+0000 7f3a5ee99640 10 monclient: monmap: 2024-04-22T08:12:52.058 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:52.058 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:12:52.058 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:12:52.058 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:12:52.058 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:52.058 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.058 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:12:52.058 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:12:52.059 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:12:52.059 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.026+0000 7f3a5ee99640 1 Processor -- start 2024-04-22T08:12:52.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.026+0000 7f3a5ee99640 1 -- start start 2024-04-22T08:12:52.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.026+0000 7f3a5ee99640 10 monclient: init 2024-04-22T08:12:52.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.027+0000 7fd9d0fac640 1 -- 172.21.15.40:0/857299001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9cc0a1ba0 msgr2=0x7fd9cc0a1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:52.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.027+0000 7fd9d0fac640 1 --2- 172.21.15.40:0/857299001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9cc0a1ba0 0x7fd9cc0a1f80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc06c970 tx=0x7fd9c002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a31cf83c79696b7e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:52.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.027+0000 7f6347dd7640 10 monclient: get_monmap_and_config 2024-04-22T08:12:52.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.027+0000 7f6347dd7640 10 monclient: build_initial_monmap 2024-04-22T08:12:52.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.027+0000 7f6347dd7640 10 monclient: monmap: 2024-04-22T08:12:52.060 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:52.060 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.060 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:52.028356+0000 2024-04-22T08:12:52.060 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:52.028356+0000 2024-04-22T08:12:52.060 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:52.060 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.060 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:12:52.060 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:12:52.060 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:12:52.061 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:12:52.061 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:12:52.061 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.027+0000 7f3a5ee99640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:52.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.027+0000 7f3a5ee99640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:12:52.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.027+0000 7f3a5ee99640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a580a1ba0 0x7f3a581203c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.027+0000 7fb962066640 10 monclient: get_monmap_and_config 2024-04-22T08:12:52.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.027+0000 7fb962066640 10 monclient: build_initial_monmap 2024-04-22T08:12:52.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.027+0000 7fb962066640 10 monclient: monmap: 2024-04-22T08:12:52.062 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:52.062 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.062 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:52.028466+0000 2024-04-22T08:12:52.062 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:52.028466+0000 2024-04-22T08:12:52.062 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:52.062 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.062 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:12:52.062 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:12:52.062 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:12:52.063 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:12:52.063 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:12:52.063 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.027+0000 7fd9cb7fe640 1 -- 172.21.15.40:0/857299001 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c002cc00 con 0x7fd9cc0a1ba0 2024-04-22T08:12:52.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.027+0000 7fd9cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:52.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.027+0000 7fd9cb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:12:52.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.027+0000 7fd9cb7fe640 10 monclient: dump: 2024-04-22T08:12:52.064 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:52.064 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.064 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.064 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.064 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:52.064 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.064 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:52.064 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:52.064 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:52.065 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.027+0000 7fd9cb7fe640 10 monclient: _finish_auth 0 2024-04-22T08:12:52.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.027+0000 7fd9cb7fe640 10 monclient: _check_auth_tickets 2024-04-22T08:12:52.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.027+0000 7fd9cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:52.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.027+0000 7fd9d3210640 5 monclient: authenticate success, global_id 4239 2024-04-22T08:12:52.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.027+0000 7fd9d3210640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:12:52.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.027+0000 7fd9cb7fe640 1 -- 172.21.15.40:0/857299001 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9c002cd60 con 0x7fd9cc0a1ba0 2024-04-22T08:12:52.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.027+0000 7fd9cb7fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:52.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.027+0000 7fd9d3210640 10 monclient: get_monmap_and_config success 2024-04-22T08:12:52.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.027+0000 7fd9d3210640 4 set_mon_vals no callback set 2024-04-22T08:12:52.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.027+0000 7fd9cb7fe640 1 -- 172.21.15.40:0/857299001 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c0034610 con 0x7fd9cc0a1ba0 2024-04-22T08:12:52.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.027+0000 7fd9cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:52.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.027+0000 7fd9cb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:12:52.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.027+0000 7fd9cb7fe640 10 monclient: dump: 2024-04-22T08:12:52.066 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:52.066 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.066 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.066 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.066 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:52.066 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.066 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:52.066 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:52.066 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:52.066 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.027+0000 7f3a5cc35640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a580a1ba0 0x7f3a581203c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.027+0000 7f3a5ee99640 10 monclient(hunting): picked mon.noname-a con 0x7f3a580a1ba0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:52.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.027+0000 7f3a5ee99640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a58136230 0x7f3a58120900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.028+0000 7f3a5cc35640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a580a1ba0 0x7f3a581203c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57844/0 (socket says 172.21.15.40:57844) 2024-04-22T08:12:52.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.028+0000 7f3a5ee99640 10 monclient(hunting): picked mon.noname-b con 0x7f3a58136230 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:52.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.028+0000 7f3a5cc35640 1 -- 172.21.15.40:0/3999428160 learned_addr learned my addr 172.21.15.40:0/3999428160 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:52.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.028+0000 7f3a5ee99640 1 --2- 172.21.15.40:0/3999428160 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3a58136670 0x7f3a58120e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.028+0000 7f3a5ee99640 10 monclient(hunting): picked mon.noname-c con 0x7f3a58136670 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:52.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.028+0000 7f3a5ee99640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.028+0000 7f3a5ee99640 1 -- 172.21.15.40:0/3999428160 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a580a2780 con 0x7f3a580a1ba0 2024-04-22T08:12:52.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.028+0000 7f3a57fff640 1 --2- 172.21.15.40:0/3999428160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a58136230 0x7f3a58120900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.028+0000 7f3a5d436640 1 --2- 172.21.15.40:0/3999428160 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3a58136670 0x7f3a58120e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.028+0000 7f3a5ee99640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.028+0000 7f3a5ee99640 1 -- 172.21.15.40:0/3999428160 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a58124360 con 0x7f3a58136230 2024-04-22T08:12:52.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.028+0000 7f3a5ee99640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.028+0000 7f3a5ee99640 1 -- 172.21.15.40:0/3999428160 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a58135110 con 0x7f3a58136670 2024-04-22T08:12:52.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.028+0000 7f6347dd7640 1 Processor -- start 2024-04-22T08:12:52.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.028+0000 7f3a5ee99640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:52.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.028+0000 7f3a5ee99640 10 monclient(hunting): authenticate will time out at 846.022223s 2024-04-22T08:12:52.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.028+0000 7f3a5cc35640 10 monclient(hunting): get_auth_request con 0x7f3a580a1ba0 auth_method 0 2024-04-22T08:12:52.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.028+0000 7f3a5cc35640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:52.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.028+0000 7f3a5cc35640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:52.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.028+0000 7f3a5cc35640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:52.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.028+0000 7f3a57fff640 10 monclient(hunting): get_auth_request con 0x7f3a58136230 auth_method 0 2024-04-22T08:12:52.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.028+0000 7f3a57fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:52.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.028+0000 7f3a57fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:52.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.028+0000 7f3a57fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:52.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.028+0000 7f3a5d436640 10 monclient(hunting): get_auth_request con 0x7f3a58136670 auth_method 0 2024-04-22T08:12:52.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.028+0000 7fd9d3210640 10 monclient: shutdown 2024-04-22T08:12:52.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.028+0000 7fd9d3210640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:52.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.028+0000 7f3a5d436640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:52.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.028+0000 7f3a5d436640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:52.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.028+0000 7f3a5d436640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:52.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.028+0000 7f3a57fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:52.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.028+0000 7f3a57fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:52.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.028+0000 7f3a57fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:52.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.028+0000 7f3a5cc35640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:52.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.028+0000 7f3a5cc35640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:52.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.028+0000 7f3a5cc35640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:52.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.028+0000 7f3a5d436640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:52.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.028+0000 7f3a5d436640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:52.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.028+0000 7f3a5d436640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:52.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.028+0000 7fb962066640 1 Processor -- start 2024-04-22T08:12:52.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.028+0000 7fb962066640 1 -- start start 2024-04-22T08:12:52.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.028+0000 7fb962066640 10 monclient: init 2024-04-22T08:12:52.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.028+0000 7fd9d3210640 1 -- 172.21.15.40:0/857299001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9cc0a1ba0 msgr2=0x7fd9cc0a1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.028+0000 7f43133a5640 1 -- 172.21.15.40:0/182044375 --> v2:172.21.15.40:6804/30669 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f430c04e9b0 con 0x7f42f003d2c0 2024-04-22T08:12:52.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.028+0000 7fd9d3210640 1 --2- 172.21.15.40:0/857299001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9cc0a1ba0 0x7fd9cc0a1f80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc06c970 tx=0x7fd9c002be50 comp rx=0 tx=0).stop 2024-04-22T08:12:52.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7fd9d3210640 1 -- 172.21.15.40:0/857299001 shutdown_connections 2024-04-22T08:12:52.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7fd9d3210640 1 --2- 172.21.15.40:0/857299001 >> v2:172.21.15.189:3300/0 conn(0x7fd9cc136670 0x7fd9cc0a3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7fd9d3210640 1 --2- 172.21.15.40:0/857299001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9cc0a1ba0 0x7fd9cc0a1f80 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7fd9d3210640 1 -- 172.21.15.40:0/857299001 >> 172.21.15.40:0/857299001 conn(0x7fd9cc0a4e80 msgr2=0x7fd9cc0a5290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:52.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7f3a57fff640 10 monclient(hunting): handle_auth_done global_id 4225 payload 824 2024-04-22T08:12:52.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7f3a57fff640 1 -- 172.21.15.40:0/3999428160 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3a58136670 msgr2=0x7f3a58120e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7f3a57fff640 1 --2- 172.21.15.40:0/3999428160 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3a58136670 0x7f3a58120e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7f3a57fff640 1 -- 172.21.15.40:0/3999428160 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a580a1ba0 msgr2=0x7f3a581203c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7f3a57fff640 1 --2- 172.21.15.40:0/3999428160 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a580a1ba0 0x7f3a581203c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7f3a57fff640 10 monclient: _finish_hunting 0 2024-04-22T08:12:52.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7f3a57fff640 1 monclient: found mon.noname-b 2024-04-22T08:12:52.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7f3a57fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:52.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7f3a57fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.189:3300/0 2024-04-22T08:12:52.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7f3a57fff640 1 -- 172.21.15.40:0/3999428160 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a4c009400 con 0x7f3a58136230 2024-04-22T08:12:52.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7fd9d3210640 1 -- 172.21.15.40:0/857299001 shutdown_connections 2024-04-22T08:12:52.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7f3a5cc35640 1 --2- 172.21.15.40:0/3999428160 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a580a1ba0 0x7f3a581203c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:12:52.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7f4301ffb640 1 -- 172.21.15.40:0/182044375 <== osd.0 v2:172.21.15.40:6804/30669 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f430c04e9b0 con 0x7f42f003d2c0 2024-04-22T08:12:52.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7fd9d3210640 1 -- 172.21.15.40:0/857299001 wait complete. 2024-04-22T08:12:52.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.028+0000 7f6347dd7640 1 -- start start 2024-04-22T08:12:52.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.028+0000 7f6347dd7640 10 monclient: init 2024-04-22T08:12:52.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7f6347dd7640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:52.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7f6347dd7640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-04-22T08:12:52.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7f6347dd7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f634009e960 addr v1:172.21.15.189:6789/0 2024-04-22T08:12:52.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7f6347dd7640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6340132d00 0x7f6340137b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7f6347dd7640 10 monclient(hunting): picked mon.noname-a con 0x7f6340132d00 addr v2:172.21.15.40:3300/0 2024-04-22T08:12:52.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7f6347dd7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6340138080 addr v1:172.21.15.40:6789/0 2024-04-22T08:12:52.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7f6347dd7640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f634009bb50 con 0x7f6340138080 2024-04-22T08:12:52.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7f6347dd7640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f634009aa50 con 0x7f634009e960 2024-04-22T08:12:52.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7f6347dd7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7f6347dd7640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f634009a5c0 con 0x7f6340132d00 2024-04-22T08:12:52.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7f6347dd7640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:52.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7f6347dd7640 10 monclient(hunting): authenticate will time out at 846.023324s 2024-04-22T08:12:52.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7fb962066640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:52.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7fb962066640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-04-22T08:12:52.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7fb962066640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb95c133d80 0x7fb95c134160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7fb962066640 10 monclient(hunting): picked mon.noname-a con 0x7fb95c133d80 addr v2:172.21.15.40:3300/0 2024-04-22T08:12:52.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7fb962066640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb95c09f9c0 0x7fb95c09fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7fb962066640 10 monclient(hunting): picked mon.noname-c con 0x7fb95c09f9c0 addr v2:172.21.15.189:3300/0 2024-04-22T08:12:52.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7fb962066640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb95c0a0b70 addr v1:172.21.15.189:6789/0 2024-04-22T08:12:52.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7fb962066640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb95c09ec70 con 0x7fb95c0a0b70 2024-04-22T08:12:52.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7fb962066640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7fb962066640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb95c0a2aa0 con 0x7fb95c133d80 2024-04-22T08:12:52.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7fb962066640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7fb962066640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb95c0a2800 con 0x7fb95c09f9c0 2024-04-22T08:12:52.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7fb962066640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:52.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7fb962066640 10 monclient(hunting): authenticate will time out at 846.023422s 2024-04-22T08:12:52.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7fb95b7fe640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb95c133d80 0x7fb95c134160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7fb95b7fe640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb95c133d80 0x7fb95c134160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57860/0 (socket says 172.21.15.40:57860) 2024-04-22T08:12:52.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7fb95b7fe640 1 -- 172.21.15.40:0/598358128 learned_addr learned my addr 172.21.15.40:0/598358128 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:52.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7fb95b7fe640 10 monclient(hunting): get_auth_request con 0x7fb95c133d80 auth_method 0 2024-04-22T08:12:52.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7fb95b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:52.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7fb95b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:52.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7fb95b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:52.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7f6346374640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f6340138080 0x7f634013a470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:38884/0 (socket says 172.21.15.40:38884) 2024-04-22T08:12:52.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7f6346374640 1 -- 172.21.15.40:0/3772186029 learned_addr learned my addr 172.21.15.40:0/3772186029 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:12:52.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7fb95b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:52.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7fb95b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:52.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.029+0000 7fb95b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:52.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.030+0000 7fb95b7fe640 10 monclient(hunting): handle_auth_done global_id 4245 payload 274 2024-04-22T08:12:52.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.030+0000 7fb95b7fe640 1 -- 172.21.15.40:0/598358128 >> v2:172.21.15.189:3300/0 conn(0x7fb95c09f9c0 msgr2=0x7fb95c09fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:52.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.030+0000 7fb95b7fe640 1 --2- 172.21.15.40:0/598358128 >> v2:172.21.15.189:3300/0 conn(0x7fb95c09f9c0 0x7fb95c09fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.030+0000 7fb95b7fe640 1 -- 172.21.15.40:0/598358128 >> v1:172.21.15.189:6789/0 conn(0x7fb95c0a0b70 legacy=0x7fb95c0a0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:52.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.030+0000 7fb95b7fe640 10 monclient: _finish_hunting 0 2024-04-22T08:12:52.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.030+0000 7fb95b7fe640 1 monclient: found mon.noname-a 2024-04-22T08:12:52.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.030+0000 7fb95b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:52.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.030+0000 7fb95b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:12:52.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.030+0000 7fb95b7fe640 1 -- 172.21.15.40:0/598358128 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb95c09e990 con 0x7fb95c133d80 2024-04-22T08:12:52.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.030+0000 7f6345372640 1 --2- 172.21.15.40:0/3772186029 >> v2:172.21.15.40:3300/0 conn(0x7f6340132d00 0x7f6340137b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.030+0000 7f42ef7fe640 1 -- 172.21.15.40:0/182044375 >> v2:172.21.15.40:6804/30669 conn(0x7f42f003d2c0 msgr2=0x7f42f003f6e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.030+0000 7f6345372640 10 monclient(hunting): get_auth_request con 0x7f6340132d00 auth_method 0 2024-04-22T08:12:52.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.030+0000 7f6345372640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:52.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.030+0000 7f42ef7fe640 1 --2- 172.21.15.40:0/182044375 >> v2:172.21.15.40:6804/30669 conn(0x7f42f003d2c0 0x7f42f003f6e0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.030+0000 7f6345372640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:52.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.030+0000 7f6345372640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:52.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.030+0000 7fd9d3210640 10 monclient: build_initial_monmap 2024-04-22T08:12:52.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.030+0000 7fd9d3210640 10 monclient: monmap: 2024-04-22T08:12:52.086 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:52.086 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:12:52.086 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:12:52.086 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:12:52.086 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:52.086 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.086 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:12:52.086 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:12:52.086 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:12:52.086 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.030+0000 7f6345372640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:52.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.030+0000 7f6345372640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:52.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.030+0000 7f6345372640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:52.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.030+0000 7f42ef7fe640 1 -- 172.21.15.40:0/182044375 >> v2:172.21.15.189:6800/30168 conn(0x7f42f0037790 msgr2=0x7f42f0039c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.030+0000 7f42ef7fe640 1 --2- 172.21.15.40:0/182044375 >> v2:172.21.15.189:6800/30168 conn(0x7f42f0037790 0x7f42f0039c50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4308009870 tx=0x7f430803d040 comp rx=0 tx=0).stop 2024-04-22T08:12:52.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.030+0000 7f42ef7fe640 10 monclient: shutdown 2024-04-22T08:12:52.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.030+0000 7f42ef7fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:52.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.030+0000 7f42ef7fe640 1 -- 172.21.15.40:0/182044375 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f430c136230 msgr2=0x7f430c1cb1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.030+0000 7f42ef7fe640 1 --2- 172.21.15.40:0/182044375 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f430c136230 0x7f430c1cb1a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f42fc00e810 tx=0x7f42fc00ecd0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.030+0000 7f4311942640 1 -- 172.21.15.40:0/182044375 reap_dead start 2024-04-22T08:12:52.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.030+0000 7f42ef7fe640 1 -- 172.21.15.40:0/182044375 shutdown_connections 2024-04-22T08:12:52.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.030+0000 7f6345372640 10 monclient(hunting): handle_auth_done global_id 4248 payload 274 2024-04-22T08:12:52.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.030+0000 7f6345372640 1 -- 172.21.15.40:0/3772186029 >> v1:172.21.15.189:6789/0 conn(0x7f634009e960 legacy=0x7f634009ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:52.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.030+0000 7f42ef7fe640 1 -- 172.21.15.40:0/182044375 >> 172.21.15.40:0/182044375 conn(0x7f430c0a4e80 msgr2=0x7f430c06ed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:52.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.030+0000 7f6345372640 1 -- 172.21.15.40:0/3772186029 >> v1:172.21.15.40:6789/0 conn(0x7f6340138080 legacy=0x7f634013a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:52.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.030+0000 7f6345372640 10 monclient: _finish_hunting 0 2024-04-22T08:12:52.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.030+0000 7f6345372640 1 monclient: found mon.noname-a 2024-04-22T08:12:52.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.030+0000 7fd9d3210640 1 Processor -- start 2024-04-22T08:12:52.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.030+0000 7f42ef7fe640 1 -- 172.21.15.40:0/182044375 shutdown_connections 2024-04-22T08:12:52.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.030+0000 7fd9d3210640 1 -- start start 2024-04-22T08:12:52.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.030+0000 7f42ef7fe640 1 -- 172.21.15.40:0/182044375 wait complete. 2024-04-22T08:12:52.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.030+0000 7fd9d3210640 10 monclient: init 2024-04-22T08:12:52.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.030+0000 7f6345372640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:52.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.030+0000 7f6345372640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:12:52.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.030+0000 7f6345372640 1 -- 172.21.15.40:0/3772186029 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f634009b870 con 0x7f6340132d00 2024-04-22T08:12:52.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.031+0000 7fad3d3d0640 1 -- 172.21.15.40:0/1972830962 --> v2:172.21.15.189:6805/30673 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fad38136ac0 con 0x7fad2403d2c0 2024-04-22T08:12:52.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.031+0000 7fd9d3210640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:52.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.031+0000 7fd9d3210640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-22T08:12:52.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.031+0000 7f3a57fff640 1 -- 172.21.15.40:0/3999428160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a58136230 msgr2=0x7f3a58120900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:52.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.031+0000 7f3a57fff640 1 --2- 172.21.15.40:0/3999428160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a58136230 0x7f3a58120900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3a4400c3e0 tx=0x7f3a4400c8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:52.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.031+0000 7fd9d3210640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9cc136230 0x7fd9cc1cab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.031+0000 7fd9d3210640 10 monclient(hunting): picked mon.noname-b con 0x7fd9cc136230 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:52.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.031+0000 7fd9d3210640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9cc136670 0x7fd9cc1cb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.031+0000 7fd9d3210640 10 monclient(hunting): picked mon.noname-a con 0x7fd9cc136670 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:52.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.031+0000 7fd9d3210640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9cc1223d0 0x7fd9cc122840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.031+0000 7f3a557fa640 1 -- 172.21.15.40:0/3999428160 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a440070d0 con 0x7f3a58136230 2024-04-22T08:12:52.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.032+0000 7f3a557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:52.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.032+0000 7f3a557fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:12:52.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.032+0000 7f3a557fa640 10 monclient: dump: 2024-04-22T08:12:52.094 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:52.094 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.094 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.094 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.094 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:52.094 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.095 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:52.095 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:52.095 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:52.095 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.032+0000 7f3a557fa640 10 monclient: _finish_auth 0 2024-04-22T08:12:52.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.032+0000 7f3a557fa640 10 monclient: _check_auth_tickets 2024-04-22T08:12:52.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.032+0000 7f3a557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:52.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.032+0000 7fad1f7fe640 1 -- 172.21.15.40:0/1972830962 <== osd.3 v2:172.21.15.189:6805/30673 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fad38136ac0 con 0x7fad2403d2c0 2024-04-22T08:12:52.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.032+0000 7f3a557fa640 1 -- 172.21.15.40:0/3999428160 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a44007230 con 0x7f3a58136230 2024-04-22T08:12:52.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.032+0000 7f3a5ee99640 5 monclient: authenticate success, global_id 4225 2024-04-22T08:12:52.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.032+0000 7f3a5ee99640 10 monclient: _renew_subs 2024-04-22T08:12:52.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.032+0000 7f3a5ee99640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.189:3300/0 2024-04-22T08:12:52.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.032+0000 7f3a5ee99640 1 -- 172.21.15.40:0/3999428160 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a581236a0 con 0x7f3a58136230 2024-04-22T08:12:52.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.031+0000 7fd9d0fac640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9cc136230 0x7fd9cc1cab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.032+0000 7f3a557fa640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:52.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.032+0000 7f3a557fa640 1 -- 172.21.15.40:0/3999428160 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a440075e0 con 0x7f3a58136230 2024-04-22T08:12:52.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.032+0000 7f3a557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:52.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.032+0000 7f3a557fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:12:52.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.032+0000 7f3a557fa640 10 monclient: dump: 2024-04-22T08:12:52.098 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:52.098 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.098 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.098 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.098 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:52.098 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.098 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:52.098 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:52.098 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:52.098 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.032+0000 7f3a56ffd640 4 set_mon_vals no callback set 2024-04-22T08:12:52.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.032+0000 7f3a5ee99640 10 monclient: _renew_subs 2024-04-22T08:12:52.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.032+0000 7f3a5ee99640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.189:3300/0 2024-04-22T08:12:52.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.032+0000 7f3a5ee99640 1 -- 172.21.15.40:0/3999428160 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a58121380 con 0x7f3a58136230 2024-04-22T08:12:52.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.031+0000 7fd9d0fac640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9cc136230 0x7fd9cc1cab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.40:42682/0 (socket says 172.21.15.40:42682) 2024-04-22T08:12:52.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.032+0000 7fd9cbfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9cc136670 0x7fd9cc1cb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.032+0000 7fd9cbfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9cc136670 0x7fd9cc1cb0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57874/0 (socket says 172.21.15.40:57874) 2024-04-22T08:12:52.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.032+0000 7fd9d17ad640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9cc1223d0 0x7fd9cc122840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.032+0000 7fd9d3210640 10 monclient(hunting): picked mon.noname-c con 0x7fd9cc1223d0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:52.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.032+0000 7fd9d3210640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.032+0000 7fad3d3d0640 1 -- 172.21.15.40:0/1972830962 >> v2:172.21.15.189:6805/30673 conn(0x7fad2403d2c0 msgr2=0x7fad2403f6e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.032+0000 7fad3d3d0640 1 --2- 172.21.15.40:0/1972830962 >> v2:172.21.15.189:6805/30673 conn(0x7fad2403d2c0 0x7fad2403f6e0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.032+0000 7fd9d3210640 1 -- 172.21.15.40:0/434605762 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9cc125a60 con 0x7fd9cc136670 2024-04-22T08:12:52.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.032+0000 7fd9d3210640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.032+0000 7fd9d3210640 1 -- 172.21.15.40:0/434605762 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9cc124b00 con 0x7fd9cc136230 2024-04-22T08:12:52.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.032+0000 7fad3d3d0640 1 -- 172.21.15.40:0/1972830962 >> v2:172.21.15.189:6800/30168 conn(0x7fad24037790 msgr2=0x7fad24039c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.032+0000 7fad3d3d0640 1 --2- 172.21.15.40:0/1972830962 >> v2:172.21.15.189:6800/30168 conn(0x7fad24037790 0x7fad24039c50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fad28005160 tx=0x7fad28009040 comp rx=0 tx=0).stop 2024-04-22T08:12:52.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.032+0000 7fad3d3d0640 10 monclient: shutdown 2024-04-22T08:12:52.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.032+0000 7fad3d3d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:52.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.032+0000 7fad3d3d0640 1 -- 172.21.15.40:0/1972830962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad3813b410 msgr2=0x7fad381ca680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.032+0000 7fad3d3d0640 1 --2- 172.21.15.40:0/1972830962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad3813b410 0x7fad381ca680 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fad3000c750 tx=0x7fad3000cc10 comp rx=0 tx=0).stop 2024-04-22T08:12:52.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.032+0000 7fd9d0fac640 1 -- 172.21.15.40:0/434605762 learned_addr learned my addr 172.21.15.40:0/434605762 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:52.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.032+0000 7fd9d3210640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.032+0000 7fd9d3210640 1 -- 172.21.15.40:0/434605762 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9cc0a2710 con 0x7fd9cc1223d0 2024-04-22T08:12:52.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.032+0000 7fd9d3210640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:52.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.032+0000 7fad377fe640 1 -- 172.21.15.40:0/1972830962 reap_dead start 2024-04-22T08:12:52.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.032+0000 7fd9d3210640 10 monclient(hunting): authenticate will time out at 846.026737s 2024-04-22T08:12:52.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.032+0000 7fd9cbfff640 10 monclient(hunting): get_auth_request con 0x7fd9cc136670 auth_method 0 2024-04-22T08:12:52.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.033+0000 7fd9cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:52.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.033+0000 7fd9cbfff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:52.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.033+0000 7fd9cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:52.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.033+0000 7fd9d0fac640 10 monclient(hunting): get_auth_request con 0x7fd9cc136230 auth_method 0 2024-04-22T08:12:52.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.033+0000 7fd9d0fac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:52.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.033+0000 7fd9d0fac640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:52.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.033+0000 7fd9d0fac640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:52.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.033+0000 7fad3d3d0640 1 -- 172.21.15.40:0/1972830962 shutdown_connections 2024-04-22T08:12:52.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.033+0000 7fad3d3d0640 1 -- 172.21.15.40:0/1972830962 >> 172.21.15.40:0/1972830962 conn(0x7fad380a89a0 msgr2=0x7fad380a66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:52.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.033+0000 7f3a557fa640 1 -- 172.21.15.40:0/3999428160 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 44726+0+0 (secure 0 0 0) 0x7f3a4401b3b0 con 0x7f3a58136230 2024-04-22T08:12:52.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.033+0000 7fd9d17ad640 10 monclient(hunting): get_auth_request con 0x7fd9cc1223d0 auth_method 0 2024-04-22T08:12:52.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.033+0000 7fb95b7fe640 1 -- 172.21.15.40:0/598358128 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb95c133d80 msgr2=0x7fb95c134160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:52.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.033+0000 7fb95b7fe640 1 --2- 172.21.15.40:0/598358128 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb95c133d80 0x7fb95c134160 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb95c0725f0 tx=0x7fb94c02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23b035d7fc7dc058 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:52.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.033+0000 7f3a5ee99640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.189:3300/0 2024-04-22T08:12:52.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.033+0000 7f3a5ee99640 1 -- 172.21.15.40:0/3999428160 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3a580a24c0 con 0x7f3a58136230 2024-04-22T08:12:52.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.033+0000 7fad3d3d0640 1 -- 172.21.15.40:0/1972830962 shutdown_connections 2024-04-22T08:12:52.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.033+0000 7fd9d17ad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:52.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.033+0000 7fad3d3d0640 1 -- 172.21.15.40:0/1972830962 wait complete. 2024-04-22T08:12:52.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.033+0000 7f3a557fa640 1 --2- 172.21.15.40:0/3999428160 >> v2:172.21.15.189:6800/30168 conn(0x7f3a3c037790 0x7f3a3c039c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.033+0000 7fd9d17ad640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:52.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.033+0000 7fd9d17ad640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:52.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.033+0000 7fd9cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:52.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.033+0000 7fd9cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:52.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.033+0000 7fd9cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:52.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.033+0000 7fd9d17ad640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:52.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.033+0000 7fd9d17ad640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:52.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.033+0000 7fd9d17ad640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:52.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.033+0000 7f3a557fa640 1 -- 172.21.15.40:0/3999428160 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f3a440497d0 con 0x7f3a58136230 2024-04-22T08:12:52.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.033+0000 7fd9cbfff640 10 monclient(hunting): handle_auth_done global_id 4251 payload 824 2024-04-22T08:12:52.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.033+0000 7fd9cbfff640 1 -- 172.21.15.40:0/434605762 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9cc1223d0 msgr2=0x7fd9cc122840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.033+0000 7fd9cbfff640 1 --2- 172.21.15.40:0/434605762 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9cc1223d0 0x7fd9cc122840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.033+0000 7fd9cbfff640 1 -- 172.21.15.40:0/434605762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9cc136230 msgr2=0x7fd9cc1cab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.033+0000 7fd9cbfff640 1 --2- 172.21.15.40:0/434605762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9cc136230 0x7fd9cc1cab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.033+0000 7fd9cbfff640 10 monclient: _finish_hunting 0 2024-04-22T08:12:52.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.033+0000 7fd9cbfff640 1 monclient: found mon.noname-a 2024-04-22T08:12:52.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.033+0000 7fd9cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:52.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.033+0000 7fd9cbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:12:52.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.033+0000 7fd9cbfff640 1 -- 172.21.15.40:0/434605762 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9c0009b30 con 0x7fd9cc136670 2024-04-22T08:12:52.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.033+0000 7fd9d17ad640 1 --2- 172.21.15.40:0/434605762 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9cc1223d0 0x7fd9cc122840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:12:52.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.033+0000 7f3a5cc35640 1 --2- 172.21.15.40:0/3999428160 >> v2:172.21.15.189:6800/30168 conn(0x7f3a3c037790 0x7f3a3c039c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.033+0000 7f3a5cc35640 10 monclient: get_auth_request con 0x7f3a3c037790 auth_method 0 2024-04-22T08:12:52.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.034+0000 7f3a557fa640 1 --2- 172.21.15.40:0/3999428160 >> v2:172.21.15.189:6806/30674 conn(0x7f3a3c03d240 0x7f3a3c03f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.034+0000 7f3a557fa640 1 -- 172.21.15.40:0/3999428160 --> v2:172.21.15.189:6806/30674 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3a3c03fcd0 con 0x7f3a3c03d240 2024-04-22T08:12:52.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.034+0000 7f3a5d436640 1 --2- 172.21.15.40:0/3999428160 >> v2:172.21.15.189:6806/30674 conn(0x7f3a3c03d240 0x7f3a3c03f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.034+0000 7f3a5d436640 10 monclient: get_auth_request con 0x7f3a3c03d240 auth_method 0 2024-04-22T08:12:52.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.035+0000 7f6345372640 1 -- 172.21.15.40:0/3772186029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6340132d00 msgr2=0x7f6340137b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:52.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.035+0000 7f6345372640 1 --2- 172.21.15.40:0/3772186029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6340132d00 0x7f6340137b40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6330009d90 tx=0x7f633002bd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6cc790a790e253b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:52.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.035+0000 7fb95a7fc640 1 -- 172.21.15.40:0/598358128 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb94c031a40 con 0x7fb95c133d80 2024-04-22T08:12:52.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.035+0000 7fb95a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:52.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.035+0000 7fb95a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:12:52.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.035+0000 7fb95a7fc640 10 monclient: dump: 2024-04-22T08:12:52.111 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:52.112 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.112 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.112 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.112 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:52.112 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.112 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:52.112 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:52.112 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:52.112 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.035+0000 7fb95a7fc640 10 monclient: _finish_auth 0 2024-04-22T08:12:52.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.035+0000 7fb95a7fc640 10 monclient: _check_auth_tickets 2024-04-22T08:12:52.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.035+0000 7fb95a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:52.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.035+0000 7fb95a7fc640 1 -- 172.21.15.40:0/598358128 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb94c031ba0 con 0x7fb95c133d80 2024-04-22T08:12:52.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.035+0000 7fb95a7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:52.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.035+0000 7fb95a7fc640 1 -- 172.21.15.40:0/598358128 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb94c033440 con 0x7fb95c133d80 2024-04-22T08:12:52.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.035+0000 7fb95a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:52.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.035+0000 7fb95a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:12:52.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.035+0000 7fb95a7fc640 10 monclient: dump: 2024-04-22T08:12:52.113 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:52.113 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.113 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.113 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.113 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:52.113 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.113 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:52.113 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:52.113 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:52.113 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.035+0000 7fb962066640 5 monclient: authenticate success, global_id 4245 2024-04-22T08:12:52.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.035+0000 7fb962066640 10 monclient: get_monmap_and_config success 2024-04-22T08:12:52.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.035+0000 7fb962066640 4 set_mon_vals no callback set 2024-04-22T08:12:52.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.035+0000 7f6344b71640 1 -- 172.21.15.40:0/3772186029 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f633002cda0 con 0x7f6340132d00 2024-04-22T08:12:52.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.035+0000 7f6344b71640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:52.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.035+0000 7f6344b71640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:12:52.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.035+0000 7f6344b71640 10 monclient: dump: 2024-04-22T08:12:52.115 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:52.115 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.115 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.115 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.115 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:52.115 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.115 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:52.116 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:52.116 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:52.116 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.035+0000 7f6344b71640 10 monclient: _finish_auth 0 2024-04-22T08:12:52.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.035+0000 7f6344b71640 10 monclient: _check_auth_tickets 2024-04-22T08:12:52.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.035+0000 7f6344b71640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:52.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.035+0000 7f6347dd7640 5 monclient: authenticate success, global_id 4248 2024-04-22T08:12:52.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.035+0000 7f6347dd7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:12:52.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.035+0000 7f6344b71640 1 -- 172.21.15.40:0/3772186029 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6330034440 con 0x7f6340132d00 2024-04-22T08:12:52.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.035+0000 7f6344b71640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:52.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.035+0000 7f6347dd7640 10 monclient: get_monmap_and_config success 2024-04-22T08:12:52.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.035+0000 7f6347dd7640 4 set_mon_vals no callback set 2024-04-22T08:12:52.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.035+0000 7f6344b71640 1 -- 172.21.15.40:0/3772186029 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63300347d0 con 0x7f6340132d00 2024-04-22T08:12:52.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.036+0000 7f6344b71640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:52.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.036+0000 7f6344b71640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:12:52.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.036+0000 7f6344b71640 10 monclient: dump: 2024-04-22T08:12:52.117 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:52.117 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.117 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.117 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.117 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:52.117 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.117 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:52.117 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:52.117 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:52.117 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.036+0000 7fd9cbfff640 1 -- 172.21.15.40:0/434605762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9cc136670 msgr2=0x7fd9cc1cb0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:52.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.036+0000 7fd9cbfff640 1 --2- 172.21.15.40:0/434605762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9cc136670 0x7fd9cc1cb0c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd9b800c410 tx=0x7fd9b800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:52.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.036+0000 7fd9c97fa640 1 -- 172.21.15.40:0/434605762 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b80037f0 con 0x7fd9cc136670 2024-04-22T08:12:52.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.036+0000 7fd9c97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:52.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.036+0000 7fd9c97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:12:52.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.036+0000 7fd9c97fa640 10 monclient: dump: 2024-04-22T08:12:52.119 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:52.119 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.119 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.119 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.119 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:52.119 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.119 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:52.119 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:52.119 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:52.119 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.036+0000 7fd9c97fa640 10 monclient: _finish_auth 0 2024-04-22T08:12:52.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.036+0000 7fd9c97fa640 10 monclient: _check_auth_tickets 2024-04-22T08:12:52.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.036+0000 7fd9c97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:52.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.036+0000 7fd9c97fa640 1 -- 172.21.15.40:0/434605762 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9b8003970 con 0x7fd9cc136670 2024-04-22T08:12:52.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.036+0000 7fd9c97fa640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:52.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.036+0000 7fd9c97fa640 1 -- 172.21.15.40:0/434605762 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b8003d20 con 0x7fd9cc136670 2024-04-22T08:12:52.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.036+0000 7fd9c97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:52.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.036+0000 7fd9c97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:12:52.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.036+0000 7fd9c97fa640 10 monclient: dump: 2024-04-22T08:12:52.120 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:52.120 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.120 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.120 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.120 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:52.120 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.120 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:52.120 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:52.120 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:52.121 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.036+0000 7fd9caffd640 4 set_mon_vals no callback set 2024-04-22T08:12:52.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.036+0000 7f3a5cc35640 1 --2- 172.21.15.40:0/3999428160 >> v2:172.21.15.189:6800/30168 conn(0x7f3a3c037790 0x7f3a3c039c50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c0091d0 tx=0x7f3a4c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:52.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.036+0000 7fb962066640 10 monclient: shutdown 2024-04-22T08:12:52.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.036+0000 7f6347dd7640 10 monclient: shutdown 2024-04-22T08:12:52.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.036+0000 7fb962066640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:52.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.036+0000 7fb962066640 1 -- 172.21.15.40:0/598358128 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb95c133d80 msgr2=0x7fb95c134160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.036+0000 7fb962066640 1 --2- 172.21.15.40:0/598358128 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb95c133d80 0x7fb95c134160 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb95c0725f0 tx=0x7fb94c02b7f0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.036+0000 7fb962066640 1 -- 172.21.15.40:0/598358128 shutdown_connections 2024-04-22T08:12:52.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.036+0000 7f6347dd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:52.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.036+0000 7f6347dd7640 1 -- 172.21.15.40:0/3772186029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6340132d00 msgr2=0x7f6340137b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.036+0000 7f6347dd7640 1 --2- 172.21.15.40:0/3772186029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6340132d00 0x7f6340137b40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6330009d90 tx=0x7f633002bd00 comp rx=0 tx=0).stop 2024-04-22T08:12:52.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.036+0000 7fb962066640 1 --2- 172.21.15.40:0/598358128 >> v2:172.21.15.189:3300/0 conn(0x7fb95c09f9c0 0x7fb95c09fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.036+0000 7fb962066640 1 --2- 172.21.15.40:0/598358128 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb95c133d80 0x7fb95c134160 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.036+0000 7fb962066640 1 -- 172.21.15.40:0/598358128 >> 172.21.15.40:0/598358128 conn(0x7fb95c0a6170 msgr2=0x7fb95c0a6580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:52.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.036+0000 7f6347dd7640 1 -- 172.21.15.40:0/3772186029 shutdown_connections 2024-04-22T08:12:52.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.036+0000 7f6347dd7640 1 --2- 172.21.15.40:0/3772186029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6340132d00 0x7f6340137b40 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.036+0000 7f6347dd7640 1 -- 172.21.15.40:0/3772186029 >> 172.21.15.40:0/3772186029 conn(0x7f63400a57e0 msgr2=0x7f63400a1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:52.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.036+0000 7fb962066640 1 -- 172.21.15.40:0/598358128 shutdown_connections 2024-04-22T08:12:52.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.036+0000 7fd9d3210640 5 monclient: authenticate success, global_id 4251 2024-04-22T08:12:52.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.037+0000 7fd9d3210640 10 monclient: _renew_subs 2024-04-22T08:12:52.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.037+0000 7fd9d3210640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:52.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.037+0000 7fd9d3210640 1 -- 172.21.15.40:0/434605762 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9cc0717a0 con 0x7fd9cc136670 2024-04-22T08:12:52.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.037+0000 7f6347dd7640 1 -- 172.21.15.40:0/3772186029 shutdown_connections 2024-04-22T08:12:52.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.037+0000 7f6347dd7640 1 -- 172.21.15.40:0/3772186029 wait complete. 2024-04-22T08:12:52.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.037+0000 7fd9d3210640 10 monclient: _renew_subs 2024-04-22T08:12:52.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.037+0000 7fd9d3210640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:52.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.037+0000 7fd9d3210640 1 -- 172.21.15.40:0/434605762 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9cc123f30 con 0x7fd9cc136670 2024-04-22T08:12:52.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.037+0000 7fb962066640 1 -- 172.21.15.40:0/598358128 wait complete. 2024-04-22T08:12:52.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.037+0000 7f6347dd7640 10 monclient: build_initial_monmap 2024-04-22T08:12:52.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.037+0000 7f6347dd7640 10 monclient: monmap: 2024-04-22T08:12:52.126 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:52.126 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:12:52.126 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:12:52.126 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:12:52.126 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:52.127 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.127 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:12:52.127 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:12:52.127 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:12:52.127 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.037+0000 7fb962066640 10 monclient: build_initial_monmap 2024-04-22T08:12:52.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.037+0000 7fb962066640 10 monclient: monmap: 2024-04-22T08:12:52.127 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:52.127 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:12:52.127 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:12:52.128 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:12:52.128 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:52.128 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.128 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:12:52.128 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:12:52.128 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:12:52.128 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.038+0000 7f6347dd7640 1 Processor -- start 2024-04-22T08:12:52.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.038+0000 7fd9d3210640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:52.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.038+0000 7fd9c97fa640 1 -- 172.21.15.40:0/434605762 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 4) v1 ==== 44726+0+0 (secure 0 0 0) 0x7fd9b801b3b0 con 0x7fd9cc136670 2024-04-22T08:12:52.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.038+0000 7fd9d3210640 1 -- 172.21.15.40:0/434605762 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd9cc0a24c0 con 0x7fd9cc136670 2024-04-22T08:12:52.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.038+0000 7fb962066640 1 Processor -- start 2024-04-22T08:12:52.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.038+0000 7fd9c97fa640 1 --2- 172.21.15.40:0/434605762 >> v2:172.21.15.189:6800/30168 conn(0x7fd9b0037790 0x7fd9b0039c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.038+0000 7fb962066640 1 -- start start 2024-04-22T08:12:52.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.038+0000 7fb962066640 10 monclient: init 2024-04-22T08:12:52.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.038+0000 7fd9c97fa640 1 -- 172.21.15.40:0/434605762 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3506+0+0 (secure 0 0 0) 0x7fd9b80497e0 con 0x7fd9cc136670 2024-04-22T08:12:52.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.038+0000 7fd9c97fa640 1 --2- 172.21.15.40:0/434605762 >> v2:172.21.15.189:6801/30672 conn(0x7fd9b003d240 0x7fd9b003f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.038+0000 7fd9c97fa640 1 -- 172.21.15.40:0/434605762 --> v2:172.21.15.189:6801/30672 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd9b003fcd0 con 0x7fd9b003d240 2024-04-22T08:12:52.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7f3a5d436640 1 --2- 172.21.15.40:0/3999428160 >> v2:172.21.15.189:6806/30674 conn(0x7f3a3c03d240 0x7f3a3c03f660 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:52.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7fd9d0fac640 1 --2- 172.21.15.40:0/434605762 >> v2:172.21.15.189:6800/30168 conn(0x7fd9b0037790 0x7fd9b0039c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7fd9d0fac640 10 monclient: get_auth_request con 0x7fd9b0037790 auth_method 0 2024-04-22T08:12:52.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.038+0000 7f6347dd7640 1 -- start start 2024-04-22T08:12:52.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.038+0000 7f6347dd7640 10 monclient: init 2024-04-22T08:12:52.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7f6347dd7640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:52.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7f6347dd7640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:12:52.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7fd9d17ad640 1 --2- 172.21.15.40:0/434605762 >> v2:172.21.15.189:6801/30672 conn(0x7fd9b003d240 0x7fd9b003f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7f6347dd7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f634009e960 0x7f63401c8d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7f6347dd7640 10 monclient(hunting): picked mon.noname-a con 0x7f634009e960 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:52.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7f6347dd7640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6340132d00 0x7f63401c92c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7f6347dd7640 10 monclient(hunting): picked mon.noname-b con 0x7f6340132d00 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:52.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7f6347dd7640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6340138080 0x7f634006f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7f6347dd7640 10 monclient(hunting): picked mon.noname-c con 0x7f6340138080 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:52.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7f6347dd7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7f6347dd7640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6340071d30 con 0x7f634009e960 2024-04-22T08:12:52.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7f6347dd7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7f6347dd7640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63401399c0 con 0x7f6340132d00 2024-04-22T08:12:52.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7f6347dd7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7f6347dd7640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6340070c30 con 0x7f6340138080 2024-04-22T08:12:52.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7f6347dd7640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:52.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7f6347dd7640 10 monclient(hunting): authenticate will time out at 846.033120s 2024-04-22T08:12:52.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7fd9d17ad640 10 monclient: get_auth_request con 0x7fd9b003d240 auth_method 0 2024-04-22T08:12:52.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7fb962066640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:52.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7f6346374640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6340138080 0x7f634006f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7f6346374640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6340138080 0x7f634006f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49680/0 (socket says 172.21.15.40:49680) 2024-04-22T08:12:52.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7f6346374640 1 -- 172.21.15.40:0/3340050021 learned_addr learned my addr 172.21.15.40:0/3340050021 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:52.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7f6346374640 10 monclient(hunting): get_auth_request con 0x7f6340138080 auth_method 0 2024-04-22T08:12:52.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7f6346374640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:52.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7f6346374640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:52.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7f6346374640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:52.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7fd9c97fa640 1 -- 172.21.15.40:0/434605762 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd9b8019e80 con 0x7fd9cc136670 2024-04-22T08:12:52.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7fd9c97fa640 10 monclient: handle_get_version_reply finishing 1 version 15 2024-04-22T08:12:52.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7f6345b73640 1 --2- 172.21.15.40:0/3340050021 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f634009e960 0x7f63401c8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7f6345b73640 10 monclient(hunting): get_auth_request con 0x7f634009e960 auth_method 0 2024-04-22T08:12:52.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7f6345b73640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:52.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7f6345b73640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:52.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7f6345b73640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:52.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7f6346374640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:52.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7f6346374640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:52.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7f6346374640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:52.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7fb962066640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-22T08:12:52.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7fb962066640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb95c09f9c0 0x7fb95c127d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7f6345372640 1 --2- 172.21.15.40:0/3340050021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6340132d00 0x7f63401c92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7f6345372640 10 monclient(hunting): get_auth_request con 0x7f6340132d00 auth_method 0 2024-04-22T08:12:52.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7f6345372640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:52.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7f6345372640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:52.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7f6345372640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:52.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7f6345b73640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:52.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7f6345b73640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:52.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7f6345b73640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:52.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7fb962066640 10 monclient(hunting): picked mon.noname-b con 0x7fb95c09f9c0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:52.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7f6345372640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:52.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7f6345372640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:52.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7f6345372640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:52.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7fb962066640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb95c0a0b70 0x7fb95c128260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7fb962066640 10 monclient(hunting): picked mon.noname-a con 0x7fb95c0a0b70 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:52.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7fb962066640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb95c133d80 0x7fb95c1287a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7fb962066640 10 monclient(hunting): picked mon.noname-c con 0x7fb95c133d80 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:52.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7fb962066640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7fb962066640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb95c070850 con 0x7fb95c0a0b70 2024-04-22T08:12:52.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7fb962066640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7fb962066640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb95c0a03f0 con 0x7fb95c09f9c0 2024-04-22T08:12:52.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7fb962066640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7fb962066640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb95c06e0b0 con 0x7fb95c133d80 2024-04-22T08:12:52.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7fb962066640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:52.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.039+0000 7fb962066640 10 monclient(hunting): authenticate will time out at 846.033682s 2024-04-22T08:12:52.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7fb95affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb95c0a0b70 0x7fb95c128260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7fb95affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb95c0a0b70 0x7fb95c128260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57892/0 (socket says 172.21.15.40:57892) 2024-04-22T08:12:52.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7fb95affd640 1 -- 172.21.15.40:0/2190717234 learned_addr learned my addr 172.21.15.40:0/2190717234 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:52.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7fb95affd640 10 monclient(hunting): get_auth_request con 0x7fb95c0a0b70 auth_method 0 2024-04-22T08:12:52.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7fb95affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:52.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7fb95affd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:52.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7fb95affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:52.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7f6346374640 10 monclient(hunting): handle_auth_done global_id 4235 payload 824 2024-04-22T08:12:52.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7f6346374640 1 -- 172.21.15.40:0/3340050021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6340132d00 msgr2=0x7f63401c92c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7f6346374640 1 --2- 172.21.15.40:0/3340050021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6340132d00 0x7f63401c92c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7f6346374640 1 -- 172.21.15.40:0/3340050021 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f634009e960 msgr2=0x7f63401c8d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7f6346374640 1 --2- 172.21.15.40:0/3340050021 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f634009e960 0x7f63401c8d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7f6346374640 10 monclient: _finish_hunting 0 2024-04-22T08:12:52.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7f6346374640 1 monclient: found mon.noname-c 2024-04-22T08:12:52.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7f6346374640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:52.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7f6346374640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:12:52.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7f6346374640 1 -- 172.21.15.40:0/3340050021 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6330009a70 con 0x7f6340138080 2024-04-22T08:12:52.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7fb95bfff640 1 --2- 172.21.15.40:0/2190717234 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb95c133d80 0x7fb95c1287a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7fb95bfff640 10 monclient(hunting): get_auth_request con 0x7fb95c133d80 auth_method 0 2024-04-22T08:12:52.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7fb95bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:52.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7fb95bfff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:52.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7fb95b7fe640 1 --2- 172.21.15.40:0/2190717234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb95c09f9c0 0x7fb95c127d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7fb95bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:52.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7fb95affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:52.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7fb95affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:52.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7fb95affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:52.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7fb95b7fe640 10 monclient(hunting): get_auth_request con 0x7fb95c09f9c0 auth_method 0 2024-04-22T08:12:52.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7fb95b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:52.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7fb95b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:52.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7fb95b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:52.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7fb95b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:52.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7fb95b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:52.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7fb95b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:52.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7fb95bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:52.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7fb95bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:52.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7fb95bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:52.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7fb95affd640 10 monclient(hunting): handle_auth_done global_id 4257 payload 824 2024-04-22T08:12:52.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7fb95affd640 1 -- 172.21.15.40:0/2190717234 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb95c133d80 msgr2=0x7fb95c1287a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7fb95affd640 1 --2- 172.21.15.40:0/2190717234 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb95c133d80 0x7fb95c1287a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7fb95affd640 1 -- 172.21.15.40:0/2190717234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb95c09f9c0 msgr2=0x7fb95c127d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7fb95affd640 1 --2- 172.21.15.40:0/2190717234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb95c09f9c0 0x7fb95c127d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7fb95affd640 10 monclient: _finish_hunting 0 2024-04-22T08:12:52.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7fb95affd640 1 monclient: found mon.noname-a 2024-04-22T08:12:52.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7fb95affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:52.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7fb95affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:12:52.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.040+0000 7fb95affd640 1 -- 172.21.15.40:0/2190717234 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb94c0094d0 con 0x7fb95c0a0b70 2024-04-22T08:12:52.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.041+0000 7f3a557fa640 1 -- 172.21.15.40:0/3999428160 <== osd.5 v2:172.21.15.189:6806/30674 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f3a3c03fcd0 con 0x7f3a3c03d240 2024-04-22T08:12:52.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.041+0000 7fd9d0fac640 1 --2- 172.21.15.40:0/434605762 >> v2:172.21.15.189:6800/30168 conn(0x7fd9b0037790 0x7fd9b0039c50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd9c0000f80 tx=0x7fd9c0030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:52.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.042+0000 7f6346374640 1 -- 172.21.15.40:0/3340050021 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6340138080 msgr2=0x7f634006f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:52.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.042+0000 7f6346374640 1 --2- 172.21.15.40:0/3340050021 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6340138080 0x7f634006f140 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f633400c3e0 tx=0x7f633400c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:52.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.042+0000 7f633e7fc640 1 -- 172.21.15.40:0/3340050021 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63340070d0 con 0x7f6340138080 2024-04-22T08:12:52.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.042+0000 7f633e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:52.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.042+0000 7f633e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:12:52.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.042+0000 7f633e7fc640 10 monclient: dump: 2024-04-22T08:12:52.148 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:52.148 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.148 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.149 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.149 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:52.149 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.149 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:52.149 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:52.149 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:52.149 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.042+0000 7f633e7fc640 10 monclient: _finish_auth 0 2024-04-22T08:12:52.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.042+0000 7f633e7fc640 10 monclient: _check_auth_tickets 2024-04-22T08:12:52.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.043+0000 7f633e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:52.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.043+0000 7f6347dd7640 5 monclient: authenticate success, global_id 4235 2024-04-22T08:12:52.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.043+0000 7f6347dd7640 10 monclient: _renew_subs 2024-04-22T08:12:52.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.043+0000 7f6347dd7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:52.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.043+0000 7f6347dd7640 1 -- 172.21.15.40:0/3340050021 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6340139150 con 0x7f6340138080 2024-04-22T08:12:52.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.043+0000 7f633e7fc640 1 -- 172.21.15.40:0/3340050021 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6334007250 con 0x7f6340138080 2024-04-22T08:12:52.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.043+0000 7f633e7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:52.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.043+0000 7f633ffff640 4 set_mon_vals no callback set 2024-04-22T08:12:52.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.043+0000 7fb95affd640 1 -- 172.21.15.40:0/2190717234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb95c0a0b70 msgr2=0x7fb95c128260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:52.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.043+0000 7fb95affd640 1 --2- 172.21.15.40:0/2190717234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb95c0a0b70 0x7fb95c128260 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb95000b4a0 tx=0x7fb95000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:52.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.043+0000 7fb943fff640 1 -- 172.21.15.40:0/2190717234 DispatchQueue::entry inject delay of 0.002000 2024-04-22T08:12:52.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.043+0000 7f633e7fc640 1 -- 172.21.15.40:0/3340050021 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6334007600 con 0x7f6340138080 2024-04-22T08:12:52.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.043+0000 7f6347dd7640 10 monclient: _renew_subs 2024-04-22T08:12:52.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.043+0000 7f6347dd7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:52.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.043+0000 7f6347dd7640 1 -- 172.21.15.40:0/3340050021 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6340139400 con 0x7f6340138080 2024-04-22T08:12:52.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.043+0000 7f633e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:52.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.043+0000 7f633e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:12:52.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.043+0000 7f633e7fc640 10 monclient: dump: 2024-04-22T08:12:52.151 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:52.151 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.151 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.151 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.152 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:52.152 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.152 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:52.152 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:52.152 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:52.152 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.044+0000 7fd9d17ad640 1 --2- 172.21.15.40:0/434605762 >> v2:172.21.15.189:6801/30672 conn(0x7fd9b003d240 0x7fd9b003f660 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:52.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.044+0000 7f6347dd7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:52.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.044+0000 7f6347dd7640 1 -- 172.21.15.40:0/3340050021 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f63400718e0 con 0x7f6340138080 2024-04-22T08:12:52.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.045+0000 7fd9c97fa640 1 -- 172.21.15.40:0/434605762 <== osd.4 v2:172.21.15.189:6801/30672 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fd9b003fcd0 con 0x7fd9b003d240 2024-04-22T08:12:52.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.045+0000 7fb943fff640 1 -- 172.21.15.40:0/2190717234 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb950010820 con 0x7fb95c0a0b70 2024-04-22T08:12:52.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.045+0000 7fb943fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:52.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.045+0000 7f633e7fc640 1 -- 172.21.15.40:0/3340050021 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 4) v1 ==== 44726+0+0 (secure 0 0 0) 0x7f6334017020 con 0x7f6340138080 2024-04-22T08:12:52.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.045+0000 7f633e7fc640 1 --2- 172.21.15.40:0/3340050021 >> v2:172.21.15.189:6800/30168 conn(0x7f632c037740 0x7f632c039c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.045+0000 7f633e7fc640 1 -- 172.21.15.40:0/3340050021 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f633404d8e0 con 0x7f6340138080 2024-04-22T08:12:52.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.045+0000 7f633e7fc640 1 --2- 172.21.15.40:0/3340050021 >> v2:172.21.15.40:6808/30671 conn(0x7f632c03d1f0 0x7f632c03f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.045+0000 7f633e7fc640 1 -- 172.21.15.40:0/3340050021 --> v2:172.21.15.40:6808/30671 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f632c03fc80 con 0x7f632c03d1f0 2024-04-22T08:12:52.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.045+0000 7f633e7fc640 1 -- 172.21.15.40:0/3340050021 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7f633404fda0 con 0x7f6340138080 2024-04-22T08:12:52.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.045+0000 7f633e7fc640 10 monclient: handle_get_version_reply finishing 1 version 15 2024-04-22T08:12:52.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.046+0000 7f6345b73640 1 --2- 172.21.15.40:0/3340050021 >> v2:172.21.15.189:6800/30168 conn(0x7f632c037740 0x7f632c039c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.046+0000 7f6345b73640 10 monclient: get_auth_request con 0x7f632c037740 auth_method 0 2024-04-22T08:12:52.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.046+0000 7f6345372640 1 --2- 172.21.15.40:0/3340050021 >> v2:172.21.15.40:6808/30671 conn(0x7f632c03d1f0 0x7f632c03f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.046+0000 7f6345372640 10 monclient: get_auth_request con 0x7f632c03d1f0 auth_method 0 2024-04-22T08:12:52.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.046+0000 7fb943fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:12:52.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.046+0000 7fb943fff640 10 monclient: dump: 2024-04-22T08:12:52.156 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:52.156 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.156 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.156 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.156 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:52.156 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.156 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:52.157 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:52.157 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:52.157 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.046+0000 7fb943fff640 10 monclient: _finish_auth 0 2024-04-22T08:12:52.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.046+0000 7fb943fff640 10 monclient: _check_auth_tickets 2024-04-22T08:12:52.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.046+0000 7fb943fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:52.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.046+0000 7fb943fff640 1 -- 172.21.15.40:0/2190717234 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb950010980 con 0x7fb95c0a0b70 2024-04-22T08:12:52.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.046+0000 7fb943fff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:52.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.046+0000 7fb943fff640 1 -- 172.21.15.40:0/2190717234 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb950010d30 con 0x7fb95c0a0b70 2024-04-22T08:12:52.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.046+0000 7fb943fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:52.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.046+0000 7fb943fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:12:52.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.046+0000 7fb943fff640 10 monclient: dump: 2024-04-22T08:12:52.157 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:52.157 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.157 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.157 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.158 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:52.158 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.158 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:52.158 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:52.158 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:52.158 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.046+0000 7fb959ffb640 4 set_mon_vals no callback set 2024-04-22T08:12:52.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.046+0000 7fb962066640 5 monclient: authenticate success, global_id 4257 2024-04-22T08:12:52.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.046+0000 7fb962066640 10 monclient: _renew_subs 2024-04-22T08:12:52.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.046+0000 7fb962066640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:52.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.046+0000 7fb962066640 1 -- 172.21.15.40:0/2190717234 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb95c133820 con 0x7fb95c0a0b70 2024-04-22T08:12:52.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.046+0000 7fb962066640 10 monclient: _renew_subs 2024-04-22T08:12:52.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.046+0000 7fb962066640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:52.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.046+0000 7fb962066640 1 -- 172.21.15.40:0/2190717234 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb95c06f8f0 con 0x7fb95c0a0b70 2024-04-22T08:12:52.159 INFO:teuthology.orchestra.run.smithi040.stdout:47244640260 2024-04-22T08:12:52.160 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-22T08:12:52.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.047+0000 7fb943fff640 1 -- 172.21.15.40:0/2190717234 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 4) v1 ==== 44726+0+0 (secure 0 0 0) 0x7fb950017360 con 0x7fb95c0a0b70 2024-04-22T08:12:52.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.048+0000 7fb962066640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:52.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.048+0000 7fb962066640 1 -- 172.21.15.40:0/2190717234 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb95c1346a0 con 0x7fb95c0a0b70 2024-04-22T08:12:52.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.048+0000 7fb943fff640 1 --2- 172.21.15.40:0/2190717234 >> v2:172.21.15.189:6800/30168 conn(0x7fb948037790 0x7fb948039c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.048+0000 7fb943fff640 1 -- 172.21.15.40:0/2190717234 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3506+0+0 (secure 0 0 0) 0x7fb950049df0 con 0x7fb95c0a0b70 2024-04-22T08:12:52.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.048+0000 7fb943fff640 1 --2- 172.21.15.40:0/2190717234 >> v2:172.21.15.40:6800/30670 conn(0x7fb94803d240 0x7fb94803f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.048+0000 7fb943fff640 1 -- 172.21.15.40:0/2190717234 --> v2:172.21.15.40:6800/30670 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb94803fcd0 con 0x7fb94803d240 2024-04-22T08:12:52.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.048+0000 7fb95bfff640 1 --2- 172.21.15.40:0/2190717234 >> v2:172.21.15.40:6800/30670 conn(0x7fb94803d240 0x7fb94803f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.048+0000 7fb943fff640 1 -- 172.21.15.40:0/2190717234 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb95000d300 con 0x7fb95c0a0b70 2024-04-22T08:12:52.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.048+0000 7fb943fff640 10 monclient: handle_get_version_reply finishing 1 version 15 2024-04-22T08:12:52.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.048+0000 7fb95b7fe640 1 --2- 172.21.15.40:0/2190717234 >> v2:172.21.15.189:6800/30168 conn(0x7fb948037790 0x7fb948039c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.048+0000 7fb95b7fe640 10 monclient: get_auth_request con 0x7fb948037790 auth_method 0 2024-04-22T08:12:52.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.048+0000 7fb95bfff640 10 monclient: get_auth_request con 0x7fb94803d240 auth_method 0 2024-04-22T08:12:52.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.048+0000 7f6345b73640 1 --2- 172.21.15.40:0/3340050021 >> v2:172.21.15.189:6800/30168 conn(0x7f632c037740 0x7f632c039c00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f63380017f0 tx=0x7f6338009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:52.163 INFO:teuthology.orchestra.run.smithi040.stdout:47244640259 2024-04-22T08:12:52.163 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-22T08:12:52.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.050+0000 7f6345372640 1 --2- 172.21.15.40:0/3340050021 >> v2:172.21.15.40:6808/30671 conn(0x7f632c03d1f0 0x7f632c03f610 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:52.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.051+0000 7fb95b7fe640 1 --2- 172.21.15.40:0/2190717234 >> v2:172.21.15.189:6800/30168 conn(0x7fb948037790 0x7fb948039c50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb94c033000 tx=0x7fb94c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:52.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.052+0000 7f633e7fc640 1 -- 172.21.15.40:0/3340050021 <== osd.2 v2:172.21.15.40:6808/30671 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f632c03fc80 con 0x7f632c03d1f0 2024-04-22T08:12:52.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.053+0000 7fb95bfff640 1 --2- 172.21.15.40:0/2190717234 >> v2:172.21.15.40:6800/30670 conn(0x7fb94803d240 0x7fb94803f660 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:52.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.054+0000 7fb943fff640 1 -- 172.21.15.40:0/2190717234 <== osd.1 v2:172.21.15.40:6800/30670 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fb94803fcd0 con 0x7fb94803d240 2024-04-22T08:12:52.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.060+0000 7fd9d3210640 1 -- 172.21.15.40:0/434605762 --> v2:172.21.15.189:6801/30672 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd9cc04e9b0 con 0x7fd9b003d240 2024-04-22T08:12:52.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.061+0000 7fd9c97fa640 1 -- 172.21.15.40:0/434605762 <== osd.4 v2:172.21.15.189:6801/30672 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd9cc04e9b0 con 0x7fd9b003d240 2024-04-22T08:12:52.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.061+0000 7fd9d3210640 1 -- 172.21.15.40:0/434605762 >> v2:172.21.15.189:6801/30672 conn(0x7fd9b003d240 msgr2=0x7fd9b003f660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.061+0000 7fd9d3210640 1 --2- 172.21.15.40:0/434605762 >> v2:172.21.15.189:6801/30672 conn(0x7fd9b003d240 0x7fd9b003f660 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.061+0000 7fd9d3210640 1 -- 172.21.15.40:0/434605762 >> v2:172.21.15.189:6800/30168 conn(0x7fd9b0037790 msgr2=0x7fd9b0039c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.061+0000 7fd9d3210640 1 --2- 172.21.15.40:0/434605762 >> v2:172.21.15.189:6800/30168 conn(0x7fd9b0037790 0x7fd9b0039c50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd9c0000f80 tx=0x7fd9c0030000 comp rx=0 tx=0).stop 2024-04-22T08:12:52.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.061+0000 7fd9d3210640 10 monclient: shutdown 2024-04-22T08:12:52.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.061+0000 7fd9d3210640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:52.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.061+0000 7fd9d3210640 1 -- 172.21.15.40:0/434605762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9cc136670 msgr2=0x7fd9cc1cb0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.061+0000 7fd9d3210640 1 --2- 172.21.15.40:0/434605762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9cc136670 0x7fd9cc1cb0c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd9b800c410 tx=0x7fd9b800c810 comp rx=0 tx=0).stop 2024-04-22T08:12:52.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.061+0000 7fd9d17ad640 1 -- 172.21.15.40:0/434605762 reap_dead start 2024-04-22T08:12:52.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.062+0000 7fd9d3210640 1 -- 172.21.15.40:0/434605762 shutdown_connections 2024-04-22T08:12:52.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.062+0000 7fd9d3210640 1 -- 172.21.15.40:0/434605762 >> 172.21.15.40:0/434605762 conn(0x7fd9cc0a4e80 msgr2=0x7fd9cc0a7030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:52.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.062+0000 7fd9d3210640 1 -- 172.21.15.40:0/434605762 shutdown_connections 2024-04-22T08:12:52.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.062+0000 7fd9d3210640 1 -- 172.21.15.40:0/434605762 wait complete. 2024-04-22T08:12:52.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.064+0000 7f3a5ee99640 1 -- 172.21.15.40:0/3999428160 --> v2:172.21.15.189:6806/30674 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3a58121540 con 0x7f3a3c03d240 2024-04-22T08:12:52.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.065+0000 7f3a557fa640 1 -- 172.21.15.40:0/3999428160 <== osd.5 v2:172.21.15.189:6806/30674 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3a58121540 con 0x7f3a3c03d240 2024-04-22T08:12:52.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.065+0000 7f3a5ee99640 1 -- 172.21.15.40:0/3999428160 >> v2:172.21.15.189:6806/30674 conn(0x7f3a3c03d240 msgr2=0x7f3a3c03f660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.065+0000 7f3a5ee99640 1 --2- 172.21.15.40:0/3999428160 >> v2:172.21.15.189:6806/30674 conn(0x7f3a3c03d240 0x7f3a3c03f660 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.065+0000 7f3a5ee99640 1 -- 172.21.15.40:0/3999428160 >> v2:172.21.15.189:6800/30168 conn(0x7f3a3c037790 msgr2=0x7f3a3c039c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.065+0000 7f3a5ee99640 1 --2- 172.21.15.40:0/3999428160 >> v2:172.21.15.189:6800/30168 conn(0x7f3a3c037790 0x7f3a3c039c50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c0091d0 tx=0x7f3a4c008030 comp rx=0 tx=0).stop 2024-04-22T08:12:52.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.065+0000 7f3a5ee99640 10 monclient: shutdown 2024-04-22T08:12:52.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.065+0000 7f3a5ee99640 20 monclient: shutdown canceling and discarding version request 1 2024-04-22T08:12:52.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.065+0000 7f3a5ee99640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:52.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.066+0000 7f6347dd7640 1 -- 172.21.15.40:0/3340050021 --> v2:172.21.15.40:6808/30671 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f63401395f0 con 0x7f632c03d1f0 2024-04-22T08:12:52.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.065+0000 7f3a5ee99640 1 -- 172.21.15.40:0/3999428160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a58136230 msgr2=0x7f3a58120900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.065+0000 7f3a5ee99640 1 --2- 172.21.15.40:0/3999428160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a58136230 0x7f3a58120900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3a4400c3e0 tx=0x7f3a4400c8a0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.066+0000 7f3a5d436640 1 -- 172.21.15.40:0/3999428160 reap_dead start 2024-04-22T08:12:52.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.066+0000 7f3a5ee99640 1 -- 172.21.15.40:0/3999428160 shutdown_connections 2024-04-22T08:12:52.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.066+0000 7f3a5ee99640 1 -- 172.21.15.40:0/3999428160 >> 172.21.15.40:0/3999428160 conn(0x7f3a580a4e80 msgr2=0x7f3a58134480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:52.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.066+0000 7f3a5ee99640 1 -- 172.21.15.40:0/3999428160 shutdown_connections 2024-04-22T08:12:52.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.066+0000 7f3a5ee99640 1 -- 172.21.15.40:0/3999428160 wait complete. 2024-04-22T08:12:52.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.066+0000 7f633e7fc640 1 -- 172.21.15.40:0/3340050021 <== osd.2 v2:172.21.15.40:6808/30671 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f63401395f0 con 0x7f632c03d1f0 2024-04-22T08:12:52.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.066+0000 7f6347dd7640 1 -- 172.21.15.40:0/3340050021 >> v2:172.21.15.40:6808/30671 conn(0x7f632c03d1f0 msgr2=0x7f632c03f610 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.066+0000 7f6347dd7640 1 --2- 172.21.15.40:0/3340050021 >> v2:172.21.15.40:6808/30671 conn(0x7f632c03d1f0 0x7f632c03f610 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.067+0000 7f6347dd7640 1 -- 172.21.15.40:0/3340050021 >> v2:172.21.15.189:6800/30168 conn(0x7f632c037740 msgr2=0x7f632c039c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.067+0000 7f6347dd7640 1 --2- 172.21.15.40:0/3340050021 >> v2:172.21.15.189:6800/30168 conn(0x7f632c037740 0x7f632c039c00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f63380017f0 tx=0x7f6338009040 comp rx=0 tx=0).stop 2024-04-22T08:12:52.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.067+0000 7f6347dd7640 10 monclient: shutdown 2024-04-22T08:12:52.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.067+0000 7f6347dd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:52.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.067+0000 7f6347dd7640 1 -- 172.21.15.40:0/3340050021 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6340138080 msgr2=0x7f634006f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.067+0000 7f6347dd7640 1 --2- 172.21.15.40:0/3340050021 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6340138080 0x7f634006f140 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f633400c3e0 tx=0x7f633400c8a0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.067+0000 7f6346374640 1 -- 172.21.15.40:0/3340050021 reap_dead start 2024-04-22T08:12:52.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.067+0000 7f6347dd7640 1 -- 172.21.15.40:0/3340050021 shutdown_connections 2024-04-22T08:12:52.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.067+0000 7f6347dd7640 1 -- 172.21.15.40:0/3340050021 >> 172.21.15.40:0/3340050021 conn(0x7f63400a57e0 msgr2=0x7f634009d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:52.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.067+0000 7f6347dd7640 1 -- 172.21.15.40:0/3340050021 shutdown_connections 2024-04-22T08:12:52.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.067+0000 7f6347dd7640 1 -- 172.21.15.40:0/3340050021 wait complete. 2024-04-22T08:12:52.173 INFO:teuthology.orchestra.run.smithi040.stdout:47244640259 2024-04-22T08:12:52.174 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-22T08:12:52.174 INFO:teuthology.orchestra.run.smithi040.stdout:47244640261 2024-04-22T08:12:52.175 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-22T08:12:52.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.081+0000 7fb962066640 1 -- 172.21.15.40:0/2190717234 --> v2:172.21.15.40:6800/30670 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb95c04e9b0 con 0x7fb94803d240 2024-04-22T08:12:52.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.081+0000 7fb943fff640 1 -- 172.21.15.40:0/2190717234 <== osd.1 v2:172.21.15.40:6800/30670 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb95c04e9b0 con 0x7fb94803d240 2024-04-22T08:12:52.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.082+0000 7fb962066640 1 -- 172.21.15.40:0/2190717234 >> v2:172.21.15.40:6800/30670 conn(0x7fb94803d240 msgr2=0x7fb94803f660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.082+0000 7fb962066640 1 --2- 172.21.15.40:0/2190717234 >> v2:172.21.15.40:6800/30670 conn(0x7fb94803d240 0x7fb94803f660 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.082+0000 7fb962066640 1 -- 172.21.15.40:0/2190717234 >> v2:172.21.15.189:6800/30168 conn(0x7fb948037790 msgr2=0x7fb948039c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.082+0000 7fb962066640 1 --2- 172.21.15.40:0/2190717234 >> v2:172.21.15.189:6800/30168 conn(0x7fb948037790 0x7fb948039c50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb94c033000 tx=0x7fb94c008030 comp rx=0 tx=0).stop 2024-04-22T08:12:52.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.082+0000 7fb962066640 10 monclient: shutdown 2024-04-22T08:12:52.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.082+0000 7fb962066640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:52.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.082+0000 7fb962066640 1 -- 172.21.15.40:0/2190717234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb95c0a0b70 msgr2=0x7fb95c128260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.082+0000 7fb962066640 1 --2- 172.21.15.40:0/2190717234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb95c0a0b70 0x7fb95c128260 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb95000b4a0 tx=0x7fb95000b960 comp rx=0 tx=0).stop 2024-04-22T08:12:52.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.082+0000 7fb95bfff640 1 -- 172.21.15.40:0/2190717234 reap_dead start 2024-04-22T08:12:52.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.082+0000 7fb962066640 1 -- 172.21.15.40:0/2190717234 shutdown_connections 2024-04-22T08:12:52.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.082+0000 7fb962066640 1 -- 172.21.15.40:0/2190717234 >> 172.21.15.40:0/2190717234 conn(0x7fb95c0a6170 msgr2=0x7fb95c0a4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:52.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.082+0000 7fb962066640 1 -- 172.21.15.40:0/2190717234 shutdown_connections 2024-04-22T08:12:52.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.082+0000 7fb962066640 1 -- 172.21.15.40:0/2190717234 wait complete. 2024-04-22T08:12:52.178 INFO:teuthology.orchestra.run.smithi040.stdout:47244640260 2024-04-22T08:12:52.178 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-22T08:12:52.179 INFO:teuthology.orchestra.run.smithi040.stdout:47244640259 2024-04-22T08:12:52.179 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-22T08:12:52.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.318+0000 7f33773ae640 10 monclient: get_monmap_and_config 2024-04-22T08:12:52.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.318+0000 7f5a11870640 10 monclient: get_monmap_and_config 2024-04-22T08:12:52.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.318+0000 7f5a11870640 10 monclient: build_initial_monmap 2024-04-22T08:12:52.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.318+0000 7f5a11870640 10 monclient: monmap: 2024-04-22T08:12:52.318 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:52.318 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.318 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:52.319003+0000 2024-04-22T08:12:52.318 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:52.319003+0000 2024-04-22T08:12:52.318 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:52.318 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.318 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:12:52.318 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:12:52.318 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:12:52.319 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:12:52.319 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:12:52.319 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.318+0000 7f33773ae640 10 monclient: build_initial_monmap 2024-04-22T08:12:52.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.318+0000 7f33773ae640 10 monclient: monmap: 2024-04-22T08:12:52.319 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:52.319 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.320 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:52.318941+0000 2024-04-22T08:12:52.320 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:52.318941+0000 2024-04-22T08:12:52.320 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:52.320 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.320 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:12:52.320 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:12:52.320 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:12:52.320 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:12:52.320 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:12:52.320 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.318+0000 7f33773ae640 1 Processor -- start 2024-04-22T08:12:52.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.318+0000 7f5a11870640 1 Processor -- start 2024-04-22T08:12:52.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.319+0000 7f5a11870640 1 -- start start 2024-04-22T08:12:52.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.319+0000 7f5a11870640 10 monclient: init 2024-04-22T08:12:52.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.319+0000 7f33773ae640 1 -- start start 2024-04-22T08:12:52.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.319+0000 7f33773ae640 10 monclient: init 2024-04-22T08:12:52.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.319+0000 7f5a11870640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:52.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.319+0000 7f5a11870640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-04-22T08:12:52.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.319+0000 7f33773ae640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:52.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.319+0000 7f5a11870640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5a0c09ee10 addr v1:172.21.15.40:6789/0 2024-04-22T08:12:52.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.319+0000 7f5a11870640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5a0c0a1f30 0x7f5a0c0a3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.319+0000 7f5a11870640 10 monclient(hunting): picked mon.noname-c con 0x7f5a0c0a1f30 addr v2:172.21.15.189:3300/0 2024-04-22T08:12:52.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.319+0000 7f5a11870640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5a0c13b0e0 0x7f5a0c13d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.319+0000 7f5a11870640 10 monclient(hunting): picked mon.noname-a con 0x7f5a0c13b0e0 addr v2:172.21.15.40:3300/0 2024-04-22T08:12:52.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.319+0000 7f5a11870640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a0c0a4470 con 0x7f5a0c09ee10 2024-04-22T08:12:52.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.319+0000 7f5a11870640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.319+0000 7f5a11870640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a0c09f1f0 con 0x7f5a0c13b0e0 2024-04-22T08:12:52.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.319+0000 7f5a11870640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.319+0000 7f5a11870640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a0c09fd30 con 0x7f5a0c0a1f30 2024-04-22T08:12:52.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.319+0000 7f5a11870640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:52.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.319+0000 7f5a11870640 10 monclient(hunting): authenticate will time out at 846.313509s 2024-04-22T08:12:52.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.319+0000 7f33773ae640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-04-22T08:12:52.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.319+0000 7f5a0affd640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f5a0c09ee10 0x7f5a0c0a1820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:38892/0 (socket says 172.21.15.40:38892) 2024-04-22T08:12:52.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.319+0000 7f5a0affd640 1 -- 172.21.15.40:0/2700847048 learned_addr learned my addr 172.21.15.40:0/2700847048 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:12:52.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.319+0000 7f33773ae640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f337009ee10 addr v1:172.21.15.40:6789/0 2024-04-22T08:12:52.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.319+0000 7f5a0b7fe640 1 --2- 172.21.15.40:0/2700847048 >> v2:172.21.15.40:3300/0 conn(0x7f5a0c13b0e0 0x7f5a0c13d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.319+0000 7f33773ae640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33700a1f30 0x7f33700a3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.319+0000 7f33773ae640 10 monclient(hunting): picked mon.noname-b con 0x7f33700a1f30 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:52.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.319+0000 7f33773ae640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f337013b0e0 0x7f337013d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.319+0000 7f33773ae640 10 monclient(hunting): picked mon.noname-a con 0x7f337013b0e0 addr v2:172.21.15.40:3300/0 2024-04-22T08:12:52.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.319+0000 7f33773ae640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33700a4470 con 0x7f337009ee10 2024-04-22T08:12:52.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f33773ae640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f33773ae640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33700a2310 con 0x7f337013b0e0 2024-04-22T08:12:52.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f33773ae640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f33773ae640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f337009f1f0 con 0x7f33700a1f30 2024-04-22T08:12:52.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f33773ae640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:52.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f33773ae640 10 monclient(hunting): authenticate will time out at 846.313804s 2024-04-22T08:12:52.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f5a0a7fc640 1 --2- 172.21.15.40:0/2700847048 >> v2:172.21.15.189:3300/0 conn(0x7f5a0c0a1f30 0x7f5a0c0a3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f337594b640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f337013b0e0 0x7f337013d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f3374949640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33700a1f30 0x7f33700a3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f5a0b7fe640 10 monclient(hunting): get_auth_request con 0x7f5a0c13b0e0 auth_method 0 2024-04-22T08:12:52.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f337594b640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f337013b0e0 0x7f337013d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57898/0 (socket says 172.21.15.40:57898) 2024-04-22T08:12:52.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f337594b640 1 -- 172.21.15.40:0/3379446409 learned_addr learned my addr 172.21.15.40:0/3379446409 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:52.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f337594b640 10 monclient(hunting): get_auth_request con 0x7f337013b0e0 auth_method 0 2024-04-22T08:12:52.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f337594b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:52.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f337594b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:52.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f337594b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:52.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f3374949640 10 monclient(hunting): get_auth_request con 0x7f33700a1f30 auth_method 0 2024-04-22T08:12:52.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f3374949640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:52.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f3374949640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:52.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f3374949640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:52.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f5a0b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:52.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f5a0b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:52.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f5a0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:52.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f5a0a7fc640 10 monclient(hunting): get_auth_request con 0x7f5a0c0a1f30 auth_method 0 2024-04-22T08:12:52.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f3374949640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:52.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f3374949640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:52.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f3374949640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:52.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f337594b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:52.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f337594b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:52.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f337594b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:52.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f3374949640 10 monclient(hunting): handle_auth_done global_id 4241 payload 274 2024-04-22T08:12:52.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f3374949640 1 -- 172.21.15.40:0/3379446409 >> v2:172.21.15.40:3300/0 conn(0x7f337013b0e0 msgr2=0x7f337013d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:52.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f3374949640 1 --2- 172.21.15.40:0/3379446409 >> v2:172.21.15.40:3300/0 conn(0x7f337013b0e0 0x7f337013d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f3374949640 1 -- 172.21.15.40:0/3379446409 >> v1:172.21.15.40:6789/0 conn(0x7f337009ee10 legacy=0x7f33700a1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:52.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f5a0a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:52.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f5a0a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:52.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f5a0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:52.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f5a0a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:52.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f5a0a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:52.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f5a0a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:52.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f5a0b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:52.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f5a0b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:52.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f5a0b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:52.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f5a0a7fc640 10 monclient(hunting): handle_auth_done global_id 4237 payload 274 2024-04-22T08:12:52.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f5a0a7fc640 1 -- 172.21.15.40:0/2700847048 >> v2:172.21.15.40:3300/0 conn(0x7f5a0c13b0e0 msgr2=0x7f5a0c13d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:52.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f5a0a7fc640 1 --2- 172.21.15.40:0/2700847048 >> v2:172.21.15.40:3300/0 conn(0x7f5a0c13b0e0 0x7f5a0c13d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f5a0a7fc640 1 -- 172.21.15.40:0/2700847048 >> v1:172.21.15.40:6789/0 conn(0x7f5a0c09ee10 legacy=0x7f5a0c0a1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:52.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f3374949640 10 monclient: _finish_hunting 0 2024-04-22T08:12:52.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f3374949640 1 monclient: found mon.noname-b 2024-04-22T08:12:52.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f3374949640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:52.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f3374949640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:12:52.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f3374949640 1 -- 172.21.15.40:0/3379446409 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f337009fd30 con 0x7f33700a1f30 2024-04-22T08:12:52.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f5a0a7fc640 10 monclient: _finish_hunting 0 2024-04-22T08:12:52.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f5a0a7fc640 1 monclient: found mon.noname-c 2024-04-22T08:12:52.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f5a0a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:52.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f5a0a7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.189:3300/0 2024-04-22T08:12:52.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.320+0000 7f5a0a7fc640 1 -- 172.21.15.40:0/2700847048 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a0c0a0010 con 0x7f5a0c0a1f30 2024-04-22T08:12:52.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.324+0000 7f3374949640 1 -- 172.21.15.40:0/3379446409 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33700a1f30 msgr2=0x7f33700a3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:52.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.324+0000 7f3374949640 1 --2- 172.21.15.40:0/3379446409 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33700a1f30 0x7f33700a3240 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f33600097a0 tx=0x7f336002b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=939c500a2826a3d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:52.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.324+0000 7f3367fff640 1 -- 172.21.15.40:0/3379446409 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3360030920 con 0x7f33700a1f30 2024-04-22T08:12:52.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.324+0000 7f3367fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:52.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.324+0000 7f3367fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:12:52.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.324+0000 7f3367fff640 10 monclient: dump: 2024-04-22T08:12:52.335 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:52.335 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.335 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.335 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.335 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:52.335 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.335 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:52.335 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:52.335 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:52.335 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.324+0000 7f3367fff640 10 monclient: _finish_auth 0 2024-04-22T08:12:52.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.324+0000 7f3367fff640 10 monclient: _check_auth_tickets 2024-04-22T08:12:52.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.324+0000 7f3367fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:52.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.324+0000 7f33773ae640 5 monclient: authenticate success, global_id 4241 2024-04-22T08:12:52.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.324+0000 7f33773ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:12:52.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.324+0000 7f3367fff640 1 -- 172.21.15.40:0/3379446409 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f336002e050 con 0x7f33700a1f30 2024-04-22T08:12:52.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.324+0000 7f3367fff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:52.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.324+0000 7f33773ae640 10 monclient: get_monmap_and_config success 2024-04-22T08:12:52.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.324+0000 7f33773ae640 4 set_mon_vals no callback set 2024-04-22T08:12:52.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.324+0000 7f3367fff640 1 -- 172.21.15.40:0/3379446409 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3360030ca0 con 0x7f33700a1f30 2024-04-22T08:12:52.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.324+0000 7f3367fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:52.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.324+0000 7f3367fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:12:52.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.324+0000 7f3367fff640 10 monclient: dump: 2024-04-22T08:12:52.336 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:52.337 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.337 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.337 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.337 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:52.337 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.337 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:52.337 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:52.337 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:52.337 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.324+0000 7f5a0a7fc640 1 -- 172.21.15.40:0/2700847048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a0c0a1f30 msgr2=0x7f5a0c0a3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:52.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.324+0000 7f5a0a7fc640 1 --2- 172.21.15.40:0/2700847048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a0c0a1f30 0x7f5a0c0a3240 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c06c520 tx=0x7f59f402b730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df0f3dfe47249c5c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:52.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.324+0000 7f5a09ffb640 1 -- 172.21.15.40:0/2700847048 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f4030920 con 0x7f5a0c0a1f30 2024-04-22T08:12:52.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.324+0000 7f5a09ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:52.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.324+0000 7f5a09ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:12:52.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.324+0000 7f5a09ffb640 10 monclient: dump: 2024-04-22T08:12:52.339 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:52.339 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.339 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.339 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.339 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:52.339 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.339 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:52.339 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:52.339 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:52.339 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.324+0000 7f5a09ffb640 10 monclient: _finish_auth 0 2024-04-22T08:12:52.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.324+0000 7f5a09ffb640 10 monclient: _check_auth_tickets 2024-04-22T08:12:52.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.324+0000 7f5a09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:52.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.324+0000 7f5a09ffb640 1 -- 172.21.15.40:0/2700847048 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59f402e050 con 0x7f5a0c0a1f30 2024-04-22T08:12:52.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.324+0000 7f5a11870640 5 monclient: authenticate success, global_id 4237 2024-04-22T08:12:52.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.324+0000 7f5a11870640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:12:52.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.324+0000 7f5a09ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:52.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.324+0000 7f5a09ffb640 1 -- 172.21.15.40:0/2700847048 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f4030ca0 con 0x7f5a0c0a1f30 2024-04-22T08:12:52.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.324+0000 7f5a09ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:52.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.324+0000 7f5a09ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:12:52.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.324+0000 7f5a09ffb640 10 monclient: dump: 2024-04-22T08:12:52.340 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:52.340 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.340 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.340 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.340 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:52.341 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.341 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:52.341 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:52.341 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:52.341 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.325+0000 7f33773ae640 10 monclient: shutdown 2024-04-22T08:12:52.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.325+0000 7f33773ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:52.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.325+0000 7f33773ae640 1 -- 172.21.15.40:0/3379446409 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33700a1f30 msgr2=0x7f33700a3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.325+0000 7f33773ae640 1 --2- 172.21.15.40:0/3379446409 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33700a1f30 0x7f33700a3240 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f33600097a0 tx=0x7f336002b710 comp rx=0 tx=0).stop 2024-04-22T08:12:52.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.324+0000 7f5a11870640 10 monclient: get_monmap_and_config success 2024-04-22T08:12:52.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.324+0000 7f5a11870640 4 set_mon_vals no callback set 2024-04-22T08:12:52.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.325+0000 7f33773ae640 1 -- 172.21.15.40:0/3379446409 shutdown_connections 2024-04-22T08:12:52.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.325+0000 7f33773ae640 1 --2- 172.21.15.40:0/3379446409 >> v2:172.21.15.40:3300/0 conn(0x7f337013b0e0 0x7f337013d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.325+0000 7f33773ae640 1 --2- 172.21.15.40:0/3379446409 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33700a1f30 0x7f33700a3240 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.325+0000 7f33773ae640 1 -- 172.21.15.40:0/3379446409 >> 172.21.15.40:0/3379446409 conn(0x7f33700a4e80 msgr2=0x7f33700a5290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:52.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.325+0000 7f33773ae640 1 -- 172.21.15.40:0/3379446409 shutdown_connections 2024-04-22T08:12:52.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.325+0000 7f33773ae640 1 -- 172.21.15.40:0/3379446409 wait complete. 2024-04-22T08:12:52.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.325+0000 7f5a11870640 10 monclient: shutdown 2024-04-22T08:12:52.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.325+0000 7f5a11870640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:52.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.325+0000 7f5a11870640 1 -- 172.21.15.40:0/2700847048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a0c0a1f30 msgr2=0x7f5a0c0a3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.325+0000 7f5a11870640 1 --2- 172.21.15.40:0/2700847048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a0c0a1f30 0x7f5a0c0a3240 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c06c520 tx=0x7f59f402b730 comp rx=0 tx=0).stop 2024-04-22T08:12:52.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.325+0000 7f5a11870640 1 -- 172.21.15.40:0/2700847048 shutdown_connections 2024-04-22T08:12:52.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.325+0000 7f5a11870640 1 --2- 172.21.15.40:0/2700847048 >> v2:172.21.15.40:3300/0 conn(0x7f5a0c13b0e0 0x7f5a0c13d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.325+0000 7f5a11870640 1 --2- 172.21.15.40:0/2700847048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a0c0a1f30 0x7f5a0c0a3240 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.325+0000 7f5a11870640 1 -- 172.21.15.40:0/2700847048 >> 172.21.15.40:0/2700847048 conn(0x7f5a0c0a4e80 msgr2=0x7f5a0c0a5290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:52.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.325+0000 7f33773ae640 10 monclient: build_initial_monmap 2024-04-22T08:12:52.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.325+0000 7f33773ae640 10 monclient: monmap: 2024-04-22T08:12:52.345 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:52.345 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:12:52.345 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:12:52.345 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:12:52.345 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:52.345 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.345 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:12:52.345 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:12:52.345 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:12:52.345 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.325+0000 7f5a11870640 1 -- 172.21.15.40:0/2700847048 shutdown_connections 2024-04-22T08:12:52.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.326+0000 7f5a11870640 1 -- 172.21.15.40:0/2700847048 wait complete. 2024-04-22T08:12:52.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.326+0000 7f33773ae640 1 Processor -- start 2024-04-22T08:12:52.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.326+0000 7f33773ae640 1 -- start start 2024-04-22T08:12:52.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.326+0000 7f33773ae640 10 monclient: init 2024-04-22T08:12:52.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.326+0000 7f5a11870640 10 monclient: build_initial_monmap 2024-04-22T08:12:52.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.326+0000 7f5a11870640 10 monclient: monmap: 2024-04-22T08:12:52.347 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:52.347 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:12:52.347 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:12:52.347 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:12:52.347 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:52.347 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.347 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:12:52.347 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:12:52.347 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:12:52.348 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.326+0000 7f33773ae640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:52.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.326+0000 7f33773ae640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-22T08:12:52.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.326+0000 7f33773ae640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f337009ee10 0x7f3370139bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.326+0000 7f33773ae640 10 monclient(hunting): picked mon.noname-c con 0x7f337009ee10 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:52.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.327+0000 7f33773ae640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33700a1f30 0x7f3370120620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.327+0000 7f33773ae640 10 monclient(hunting): picked mon.noname-a con 0x7f33700a1f30 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:52.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.327+0000 7f33773ae640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f337013b0e0 0x7f3370120b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.327+0000 7f337514a640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f337009ee10 0x7f3370139bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.327+0000 7f33773ae640 10 monclient(hunting): picked mon.noname-b con 0x7f337013b0e0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:52.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.327+0000 7f33773ae640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.327+0000 7f33773ae640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33701cb740 con 0x7f33700a1f30 2024-04-22T08:12:52.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.327+0000 7f33773ae640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.327+0000 7f33773ae640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33700a4260 con 0x7f337013b0e0 2024-04-22T08:12:52.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.327+0000 7f33773ae640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.327+0000 7f33773ae640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f337013d040 con 0x7f337009ee10 2024-04-22T08:12:52.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.327+0000 7f337514a640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f337009ee10 0x7f3370139bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49706/0 (socket says 172.21.15.40:49706) 2024-04-22T08:12:52.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.327+0000 7f337514a640 1 -- 172.21.15.40:0/958802222 learned_addr learned my addr 172.21.15.40:0/958802222 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:52.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.327+0000 7f33773ae640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:52.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.327+0000 7f33773ae640 10 monclient(hunting): authenticate will time out at 846.320932s 2024-04-22T08:12:52.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.327+0000 7f337514a640 10 monclient(hunting): get_auth_request con 0x7f337009ee10 auth_method 0 2024-04-22T08:12:52.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.327+0000 7f337514a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:52.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.327+0000 7f337514a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:52.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.327+0000 7f337514a640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:52.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.327+0000 7f5a11870640 1 Processor -- start 2024-04-22T08:12:52.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.327+0000 7f337594b640 1 --2- 172.21.15.40:0/958802222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f337013b0e0 0x7f3370120b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.327+0000 7f337514a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:52.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.327+0000 7f337514a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:52.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.327+0000 7f337514a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:52.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.327+0000 7f5a11870640 1 -- start start 2024-04-22T08:12:52.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.327+0000 7f5a11870640 10 monclient: init 2024-04-22T08:12:52.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.327+0000 7f337594b640 10 monclient(hunting): get_auth_request con 0x7f337013b0e0 auth_method 0 2024-04-22T08:12:52.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.327+0000 7f337594b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:52.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.327+0000 7f337594b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:52.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.327+0000 7f337594b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:52.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.327+0000 7f337594b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:52.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.327+0000 7f337594b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:52.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.327+0000 7f337594b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:52.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.327+0000 7f337514a640 10 monclient(hunting): handle_auth_done global_id 4244 payload 824 2024-04-22T08:12:52.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.327+0000 7f337514a640 1 -- 172.21.15.40:0/958802222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f337013b0e0 msgr2=0x7f3370120b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.327+0000 7f337514a640 1 --2- 172.21.15.40:0/958802222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f337013b0e0 0x7f3370120b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.327+0000 7f337514a640 1 -- 172.21.15.40:0/958802222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33700a1f30 msgr2=0x7f3370120620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.327+0000 7f337514a640 1 --2- 172.21.15.40:0/958802222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33700a1f30 0x7f3370120620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.327+0000 7f337514a640 10 monclient: _finish_hunting 0 2024-04-22T08:12:52.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.327+0000 7f337514a640 1 monclient: found mon.noname-c 2024-04-22T08:12:52.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.328+0000 7f5a11870640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:52.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.328+0000 7f5a11870640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:12:52.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.328+0000 7f5a11870640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a0c09ee10 0x7f5a0c1cf070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.328+0000 7f5a11870640 10 monclient(hunting): picked mon.noname-a con 0x7f5a0c09ee10 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:52.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.328+0000 7f5a11870640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a0c0a1f30 0x7f5a0c13a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.328+0000 7f5a11870640 10 monclient(hunting): picked mon.noname-b con 0x7f5a0c0a1f30 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:52.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.328+0000 7f5a11870640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a0c13b0e0 0x7f5a0c13a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.328+0000 7f5a11870640 10 monclient(hunting): picked mon.noname-c con 0x7f5a0c13b0e0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:52.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.328+0000 7f5a11870640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.328+0000 7f5a11870640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0c133fe0 con 0x7f5a0c09ee10 2024-04-22T08:12:52.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.328+0000 7f5a11870640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.328+0000 7f5a11870640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0c13cfb0 con 0x7f5a0c0a1f30 2024-04-22T08:12:52.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.328+0000 7f5a11870640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.328+0000 7f5a11870640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0c134280 con 0x7f5a0c13b0e0 2024-04-22T08:12:52.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.328+0000 7f5a11870640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:52.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.328+0000 7f5a11870640 10 monclient(hunting): authenticate will time out at 846.322084s 2024-04-22T08:12:52.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.327+0000 7f337514a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:52.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.327+0000 7f337514a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:12:52.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.327+0000 7f337514a640 1 -- 172.21.15.40:0/958802222 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3360009400 con 0x7f337009ee10 2024-04-22T08:12:52.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.328+0000 7f5a0a7fc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a0c0a1f30 0x7f5a0c13a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.328+0000 7f5a0affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a0c09ee10 0x7f5a0c1cf070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.328+0000 7f5a0a7fc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a0c0a1f30 0x7f5a0c13a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.40:42738/0 (socket says 172.21.15.40:42738) 2024-04-22T08:12:52.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.328+0000 7f5a0a7fc640 1 -- 172.21.15.40:0/4245641914 learned_addr learned my addr 172.21.15.40:0/4245641914 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:52.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.328+0000 7f5a0affd640 10 monclient(hunting): get_auth_request con 0x7f5a0c09ee10 auth_method 0 2024-04-22T08:12:52.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.328+0000 7f5a0affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:52.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.328+0000 7f5a0affd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:52.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.328+0000 7f5a0affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:52.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.328+0000 7f5a0a7fc640 10 monclient(hunting): get_auth_request con 0x7f5a0c0a1f30 auth_method 0 2024-04-22T08:12:52.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.328+0000 7f5a0a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:52.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.328+0000 7f5a0a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:52.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.328+0000 7f5a0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:52.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.328+0000 7f5a0affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:52.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.328+0000 7f5a0affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:52.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.328+0000 7f5a0affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:52.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.328+0000 7f5a0a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:52.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.328+0000 7f5a0a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:52.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.328+0000 7f5a0a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:52.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.328+0000 7f5a0affd640 10 monclient(hunting): handle_auth_done global_id 4266 payload 824 2024-04-22T08:12:52.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.329+0000 7f5a0affd640 1 -- 172.21.15.40:0/4245641914 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a0c13b0e0 msgr2=0x7f5a0c13a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.329+0000 7f5a0affd640 1 --2- 172.21.15.40:0/4245641914 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a0c13b0e0 0x7f5a0c13a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.329+0000 7f5a0affd640 1 -- 172.21.15.40:0/4245641914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a0c0a1f30 msgr2=0x7f5a0c13a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.329+0000 7f5a0affd640 1 --2- 172.21.15.40:0/4245641914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a0c0a1f30 0x7f5a0c13a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.329+0000 7f5a0affd640 10 monclient: _finish_hunting 0 2024-04-22T08:12:52.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.329+0000 7f5a0affd640 1 monclient: found mon.noname-a 2024-04-22T08:12:52.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.329+0000 7f5a0affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:52.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.329+0000 7f5a0affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:12:52.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.329+0000 7f5a0affd640 1 -- 172.21.15.40:0/4245641914 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59f4009400 con 0x7f5a0c09ee10 2024-04-22T08:12:52.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.329+0000 7f5a0a7fc640 1 --2- 172.21.15.40:0/4245641914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a0c0a1f30 0x7f5a0c13a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:12:52.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.330+0000 7f337514a640 1 -- 172.21.15.40:0/958802222 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f337009ee10 msgr2=0x7f3370139bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:52.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.330+0000 7f337514a640 1 --2- 172.21.15.40:0/958802222 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f337009ee10 0x7f3370139bf0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f336c00c410 tx=0x7f336c00c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:52.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.330+0000 7f3365ffb640 1 -- 172.21.15.40:0/958802222 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f336c0037f0 con 0x7f337009ee10 2024-04-22T08:12:52.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.330+0000 7f3365ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:52.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.330+0000 7f3365ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:12:52.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.330+0000 7f3365ffb640 10 monclient: dump: 2024-04-22T08:12:52.363 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:52.363 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.363 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.363 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.363 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:52.363 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.363 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:52.363 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:52.363 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:52.363 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.330+0000 7f3365ffb640 10 monclient: _finish_auth 0 2024-04-22T08:12:52.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.330+0000 7f3365ffb640 10 monclient: _check_auth_tickets 2024-04-22T08:12:52.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.330+0000 7f3365ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:52.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.330+0000 7f3365ffb640 1 -- 172.21.15.40:0/958802222 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f336c003950 con 0x7f337009ee10 2024-04-22T08:12:52.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.330+0000 7f33773ae640 5 monclient: authenticate success, global_id 4244 2024-04-22T08:12:52.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.330+0000 7f3365ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:52.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.330+0000 7f33773ae640 10 monclient: _renew_subs 2024-04-22T08:12:52.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.330+0000 7f3365ffb640 1 -- 172.21.15.40:0/958802222 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f336c003d00 con 0x7f337009ee10 2024-04-22T08:12:52.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.330+0000 7f33773ae640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:52.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.330+0000 7f33773ae640 1 -- 172.21.15.40:0/958802222 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3370122220 con 0x7f337009ee10 2024-04-22T08:12:52.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.330+0000 7f33677fe640 4 set_mon_vals no callback set 2024-04-22T08:12:52.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.330+0000 7f33773ae640 10 monclient: _renew_subs 2024-04-22T08:12:52.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.330+0000 7f33773ae640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:52.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.330+0000 7f33773ae640 1 -- 172.21.15.40:0/958802222 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3370123ce0 con 0x7f337009ee10 2024-04-22T08:12:52.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.330+0000 7f3365ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:52.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.330+0000 7f3365ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:12:52.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.330+0000 7f3365ffb640 10 monclient: dump: 2024-04-22T08:12:52.365 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:52.365 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.365 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.365 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.365 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:52.365 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.365 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:52.365 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:52.365 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:52.365 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.331+0000 7f5a0affd640 1 -- 172.21.15.40:0/4245641914 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a0c09ee10 msgr2=0x7f5a0c1cf070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:52.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.331+0000 7f5a0affd640 1 --2- 172.21.15.40:0/4245641914 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a0c09ee10 0x7f5a0c1cf070 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f59fc00c360 tx=0x7f59fc00c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:52.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.331+0000 7f5a03fff640 1 -- 172.21.15.40:0/4245641914 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59fc0070d0 con 0x7f5a0c09ee10 2024-04-22T08:12:52.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.331+0000 7f5a03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:52.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.331+0000 7f5a03fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:12:52.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.331+0000 7f5a03fff640 10 monclient: dump: 2024-04-22T08:12:52.366 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:52.366 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.366 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.366 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.367 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:52.367 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.367 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:52.367 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:52.367 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:52.367 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.331+0000 7f5a03fff640 10 monclient: _finish_auth 0 2024-04-22T08:12:52.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.331+0000 7f5a03fff640 10 monclient: _check_auth_tickets 2024-04-22T08:12:52.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.331+0000 7f5a03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:52.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.331+0000 7f5a03fff640 1 -- 172.21.15.40:0/4245641914 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59fc007230 con 0x7f5a0c09ee10 2024-04-22T08:12:52.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.331+0000 7f5a03fff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:52.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.331+0000 7f5a03fff640 1 -- 172.21.15.40:0/4245641914 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59fc0075e0 con 0x7f5a0c09ee10 2024-04-22T08:12:52.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.331+0000 7f5a03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:52.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.331+0000 7f5a03fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:12:52.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.331+0000 7f5a03fff640 10 monclient: dump: 2024-04-22T08:12:52.368 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:52.368 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.368 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.368 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.368 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:52.368 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.368 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:52.368 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:52.368 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:52.368 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.331+0000 7f5a097fa640 4 set_mon_vals no callback set 2024-04-22T08:12:52.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.331+0000 7f3365ffb640 1 -- 172.21.15.40:0/958802222 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 4) v1 ==== 44726+0+0 (secure 0 0 0) 0x7f336c01a360 con 0x7f337009ee10 2024-04-22T08:12:52.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.331+0000 7f3365ffb640 1 --2- 172.21.15.40:0/958802222 >> v2:172.21.15.189:6800/30168 conn(0x7f33540377a0 0x7f3354039c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.331+0000 7f3365ffb640 1 -- 172.21.15.40:0/958802222 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f336c0493b0 con 0x7f337009ee10 2024-04-22T08:12:52.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.332+0000 7f33773ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:52.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.332+0000 7f33773ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:52.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.332+0000 7f33773ae640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:52.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.332+0000 7f33773ae640 1 -- 172.21.15.40:0/958802222 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f337009eba0 con 0x7f337009ee10 2024-04-22T08:12:52.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.332+0000 7f5a11870640 5 monclient: authenticate success, global_id 4266 2024-04-22T08:12:52.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.332+0000 7f5a11870640 10 monclient: _renew_subs 2024-04-22T08:12:52.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.332+0000 7f5a11870640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:52.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.332+0000 7f5a11870640 1 -- 172.21.15.40:0/4245641914 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a0c134560 con 0x7f5a0c09ee10 2024-04-22T08:12:52.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.332+0000 7f5a11870640 10 monclient: _renew_subs 2024-04-22T08:12:52.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.332+0000 7f5a11870640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:52.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.332+0000 7f5a11870640 1 -- 172.21.15.40:0/4245641914 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a0c071050 con 0x7f5a0c09ee10 2024-04-22T08:12:52.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.332+0000 7f3374949640 1 --2- 172.21.15.40:0/958802222 >> v2:172.21.15.189:6800/30168 conn(0x7f33540377a0 0x7f3354039c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.332+0000 7f3374949640 10 monclient: get_auth_request con 0x7f33540377a0 auth_method 0 2024-04-22T08:12:52.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.333+0000 7f5a11870640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:52.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.333+0000 7f5a11870640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:52.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.333+0000 7f5a11870640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:52.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.333+0000 7f5a11870640 1 -- 172.21.15.40:0/4245641914 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a0c134960 con 0x7f5a0c09ee10 2024-04-22T08:12:52.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.333+0000 7f5a03fff640 1 -- 172.21.15.40:0/4245641914 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 4) v1 ==== 44726+0+0 (secure 0 0 0) 0x7f59fc01b3b0 con 0x7f5a0c09ee10 2024-04-22T08:12:52.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.333+0000 7f5a03fff640 1 --2- 172.21.15.40:0/4245641914 >> v2:172.21.15.189:6800/30168 conn(0x7f59ec037740 0x7f59ec039c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.334+0000 7f5a03fff640 1 -- 172.21.15.40:0/4245641914 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f59fc049720 con 0x7f5a0c09ee10 2024-04-22T08:12:52.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.335+0000 7f3374949640 1 --2- 172.21.15.40:0/958802222 >> v2:172.21.15.189:6800/30168 conn(0x7f33540377a0 0x7f3354039c60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3360032000 tx=0x7f336002d000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:52.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.335+0000 7f5a0a7fc640 1 --2- 172.21.15.40:0/4245641914 >> v2:172.21.15.189:6800/30168 conn(0x7f59ec037740 0x7f59ec039c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.335+0000 7f5a0a7fc640 10 monclient: get_auth_request con 0x7f59ec037740 auth_method 0 2024-04-22T08:12:52.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.338+0000 7f5a0a7fc640 1 --2- 172.21.15.40:0/4245641914 >> v2:172.21.15.189:6800/30168 conn(0x7f59ec037740 0x7f59ec039c00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c0a35b0 tx=0x7f59f4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:52.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.342+0000 7f3365ffb640 1 -- 172.21.15.40:0/958802222 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f336c017070 con 0x7f337009ee10 2024-04-22T08:12:52.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.342+0000 7f3365ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:52.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.342+0000 7f3365ffb640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:12:52.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.343+0000 7f5a03fff640 1 -- 172.21.15.40:0/4245641914 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f59fc01a360 con 0x7f5a0c09ee10 2024-04-22T08:12:52.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.343+0000 7f5a03fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:52.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.343+0000 7f5a03fff640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:12:52.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.359+0000 7f1c8876c640 10 monclient: get_monmap_and_config 2024-04-22T08:12:52.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.359+0000 7f1c8876c640 10 monclient: build_initial_monmap 2024-04-22T08:12:52.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.359+0000 7f1c8876c640 10 monclient: monmap: 2024-04-22T08:12:52.376 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:52.376 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.376 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:52.359893+0000 2024-04-22T08:12:52.376 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:52.359893+0000 2024-04-22T08:12:52.376 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:52.376 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.376 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:12:52.376 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:12:52.376 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:12:52.376 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:12:52.377 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:12:52.377 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.359+0000 7f1c8876c640 1 Processor -- start 2024-04-22T08:12:52.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.359+0000 7f1c8876c640 1 -- start start 2024-04-22T08:12:52.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.360+0000 7f1c8876c640 10 monclient: init 2024-04-22T08:12:52.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.361+0000 7f1c8876c640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:52.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.361+0000 7f1c8876c640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-04-22T08:12:52.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.361+0000 7f1c8876c640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1c8009ee10 0x7f1c800a3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.361+0000 7f1c8876c640 10 monclient(hunting): picked mon.noname-a con 0x7f1c8009ee10 addr v2:172.21.15.40:3300/0 2024-04-22T08:12:52.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.361+0000 7f1c8876c640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1c800a1820 0x7f1c800a1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.361+0000 7f1c8876c640 10 monclient(hunting): picked mon.noname-c con 0x7f1c800a1820 addr v2:172.21.15.189:3300/0 2024-04-22T08:12:52.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.361+0000 7f1c8876c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1c80135dc0 addr v1:172.21.15.189:6789/0 2024-04-22T08:12:52.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.361+0000 7f1c85d07640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1c800a1820 0x7f1c800a1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.361+0000 7f1c85d07640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1c800a1820 0x7f1c800a1c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.40:42752/0 (socket says 172.21.15.40:42752) 2024-04-22T08:12:52.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.361+0000 7f1c85d07640 1 -- 172.21.15.40:0/1687129872 learned_addr learned my addr 172.21.15.40:0/1687129872 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:52.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.361+0000 7f1c8876c640 1 -- 172.21.15.40:0/1687129872 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c800a4470 con 0x7f1c80135dc0 2024-04-22T08:12:52.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.361+0000 7f1c8876c640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.361+0000 7f1c8876c640 1 -- 172.21.15.40:0/1687129872 --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c8009f1f0 con 0x7f1c8009ee10 2024-04-22T08:12:52.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.361+0000 7f1c8876c640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.361+0000 7f1c8876c640 1 -- 172.21.15.40:0/1687129872 --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c8009fd30 con 0x7f1c800a1820 2024-04-22T08:12:52.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.361+0000 7f1c8876c640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:52.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.361+0000 7f1c8876c640 10 monclient(hunting): authenticate will time out at 846.355396s 2024-04-22T08:12:52.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.361+0000 7f1c85d07640 10 monclient(hunting): get_auth_request con 0x7f1c800a1820 auth_method 0 2024-04-22T08:12:52.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.361+0000 7f1c85d07640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:52.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.361+0000 7f1c85d07640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:52.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.361+0000 7f1c85d07640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:52.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.361+0000 7f1c86508640 1 --2- 172.21.15.40:0/1687129872 >> v2:172.21.15.40:3300/0 conn(0x7f1c8009ee10 0x7f1c800a3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.361+0000 7f1c85d07640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:52.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.361+0000 7f1c85d07640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:52.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.361+0000 7f1c85d07640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:52.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.361+0000 7f1c86508640 10 monclient(hunting): get_auth_request con 0x7f1c8009ee10 auth_method 0 2024-04-22T08:12:52.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.361+0000 7f1c86508640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:52.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.361+0000 7f1c86508640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:52.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.361+0000 7f1c86508640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:52.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.361+0000 7f1c85d07640 10 monclient(hunting): handle_auth_done global_id 4246 payload 274 2024-04-22T08:12:52.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.361+0000 7f1c85d07640 1 -- 172.21.15.40:0/1687129872 >> v2:172.21.15.40:3300/0 conn(0x7f1c8009ee10 msgr2=0x7f1c800a3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:52.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.361+0000 7f1c85d07640 1 --2- 172.21.15.40:0/1687129872 >> v2:172.21.15.40:3300/0 conn(0x7f1c8009ee10 0x7f1c800a3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.361+0000 7f1c85d07640 1 -- 172.21.15.40:0/1687129872 >> v1:172.21.15.189:6789/0 conn(0x7f1c80135dc0 legacy=0x7f1c80136210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:52.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.361+0000 7f1c85d07640 10 monclient: _finish_hunting 0 2024-04-22T08:12:52.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.361+0000 7f1c85d07640 1 monclient: found mon.noname-c 2024-04-22T08:12:52.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.361+0000 7f1c85d07640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:52.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.361+0000 7f1c85d07640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.189:3300/0 2024-04-22T08:12:52.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.361+0000 7f1c85d07640 1 -- 172.21.15.40:0/1687129872 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c800a0010 con 0x7f1c800a1820 2024-04-22T08:12:52.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.363+0000 7efd37270640 10 monclient: get_monmap_and_config 2024-04-22T08:12:52.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.363+0000 7efd37270640 10 monclient: build_initial_monmap 2024-04-22T08:12:52.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.364+0000 7efd37270640 10 monclient: monmap: 2024-04-22T08:12:52.380 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:52.380 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.380 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:52.364787+0000 2024-04-22T08:12:52.381 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:52.364787+0000 2024-04-22T08:12:52.381 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:52.381 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.381 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:12:52.381 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:12:52.381 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:12:52.381 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:12:52.381 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:12:52.381 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.364+0000 7efd37270640 1 Processor -- start 2024-04-22T08:12:52.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.365+0000 7f08a03f5640 10 monclient: get_monmap_and_config 2024-04-22T08:12:52.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.365+0000 7f08a03f5640 10 monclient: build_initial_monmap 2024-04-22T08:12:52.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.365+0000 7f08a03f5640 10 monclient: monmap: 2024-04-22T08:12:52.382 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:52.382 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.382 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:52.365818+0000 2024-04-22T08:12:52.382 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:52.365818+0000 2024-04-22T08:12:52.382 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:52.382 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.382 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:12:52.382 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:12:52.382 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:12:52.382 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:12:52.383 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:12:52.383 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.365+0000 7efd37270640 1 -- start start 2024-04-22T08:12:52.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.365+0000 7efd37270640 10 monclient: init 2024-04-22T08:12:52.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.365+0000 7efd37270640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:52.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.365+0000 7efd37270640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-04-22T08:12:52.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.365+0000 7efd37270640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efd3009ee10 addr v1:172.21.15.189:6789/0 2024-04-22T08:12:52.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.365+0000 7efd37270640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efd300a0d20 0x7efd300a1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.365+0000 7f1c85d07640 1 -- 172.21.15.40:0/1687129872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c800a1820 msgr2=0x7f1c800a1c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:52.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.365+0000 7f1c85d07640 1 --2- 172.21.15.40:0/1687129872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c800a1820 0x7f1c800a1c30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1c8006c520 tx=0x7f1c7002b7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4b2b1ca6ca56eb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:52.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.365+0000 7efd37270640 10 monclient(hunting): picked mon.noname-a con 0x7efd300a0d20 addr v2:172.21.15.40:3300/0 2024-04-22T08:12:52.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.365+0000 7efd37270640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efd3013f550 addr v1:172.21.15.40:6789/0 2024-04-22T08:12:52.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.365+0000 7efd37270640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd300a3d90 con 0x7efd3013f550 2024-04-22T08:12:52.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.365+0000 7efd37270640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd3009f1f0 con 0x7efd3009ee10 2024-04-22T08:12:52.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.365+0000 7efd37270640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.365+0000 7f08a03f5640 1 Processor -- start 2024-04-22T08:12:52.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.365+0000 7efd37270640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd3009fd30 con 0x7efd300a0d20 2024-04-22T08:12:52.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.365+0000 7efd37270640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:52.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.365+0000 7efd37270640 10 monclient(hunting): authenticate will time out at 846.359299s 2024-04-22T08:12:52.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.365+0000 7efd3480b640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efd300a0d20 0x7efd300a1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.365+0000 7efd3480b640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efd300a0d20 0x7efd300a1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57934/0 (socket says 172.21.15.40:57934) 2024-04-22T08:12:52.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.365+0000 7efd3480b640 1 -- 172.21.15.40:0/3990490466 learned_addr learned my addr 172.21.15.40:0/3990490466 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:52.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.365+0000 7efd3480b640 10 monclient(hunting): get_auth_request con 0x7efd300a0d20 auth_method 0 2024-04-22T08:12:52.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.365+0000 7efd3480b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:52.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.365+0000 7efd3480b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:52.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.365+0000 7efd3480b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:52.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.365+0000 7efd3480b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:52.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.365+0000 7efd3480b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:52.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.365+0000 7efd3480b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:52.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.366+0000 7efd3480b640 10 monclient(hunting): handle_auth_done global_id 4272 payload 274 2024-04-22T08:12:52.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.366+0000 7f1c85506640 1 -- 172.21.15.40:0/1687129872 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c70031960 con 0x7f1c800a1820 2024-04-22T08:12:52.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.366+0000 7f1c85506640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:52.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.366+0000 7f1c85506640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:12:52.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.366+0000 7f1c85506640 10 monclient: dump: 2024-04-22T08:12:52.388 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:52.388 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.388 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.388 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.388 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:52.388 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.388 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:52.388 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:52.388 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:52.388 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.366+0000 7f1c85506640 10 monclient: _finish_auth 0 2024-04-22T08:12:52.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.366+0000 7f1c85506640 10 monclient: _check_auth_tickets 2024-04-22T08:12:52.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.366+0000 7f1c85506640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:52.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.366+0000 7f1c85506640 1 -- 172.21.15.40:0/1687129872 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c70031ac0 con 0x7f1c800a1820 2024-04-22T08:12:52.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.366+0000 7f1c85506640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:52.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.366+0000 7f1c85506640 1 -- 172.21.15.40:0/1687129872 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c70031e50 con 0x7f1c800a1820 2024-04-22T08:12:52.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.366+0000 7f1c85506640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:52.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.366+0000 7f31de6fc640 10 monclient: get_monmap_and_config 2024-04-22T08:12:52.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.366+0000 7f31de6fc640 10 monclient: build_initial_monmap 2024-04-22T08:12:52.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.366+0000 7f1c85506640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:12:52.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.366+0000 7f1c85506640 10 monclient: dump: 2024-04-22T08:12:52.390 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:52.390 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.390 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.390 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.390 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:52.390 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.390 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:52.390 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:52.390 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:52.390 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.366+0000 7f1c8876c640 5 monclient: authenticate success, global_id 4246 2024-04-22T08:12:52.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.366+0000 7f1c8876c640 10 monclient: get_monmap_and_config success 2024-04-22T08:12:52.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.366+0000 7f1c8876c640 4 set_mon_vals no callback set 2024-04-22T08:12:52.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.366+0000 7f31de6fc640 10 monclient: monmap: 2024-04-22T08:12:52.391 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:52.391 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.391 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:52.366979+0000 2024-04-22T08:12:52.391 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:52.366979+0000 2024-04-22T08:12:52.391 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:52.391 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.391 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:12:52.391 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:12:52.391 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:12:52.392 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:12:52.392 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:12:52.392 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.366+0000 7efd3480b640 1 -- 172.21.15.40:0/3990490466 >> v1:172.21.15.189:6789/0 conn(0x7efd3009ee10 legacy=0x7efd300a1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:52.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.366+0000 7efd3480b640 1 -- 172.21.15.40:0/3990490466 >> v1:172.21.15.40:6789/0 conn(0x7efd3013f550 legacy=0x7efd30141940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:52.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.366+0000 7efd3480b640 10 monclient: _finish_hunting 0 2024-04-22T08:12:52.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.366+0000 7efd3480b640 1 monclient: found mon.noname-a 2024-04-22T08:12:52.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.366+0000 7efd3480b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:52.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.366+0000 7efd3480b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:12:52.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.366+0000 7efd3480b640 1 -- 172.21.15.40:0/3990490466 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd300a27f0 con 0x7efd300a0d20 2024-04-22T08:12:52.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.366+0000 7f1c8876c640 10 monclient: shutdown 2024-04-22T08:12:52.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.366+0000 7f1c8876c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:52.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.366+0000 7f1c8876c640 1 -- 172.21.15.40:0/1687129872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c800a1820 msgr2=0x7f1c800a1c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.366+0000 7f1c8876c640 1 --2- 172.21.15.40:0/1687129872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c800a1820 0x7f1c800a1c30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1c8006c520 tx=0x7f1c7002b7f0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.367+0000 7f1c8876c640 1 -- 172.21.15.40:0/1687129872 shutdown_connections 2024-04-22T08:12:52.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.367+0000 7f1c8876c640 1 --2- 172.21.15.40:0/1687129872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c800a1820 0x7f1c800a1c30 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.367+0000 7f1c8876c640 1 --2- 172.21.15.40:0/1687129872 >> v2:172.21.15.40:3300/0 conn(0x7f1c8009ee10 0x7f1c800a3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.367+0000 7f1c8876c640 1 -- 172.21.15.40:0/1687129872 >> 172.21.15.40:0/1687129872 conn(0x7f1c800a4e80 msgr2=0x7f1c800a5290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:52.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.366+0000 7f08a03f5640 1 -- start start 2024-04-22T08:12:52.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.366+0000 7f08a03f5640 10 monclient: init 2024-04-22T08:12:52.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.367+0000 7f08a03f5640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:52.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.367+0000 7f08a03f5640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-04-22T08:12:52.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.367+0000 7f08a03f5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f089809d710 addr v1:172.21.15.189:6789/0 2024-04-22T08:12:52.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.367+0000 7f08a03f5640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f089809ef20 0x7f089813f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.367+0000 7f08a03f5640 10 monclient(hunting): picked mon.noname-b con 0x7f089809ef20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:52.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.367+0000 7f08a03f5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f089809e200 addr v1:172.21.15.40:6789/0 2024-04-22T08:12:52.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.367+0000 7f08a03f5640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0898099a60 con 0x7f089809e200 2024-04-22T08:12:52.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.367+0000 7f08a03f5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08980a0020 con 0x7f089809d710 2024-04-22T08:12:52.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.367+0000 7f08a03f5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.367+0000 7f08a03f5640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f089809fd40 con 0x7f089809ef20 2024-04-22T08:12:52.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.367+0000 7f08a03f5640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:52.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.367+0000 7f08a03f5640 10 monclient(hunting): authenticate will time out at 846.361053s 2024-04-22T08:12:52.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.367+0000 7f31de6fc640 1 Processor -- start 2024-04-22T08:12:52.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.367+0000 7f1c8876c640 1 -- 172.21.15.40:0/1687129872 shutdown_connections 2024-04-22T08:12:52.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.367+0000 7f31de6fc640 1 -- start start 2024-04-22T08:12:52.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.367+0000 7f31de6fc640 10 monclient: init 2024-04-22T08:12:52.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.367+0000 7f1c8876c640 1 -- 172.21.15.40:0/1687129872 wait complete. 2024-04-22T08:12:52.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.367+0000 7f089e191640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f089809d710 0x7f089809daf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.40:43642/0 (socket says 172.21.15.40:43642) 2024-04-22T08:12:52.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.367+0000 7f089e992640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f089809e200 0x7f0898141b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:38920/0 (socket says 172.21.15.40:38920) 2024-04-22T08:12:52.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.367+0000 7f089d990640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f089809ef20 0x7f089813f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.367+0000 7f089e191640 1 -- 172.21.15.40:0/2156347506 learned_addr learned my addr 172.21.15.40:0/2156347506 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:12:52.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.367+0000 7f089d990640 10 monclient(hunting): get_auth_request con 0x7f089809ef20 auth_method 0 2024-04-22T08:12:52.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.367+0000 7f089d990640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:52.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.367+0000 7f089d990640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:52.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.367+0000 7f089d990640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:52.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.367+0000 7f089d990640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:52.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.367+0000 7f089d990640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:52.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.367+0000 7f089d990640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:52.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.368+0000 7f31de6fc640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:52.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.368+0000 7f31de6fc640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-04-22T08:12:52.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.368+0000 7f31de6fc640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31d809d710 0x7f31d809daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.368+0000 7f1c8876c640 10 monclient: build_initial_monmap 2024-04-22T08:12:52.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.368+0000 7f1c8876c640 10 monclient: monmap: 2024-04-22T08:12:52.400 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:52.400 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:12:52.400 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:12:52.400 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:12:52.400 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:52.400 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.400 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:12:52.400 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:12:52.400 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:12:52.400 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.367+0000 7f089d990640 10 monclient(hunting): handle_auth_done global_id 4247 payload 274 2024-04-22T08:12:52.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.368+0000 7f089d990640 1 -- 172.21.15.40:0/2156347506 >> v1:172.21.15.189:6789/0 conn(0x7f089809d710 legacy=0x7f089809daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:52.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.368+0000 7f089d990640 1 -- 172.21.15.40:0/2156347506 >> v1:172.21.15.40:6789/0 conn(0x7f089809e200 legacy=0x7f0898141b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:52.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.368+0000 7f089d990640 10 monclient: _finish_hunting 0 2024-04-22T08:12:52.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.368+0000 7f089d990640 1 monclient: found mon.noname-b 2024-04-22T08:12:52.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.368+0000 7f089d990640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:52.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.368+0000 7f089d990640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:12:52.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.368+0000 7f089d990640 1 -- 172.21.15.40:0/2156347506 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08980a0300 con 0x7f089809ef20 2024-04-22T08:12:52.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.368+0000 7f31d7fff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31d809d710 0x7f31d809daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.368+0000 7f31d7fff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31d809d710 0x7f31d809daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49730/0 (socket says 172.21.15.40:49730) 2024-04-22T08:12:52.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.368+0000 7f1c8876c640 1 Processor -- start 2024-04-22T08:12:52.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.368+0000 7f1c8876c640 1 -- start start 2024-04-22T08:12:52.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.368+0000 7f1c8876c640 10 monclient: init 2024-04-22T08:12:52.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.368+0000 7f31d7fff640 1 -- 172.21.15.40:0/2714820949 learned_addr learned my addr 172.21.15.40:0/2714820949 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:52.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.368+0000 7f31de6fc640 10 monclient(hunting): picked mon.noname-b con 0x7f31d809d710 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:52.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.368+0000 7f31de6fc640 1 --2- 172.21.15.40:0/2714820949 >> v2:172.21.15.189:3300/0 conn(0x7f31d809ef20 0x7f31d809e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f31de6fc640 10 monclient(hunting): picked mon.noname-c con 0x7f31d809ef20 addr v2:172.21.15.189:3300/0 2024-04-22T08:12:52.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f1c8876c640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:52.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f1c8876c640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-22T08:12:52.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f31de6fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f31d813e5b0 addr v1:172.21.15.40:6789/0 2024-04-22T08:12:52.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f31de6fc640 1 -- 172.21.15.40:0/2714820949 --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31d8099a60 con 0x7f31d813e5b0 2024-04-22T08:12:52.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f31de6fc640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f31de6fc640 1 -- 172.21.15.40:0/2714820949 --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31d809e570 con 0x7f31d809ef20 2024-04-22T08:12:52.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f31de6fc640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f31de6fc640 1 -- 172.21.15.40:0/2714820949 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31d80a0020 con 0x7f31d809d710 2024-04-22T08:12:52.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f31de6fc640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:52.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f1c8876c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c8009ee10 0x7f1c80133f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f1c8876c640 10 monclient(hunting): picked mon.noname-a con 0x7f1c8009ee10 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:52.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f1c8876c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1c800a1820 0x7f1c80134460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f1c8876c640 10 monclient(hunting): picked mon.noname-c con 0x7f1c800a1820 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:52.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f1c8876c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c80135dc0 0x7f1c8011e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f1c8876c640 10 monclient(hunting): picked mon.noname-b con 0x7f1c80135dc0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:52.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f1c8876c640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f1c8876c640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c80071c40 con 0x7f1c8009ee10 2024-04-22T08:12:52.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f1c8876c640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f1c8876c640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c800a7260 con 0x7f1c80135dc0 2024-04-22T08:12:52.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f1c8876c640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f31de6fc640 10 monclient(hunting): authenticate will time out at 846.363344s 2024-04-22T08:12:52.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f1c8876c640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c80121040 con 0x7f1c800a1820 2024-04-22T08:12:52.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f1c8876c640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:52.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f1c8876c640 10 monclient(hunting): authenticate will time out at 846.363309s 2024-04-22T08:12:52.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f1c85d07640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1c800a1820 0x7f1c80134460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f1c85d07640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1c800a1820 0x7f1c80134460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49736/0 (socket says 172.21.15.40:49736) 2024-04-22T08:12:52.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f1c85d07640 1 -- 172.21.15.40:0/4260163537 learned_addr learned my addr 172.21.15.40:0/4260163537 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:52.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f1c85d07640 10 monclient(hunting): get_auth_request con 0x7f1c800a1820 auth_method 0 2024-04-22T08:12:52.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f1c85d07640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:52.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f1c85d07640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:52.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f1c85d07640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:52.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f31d7fff640 10 monclient(hunting): get_auth_request con 0x7f31d809d710 auth_method 0 2024-04-22T08:12:52.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f31d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:52.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f31d7fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:52.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f31d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:52.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f1c85d07640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:52.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f1c85d07640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:52.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f31d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:52.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f31d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:52.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f31d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:52.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f31d7fff640 10 monclient(hunting): handle_auth_done global_id 4253 payload 274 2024-04-22T08:12:52.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f31d7fff640 1 -- 172.21.15.40:0/2714820949 >> v2:172.21.15.189:3300/0 conn(0x7f31d809ef20 msgr2=0x7f31d809e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:52.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f31d77fe640 1 --2- 172.21.15.40:0/2714820949 >> v2:172.21.15.189:3300/0 conn(0x7f31d809ef20 0x7f31d809e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f1c85d07640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:52.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f1c86d09640 1 --2- 172.21.15.40:0/4260163537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c80135dc0 0x7f1c8011e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f1c86d09640 10 monclient(hunting): get_auth_request con 0x7f1c80135dc0 auth_method 0 2024-04-22T08:12:52.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f1c86d09640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:52.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f1c86d09640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:52.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f1c86d09640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:52.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.370+0000 7f1c86d09640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:52.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.370+0000 7f1c86d09640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:52.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.370+0000 7f1c86d09640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:52.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.370+0000 7f1c85d07640 10 monclient(hunting): handle_auth_done global_id 4250 payload 824 2024-04-22T08:12:52.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.370+0000 7f1c85d07640 1 -- 172.21.15.40:0/4260163537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c80135dc0 msgr2=0x7f1c8011e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.370+0000 7f1c85d07640 1 --2- 172.21.15.40:0/4260163537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c80135dc0 0x7f1c8011e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.370+0000 7f1c85d07640 1 -- 172.21.15.40:0/4260163537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c8009ee10 msgr2=0x7f1c80133f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.370+0000 7f1c85d07640 1 --2- 172.21.15.40:0/4260163537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c8009ee10 0x7f1c80133f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.370+0000 7f1c85d07640 10 monclient: _finish_hunting 0 2024-04-22T08:12:52.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.370+0000 7f1c85d07640 1 monclient: found mon.noname-c 2024-04-22T08:12:52.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.370+0000 7f1c85d07640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:52.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.370+0000 7f1c85d07640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:12:52.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.370+0000 7f1c85d07640 1 -- 172.21.15.40:0/4260163537 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c700094d0 con 0x7f1c800a1820 2024-04-22T08:12:52.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.370+0000 7efd3480b640 1 -- 172.21.15.40:0/3990490466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd300a0d20 msgr2=0x7efd300a1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:52.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f31d7fff640 1 --2- 172.21.15.40:0/2714820949 >> v2:172.21.15.189:3300/0 conn(0x7f31d809ef20 0x7f31d809e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f31d7fff640 1 -- 172.21.15.40:0/2714820949 >> v1:172.21.15.40:6789/0 conn(0x7f31d813e5b0 legacy=0x7f31d81409a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:52.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f31d7fff640 10 monclient: _finish_hunting 0 2024-04-22T08:12:52.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.370+0000 7efd3480b640 1 --2- 172.21.15.40:0/3990490466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd300a0d20 0x7efd300a1e30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7efd20009870 tx=0x7efd2002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c3e0279e5339113 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:52.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f31d7fff640 1 monclient: found mon.noname-b 2024-04-22T08:12:52.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f31d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:52.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f31d7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:12:52.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f31d7fff640 1 -- 172.21.15.40:0/2714820949 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d809fd40 con 0x7f31d809d710 2024-04-22T08:12:52.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f31d77fe640 10 monclient: get_auth_request con 0x7f31d809ef20 auth_method 0 2024-04-22T08:12:52.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.369+0000 7f31d77fe640 1 --2- 172.21.15.40:0/2714820949 >> v2:172.21.15.189:3300/0 conn(0x7f31d809ef20 0x7f31d809e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T08:12:52.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.370+0000 7efd27fff640 1 -- 172.21.15.40:0/3990490466 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd20031a40 con 0x7efd300a0d20 2024-04-22T08:12:52.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.370+0000 7efd27fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:52.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.370+0000 7efd27fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:12:52.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.370+0000 7efd27fff640 10 monclient: dump: 2024-04-22T08:12:52.416 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:52.417 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.417 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.417 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.417 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:52.417 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.417 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:52.417 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:52.417 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:52.417 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.370+0000 7efd27fff640 10 monclient: _finish_auth 0 2024-04-22T08:12:52.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.370+0000 7efd27fff640 10 monclient: _check_auth_tickets 2024-04-22T08:12:52.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.370+0000 7efd27fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:52.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.370+0000 7efd37270640 5 monclient: authenticate success, global_id 4272 2024-04-22T08:12:52.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.370+0000 7efd37270640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:12:52.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.370+0000 7efd27fff640 1 -- 172.21.15.40:0/3990490466 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd20031ba0 con 0x7efd300a0d20 2024-04-22T08:12:52.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.370+0000 7efd27fff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:52.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.370+0000 7efd27fff640 1 -- 172.21.15.40:0/3990490466 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd20033440 con 0x7efd300a0d20 2024-04-22T08:12:52.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.370+0000 7efd27fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:52.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.370+0000 7efd27fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:12:52.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.370+0000 7efd27fff640 10 monclient: dump: 2024-04-22T08:12:52.418 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:52.418 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.418 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.418 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.418 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:52.418 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.419 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:52.419 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:52.419 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:52.419 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.370+0000 7efd37270640 10 monclient: get_monmap_and_config success 2024-04-22T08:12:52.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.370+0000 7efd37270640 4 set_mon_vals no callback set 2024-04-22T08:12:52.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.371+0000 7f089d990640 1 -- 172.21.15.40:0/2156347506 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f089809ef20 msgr2=0x7f089813f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:52.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.371+0000 7f089d990640 1 --2- 172.21.15.40:0/2156347506 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f089809ef20 0x7f089813f5c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f08940097a0 tx=0x7f089402b6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6498202fec734ba8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:52.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.371+0000 7f089d18f640 1 -- 172.21.15.40:0/2156347506 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0894030a00 con 0x7f089809ef20 2024-04-22T08:12:52.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.371+0000 7f089d18f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:52.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.371+0000 7f089d18f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:12:52.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.371+0000 7f089d18f640 10 monclient: dump: 2024-04-22T08:12:52.420 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:52.420 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.420 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.420 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.420 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:52.420 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.420 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:52.420 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:52.420 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:52.420 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.371+0000 7f089d18f640 10 monclient: _finish_auth 0 2024-04-22T08:12:52.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.371+0000 7f089d18f640 10 monclient: _check_auth_tickets 2024-04-22T08:12:52.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.371+0000 7f089d18f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:52.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.371+0000 7f089d18f640 1 -- 172.21.15.40:0/2156347506 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f089402e050 con 0x7f089809ef20 2024-04-22T08:12:52.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.371+0000 7f089d18f640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:52.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.371+0000 7f089d18f640 1 -- 172.21.15.40:0/2156347506 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0894030d80 con 0x7f089809ef20 2024-04-22T08:12:52.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.371+0000 7f089d18f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:52.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.371+0000 7f089d18f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:12:52.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.371+0000 7f089d18f640 10 monclient: dump: 2024-04-22T08:12:52.421 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:52.421 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.421 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.421 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.421 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:52.421 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.421 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:52.421 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:52.421 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:52.422 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.371+0000 7f08a03f5640 5 monclient: authenticate success, global_id 4247 2024-04-22T08:12:52.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.371+0000 7f08a03f5640 10 monclient: get_monmap_and_config success 2024-04-22T08:12:52.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.371+0000 7f08a03f5640 4 set_mon_vals no callback set 2024-04-22T08:12:52.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.371+0000 7efd37270640 10 monclient: shutdown 2024-04-22T08:12:52.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.371+0000 7efd37270640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:52.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.371+0000 7efd37270640 1 -- 172.21.15.40:0/3990490466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd300a0d20 msgr2=0x7efd300a1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.371+0000 7efd37270640 1 --2- 172.21.15.40:0/3990490466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd300a0d20 0x7efd300a1e30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7efd20009870 tx=0x7efd2002b7f0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.371+0000 7efd37270640 1 -- 172.21.15.40:0/3990490466 shutdown_connections 2024-04-22T08:12:52.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.371+0000 7efd37270640 1 --2- 172.21.15.40:0/3990490466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd300a0d20 0x7efd300a1e30 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.371+0000 7efd37270640 1 -- 172.21.15.40:0/3990490466 >> 172.21.15.40:0/3990490466 conn(0x7efd300a4e80 msgr2=0x7efd300a5290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:52.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.371+0000 7efd37270640 1 -- 172.21.15.40:0/3990490466 shutdown_connections 2024-04-22T08:12:52.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.371+0000 7efd37270640 1 -- 172.21.15.40:0/3990490466 wait complete. 2024-04-22T08:12:52.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.372+0000 7efd37270640 10 monclient: build_initial_monmap 2024-04-22T08:12:52.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.372+0000 7efd37270640 10 monclient: monmap: 2024-04-22T08:12:52.423 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:52.423 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:12:52.423 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:12:52.423 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:12:52.423 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:52.423 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.424 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:12:52.424 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:12:52.424 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:12:52.424 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.372+0000 7f1c85d07640 1 -- 172.21.15.40:0/4260163537 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1c800a1820 msgr2=0x7f1c80134460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:52.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.372+0000 7f1c85d07640 1 --2- 172.21.15.40:0/4260163537 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1c800a1820 0x7f1c80134460 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1c700090b0 tx=0x7f1c700046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:52.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.372+0000 7f08a03f5640 10 monclient: shutdown 2024-04-22T08:12:52.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.372+0000 7f08a03f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:52.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.372+0000 7f08a03f5640 1 -- 172.21.15.40:0/2156347506 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f089809ef20 msgr2=0x7f089813f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.372+0000 7f08a03f5640 1 --2- 172.21.15.40:0/2156347506 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f089809ef20 0x7f089813f5c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f08940097a0 tx=0x7f089402b6d0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.372+0000 7f08a03f5640 1 -- 172.21.15.40:0/2156347506 shutdown_connections 2024-04-22T08:12:52.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.372+0000 7f08a03f5640 1 --2- 172.21.15.40:0/2156347506 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f089809ef20 0x7f089813f5c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.372+0000 7efd37270640 1 Processor -- start 2024-04-22T08:12:52.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.372+0000 7f08a03f5640 1 -- 172.21.15.40:0/2156347506 >> 172.21.15.40:0/2156347506 conn(0x7f08980a3370 msgr2=0x7f089806ba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:52.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.372+0000 7f1c76ffd640 1 -- 172.21.15.40:0/4260163537 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c7003c400 con 0x7f1c800a1820 2024-04-22T08:12:52.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.372+0000 7f1c76ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:52.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.372+0000 7f1c76ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:12:52.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.372+0000 7f1c76ffd640 10 monclient: dump: 2024-04-22T08:12:52.426 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:52.426 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.426 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.426 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.426 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:52.426 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.427 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:52.427 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:52.427 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:52.427 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.372+0000 7f1c76ffd640 10 monclient: _finish_auth 0 2024-04-22T08:12:52.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.372+0000 7f1c76ffd640 10 monclient: _check_auth_tickets 2024-04-22T08:12:52.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.372+0000 7f1c76ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:52.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.372+0000 7f1c76ffd640 1 -- 172.21.15.40:0/4260163537 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c7003c560 con 0x7f1c800a1820 2024-04-22T08:12:52.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.372+0000 7f1c76ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:52.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.372+0000 7f1c76ffd640 1 -- 172.21.15.40:0/4260163537 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c7003c8f0 con 0x7f1c800a1820 2024-04-22T08:12:52.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.372+0000 7f1c76ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:52.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.372+0000 7f1c76ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:12:52.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.372+0000 7f1c76ffd640 10 monclient: dump: 2024-04-22T08:12:52.428 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:52.428 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.428 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.428 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.428 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:52.428 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.428 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:52.428 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:52.428 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:52.428 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.372+0000 7f1c8876c640 5 monclient: authenticate success, global_id 4250 2024-04-22T08:12:52.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.372+0000 7f1c8876c640 10 monclient: _renew_subs 2024-04-22T08:12:52.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.372+0000 7efd37270640 1 -- start start 2024-04-22T08:12:52.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.372+0000 7f1c84d05640 4 set_mon_vals no callback set 2024-04-22T08:12:52.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.372+0000 7f1c8876c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:52.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.372+0000 7f1c8876c640 1 -- 172.21.15.40:0/4260163537 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c80121320 con 0x7f1c800a1820 2024-04-22T08:12:52.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.373+0000 7efd37270640 10 monclient: init 2024-04-22T08:12:52.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.373+0000 7f31d7fff640 1 -- 172.21.15.40:0/2714820949 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31d809d710 msgr2=0x7f31d809daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:52.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.373+0000 7f31d7fff640 1 --2- 172.21.15.40:0/2714820949 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31d809d710 0x7f31d809daf0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f31d0009e50 tx=0x7f31d002bdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e3e44253ae05b05 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:52.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.373+0000 7efd37270640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:52.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.373+0000 7efd37270640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-22T08:12:52.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.373+0000 7f31d6ffd640 1 -- 172.21.15.40:0/2714820949 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d002cb60 con 0x7f31d809d710 2024-04-22T08:12:52.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.373+0000 7efd37270640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd3009ee10 0x7efd30071160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.373+0000 7f08a03f5640 1 -- 172.21.15.40:0/2156347506 shutdown_connections 2024-04-22T08:12:52.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.373+0000 7f08a03f5640 1 -- 172.21.15.40:0/2156347506 wait complete. 2024-04-22T08:12:52.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.373+0000 7efd37270640 10 monclient(hunting): picked mon.noname-a con 0x7efd3009ee10 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:52.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.373+0000 7efd37270640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efd300a0d20 0x7efd301cf740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.373+0000 7f31d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:52.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7efd37270640 10 monclient(hunting): picked mon.noname-c con 0x7efd300a0d20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:52.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7efd37270640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd3013f550 0x7efd3006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7f1c8876c640 10 monclient: _renew_subs 2024-04-22T08:12:52.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7f1c8876c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:52.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7f1c8876c640 1 -- 172.21.15.40:0/4260163537 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c800718e0 con 0x7f1c800a1820 2024-04-22T08:12:52.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7efd37270640 10 monclient(hunting): picked mon.noname-b con 0x7efd3013f550 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:52.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7efd3500c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd3009ee10 0x7efd30071160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7efd37270640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7efd3500c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd3009ee10 0x7efd30071160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57948/0 (socket says 172.21.15.40:57948) 2024-04-22T08:12:52.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7efd37270640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd300a73b0 con 0x7efd3009ee10 2024-04-22T08:12:52.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7efd37270640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7efd3500c640 1 -- 172.21.15.40:0/2123648794 learned_addr learned my addr 172.21.15.40:0/2123648794 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:52.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7efd37270640 1 -- 172.21.15.40:0/2123648794 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd300a24c0 con 0x7efd3013f550 2024-04-22T08:12:52.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7efd37270640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7efd37270640 1 -- 172.21.15.40:0/2123648794 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd30071b70 con 0x7efd300a0d20 2024-04-22T08:12:52.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7efd37270640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:52.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7efd37270640 10 monclient(hunting): authenticate will time out at 846.368023s 2024-04-22T08:12:52.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7efd3480b640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efd300a0d20 0x7efd301cf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7efd3500c640 10 monclient(hunting): get_auth_request con 0x7efd3009ee10 auth_method 0 2024-04-22T08:12:52.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7efd3500c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:52.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7efd3500c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:52.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7efd3500c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:52.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7efd3480b640 10 monclient(hunting): get_auth_request con 0x7efd300a0d20 auth_method 0 2024-04-22T08:12:52.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7efd3480b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:52.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7efd3480b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:52.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7efd3480b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:52.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7efd3500c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:52.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7efd3500c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:52.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7efd3500c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:52.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7efd3480b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:52.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7efd3480b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:52.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7efd3480b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:52.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.373+0000 7f31d6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:12:52.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.373+0000 7f31d6ffd640 10 monclient: dump: 2024-04-22T08:12:52.436 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:52.437 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.437 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.437 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.437 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:52.437 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.437 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:52.437 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:52.437 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:52.437 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.373+0000 7f31d6ffd640 10 monclient: _finish_auth 0 2024-04-22T08:12:52.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.373+0000 7f31d6ffd640 10 monclient: _check_auth_tickets 2024-04-22T08:12:52.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.373+0000 7f31d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:52.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7f31d6ffd640 1 -- 172.21.15.40:0/2714820949 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31d002ccc0 con 0x7f31d809d710 2024-04-22T08:12:52.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7f31d6ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:52.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7f31d6ffd640 1 -- 172.21.15.40:0/2714820949 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d0034610 con 0x7f31d809d710 2024-04-22T08:12:52.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7f31d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:52.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7f31d6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:12:52.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7f31d6ffd640 10 monclient: dump: 2024-04-22T08:12:52.438 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:52.438 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.438 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.438 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.438 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:52.438 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.438 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:52.438 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:52.438 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:52.439 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7efd3500c640 10 monclient(hunting): handle_auth_done global_id 4275 payload 824 2024-04-22T08:12:52.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7efd3500c640 1 -- 172.21.15.40:0/2123648794 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efd300a0d20 msgr2=0x7efd301cf740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7efd3500c640 1 --2- 172.21.15.40:0/2123648794 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efd300a0d20 0x7efd301cf740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7efd3500c640 1 -- 172.21.15.40:0/2123648794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd3013f550 msgr2=0x7efd3006eb30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T08:12:52.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7efd3500c640 1 --2- 172.21.15.40:0/2123648794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd3013f550 0x7efd3006eb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7efd3500c640 10 monclient: _finish_hunting 0 2024-04-22T08:12:52.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7efd3500c640 1 monclient: found mon.noname-a 2024-04-22T08:12:52.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7efd3500c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:52.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7efd3500c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:12:52.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7efd3500c640 1 -- 172.21.15.40:0/2123648794 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd200094d0 con 0x7efd3009ee10 2024-04-22T08:12:52.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7f31de6fc640 5 monclient: authenticate success, global_id 4253 2024-04-22T08:12:52.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7f31de6fc640 10 monclient: get_monmap_and_config success 2024-04-22T08:12:52.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.375+0000 7f31de6fc640 4 set_mon_vals no callback set 2024-04-22T08:12:52.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.375+0000 7f08a03f5640 10 monclient: build_initial_monmap 2024-04-22T08:12:52.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7f1c76ffd640 1 -- 172.21.15.40:0/4260163537 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 4) v1 ==== 44726+0+0 (secure 0 0 0) 0x7f1c7003a040 con 0x7f1c800a1820 2024-04-22T08:12:52.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.375+0000 7f08a03f5640 10 monclient: monmap: 2024-04-22T08:12:52.441 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:52.441 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:12:52.441 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:12:52.441 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:12:52.441 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:52.441 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.441 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:12:52.442 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:12:52.442 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:12:52.442 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.374+0000 7f1c76ffd640 1 --2- 172.21.15.40:0/4260163537 >> v2:172.21.15.189:6800/30168 conn(0x7f1c64037790 0x7f1c64039c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.375+0000 7f1c76ffd640 1 -- 172.21.15.40:0/4260163537 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f1c7003b760 con 0x7f1c800a1820 2024-04-22T08:12:52.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.375+0000 7f1c86508640 1 --2- 172.21.15.40:0/4260163537 >> v2:172.21.15.189:6800/30168 conn(0x7f1c64037790 0x7f1c64039c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.375+0000 7f1c86508640 10 monclient: get_auth_request con 0x7f1c64037790 auth_method 0 2024-04-22T08:12:52.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.375+0000 7f1c8876c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:52.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.375+0000 7f1c8876c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:52.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.375+0000 7f1c8876c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:52.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.375+0000 7f1c8876c640 1 -- 172.21.15.40:0/4260163537 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c8004e9b0 con 0x7f1c800a1820 2024-04-22T08:12:52.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.375+0000 7f08a03f5640 1 Processor -- start 2024-04-22T08:12:52.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.375+0000 7f08a03f5640 1 -- start start 2024-04-22T08:12:52.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.375+0000 7f31de6fc640 10 monclient: shutdown 2024-04-22T08:12:52.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.375+0000 7f08a03f5640 10 monclient: init 2024-04-22T08:12:52.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.375+0000 7f31de6fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:52.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.376+0000 7f31de6fc640 1 -- 172.21.15.40:0/2714820949 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31d809d710 msgr2=0x7f31d809daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.376+0000 7f31de6fc640 1 --2- 172.21.15.40:0/2714820949 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31d809d710 0x7f31d809daf0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f31d0009e50 tx=0x7f31d002bdb0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.376+0000 7f31de6fc640 1 -- 172.21.15.40:0/2714820949 shutdown_connections 2024-04-22T08:12:52.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.376+0000 7f31de6fc640 1 --2- 172.21.15.40:0/2714820949 >> v2:172.21.15.189:3300/0 conn(0x7f31d809ef20 0x7f31d809e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.376+0000 7f31de6fc640 1 --2- 172.21.15.40:0/2714820949 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31d809d710 0x7f31d809daf0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.376+0000 7f31de6fc640 1 -- 172.21.15.40:0/2714820949 >> 172.21.15.40:0/2714820949 conn(0x7f31d80a3370 msgr2=0x7f31d806ba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:52.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.376+0000 7f31de6fc640 1 -- 172.21.15.40:0/2714820949 shutdown_connections 2024-04-22T08:12:52.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.376+0000 7f31de6fc640 1 -- 172.21.15.40:0/2714820949 wait complete. 2024-04-22T08:12:52.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.376+0000 7f08a03f5640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:52.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.376+0000 7f08a03f5640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-22T08:12:52.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.376+0000 7f08a03f5640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f089809d710 0x7f08981ceb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.376+0000 7f08a03f5640 10 monclient(hunting): picked mon.noname-c con 0x7f089809d710 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:52.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.376+0000 7f08a03f5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f089809e200 0x7f08981cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.377+0000 7f08a03f5640 10 monclient(hunting): picked mon.noname-b con 0x7f089809e200 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:52.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.377+0000 7f089e191640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f089809d710 0x7f08981ceb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.377+0000 7f089e191640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f089809d710 0x7f08981ceb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49764/0 (socket says 172.21.15.40:49764) 2024-04-22T08:12:52.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.377+0000 7f089e191640 1 -- 172.21.15.40:0/1029789542 learned_addr learned my addr 172.21.15.40:0/1029789542 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:52.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.377+0000 7f089d990640 1 --2- 172.21.15.40:0/1029789542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f089809e200 0x7f08981cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.377+0000 7efd3500c640 1 -- 172.21.15.40:0/2123648794 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd3009ee10 msgr2=0x7efd30071160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:52.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.377+0000 7efd3500c640 1 --2- 172.21.15.40:0/2123648794 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd3009ee10 0x7efd30071160 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7efd2c00ba50 tx=0x7efd2c00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:52.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.377+0000 7f31de6fc640 10 monclient: build_initial_monmap 2024-04-22T08:12:52.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.377+0000 7f31de6fc640 10 monclient: monmap: 2024-04-22T08:12:52.448 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:52.448 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:12:52.448 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:12:52.448 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:12:52.448 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:52.449 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.449 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:12:52.449 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:12:52.449 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:12:52.449 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.377+0000 7efd25ffb640 1 -- 172.21.15.40:0/2123648794 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd2c00cdf0 con 0x7efd3009ee10 2024-04-22T08:12:52.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.377+0000 7efd25ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:52.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.377+0000 7efd25ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:12:52.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.377+0000 7efd25ffb640 10 monclient: dump: 2024-04-22T08:12:52.450 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:52.450 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.450 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.450 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.450 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:52.450 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.450 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:52.450 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:52.450 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:52.450 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.377+0000 7efd25ffb640 10 monclient: _finish_auth 0 2024-04-22T08:12:52.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.377+0000 7efd25ffb640 10 monclient: _check_auth_tickets 2024-04-22T08:12:52.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.377+0000 7efd25ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:52.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.377+0000 7efd25ffb640 1 -- 172.21.15.40:0/2123648794 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd2c014440 con 0x7efd3009ee10 2024-04-22T08:12:52.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.377+0000 7efd25ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:52.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.377+0000 7efd25ffb640 1 -- 172.21.15.40:0/2123648794 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd2c0147f0 con 0x7efd3009ee10 2024-04-22T08:12:52.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.377+0000 7efd25ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:52.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.377+0000 7efd25ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:12:52.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.377+0000 7efd25ffb640 10 monclient: dump: 2024-04-22T08:12:52.451 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:52.451 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.451 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.451 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.451 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:52.451 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.451 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:52.452 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:52.452 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:52.452 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.377+0000 7efd277fe640 4 set_mon_vals no callback set 2024-04-22T08:12:52.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.377+0000 7f08a03f5640 1 --2- 172.21.15.40:0/1029789542 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f089809ef20 0x7f08981cf5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.377+0000 7f08a03f5640 10 monclient(hunting): picked mon.noname-a con 0x7f089809ef20 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:52.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.378+0000 7f1c86508640 1 --2- 172.21.15.40:0/4260163537 >> v2:172.21.15.189:6800/30168 conn(0x7f1c64037790 0x7f1c64039c50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c0098f0 tx=0x7f1c7c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:52.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.378+0000 7f08a03f5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.378+0000 7f08a03f5640 1 -- 172.21.15.40:0/1029789542 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08980704b0 con 0x7f089809ef20 2024-04-22T08:12:52.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.378+0000 7f08a03f5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.378+0000 7f089e992640 1 --2- 172.21.15.40:0/1029789542 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f089809ef20 0x7f08981cf5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.378+0000 7f08a03f5640 1 -- 172.21.15.40:0/1029789542 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0898070700 con 0x7f089809e200 2024-04-22T08:12:52.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.378+0000 7f08a03f5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.378+0000 7f08a03f5640 1 -- 172.21.15.40:0/1029789542 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f089809f380 con 0x7f089809d710 2024-04-22T08:12:52.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.378+0000 7f08a03f5640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:52.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.378+0000 7f08a03f5640 10 monclient(hunting): authenticate will time out at 846.372022s 2024-04-22T08:12:52.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.378+0000 7f31de6fc640 1 Processor -- start 2024-04-22T08:12:52.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.378+0000 7f089e191640 10 monclient(hunting): get_auth_request con 0x7f089809d710 auth_method 0 2024-04-22T08:12:52.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.378+0000 7f089e191640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:52.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.378+0000 7f089e191640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:52.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.378+0000 7f089e191640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:52.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.378+0000 7f089d990640 10 monclient(hunting): get_auth_request con 0x7f089809e200 auth_method 0 2024-04-22T08:12:52.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.378+0000 7f089d990640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:52.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.378+0000 7f089d990640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:52.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.378+0000 7f089d990640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:52.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.378+0000 7f089e992640 10 monclient(hunting): get_auth_request con 0x7f089809ef20 auth_method 0 2024-04-22T08:12:52.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.378+0000 7f089e992640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:52.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.378+0000 7f089e992640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:52.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.378+0000 7f089e992640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:52.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.378+0000 7f089e191640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:52.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.378+0000 7f089e191640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:52.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.378+0000 7f089e191640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:52.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.378+0000 7f089d990640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:52.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.378+0000 7f089d990640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:52.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.378+0000 7f089d990640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:52.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.378+0000 7f089e191640 10 monclient(hunting): handle_auth_done global_id 4259 payload 824 2024-04-22T08:12:52.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.378+0000 7f089e191640 1 -- 172.21.15.40:0/1029789542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f089809e200 msgr2=0x7f08981cf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.378+0000 7f089e191640 1 --2- 172.21.15.40:0/1029789542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f089809e200 0x7f08981cf0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.378+0000 7f089e191640 1 -- 172.21.15.40:0/1029789542 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f089809ef20 msgr2=0x7f08981cf5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.378+0000 7f089e191640 1 --2- 172.21.15.40:0/1029789542 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f089809ef20 0x7f08981cf5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.378+0000 7f089e191640 10 monclient: _finish_hunting 0 2024-04-22T08:12:52.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.378+0000 7f089e191640 1 monclient: found mon.noname-c 2024-04-22T08:12:52.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.378+0000 7f089e191640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:52.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.378+0000 7f089e191640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:12:52.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.378+0000 7f089e191640 1 -- 172.21.15.40:0/1029789542 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0894009400 con 0x7f089809d710 2024-04-22T08:12:52.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.378+0000 7f089d990640 1 --2- 172.21.15.40:0/1029789542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f089809e200 0x7f08981cf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:12:52.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.378+0000 7f31de6fc640 1 -- start start 2024-04-22T08:12:52.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.378+0000 7f31de6fc640 10 monclient: init 2024-04-22T08:12:52.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.379+0000 7efd37270640 5 monclient: authenticate success, global_id 4275 2024-04-22T08:12:52.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.379+0000 7efd37270640 10 monclient: _renew_subs 2024-04-22T08:12:52.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.379+0000 7efd37270640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:52.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.379+0000 7efd37270640 1 -- 172.21.15.40:0/2123648794 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd301279f0 con 0x7efd3009ee10 2024-04-22T08:12:52.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.379+0000 7f31de6fc640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:52.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.379+0000 7f31de6fc640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:12:52.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.379+0000 7f31de6fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31d809d710 0x7f31d81ce9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.379+0000 7f31de6fc640 10 monclient(hunting): picked mon.noname-a con 0x7f31d809d710 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:52.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.379+0000 7efd37270640 10 monclient: _renew_subs 2024-04-22T08:12:52.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.379+0000 7efd37270640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:52.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.379+0000 7efd37270640 1 -- 172.21.15.40:0/2123648794 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd30070590 con 0x7efd3009ee10 2024-04-22T08:12:52.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.379+0000 7f31de6fc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31d809ef20 0x7f31d813d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.379+0000 7f31de6fc640 10 monclient(hunting): picked mon.noname-b con 0x7f31d809ef20 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:52.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.379+0000 7f31de6fc640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31d813e5b0 0x7f31d813da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.379+0000 7f31de6fc640 10 monclient(hunting): picked mon.noname-c con 0x7f31d813e5b0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:52.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.379+0000 7f31de6fc640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.379+0000 7f31de6fc640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d809ac70 con 0x7f31d809d710 2024-04-22T08:12:52.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.379+0000 7f31de6fc640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.379+0000 7f31de6fc640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d8071200 con 0x7f31d809ef20 2024-04-22T08:12:52.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.379+0000 7f31de6fc640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:52.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.379+0000 7f31de6fc640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31d812ae80 con 0x7f31d813e5b0 2024-04-22T08:12:52.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.379+0000 7f31de6fc640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:52.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.379+0000 7f31de6fc640 10 monclient(hunting): authenticate will time out at 846.373741s 2024-04-22T08:12:52.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.380+0000 7f31d77fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31d809ef20 0x7f31d813d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.380+0000 7f31d77fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31d809ef20 0x7f31d813d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.40:42794/0 (socket says 172.21.15.40:42794) 2024-04-22T08:12:52.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.380+0000 7f31d77fe640 1 -- 172.21.15.40:0/724429862 learned_addr learned my addr 172.21.15.40:0/724429862 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:52.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.380+0000 7f31d77fe640 10 monclient(hunting): get_auth_request con 0x7f31d809ef20 auth_method 0 2024-04-22T08:12:52.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.380+0000 7f31d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:52.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.380+0000 7f31d77fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:52.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.380+0000 7f31d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:52.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.380+0000 7f31d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:52.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.380+0000 7f31d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:52.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.380+0000 7f31d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:52.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.380+0000 7f31dcc99640 1 --2- 172.21.15.40:0/724429862 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31d813e5b0 0x7f31d813da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.380+0000 7f31dcc99640 10 monclient(hunting): get_auth_request con 0x7f31d813e5b0 auth_method 0 2024-04-22T08:12:52.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.380+0000 7f31dcc99640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:52.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.380+0000 7f31dcc99640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:52.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.380+0000 7f31dcc99640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:52.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.380+0000 7f31d77fe640 10 monclient(hunting): handle_auth_done global_id 4255 payload 824 2024-04-22T08:12:52.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.380+0000 7efd25ffb640 1 -- 172.21.15.40:0/2123648794 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 4) v1 ==== 44726+0+0 (secure 0 0 0) 0x7efd2c014a10 con 0x7efd3009ee10 2024-04-22T08:12:52.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.380+0000 7f31d77fe640 1 -- 172.21.15.40:0/724429862 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31d813e5b0 msgr2=0x7f31d813da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.380+0000 7f31d77fe640 1 --2- 172.21.15.40:0/724429862 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31d813e5b0 0x7f31d813da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.380+0000 7f31d77fe640 1 -- 172.21.15.40:0/724429862 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31d809d710 msgr2=0x7f31d81ce9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.380+0000 7f31d77fe640 1 --2- 172.21.15.40:0/724429862 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31d809d710 0x7f31d81ce9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.380+0000 7f31d77fe640 10 monclient: _finish_hunting 0 2024-04-22T08:12:52.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.380+0000 7f31d77fe640 1 monclient: found mon.noname-b 2024-04-22T08:12:52.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.380+0000 7efd25ffb640 1 --2- 172.21.15.40:0/2123648794 >> v2:172.21.15.189:6800/30168 conn(0x7efd14037770 0x7efd14039c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.380+0000 7f31d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:52.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.380+0000 7efd25ffb640 1 -- 172.21.15.40:0/2123648794 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3506+0+0 (secure 0 0 0) 0x7efd2c049fe0 con 0x7efd3009ee10 2024-04-22T08:12:52.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.380+0000 7f31d77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.189:3300/0 2024-04-22T08:12:52.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.380+0000 7f31d77fe640 1 -- 172.21.15.40:0/724429862 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d0009b30 con 0x7f31d809ef20 2024-04-22T08:12:52.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.380+0000 7efd37270640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:52.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.380+0000 7efd37270640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:52.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.380+0000 7efd37270640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:52.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.380+0000 7f31dcc99640 1 --2- 172.21.15.40:0/724429862 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31d813e5b0 0x7f31d813da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T08:12:52.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.381+0000 7f089e191640 1 -- 172.21.15.40:0/1029789542 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f089809d710 msgr2=0x7f08981ceb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:52.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.381+0000 7f089e191640 1 --2- 172.21.15.40:0/1029789542 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f089809d710 0x7f08981ceb70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f089000cc40 tx=0x7f089000a430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:52.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.380+0000 7efd37270640 1 -- 172.21.15.40:0/2123648794 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd300673a0 con 0x7efd3009ee10 2024-04-22T08:12:52.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.381+0000 7efd3480b640 1 --2- 172.21.15.40:0/2123648794 >> v2:172.21.15.189:6800/30168 conn(0x7efd14037770 0x7efd14039c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.381+0000 7efd3480b640 10 monclient: get_auth_request con 0x7efd14037770 auth_method 0 2024-04-22T08:12:52.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.382+0000 7f088effd640 1 -- 172.21.15.40:0/1029789542 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0890012400 con 0x7f089809d710 2024-04-22T08:12:52.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.382+0000 7f088effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:52.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.382+0000 7f088effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:12:52.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.382+0000 7f088effd640 10 monclient: dump: 2024-04-22T08:12:52.470 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:52.470 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.470 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.470 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.470 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:52.470 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.470 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:52.470 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:52.471 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:52.471 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.382+0000 7f088effd640 10 monclient: _finish_auth 0 2024-04-22T08:12:52.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.382+0000 7f088effd640 10 monclient: _check_auth_tickets 2024-04-22T08:12:52.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.382+0000 7f088effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:52.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.382+0000 7f088effd640 1 -- 172.21.15.40:0/1029789542 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0890012560 con 0x7f089809d710 2024-04-22T08:12:52.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.382+0000 7f08a03f5640 5 monclient: authenticate success, global_id 4259 2024-04-22T08:12:52.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.382+0000 7f08a03f5640 10 monclient: _renew_subs 2024-04-22T08:12:52.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.382+0000 7f08a03f5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:52.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.382+0000 7f08a03f5640 1 -- 172.21.15.40:0/1029789542 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f089809b110 con 0x7f089809d710 2024-04-22T08:12:52.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.382+0000 7f08a03f5640 10 monclient: _renew_subs 2024-04-22T08:12:52.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.382+0000 7f1c76ffd640 1 -- 172.21.15.40:0/4260163537 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f1c70038080 con 0x7f1c800a1820 2024-04-22T08:12:52.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.382+0000 7f1c76ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:52.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.382+0000 7f1c76ffd640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:12:52.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.382+0000 7f08a03f5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:52.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.382+0000 7f08a03f5640 1 -- 172.21.15.40:0/1029789542 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0898071660 con 0x7f089809d710 2024-04-22T08:12:52.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.382+0000 7f088effd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:52.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.382+0000 7f088effd640 1 -- 172.21.15.40:0/1029789542 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0890012910 con 0x7f089809d710 2024-04-22T08:12:52.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.382+0000 7f088effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:52.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.382+0000 7f088ffff640 4 set_mon_vals no callback set 2024-04-22T08:12:52.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.382+0000 7f088effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:12:52.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.382+0000 7f088effd640 10 monclient: dump: 2024-04-22T08:12:52.473 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:52.473 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.473 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.473 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.473 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:52.473 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.473 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:52.474 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:52.474 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:52.474 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.383+0000 7f31d77fe640 1 -- 172.21.15.40:0/724429862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31d809ef20 msgr2=0x7f31d813d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:52.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.383+0000 7f31d77fe640 1 --2- 172.21.15.40:0/724429862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31d809ef20 0x7f31d813d4c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f31cc00c7f0 tx=0x7f31cc00cbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:52.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.383+0000 7f31d4ff9640 1 -- 172.21.15.40:0/724429862 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31cc007c40 con 0x7f31d809ef20 2024-04-22T08:12:52.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.383+0000 7f31d4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:52.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.383+0000 7f31d4ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:12:52.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.383+0000 7f31d4ff9640 10 monclient: dump: 2024-04-22T08:12:52.475 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:52.475 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.475 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.475 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.475 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:52.475 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.475 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:52.475 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:52.475 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:52.475 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.383+0000 7f31d4ff9640 10 monclient: _finish_auth 0 2024-04-22T08:12:52.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.383+0000 7f31d4ff9640 10 monclient: _check_auth_tickets 2024-04-22T08:12:52.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.383+0000 7f31d4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:52.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.383+0000 7f31de6fc640 5 monclient: authenticate success, global_id 4255 2024-04-22T08:12:52.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.383+0000 7f31de6fc640 10 monclient: _renew_subs 2024-04-22T08:12:52.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.383+0000 7f31d4ff9640 1 -- 172.21.15.40:0/724429862 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31cc007da0 con 0x7f31d809ef20 2024-04-22T08:12:52.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.383+0000 7f31de6fc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.189:3300/0 2024-04-22T08:12:52.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.383+0000 7f31de6fc640 1 -- 172.21.15.40:0/724429862 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31d812a5e0 con 0x7f31d809ef20 2024-04-22T08:12:52.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.383+0000 7f31d4ff9640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:52.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.383+0000 7f31de6fc640 10 monclient: _renew_subs 2024-04-22T08:12:52.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.383+0000 7f31de6fc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.189:3300/0 2024-04-22T08:12:52.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.383+0000 7f31de6fc640 1 -- 172.21.15.40:0/724429862 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31d813df40 con 0x7f31d809ef20 2024-04-22T08:12:52.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.383+0000 7f088effd640 1 -- 172.21.15.40:0/1029789542 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 4) v1 ==== 44726+0+0 (secure 0 0 0) 0x7f0890012af0 con 0x7f089809d710 2024-04-22T08:12:52.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.383+0000 7f31d4ff9640 1 -- 172.21.15.40:0/724429862 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31cc0115f0 con 0x7f31d809ef20 2024-04-22T08:12:52.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.384+0000 7f31d4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:52.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.384+0000 7f31d4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:12:52.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.384+0000 7f08a03f5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:52.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.384+0000 7f088effd640 1 --2- 172.21.15.40:0/1029789542 >> v2:172.21.15.189:6800/30168 conn(0x7f087c037790 0x7f087c039c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.384+0000 7f08a03f5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:52.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.384+0000 7f08a03f5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:52.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.384+0000 7f08a03f5640 1 -- 172.21.15.40:0/1029789542 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f089804e9b0 con 0x7f089809d710 2024-04-22T08:12:52.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.384+0000 7efd3480b640 1 --2- 172.21.15.40:0/2123648794 >> v2:172.21.15.189:6800/30168 conn(0x7efd14037770 0x7efd14039c30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7efd300a4720 tx=0x7efd20008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:52.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.384+0000 7f088effd640 1 -- 172.21.15.40:0/1029789542 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f089004a710 con 0x7f089809d710 2024-04-22T08:12:52.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.384+0000 7f089d990640 1 --2- 172.21.15.40:0/1029789542 >> v2:172.21.15.189:6800/30168 conn(0x7f087c037790 0x7f087c039c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.384+0000 7f089d990640 10 monclient: get_auth_request con 0x7f087c037790 auth_method 0 2024-04-22T08:12:52.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.384+0000 7f31d4ff9640 10 monclient: dump: 2024-04-22T08:12:52.480 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:52.480 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:52.480 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.480 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:52.480 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:52.480 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:52.480 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:52.481 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:52.481 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:52.481 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:52.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.384+0000 7f31d67fc640 4 set_mon_vals no callback set 2024-04-22T08:12:52.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.385+0000 7f31de6fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:52.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.385+0000 7f31de6fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:52.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.385+0000 7f31de6fc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.189:3300/0 2024-04-22T08:12:52.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.385+0000 7f31de6fc640 1 -- 172.21.15.40:0/724429862 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31b8005040 con 0x7f31d809ef20 2024-04-22T08:12:52.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.386+0000 7f31d4ff9640 1 -- 172.21.15.40:0/724429862 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 44726+0+0 (secure 0 0 0) 0x7f31cc011750 con 0x7f31d809ef20 2024-04-22T08:12:52.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.386+0000 7f31d4ff9640 1 --2- 172.21.15.40:0/724429862 >> v2:172.21.15.189:6800/30168 conn(0x7f31c4037740 0x7f31c4039c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:52.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.386+0000 7f31d4ff9640 1 -- 172.21.15.40:0/724429862 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f31cc04a3a0 con 0x7f31d809ef20 2024-04-22T08:12:52.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.387+0000 7f31d7fff640 1 --2- 172.21.15.40:0/724429862 >> v2:172.21.15.189:6800/30168 conn(0x7f31c4037740 0x7f31c4039c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:52.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.387+0000 7f31d7fff640 10 monclient: get_auth_request con 0x7f31c4037740 auth_method 0 2024-04-22T08:12:52.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.387+0000 7f089d990640 1 --2- 172.21.15.40:0/1029789542 >> v2:172.21.15.189:6800/30168 conn(0x7f087c037790 0x7f087c039c50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0894032000 tx=0x7f0894008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:52.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.388+0000 7efd25ffb640 1 -- 172.21.15.40:0/2123648794 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7efd2c019070 con 0x7efd3009ee10 2024-04-22T08:12:52.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.388+0000 7efd25ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:52.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.388+0000 7efd25ffb640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:12:52.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.389+0000 7f31d7fff640 1 --2- 172.21.15.40:0/724429862 >> v2:172.21.15.189:6800/30168 conn(0x7f31c4037740 0x7f31c4039c00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f31d0032000 tx=0x7f31d002e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:52.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.391+0000 7f31d4ff9640 1 -- 172.21.15.40:0/724429862 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f31cc011a00 con 0x7f31d809ef20 2024-04-22T08:12:52.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.391+0000 7f31d4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:52.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.391+0000 7f31d4ff9640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:12:52.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.391+0000 7f088effd640 1 -- 172.21.15.40:0/1029789542 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f0890019cd0 con 0x7f089809d710 2024-04-22T08:12:52.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.392+0000 7f088effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:52.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.392+0000 7f088effd640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:12:52.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.530+0000 7f33773ae640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-22T08:12:52.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.530+0000 7f33773ae640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-22T08:12:52.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.530+0000 7f33773ae640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:52.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.530+0000 7f33773ae640 1 -- 172.21.15.40:0/958802222 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f337004e9b0 con 0x7f337009ee10 2024-04-22T08:12:52.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.531+0000 7f3365ffb640 1 -- 172.21.15.40:0/958802222 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f336c019310 con 0x7f337009ee10 2024-04-22T08:12:52.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.531+0000 7f3365ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-22T08:12:52.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.531+0000 7f3365ffb640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:12:52.532 INFO:teuthology.orchestra.run.smithi040.stdout:47244640258 2024-04-22T08:12:52.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.533+0000 7f33773ae640 1 -- 172.21.15.40:0/958802222 >> v2:172.21.15.189:6800/30168 conn(0x7f33540377a0 msgr2=0x7f3354039c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.533+0000 7f33773ae640 1 --2- 172.21.15.40:0/958802222 >> v2:172.21.15.189:6800/30168 conn(0x7f33540377a0 0x7f3354039c60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3360032000 tx=0x7f336002d000 comp rx=0 tx=0).stop 2024-04-22T08:12:52.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.533+0000 7f33773ae640 10 monclient: shutdown 2024-04-22T08:12:52.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.533+0000 7f33773ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:52.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.533+0000 7f33773ae640 1 -- 172.21.15.40:0/958802222 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f337009ee10 msgr2=0x7f3370139bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.533+0000 7f33773ae640 1 --2- 172.21.15.40:0/958802222 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f337009ee10 0x7f3370139bf0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f336c00c410 tx=0x7f336c00c8d0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.533+0000 7f33773ae640 1 -- 172.21.15.40:0/958802222 shutdown_connections 2024-04-22T08:12:52.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.533+0000 7f33773ae640 1 --2- 172.21.15.40:0/958802222 >> v2:172.21.15.189:6800/30168 conn(0x7f33540377a0 0x7f3354039c60 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.533+0000 7f33773ae640 1 --2- 172.21.15.40:0/958802222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f337013b0e0 0x7f3370120b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.533+0000 7f33773ae640 1 --2- 172.21.15.40:0/958802222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33700a1f30 0x7f3370120620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.533+0000 7f33773ae640 1 --2- 172.21.15.40:0/958802222 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f337009ee10 0x7f3370139bf0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.533+0000 7f33773ae640 1 -- 172.21.15.40:0/958802222 >> 172.21.15.40:0/958802222 conn(0x7f33700a4e80 msgr2=0x7f337013b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:52.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.533+0000 7f33773ae640 1 -- 172.21.15.40:0/958802222 shutdown_connections 2024-04-22T08:12:52.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.533+0000 7f33773ae640 1 -- 172.21.15.40:0/958802222 wait complete. 2024-04-22T08:12:52.544 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.4 2024-04-22T08:12:52.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.575+0000 7efd37270640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-22T08:12:52.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.575+0000 7efd37270640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-22T08:12:52.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.575+0000 7efd37270640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:52.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.575+0000 7efd37270640 1 -- 172.21.15.40:0/2123648794 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7efd3004e9b0 con 0x7efd3009ee10 2024-04-22T08:12:52.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.575+0000 7efd25ffb640 1 -- 172.21.15.40:0/2123648794 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7efd2c03d090 con 0x7efd3009ee10 2024-04-22T08:12:52.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.575+0000 7efd25ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-22T08:12:52.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.575+0000 7efd25ffb640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:12:52.575 INFO:teuthology.orchestra.run.smithi040.stdout:47244640260 2024-04-22T08:12:52.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.577+0000 7efd37270640 1 -- 172.21.15.40:0/2123648794 >> v2:172.21.15.189:6800/30168 conn(0x7efd14037770 msgr2=0x7efd14039c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.577+0000 7efd37270640 1 --2- 172.21.15.40:0/2123648794 >> v2:172.21.15.189:6800/30168 conn(0x7efd14037770 0x7efd14039c30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7efd300a4720 tx=0x7efd20008030 comp rx=0 tx=0).stop 2024-04-22T08:12:52.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.577+0000 7efd37270640 10 monclient: shutdown 2024-04-22T08:12:52.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.577+0000 7efd37270640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:52.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.577+0000 7efd37270640 1 -- 172.21.15.40:0/2123648794 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd3009ee10 msgr2=0x7efd30071160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.577+0000 7efd37270640 1 --2- 172.21.15.40:0/2123648794 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd3009ee10 0x7efd30071160 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7efd2c00ba50 tx=0x7efd2c00bf10 comp rx=0 tx=0).stop 2024-04-22T08:12:52.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.577+0000 7efd37270640 1 -- 172.21.15.40:0/2123648794 shutdown_connections 2024-04-22T08:12:52.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.577+0000 7efd37270640 1 --2- 172.21.15.40:0/2123648794 >> v2:172.21.15.189:6800/30168 conn(0x7efd14037770 0x7efd14039c30 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.577+0000 7efd37270640 1 --2- 172.21.15.40:0/2123648794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd3013f550 0x7efd3006eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.577+0000 7efd37270640 1 --2- 172.21.15.40:0/2123648794 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efd300a0d20 0x7efd301cf740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.577+0000 7efd37270640 1 --2- 172.21.15.40:0/2123648794 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd3009ee10 0x7efd30071160 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.577+0000 7efd37270640 1 -- 172.21.15.40:0/2123648794 >> 172.21.15.40:0/2123648794 conn(0x7efd300a4e80 msgr2=0x7efd300a2c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:52.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.577+0000 7efd37270640 1 -- 172.21.15.40:0/2123648794 shutdown_connections 2024-04-22T08:12:52.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.577+0000 7efd37270640 1 -- 172.21.15.40:0/2123648794 wait complete. 2024-04-22T08:12:52.588 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640260 for osd.2 2024-04-22T08:12:52.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.720+0000 7f31de6fc640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-22T08:12:52.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.720+0000 7f31de6fc640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-22T08:12:52.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.720+0000 7f31de6fc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.189:3300/0 2024-04-22T08:12:52.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.720+0000 7f08a03f5640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-22T08:12:52.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.720+0000 7f08a03f5640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-22T08:12:52.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.720+0000 7f08a03f5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:52.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.720+0000 7f31de6fc640 1 -- 172.21.15.40:0/724429862 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f31b80052f0 con 0x7f31d809ef20 2024-04-22T08:12:52.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.720+0000 7f08a03f5640 1 -- 172.21.15.40:0/1029789542 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f0898070840 con 0x7f089809d710 2024-04-22T08:12:52.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.720+0000 7f088effd640 1 -- 172.21.15.40:0/1029789542 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0898070840 con 0x7f089809d710 2024-04-22T08:12:52.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.720+0000 7f088effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-22T08:12:52.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.720+0000 7f088effd640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:12:52.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.721+0000 7f31d4ff9640 1 -- 172.21.15.40:0/724429862 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f31cc0193b0 con 0x7f31d809ef20 2024-04-22T08:12:52.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.721+0000 7f31d4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-22T08:12:52.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.721+0000 7f31d4ff9640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:12:52.722 INFO:teuthology.orchestra.run.smithi040.stdout:47244640259 2024-04-22T08:12:52.722 INFO:teuthology.orchestra.run.smithi040.stdout:47244640259 2024-04-22T08:12:52.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.723+0000 7f31de6fc640 1 -- 172.21.15.40:0/724429862 >> v2:172.21.15.189:6800/30168 conn(0x7f31c4037740 msgr2=0x7f31c4039c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.723+0000 7f31de6fc640 1 --2- 172.21.15.40:0/724429862 >> v2:172.21.15.189:6800/30168 conn(0x7f31c4037740 0x7f31c4039c00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f31d0032000 tx=0x7f31d002e000 comp rx=0 tx=0).stop 2024-04-22T08:12:52.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.723+0000 7f31de6fc640 10 monclient: shutdown 2024-04-22T08:12:52.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.723+0000 7f31de6fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:52.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.723+0000 7f31de6fc640 1 -- 172.21.15.40:0/724429862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31d809ef20 msgr2=0x7f31d813d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.723+0000 7f31de6fc640 1 --2- 172.21.15.40:0/724429862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31d809ef20 0x7f31d813d4c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f31cc00c7f0 tx=0x7f31cc00cbf0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.723+0000 7f31de6fc640 1 -- 172.21.15.40:0/724429862 shutdown_connections 2024-04-22T08:12:52.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.723+0000 7f31de6fc640 1 --2- 172.21.15.40:0/724429862 >> v2:172.21.15.189:6800/30168 conn(0x7f31c4037740 0x7f31c4039c00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.723+0000 7f31de6fc640 1 --2- 172.21.15.40:0/724429862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31d809ef20 0x7f31d813d4c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.723+0000 7f31de6fc640 1 --2- 172.21.15.40:0/724429862 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31d813e5b0 0x7f31d813da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.723+0000 7f31de6fc640 1 --2- 172.21.15.40:0/724429862 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31d809d710 0x7f31d81ce9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.723+0000 7f31de6fc640 1 -- 172.21.15.40:0/724429862 >> 172.21.15.40:0/724429862 conn(0x7f31d80a3370 msgr2=0x7f31d80a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:52.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.723+0000 7f08a03f5640 1 -- 172.21.15.40:0/1029789542 >> v2:172.21.15.189:6800/30168 conn(0x7f087c037790 msgr2=0x7f087c039c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.723+0000 7f08a03f5640 1 --2- 172.21.15.40:0/1029789542 >> v2:172.21.15.189:6800/30168 conn(0x7f087c037790 0x7f087c039c50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0894032000 tx=0x7f0894008030 comp rx=0 tx=0).stop 2024-04-22T08:12:52.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.723+0000 7f08a03f5640 10 monclient: shutdown 2024-04-22T08:12:52.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.723+0000 7f08a03f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:52.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.723+0000 7f08a03f5640 1 -- 172.21.15.40:0/1029789542 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f089809d710 msgr2=0x7f08981ceb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.723+0000 7f08a03f5640 1 --2- 172.21.15.40:0/1029789542 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f089809d710 0x7f08981ceb70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f089000cc40 tx=0x7f089000a430 comp rx=0 tx=0).stop 2024-04-22T08:12:52.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.723+0000 7f08a03f5640 1 -- 172.21.15.40:0/1029789542 shutdown_connections 2024-04-22T08:12:52.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.723+0000 7f08a03f5640 1 --2- 172.21.15.40:0/1029789542 >> v2:172.21.15.189:6800/30168 conn(0x7f087c037790 0x7f087c039c50 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.723+0000 7f08a03f5640 1 --2- 172.21.15.40:0/1029789542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f089809e200 0x7f08981cf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.723+0000 7f08a03f5640 1 --2- 172.21.15.40:0/1029789542 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f089809ef20 0x7f08981cf5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.723+0000 7f08a03f5640 1 --2- 172.21.15.40:0/1029789542 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f089809d710 0x7f08981ceb70 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.723+0000 7f08a03f5640 1 -- 172.21.15.40:0/1029789542 >> 172.21.15.40:0/1029789542 conn(0x7f08980a3370 msgr2=0x7f08980a5730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:52.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.723+0000 7f31de6fc640 1 -- 172.21.15.40:0/724429862 shutdown_connections 2024-04-22T08:12:52.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.723+0000 7f31de6fc640 1 -- 172.21.15.40:0/724429862 wait complete. 2024-04-22T08:12:52.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.723+0000 7f08a03f5640 1 -- 172.21.15.40:0/1029789542 shutdown_connections 2024-04-22T08:12:52.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.723+0000 7f08a03f5640 1 -- 172.21.15.40:0/1029789542 wait complete. 2024-04-22T08:12:52.739 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640259 for osd.5 2024-04-22T08:12:52.739 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640259 for osd.0 2024-04-22T08:12:52.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.754+0000 7f5a11870640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-22T08:12:52.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.754+0000 7f5a11870640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-22T08:12:52.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.754+0000 7f5a11870640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:52.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.755+0000 7f5a11870640 1 -- 172.21.15.40:0/4245641914 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f5a0c134c40 con 0x7f5a0c09ee10 2024-04-22T08:12:52.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.755+0000 7f5a03fff640 1 -- 172.21.15.40:0/4245641914 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f59fc017070 con 0x7f5a0c09ee10 2024-04-22T08:12:52.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.755+0000 7f5a03fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-22T08:12:52.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.755+0000 7f5a03fff640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:12:52.755 INFO:teuthology.orchestra.run.smithi040.stdout:47244640258 2024-04-22T08:12:52.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.757+0000 7f5a11870640 1 -- 172.21.15.40:0/4245641914 >> v2:172.21.15.189:6800/30168 conn(0x7f59ec037740 msgr2=0x7f59ec039c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.757+0000 7f5a11870640 1 --2- 172.21.15.40:0/4245641914 >> v2:172.21.15.189:6800/30168 conn(0x7f59ec037740 0x7f59ec039c00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c0a35b0 tx=0x7f59f4008030 comp rx=0 tx=0).stop 2024-04-22T08:12:52.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.757+0000 7f5a11870640 10 monclient: shutdown 2024-04-22T08:12:52.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.757+0000 7f5a11870640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:52.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.757+0000 7f5a11870640 1 -- 172.21.15.40:0/4245641914 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a0c09ee10 msgr2=0x7f5a0c1cf070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.757+0000 7f5a11870640 1 --2- 172.21.15.40:0/4245641914 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a0c09ee10 0x7f5a0c1cf070 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f59fc00c360 tx=0x7f59fc00c820 comp rx=0 tx=0).stop 2024-04-22T08:12:52.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.757+0000 7f5a11870640 1 -- 172.21.15.40:0/4245641914 shutdown_connections 2024-04-22T08:12:52.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.757+0000 7f5a11870640 1 --2- 172.21.15.40:0/4245641914 >> v2:172.21.15.189:6800/30168 conn(0x7f59ec037740 0x7f59ec039c00 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.757+0000 7f5a11870640 1 --2- 172.21.15.40:0/4245641914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a0c0a1f30 0x7f5a0c13a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.757+0000 7f5a11870640 1 --2- 172.21.15.40:0/4245641914 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a0c13b0e0 0x7f5a0c13a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.757+0000 7f5a11870640 1 --2- 172.21.15.40:0/4245641914 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a0c09ee10 0x7f5a0c1cf070 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.757+0000 7f5a11870640 1 -- 172.21.15.40:0/4245641914 >> 172.21.15.40:0/4245641914 conn(0x7f5a0c0a4e80 msgr2=0x7f5a0c0a6b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:52.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.757+0000 7f5a11870640 1 -- 172.21.15.40:0/4245641914 shutdown_connections 2024-04-22T08:12:52.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.757+0000 7f5a11870640 1 -- 172.21.15.40:0/4245641914 wait complete. 2024-04-22T08:12:52.767 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.1 2024-04-22T08:12:52.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.777+0000 7f1c8876c640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-22T08:12:52.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.777+0000 7f1c8876c640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-22T08:12:52.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.777+0000 7f1c8876c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:52.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.778+0000 7f1c8876c640 1 -- 172.21.15.40:0/4260163537 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f1c8006a030 con 0x7f1c800a1820 2024-04-22T08:12:52.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.778+0000 7f1c76ffd640 1 -- 172.21.15.40:0/4260163537 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1c70043030 con 0x7f1c800a1820 2024-04-22T08:12:52.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.778+0000 7f1c76ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-22T08:12:52.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.778+0000 7f1c76ffd640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:12:52.778 INFO:teuthology.orchestra.run.smithi040.stdout:47244640258 2024-04-22T08:12:52.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.780+0000 7f1c8876c640 1 -- 172.21.15.40:0/4260163537 >> v2:172.21.15.189:6800/30168 conn(0x7f1c64037790 msgr2=0x7f1c64039c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.780+0000 7f1c8876c640 1 --2- 172.21.15.40:0/4260163537 >> v2:172.21.15.189:6800/30168 conn(0x7f1c64037790 0x7f1c64039c50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c0098f0 tx=0x7f1c7c008040 comp rx=0 tx=0).stop 2024-04-22T08:12:52.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.780+0000 7f1c8876c640 10 monclient: shutdown 2024-04-22T08:12:52.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.780+0000 7f1c8876c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:52.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.780+0000 7f1c8876c640 1 -- 172.21.15.40:0/4260163537 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1c800a1820 msgr2=0x7f1c80134460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:52.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.780+0000 7f1c8876c640 1 --2- 172.21.15.40:0/4260163537 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1c800a1820 0x7f1c80134460 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1c700090b0 tx=0x7f1c700046c0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.780+0000 7f1c8876c640 1 -- 172.21.15.40:0/4260163537 shutdown_connections 2024-04-22T08:12:52.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.780+0000 7f1c8876c640 1 --2- 172.21.15.40:0/4260163537 >> v2:172.21.15.189:6800/30168 conn(0x7f1c64037790 0x7f1c64039c50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.780+0000 7f1c8876c640 1 --2- 172.21.15.40:0/4260163537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c80135dc0 0x7f1c8011e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.780+0000 7f1c8876c640 1 --2- 172.21.15.40:0/4260163537 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1c800a1820 0x7f1c80134460 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.780+0000 7f1c8876c640 1 --2- 172.21.15.40:0/4260163537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c8009ee10 0x7f1c80133f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:52.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.780+0000 7f1c8876c640 1 -- 172.21.15.40:0/4260163537 >> 172.21.15.40:0/4260163537 conn(0x7f1c800a4e80 msgr2=0x7f1c800a7470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:52.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.780+0000 7f1c8876c640 1 -- 172.21.15.40:0/4260163537 shutdown_connections 2024-04-22T08:12:52.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:52.780+0000 7f1c8876c640 1 -- 172.21.15.40:0/4260163537 wait complete. 2024-04-22T08:12:52.790 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.3 2024-04-22T08:12:53.545 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-22T08:12:53.589 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-22T08:12:53.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.658+0000 7f57544af640 10 monclient: get_monmap_and_config 2024-04-22T08:12:53.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.658+0000 7f57544af640 10 monclient: build_initial_monmap 2024-04-22T08:12:53.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.658+0000 7f57544af640 10 monclient: monmap: 2024-04-22T08:12:53.658 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:53.659 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:53.659 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:53.659299+0000 2024-04-22T08:12:53.659 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:53.659299+0000 2024-04-22T08:12:53.659 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:53.659 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:53.659 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:12:53.659 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:12:53.659 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:12:53.659 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:12:53.659 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:12:53.659 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:53.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.659+0000 7f57544af640 1 Processor -- start 2024-04-22T08:12:53.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.659+0000 7f57544af640 1 -- start start 2024-04-22T08:12:53.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.659+0000 7f57544af640 10 monclient: init 2024-04-22T08:12:53.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.659+0000 7f57544af640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:53.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.659+0000 7f57544af640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-04-22T08:12:53.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.659+0000 7f57544af640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f574c0a54b0 addr v1:172.21.15.189:6789/0 2024-04-22T08:12:53.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.659+0000 7f57544af640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f574c0a33b0 0x7f574c13aef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:53.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.659+0000 7f57544af640 10 monclient(hunting): picked mon.noname-c con 0x7f574c0a33b0 addr v2:172.21.15.189:3300/0 2024-04-22T08:12:53.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.659+0000 7f57544af640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f574c0a3820 0x7f574c13d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:53.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.659+0000 7f57544af640 10 monclient(hunting): picked mon.noname-a con 0x7f574c0a3820 addr v2:172.21.15.40:3300/0 2024-04-22T08:12:53.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.659+0000 7f57544af640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f574c0a5890 con 0x7f574c0a54b0 2024-04-22T08:12:53.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.659+0000 7f57544af640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:53.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.659+0000 7f57544af640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f574c0a4370 con 0x7f574c0a3820 2024-04-22T08:12:53.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.659+0000 7f57544af640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:53.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.659+0000 7f57544af640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f574c0a0160 con 0x7f574c0a33b0 2024-04-22T08:12:53.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.659+0000 7f57544af640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:53.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.659+0000 7f57544af640 10 monclient(hunting): authenticate will time out at 847.653596s 2024-04-22T08:12:53.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.659+0000 7f5752a4c640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f574c0a3820 0x7f574c13d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:53.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.660+0000 7f5752a4c640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f574c0a3820 0x7f574c13d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57970/0 (socket says 172.21.15.40:57970) 2024-04-22T08:12:53.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.660+0000 7f5752a4c640 1 -- 172.21.15.40:0/1396469416 learned_addr learned my addr 172.21.15.40:0/1396469416 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:53.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.660+0000 7f5752a4c640 10 monclient(hunting): get_auth_request con 0x7f574c0a3820 auth_method 0 2024-04-22T08:12:53.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.660+0000 7f5752a4c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:53.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.660+0000 7f5752a4c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:53.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.660+0000 7f5752a4c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:53.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.660+0000 7f5752a4c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:53.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.660+0000 7f5752a4c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:53.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.660+0000 7f5752a4c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:53.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.660+0000 7f5751a4a640 1 --2- 172.21.15.40:0/1396469416 >> v2:172.21.15.189:3300/0 conn(0x7f574c0a33b0 0x7f574c13aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:53.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.660+0000 7f5752a4c640 10 monclient(hunting): handle_auth_done global_id 4284 payload 274 2024-04-22T08:12:53.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.660+0000 7f5752a4c640 1 -- 172.21.15.40:0/1396469416 >> v2:172.21.15.189:3300/0 conn(0x7f574c0a33b0 msgr2=0x7f574c13aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:53.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.660+0000 7f5752a4c640 1 --2- 172.21.15.40:0/1396469416 >> v2:172.21.15.189:3300/0 conn(0x7f574c0a33b0 0x7f574c13aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:53.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.660+0000 7f5752a4c640 1 -- 172.21.15.40:0/1396469416 >> v1:172.21.15.189:6789/0 conn(0x7f574c0a54b0 legacy=0x7f574c0a2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:53.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.660+0000 7f5752a4c640 10 monclient: _finish_hunting 0 2024-04-22T08:12:53.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.660+0000 7f5752a4c640 1 monclient: found mon.noname-a 2024-04-22T08:12:53.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.660+0000 7f5752a4c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:53.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.660+0000 7f5752a4c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:12:53.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.660+0000 7f5752a4c640 1 -- 172.21.15.40:0/1396469416 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f574c0a0440 con 0x7f574c0a3820 2024-04-22T08:12:53.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.663+0000 7f5752a4c640 1 -- 172.21.15.40:0/1396469416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f574c0a3820 msgr2=0x7f574c13d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:53.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.663+0000 7f5752a4c640 1 --2- 172.21.15.40:0/1396469416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f574c0a3820 0x7f574c13d440 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f574c0722c0 tx=0x7f574402bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bfd69d6246b74ff server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:53.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.663+0000 7f5751249640 1 -- 172.21.15.40:0/1396469416 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f574402cb70 con 0x7f574c0a3820 2024-04-22T08:12:53.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.663+0000 7f5751249640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:53.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.663+0000 7f5751249640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:12:53.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.663+0000 7f5751249640 10 monclient: dump: 2024-04-22T08:12:53.669 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:53.669 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:53.669 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:53.670 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:53.670 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:53.670 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:53.670 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:53.670 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:53.670 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:53.670 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:53.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.663+0000 7f5751249640 10 monclient: _finish_auth 0 2024-04-22T08:12:53.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.663+0000 7f5751249640 10 monclient: _check_auth_tickets 2024-04-22T08:12:53.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.663+0000 7f5751249640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:53.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.663+0000 7f5751249640 1 -- 172.21.15.40:0/1396469416 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f574402ccd0 con 0x7f574c0a3820 2024-04-22T08:12:53.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.663+0000 7f57544af640 5 monclient: authenticate success, global_id 4284 2024-04-22T08:12:53.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.663+0000 7f57544af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:12:53.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.663+0000 7f5751249640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:53.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.663+0000 7f57544af640 10 monclient: get_monmap_and_config success 2024-04-22T08:12:53.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.663+0000 7f5751249640 1 -- 172.21.15.40:0/1396469416 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5744034610 con 0x7f574c0a3820 2024-04-22T08:12:53.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.663+0000 7f5751249640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:53.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.663+0000 7f57544af640 4 set_mon_vals no callback set 2024-04-22T08:12:53.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.663+0000 7f5751249640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:12:53.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.663+0000 7f5751249640 10 monclient: dump: 2024-04-22T08:12:53.671 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:53.671 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:53.671 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:53.671 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:53.671 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:53.671 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:53.671 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:53.672 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:53.672 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:53.672 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:53.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.664+0000 7f57544af640 10 monclient: shutdown 2024-04-22T08:12:53.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.664+0000 7f57544af640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:53.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.664+0000 7f57544af640 1 -- 172.21.15.40:0/1396469416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f574c0a3820 msgr2=0x7f574c13d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:53.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.664+0000 7f57544af640 1 --2- 172.21.15.40:0/1396469416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f574c0a3820 0x7f574c13d440 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f574c0722c0 tx=0x7f574402bdc0 comp rx=0 tx=0).stop 2024-04-22T08:12:53.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.664+0000 7f57544af640 1 -- 172.21.15.40:0/1396469416 shutdown_connections 2024-04-22T08:12:53.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.664+0000 7f57544af640 1 --2- 172.21.15.40:0/1396469416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f574c0a3820 0x7f574c13d440 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:53.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.664+0000 7f57544af640 1 --2- 172.21.15.40:0/1396469416 >> v2:172.21.15.189:3300/0 conn(0x7f574c0a33b0 0x7f574c13aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:53.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.664+0000 7f57544af640 1 -- 172.21.15.40:0/1396469416 >> 172.21.15.40:0/1396469416 conn(0x7f574c0a8bc0 msgr2=0x7f574c0a4da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:53.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.664+0000 7f57544af640 1 -- 172.21.15.40:0/1396469416 shutdown_connections 2024-04-22T08:12:53.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.664+0000 7f57544af640 1 -- 172.21.15.40:0/1396469416 wait complete. 2024-04-22T08:12:53.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.665+0000 7f57544af640 10 monclient: build_initial_monmap 2024-04-22T08:12:53.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.665+0000 7f57544af640 10 monclient: monmap: 2024-04-22T08:12:53.673 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:53.673 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:12:53.673 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:12:53.673 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:12:53.673 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:53.673 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:53.673 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:12:53.673 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:12:53.673 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:12:53.673 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:53.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.665+0000 7f57544af640 1 Processor -- start 2024-04-22T08:12:53.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.665+0000 7f57544af640 1 -- start start 2024-04-22T08:12:53.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.665+0000 7f57544af640 10 monclient: init 2024-04-22T08:12:53.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.665+0000 7f57544af640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:53.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.665+0000 7f57544af640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:12:53.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.665+0000 7f57544af640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f574c0a33b0 0x7f574c134f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:53.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f57544af640 10 monclient(hunting): picked mon.noname-a con 0x7f574c0a33b0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:53.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f57544af640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f574c0a3820 0x7f574c135450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:53.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f57544af640 10 monclient(hunting): picked mon.noname-b con 0x7f574c0a3820 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:53.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f57544af640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f574c0a54b0 0x7f574c13a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:53.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f57544af640 10 monclient(hunting): picked mon.noname-c con 0x7f574c0a54b0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:53.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f57544af640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:53.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f57544af640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574c11fd80 con 0x7f574c0a33b0 2024-04-22T08:12:53.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f57544af640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:53.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f57544af640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574c13d170 con 0x7f574c0a3820 2024-04-22T08:12:53.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f57544af640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:53.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f57544af640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f574c11f730 con 0x7f574c0a54b0 2024-04-22T08:12:53.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f57544af640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:53.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f57544af640 10 monclient(hunting): authenticate will time out at 847.659848s 2024-04-22T08:12:53.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f575224b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f574c0a33b0 0x7f574c134f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:53.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f575224b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f574c0a33b0 0x7f574c134f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57982/0 (socket says 172.21.15.40:57982) 2024-04-22T08:12:53.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f5752a4c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f574c0a54b0 0x7f574c13a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:53.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f575224b640 1 -- 172.21.15.40:0/3127029691 learned_addr learned my addr 172.21.15.40:0/3127029691 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:53.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f575224b640 10 monclient(hunting): get_auth_request con 0x7f574c0a33b0 auth_method 0 2024-04-22T08:12:53.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f575224b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:53.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f575224b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:53.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f575224b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:53.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f5752a4c640 10 monclient(hunting): get_auth_request con 0x7f574c0a54b0 auth_method 0 2024-04-22T08:12:53.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f5752a4c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:53.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f5752a4c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:53.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f5752a4c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:53.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f5752a4c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:53.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f5752a4c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:53.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f5752a4c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:53.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f575224b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:53.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f575224b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:53.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f575224b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:53.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f5751a4a640 1 --2- 172.21.15.40:0/3127029691 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f574c0a3820 0x7f574c135450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:53.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f5751a4a640 10 monclient(hunting): get_auth_request con 0x7f574c0a3820 auth_method 0 2024-04-22T08:12:53.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f5751a4a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:53.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f5751a4a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:53.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f5751a4a640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:53.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f5751a4a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:53.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f5751a4a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:53.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f5751a4a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:53.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f5752a4c640 10 monclient(hunting): handle_auth_done global_id 4268 payload 824 2024-04-22T08:12:53.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f5752a4c640 1 -- 172.21.15.40:0/3127029691 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f574c0a3820 msgr2=0x7f574c135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:53.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f5752a4c640 1 --2- 172.21.15.40:0/3127029691 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f574c0a3820 0x7f574c135450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:53.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f5752a4c640 1 -- 172.21.15.40:0/3127029691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f574c0a33b0 msgr2=0x7f574c134f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:53.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f5752a4c640 1 --2- 172.21.15.40:0/3127029691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f574c0a33b0 0x7f574c134f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:53.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f5752a4c640 10 monclient: _finish_hunting 0 2024-04-22T08:12:53.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f5752a4c640 1 monclient: found mon.noname-c 2024-04-22T08:12:53.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f5752a4c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:53.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f5752a4c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:12:53.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f5752a4c640 1 -- 172.21.15.40:0/3127029691 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5744009b30 con 0x7f574c0a54b0 2024-04-22T08:12:53.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.666+0000 7f575224b640 1 --2- 172.21.15.40:0/3127029691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f574c0a33b0 0x7f574c134f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:12:53.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.668+0000 7f5752a4c640 1 -- 172.21.15.40:0/3127029691 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f574c0a54b0 msgr2=0x7f574c13a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:53.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.668+0000 7f5752a4c640 1 --2- 172.21.15.40:0/3127029691 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f574c0a54b0 0x7f574c13a470 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5744035000 tx=0x7f5744004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:53.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.669+0000 7f5742ffd640 1 -- 172.21.15.40:0/3127029691 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5744007e30 con 0x7f574c0a54b0 2024-04-22T08:12:53.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.669+0000 7f5742ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:53.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.669+0000 7f5742ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:12:53.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.669+0000 7f5742ffd640 10 monclient: dump: 2024-04-22T08:12:53.678 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:53.678 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:53.678 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:53.678 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:53.678 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:53.678 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:53.678 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:53.678 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:53.678 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:53.678 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:53.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.669+0000 7f5742ffd640 10 monclient: _finish_auth 0 2024-04-22T08:12:53.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.669+0000 7f5742ffd640 10 monclient: _check_auth_tickets 2024-04-22T08:12:53.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.669+0000 7f5742ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:53.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.669+0000 7f57544af640 5 monclient: authenticate success, global_id 4268 2024-04-22T08:12:53.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.669+0000 7f5742ffd640 1 -- 172.21.15.40:0/3127029691 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f574403e660 con 0x7f574c0a54b0 2024-04-22T08:12:53.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.669+0000 7f57544af640 10 monclient: _renew_subs 2024-04-22T08:12:53.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.669+0000 7f57544af640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:53.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.669+0000 7f57544af640 1 -- 172.21.15.40:0/3127029691 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f574c11fa10 con 0x7f574c0a54b0 2024-04-22T08:12:53.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.669+0000 7f5742ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:53.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.669+0000 7f5742ffd640 1 -- 172.21.15.40:0/3127029691 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f574403ea10 con 0x7f574c0a54b0 2024-04-22T08:12:53.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.669+0000 7f5742ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:53.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.669+0000 7f5742ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:12:53.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.669+0000 7f5742ffd640 10 monclient: dump: 2024-04-22T08:12:53.679 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:53.679 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:53.679 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:53.680 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:53.680 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:53.680 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:53.680 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:53.680 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:53.680 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:53.680 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:53.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.669+0000 7f5750a48640 4 set_mon_vals no callback set 2024-04-22T08:12:53.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.669+0000 7f57544af640 10 monclient: _renew_subs 2024-04-22T08:12:53.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.669+0000 7f57544af640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:53.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.669+0000 7f57544af640 1 -- 172.21.15.40:0/3127029691 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f574c11f400 con 0x7f574c0a54b0 2024-04-22T08:12:53.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.670+0000 7f5742ffd640 1 -- 172.21.15.40:0/3127029691 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f574403ebb0 con 0x7f574c0a54b0 2024-04-22T08:12:53.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.670+0000 7f57544af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:53.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.670+0000 7f57544af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:53.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.670+0000 7f57544af640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:53.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.670+0000 7f5742ffd640 1 --2- 172.21.15.40:0/3127029691 >> v2:172.21.15.189:6800/30168 conn(0x7f57300377d0 0x7f5730039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:53.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.670+0000 7f57544af640 1 -- 172.21.15.40:0/3127029691 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f574c11fef0 con 0x7f574c0a54b0 2024-04-22T08:12:53.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.670+0000 7f5742ffd640 1 -- 172.21.15.40:0/3127029691 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f574402ce40 con 0x7f574c0a54b0 2024-04-22T08:12:53.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.670+0000 7f575224b640 1 --2- 172.21.15.40:0/3127029691 >> v2:172.21.15.189:6800/30168 conn(0x7f57300377d0 0x7f5730039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:53.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.670+0000 7f575224b640 10 monclient: get_auth_request con 0x7f57300377d0 auth_method 0 2024-04-22T08:12:53.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.673+0000 7f575224b640 1 --2- 172.21.15.40:0/3127029691 >> v2:172.21.15.189:6800/30168 conn(0x7f57300377d0 0x7f5730039c90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f57480043e0 tx=0x7f5748007ef0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:53.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.675+0000 7f5742ffd640 1 -- 172.21.15.40:0/3127029691 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f5744043030 con 0x7f574c0a54b0 2024-04-22T08:12:53.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.675+0000 7f5742ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:53.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.675+0000 7f5742ffd640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:12:53.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.699+0000 7fc010a7d640 10 monclient: get_monmap_and_config 2024-04-22T08:12:53.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.699+0000 7fc010a7d640 10 monclient: build_initial_monmap 2024-04-22T08:12:53.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.699+0000 7fc010a7d640 10 monclient: monmap: 2024-04-22T08:12:53.699 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:53.700 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:53.700 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:53.699932+0000 2024-04-22T08:12:53.700 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:53.699932+0000 2024-04-22T08:12:53.700 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:53.700 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:53.700 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:12:53.701 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:12:53.701 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:12:53.701 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:12:53.701 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:12:53.701 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:53.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.699+0000 7fc010a7d640 1 Processor -- start 2024-04-22T08:12:53.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.699+0000 7fc010a7d640 1 -- start start 2024-04-22T08:12:53.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.699+0000 7fc010a7d640 10 monclient: init 2024-04-22T08:12:53.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.700+0000 7fc010a7d640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:53.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.700+0000 7fc010a7d640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-04-22T08:12:53.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.700+0000 7fc010a7d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc00c09ee10 0x7fc00c0a32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:53.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.700+0000 7fc010a7d640 10 monclient(hunting): picked mon.noname-b con 0x7fc00c09ee10 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:53.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.700+0000 7fc010a7d640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc00c0a1d30 0x7fc00c0a2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:53.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.700+0000 7fc010a7d640 10 monclient(hunting): picked mon.noname-a con 0x7fc00c0a1d30 addr v2:172.21.15.40:3300/0 2024-04-22T08:12:53.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.700+0000 7fc010a7d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc00c135d90 addr v1:172.21.15.189:6789/0 2024-04-22T08:12:53.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.700+0000 7fc010a7d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc00c09f1f0 con 0x7fc00c135d90 2024-04-22T08:12:53.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.700+0000 7fc010a7d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:53.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.700+0000 7fc010a7d640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc00c0a39c0 con 0x7fc00c0a1d30 2024-04-22T08:12:53.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.700+0000 7fc010a7d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:53.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.700+0000 7fc010a7d640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc00c09fd30 con 0x7fc00c09ee10 2024-04-22T08:12:53.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.700+0000 7fc010a7d640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:53.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.700+0000 7fc010a7d640 10 monclient(hunting): authenticate will time out at 847.694276s 2024-04-22T08:12:53.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.700+0000 7fc00a59c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc00c09ee10 0x7fc00c0a32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:53.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.700+0000 7fc009d9b640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc00c0a1d30 0x7fc00c0a2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:53.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.700+0000 7fc00a59c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc00c09ee10 0x7fc00c0a32b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49790/0 (socket says 172.21.15.40:49790) 2024-04-22T08:12:53.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.700+0000 7fc00a59c640 1 -- 172.21.15.40:0/699790687 learned_addr learned my addr 172.21.15.40:0/699790687 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:53.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.700+0000 7fc009d9b640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc00c0a1d30 0x7fc00c0a2150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57986/0 (socket says 172.21.15.40:57986) 2024-04-22T08:12:53.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.700+0000 7fc00a59c640 10 monclient(hunting): get_auth_request con 0x7fc00c09ee10 auth_method 0 2024-04-22T08:12:53.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.700+0000 7fc00a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:53.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.700+0000 7fc00a59c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:53.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.700+0000 7fc00a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:53.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.700+0000 7fc009d9b640 10 monclient(hunting): get_auth_request con 0x7fc00c0a1d30 auth_method 0 2024-04-22T08:12:53.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.700+0000 7fc009d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:53.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.700+0000 7fc009d9b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:53.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.700+0000 7fc009d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:53.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.700+0000 7fc00a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:53.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.700+0000 7fc00a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:53.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.700+0000 7fc00a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:53.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.700+0000 7fc009d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:53.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.700+0000 7fc009d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:53.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.700+0000 7fc009d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:53.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.700+0000 7fc00a59c640 10 monclient(hunting): handle_auth_done global_id 4271 payload 274 2024-04-22T08:12:53.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.700+0000 7fc00a59c640 1 -- 172.21.15.40:0/699790687 >> v2:172.21.15.40:3300/0 conn(0x7fc00c0a1d30 msgr2=0x7fc00c0a2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:53.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.700+0000 7fc00a59c640 1 --2- 172.21.15.40:0/699790687 >> v2:172.21.15.40:3300/0 conn(0x7fc00c0a1d30 0x7fc00c0a2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:53.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.700+0000 7fc00a59c640 1 -- 172.21.15.40:0/699790687 >> v1:172.21.15.189:6789/0 conn(0x7fc00c135d90 legacy=0x7fc00c1361c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:53.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.700+0000 7fc00a59c640 10 monclient: _finish_hunting 0 2024-04-22T08:12:53.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.700+0000 7fc00a59c640 1 monclient: found mon.noname-b 2024-04-22T08:12:53.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.700+0000 7fc00a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:53.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.701+0000 7fc00a59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:12:53.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.701+0000 7fc00a59c640 1 -- 172.21.15.40:0/699790687 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc00c0a02f0 con 0x7fc00c09ee10 2024-04-22T08:12:53.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.701+0000 7fc009d9b640 1 --2- 172.21.15.40:0/699790687 >> v2:172.21.15.40:3300/0 conn(0x7fc00c0a1d30 0x7fc00c0a2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:12:53.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.703+0000 7fc00a59c640 1 -- 172.21.15.40:0/699790687 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc00c09ee10 msgr2=0x7fc00c0a32b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:53.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.703+0000 7fc00a59c640 1 --2- 172.21.15.40:0/699790687 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc00c09ee10 0x7fc00c0a32b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc004009e50 tx=0x7fc00402bdc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b9124ca5dc9f89e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:53.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.703+0000 7fc00959a640 1 -- 172.21.15.40:0/699790687 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc00402cb70 con 0x7fc00c09ee10 2024-04-22T08:12:53.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.703+0000 7fc00959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:53.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.704+0000 7fc00959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:12:53.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.704+0000 7fc00959a640 10 monclient: dump: 2024-04-22T08:12:53.726 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:53.726 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:53.727 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:53.727 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:53.727 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:53.727 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:53.727 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:53.728 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:53.728 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:53.728 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:53.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.704+0000 7fc00959a640 10 monclient: _finish_auth 0 2024-04-22T08:12:53.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.704+0000 7fc00959a640 10 monclient: _check_auth_tickets 2024-04-22T08:12:53.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.704+0000 7fc00959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:53.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.704+0000 7fc010a7d640 5 monclient: authenticate success, global_id 4271 2024-04-22T08:12:53.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.704+0000 7fc00959a640 1 -- 172.21.15.40:0/699790687 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc00402ccd0 con 0x7fc00c09ee10 2024-04-22T08:12:53.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.704+0000 7fc010a7d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:12:53.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.704+0000 7fc00959a640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:53.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.704+0000 7fc010a7d640 10 monclient: get_monmap_and_config success 2024-04-22T08:12:53.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.704+0000 7fc010a7d640 4 set_mon_vals no callback set 2024-04-22T08:12:53.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.704+0000 7fc00959a640 1 -- 172.21.15.40:0/699790687 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc004034610 con 0x7fc00c09ee10 2024-04-22T08:12:53.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.704+0000 7fc00959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:53.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.704+0000 7fc00959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:12:53.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.704+0000 7fc00959a640 10 monclient: dump: 2024-04-22T08:12:53.731 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:53.731 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:53.731 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:53.732 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:53.732 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:53.732 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:53.732 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:53.732 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:53.733 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:53.733 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:53.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.704+0000 7fc010a7d640 10 monclient: shutdown 2024-04-22T08:12:53.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.704+0000 7fc010a7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:53.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.704+0000 7fc010a7d640 1 -- 172.21.15.40:0/699790687 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc00c09ee10 msgr2=0x7fc00c0a32b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:53.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.704+0000 7fc010a7d640 1 --2- 172.21.15.40:0/699790687 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc00c09ee10 0x7fc00c0a32b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc004009e50 tx=0x7fc00402bdc0 comp rx=0 tx=0).stop 2024-04-22T08:12:53.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.704+0000 7fc010a7d640 1 -- 172.21.15.40:0/699790687 shutdown_connections 2024-04-22T08:12:53.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.704+0000 7fc010a7d640 1 --2- 172.21.15.40:0/699790687 >> v2:172.21.15.40:3300/0 conn(0x7fc00c0a1d30 0x7fc00c0a2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:53.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.704+0000 7fc010a7d640 1 --2- 172.21.15.40:0/699790687 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc00c09ee10 0x7fc00c0a32b0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:53.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.704+0000 7fc010a7d640 1 -- 172.21.15.40:0/699790687 >> 172.21.15.40:0/699790687 conn(0x7fc00c0a4ef0 msgr2=0x7fc00c0a5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:53.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.704+0000 7fc010a7d640 1 -- 172.21.15.40:0/699790687 shutdown_connections 2024-04-22T08:12:53.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.704+0000 7fc010a7d640 1 -- 172.21.15.40:0/699790687 wait complete. 2024-04-22T08:12:53.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.705+0000 7fc010a7d640 10 monclient: build_initial_monmap 2024-04-22T08:12:53.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.705+0000 7fc010a7d640 10 monclient: monmap: 2024-04-22T08:12:53.736 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:53.736 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:12:53.736 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:12:53.736 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:12:53.736 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:53.737 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:53.737 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:12:53.737 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:12:53.737 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:12:53.737 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:53.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.705+0000 7fc010a7d640 1 Processor -- start 2024-04-22T08:12:53.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.705+0000 7fc010a7d640 1 -- start start 2024-04-22T08:12:53.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.705+0000 7fc010a7d640 10 monclient: init 2024-04-22T08:12:53.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc010a7d640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:53.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc010a7d640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-22T08:12:53.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc010a7d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc00c09ee10 0x7fc00c1cadd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:53.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc010a7d640 10 monclient(hunting): picked mon.noname-b con 0x7fc00c09ee10 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:53.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc010a7d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc00c0a1d30 0x7fc00c1cb310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:53.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc010a7d640 10 monclient(hunting): picked mon.noname-c con 0x7fc00c0a1d30 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:53.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc010a7d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc00c135d90 0x7fc00c122130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:53.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc010a7d640 10 monclient(hunting): picked mon.noname-a con 0x7fc00c135d90 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:53.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc010a7d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:53.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc010a7d640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc00c134e00 con 0x7fc00c135d90 2024-04-22T08:12:53.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc010a7d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:53.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc010a7d640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc00c125810 con 0x7fc00c09ee10 2024-04-22T08:12:53.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc010a7d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:53.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc010a7d640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc00c1260e0 con 0x7fc00c0a1d30 2024-04-22T08:12:53.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc010a7d640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:53.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc010a7d640 10 monclient(hunting): authenticate will time out at 847.700039s 2024-04-22T08:12:53.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc009d9b640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc00c0a1d30 0x7fc00c1cb310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:53.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc009d9b640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc00c0a1d30 0x7fc00c1cb310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49792/0 (socket says 172.21.15.40:49792) 2024-04-22T08:12:53.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc009d9b640 1 -- 172.21.15.40:0/2117605109 learned_addr learned my addr 172.21.15.40:0/2117605109 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:53.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc009d9b640 10 monclient(hunting): get_auth_request con 0x7fc00c0a1d30 auth_method 0 2024-04-22T08:12:53.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc009d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:53.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc009d9b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:53.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc009d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:53.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc00ad9d640 1 --2- 172.21.15.40:0/2117605109 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc00c135d90 0x7fc00c122130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:53.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc009d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:53.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc009d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:53.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc009d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:53.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc00ad9d640 10 monclient(hunting): get_auth_request con 0x7fc00c135d90 auth_method 0 2024-04-22T08:12:53.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc00ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:53.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc00ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:53.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc00ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:53.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc00a59c640 1 --2- 172.21.15.40:0/2117605109 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc00c09ee10 0x7fc00c1cadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:53.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc00a59c640 10 monclient(hunting): get_auth_request con 0x7fc00c09ee10 auth_method 0 2024-04-22T08:12:53.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc00a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:53.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc00a59c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:53.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc00a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:53.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc00ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:53.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc00ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:53.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc00ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:53.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc00a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:53.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc00a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:53.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc00a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:53.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc009d9b640 10 monclient(hunting): handle_auth_done global_id 4274 payload 824 2024-04-22T08:12:53.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc009d9b640 1 -- 172.21.15.40:0/2117605109 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc00c09ee10 msgr2=0x7fc00c1cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:53.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc009d9b640 1 --2- 172.21.15.40:0/2117605109 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc00c09ee10 0x7fc00c1cadd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:53.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc009d9b640 1 -- 172.21.15.40:0/2117605109 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc00c135d90 msgr2=0x7fc00c122130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:53.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc009d9b640 1 --2- 172.21.15.40:0/2117605109 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc00c135d90 0x7fc00c122130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:53.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc009d9b640 10 monclient: _finish_hunting 0 2024-04-22T08:12:53.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc009d9b640 1 monclient: found mon.noname-c 2024-04-22T08:12:53.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc009d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:53.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc009d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:12:53.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.706+0000 7fc009d9b640 1 -- 172.21.15.40:0/2117605109 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc000009400 con 0x7fc00c0a1d30 2024-04-22T08:12:53.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.708+0000 7fc009d9b640 1 -- 172.21.15.40:0/2117605109 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc00c0a1d30 msgr2=0x7fc00c1cb310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:53.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.708+0000 7fc009d9b640 1 --2- 172.21.15.40:0/2117605109 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc00c0a1d30 0x7fc00c1cb310 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc00000ebe0 tx=0x7fc00000c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:53.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.709+0000 7fbffaffd640 1 -- 172.21.15.40:0/2117605109 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc000009760 con 0x7fc00c0a1d30 2024-04-22T08:12:53.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.709+0000 7fbffaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:53.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.709+0000 7fbffaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:12:53.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.709+0000 7fbffaffd640 10 monclient: dump: 2024-04-22T08:12:53.749 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:53.749 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:53.749 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:53.749 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:53.749 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:53.750 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:53.750 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:53.750 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:53.750 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:53.750 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:53.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.709+0000 7fbffaffd640 10 monclient: _finish_auth 0 2024-04-22T08:12:53.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.709+0000 7fbffaffd640 10 monclient: _check_auth_tickets 2024-04-22T08:12:53.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.709+0000 7fbffaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:53.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.709+0000 7fbffaffd640 1 -- 172.21.15.40:0/2117605109 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc00000ce40 con 0x7fc00c0a1d30 2024-04-22T08:12:53.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.709+0000 7fc010a7d640 5 monclient: authenticate success, global_id 4274 2024-04-22T08:12:53.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.709+0000 7fc010a7d640 10 monclient: _renew_subs 2024-04-22T08:12:53.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.709+0000 7fc010a7d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:53.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.709+0000 7fc010a7d640 1 -- 172.21.15.40:0/2117605109 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc004009b30 con 0x7fc00c0a1d30 2024-04-22T08:12:53.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.709+0000 7fbffaffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:53.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.709+0000 7fbffaffd640 1 -- 172.21.15.40:0/2117605109 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc00000eee0 con 0x7fc00c0a1d30 2024-04-22T08:12:53.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.709+0000 7fbffaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:53.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.709+0000 7fbffaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:12:53.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.709+0000 7fbffaffd640 10 monclient: dump: 2024-04-22T08:12:53.753 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:53.753 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:53.753 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:53.753 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:53.754 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:53.754 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:53.754 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:53.754 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:53.754 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:53.755 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:53.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.709+0000 7fc008d99640 4 set_mon_vals no callback set 2024-04-22T08:12:53.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.709+0000 7fc010a7d640 10 monclient: _renew_subs 2024-04-22T08:12:53.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.709+0000 7fc010a7d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:53.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.709+0000 7fc010a7d640 1 -- 172.21.15.40:0/2117605109 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc00c123ce0 con 0x7fc00c0a1d30 2024-04-22T08:12:53.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.710+0000 7fc010a7d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:53.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.710+0000 7fc010a7d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:53.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.710+0000 7fc010a7d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:53.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.710+0000 7fc010a7d640 1 -- 172.21.15.40:0/2117605109 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc00c04e9b0 con 0x7fc00c0a1d30 2024-04-22T08:12:53.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.710+0000 7fbffaffd640 1 -- 172.21.15.40:0/2117605109 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fc000013690 con 0x7fc00c0a1d30 2024-04-22T08:12:53.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.710+0000 7fbffaffd640 1 --2- 172.21.15.40:0/2117605109 >> v2:172.21.15.189:6800/30168 conn(0x7fbfec037800 0x7fbfec039cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:53.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.710+0000 7fbffaffd640 1 -- 172.21.15.40:0/2117605109 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3506+0+0 (secure 0 0 0) 0x7fc00004a5e0 con 0x7fc00c0a1d30 2024-04-22T08:12:53.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.710+0000 7fc00a59c640 1 --2- 172.21.15.40:0/2117605109 >> v2:172.21.15.189:6800/30168 conn(0x7fbfec037800 0x7fbfec039cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:53.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.710+0000 7fc00a59c640 10 monclient: get_auth_request con 0x7fbfec037800 auth_method 0 2024-04-22T08:12:53.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.713+0000 7fc00a59c640 1 --2- 172.21.15.40:0/2117605109 >> v2:172.21.15.189:6800/30168 conn(0x7fbfec037800 0x7fbfec039cc0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc004032000 tx=0x7fc004030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:53.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.715+0000 7fbffaffd640 1 -- 172.21.15.40:0/2117605109 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7fc0000238e0 con 0x7fc00c0a1d30 2024-04-22T08:12:53.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.715+0000 7fbffaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:53.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.715+0000 7fbffaffd640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:12:53.758 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-22T08:12:53.758 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-22T08:12:53.768 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-22T08:12:53.791 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-22T08:12:53.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.874+0000 7f0886bb9640 10 monclient: get_monmap_and_config 2024-04-22T08:12:53.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.874+0000 7f0886bb9640 10 monclient: build_initial_monmap 2024-04-22T08:12:53.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.874+0000 7f0886bb9640 10 monclient: monmap: 2024-04-22T08:12:53.874 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:53.874 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:53.875 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:53.875074+0000 2024-04-22T08:12:53.875 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:53.875074+0000 2024-04-22T08:12:53.875 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:53.875 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:53.875 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:12:53.875 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:12:53.875 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:12:53.875 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:12:53.875 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:12:53.875 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:53.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.875+0000 7f0886bb9640 1 Processor -- start 2024-04-22T08:12:53.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.875+0000 7f0886bb9640 1 -- start start 2024-04-22T08:12:53.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.875+0000 7f0886bb9640 10 monclient: init 2024-04-22T08:12:53.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.875+0000 7f0886bb9640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:53.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.875+0000 7f0886bb9640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-04-22T08:12:53.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.875+0000 7f0886bb9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f088009ee10 addr v1:172.21.15.40:6789/0 2024-04-22T08:12:53.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.875+0000 7f0886bb9640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f08800a1f30 0x7f08800a3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:53.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.875+0000 7f0886bb9640 10 monclient(hunting): picked mon.noname-b con 0x7f08800a1f30 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:53.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.876+0000 7f087ffff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f08800a1f30 0x7f08800a3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:53.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.876+0000 7f0886bb9640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f088013b0e0 0x7f088013d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:53.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.876+0000 7f0886bb9640 10 monclient(hunting): picked mon.noname-c con 0x7f088013b0e0 addr v2:172.21.15.189:3300/0 2024-04-22T08:12:53.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.876+0000 7f0884955640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f088009ee10 0x7f08800a1820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:38944/0 (socket says 172.21.15.40:38944) 2024-04-22T08:12:53.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.876+0000 7f0884955640 1 -- 172.21.15.40:0/3617703988 learned_addr learned my addr 172.21.15.40:0/3617703988 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:12:53.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.876+0000 7f0886bb9640 1 -- 172.21.15.40:0/3617703988 --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08800a4470 con 0x7f088009ee10 2024-04-22T08:12:53.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.876+0000 7f0886bb9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:53.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.876+0000 7f0886bb9640 1 -- 172.21.15.40:0/3617703988 --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08800a2310 con 0x7f088013b0e0 2024-04-22T08:12:53.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.876+0000 7f0886bb9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:53.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.876+0000 7f0886bb9640 1 -- 172.21.15.40:0/3617703988 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f088009f1f0 con 0x7f08800a1f30 2024-04-22T08:12:53.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.876+0000 7f0886bb9640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:53.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.876+0000 7f0886bb9640 10 monclient(hunting): authenticate will time out at 847.869994s 2024-04-22T08:12:53.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.876+0000 7f0885156640 1 --2- 172.21.15.40:0/3617703988 >> v2:172.21.15.189:3300/0 conn(0x7f088013b0e0 0x7f088013d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:53.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.876+0000 7f0885156640 10 monclient(hunting): get_auth_request con 0x7f088013b0e0 auth_method 0 2024-04-22T08:12:53.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.876+0000 7f0885156640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:53.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.876+0000 7f0885156640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:53.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.876+0000 7f0885156640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:53.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.876+0000 7f087ffff640 10 monclient(hunting): get_auth_request con 0x7f08800a1f30 auth_method 0 2024-04-22T08:12:53.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.876+0000 7f087ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:53.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.876+0000 7f087ffff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:53.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.876+0000 7f087ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:53.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.876+0000 7f087ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:53.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.876+0000 7f087ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:53.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.876+0000 7f087ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:53.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.877+0000 7f0885156640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:53.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.877+0000 7f0885156640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:53.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.877+0000 7f0885156640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:53.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.877+0000 7f0885156640 10 monclient(hunting): handle_auth_done global_id 4267 payload 274 2024-04-22T08:12:53.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.877+0000 7f0885156640 1 -- 172.21.15.40:0/3617703988 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f08800a1f30 msgr2=0x7f08800a3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:53.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.877+0000 7f0885156640 1 --2- 172.21.15.40:0/3617703988 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f08800a1f30 0x7f08800a3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:53.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.877+0000 7f0885156640 1 -- 172.21.15.40:0/3617703988 >> v1:172.21.15.40:6789/0 conn(0x7f088009ee10 legacy=0x7f08800a1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:53.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.877+0000 7f0885156640 10 monclient: _finish_hunting 0 2024-04-22T08:12:53.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.877+0000 7f0885156640 1 monclient: found mon.noname-c 2024-04-22T08:12:53.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.877+0000 7f0885156640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:53.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.877+0000 7f0885156640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.189:3300/0 2024-04-22T08:12:53.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.877+0000 7f0885156640 1 -- 172.21.15.40:0/3617703988 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f088009fd30 con 0x7f088013b0e0 2024-04-22T08:12:53.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.883+0000 7f3f55244640 10 monclient: get_monmap_and_config 2024-04-22T08:12:53.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.883+0000 7f3f55244640 10 monclient: build_initial_monmap 2024-04-22T08:12:53.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.883+0000 7f3f55244640 10 monclient: monmap: 2024-04-22T08:12:53.882 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:53.883 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:53.883 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:53.883952+0000 2024-04-22T08:12:53.883 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:53.883952+0000 2024-04-22T08:12:53.883 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:53.883 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:53.884 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:12:53.884 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:12:53.884 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:12:53.884 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:12:53.884 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:12:53.884 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:53.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.883+0000 7f3f55244640 1 Processor -- start 2024-04-22T08:12:53.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.883+0000 7f0885156640 1 -- 172.21.15.40:0/3617703988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f088013b0e0 msgr2=0x7f088013d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:53.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.883+0000 7f0885156640 1 --2- 172.21.15.40:0/3617703988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f088013b0e0 0x7f088013d4d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0878009d90 tx=0x7f087802bd20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4996081d0b868176 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:53.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.884+0000 7f3f55244640 1 -- start start 2024-04-22T08:12:53.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.884+0000 7f3f55244640 10 monclient: init 2024-04-22T08:12:53.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.884+0000 7f087f7fe640 1 -- 172.21.15.40:0/3617703988 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087802cce0 con 0x7f088013b0e0 2024-04-22T08:12:53.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.884+0000 7f087f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:53.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.884+0000 7f087f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:12:53.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.884+0000 7f087f7fe640 10 monclient: dump: 2024-04-22T08:12:53.889 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:53.889 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:53.889 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:53.889 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:53.889 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:53.889 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:53.889 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:53.889 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:53.889 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:53.890 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:53.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.884+0000 7f087f7fe640 10 monclient: _finish_auth 0 2024-04-22T08:12:53.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.884+0000 7f087f7fe640 10 monclient: _check_auth_tickets 2024-04-22T08:12:53.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.884+0000 7f087f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:53.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.884+0000 7f087f7fe640 1 -- 172.21.15.40:0/3617703988 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f087802ce40 con 0x7f088013b0e0 2024-04-22T08:12:53.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.884+0000 7f087f7fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:53.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.884+0000 7f087f7fe640 1 -- 172.21.15.40:0/3617703988 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0878034660 con 0x7f088013b0e0 2024-04-22T08:12:53.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.884+0000 7f087f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:53.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.884+0000 7f087f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:12:53.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.884+0000 7f087f7fe640 10 monclient: dump: 2024-04-22T08:12:53.890 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:53.890 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:53.890 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:53.891 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:53.891 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:53.891 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:53.891 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:53.891 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:53.891 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:53.891 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:53.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.884+0000 7f0886bb9640 5 monclient: authenticate success, global_id 4267 2024-04-22T08:12:53.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.884+0000 7f0886bb9640 10 monclient: get_monmap_and_config success 2024-04-22T08:12:53.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.884+0000 7f0886bb9640 4 set_mon_vals no callback set 2024-04-22T08:12:53.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.884+0000 7f3f55244640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:53.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.884+0000 7f3f55244640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-04-22T08:12:53.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.884+0000 7f3f55244640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3f5009ee10 0x7f3f500a3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:53.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.884+0000 7f3f55244640 10 monclient(hunting): picked mon.noname-c con 0x7f3f5009ee10 addr v2:172.21.15.189:3300/0 2024-04-22T08:12:53.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.884+0000 7f3f55244640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3f500a1820 addr v1:172.21.15.189:6789/0 2024-04-22T08:12:53.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.884+0000 7f3f55244640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3f5013b0e0 0x7f3f5013d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:53.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.884+0000 7f3f55244640 10 monclient(hunting): picked mon.noname-a con 0x7f3f5013b0e0 addr v2:172.21.15.40:3300/0 2024-04-22T08:12:53.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.884+0000 7f3f55244640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f500a4470 con 0x7f3f500a1820 2024-04-22T08:12:53.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.884+0000 7f3f55244640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:53.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.884+0000 7f3f55244640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f5009f1f0 con 0x7f3f5013b0e0 2024-04-22T08:12:53.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.884+0000 7f3f55244640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:53.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.884+0000 7f3f55244640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f5009fd30 con 0x7f3f5009ee10 2024-04-22T08:12:53.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.884+0000 7f3f55244640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:53.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.885+0000 7f0886bb9640 10 monclient: shutdown 2024-04-22T08:12:53.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.884+0000 7f3f55244640 10 monclient(hunting): authenticate will time out at 847.878636s 2024-04-22T08:12:53.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.885+0000 7f0886bb9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:53.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.885+0000 7f0886bb9640 1 -- 172.21.15.40:0/3617703988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f088013b0e0 msgr2=0x7f088013d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:53.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.885+0000 7f0886bb9640 1 --2- 172.21.15.40:0/3617703988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f088013b0e0 0x7f088013d4d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0878009d90 tx=0x7f087802bd20 comp rx=0 tx=0).stop 2024-04-22T08:12:53.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.885+0000 7f3f4f59e640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3f5013b0e0 0x7f3f5013d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:53.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.885+0000 7f3f4f59e640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3f5013b0e0 0x7f3f5013d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58004/0 (socket says 172.21.15.40:58004) 2024-04-22T08:12:53.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.885+0000 7f3f4f59e640 1 -- 172.21.15.40:0/691819748 learned_addr learned my addr 172.21.15.40:0/691819748 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:53.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.885+0000 7f3f4f59e640 10 monclient(hunting): get_auth_request con 0x7f3f5013b0e0 auth_method 0 2024-04-22T08:12:53.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.885+0000 7f3f4f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:53.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.885+0000 7f3f4f59e640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:53.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.885+0000 7f3f4f59e640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:53.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.885+0000 7f3f4ed9d640 1 --2- 172.21.15.40:0/691819748 >> v2:172.21.15.189:3300/0 conn(0x7f3f5009ee10 0x7f3f500a3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:53.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.885+0000 7f3f4f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:53.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.885+0000 7f3f4f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:53.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.885+0000 7f3f4f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:53.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.885+0000 7f3f4ed9d640 10 monclient(hunting): get_auth_request con 0x7f3f5009ee10 auth_method 0 2024-04-22T08:12:53.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.885+0000 7f3f4ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:53.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.885+0000 7f0886bb9640 1 -- 172.21.15.40:0/3617703988 shutdown_connections 2024-04-22T08:12:53.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.885+0000 7f3f4ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:53.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.885+0000 7f3f4ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:53.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.885+0000 7f3f4ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:53.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.885+0000 7f3f4ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:53.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.885+0000 7f3f4ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:53.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.885+0000 7f3f4f59e640 10 monclient(hunting): handle_auth_done global_id 4296 payload 274 2024-04-22T08:12:53.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.885+0000 7f3f4f59e640 1 -- 172.21.15.40:0/691819748 >> v2:172.21.15.189:3300/0 conn(0x7f3f5009ee10 msgr2=0x7f3f500a3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:53.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.885+0000 7f0886bb9640 1 --2- 172.21.15.40:0/3617703988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f088013b0e0 0x7f088013d4d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:53.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.885+0000 7f3f4f59e640 1 --2- 172.21.15.40:0/691819748 >> v2:172.21.15.189:3300/0 conn(0x7f3f5009ee10 0x7f3f500a3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:53.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.885+0000 7f3f4f59e640 1 -- 172.21.15.40:0/691819748 >> v1:172.21.15.189:6789/0 conn(0x7f3f500a1820 legacy=0x7f3f500a1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:53.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.885+0000 7f3f4f59e640 10 monclient: _finish_hunting 0 2024-04-22T08:12:53.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.885+0000 7f3f4f59e640 1 monclient: found mon.noname-a 2024-04-22T08:12:53.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.885+0000 7f3f4f59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:53.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.885+0000 7f3f4f59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:12:53.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.885+0000 7f3f4f59e640 1 -- 172.21.15.40:0/691819748 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f500a0010 con 0x7f3f5013b0e0 2024-04-22T08:12:53.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.885+0000 7f0886bb9640 1 --2- 172.21.15.40:0/3617703988 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f08800a1f30 0x7f08800a3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:53.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.885+0000 7f0886bb9640 1 -- 172.21.15.40:0/3617703988 >> 172.21.15.40:0/3617703988 conn(0x7f08800a4e80 msgr2=0x7f08800a5290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:53.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.885+0000 7f3f4ed9d640 1 --2- 172.21.15.40:0/691819748 >> v2:172.21.15.189:3300/0 conn(0x7f3f5009ee10 0x7f3f500a3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:12:53.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.885+0000 7f0886bb9640 1 -- 172.21.15.40:0/3617703988 shutdown_connections 2024-04-22T08:12:53.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.885+0000 7f0886bb9640 1 -- 172.21.15.40:0/3617703988 wait complete. 2024-04-22T08:12:53.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.886+0000 7f0886bb9640 10 monclient: build_initial_monmap 2024-04-22T08:12:53.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.887+0000 7f0886bb9640 10 monclient: monmap: 2024-04-22T08:12:53.901 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:53.901 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:12:53.901 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:12:53.901 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:12:53.901 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:53.901 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:53.901 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:12:53.902 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:12:53.902 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:12:53.902 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:53.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.887+0000 7f0886bb9640 1 Processor -- start 2024-04-22T08:12:53.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.887+0000 7f0886bb9640 1 -- start start 2024-04-22T08:12:53.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.887+0000 7f0886bb9640 10 monclient: init 2024-04-22T08:12:53.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.888+0000 7f0886bb9640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:53.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.888+0000 7f0886bb9640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:12:53.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.888+0000 7f0886bb9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f088009ee10 0x7f08801cb020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:53.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.889+0000 7f0886bb9640 10 monclient(hunting): picked mon.noname-a con 0x7f088009ee10 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:53.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.889+0000 7f0886bb9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08800a1f30 0x7f0880120590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:53.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.889+0000 7f0884955640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f088009ee10 0x7f08801cb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:53.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.889+0000 7f3f4f59e640 1 -- 172.21.15.40:0/691819748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f5013b0e0 msgr2=0x7f3f5013d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:53.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.889+0000 7f3f4f59e640 1 --2- 172.21.15.40:0/691819748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f5013b0e0 0x7f3f5013d4d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3f500a5ea0 tx=0x7f3f4802c020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa878d1a48672c47 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:53.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.889+0000 7f0884955640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f088009ee10 0x7f08801cb020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58020/0 (socket says 172.21.15.40:58020) 2024-04-22T08:12:53.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.889+0000 7f3f4dd9b640 1 -- 172.21.15.40:0/691819748 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f4802cbc0 con 0x7f3f5013b0e0 2024-04-22T08:12:53.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.889+0000 7f3f4dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:53.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.889+0000 7f0886bb9640 10 monclient(hunting): picked mon.noname-b con 0x7f08800a1f30 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:53.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.889+0000 7f0884955640 1 -- 172.21.15.40:0/2463889770 learned_addr learned my addr 172.21.15.40:0/2463889770 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:53.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.889+0000 7f3f4dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:12:53.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.889+0000 7f3f4dd9b640 10 monclient: dump: 2024-04-22T08:12:53.905 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:53.905 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:53.905 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:53.905 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:53.905 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:53.905 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:53.905 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:53.905 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:53.905 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:53.905 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:53.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.889+0000 7f0886bb9640 1 --2- 172.21.15.40:0/2463889770 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f088013b0e0 0x7f0880120ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:53.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.889+0000 7f3f4dd9b640 10 monclient: _finish_auth 0 2024-04-22T08:12:53.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.889+0000 7f3f4dd9b640 10 monclient: _check_auth_tickets 2024-04-22T08:12:53.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.889+0000 7f3f4dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:53.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.889+0000 7f0886bb9640 10 monclient(hunting): picked mon.noname-c con 0x7f088013b0e0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:53.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.889+0000 7f0886bb9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:53.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.889+0000 7f0886bb9640 1 -- 172.21.15.40:0/2463889770 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08801243e0 con 0x7f088009ee10 2024-04-22T08:12:53.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.889+0000 7f3f4dd9b640 1 -- 172.21.15.40:0/691819748 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f4802cd20 con 0x7f3f5013b0e0 2024-04-22T08:12:53.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.889+0000 7f3f4dd9b640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:53.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.889+0000 7f0886bb9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:53.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.889+0000 7f3f4dd9b640 1 -- 172.21.15.40:0/691819748 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f48034610 con 0x7f3f5013b0e0 2024-04-22T08:12:53.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.889+0000 7f0886bb9640 1 -- 172.21.15.40:0/2463889770 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0880124170 con 0x7f08800a1f30 2024-04-22T08:12:53.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.889+0000 7f3f4dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:53.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.889+0000 7f0886bb9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:53.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.889+0000 7f0886bb9640 1 -- 172.21.15.40:0/2463889770 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08800a4260 con 0x7f088013b0e0 2024-04-22T08:12:53.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.889+0000 7f3f4dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:12:53.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.889+0000 7f0886bb9640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:53.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.889+0000 7f3f4dd9b640 10 monclient: dump: 2024-04-22T08:12:53.909 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:53.910 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:53.910 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:53.910 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:53.910 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:53.910 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:53.910 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:53.910 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:53.910 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:53.910 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:53.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.889+0000 7f087ffff640 1 --2- 172.21.15.40:0/2463889770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08800a1f30 0x7f0880120590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:53.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.889+0000 7f0886bb9640 10 monclient(hunting): authenticate will time out at 847.883596s 2024-04-22T08:12:53.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.889+0000 7f3f55244640 5 monclient: authenticate success, global_id 4296 2024-04-22T08:12:53.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.889+0000 7f0884955640 10 monclient(hunting): get_auth_request con 0x7f088009ee10 auth_method 0 2024-04-22T08:12:53.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.889+0000 7f0884955640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:53.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.889+0000 7f0884955640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:53.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.889+0000 7f0884955640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:53.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.889+0000 7f087ffff640 10 monclient(hunting): get_auth_request con 0x7f08800a1f30 auth_method 0 2024-04-22T08:12:53.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.889+0000 7f087ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:53.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.889+0000 7f087ffff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:53.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.889+0000 7f087ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:53.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.889+0000 7f3f55244640 10 monclient: get_monmap_and_config success 2024-04-22T08:12:53.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.889+0000 7f3f55244640 4 set_mon_vals no callback set 2024-04-22T08:12:53.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.890+0000 7f0885156640 1 --2- 172.21.15.40:0/2463889770 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f088013b0e0 0x7f0880120ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:53.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.890+0000 7f087ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:53.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.890+0000 7f087ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:53.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.890+0000 7f087ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:53.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.890+0000 7f0885156640 10 monclient(hunting): get_auth_request con 0x7f088013b0e0 auth_method 0 2024-04-22T08:12:53.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.890+0000 7f0885156640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:53.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.890+0000 7f0885156640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:53.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.890+0000 7f0885156640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:53.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.890+0000 7f0884955640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:53.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.890+0000 7f0884955640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:53.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.890+0000 7f0884955640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:53.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.890+0000 7f0885156640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:53.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.890+0000 7f0885156640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:53.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.890+0000 7f0885156640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:53.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.890+0000 7f3f55244640 10 monclient: shutdown 2024-04-22T08:12:53.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.890+0000 7f3f55244640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:53.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.890+0000 7f087ffff640 10 monclient(hunting): handle_auth_done global_id 4273 payload 824 2024-04-22T08:12:53.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.890+0000 7f3f55244640 1 -- 172.21.15.40:0/691819748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f5013b0e0 msgr2=0x7f3f5013d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:53.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.890+0000 7f3f55244640 1 --2- 172.21.15.40:0/691819748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f5013b0e0 0x7f3f5013d4d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3f500a5ea0 tx=0x7f3f4802c020 comp rx=0 tx=0).stop 2024-04-22T08:12:53.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.890+0000 7f087ffff640 1 -- 172.21.15.40:0/2463889770 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f088013b0e0 msgr2=0x7f0880120ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:53.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.891+0000 7f3f55244640 1 -- 172.21.15.40:0/691819748 shutdown_connections 2024-04-22T08:12:53.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.891+0000 7f087ffff640 1 --2- 172.21.15.40:0/2463889770 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f088013b0e0 0x7f0880120ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:53.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.891+0000 7f3f55244640 1 --2- 172.21.15.40:0/691819748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f5013b0e0 0x7f3f5013d4d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:53.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.891+0000 7f3f55244640 1 --2- 172.21.15.40:0/691819748 >> v2:172.21.15.189:3300/0 conn(0x7f3f5009ee10 0x7f3f500a3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:53.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.891+0000 7f087ffff640 1 -- 172.21.15.40:0/2463889770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f088009ee10 msgr2=0x7f08801cb020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:53.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.891+0000 7f087ffff640 1 --2- 172.21.15.40:0/2463889770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f088009ee10 0x7f08801cb020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:53.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.891+0000 7f087ffff640 10 monclient: _finish_hunting 0 2024-04-22T08:12:53.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.891+0000 7f087ffff640 1 monclient: found mon.noname-b 2024-04-22T08:12:53.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.891+0000 7f087ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:53.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.891+0000 7f087ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.189:3300/0 2024-04-22T08:12:53.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.891+0000 7f087ffff640 1 -- 172.21.15.40:0/2463889770 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0878009a70 con 0x7f08800a1f30 2024-04-22T08:12:53.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.891+0000 7f3f55244640 1 -- 172.21.15.40:0/691819748 >> 172.21.15.40:0/691819748 conn(0x7f3f500a4e80 msgr2=0x7f3f500a5290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:53.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.891+0000 7f0884955640 1 --2- 172.21.15.40:0/2463889770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f088009ee10 0x7f08801cb020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:12:53.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.891+0000 7f0885156640 1 --2- 172.21.15.40:0/2463889770 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f088013b0e0 0x7f0880120ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:12:53.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.891+0000 7f3f55244640 1 -- 172.21.15.40:0/691819748 shutdown_connections 2024-04-22T08:12:53.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.891+0000 7f3f55244640 1 -- 172.21.15.40:0/691819748 wait complete. 2024-04-22T08:12:53.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.892+0000 7f3f55244640 10 monclient: build_initial_monmap 2024-04-22T08:12:53.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.892+0000 7f3f55244640 10 monclient: monmap: 2024-04-22T08:12:53.920 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:53.920 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:12:53.920 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:12:53.920 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:12:53.920 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:53.920 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:53.920 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:12:53.920 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:12:53.920 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:12:53.920 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:53.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.893+0000 7f087ffff640 1 -- 172.21.15.40:0/2463889770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08800a1f30 msgr2=0x7f0880120590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:53.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.893+0000 7f087ffff640 1 --2- 172.21.15.40:0/2463889770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08800a1f30 0x7f0880120590 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f086c00e870 tx=0x7f086c00ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:53.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.893+0000 7f3f55244640 1 Processor -- start 2024-04-22T08:12:53.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.893+0000 7f3f55244640 1 -- start start 2024-04-22T08:12:53.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.893+0000 7f087d7fa640 1 -- 172.21.15.40:0/2463889770 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086c00cb50 con 0x7f08800a1f30 2024-04-22T08:12:53.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.893+0000 7f087d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:53.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.893+0000 7f087d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:12:53.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.893+0000 7f087d7fa640 10 monclient: dump: 2024-04-22T08:12:53.922 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:53.923 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:53.923 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:53.923 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:53.923 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:53.923 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:53.923 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:53.923 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:53.923 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:53.923 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:53.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.893+0000 7f087d7fa640 10 monclient: _finish_auth 0 2024-04-22T08:12:53.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.893+0000 7f087d7fa640 10 monclient: _check_auth_tickets 2024-04-22T08:12:53.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.893+0000 7f087d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:53.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.893+0000 7f087d7fa640 1 -- 172.21.15.40:0/2463889770 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f086c00ccb0 con 0x7f08800a1f30 2024-04-22T08:12:53.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.893+0000 7f087d7fa640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:53.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.893+0000 7f087d7fa640 1 -- 172.21.15.40:0/2463889770 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086c013670 con 0x7f08800a1f30 2024-04-22T08:12:53.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.893+0000 7f087d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:53.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.893+0000 7f087d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:12:53.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.893+0000 7f087d7fa640 10 monclient: dump: 2024-04-22T08:12:53.924 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:53.924 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:53.924 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:53.924 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:53.924 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:53.924 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:53.924 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:53.924 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:53.924 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:53.924 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:53.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.893+0000 7f087effd640 4 set_mon_vals no callback set 2024-04-22T08:12:53.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.894+0000 7f0886bb9640 5 monclient: authenticate success, global_id 4273 2024-04-22T08:12:53.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.894+0000 7f0886bb9640 10 monclient: _renew_subs 2024-04-22T08:12:53.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.894+0000 7f0886bb9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.189:3300/0 2024-04-22T08:12:53.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.894+0000 7f0886bb9640 1 -- 172.21.15.40:0/2463889770 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f088013cdb0 con 0x7f08800a1f30 2024-04-22T08:12:53.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.894+0000 7f3f55244640 10 monclient: init 2024-04-22T08:12:53.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.894+0000 7f0886bb9640 10 monclient: _renew_subs 2024-04-22T08:12:53.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.894+0000 7f0886bb9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.189:3300/0 2024-04-22T08:12:53.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.894+0000 7f0886bb9640 1 -- 172.21.15.40:0/2463889770 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0880122010 con 0x7f08800a1f30 2024-04-22T08:12:53.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.895+0000 7f3f55244640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:53.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.895+0000 7f3f55244640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:12:53.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.895+0000 7f3f55244640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f5009ee10 0x7f3f501c68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:53.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.895+0000 7f3f55244640 10 monclient(hunting): picked mon.noname-a con 0x7f3f5009ee10 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:53.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.895+0000 7f3f55244640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f500a1820 0x7f3f5011e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:53.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.895+0000 7f3f55244640 10 monclient(hunting): picked mon.noname-b con 0x7f3f500a1820 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:53.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.895+0000 7f3f55244640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f5013b0e0 0x7f3f5011ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:53.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.895+0000 7f3f4e59c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f500a1820 0x7f3f5011e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:53.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.895+0000 7f3f4e59c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f500a1820 0x7f3f5011e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.40:42844/0 (socket says 172.21.15.40:42844) 2024-04-22T08:12:53.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.895+0000 7f3f4e59c640 1 -- 172.21.15.40:0/3439132168 learned_addr learned my addr 172.21.15.40:0/3439132168 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:53.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.895+0000 7f3f55244640 10 monclient(hunting): picked mon.noname-c con 0x7f3f5013b0e0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:53.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.895+0000 7f3f4f59e640 1 --2- 172.21.15.40:0/3439132168 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f5013b0e0 0x7f3f5011ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:53.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.895+0000 7f087d7fa640 1 -- 172.21.15.40:0/2463889770 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f086c0137d0 con 0x7f08800a1f30 2024-04-22T08:12:53.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.895+0000 7f087d7fa640 1 --2- 172.21.15.40:0/2463889770 >> v2:172.21.15.189:6800/30168 conn(0x7f0864037850 0x7f0864039d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:53.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.895+0000 7f087d7fa640 1 -- 172.21.15.40:0/2463889770 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f086c04a3e0 con 0x7f08800a1f30 2024-04-22T08:12:53.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.895+0000 7f3f4ed9d640 1 --2- 172.21.15.40:0/3439132168 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f5009ee10 0x7f3f501c68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:53.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.895+0000 7f0884955640 1 --2- 172.21.15.40:0/2463889770 >> v2:172.21.15.189:6800/30168 conn(0x7f0864037850 0x7f0864039d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:53.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.895+0000 7f0884955640 10 monclient: get_auth_request con 0x7f0864037850 auth_method 0 2024-04-22T08:12:53.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.895+0000 7f3f55244640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:53.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.896+0000 7f3f55244640 1 -- 172.21.15.40:0/3439132168 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f50121be0 con 0x7f3f5009ee10 2024-04-22T08:12:53.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.896+0000 7f3f55244640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:53.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.896+0000 7f0886bb9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:53.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.896+0000 7f0886bb9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:53.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.896+0000 7f0886bb9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.189:3300/0 2024-04-22T08:12:53.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.896+0000 7f0886bb9640 1 -- 172.21.15.40:0/2463889770 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08801229f0 con 0x7f08800a1f30 2024-04-22T08:12:53.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.896+0000 7f3f55244640 1 -- 172.21.15.40:0/3439132168 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f501223f0 con 0x7f3f500a1820 2024-04-22T08:12:53.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.896+0000 7f3f55244640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:53.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.896+0000 7f3f55244640 1 -- 172.21.15.40:0/3439132168 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f500a4260 con 0x7f3f5013b0e0 2024-04-22T08:12:53.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.896+0000 7f3f55244640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:53.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.896+0000 7f3f55244640 10 monclient(hunting): authenticate will time out at 847.890498s 2024-04-22T08:12:53.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.896+0000 7f3f4e59c640 10 monclient(hunting): get_auth_request con 0x7f3f500a1820 auth_method 0 2024-04-22T08:12:53.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.896+0000 7f3f4e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:53.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.896+0000 7f3f4e59c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:53.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.897+0000 7f3f4e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:53.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.897+0000 7f3f4f59e640 10 monclient(hunting): get_auth_request con 0x7f3f5013b0e0 auth_method 0 2024-04-22T08:12:53.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.897+0000 7f3f4f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:53.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.897+0000 7f3f4f59e640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:53.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.897+0000 7f3f4f59e640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:53.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.897+0000 7f3f4ed9d640 10 monclient(hunting): get_auth_request con 0x7f3f5009ee10 auth_method 0 2024-04-22T08:12:53.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.897+0000 7f3f4ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:53.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.897+0000 7f3f4ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:53.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.897+0000 7f3f4ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:53.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.897+0000 7f3f4f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:53.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.897+0000 7f3f4f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:53.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.897+0000 7f3f4f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:53.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.897+0000 7f3f4ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:53.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.897+0000 7f3f4ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:53.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.897+0000 7f3f4ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:53.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.897+0000 7f3f4ed9d640 10 monclient(hunting): handle_auth_done global_id 4302 payload 824 2024-04-22T08:12:53.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.897+0000 7f3f4ed9d640 1 -- 172.21.15.40:0/3439132168 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f5013b0e0 msgr2=0x7f3f5011ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:53.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.897+0000 7f3f4ed9d640 1 --2- 172.21.15.40:0/3439132168 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f5013b0e0 0x7f3f5011ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:53.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.897+0000 7f3f4ed9d640 1 -- 172.21.15.40:0/3439132168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f500a1820 msgr2=0x7f3f5011e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:53.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.897+0000 7f3f4ed9d640 1 --2- 172.21.15.40:0/3439132168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f500a1820 0x7f3f5011e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:53.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.897+0000 7f3f4ed9d640 10 monclient: _finish_hunting 0 2024-04-22T08:12:53.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.897+0000 7f3f4ed9d640 1 monclient: found mon.noname-a 2024-04-22T08:12:53.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.897+0000 7f3f4ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:53.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.897+0000 7f3f4ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:12:53.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.897+0000 7f3f4ed9d640 1 -- 172.21.15.40:0/3439132168 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f40013a10 con 0x7f3f5009ee10 2024-04-22T08:12:53.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.897+0000 7f3f4f59e640 1 --2- 172.21.15.40:0/3439132168 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f5013b0e0 0x7f3f5011ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:12:53.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.898+0000 7f0884955640 1 --2- 172.21.15.40:0/2463889770 >> v2:172.21.15.189:6800/30168 conn(0x7f0864037850 0x7f0864039d10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0874007d50 tx=0x7f0874007ca0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:53.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.900+0000 7f3f4ed9d640 1 -- 172.21.15.40:0/3439132168 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f5009ee10 msgr2=0x7f3f501c68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:53.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.900+0000 7f3f4ed9d640 1 --2- 172.21.15.40:0/3439132168 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f5009ee10 0x7f3f501c68d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3f40017350 tx=0x7f3f40017810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:53.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.900+0000 7f3f377fe640 1 -- 172.21.15.40:0/3439132168 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f40004230 con 0x7f3f5009ee10 2024-04-22T08:12:53.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.900+0000 7f3f377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:53.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.900+0000 7f3f377fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:12:53.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.900+0000 7f3f377fe640 10 monclient: dump: 2024-04-22T08:12:53.939 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:53.939 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:53.939 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:53.939 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:53.939 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:53.939 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:53.939 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:53.939 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:53.939 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:53.939 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:53.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.900+0000 7f3f377fe640 10 monclient: _finish_auth 0 2024-04-22T08:12:53.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.900+0000 7f3f377fe640 10 monclient: _check_auth_tickets 2024-04-22T08:12:53.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.900+0000 7f3f377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:53.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.900+0000 7f3f377fe640 1 -- 172.21.15.40:0/3439132168 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f40019040 con 0x7f3f5009ee10 2024-04-22T08:12:53.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.900+0000 7f3f377fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:53.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.900+0000 7f3f377fe640 1 -- 172.21.15.40:0/3439132168 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f40003610 con 0x7f3f5009ee10 2024-04-22T08:12:53.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.900+0000 7f3f377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:53.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.900+0000 7f3f377fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:12:53.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.900+0000 7f3f377fe640 10 monclient: dump: 2024-04-22T08:12:53.940 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:53.940 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:53.940 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:53.940 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:53.940 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:53.940 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:53.941 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:53.941 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:53.941 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:53.941 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:53.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.900+0000 7f3f55244640 5 monclient: authenticate success, global_id 4302 2024-04-22T08:12:53.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.900+0000 7f3f55244640 10 monclient: _renew_subs 2024-04-22T08:12:53.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.900+0000 7f3f55244640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:53.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.900+0000 7f3f55244640 1 -- 172.21.15.40:0/3439132168 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f48009b30 con 0x7f3f5009ee10 2024-04-22T08:12:53.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.900+0000 7f3f4d59a640 4 set_mon_vals no callback set 2024-04-22T08:12:53.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.900+0000 7f3f55244640 10 monclient: _renew_subs 2024-04-22T08:12:53.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.900+0000 7f3f55244640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:53.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.900+0000 7f3f55244640 1 -- 172.21.15.40:0/3439132168 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f501209e0 con 0x7f3f5009ee10 2024-04-22T08:12:53.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.901+0000 7f087d7fa640 1 -- 172.21.15.40:0/2463889770 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f086c017080 con 0x7f08800a1f30 2024-04-22T08:12:53.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.901+0000 7f087d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:53.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.901+0000 7f087d7fa640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:12:53.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.901+0000 7f3f55244640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:53.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.901+0000 7f3f55244640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:53.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.901+0000 7f3f55244640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:53.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.901+0000 7f3f377fe640 1 -- 172.21.15.40:0/3439132168 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f3f40003770 con 0x7f3f5009ee10 2024-04-22T08:12:53.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.902+0000 7f3f377fe640 1 --2- 172.21.15.40:0/3439132168 >> v2:172.21.15.189:6800/30168 conn(0x7f3f3c037820 0x7f3f3c039ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:53.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.902+0000 7f3f377fe640 1 -- 172.21.15.40:0/3439132168 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f3f400526f0 con 0x7f3f5009ee10 2024-04-22T08:12:53.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.901+0000 7f3f55244640 1 -- 172.21.15.40:0/3439132168 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f5004e9b0 con 0x7f3f5009ee10 2024-04-22T08:12:53.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.902+0000 7f3f4e59c640 1 --2- 172.21.15.40:0/3439132168 >> v2:172.21.15.189:6800/30168 conn(0x7f3f3c037820 0x7f3f3c039ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:53.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.902+0000 7f3f4e59c640 10 monclient: get_auth_request con 0x7f3f3c037820 auth_method 0 2024-04-22T08:12:53.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.904+0000 7f3f4e59c640 1 --2- 172.21.15.40:0/3439132168 >> v2:172.21.15.189:6800/30168 conn(0x7f3f3c037820 0x7f3f3c039ce0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3f50071ad0 tx=0x7f3f44009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:53.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.911+0000 7f3f377fe640 1 -- 172.21.15.40:0/3439132168 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f3f40024080 con 0x7f3f5009ee10 2024-04-22T08:12:53.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.912+0000 7f3f377fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:53.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.912+0000 7f3f377fe640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:12:53.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.915+0000 7f57544af640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-22T08:12:53.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.915+0000 7f57544af640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-22T08:12:53.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.915+0000 7f57544af640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:53.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.915+0000 7f57544af640 1 -- 172.21.15.40:0/3127029691 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f574c11f870 con 0x7f574c0a54b0 2024-04-22T08:12:53.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.916+0000 7f5742ffd640 1 -- 172.21.15.40:0/3127029691 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f574c11f870 con 0x7f574c0a54b0 2024-04-22T08:12:53.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.916+0000 7f5742ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-22T08:12:53.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.916+0000 7f5742ffd640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:12:53.945 INFO:teuthology.orchestra.run.smithi040.stdout:47244640259 2024-04-22T08:12:53.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.919+0000 7f57544af640 1 -- 172.21.15.40:0/3127029691 >> v2:172.21.15.189:6800/30168 conn(0x7f57300377d0 msgr2=0x7f5730039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:53.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.919+0000 7f57544af640 1 --2- 172.21.15.40:0/3127029691 >> v2:172.21.15.189:6800/30168 conn(0x7f57300377d0 0x7f5730039c90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f57480043e0 tx=0x7f5748007ef0 comp rx=0 tx=0).stop 2024-04-22T08:12:53.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.919+0000 7f57544af640 10 monclient: shutdown 2024-04-22T08:12:53.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.919+0000 7f57544af640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:53.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.919+0000 7f57544af640 1 -- 172.21.15.40:0/3127029691 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f574c0a54b0 msgr2=0x7f574c13a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:53.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.919+0000 7f57544af640 1 --2- 172.21.15.40:0/3127029691 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f574c0a54b0 0x7f574c13a470 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5744035000 tx=0x7f5744004a40 comp rx=0 tx=0).stop 2024-04-22T08:12:53.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.919+0000 7f57544af640 1 -- 172.21.15.40:0/3127029691 shutdown_connections 2024-04-22T08:12:53.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.919+0000 7f57544af640 1 --2- 172.21.15.40:0/3127029691 >> v2:172.21.15.189:6800/30168 conn(0x7f57300377d0 0x7f5730039c90 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:53.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.919+0000 7f57544af640 1 --2- 172.21.15.40:0/3127029691 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f574c0a3820 0x7f574c135450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:53.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.919+0000 7f57544af640 1 --2- 172.21.15.40:0/3127029691 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f574c0a54b0 0x7f574c13a470 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:53.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.919+0000 7f57544af640 1 --2- 172.21.15.40:0/3127029691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f574c0a33b0 0x7f574c134f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:53.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.919+0000 7f57544af640 1 -- 172.21.15.40:0/3127029691 >> 172.21.15.40:0/3127029691 conn(0x7f574c0a8bc0 msgr2=0x7f574c06e960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:53.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.919+0000 7f57544af640 1 -- 172.21.15.40:0/3127029691 shutdown_connections 2024-04-22T08:12:53.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.919+0000 7f57544af640 1 -- 172.21.15.40:0/3127029691 wait complete. 2024-04-22T08:12:53.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.928+0000 7fc010a7d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-22T08:12:53.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.928+0000 7fc010a7d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-22T08:12:53.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.928+0000 7fc010a7d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:53.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.928+0000 7fc010a7d640 1 -- 172.21.15.40:0/2117605109 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fc00c06a1e0 con 0x7fc00c0a1d30 2024-04-22T08:12:53.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.929+0000 7fbffaffd640 1 -- 172.21.15.40:0/2117605109 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc000017020 con 0x7fc00c0a1d30 2024-04-22T08:12:53.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.929+0000 7fbffaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-22T08:12:53.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.929+0000 7fbffaffd640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:12:53.947 INFO:teuthology.orchestra.run.smithi040.stdout:47244640261 2024-04-22T08:12:53.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.931+0000 7fc010a7d640 1 -- 172.21.15.40:0/2117605109 >> v2:172.21.15.189:6800/30168 conn(0x7fbfec037800 msgr2=0x7fbfec039cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:53.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.931+0000 7fc010a7d640 1 --2- 172.21.15.40:0/2117605109 >> v2:172.21.15.189:6800/30168 conn(0x7fbfec037800 0x7fbfec039cc0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc004032000 tx=0x7fc004030000 comp rx=0 tx=0).stop 2024-04-22T08:12:53.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.931+0000 7fc010a7d640 10 monclient: shutdown 2024-04-22T08:12:53.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.931+0000 7fc010a7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:53.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.931+0000 7fc010a7d640 1 -- 172.21.15.40:0/2117605109 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc00c0a1d30 msgr2=0x7fc00c1cb310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:53.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.931+0000 7fc010a7d640 1 --2- 172.21.15.40:0/2117605109 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc00c0a1d30 0x7fc00c1cb310 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc00000ebe0 tx=0x7fc00000c520 comp rx=0 tx=0).stop 2024-04-22T08:12:53.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.931+0000 7fc010a7d640 1 -- 172.21.15.40:0/2117605109 shutdown_connections 2024-04-22T08:12:53.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.931+0000 7fc010a7d640 1 --2- 172.21.15.40:0/2117605109 >> v2:172.21.15.189:6800/30168 conn(0x7fbfec037800 0x7fbfec039cc0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:53.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.931+0000 7fc010a7d640 1 --2- 172.21.15.40:0/2117605109 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc00c135d90 0x7fc00c122130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:53.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.931+0000 7fc010a7d640 1 --2- 172.21.15.40:0/2117605109 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc00c0a1d30 0x7fc00c1cb310 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:53.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.931+0000 7fc010a7d640 1 --2- 172.21.15.40:0/2117605109 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc00c09ee10 0x7fc00c1cadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:53.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.931+0000 7fc010a7d640 1 -- 172.21.15.40:0/2117605109 >> 172.21.15.40:0/2117605109 conn(0x7fc00c0a4ef0 msgr2=0x7fc00c06eea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:53.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.931+0000 7fc010a7d640 1 -- 172.21.15.40:0/2117605109 shutdown_connections 2024-04-22T08:12:53.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.931+0000 7fc010a7d640 1 -- 172.21.15.40:0/2117605109 wait complete. 2024-04-22T08:12:53.950 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.4 2024-04-22T08:12:53.950 DEBUG:teuthology.parallel:result is None 2024-04-22T08:12:53.951 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640261 for osd.2 2024-04-22T08:12:53.951 DEBUG:teuthology.parallel:result is None 2024-04-22T08:12:53.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.953+0000 7fb5e3ebd640 10 monclient: get_monmap_and_config 2024-04-22T08:12:53.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.953+0000 7fb5e3ebd640 10 monclient: build_initial_monmap 2024-04-22T08:12:53.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.953+0000 7fb5e3ebd640 10 monclient: monmap: 2024-04-22T08:12:53.952 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:53.952 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:53.953 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:53.953961+0000 2024-04-22T08:12:53.953 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:53.953961+0000 2024-04-22T08:12:53.953 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:53.953 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:53.953 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:12:53.953 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:12:53.953 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:12:53.953 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:12:53.953 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:12:53.953 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:53.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.953+0000 7fb5e3ebd640 1 Processor -- start 2024-04-22T08:12:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.954+0000 7fb5e3ebd640 1 -- start start 2024-04-22T08:12:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.954+0000 7fb5e3ebd640 10 monclient: init 2024-04-22T08:12:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.954+0000 7fb5e3ebd640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.954+0000 7fb5e3ebd640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-04-22T08:12:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.954+0000 7fb5e3ebd640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb5dc09ee10 0x7fb5dc0a32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.954+0000 7fb5e3ebd640 10 monclient(hunting): picked mon.noname-a con 0x7fb5dc09ee10 addr v2:172.21.15.40:3300/0 2024-04-22T08:12:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.954+0000 7fb5e3ebd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb5dc0a1d30 addr v1:172.21.15.40:6789/0 2024-04-22T08:12:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.954+0000 7fb5e3ebd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb5dc0a2180 addr v1:172.21.15.189:6789/0 2024-04-22T08:12:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.954+0000 7fb5e3ebd640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5dc0a2590 con 0x7fb5dc0a1d30 2024-04-22T08:12:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.954+0000 7fb5e3ebd640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5dc09f1f0 con 0x7fb5dc0a2180 2024-04-22T08:12:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.954+0000 7fb5e3ebd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.954+0000 7fb5e3ebd640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5dc136bb0 con 0x7fb5dc09ee10 2024-04-22T08:12:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.954+0000 7fb5e3ebd640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:53.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.954+0000 7fb5e3ebd640 10 monclient(hunting): authenticate will time out at 847.948456s 2024-04-22T08:12:53.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.954+0000 7fb5e1458640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fb5dc0a1d30 0x7fb5dc135d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:38948/0 (socket says 172.21.15.40:38948) 2024-04-22T08:12:53.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.954+0000 7fb5e1458640 1 -- 172.21.15.40:0/2493724726 learned_addr learned my addr 172.21.15.40:0/2493724726 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:12:53.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.954+0000 7fb5e1c59640 1 --2- 172.21.15.40:0/2493724726 >> v2:172.21.15.40:3300/0 conn(0x7fb5dc09ee10 0x7fb5dc0a32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:53.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.954+0000 7fb5e1c59640 10 monclient(hunting): get_auth_request con 0x7fb5dc09ee10 auth_method 0 2024-04-22T08:12:53.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.954+0000 7fb5e1c59640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:53.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.954+0000 7fb5e1c59640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:53.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.954+0000 7fb5e1c59640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:53.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.954+0000 7fb5e1c59640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:53.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.954+0000 7fb5e1c59640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:53.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.954+0000 7fb5e1c59640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:53.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.955+0000 7fb5e1c59640 10 monclient(hunting): handle_auth_done global_id 4305 payload 274 2024-04-22T08:12:53.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.955+0000 7fb5e1c59640 1 -- 172.21.15.40:0/2493724726 >> v1:172.21.15.189:6789/0 conn(0x7fb5dc0a2180 legacy=0x7fb5dc1364a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:53.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.955+0000 7fb5e1c59640 1 -- 172.21.15.40:0/2493724726 >> v1:172.21.15.40:6789/0 conn(0x7fb5dc0a1d30 legacy=0x7fb5dc135d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:53.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.955+0000 7fb5e1c59640 10 monclient: _finish_hunting 0 2024-04-22T08:12:53.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.955+0000 7fb5e1c59640 1 monclient: found mon.noname-a 2024-04-22T08:12:53.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.955+0000 7fb5e1c59640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:53.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.955+0000 7fb5e1c59640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:12:53.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.955+0000 7fb5e1c59640 1 -- 172.21.15.40:0/2493724726 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5dc0a39c0 con 0x7fb5dc09ee10 2024-04-22T08:12:53.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.958+0000 7f8a8384a640 10 monclient: get_monmap_and_config 2024-04-22T08:12:53.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.958+0000 7f8a8384a640 10 monclient: build_initial_monmap 2024-04-22T08:12:53.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.958+0000 7f8a8384a640 10 monclient: monmap: 2024-04-22T08:12:53.959 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:53.959 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:53.959 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:53.958926+0000 2024-04-22T08:12:53.959 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:53.958926+0000 2024-04-22T08:12:53.959 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:53.959 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:53.959 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:12:53.959 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:12:53.959 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:12:53.959 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:12:53.960 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:12:53.960 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:53.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.958+0000 7fb5e1c59640 1 -- 172.21.15.40:0/2493724726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5dc09ee10 msgr2=0x7fb5dc0a32b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:53.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.958+0000 7fb5e1c59640 1 --2- 172.21.15.40:0/2493724726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5dc09ee10 0x7fb5dc0a32b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc09f380 tx=0x7fb5d802b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=764c89885c825c8c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:53.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.958+0000 7f8a8384a640 1 Processor -- start 2024-04-22T08:12:53.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.958+0000 7f8a8384a640 1 -- start start 2024-04-22T08:12:53.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.958+0000 7fb5e0c57640 1 -- 172.21.15.40:0/2493724726 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d8030a00 con 0x7fb5dc09ee10 2024-04-22T08:12:53.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.958+0000 7fb5e0c57640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:53.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7fb5e0c57640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:12:53.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.958+0000 7f8a8384a640 10 monclient: init 2024-04-22T08:12:53.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7fb5e0c57640 10 monclient: dump: 2024-04-22T08:12:53.962 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:53.962 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:53.962 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:53.962 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:53.963 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:53.963 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:53.963 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:53.963 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:53.963 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:53.963 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:53.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7fb5e0c57640 10 monclient: _finish_auth 0 2024-04-22T08:12:53.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7fb5e0c57640 10 monclient: _check_auth_tickets 2024-04-22T08:12:53.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7fb5e0c57640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:53.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7fb5e0c57640 1 -- 172.21.15.40:0/2493724726 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5d802e050 con 0x7fb5dc09ee10 2024-04-22T08:12:53.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7fb5e3ebd640 5 monclient: authenticate success, global_id 4305 2024-04-22T08:12:53.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7fb5e3ebd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:12:53.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7fb5e0c57640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:53.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7fb5e3ebd640 10 monclient: get_monmap_and_config success 2024-04-22T08:12:53.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7fb5e3ebd640 4 set_mon_vals no callback set 2024-04-22T08:12:53.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7fb5e0c57640 1 -- 172.21.15.40:0/2493724726 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d8030d80 con 0x7fb5dc09ee10 2024-04-22T08:12:53.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7fb5e0c57640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:53.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7fb5e0c57640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:12:53.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7fb5e0c57640 10 monclient: dump: 2024-04-22T08:12:53.964 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:53.964 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:53.964 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:53.964 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:53.964 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:53.964 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:53.964 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:53.964 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:53.964 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:53.964 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:53.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7fb5e3ebd640 10 monclient: shutdown 2024-04-22T08:12:53.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7fb5e3ebd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:53.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7fb5e3ebd640 1 -- 172.21.15.40:0/2493724726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5dc09ee10 msgr2=0x7fb5dc0a32b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:53.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7fb5e3ebd640 1 --2- 172.21.15.40:0/2493724726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5dc09ee10 0x7fb5dc0a32b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc09f380 tx=0x7fb5d802b710 comp rx=0 tx=0).stop 2024-04-22T08:12:53.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7f8a8384a640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:53.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7f8a8384a640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-04-22T08:12:53.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7f8a8384a640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8a7c09ee10 0x7f8a7c0a3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:53.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7f8a8384a640 10 monclient(hunting): picked mon.noname-b con 0x7f8a7c09ee10 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:53.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7f8a8384a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8a7c0a1820 addr v1:172.21.15.40:6789/0 2024-04-22T08:12:53.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7f8a8384a640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8a7c13b0e0 0x7f8a7c13d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:53.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7f8a8384a640 10 monclient(hunting): picked mon.noname-a con 0x7f8a7c13b0e0 addr v2:172.21.15.40:3300/0 2024-04-22T08:12:53.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7f8a8384a640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a7c0a4470 con 0x7f8a7c0a1820 2024-04-22T08:12:53.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7f8a8384a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:53.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7f8a8384a640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a7c0a2310 con 0x7f8a7c13b0e0 2024-04-22T08:12:53.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7f8a8384a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:53.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7f8a8384a640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a7c09f1f0 con 0x7f8a7c09ee10 2024-04-22T08:12:53.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7f8a8384a640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:53.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7f8a8384a640 10 monclient(hunting): authenticate will time out at 847.953299s 2024-04-22T08:12:53.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7f8a81de7640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8a7c13b0e0 0x7f8a7c13d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:53.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7f8a815e6640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8a7c09ee10 0x7f8a7c0a3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:53.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7f8a81de7640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8a7c13b0e0 0x7f8a7c13d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58050/0 (socket says 172.21.15.40:58050) 2024-04-22T08:12:53.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7f8a81de7640 1 -- 172.21.15.40:0/2878734 learned_addr learned my addr 172.21.15.40:0/2878734 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:53.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7f8a81de7640 10 monclient(hunting): get_auth_request con 0x7f8a7c13b0e0 auth_method 0 2024-04-22T08:12:53.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7f8a81de7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:53.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7f8a81de7640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:53.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7f8a81de7640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:53.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7f8a815e6640 10 monclient(hunting): get_auth_request con 0x7f8a7c09ee10 auth_method 0 2024-04-22T08:12:53.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7f8a815e6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:53.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7f8a815e6640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:53.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7f8a815e6640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:53.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7f8a81de7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:53.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7f8a81de7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:53.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7f8a81de7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:53.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7fb5e3ebd640 1 -- 172.21.15.40:0/2493724726 shutdown_connections 2024-04-22T08:12:53.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7f8a815e6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:53.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7f8a815e6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:53.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7f8a815e6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:53.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.960+0000 7f8a81de7640 10 monclient(hunting): handle_auth_done global_id 4308 payload 274 2024-04-22T08:12:53.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.960+0000 7f8a81de7640 1 -- 172.21.15.40:0/2878734 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8a7c09ee10 msgr2=0x7f8a7c0a3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:53.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.960+0000 7f8a81de7640 1 --2- 172.21.15.40:0/2878734 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8a7c09ee10 0x7f8a7c0a3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:53.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.960+0000 7f8a81de7640 1 -- 172.21.15.40:0/2878734 >> v1:172.21.15.40:6789/0 conn(0x7f8a7c0a1820 legacy=0x7f8a7c0a1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:53.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.960+0000 7f8a81de7640 10 monclient: _finish_hunting 0 2024-04-22T08:12:53.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.960+0000 7f8a81de7640 1 monclient: found mon.noname-a 2024-04-22T08:12:53.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.960+0000 7f8a81de7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:53.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.960+0000 7f8a81de7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:12:53.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.960+0000 7f8a81de7640 1 -- 172.21.15.40:0/2878734 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a7c09fd30 con 0x7f8a7c13b0e0 2024-04-22T08:12:53.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7fb5e3ebd640 1 --2- 172.21.15.40:0/2493724726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5dc09ee10 0x7fb5dc0a32b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:53.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.959+0000 7fb5e3ebd640 1 -- 172.21.15.40:0/2493724726 >> 172.21.15.40:0/2493724726 conn(0x7fb5dc0a4ef0 msgr2=0x7fb5dc0a5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:53.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.960+0000 7fb5e3ebd640 1 -- 172.21.15.40:0/2493724726 shutdown_connections 2024-04-22T08:12:53.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.960+0000 7fb5e3ebd640 1 -- 172.21.15.40:0/2493724726 wait complete. 2024-04-22T08:12:53.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.961+0000 7fb5e3ebd640 10 monclient: build_initial_monmap 2024-04-22T08:12:53.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.961+0000 7fb5e3ebd640 10 monclient: monmap: 2024-04-22T08:12:53.973 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:53.973 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:12:53.973 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:12:53.973 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:12:53.973 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:53.973 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:53.973 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:12:53.973 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:12:53.973 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:12:53.973 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:53.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.961+0000 7fb5e3ebd640 1 Processor -- start 2024-04-22T08:12:53.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.961+0000 7fb5e3ebd640 1 -- start start 2024-04-22T08:12:53.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.961+0000 7fb5e3ebd640 10 monclient: init 2024-04-22T08:12:53.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e3ebd640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:53.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e3ebd640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-22T08:12:53.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e3ebd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5dc09ee10 0x7fb5dc1cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:53.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e3ebd640 10 monclient(hunting): picked mon.noname-a con 0x7fb5dc09ee10 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:53.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e3ebd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb5dc0a1d30 0x7fb5dc1cb350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:53.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e3ebd640 10 monclient(hunting): picked mon.noname-c con 0x7fb5dc0a1d30 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:53.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e3ebd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5dc0a2180 0x7fb5dc123330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:53.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e1c59640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5dc09ee10 0x7fb5dc1cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:53.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e1c59640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5dc09ee10 0x7fb5dc1cae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58052/0 (socket says 172.21.15.40:58052) 2024-04-22T08:12:53.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e1458640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb5dc0a1d30 0x7fb5dc1cb350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:53.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e1458640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb5dc0a1d30 0x7fb5dc1cb350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49826/0 (socket says 172.21.15.40:49826) 2024-04-22T08:12:53.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e3ebd640 10 monclient(hunting): picked mon.noname-b con 0x7fb5dc0a2180 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:53.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e1c59640 1 -- 172.21.15.40:0/3170709029 learned_addr learned my addr 172.21.15.40:0/3170709029 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:53.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e3ebd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:53.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e3ebd640 1 -- 172.21.15.40:0/3170709029 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5dc126920 con 0x7fb5dc09ee10 2024-04-22T08:12:53.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e3ebd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:53.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e3ebd640 1 -- 172.21.15.40:0/3170709029 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5dc0a27a0 con 0x7fb5dc0a2180 2024-04-22T08:12:53.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e3ebd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:53.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e3ebd640 1 -- 172.21.15.40:0/3170709029 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5dc0718c0 con 0x7fb5dc0a1d30 2024-04-22T08:12:53.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7f8a81de7640 1 -- 172.21.15.40:0/2878734 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a7c13b0e0 msgr2=0x7f8a7c13d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:53.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7f8a81de7640 1 --2- 172.21.15.40:0/2878734 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a7c13b0e0 0x7f8a7c13d4d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8a74009e50 tx=0x7f8a7402bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70ab1a339fdfa933 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:53.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e3ebd640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:53.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e3ebd640 10 monclient(hunting): authenticate will time out at 847.956161s 2024-04-22T08:12:53.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e1c59640 10 monclient(hunting): get_auth_request con 0x7fb5dc09ee10 auth_method 0 2024-04-22T08:12:53.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e1c59640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:53.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e1c59640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:53.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e1c59640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:53.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.963+0000 7f8a73fff640 1 -- 172.21.15.40:0/2878734 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a7402cb70 con 0x7f8a7c13b0e0 2024-04-22T08:12:53.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.963+0000 7f8a73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:53.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.963+0000 7f8a73fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:12:53.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.963+0000 7f8a73fff640 10 monclient: dump: 2024-04-22T08:12:53.977 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:53.977 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:53.977 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:53.978 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:53.978 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:53.978 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:53.978 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:53.978 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:53.978 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:53.978 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:53.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.963+0000 7f8a73fff640 10 monclient: _finish_auth 0 2024-04-22T08:12:53.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.963+0000 7f8a73fff640 10 monclient: _check_auth_tickets 2024-04-22T08:12:53.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.963+0000 7f8a73fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:53.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.963+0000 7f8a73fff640 1 -- 172.21.15.40:0/2878734 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a7402ccd0 con 0x7f8a7c13b0e0 2024-04-22T08:12:53.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.963+0000 7f8a8384a640 5 monclient: authenticate success, global_id 4308 2024-04-22T08:12:53.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.963+0000 7f8a8384a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:12:53.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.963+0000 7f8a73fff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:53.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.963+0000 7f8a8384a640 10 monclient: get_monmap_and_config success 2024-04-22T08:12:53.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.963+0000 7f8a73fff640 1 -- 172.21.15.40:0/2878734 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a74034610 con 0x7f8a7c13b0e0 2024-04-22T08:12:53.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.963+0000 7f8a73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:53.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.963+0000 7f8a8384a640 4 set_mon_vals no callback set 2024-04-22T08:12:53.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.963+0000 7f8a73fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:12:53.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.963+0000 7f8a73fff640 10 monclient: dump: 2024-04-22T08:12:53.979 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:53.979 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:53.979 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:53.979 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:53.979 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:53.979 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:53.979 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:53.979 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:53.979 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:53.980 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:53.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e1458640 10 monclient(hunting): get_auth_request con 0x7fb5dc0a1d30 auth_method 0 2024-04-22T08:12:53.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e1458640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:53.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e1458640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:53.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e1458640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:53.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e1c59640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:53.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e1c59640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:53.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e1c59640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:53.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e1458640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:53.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e1458640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:53.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e245a640 1 --2- 172.21.15.40:0/3170709029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5dc0a2180 0x7fb5dc123330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:53.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e1458640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:53.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e245a640 10 monclient(hunting): get_auth_request con 0x7fb5dc0a2180 auth_method 0 2024-04-22T08:12:53.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e245a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:53.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e245a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:53.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e245a640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:53.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e245a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:53.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e245a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:53.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.963+0000 7f8a8384a640 10 monclient: shutdown 2024-04-22T08:12:53.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.963+0000 7f8a8384a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:53.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.963+0000 7f8a8384a640 1 -- 172.21.15.40:0/2878734 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a7c13b0e0 msgr2=0x7f8a7c13d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:53.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.963+0000 7f8a8384a640 1 --2- 172.21.15.40:0/2878734 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a7c13b0e0 0x7f8a7c13d4d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8a74009e50 tx=0x7f8a7402bdc0 comp rx=0 tx=0).stop 2024-04-22T08:12:53.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e245a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:53.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e1c59640 10 monclient(hunting): handle_auth_done global_id 4311 payload 824 2024-04-22T08:12:53.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e1c59640 1 -- 172.21.15.40:0/3170709029 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb5dc0a1d30 msgr2=0x7fb5dc1cb350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:53.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e1c59640 1 --2- 172.21.15.40:0/3170709029 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb5dc0a1d30 0x7fb5dc1cb350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:53.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e1c59640 1 -- 172.21.15.40:0/3170709029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5dc0a2180 msgr2=0x7fb5dc123330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:53.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e1c59640 1 --2- 172.21.15.40:0/3170709029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5dc0a2180 0x7fb5dc123330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:53.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e1c59640 10 monclient: _finish_hunting 0 2024-04-22T08:12:53.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e1c59640 1 monclient: found mon.noname-a 2024-04-22T08:12:53.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e1c59640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:53.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e1c59640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:12:53.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.962+0000 7fb5e1c59640 1 -- 172.21.15.40:0/3170709029 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5d8009400 con 0x7fb5dc09ee10 2024-04-22T08:12:53.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.963+0000 7f8a8384a640 1 -- 172.21.15.40:0/2878734 shutdown_connections 2024-04-22T08:12:53.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.963+0000 7f8a8384a640 1 --2- 172.21.15.40:0/2878734 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a7c13b0e0 0x7f8a7c13d4d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:53.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.963+0000 7f8a8384a640 1 --2- 172.21.15.40:0/2878734 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8a7c09ee10 0x7f8a7c0a3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:53.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.963+0000 7f8a8384a640 1 -- 172.21.15.40:0/2878734 >> 172.21.15.40:0/2878734 conn(0x7f8a7c0a4e80 msgr2=0x7f8a7c0a5290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:53.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.964+0000 7f8a8384a640 1 -- 172.21.15.40:0/2878734 shutdown_connections 2024-04-22T08:12:53.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.964+0000 7f8a8384a640 1 -- 172.21.15.40:0/2878734 wait complete. 2024-04-22T08:12:53.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.964+0000 7f8a8384a640 10 monclient: build_initial_monmap 2024-04-22T08:12:53.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.964+0000 7f8a8384a640 10 monclient: monmap: 2024-04-22T08:12:53.985 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:53.985 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:12:53.986 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:12:53.986 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:12:53.986 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:53.986 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:53.986 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:12:53.986 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:12:53.986 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:12:53.986 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:53.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.964+0000 7f8a8384a640 1 Processor -- start 2024-04-22T08:12:53.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.965+0000 7f8a8384a640 1 -- start start 2024-04-22T08:12:53.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.965+0000 7f8a8384a640 10 monclient: init 2024-04-22T08:12:53.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.965+0000 7fb5e1c59640 1 -- 172.21.15.40:0/3170709029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5dc09ee10 msgr2=0x7fb5dc1cae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:53.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.965+0000 7fb5e1c59640 1 --2- 172.21.15.40:0/3170709029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5dc09ee10 0x7fb5dc1cae10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8009890 tx=0x7fb5d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:53.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.965+0000 7fb5d27fc640 1 -- 172.21.15.40:0/3170709029 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d803db60 con 0x7fb5dc09ee10 2024-04-22T08:12:53.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.965+0000 7fb5d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:53.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.965+0000 7fb5d27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:12:53.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.965+0000 7fb5d27fc640 10 monclient: dump: 2024-04-22T08:12:53.988 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:53.988 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:53.988 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:53.988 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:53.988 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:53.988 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:53.988 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:53.988 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:53.988 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:53.988 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:53.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.965+0000 7fb5d27fc640 10 monclient: _finish_auth 0 2024-04-22T08:12:53.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.965+0000 7fb5d27fc640 10 monclient: _check_auth_tickets 2024-04-22T08:12:53.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.965+0000 7fb5d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:53.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.965+0000 7fb5d27fc640 1 -- 172.21.15.40:0/3170709029 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5d803dcc0 con 0x7fb5dc09ee10 2024-04-22T08:12:53.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.965+0000 7fb5e3ebd640 5 monclient: authenticate success, global_id 4311 2024-04-22T08:12:53.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.965+0000 7fb5e3ebd640 10 monclient: _renew_subs 2024-04-22T08:12:53.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.965+0000 7fb5e3ebd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:53.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.965+0000 7fb5e3ebd640 1 -- 172.21.15.40:0/3170709029 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5dc1259c0 con 0x7fb5dc09ee10 2024-04-22T08:12:53.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.965+0000 7fb5d27fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:53.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.965+0000 7fb5d27fc640 1 -- 172.21.15.40:0/3170709029 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d80326a0 con 0x7fb5dc09ee10 2024-04-22T08:12:53.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.965+0000 7fb5d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:53.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.965+0000 7fb5d27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:12:53.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.965+0000 7fb5d27fc640 10 monclient: dump: 2024-04-22T08:12:53.989 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:53.989 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:53.989 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:53.989 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:53.989 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:53.990 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:53.990 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:53.990 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:53.990 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:53.990 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:53.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.965+0000 7fb5e3ebd640 10 monclient: _renew_subs 2024-04-22T08:12:53.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.965+0000 7f8a8384a640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:53.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.965+0000 7f8a8384a640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:12:53.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.965+0000 7f8a8384a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a7c09ee10 0x7f8a7c1cabc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:53.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.965+0000 7f8a8384a640 10 monclient(hunting): picked mon.noname-a con 0x7f8a7c09ee10 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:53.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.965+0000 7f8a8384a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a7c0a1820 0x7f8a7c1cb100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:53.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.965+0000 7f8a8384a640 10 monclient(hunting): picked mon.noname-b con 0x7f8a7c0a1820 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:53.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.965+0000 7f8a8384a640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8a7c13b0e0 0x7f8a7c120390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:53.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.965+0000 7f8a8384a640 10 monclient(hunting): picked mon.noname-c con 0x7f8a7c13b0e0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:53.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.965+0000 7f8a81de7640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8a7c13b0e0 0x7f8a7c120390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:53.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.965+0000 7f8a81de7640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8a7c13b0e0 0x7f8a7c120390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49834/0 (socket says 172.21.15.40:49834) 2024-04-22T08:12:53.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.965+0000 7f8a815e6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a7c09ee10 0x7f8a7c1cabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:53.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.965+0000 7f8a81de7640 1 -- 172.21.15.40:0/425098404 learned_addr learned my addr 172.21.15.40:0/425098404 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:53.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.965+0000 7f8a8384a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:53.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.965+0000 7fb5e3ebd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:53.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.965+0000 7fb5e3ebd640 1 -- 172.21.15.40:0/3170709029 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5dc124df0 con 0x7fb5dc09ee10 2024-04-22T08:12:53.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.965+0000 7fb5d3fff640 4 set_mon_vals no callback set 2024-04-22T08:12:53.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.965+0000 7f8a8384a640 1 -- 172.21.15.40:0/425098404 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a7c1241d0 con 0x7f8a7c09ee10 2024-04-22T08:12:53.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.965+0000 7f8a8384a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:53.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.965+0000 7f8a8384a640 1 -- 172.21.15.40:0/425098404 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a7c123f60 con 0x7f8a7c0a1820 2024-04-22T08:12:53.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.966+0000 7f8a8384a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:53.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.966+0000 7f8a8384a640 1 -- 172.21.15.40:0/425098404 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a7c0a4260 con 0x7f8a7c13b0e0 2024-04-22T08:12:53.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.966+0000 7f8a8384a640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:53.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.966+0000 7f8a8384a640 10 monclient(hunting): authenticate will time out at 847.959836s 2024-04-22T08:12:53.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.966+0000 7f8a81de7640 10 monclient(hunting): get_auth_request con 0x7f8a7c13b0e0 auth_method 0 2024-04-22T08:12:53.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.966+0000 7f8a81de7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:53.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.966+0000 7f8a81de7640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:53.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.966+0000 7f8a81de7640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:53.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.966+0000 7f8a815e6640 10 monclient(hunting): get_auth_request con 0x7f8a7c09ee10 auth_method 0 2024-04-22T08:12:53.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.966+0000 7f8a815e6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:53.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.966+0000 7f8a815e6640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:53.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.966+0000 7f8a815e6640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:53.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.966+0000 7f8a81de7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:53.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.966+0000 7f8a81de7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:53.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.966+0000 7f8a81de7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:53.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.966+0000 7f8a815e6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:53.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.966+0000 7f8a815e6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:53.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.966+0000 7f8a815e6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:53.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.966+0000 7f8a815e6640 10 monclient(hunting): handle_auth_done global_id 4314 payload 824 2024-04-22T08:12:53.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.966+0000 7f8a815e6640 1 -- 172.21.15.40:0/425098404 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8a7c13b0e0 msgr2=0x7f8a7c120390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:53.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.966+0000 7f8a815e6640 1 --2- 172.21.15.40:0/425098404 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8a7c13b0e0 0x7f8a7c120390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:53.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.966+0000 7f8a815e6640 1 -- 172.21.15.40:0/425098404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a7c0a1820 msgr2=0x7f8a7c1cb100 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T08:12:53.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.966+0000 7fb5e3ebd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:53.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.966+0000 7fb5e3ebd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:53.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.966+0000 7fb5e3ebd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:53.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.966+0000 7fb5e3ebd640 1 -- 172.21.15.40:0/3170709029 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5dc04e9b0 con 0x7fb5dc09ee10 2024-04-22T08:12:53.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.966+0000 7f8a815e6640 1 --2- 172.21.15.40:0/425098404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a7c0a1820 0x7f8a7c1cb100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:53.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.966+0000 7f8a815e6640 10 monclient: _finish_hunting 0 2024-04-22T08:12:53.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.966+0000 7f8a815e6640 1 monclient: found mon.noname-a 2024-04-22T08:12:53.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.966+0000 7f8a815e6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:53.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.966+0000 7f8a815e6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:12:53.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.966+0000 7f8a815e6640 1 -- 172.21.15.40:0/425098404 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a74009b30 con 0x7f8a7c09ee10 2024-04-22T08:12:53.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.966+0000 7f8a81de7640 1 --2- 172.21.15.40:0/425098404 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8a7c13b0e0 0x7f8a7c120390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:12:53.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.967+0000 7fb5d27fc640 1 -- 172.21.15.40:0/3170709029 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fb5d8039040 con 0x7fb5dc09ee10 2024-04-22T08:12:53.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.967+0000 7fb5d27fc640 1 --2- 172.21.15.40:0/3170709029 >> v2:172.21.15.189:6800/30168 conn(0x7fb5c0037820 0x7fb5c0039ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:53.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.967+0000 7fb5d27fc640 1 -- 172.21.15.40:0/3170709029 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3506+0+0 (secure 0 0 0) 0x7fb5d806c660 con 0x7fb5dc09ee10 2024-04-22T08:12:53.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.967+0000 7fb5e1458640 1 --2- 172.21.15.40:0/3170709029 >> v2:172.21.15.189:6800/30168 conn(0x7fb5c0037820 0x7fb5c0039ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:53.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.967+0000 7fb5e1458640 10 monclient: get_auth_request con 0x7fb5c0037820 auth_method 0 2024-04-22T08:12:53.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.968+0000 7f8a815e6640 1 -- 172.21.15.40:0/425098404 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a7c09ee10 msgr2=0x7f8a7c1cabc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:53.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.968+0000 7f8a815e6640 1 --2- 172.21.15.40:0/425098404 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a7c09ee10 0x7f8a7c1cabc0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8a7800e880 tx=0x7f8a7800ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:54.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.970+0000 7fb5e1458640 1 --2- 172.21.15.40:0/3170709029 >> v2:172.21.15.189:6800/30168 conn(0x7fb5c0037820 0x7fb5c0039ce0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc007c90 tx=0x7fb5cc007be0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:54.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.971+0000 7f8a71ffb640 1 -- 172.21.15.40:0/425098404 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a7800cb50 con 0x7f8a7c09ee10 2024-04-22T08:12:54.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.971+0000 7f8a71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:54.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.971+0000 7f8a71ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:12:54.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.971+0000 7f8a71ffb640 10 monclient: dump: 2024-04-22T08:12:54.000 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:54.001 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:54.001 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:54.001 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:54.001 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:54.001 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:54.001 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:54.001 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:54.001 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:54.001 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:54.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.971+0000 7f8a71ffb640 10 monclient: _finish_auth 0 2024-04-22T08:12:54.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.971+0000 7f8a71ffb640 10 monclient: _check_auth_tickets 2024-04-22T08:12:54.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.971+0000 7f8a71ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:54.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.971+0000 7f8a8384a640 5 monclient: authenticate success, global_id 4314 2024-04-22T08:12:54.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.971+0000 7f8a8384a640 10 monclient: _renew_subs 2024-04-22T08:12:54.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.971+0000 7f8a8384a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:54.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.971+0000 7f8a8384a640 1 -- 172.21.15.40:0/425098404 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a7c13cfd0 con 0x7f8a7c09ee10 2024-04-22T08:12:54.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.971+0000 7fb5d27fc640 1 -- 172.21.15.40:0/3170709029 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7fb5d8032800 con 0x7fb5dc09ee10 2024-04-22T08:12:54.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.971+0000 7fb5d27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:54.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.971+0000 7fb5d27fc640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:12:54.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.971+0000 7f8a71ffb640 1 -- 172.21.15.40:0/425098404 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a7800ccb0 con 0x7f8a7c09ee10 2024-04-22T08:12:54.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.971+0000 7f8a8384a640 10 monclient: _renew_subs 2024-04-22T08:12:54.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.971+0000 7f8a8384a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:54.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.971+0000 7f8a8384a640 1 -- 172.21.15.40:0/425098404 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a7c121e00 con 0x7f8a7c09ee10 2024-04-22T08:12:54.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.972+0000 7f8a71ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:54.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.972+0000 7f8a71ffb640 1 -- 172.21.15.40:0/425098404 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a78013470 con 0x7f8a7c09ee10 2024-04-22T08:12:54.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.972+0000 7f8a737fe640 4 set_mon_vals no callback set 2024-04-22T08:12:54.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.972+0000 7f8a71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:54.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.972+0000 7f8a71ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:12:54.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.972+0000 7f8a71ffb640 10 monclient: dump: 2024-04-22T08:12:54.005 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:54.005 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:54.005 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:54.005 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:54.005 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:54.005 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:54.006 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:54.006 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:54.006 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:54.006 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:54.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.973+0000 7f8a8384a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:54.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.973+0000 7f8a71ffb640 1 -- 172.21.15.40:0/425098404 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f8a780135d0 con 0x7f8a7c09ee10 2024-04-22T08:12:54.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.973+0000 7f8a8384a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:54.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.973+0000 7f8a8384a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:54.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.973+0000 7f8a8384a640 1 -- 172.21.15.40:0/425098404 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a7c04e9b0 con 0x7f8a7c09ee10 2024-04-22T08:12:54.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.973+0000 7f8a71ffb640 1 --2- 172.21.15.40:0/425098404 >> v2:172.21.15.189:6800/30168 conn(0x7f8a60037800 0x7f8a60039cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:54.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.973+0000 7f8a71ffb640 1 -- 172.21.15.40:0/425098404 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f8a7804a450 con 0x7f8a7c09ee10 2024-04-22T08:12:54.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.973+0000 7f8a80de5640 1 --2- 172.21.15.40:0/425098404 >> v2:172.21.15.189:6800/30168 conn(0x7f8a60037800 0x7f8a60039cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:54.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.973+0000 7f8a80de5640 10 monclient: get_auth_request con 0x7f8a60037800 auth_method 0 2024-04-22T08:12:54.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.976+0000 7f8a80de5640 1 --2- 172.21.15.40:0/425098404 >> v2:172.21.15.189:6800/30168 conn(0x7f8a60037800 0x7f8a60039cc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c04e8a0 tx=0x7f8a6c0079b0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:54.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.977+0000 7f8a71ffb640 1 -- 172.21.15.40:0/425098404 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f8a780123d0 con 0x7f8a7c09ee10 2024-04-22T08:12:54.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.978+0000 7f8a71ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:54.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:53.978+0000 7f8a71ffb640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:12:54.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.093+0000 7f0886bb9640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-22T08:12:54.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.093+0000 7f0886bb9640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-22T08:12:54.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.093+0000 7f0886bb9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.189:3300/0 2024-04-22T08:12:54.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.093+0000 7f0886bb9640 1 -- 172.21.15.40:0/2463889770 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f088004e9b0 con 0x7f08800a1f30 2024-04-22T08:12:54.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.093+0000 7f087d7fa640 1 -- 172.21.15.40:0/2463889770 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f086c021020 con 0x7f08800a1f30 2024-04-22T08:12:54.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.093+0000 7f087d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-22T08:12:54.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.093+0000 7f087d7fa640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:12:54.093 INFO:teuthology.orchestra.run.smithi040.stdout:47244640260 2024-04-22T08:12:54.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.096+0000 7f0886bb9640 1 -- 172.21.15.40:0/2463889770 >> v2:172.21.15.189:6800/30168 conn(0x7f0864037850 msgr2=0x7f0864039d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:54.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.096+0000 7f0886bb9640 1 --2- 172.21.15.40:0/2463889770 >> v2:172.21.15.189:6800/30168 conn(0x7f0864037850 0x7f0864039d10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0874007d50 tx=0x7f0874007ca0 comp rx=0 tx=0).stop 2024-04-22T08:12:54.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.096+0000 7f0886bb9640 10 monclient: shutdown 2024-04-22T08:12:54.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.096+0000 7f0886bb9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:54.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.096+0000 7f0886bb9640 1 -- 172.21.15.40:0/2463889770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08800a1f30 msgr2=0x7f0880120590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:54.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.096+0000 7f0886bb9640 1 --2- 172.21.15.40:0/2463889770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08800a1f30 0x7f0880120590 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f086c00e870 tx=0x7f086c00ed30 comp rx=0 tx=0).stop 2024-04-22T08:12:54.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.096+0000 7f0886bb9640 1 -- 172.21.15.40:0/2463889770 shutdown_connections 2024-04-22T08:12:54.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.096+0000 7f0886bb9640 1 --2- 172.21.15.40:0/2463889770 >> v2:172.21.15.189:6800/30168 conn(0x7f0864037850 0x7f0864039d10 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:54.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.096+0000 7f0886bb9640 1 --2- 172.21.15.40:0/2463889770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08800a1f30 0x7f0880120590 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:54.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.096+0000 7f0886bb9640 1 --2- 172.21.15.40:0/2463889770 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f088013b0e0 0x7f0880120ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:54.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.096+0000 7f0886bb9640 1 --2- 172.21.15.40:0/2463889770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f088009ee10 0x7f08801cb020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:54.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.096+0000 7f0886bb9640 1 -- 172.21.15.40:0/2463889770 >> 172.21.15.40:0/2463889770 conn(0x7f08800a4e80 msgr2=0x7f088013c620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:54.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.096+0000 7f0886bb9640 1 -- 172.21.15.40:0/2463889770 shutdown_connections 2024-04-22T08:12:54.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.096+0000 7f0886bb9640 1 -- 172.21.15.40:0/2463889770 wait complete. 2024-04-22T08:12:54.106 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.0 2024-04-22T08:12:54.107 DEBUG:teuthology.parallel:result is None 2024-04-22T08:12:54.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.156+0000 7f3f55244640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-22T08:12:54.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.156+0000 7f3f55244640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-22T08:12:54.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.156+0000 7f3f55244640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:54.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.156+0000 7f3f55244640 1 -- 172.21.15.40:0/3439132168 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f3f501c01d0 con 0x7f3f5009ee10 2024-04-22T08:12:54.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.156+0000 7f3f377fe640 1 -- 172.21.15.40:0/3439132168 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3f4001ec20 con 0x7f3f5009ee10 2024-04-22T08:12:54.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.156+0000 7f3f377fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-22T08:12:54.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.156+0000 7f3f377fe640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:12:54.156 INFO:teuthology.orchestra.run.smithi040.stdout:47244640260 2024-04-22T08:12:54.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.158+0000 7f3f55244640 1 -- 172.21.15.40:0/3439132168 >> v2:172.21.15.189:6800/30168 conn(0x7f3f3c037820 msgr2=0x7f3f3c039ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:54.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.158+0000 7f3f55244640 1 --2- 172.21.15.40:0/3439132168 >> v2:172.21.15.189:6800/30168 conn(0x7f3f3c037820 0x7f3f3c039ce0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3f50071ad0 tx=0x7f3f44009040 comp rx=0 tx=0).stop 2024-04-22T08:12:54.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.158+0000 7f3f55244640 10 monclient: shutdown 2024-04-22T08:12:54.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.158+0000 7f3f55244640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:54.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.158+0000 7f3f55244640 1 -- 172.21.15.40:0/3439132168 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f5009ee10 msgr2=0x7f3f501c68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:54.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.158+0000 7f3f55244640 1 --2- 172.21.15.40:0/3439132168 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f5009ee10 0x7f3f501c68d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3f40017350 tx=0x7f3f40017810 comp rx=0 tx=0).stop 2024-04-22T08:12:54.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.158+0000 7f3f55244640 1 -- 172.21.15.40:0/3439132168 shutdown_connections 2024-04-22T08:12:54.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.158+0000 7f3f55244640 1 --2- 172.21.15.40:0/3439132168 >> v2:172.21.15.189:6800/30168 conn(0x7f3f3c037820 0x7f3f3c039ce0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:54.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.158+0000 7f3f55244640 1 --2- 172.21.15.40:0/3439132168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f500a1820 0x7f3f5011e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:54.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.158+0000 7f3f55244640 1 --2- 172.21.15.40:0/3439132168 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f5013b0e0 0x7f3f5011ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:54.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.158+0000 7f3f55244640 1 --2- 172.21.15.40:0/3439132168 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f5009ee10 0x7f3f501c68d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:54.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.158+0000 7f3f55244640 1 -- 172.21.15.40:0/3439132168 >> 172.21.15.40:0/3439132168 conn(0x7f3f500a4e80 msgr2=0x7f3f500a70d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:54.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.159+0000 7f3f55244640 1 -- 172.21.15.40:0/3439132168 shutdown_connections 2024-04-22T08:12:54.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.159+0000 7f3f55244640 1 -- 172.21.15.40:0/3439132168 wait complete. 2024-04-22T08:12:54.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.163+0000 7f8a8384a640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-22T08:12:54.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.163+0000 7f8a8384a640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-22T08:12:54.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.163+0000 7f8a8384a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:54.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.163+0000 7f8a8384a640 1 -- 172.21.15.40:0/425098404 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f8a7c123c10 con 0x7f8a7c09ee10 2024-04-22T08:12:54.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.163+0000 7f8a71ffb640 1 -- 172.21.15.40:0/425098404 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8a78021090 con 0x7f8a7c09ee10 2024-04-22T08:12:54.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.163+0000 7f8a71ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-22T08:12:54.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.163+0000 7f8a71ffb640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:12:54.163 INFO:teuthology.orchestra.run.smithi040.stdout:47244640259 2024-04-22T08:12:54.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.165+0000 7f8a8384a640 1 -- 172.21.15.40:0/425098404 >> v2:172.21.15.189:6800/30168 conn(0x7f8a60037800 msgr2=0x7f8a60039cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:54.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.165+0000 7f8a8384a640 1 --2- 172.21.15.40:0/425098404 >> v2:172.21.15.189:6800/30168 conn(0x7f8a60037800 0x7f8a60039cc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c04e8a0 tx=0x7f8a6c0079b0 comp rx=0 tx=0).stop 2024-04-22T08:12:54.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.165+0000 7f8a8384a640 10 monclient: shutdown 2024-04-22T08:12:54.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.165+0000 7f8a8384a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:54.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.165+0000 7f8a8384a640 1 -- 172.21.15.40:0/425098404 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a7c09ee10 msgr2=0x7f8a7c1cabc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:54.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.165+0000 7f8a8384a640 1 --2- 172.21.15.40:0/425098404 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a7c09ee10 0x7f8a7c1cabc0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8a7800e880 tx=0x7f8a7800ed40 comp rx=0 tx=0).stop 2024-04-22T08:12:54.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.165+0000 7f8a8384a640 1 -- 172.21.15.40:0/425098404 shutdown_connections 2024-04-22T08:12:54.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.165+0000 7f8a8384a640 1 --2- 172.21.15.40:0/425098404 >> v2:172.21.15.189:6800/30168 conn(0x7f8a60037800 0x7f8a60039cc0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:54.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.165+0000 7f8a8384a640 1 --2- 172.21.15.40:0/425098404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a7c0a1820 0x7f8a7c1cb100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:54.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.165+0000 7f8a8384a640 1 --2- 172.21.15.40:0/425098404 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8a7c13b0e0 0x7f8a7c120390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:54.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.165+0000 7f8a8384a640 1 --2- 172.21.15.40:0/425098404 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a7c09ee10 0x7f8a7c1cabc0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:54.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.165+0000 7f8a8384a640 1 -- 172.21.15.40:0/425098404 >> 172.21.15.40:0/425098404 conn(0x7f8a7c0a4e80 msgr2=0x7f8a7c13d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:54.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.165+0000 7f8a8384a640 1 -- 172.21.15.40:0/425098404 shutdown_connections 2024-04-22T08:12:54.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.165+0000 7f8a8384a640 1 -- 172.21.15.40:0/425098404 wait complete. 2024-04-22T08:12:54.170 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.5 2024-04-22T08:12:54.170 DEBUG:teuthology.parallel:result is None 2024-04-22T08:12:54.176 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.3 2024-04-22T08:12:54.176 DEBUG:teuthology.parallel:result is None 2024-04-22T08:12:54.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.195+0000 7fb5e3ebd640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-22T08:12:54.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.195+0000 7fb5e3ebd640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-22T08:12:54.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.195+0000 7fb5e3ebd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:54.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.195+0000 7fb5e3ebd640 1 -- 172.21.15.40:0/3170709029 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fb5dc0a2590 con 0x7fb5dc09ee10 2024-04-22T08:12:54.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.196+0000 7fb5d27fc640 1 -- 172.21.15.40:0/3170709029 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb5dc0a2590 con 0x7fb5dc09ee10 2024-04-22T08:12:54.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.196+0000 7fb5d27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-22T08:12:54.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.196+0000 7fb5d27fc640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:12:54.198 INFO:teuthology.orchestra.run.smithi040.stdout:47244640258 2024-04-22T08:12:54.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.197+0000 7fb5e3ebd640 1 -- 172.21.15.40:0/3170709029 >> v2:172.21.15.189:6800/30168 conn(0x7fb5c0037820 msgr2=0x7fb5c0039ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:54.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.197+0000 7fb5e3ebd640 1 --2- 172.21.15.40:0/3170709029 >> v2:172.21.15.189:6800/30168 conn(0x7fb5c0037820 0x7fb5c0039ce0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc007c90 tx=0x7fb5cc007be0 comp rx=0 tx=0).stop 2024-04-22T08:12:54.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.197+0000 7fb5e3ebd640 10 monclient: shutdown 2024-04-22T08:12:54.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.197+0000 7fb5e3ebd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:54.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.197+0000 7fb5e3ebd640 1 -- 172.21.15.40:0/3170709029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5dc09ee10 msgr2=0x7fb5dc1cae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:54.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.197+0000 7fb5e3ebd640 1 --2- 172.21.15.40:0/3170709029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5dc09ee10 0x7fb5dc1cae10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8009890 tx=0x7fb5d80046c0 comp rx=0 tx=0).stop 2024-04-22T08:12:54.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.197+0000 7fb5e3ebd640 1 -- 172.21.15.40:0/3170709029 shutdown_connections 2024-04-22T08:12:54.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.197+0000 7fb5e3ebd640 1 --2- 172.21.15.40:0/3170709029 >> v2:172.21.15.189:6800/30168 conn(0x7fb5c0037820 0x7fb5c0039ce0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:54.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.197+0000 7fb5e3ebd640 1 --2- 172.21.15.40:0/3170709029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5dc0a2180 0x7fb5dc123330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:54.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.197+0000 7fb5e3ebd640 1 --2- 172.21.15.40:0/3170709029 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb5dc0a1d30 0x7fb5dc1cb350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:54.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.197+0000 7fb5e3ebd640 1 --2- 172.21.15.40:0/3170709029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5dc09ee10 0x7fb5dc1cae10 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:54.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.197+0000 7fb5e3ebd640 1 -- 172.21.15.40:0/3170709029 >> 172.21.15.40:0/3170709029 conn(0x7fb5dc0a4ef0 msgr2=0x7fb5dc071ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:54.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.198+0000 7fb5e3ebd640 1 -- 172.21.15.40:0/3170709029 shutdown_connections 2024-04-22T08:12:54.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:54.198+0000 7fb5e3ebd640 1 -- 172.21.15.40:0/3170709029 wait complete. 2024-04-22T08:12:54.208 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.1 2024-04-22T08:12:55.208 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-22T08:12:55.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.321+0000 7fd876cbd640 10 monclient: get_monmap_and_config 2024-04-22T08:12:55.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.321+0000 7fd876cbd640 10 monclient: build_initial_monmap 2024-04-22T08:12:55.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.321+0000 7fd876cbd640 10 monclient: monmap: 2024-04-22T08:12:55.321 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:55.321 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:55.321 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:55.322654+0000 2024-04-22T08:12:55.322 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:55.322654+0000 2024-04-22T08:12:55.322 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:55.322 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:55.322 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:12:55.322 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:12:55.322 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:12:55.322 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:12:55.322 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:12:55.322 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:55.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.322+0000 7fd876cbd640 1 Processor -- start 2024-04-22T08:12:55.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.322+0000 7fd876cbd640 1 -- start start 2024-04-22T08:12:55.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.322+0000 7fd876cbd640 10 monclient: init 2024-04-22T08:12:55.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd876cbd640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:55.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd876cbd640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-04-22T08:12:55.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd876cbd640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd8700a20d0 0x7fd8700a24b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:55.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd876cbd640 10 monclient(hunting): picked mon.noname-a con 0x7fd8700a20d0 addr v2:172.21.15.40:3300/0 2024-04-22T08:12:55.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd876cbd640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd8700a2d10 0x7fd8700a31c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:55.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd876cbd640 10 monclient(hunting): picked mon.noname-c con 0x7fd8700a2d10 addr v2:172.21.15.189:3300/0 2024-04-22T08:12:55.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd876cbd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd8700a3700 0x7fd87013cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:55.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd876cbd640 10 monclient(hunting): picked mon.noname-b con 0x7fd8700a3700 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:55.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd876cbd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:55.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd876cbd640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8700a3b40 con 0x7fd8700a20d0 2024-04-22T08:12:55.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd876cbd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:55.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd876cbd640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8700a45f0 con 0x7fd8700a2d10 2024-04-22T08:12:55.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd876cbd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:55.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd876cbd640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8700a0360 con 0x7fd8700a3700 2024-04-22T08:12:55.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd876cbd640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:55.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd876cbd640 10 monclient(hunting): authenticate will time out at 849.317094s 2024-04-22T08:12:55.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd874a59640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd8700a20d0 0x7fd8700a24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:55.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd874a59640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd8700a20d0 0x7fd8700a24b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58062/0 (socket says 172.21.15.40:58062) 2024-04-22T08:12:55.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd87525a640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd8700a3700 0x7fd87013cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:55.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd874a59640 1 -- 172.21.15.40:0/1661207956 learned_addr learned my addr 172.21.15.40:0/1661207956 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:55.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd874a59640 10 monclient(hunting): get_auth_request con 0x7fd8700a20d0 auth_method 0 2024-04-22T08:12:55.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd874a59640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:55.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd874a59640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:55.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd874a59640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:55.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd87525a640 10 monclient(hunting): get_auth_request con 0x7fd8700a3700 auth_method 0 2024-04-22T08:12:55.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd87525a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:55.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd87525a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:55.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd87525a640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:55.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd874a59640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:55.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd874a59640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:55.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd874a59640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:55.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd87525a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:55.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd87525a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:55.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd87525a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:55.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd86ffff640 1 --2- 172.21.15.40:0/1661207956 >> v2:172.21.15.189:3300/0 conn(0x7fd8700a2d10 0x7fd8700a31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:55.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd86ffff640 10 monclient(hunting): get_auth_request con 0x7fd8700a2d10 auth_method 0 2024-04-22T08:12:55.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd86ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:55.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd86ffff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:55.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd86ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:55.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd874a59640 10 monclient(hunting): handle_auth_done global_id 4317 payload 274 2024-04-22T08:12:55.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd874a59640 1 -- 172.21.15.40:0/1661207956 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd8700a3700 msgr2=0x7fd87013cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:55.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd874a59640 1 --2- 172.21.15.40:0/1661207956 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd8700a3700 0x7fd87013cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:55.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd874a59640 1 -- 172.21.15.40:0/1661207956 >> v2:172.21.15.189:3300/0 conn(0x7fd8700a2d10 msgr2=0x7fd8700a31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:55.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd874a59640 1 --2- 172.21.15.40:0/1661207956 >> v2:172.21.15.189:3300/0 conn(0x7fd8700a2d10 0x7fd8700a31c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:55.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd874a59640 10 monclient: _finish_hunting 0 2024-04-22T08:12:55.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd874a59640 1 monclient: found mon.noname-a 2024-04-22T08:12:55.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd874a59640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:55.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd874a59640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:12:55.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd874a59640 1 -- 172.21.15.40:0/1661207956 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8700a0640 con 0x7fd8700a20d0 2024-04-22T08:12:55.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.323+0000 7fd87525a640 1 --2- 172.21.15.40:0/1661207956 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd8700a3700 0x7fd87013cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:12:55.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.326+0000 7fd874a59640 1 -- 172.21.15.40:0/1661207956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8700a20d0 msgr2=0x7fd8700a24b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:55.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.326+0000 7fd874a59640 1 --2- 172.21.15.40:0/1661207956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8700a20d0 0x7fd8700a24b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd8700a5bd0 tx=0x7fd86002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63ff0542e76a3b89 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:55.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.326+0000 7fd86f7fe640 1 -- 172.21.15.40:0/1661207956 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd860031960 con 0x7fd8700a20d0 2024-04-22T08:12:55.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.326+0000 7fd86f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:55.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.326+0000 7fd86f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:12:55.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.326+0000 7fd86f7fe640 10 monclient: dump: 2024-04-22T08:12:55.334 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:55.334 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:55.334 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:55.334 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:55.334 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:55.334 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:55.335 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:55.335 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:55.335 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:55.335 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:55.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.326+0000 7fd86f7fe640 10 monclient: _finish_auth 0 2024-04-22T08:12:55.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.326+0000 7fd86f7fe640 10 monclient: _check_auth_tickets 2024-04-22T08:12:55.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.326+0000 7fd86f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:55.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.326+0000 7fd86f7fe640 1 -- 172.21.15.40:0/1661207956 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd860031ac0 con 0x7fd8700a20d0 2024-04-22T08:12:55.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.326+0000 7fd876cbd640 5 monclient: authenticate success, global_id 4317 2024-04-22T08:12:55.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.326+0000 7fd876cbd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:12:55.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.326+0000 7fd86f7fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:55.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.326+0000 7fd876cbd640 10 monclient: get_monmap_and_config success 2024-04-22T08:12:55.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.326+0000 7fd86f7fe640 1 -- 172.21.15.40:0/1661207956 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd860031e50 con 0x7fd8700a20d0 2024-04-22T08:12:55.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.326+0000 7fd86f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:55.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.326+0000 7fd876cbd640 4 set_mon_vals no callback set 2024-04-22T08:12:55.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.326+0000 7fd86f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:12:55.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.326+0000 7fd86f7fe640 10 monclient: dump: 2024-04-22T08:12:55.336 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:55.336 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:55.336 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:55.336 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:55.336 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:55.336 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:55.336 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:55.336 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:55.336 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:55.336 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:55.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.327+0000 7fd876cbd640 10 monclient: shutdown 2024-04-22T08:12:55.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.327+0000 7fd876cbd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:55.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.327+0000 7fd876cbd640 1 -- 172.21.15.40:0/1661207956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8700a20d0 msgr2=0x7fd8700a24b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:55.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.327+0000 7fd876cbd640 1 --2- 172.21.15.40:0/1661207956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8700a20d0 0x7fd8700a24b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd8700a5bd0 tx=0x7fd86002b7f0 comp rx=0 tx=0).stop 2024-04-22T08:12:55.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.327+0000 7fd876cbd640 1 -- 172.21.15.40:0/1661207956 shutdown_connections 2024-04-22T08:12:55.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.327+0000 7fd876cbd640 1 --2- 172.21.15.40:0/1661207956 >> v2:172.21.15.189:3300/0 conn(0x7fd8700a2d10 0x7fd8700a31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:55.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.327+0000 7fd876cbd640 1 --2- 172.21.15.40:0/1661207956 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd8700a3700 0x7fd87013cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:55.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.327+0000 7fd876cbd640 1 --2- 172.21.15.40:0/1661207956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8700a20d0 0x7fd8700a24b0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:55.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.327+0000 7fd876cbd640 1 -- 172.21.15.40:0/1661207956 >> 172.21.15.40:0/1661207956 conn(0x7fd8700a8e60 msgr2=0x7fd8700a4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:55.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.327+0000 7fd876cbd640 1 -- 172.21.15.40:0/1661207956 shutdown_connections 2024-04-22T08:12:55.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.327+0000 7fd876cbd640 1 -- 172.21.15.40:0/1661207956 wait complete. 2024-04-22T08:12:55.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.328+0000 7fd876cbd640 10 monclient: build_initial_monmap 2024-04-22T08:12:55.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.328+0000 7fd876cbd640 10 monclient: monmap: 2024-04-22T08:12:55.337 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:55.337 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:12:55.337 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:12:55.337 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:12:55.338 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:55.338 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:55.338 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:12:55.338 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:12:55.338 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:12:55.338 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:55.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.328+0000 7fd876cbd640 1 Processor -- start 2024-04-22T08:12:55.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.328+0000 7fd876cbd640 1 -- start start 2024-04-22T08:12:55.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.328+0000 7fd876cbd640 10 monclient: init 2024-04-22T08:12:55.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd876cbd640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:55.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd876cbd640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:12:55.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd876cbd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8700a20d0 0x7fd8701c4320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:55.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd876cbd640 10 monclient(hunting): picked mon.noname-a con 0x7fd8700a20d0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:55.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd876cbd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8700a2d10 0x7fd8701c6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:55.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd876cbd640 10 monclient(hunting): picked mon.noname-b con 0x7fd8700a2d10 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:55.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd876cbd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd8700a3700 0x7fd8701c74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:55.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd876cbd640 10 monclient(hunting): picked mon.noname-c con 0x7fd8700a3700 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:55.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd876cbd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:55.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd876cbd640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8701abd80 con 0x7fd8700a20d0 2024-04-22T08:12:55.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd876cbd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:55.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd876cbd640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd870071750 con 0x7fd8700a2d10 2024-04-22T08:12:55.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd876cbd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:55.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd876cbd640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8701a9940 con 0x7fd8700a3700 2024-04-22T08:12:55.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd876cbd640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:55.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd876cbd640 10 monclient(hunting): authenticate will time out at 849.322876s 2024-04-22T08:12:55.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd874a59640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8700a20d0 0x7fd8701c4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:55.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd87525a640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd8700a3700 0x7fd8701c74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:55.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd874a59640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8700a20d0 0x7fd8701c4320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58070/0 (socket says 172.21.15.40:58070) 2024-04-22T08:12:55.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd874a59640 1 -- 172.21.15.40:0/2727904164 learned_addr learned my addr 172.21.15.40:0/2727904164 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:55.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd874a59640 10 monclient(hunting): get_auth_request con 0x7fd8700a20d0 auth_method 0 2024-04-22T08:12:55.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd874a59640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:55.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd874a59640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:55.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd874a59640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:55.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd87525a640 10 monclient(hunting): get_auth_request con 0x7fd8700a3700 auth_method 0 2024-04-22T08:12:55.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd87525a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:55.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd87525a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:55.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd87525a640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:55.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd874a59640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:55.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd874a59640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:55.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd874a59640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:55.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd87525a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:55.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd87525a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:55.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd87525a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:55.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd86ffff640 1 --2- 172.21.15.40:0/2727904164 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8700a2d10 0x7fd8701c6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:55.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd86ffff640 10 monclient(hunting): get_auth_request con 0x7fd8700a2d10 auth_method 0 2024-04-22T08:12:55.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd86ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:55.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd86ffff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:55.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd86ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:55.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd86ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:55.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd86ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:55.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd86ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:55.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd874a59640 10 monclient(hunting): handle_auth_done global_id 4320 payload 824 2024-04-22T08:12:55.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd874a59640 1 -- 172.21.15.40:0/2727904164 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd8700a3700 msgr2=0x7fd8701c74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:55.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd874a59640 1 --2- 172.21.15.40:0/2727904164 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd8700a3700 0x7fd8701c74e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:55.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd874a59640 1 -- 172.21.15.40:0/2727904164 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8700a2d10 msgr2=0x7fd8701c6fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:55.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd874a59640 1 --2- 172.21.15.40:0/2727904164 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8700a2d10 0x7fd8701c6fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:55.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd874a59640 10 monclient: _finish_hunting 0 2024-04-22T08:12:55.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd874a59640 1 monclient: found mon.noname-a 2024-04-22T08:12:55.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd874a59640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:55.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd874a59640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:12:55.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.329+0000 7fd874a59640 1 -- 172.21.15.40:0/2727904164 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd868009a70 con 0x7fd8700a20d0 2024-04-22T08:12:55.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.331+0000 7fd874a59640 1 -- 172.21.15.40:0/2727904164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8700a20d0 msgr2=0x7fd8701c4320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:55.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.331+0000 7fd874a59640 1 --2- 172.21.15.40:0/2727904164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8700a20d0 0x7fd8701c4320 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd8600090b0 tx=0x7fd8600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:55.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.332+0000 7fd86d7fa640 1 -- 172.21.15.40:0/2727904164 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd860031e30 con 0x7fd8700a20d0 2024-04-22T08:12:55.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.332+0000 7fd86d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:55.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.332+0000 7fd86d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:12:55.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.332+0000 7fd86d7fa640 10 monclient: dump: 2024-04-22T08:12:55.342 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:55.342 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:55.342 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:55.342 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:55.343 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:55.343 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:55.343 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:55.343 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:55.343 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:55.343 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:55.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.332+0000 7fd86d7fa640 10 monclient: _finish_auth 0 2024-04-22T08:12:55.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.332+0000 7fd86d7fa640 10 monclient: _check_auth_tickets 2024-04-22T08:12:55.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.332+0000 7fd86d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:55.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.332+0000 7fd86d7fa640 1 -- 172.21.15.40:0/2727904164 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd86003c400 con 0x7fd8700a20d0 2024-04-22T08:12:55.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.332+0000 7fd876cbd640 5 monclient: authenticate success, global_id 4320 2024-04-22T08:12:55.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.332+0000 7fd876cbd640 10 monclient: _renew_subs 2024-04-22T08:12:55.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.332+0000 7fd876cbd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:55.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.332+0000 7fd876cbd640 1 -- 172.21.15.40:0/2727904164 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8600094d0 con 0x7fd8700a20d0 2024-04-22T08:12:55.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.332+0000 7fd86d7fa640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:55.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.332+0000 7fd86d7fa640 1 -- 172.21.15.40:0/2727904164 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86003c7b0 con 0x7fd8700a20d0 2024-04-22T08:12:55.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.332+0000 7fd86d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:55.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.332+0000 7fd86d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:12:55.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.332+0000 7fd86d7fa640 10 monclient: dump: 2024-04-22T08:12:55.344 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:55.344 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:55.344 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:55.344 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:55.344 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:55.344 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:55.344 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:55.344 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:55.344 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:55.344 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:55.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.332+0000 7fd876cbd640 10 monclient: _renew_subs 2024-04-22T08:12:55.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.332+0000 7fd86e7fc640 4 set_mon_vals no callback set 2024-04-22T08:12:55.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.332+0000 7fd876cbd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:55.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.332+0000 7fd876cbd640 1 -- 172.21.15.40:0/2727904164 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8701ab3e0 con 0x7fd8700a20d0 2024-04-22T08:12:55.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.333+0000 7fd86d7fa640 1 -- 172.21.15.40:0/2727904164 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fd86003a040 con 0x7fd8700a20d0 2024-04-22T08:12:55.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.333+0000 7fd876cbd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:55.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.333+0000 7fd876cbd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:55.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.333+0000 7fd876cbd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:55.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.333+0000 7fd876cbd640 1 -- 172.21.15.40:0/2727904164 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8701aba00 con 0x7fd8700a20d0 2024-04-22T08:12:55.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.333+0000 7fd86d7fa640 1 --2- 172.21.15.40:0/2727904164 >> v2:172.21.15.189:6800/30168 conn(0x7fd85c037850 0x7fd85c039d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:55.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.333+0000 7fd86d7fa640 1 -- 172.21.15.40:0/2727904164 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3506+0+0 (secure 0 0 0) 0x7fd86006b970 con 0x7fd8700a20d0 2024-04-22T08:12:55.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.333+0000 7fd86ffff640 1 --2- 172.21.15.40:0/2727904164 >> v2:172.21.15.189:6800/30168 conn(0x7fd85c037850 0x7fd85c039d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:55.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.333+0000 7fd86ffff640 10 monclient: get_auth_request con 0x7fd85c037850 auth_method 0 2024-04-22T08:12:55.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.336+0000 7fd86ffff640 1 --2- 172.21.15.40:0/2727904164 >> v2:172.21.15.189:6800/30168 conn(0x7fd85c037850 0x7fd85c039d10 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd8700a77b0 tx=0x7fd864008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:55.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.338+0000 7fd86d7fa640 1 -- 172.21.15.40:0/2727904164 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7fd86003ce30 con 0x7fd8700a20d0 2024-04-22T08:12:55.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.338+0000 7fd86d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:55.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.338+0000 7fd86d7fa640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:12:55.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.518+0000 7fd876cbd640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-22T08:12:55.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.518+0000 7fd876cbd640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-22T08:12:55.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.518+0000 7fd876cbd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:55.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.518+0000 7fd876cbd640 1 -- 172.21.15.40:0/2727904164 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fd8700a4ac0 con 0x7fd8700a20d0 2024-04-22T08:12:55.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.519+0000 7fd86d7fa640 1 -- 172.21.15.40:0/2727904164 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd860038070 con 0x7fd8700a20d0 2024-04-22T08:12:55.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.519+0000 7fd86d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-22T08:12:55.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.519+0000 7fd86d7fa640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:12:55.519 INFO:teuthology.orchestra.run.smithi040.stdout:47244640260 2024-04-22T08:12:55.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.521+0000 7fd876cbd640 1 -- 172.21.15.40:0/2727904164 >> v2:172.21.15.189:6800/30168 conn(0x7fd85c037850 msgr2=0x7fd85c039d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:55.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.521+0000 7fd876cbd640 1 --2- 172.21.15.40:0/2727904164 >> v2:172.21.15.189:6800/30168 conn(0x7fd85c037850 0x7fd85c039d10 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd8700a77b0 tx=0x7fd864008040 comp rx=0 tx=0).stop 2024-04-22T08:12:55.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.521+0000 7fd876cbd640 10 monclient: shutdown 2024-04-22T08:12:55.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.521+0000 7fd876cbd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:55.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.521+0000 7fd876cbd640 1 -- 172.21.15.40:0/2727904164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8700a20d0 msgr2=0x7fd8701c4320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:55.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.521+0000 7fd876cbd640 1 --2- 172.21.15.40:0/2727904164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8700a20d0 0x7fd8701c4320 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd8600090b0 tx=0x7fd8600046c0 comp rx=0 tx=0).stop 2024-04-22T08:12:55.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.521+0000 7fd876cbd640 1 -- 172.21.15.40:0/2727904164 shutdown_connections 2024-04-22T08:12:55.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.521+0000 7fd876cbd640 1 --2- 172.21.15.40:0/2727904164 >> v2:172.21.15.189:6800/30168 conn(0x7fd85c037850 0x7fd85c039d10 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:55.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.521+0000 7fd876cbd640 1 --2- 172.21.15.40:0/2727904164 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8700a2d10 0x7fd8701c6fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:55.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.521+0000 7fd876cbd640 1 --2- 172.21.15.40:0/2727904164 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd8700a3700 0x7fd8701c74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:55.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.521+0000 7fd876cbd640 1 --2- 172.21.15.40:0/2727904164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8700a20d0 0x7fd8701c4320 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:55.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.521+0000 7fd876cbd640 1 -- 172.21.15.40:0/2727904164 >> 172.21.15.40:0/2727904164 conn(0x7fd8700a8e60 msgr2=0x7fd87006eea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:55.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.521+0000 7fd876cbd640 1 -- 172.21.15.40:0/2727904164 shutdown_connections 2024-04-22T08:12:55.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.521+0000 7fd876cbd640 1 -- 172.21.15.40:0/2727904164 wait complete. 2024-04-22T08:12:55.531 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640260 for osd.1 2024-04-22T08:12:55.532 DEBUG:teuthology.parallel:result is None 2024-04-22T08:12:55.532 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-22T08:12:55.532 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:12:55.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.632+0000 7efe32c70640 10 monclient: get_monmap_and_config 2024-04-22T08:12:55.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.632+0000 7efe32c70640 10 monclient: build_initial_monmap 2024-04-22T08:12:55.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.632+0000 7efe32c70640 10 monclient: monmap: 2024-04-22T08:12:55.631 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:55.632 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:55.632 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:55.632837+0000 2024-04-22T08:12:55.632 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:55.632837+0000 2024-04-22T08:12:55.632 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:55.632 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:55.632 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:12:55.632 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:12:55.632 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:12:55.632 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:12:55.632 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:12:55.632 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:55.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.632+0000 7efe32c70640 1 Processor -- start 2024-04-22T08:12:55.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.632+0000 7efe32c70640 1 -- start start 2024-04-22T08:12:55.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.632+0000 7efe32c70640 10 monclient: init 2024-04-22T08:12:55.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.633+0000 7efe32c70640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:55.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.633+0000 7efe32c70640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-04-22T08:12:55.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.633+0000 7efe32c70640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe2c09c150 0x7efe2c09c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:55.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.633+0000 7efe32c70640 10 monclient(hunting): picked mon.noname-b con 0x7efe2c09c150 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:55.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.633+0000 7efe32c70640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efe2c0999f0 addr v1:172.21.15.40:6789/0 2024-04-22T08:12:55.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.633+0000 7efe32c70640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efe2c0a2190 0x7efe2c13de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:55.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.633+0000 7efe32c70640 10 monclient(hunting): picked mon.noname-a con 0x7efe2c0a2190 addr v2:172.21.15.40:3300/0 2024-04-22T08:12:55.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.633+0000 7efe32c70640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe2c09dca0 con 0x7efe2c0999f0 2024-04-22T08:12:55.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.633+0000 7efe32c70640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:55.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.633+0000 7efe32c70640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe2c072270 con 0x7efe2c0a2190 2024-04-22T08:12:55.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.633+0000 7efe32c70640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:55.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.633+0000 7efe32c70640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe2c09f580 con 0x7efe2c09c150 2024-04-22T08:12:55.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.633+0000 7efe32c70640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:55.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.633+0000 7efe32c70640 10 monclient(hunting): authenticate will time out at 849.627169s 2024-04-22T08:12:55.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.633+0000 7efe30a0c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe2c09c150 0x7efe2c09c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:55.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.633+0000 7efe3120d640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efe2c0a2190 0x7efe2c13de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:55.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.633+0000 7efe30a0c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe2c09c150 0x7efe2c09c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49870/0 (socket says 172.21.15.40:49870) 2024-04-22T08:12:55.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.633+0000 7efe2bfff640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7efe2c0999f0 0x7efe2c0a1a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:38964/0 (socket says 172.21.15.40:38964) 2024-04-22T08:12:55.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.633+0000 7efe30a0c640 1 -- 172.21.15.40:0/3095646571 learned_addr learned my addr 172.21.15.40:0/3095646571 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:55.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.633+0000 7efe3120d640 10 monclient(hunting): get_auth_request con 0x7efe2c0a2190 auth_method 0 2024-04-22T08:12:55.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.633+0000 7efe3120d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:55.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.633+0000 7efe3120d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:55.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.633+0000 7efe3120d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:55.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.633+0000 7efe30a0c640 10 monclient(hunting): get_auth_request con 0x7efe2c09c150 auth_method 0 2024-04-22T08:12:55.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.633+0000 7efe30a0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:55.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.633+0000 7efe30a0c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:55.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.633+0000 7efe30a0c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:55.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.633+0000 7efe3120d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:55.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.633+0000 7efe3120d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:55.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.633+0000 7efe3120d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:55.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.633+0000 7efe30a0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:55.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.633+0000 7efe30a0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:55.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.633+0000 7efe30a0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:55.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.633+0000 7efe3120d640 10 monclient(hunting): handle_auth_done global_id 4323 payload 274 2024-04-22T08:12:55.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.633+0000 7efe3120d640 1 -- 172.21.15.40:0/3095646571 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe2c09c150 msgr2=0x7efe2c09c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:55.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.633+0000 7efe3120d640 1 --2- 172.21.15.40:0/3095646571 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe2c09c150 0x7efe2c09c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:55.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.633+0000 7efe3120d640 1 -- 172.21.15.40:0/3095646571 >> v1:172.21.15.40:6789/0 conn(0x7efe2c0999f0 legacy=0x7efe2c0a1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:55.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.633+0000 7efe3120d640 10 monclient: _finish_hunting 0 2024-04-22T08:12:55.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.633+0000 7efe3120d640 1 monclient: found mon.noname-a 2024-04-22T08:12:55.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.633+0000 7efe3120d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:55.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.633+0000 7efe3120d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:12:55.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.633+0000 7efe3120d640 1 -- 172.21.15.40:0/3095646571 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe2c09efc0 con 0x7efe2c0a2190 2024-04-22T08:12:55.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.633+0000 7efe30a0c640 1 --2- 172.21.15.40:0/3095646571 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe2c09c150 0x7efe2c09c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:12:55.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.636+0000 7efe3120d640 1 -- 172.21.15.40:0/3095646571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe2c0a2190 msgr2=0x7efe2c13de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:55.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.636+0000 7efe3120d640 1 --2- 172.21.15.40:0/3095646571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe2c0a2190 0x7efe2c13de80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7efe24009d90 tx=0x7efe2402bd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d4360b16ff5277a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:55.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.636+0000 7efe2b7fe640 1 -- 172.21.15.40:0/3095646571 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe2402ccc0 con 0x7efe2c0a2190 2024-04-22T08:12:55.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.636+0000 7efe2b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:55.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.636+0000 7efe2b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:12:55.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.636+0000 7efe2b7fe640 10 monclient: dump: 2024-04-22T08:12:55.644 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:55.644 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:55.644 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:55.644 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:55.644 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:55.644 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:55.644 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:55.644 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:55.644 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:55.644 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:55.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.636+0000 7efe2b7fe640 10 monclient: _finish_auth 0 2024-04-22T08:12:55.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.636+0000 7efe2b7fe640 10 monclient: _check_auth_tickets 2024-04-22T08:12:55.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.636+0000 7efe2b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:55.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.636+0000 7efe2b7fe640 1 -- 172.21.15.40:0/3095646571 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe2402ce20 con 0x7efe2c0a2190 2024-04-22T08:12:55.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.636+0000 7efe32c70640 5 monclient: authenticate success, global_id 4323 2024-04-22T08:12:55.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.636+0000 7efe32c70640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:12:55.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.636+0000 7efe2b7fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:55.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.636+0000 7efe32c70640 10 monclient: get_monmap_and_config success 2024-04-22T08:12:55.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.636+0000 7efe32c70640 4 set_mon_vals no callback set 2024-04-22T08:12:55.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.636+0000 7efe2b7fe640 1 -- 172.21.15.40:0/3095646571 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe24034660 con 0x7efe2c0a2190 2024-04-22T08:12:55.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.636+0000 7efe2b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:55.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.636+0000 7efe2b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:12:55.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.636+0000 7efe2b7fe640 10 monclient: dump: 2024-04-22T08:12:55.645 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:55.645 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:55.646 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:55.646 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:55.646 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:55.646 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:55.646 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:55.646 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:55.646 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:55.646 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:55.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.637+0000 7efe32c70640 10 monclient: shutdown 2024-04-22T08:12:55.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.637+0000 7efe32c70640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:55.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.637+0000 7efe32c70640 1 -- 172.21.15.40:0/3095646571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe2c0a2190 msgr2=0x7efe2c13de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:55.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.637+0000 7efe32c70640 1 --2- 172.21.15.40:0/3095646571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe2c0a2190 0x7efe2c13de80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7efe24009d90 tx=0x7efe2402bd00 comp rx=0 tx=0).stop 2024-04-22T08:12:55.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.637+0000 7efe32c70640 1 -- 172.21.15.40:0/3095646571 shutdown_connections 2024-04-22T08:12:55.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.637+0000 7efe32c70640 1 --2- 172.21.15.40:0/3095646571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe2c0a2190 0x7efe2c13de80 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:55.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.637+0000 7efe32c70640 1 --2- 172.21.15.40:0/3095646571 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe2c09c150 0x7efe2c09c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:55.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.637+0000 7efe32c70640 1 -- 172.21.15.40:0/3095646571 >> 172.21.15.40:0/3095646571 conn(0x7efe2c0a66a0 msgr2=0x7efe2c0a2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:55.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.637+0000 7efe32c70640 1 -- 172.21.15.40:0/3095646571 shutdown_connections 2024-04-22T08:12:55.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.637+0000 7efe32c70640 1 -- 172.21.15.40:0/3095646571 wait complete. 2024-04-22T08:12:55.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.638+0000 7efe32c70640 10 monclient: build_initial_monmap 2024-04-22T08:12:55.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.638+0000 7efe32c70640 10 monclient: monmap: 2024-04-22T08:12:55.647 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:55.647 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:12:55.647 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:12:55.647 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:12:55.647 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:55.647 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:55.647 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:12:55.647 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:12:55.648 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:12:55.648 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:55.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.638+0000 7efe32c70640 1 Processor -- start 2024-04-22T08:12:55.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.638+0000 7efe32c70640 1 -- start start 2024-04-22T08:12:55.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.638+0000 7efe32c70640 10 monclient: init 2024-04-22T08:12:55.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe32c70640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:55.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe32c70640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:12:55.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe32c70640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe2c0999f0 0x7efe2c1ca980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:55.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe32c70640 10 monclient(hunting): picked mon.noname-a con 0x7efe2c0999f0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:55.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe32c70640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe2c09c150 0x7efe2c1b9c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:55.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe32c70640 10 monclient(hunting): picked mon.noname-b con 0x7efe2c09c150 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:55.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe32c70640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe2c0a2190 0x7efe2c1ba1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:55.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe32c70640 10 monclient(hunting): picked mon.noname-c con 0x7efe2c0a2190 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:55.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe32c70640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:55.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe32c70640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe2c1a70b0 con 0x7efe2c0999f0 2024-04-22T08:12:55.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe32c70640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:55.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe32c70640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe2c09cce0 con 0x7efe2c09c150 2024-04-22T08:12:55.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe32c70640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:55.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe32c70640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe2c1a5c60 con 0x7efe2c0a2190 2024-04-22T08:12:55.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe32c70640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:55.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe32c70640 10 monclient(hunting): authenticate will time out at 849.632894s 2024-04-22T08:12:55.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe30a0c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe2c0999f0 0x7efe2c1ca980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:55.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe30a0c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe2c0999f0 0x7efe2c1ca980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58100/0 (socket says 172.21.15.40:58100) 2024-04-22T08:12:55.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe30a0c640 1 -- 172.21.15.40:0/2466970962 learned_addr learned my addr 172.21.15.40:0/2466970962 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:55.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe30a0c640 10 monclient(hunting): get_auth_request con 0x7efe2c0999f0 auth_method 0 2024-04-22T08:12:55.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe30a0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:55.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe30a0c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:55.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe30a0c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:55.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe3120d640 1 --2- 172.21.15.40:0/2466970962 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe2c0a2190 0x7efe2c1ba1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:55.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe3120d640 10 monclient(hunting): get_auth_request con 0x7efe2c0a2190 auth_method 0 2024-04-22T08:12:55.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe3120d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:55.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe3120d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:55.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe3120d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:55.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe30a0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:55.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe30a0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:55.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe30a0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:55.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe3120d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:55.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe3120d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:55.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe3120d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:55.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe2bfff640 1 --2- 172.21.15.40:0/2466970962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe2c09c150 0x7efe2c1b9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:55.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe2bfff640 10 monclient(hunting): get_auth_request con 0x7efe2c09c150 auth_method 0 2024-04-22T08:12:55.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe2bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:55.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe2bfff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:55.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:55.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe30a0c640 10 monclient(hunting): handle_auth_done global_id 4326 payload 824 2024-04-22T08:12:55.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe30a0c640 1 -- 172.21.15.40:0/2466970962 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe2c0a2190 msgr2=0x7efe2c1ba1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:55.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe30a0c640 1 --2- 172.21.15.40:0/2466970962 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe2c0a2190 0x7efe2c1ba1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:55.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe30a0c640 1 -- 172.21.15.40:0/2466970962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe2c09c150 msgr2=0x7efe2c1b9c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:55.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe30a0c640 1 --2- 172.21.15.40:0/2466970962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe2c09c150 0x7efe2c1b9c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:55.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe30a0c640 10 monclient: _finish_hunting 0 2024-04-22T08:12:55.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe30a0c640 1 monclient: found mon.noname-a 2024-04-22T08:12:55.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe30a0c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:55.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe30a0c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:12:55.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.639+0000 7efe30a0c640 1 -- 172.21.15.40:0/2466970962 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe1c0094d0 con 0x7efe2c0999f0 2024-04-22T08:12:55.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.641+0000 7efe30a0c640 1 -- 172.21.15.40:0/2466970962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe2c0999f0 msgr2=0x7efe2c1ca980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:55.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.641+0000 7efe30a0c640 1 --2- 172.21.15.40:0/2466970962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe2c0999f0 0x7efe2c1ca980 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7efe1c00e980 tx=0x7efe1c00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:55.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.641+0000 7efe297fa640 1 -- 172.21.15.40:0/2466970962 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe1c00cc10 con 0x7efe2c0999f0 2024-04-22T08:12:55.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.641+0000 7efe297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:55.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.641+0000 7efe297fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:12:55.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.641+0000 7efe297fa640 10 monclient: dump: 2024-04-22T08:12:55.652 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:55.652 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:55.652 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:55.652 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:55.652 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:55.653 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:55.653 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:55.653 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:55.653 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:55.653 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:55.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.641+0000 7efe297fa640 10 monclient: _finish_auth 0 2024-04-22T08:12:55.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.641+0000 7efe297fa640 10 monclient: _check_auth_tickets 2024-04-22T08:12:55.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.641+0000 7efe297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:55.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.641+0000 7efe32c70640 5 monclient: authenticate success, global_id 4326 2024-04-22T08:12:55.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.641+0000 7efe297fa640 1 -- 172.21.15.40:0/2466970962 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe1c00cd70 con 0x7efe2c0999f0 2024-04-22T08:12:55.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.641+0000 7efe32c70640 10 monclient: _renew_subs 2024-04-22T08:12:55.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.641+0000 7efe32c70640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:55.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.641+0000 7efe32c70640 1 -- 172.21.15.40:0/2466970962 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe24009a70 con 0x7efe2c0999f0 2024-04-22T08:12:55.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.641+0000 7efe297fa640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:55.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.641+0000 7efe297fa640 1 -- 172.21.15.40:0/2466970962 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe1c004230 con 0x7efe2c0999f0 2024-04-22T08:12:55.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.641+0000 7efe297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:55.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.641+0000 7efe297fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:12:55.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.641+0000 7efe297fa640 10 monclient: dump: 2024-04-22T08:12:55.654 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:55.654 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:55.654 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:55.654 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:55.654 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:55.655 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:55.655 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:55.655 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:55.655 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:55.655 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:55.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.641+0000 7efe2a7fc640 4 set_mon_vals no callback set 2024-04-22T08:12:55.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.641+0000 7efe32c70640 10 monclient: _renew_subs 2024-04-22T08:12:55.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.641+0000 7efe32c70640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:55.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.641+0000 7efe32c70640 1 -- 172.21.15.40:0/2466970962 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe2c071880 con 0x7efe2c0999f0 2024-04-22T08:12:55.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.642+0000 7efe297fa640 1 -- 172.21.15.40:0/2466970962 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7efe1c0043d0 con 0x7efe2c0999f0 2024-04-22T08:12:55.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.642+0000 7efe297fa640 1 --2- 172.21.15.40:0/2466970962 >> v2:172.21.15.189:6800/30168 conn(0x7efe180377d0 0x7efe18039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:55.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.642+0000 7efe32c70640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:55.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.642+0000 7efe32c70640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:55.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.642+0000 7efe32c70640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:55.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.643+0000 7efe297fa640 1 -- 172.21.15.40:0/2466970962 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3506+0+0 (secure 0 0 0) 0x7efe1c04a3c0 con 0x7efe2c0999f0 2024-04-22T08:12:55.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.642+0000 7efe32c70640 1 -- 172.21.15.40:0/2466970962 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe2c075260 con 0x7efe2c0999f0 2024-04-22T08:12:55.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.643+0000 7efe2bfff640 1 --2- 172.21.15.40:0/2466970962 >> v2:172.21.15.189:6800/30168 conn(0x7efe180377d0 0x7efe18039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:55.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.643+0000 7efe2bfff640 10 monclient: get_auth_request con 0x7efe180377d0 auth_method 0 2024-04-22T08:12:55.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.645+0000 7efe2bfff640 1 --2- 172.21.15.40:0/2466970962 >> v2:172.21.15.189:6800/30168 conn(0x7efe180377d0 0x7efe18039c90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7efe2c071430 tx=0x7efe20009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:55.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.647+0000 7efe297fa640 1 -- 172.21.15.40:0/2466970962 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7efe1c013470 con 0x7efe2c0999f0 2024-04-22T08:12:55.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.647+0000 7efe297fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:55.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.647+0000 7efe297fa640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:12:55.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.822+0000 7efe32c70640 1 -- 172.21.15.40:0/2466970962 --> v2:172.21.15.189:6800/30168 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efe2c09ca70 con 0x7efe180377d0 2024-04-22T08:12:55.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.825+0000 7efe297fa640 1 -- 172.21.15.40:0/2466970962 <== mgr.4102 v2:172.21.15.189:6800/30168 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+40579 (secure 0 0 0) 0x7efe20006d10 con 0x7efe180377d0 2024-04-22T08:12:55.824 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:12:55.825 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-22T08:12:55.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.827+0000 7efe32c70640 1 -- 172.21.15.40:0/2466970962 >> v2:172.21.15.189:6800/30168 conn(0x7efe180377d0 msgr2=0x7efe18039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:55.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.827+0000 7efe32c70640 1 --2- 172.21.15.40:0/2466970962 >> v2:172.21.15.189:6800/30168 conn(0x7efe180377d0 0x7efe18039c90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7efe2c071430 tx=0x7efe20009040 comp rx=0 tx=0).stop 2024-04-22T08:12:55.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.827+0000 7efe32c70640 10 monclient: shutdown 2024-04-22T08:12:55.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.827+0000 7efe32c70640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:55.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.827+0000 7efe32c70640 1 -- 172.21.15.40:0/2466970962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe2c0999f0 msgr2=0x7efe2c1ca980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:55.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.827+0000 7efe32c70640 1 --2- 172.21.15.40:0/2466970962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe2c0999f0 0x7efe2c1ca980 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7efe1c00e980 tx=0x7efe1c00ee40 comp rx=0 tx=0).stop 2024-04-22T08:12:55.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.827+0000 7efe32c70640 1 -- 172.21.15.40:0/2466970962 shutdown_connections 2024-04-22T08:12:55.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.827+0000 7efe32c70640 1 --2- 172.21.15.40:0/2466970962 >> v2:172.21.15.189:6800/30168 conn(0x7efe180377d0 0x7efe18039c90 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:55.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.827+0000 7efe32c70640 1 --2- 172.21.15.40:0/2466970962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe2c09c150 0x7efe2c1b9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:55.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.827+0000 7efe32c70640 1 --2- 172.21.15.40:0/2466970962 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe2c0a2190 0x7efe2c1ba1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:55.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.827+0000 7efe32c70640 1 --2- 172.21.15.40:0/2466970962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe2c0999f0 0x7efe2c1ca980 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:55.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.827+0000 7efe32c70640 1 -- 172.21.15.40:0/2466970962 >> 172.21.15.40:0/2466970962 conn(0x7efe2c0a66a0 msgr2=0x7efe2c09ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:55.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.827+0000 7efe32c70640 1 -- 172.21.15.40:0/2466970962 shutdown_connections 2024-04-22T08:12:55.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.827+0000 7efe32c70640 1 -- 172.21.15.40:0/2466970962 wait complete. 2024-04-22T08:12:55.837 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-04-22T08:12:54.077447+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":8,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":164644,"kb_used_data":1744,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566066396,"statfs":{"total":579820584960,"available":579651989504,"internally_reserved":0,"allocated":1785856,"data_stored":1223245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9560,"internal_metadata":165927592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000444"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:12:52.005174+0000","last_change":"2024-04-22T08:12:51.443086+0000","last_active":"2024-04-22T08:12:52.005174+0000","last_peered":"2024-04-22T08:12:52.005174+0000","last_clean":"2024-04-22T08:12:52.005174+0000","last_became_active":"2024-04-22T08:12:51.442730+0000","last_became_peered":"2024-04-22T08:12:51.442730+0000","last_unstale":"2024-04-22T08:12:52.005174+0000","last_undegraded":"2024-04-22T08:12:52.005174+0000","last_fullsized":"2024-04-22T08:12:52.005174+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:06:13.481863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:12:52.009552+0000","last_change":"2024-04-22T08:12:50.841276+0000","last_active":"2024-04-22T08:12:52.009552+0000","last_peered":"2024-04-22T08:12:52.009552+0000","last_clean":"2024-04-22T08:12:52.009552+0000","last_became_active":"2024-04-22T08:12:50.841008+0000","last_became_peered":"2024-04-22T08:12:50.841008+0000","last_unstale":"2024-04-22T08:12:52.009552+0000","last_undegraded":"2024-04-22T08:12:52.009552+0000","last_fullsized":"2024-04-22T08:12:52.009552+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:47:00.224321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:12:52.005217+0000","last_change":"2024-04-22T08:12:51.403419+0000","last_active":"2024-04-22T08:12:52.005217+0000","last_peered":"2024-04-22T08:12:52.005217+0000","last_clean":"2024-04-22T08:12:52.005217+0000","last_became_active":"2024-04-22T08:12:51.403182+0000","last_became_peered":"2024-04-22T08:12:51.403182+0000","last_unstale":"2024-04-22T08:12:52.005217+0000","last_undegraded":"2024-04-22T08:12:52.005217+0000","last_fullsized":"2024-04-22T08:12:52.005217+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:52:39.730597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:12:52.009486+0000","last_change":"2024-04-22T08:12:51.403260+0000","last_active":"2024-04-22T08:12:52.009486+0000","last_peered":"2024-04-22T08:12:52.009486+0000","last_clean":"2024-04-22T08:12:52.009486+0000","last_became_active":"2024-04-22T08:12:51.402980+0000","last_became_peered":"2024-04-22T08:12:51.402980+0000","last_unstale":"2024-04-22T08:12:52.009486+0000","last_undegraded":"2024-04-22T08:12:52.009486+0000","last_fullsized":"2024-04-22T08:12:52.009486+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:08:04.625870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:12:52.041135+0000","last_change":"2024-04-22T08:12:50.836566+0000","last_active":"2024-04-22T08:12:52.041135+0000","last_peered":"2024-04-22T08:12:52.041135+0000","last_clean":"2024-04-22T08:12:52.041135+0000","last_became_active":"2024-04-22T08:12:50.836250+0000","last_became_peered":"2024-04-22T08:12:50.836250+0000","last_unstale":"2024-04-22T08:12:52.041135+0000","last_undegraded":"2024-04-22T08:12:52.041135+0000","last_fullsized":"2024-04-22T08:12:52.041135+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:42:13.693209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:12:52.042149+0000","last_change":"2024-04-22T08:12:50.836508+0000","last_active":"2024-04-22T08:12:52.042149+0000","last_peered":"2024-04-22T08:12:52.042149+0000","last_clean":"2024-04-22T08:12:52.042149+0000","last_became_active":"2024-04-22T08:12:50.836210+0000","last_became_peered":"2024-04-22T08:12:50.836210+0000","last_unstale":"2024-04-22T08:12:52.042149+0000","last_undegraded":"2024-04-22T08:12:52.042149+0000","last_fullsized":"2024-04-22T08:12:52.042149+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:24:53.345661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:12:52.005162+0000","last_change":"2024-04-22T08:12:50.842633+0000","last_active":"2024-04-22T08:12:52.005162+0000","last_peered":"2024-04-22T08:12:52.005162+0000","last_clean":"2024-04-22T08:12:52.005162+0000","last_became_active":"2024-04-22T08:12:50.842392+0000","last_became_peered":"2024-04-22T08:12:50.842392+0000","last_unstale":"2024-04-22T08:12:52.005162+0000","last_undegraded":"2024-04-22T08:12:52.005162+0000","last_fullsized":"2024-04-22T08:12:52.005162+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:53:14.717539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:12:52.041139+0000","last_change":"2024-04-22T08:12:50.835820+0000","last_active":"2024-04-22T08:12:52.041139+0000","last_peered":"2024-04-22T08:12:52.041139+0000","last_clean":"2024-04-22T08:12:52.041139+0000","last_became_active":"2024-04-22T08:12:50.835541+0000","last_became_peered":"2024-04-22T08:12:50.835541+0000","last_unstale":"2024-04-22T08:12:52.041139+0000","last_undegraded":"2024-04-22T08:12:52.041139+0000","last_fullsized":"2024-04-22T08:12:52.041139+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:52:21.798509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":160,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:12:53.045764+0000","last_change":"2024-04-22T08:12:51.404305+0000","last_active":"2024-04-22T08:12:53.045764+0000","last_peered":"2024-04-22T08:12:53.045764+0000","last_clean":"2024-04-22T08:12:53.045764+0000","last_became_active":"2024-04-22T08:12:51.403208+0000","last_became_peered":"2024-04-22T08:12:51.403208+0000","last_unstale":"2024-04-22T08:12:53.045764+0000","last_undegraded":"2024-04-22T08:12:53.045764+0000","last_fullsized":"2024-04-22T08:12:53.045764+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:04:43.253802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640262,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27152,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":131072,"data_stored":37965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.248}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]}]},{"osd":0,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28112,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343728,"statfs":{"total":96636764160,"available":96607977472,"internally_reserved":0,"allocated":663552,"data_stored":566893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]}]},{"osd":5,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27152,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":131072,"data_stored":37965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.528}]}]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27152,"kb_used_data":124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":126976,"data_stored":37004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30399999999999999}]}]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27924,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343916,"statfs":{"total":96636764160,"available":96608169984,"internally_reserved":0,"allocated":602112,"data_stored":505453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.877}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98399999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27152,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":131072,"data_stored":37965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2170000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.093}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:12:55.838 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:12:55.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.937+0000 7f5d3151f640 10 monclient: get_monmap_and_config 2024-04-22T08:12:55.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.937+0000 7f5d3151f640 10 monclient: build_initial_monmap 2024-04-22T08:12:55.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.937+0000 7f5d3151f640 10 monclient: monmap: 2024-04-22T08:12:55.938 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:55.938 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:55.938 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:55.938723+0000 2024-04-22T08:12:55.938 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:55.938723+0000 2024-04-22T08:12:55.938 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:55.938 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:55.938 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:12:55.938 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:12:55.938 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:12:55.938 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:12:55.938 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:12:55.939 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:55.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.938+0000 7f5d3151f640 1 Processor -- start 2024-04-22T08:12:55.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.938+0000 7f5d3151f640 1 -- start start 2024-04-22T08:12:55.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.938+0000 7f5d3151f640 10 monclient: init 2024-04-22T08:12:55.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.938+0000 7f5d3151f640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:55.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.938+0000 7f5d3151f640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-04-22T08:12:55.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.939+0000 7f5d3151f640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d2c0a1a80 0x7f5d2c0a1e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:55.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.939+0000 7f5d3151f640 10 monclient(hunting): picked mon.noname-b con 0x7f5d2c0a1a80 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:55.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.939+0000 7f5d3151f640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5d2c0a0140 0x7f5d2c0a23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:55.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.939+0000 7f5d3151f640 10 monclient(hunting): picked mon.noname-a con 0x7f5d2c0a0140 addr v2:172.21.15.40:3300/0 2024-04-22T08:12:55.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.939+0000 7f5d3151f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5d2c135c20 addr v1:172.21.15.40:6789/0 2024-04-22T08:12:55.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.939+0000 7f5d3151f640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d2c09d8d0 con 0x7f5d2c135c20 2024-04-22T08:12:55.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.939+0000 7f5d3151f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:55.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.939+0000 7f5d3151f640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d2c0a28e0 con 0x7f5d2c0a0140 2024-04-22T08:12:55.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.939+0000 7f5d3151f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:55.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.939+0000 7f5d3151f640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2c09f860 con 0x7f5d2c0a1a80 2024-04-22T08:12:55.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.939+0000 7f5d3151f640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:55.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.939+0000 7f5d3151f640 10 monclient(hunting): authenticate will time out at 849.932850s 2024-04-22T08:12:55.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.939+0000 7f5d2affd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d2c0a1a80 0x7f5d2c0a1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:55.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.939+0000 7f5d2a7fc640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5d2c0a0140 0x7f5d2c0a23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:55.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.939+0000 7f5d2affd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d2c0a1a80 0x7f5d2c0a1e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49888/0 (socket says 172.21.15.40:49888) 2024-04-22T08:12:55.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.939+0000 7f5d2affd640 1 -- 172.21.15.40:0/196538962 learned_addr learned my addr 172.21.15.40:0/196538962 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:55.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.939+0000 7f5d2affd640 10 monclient(hunting): get_auth_request con 0x7f5d2c0a1a80 auth_method 0 2024-04-22T08:12:55.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.939+0000 7f5d2affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:55.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.939+0000 7f5d2affd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:55.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.939+0000 7f5d2affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:55.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.939+0000 7f5d2a7fc640 10 monclient(hunting): get_auth_request con 0x7f5d2c0a0140 auth_method 0 2024-04-22T08:12:55.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.939+0000 7f5d2a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:55.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.939+0000 7f5d2a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:55.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.939+0000 7f5d2a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:55.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.939+0000 7f5d2affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:55.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.939+0000 7f5d2affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:55.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.939+0000 7f5d2affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:55.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.939+0000 7f5d2a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:55.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.939+0000 7f5d2a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:55.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.939+0000 7f5d2a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:55.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.939+0000 7f5d2a7fc640 1 -- 172.21.15.40:0/196538962 >> v2:172.21.15.40:3300/0 conn(0x7f5d2c0a0140 msgr2=0x7f5d2c0a23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:12:55.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.939+0000 7f5d2a7fc640 1 -- 172.21.15.40:0/196538962 >> v2:172.21.15.40:3300/0 conn(0x7f5d2c0a0140 msgr2=0x7f5d2c0a23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:12:55.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.939+0000 7f5d2a7fc640 1 --2- 172.21.15.40:0/196538962 >> v2:172.21.15.40:3300/0 conn(0x7f5d2c0a0140 0x7f5d2c0a23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:12:55.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.939+0000 7f5d2a7fc640 1 --2- 172.21.15.40:0/196538962 >> v2:172.21.15.40:3300/0 conn(0x7f5d2c0a0140 0x7f5d2c0a23a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:12:55.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.939+0000 7f5d2affd640 10 monclient(hunting): handle_auth_done global_id 4307 payload 274 2024-04-22T08:12:55.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.939+0000 7f5d2affd640 1 -- 172.21.15.40:0/196538962 >> v2:172.21.15.40:3300/0 conn(0x7f5d2c0a0140 msgr2=0x7f5d2c0a23a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:12:55.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.939+0000 7f5d2affd640 1 --2- 172.21.15.40:0/196538962 >> v2:172.21.15.40:3300/0 conn(0x7f5d2c0a0140 0x7f5d2c0a23a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:55.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.939+0000 7f5d2affd640 1 -- 172.21.15.40:0/196538962 >> v1:172.21.15.40:6789/0 conn(0x7f5d2c135c20 legacy=0x7f5d2c136070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:55.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.939+0000 7f5d2affd640 10 monclient: _finish_hunting 0 2024-04-22T08:12:55.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.939+0000 7f5d2affd640 1 monclient: found mon.noname-b 2024-04-22T08:12:55.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.939+0000 7f5d2affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:55.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.939+0000 7f5d2affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:12:55.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.939+0000 7f5d2affd640 1 -- 172.21.15.40:0/196538962 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d2c09efc0 con 0x7f5d2c0a1a80 2024-04-22T08:12:55.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.942+0000 7f5d2affd640 1 -- 172.21.15.40:0/196538962 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d2c0a1a80 msgr2=0x7f5d2c0a1e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:55.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.942+0000 7f5d2affd640 1 --2- 172.21.15.40:0/196538962 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d2c0a1a80 0x7f5d2c0a1e60 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5d24009870 tx=0x7f5d2402b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d089a504b763964 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:55.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.942+0000 7f5d29ffb640 1 -- 172.21.15.40:0/196538962 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d24031950 con 0x7f5d2c0a1a80 2024-04-22T08:12:55.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.942+0000 7f5d29ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:55.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.942+0000 7f5d29ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:12:55.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.942+0000 7f5d29ffb640 10 monclient: dump: 2024-04-22T08:12:55.951 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:55.951 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:55.951 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:55.951 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:55.951 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:55.951 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:55.951 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:55.951 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:55.951 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:55.951 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:55.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.942+0000 7f5d29ffb640 10 monclient: _finish_auth 0 2024-04-22T08:12:55.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.942+0000 7f5d29ffb640 10 monclient: _check_auth_tickets 2024-04-22T08:12:55.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.942+0000 7f5d29ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:55.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.942+0000 7f5d3151f640 5 monclient: authenticate success, global_id 4307 2024-04-22T08:12:55.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.942+0000 7f5d29ffb640 1 -- 172.21.15.40:0/196538962 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d24031ab0 con 0x7f5d2c0a1a80 2024-04-22T08:12:55.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.942+0000 7f5d3151f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:12:55.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.942+0000 7f5d29ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:55.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.942+0000 7f5d3151f640 10 monclient: get_monmap_and_config success 2024-04-22T08:12:55.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.942+0000 7f5d3151f640 4 set_mon_vals no callback set 2024-04-22T08:12:55.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.942+0000 7f5d29ffb640 1 -- 172.21.15.40:0/196538962 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d24031e40 con 0x7f5d2c0a1a80 2024-04-22T08:12:55.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.942+0000 7f5d29ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:55.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.942+0000 7f5d29ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:12:55.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.942+0000 7f5d29ffb640 10 monclient: dump: 2024-04-22T08:12:55.952 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:55.953 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:55.953 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:55.953 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:55.953 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:55.953 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:55.953 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:55.953 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:55.953 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:55.953 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:55.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.942+0000 7f5d3151f640 10 monclient: shutdown 2024-04-22T08:12:55.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.942+0000 7f5d3151f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:55.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.942+0000 7f5d3151f640 1 -- 172.21.15.40:0/196538962 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d2c0a1a80 msgr2=0x7f5d2c0a1e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:55.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.942+0000 7f5d3151f640 1 --2- 172.21.15.40:0/196538962 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d2c0a1a80 0x7f5d2c0a1e60 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5d24009870 tx=0x7f5d2402b7e0 comp rx=0 tx=0).stop 2024-04-22T08:12:55.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.943+0000 7f5d3151f640 1 -- 172.21.15.40:0/196538962 shutdown_connections 2024-04-22T08:12:55.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.943+0000 7f5d3151f640 1 --2- 172.21.15.40:0/196538962 >> v2:172.21.15.40:3300/0 conn(0x7f5d2c0a0140 0x7f5d2c0a23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:55.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.943+0000 7f5d3151f640 1 --2- 172.21.15.40:0/196538962 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d2c0a1a80 0x7f5d2c0a1e60 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:55.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.943+0000 7f5d3151f640 1 -- 172.21.15.40:0/196538962 >> 172.21.15.40:0/196538962 conn(0x7f5d2c0a66a0 msgr2=0x7f5d2c0a2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:55.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.943+0000 7f5d3151f640 1 -- 172.21.15.40:0/196538962 shutdown_connections 2024-04-22T08:12:55.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.943+0000 7f5d3151f640 1 -- 172.21.15.40:0/196538962 wait complete. 2024-04-22T08:12:55.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.943+0000 7f5d3151f640 10 monclient: build_initial_monmap 2024-04-22T08:12:55.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.943+0000 7f5d3151f640 10 monclient: monmap: 2024-04-22T08:12:55.954 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:55.954 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:12:55.954 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:12:55.954 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:12:55.954 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:55.954 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:55.954 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:12:55.955 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:12:55.955 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:12:55.955 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:55.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.944+0000 7f5d3151f640 1 Processor -- start 2024-04-22T08:12:55.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.944+0000 7f5d3151f640 1 -- start start 2024-04-22T08:12:55.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.944+0000 7f5d3151f640 10 monclient: init 2024-04-22T08:12:55.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.944+0000 7f5d3151f640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:55.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.944+0000 7f5d3151f640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-22T08:12:55.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.944+0000 7f5d3151f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d2c0a0140 0x7f5d2c1ca860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:55.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.944+0000 7f5d3151f640 10 monclient(hunting): picked mon.noname-b con 0x7f5d2c0a0140 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:55.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.944+0000 7f5d3151f640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d2c0a1a80 0x7f5d2c134c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:55.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.944+0000 7f5d3151f640 10 monclient(hunting): picked mon.noname-c con 0x7f5d2c0a1a80 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:55.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.944+0000 7f5d3151f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d2c135c20 0x7f5d2c135140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:55.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.944+0000 7f5d3151f640 10 monclient(hunting): picked mon.noname-a con 0x7f5d2c135c20 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:55.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.944+0000 7f5d3151f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:55.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.944+0000 7f5d3151f640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2c1cada0 con 0x7f5d2c135c20 2024-04-22T08:12:55.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.944+0000 7f5d3151f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:55.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.944+0000 7f5d3151f640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2c09cbd0 con 0x7f5d2c0a0140 2024-04-22T08:12:55.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.944+0000 7f5d3151f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:55.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.944+0000 7f5d3151f640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2c071340 con 0x7f5d2c0a1a80 2024-04-22T08:12:55.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.944+0000 7f5d3151f640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:55.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.944+0000 7f5d3151f640 10 monclient(hunting): authenticate will time out at 849.938378s 2024-04-22T08:12:55.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.944+0000 7f5d2a7fc640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d2c0a1a80 0x7f5d2c134c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:55.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.944+0000 7f5d2b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d2c135c20 0x7f5d2c135140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:55.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.944+0000 7f5d2a7fc640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d2c0a1a80 0x7f5d2c134c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49900/0 (socket says 172.21.15.40:49900) 2024-04-22T08:12:55.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.944+0000 7f5d2a7fc640 1 -- 172.21.15.40:0/4140049508 learned_addr learned my addr 172.21.15.40:0/4140049508 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:55.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.944+0000 7f5d2a7fc640 10 monclient(hunting): get_auth_request con 0x7f5d2c0a1a80 auth_method 0 2024-04-22T08:12:55.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.944+0000 7f5d2a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:55.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.944+0000 7f5d2a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:55.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.944+0000 7f5d2a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:55.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.944+0000 7f5d2b7fe640 10 monclient(hunting): get_auth_request con 0x7f5d2c135c20 auth_method 0 2024-04-22T08:12:55.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.944+0000 7f5d2b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:55.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.944+0000 7f5d2b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:55.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.944+0000 7f5d2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:55.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.944+0000 7f5d2a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:55.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.944+0000 7f5d2a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:55.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.944+0000 7f5d2a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:55.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.944+0000 7f5d2b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:55.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.944+0000 7f5d2b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:55.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.944+0000 7f5d2b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:55.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.944+0000 7f5d2affd640 1 --2- 172.21.15.40:0/4140049508 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d2c0a0140 0x7f5d2c1ca860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:55.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.944+0000 7f5d2affd640 10 monclient(hunting): get_auth_request con 0x7f5d2c0a0140 auth_method 0 2024-04-22T08:12:55.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.944+0000 7f5d2affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:55.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.944+0000 7f5d2affd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:55.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.944+0000 7f5d2affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:55.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.944+0000 7f5d2a7fc640 10 monclient(hunting): handle_auth_done global_id 4310 payload 824 2024-04-22T08:12:55.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.945+0000 7f5d2a7fc640 1 -- 172.21.15.40:0/4140049508 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d2c0a0140 msgr2=0x7f5d2c1ca860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:55.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.945+0000 7f5d2a7fc640 1 --2- 172.21.15.40:0/4140049508 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d2c0a0140 0x7f5d2c1ca860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:55.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.945+0000 7f5d2a7fc640 1 -- 172.21.15.40:0/4140049508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d2c135c20 msgr2=0x7f5d2c135140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:55.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.945+0000 7f5d2a7fc640 1 --2- 172.21.15.40:0/4140049508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d2c135c20 0x7f5d2c135140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:55.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.945+0000 7f5d2a7fc640 10 monclient: _finish_hunting 0 2024-04-22T08:12:55.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.945+0000 7f5d2a7fc640 1 monclient: found mon.noname-c 2024-04-22T08:12:55.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.945+0000 7f5d2a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:55.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.945+0000 7f5d2a7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:12:55.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.945+0000 7f5d2a7fc640 1 -- 172.21.15.40:0/4140049508 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d240094d0 con 0x7f5d2c0a1a80 2024-04-22T08:12:55.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.945+0000 7f5d2affd640 1 --2- 172.21.15.40:0/4140049508 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d2c0a0140 0x7f5d2c1ca860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T08:12:55.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.947+0000 7f5d2a7fc640 1 -- 172.21.15.40:0/4140049508 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d2c0a1a80 msgr2=0x7f5d2c134c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:55.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.947+0000 7f5d2a7fc640 1 --2- 172.21.15.40:0/4140049508 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d2c0a1a80 0x7f5d2c134c00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5d2000e820 tx=0x7f5d2000ece0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:55.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.947+0000 7f5d137fe640 1 -- 172.21.15.40:0/4140049508 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d2000cb50 con 0x7f5d2c0a1a80 2024-04-22T08:12:55.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.947+0000 7f5d137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:55.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.947+0000 7f5d137fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:12:55.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.947+0000 7f5d137fe640 10 monclient: dump: 2024-04-22T08:12:55.959 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:55.959 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:55.959 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:55.959 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:55.959 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:55.959 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:55.959 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:55.959 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:55.959 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:55.959 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:55.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.947+0000 7f5d137fe640 10 monclient: _finish_auth 0 2024-04-22T08:12:55.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.947+0000 7f5d137fe640 10 monclient: _check_auth_tickets 2024-04-22T08:12:55.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.947+0000 7f5d137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:55.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.947+0000 7f5d3151f640 5 monclient: authenticate success, global_id 4310 2024-04-22T08:12:55.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.947+0000 7f5d137fe640 1 -- 172.21.15.40:0/4140049508 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d2000ccd0 con 0x7f5d2c0a1a80 2024-04-22T08:12:55.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.947+0000 7f5d137fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:55.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.947+0000 7f5d137fe640 1 -- 172.21.15.40:0/4140049508 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d200136b0 con 0x7f5d2c0a1a80 2024-04-22T08:12:55.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.947+0000 7f5d137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:55.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.947+0000 7f5d137fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:12:55.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.947+0000 7f5d137fe640 10 monclient: dump: 2024-04-22T08:12:55.960 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:55.960 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:55.960 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:55.960 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:55.960 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:55.961 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:55.961 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:55.961 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:55.961 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:55.961 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:55.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.947+0000 7f5d3151f640 10 monclient: _renew_subs 2024-04-22T08:12:55.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.947+0000 7f5d297fa640 4 set_mon_vals no callback set 2024-04-22T08:12:55.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.947+0000 7f5d3151f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:55.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.947+0000 7f5d3151f640 1 -- 172.21.15.40:0/4140049508 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d2c121d20 con 0x7f5d2c0a1a80 2024-04-22T08:12:55.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.947+0000 7f5d3151f640 10 monclient: _renew_subs 2024-04-22T08:12:55.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.947+0000 7f5d3151f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:55.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.947+0000 7f5d3151f640 1 -- 172.21.15.40:0/4140049508 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d2c1222e0 con 0x7f5d2c0a1a80 2024-04-22T08:12:55.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.948+0000 7f5d137fe640 1 -- 172.21.15.40:0/4140049508 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f5d20013890 con 0x7f5d2c0a1a80 2024-04-22T08:12:55.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.948+0000 7f5d3151f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:55.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.948+0000 7f5d3151f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:55.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.948+0000 7f5d3151f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:55.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.948+0000 7f5d3151f640 1 -- 172.21.15.40:0/4140049508 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d2c04e9b0 con 0x7f5d2c0a1a80 2024-04-22T08:12:55.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.948+0000 7f5d137fe640 1 --2- 172.21.15.40:0/4140049508 >> v2:172.21.15.189:6800/30168 conn(0x7f5d180377d0 0x7f5d18039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:55.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.948+0000 7f5d137fe640 1 -- 172.21.15.40:0/4140049508 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f5d2004a4d0 con 0x7f5d2c0a1a80 2024-04-22T08:12:55.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.948+0000 7f5d2affd640 1 --2- 172.21.15.40:0/4140049508 >> v2:172.21.15.189:6800/30168 conn(0x7f5d180377d0 0x7f5d18039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:55.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.948+0000 7f5d2affd640 10 monclient: get_auth_request con 0x7f5d180377d0 auth_method 0 2024-04-22T08:12:55.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.951+0000 7f5d2affd640 1 --2- 172.21.15.40:0/4140049508 >> v2:172.21.15.189:6800/30168 conn(0x7f5d180377d0 0x7f5d18039c90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5d24033000 tx=0x7f5d24008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:55.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.953+0000 7f5d137fe640 1 -- 172.21.15.40:0/4140049508 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f5d20013b40 con 0x7f5d2c0a1a80 2024-04-22T08:12:55.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.953+0000 7f5d137fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:55.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:55.953+0000 7f5d137fe640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:12:56.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.129+0000 7f5d3151f640 1 -- 172.21.15.40:0/4140049508 --> v2:172.21.15.189:6800/30168 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5d2c1bdde0 con 0x7f5d180377d0 2024-04-22T08:12:56.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.131+0000 7f5d137fe640 1 -- 172.21.15.40:0/4140049508 <== mgr.4102 v2:172.21.15.189:6800/30168 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+40579 (secure 0 0 0) 0x7f5d2c1bdde0 con 0x7f5d180377d0 2024-04-22T08:12:56.131 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:12:56.132 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-22T08:12:56.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.133+0000 7f5d3151f640 1 -- 172.21.15.40:0/4140049508 >> v2:172.21.15.189:6800/30168 conn(0x7f5d180377d0 msgr2=0x7f5d18039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:56.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.133+0000 7f5d3151f640 1 --2- 172.21.15.40:0/4140049508 >> v2:172.21.15.189:6800/30168 conn(0x7f5d180377d0 0x7f5d18039c90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5d24033000 tx=0x7f5d24008030 comp rx=0 tx=0).stop 2024-04-22T08:12:56.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.133+0000 7f5d3151f640 10 monclient: shutdown 2024-04-22T08:12:56.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.133+0000 7f5d3151f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:56.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.133+0000 7f5d3151f640 1 -- 172.21.15.40:0/4140049508 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d2c0a1a80 msgr2=0x7f5d2c134c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:56.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.133+0000 7f5d3151f640 1 --2- 172.21.15.40:0/4140049508 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d2c0a1a80 0x7f5d2c134c00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5d2000e820 tx=0x7f5d2000ece0 comp rx=0 tx=0).stop 2024-04-22T08:12:56.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.133+0000 7f5d3151f640 1 -- 172.21.15.40:0/4140049508 shutdown_connections 2024-04-22T08:12:56.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.133+0000 7f5d3151f640 1 --2- 172.21.15.40:0/4140049508 >> v2:172.21.15.189:6800/30168 conn(0x7f5d180377d0 0x7f5d18039c90 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:56.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.133+0000 7f5d3151f640 1 --2- 172.21.15.40:0/4140049508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d2c135c20 0x7f5d2c135140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:56.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.133+0000 7f5d3151f640 1 --2- 172.21.15.40:0/4140049508 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d2c0a1a80 0x7f5d2c134c00 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:56.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.133+0000 7f5d3151f640 1 --2- 172.21.15.40:0/4140049508 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d2c0a0140 0x7f5d2c1ca860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:56.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.133+0000 7f5d3151f640 1 -- 172.21.15.40:0/4140049508 >> 172.21.15.40:0/4140049508 conn(0x7f5d2c0a66a0 msgr2=0x7f5d2c09bdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:56.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.133+0000 7f5d3151f640 1 -- 172.21.15.40:0/4140049508 shutdown_connections 2024-04-22T08:12:56.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.134+0000 7f5d3151f640 1 -- 172.21.15.40:0/4140049508 wait complete. 2024-04-22T08:12:56.144 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2024-04-22T08:12:56.078017+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":8,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":164644,"kb_used_data":1744,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566066396,"statfs":{"total":579820584960,"available":579651989504,"internally_reserved":0,"allocated":1785856,"data_stored":1223245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9560,"internal_metadata":165927592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001014"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:12:52.005174+0000","last_change":"2024-04-22T08:12:51.443086+0000","last_active":"2024-04-22T08:12:52.005174+0000","last_peered":"2024-04-22T08:12:52.005174+0000","last_clean":"2024-04-22T08:12:52.005174+0000","last_became_active":"2024-04-22T08:12:51.442730+0000","last_became_peered":"2024-04-22T08:12:51.442730+0000","last_unstale":"2024-04-22T08:12:52.005174+0000","last_undegraded":"2024-04-22T08:12:52.005174+0000","last_fullsized":"2024-04-22T08:12:52.005174+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:06:13.481863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:12:52.009552+0000","last_change":"2024-04-22T08:12:50.841276+0000","last_active":"2024-04-22T08:12:52.009552+0000","last_peered":"2024-04-22T08:12:52.009552+0000","last_clean":"2024-04-22T08:12:52.009552+0000","last_became_active":"2024-04-22T08:12:50.841008+0000","last_became_peered":"2024-04-22T08:12:50.841008+0000","last_unstale":"2024-04-22T08:12:52.009552+0000","last_undegraded":"2024-04-22T08:12:52.009552+0000","last_fullsized":"2024-04-22T08:12:52.009552+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:47:00.224321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:12:52.005217+0000","last_change":"2024-04-22T08:12:51.403419+0000","last_active":"2024-04-22T08:12:52.005217+0000","last_peered":"2024-04-22T08:12:52.005217+0000","last_clean":"2024-04-22T08:12:52.005217+0000","last_became_active":"2024-04-22T08:12:51.403182+0000","last_became_peered":"2024-04-22T08:12:51.403182+0000","last_unstale":"2024-04-22T08:12:52.005217+0000","last_undegraded":"2024-04-22T08:12:52.005217+0000","last_fullsized":"2024-04-22T08:12:52.005217+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:52:39.730597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:12:52.009486+0000","last_change":"2024-04-22T08:12:51.403260+0000","last_active":"2024-04-22T08:12:52.009486+0000","last_peered":"2024-04-22T08:12:52.009486+0000","last_clean":"2024-04-22T08:12:52.009486+0000","last_became_active":"2024-04-22T08:12:51.402980+0000","last_became_peered":"2024-04-22T08:12:51.402980+0000","last_unstale":"2024-04-22T08:12:52.009486+0000","last_undegraded":"2024-04-22T08:12:52.009486+0000","last_fullsized":"2024-04-22T08:12:52.009486+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:08:04.625870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:12:52.041135+0000","last_change":"2024-04-22T08:12:50.836566+0000","last_active":"2024-04-22T08:12:52.041135+0000","last_peered":"2024-04-22T08:12:52.041135+0000","last_clean":"2024-04-22T08:12:52.041135+0000","last_became_active":"2024-04-22T08:12:50.836250+0000","last_became_peered":"2024-04-22T08:12:50.836250+0000","last_unstale":"2024-04-22T08:12:52.041135+0000","last_undegraded":"2024-04-22T08:12:52.041135+0000","last_fullsized":"2024-04-22T08:12:52.041135+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:42:13.693209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:12:52.042149+0000","last_change":"2024-04-22T08:12:50.836508+0000","last_active":"2024-04-22T08:12:52.042149+0000","last_peered":"2024-04-22T08:12:52.042149+0000","last_clean":"2024-04-22T08:12:52.042149+0000","last_became_active":"2024-04-22T08:12:50.836210+0000","last_became_peered":"2024-04-22T08:12:50.836210+0000","last_unstale":"2024-04-22T08:12:52.042149+0000","last_undegraded":"2024-04-22T08:12:52.042149+0000","last_fullsized":"2024-04-22T08:12:52.042149+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:24:53.345661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:12:52.005162+0000","last_change":"2024-04-22T08:12:50.842633+0000","last_active":"2024-04-22T08:12:52.005162+0000","last_peered":"2024-04-22T08:12:52.005162+0000","last_clean":"2024-04-22T08:12:52.005162+0000","last_became_active":"2024-04-22T08:12:50.842392+0000","last_became_peered":"2024-04-22T08:12:50.842392+0000","last_unstale":"2024-04-22T08:12:52.005162+0000","last_undegraded":"2024-04-22T08:12:52.005162+0000","last_fullsized":"2024-04-22T08:12:52.005162+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:53:14.717539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:12:52.041139+0000","last_change":"2024-04-22T08:12:50.835820+0000","last_active":"2024-04-22T08:12:52.041139+0000","last_peered":"2024-04-22T08:12:52.041139+0000","last_clean":"2024-04-22T08:12:52.041139+0000","last_became_active":"2024-04-22T08:12:50.835541+0000","last_became_peered":"2024-04-22T08:12:50.835541+0000","last_unstale":"2024-04-22T08:12:52.041139+0000","last_undegraded":"2024-04-22T08:12:52.041139+0000","last_fullsized":"2024-04-22T08:12:52.041139+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:52:21.798509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":160,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:12:53.045764+0000","last_change":"2024-04-22T08:12:51.404305+0000","last_active":"2024-04-22T08:12:53.045764+0000","last_peered":"2024-04-22T08:12:53.045764+0000","last_clean":"2024-04-22T08:12:53.045764+0000","last_became_active":"2024-04-22T08:12:51.403208+0000","last_became_peered":"2024-04-22T08:12:51.403208+0000","last_unstale":"2024-04-22T08:12:53.045764+0000","last_undegraded":"2024-04-22T08:12:53.045764+0000","last_fullsized":"2024-04-22T08:12:53.045764+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:04:43.253802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640262,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27152,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":131072,"data_stored":37965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.248}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]}]},{"osd":0,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28112,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343728,"statfs":{"total":96636764160,"available":96607977472,"internally_reserved":0,"allocated":663552,"data_stored":566893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]}]},{"osd":5,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27152,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":131072,"data_stored":37965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.528}]}]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27152,"kb_used_data":124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":126976,"data_stored":37004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30399999999999999}]}]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27924,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343916,"statfs":{"total":96636764160,"available":96608169984,"internally_reserved":0,"allocated":602112,"data_stored":505453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.877}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98399999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27152,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":131072,"data_stored":37965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2170000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.093}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:12:56.145 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-22T08:12:56.145 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-22T08:12:56.145 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-22T08:12:56.146 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-22T08:12:56.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.244+0000 7f55602b6640 10 monclient: get_monmap_and_config 2024-04-22T08:12:56.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.244+0000 7f55602b6640 10 monclient: build_initial_monmap 2024-04-22T08:12:56.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.244+0000 7f55602b6640 10 monclient: monmap: 2024-04-22T08:12:56.244 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:56.244 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:56.245 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:56.244797+0000 2024-04-22T08:12:56.245 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:56.244797+0000 2024-04-22T08:12:56.245 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:56.245 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:56.245 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:12:56.245 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:12:56.246 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:12:56.246 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:12:56.246 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:12:56.246 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:56.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.244+0000 7f55602b6640 1 Processor -- start 2024-04-22T08:12:56.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.244+0000 7f55602b6640 1 -- start start 2024-04-22T08:12:56.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.244+0000 7f55602b6640 10 monclient: init 2024-04-22T08:12:56.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f55602b6640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:56.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f55602b6640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-04-22T08:12:56.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f55602b6640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f555809ee10 0x7f55580a32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:56.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f55602b6640 10 monclient(hunting): picked mon.noname-b con 0x7f555809ee10 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:56.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f55602b6640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f55580a1d30 0x7f55580a2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:56.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f55602b6640 10 monclient(hunting): picked mon.noname-a con 0x7f55580a1d30 addr v2:172.21.15.40:3300/0 2024-04-22T08:12:56.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f55602b6640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5558135d90 0x7f55581361c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:56.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f55602b6640 10 monclient(hunting): picked mon.noname-c con 0x7f5558135d90 addr v2:172.21.15.189:3300/0 2024-04-22T08:12:56.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f55602b6640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:56.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f55602b6640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f555809f1f0 con 0x7f55580a1d30 2024-04-22T08:12:56.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f55602b6640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:56.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f55602b6640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55580a39c0 con 0x7f5558135d90 2024-04-22T08:12:56.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f55602b6640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:56.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f55602b6640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f555809fd30 con 0x7f555809ee10 2024-04-22T08:12:56.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f55602b6640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:56.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f55602b6640 10 monclient(hunting): authenticate will time out at 850.239028s 2024-04-22T08:12:56.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f555d851640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f55580a1d30 0x7f55580a2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:56.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f555e052640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f555809ee10 0x7f55580a32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:56.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f555d851640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f55580a1d30 0x7f55580a2150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58128/0 (socket says 172.21.15.40:58128) 2024-04-22T08:12:56.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f555e052640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f555809ee10 0x7f55580a32b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49914/0 (socket says 172.21.15.40:49914) 2024-04-22T08:12:56.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f555d851640 1 -- 172.21.15.40:0/3193191095 learned_addr learned my addr 172.21.15.40:0/3193191095 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:56.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f555d851640 10 monclient(hunting): get_auth_request con 0x7f55580a1d30 auth_method 0 2024-04-22T08:12:56.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f555d851640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:56.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f555d851640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:56.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f555d851640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:56.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f555e052640 10 monclient(hunting): get_auth_request con 0x7f555809ee10 auth_method 0 2024-04-22T08:12:56.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f555e052640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:56.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f555e052640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:56.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f555e052640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:56.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f555d851640 1 -- 172.21.15.40:0/3193191095 >> v2:172.21.15.40:3300/0 conn(0x7f55580a1d30 msgr2=0x7f55580a2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:12:56.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f555d851640 1 -- 172.21.15.40:0/3193191095 >> v2:172.21.15.40:3300/0 conn(0x7f55580a1d30 msgr2=0x7f55580a2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:12:56.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f555d851640 1 --2- 172.21.15.40:0/3193191095 >> v2:172.21.15.40:3300/0 conn(0x7f55580a1d30 0x7f55580a2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:12:56.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f555d851640 1 --2- 172.21.15.40:0/3193191095 >> v2:172.21.15.40:3300/0 conn(0x7f55580a1d30 0x7f55580a2150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:12:56.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f555e052640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:56.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f555e052640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:56.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f555e052640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:56.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f555e853640 1 --2- 172.21.15.40:0/3193191095 >> v2:172.21.15.189:3300/0 conn(0x7f5558135d90 0x7f55581361c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:56.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f555e853640 10 monclient(hunting): get_auth_request con 0x7f5558135d90 auth_method 0 2024-04-22T08:12:56.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f555e853640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:56.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f555e853640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:56.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f555e853640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:56.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f555e052640 10 monclient(hunting): handle_auth_done global_id 4313 payload 274 2024-04-22T08:12:56.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f555e052640 1 -- 172.21.15.40:0/3193191095 >> v2:172.21.15.189:3300/0 conn(0x7f5558135d90 msgr2=0x7f55581361c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:56.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f555e052640 1 --2- 172.21.15.40:0/3193191095 >> v2:172.21.15.189:3300/0 conn(0x7f5558135d90 0x7f55581361c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:56.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f555e052640 1 -- 172.21.15.40:0/3193191095 >> v2:172.21.15.40:3300/0 conn(0x7f55580a1d30 msgr2=0x7f55580a2150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:12:56.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f555e052640 1 --2- 172.21.15.40:0/3193191095 >> v2:172.21.15.40:3300/0 conn(0x7f55580a1d30 0x7f55580a2150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:56.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f555e052640 10 monclient: _finish_hunting 0 2024-04-22T08:12:56.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f555e052640 1 monclient: found mon.noname-b 2024-04-22T08:12:56.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f555e052640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:56.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f555e052640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:12:56.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.245+0000 7f555e052640 1 -- 172.21.15.40:0/3193191095 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55580a02f0 con 0x7f555809ee10 2024-04-22T08:12:56.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.248+0000 7f555e052640 1 -- 172.21.15.40:0/3193191095 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f555809ee10 msgr2=0x7f55580a32b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:56.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.248+0000 7f555e052640 1 --2- 172.21.15.40:0/3193191095 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f555809ee10 0x7f55580a32b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5558074690 tx=0x7f555002b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31f9460b00e696bd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:56.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.248+0000 7f555d050640 1 -- 172.21.15.40:0/3193191095 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5550030920 con 0x7f555809ee10 2024-04-22T08:12:56.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.248+0000 7f555d050640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:56.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.248+0000 7f555d050640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:12:56.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.248+0000 7f555d050640 10 monclient: dump: 2024-04-22T08:12:56.266 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:56.266 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:56.266 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:56.266 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:56.266 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:56.266 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:56.266 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:56.266 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:56.266 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:56.267 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:56.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.248+0000 7f555d050640 10 monclient: _finish_auth 0 2024-04-22T08:12:56.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.248+0000 7f555d050640 10 monclient: _check_auth_tickets 2024-04-22T08:12:56.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.248+0000 7f555d050640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:56.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.248+0000 7f55602b6640 5 monclient: authenticate success, global_id 4313 2024-04-22T08:12:56.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.248+0000 7f555d050640 1 -- 172.21.15.40:0/3193191095 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f555002e050 con 0x7f555809ee10 2024-04-22T08:12:56.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.248+0000 7f55602b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:12:56.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.248+0000 7f555d050640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:56.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.248+0000 7f55602b6640 10 monclient: get_monmap_and_config success 2024-04-22T08:12:56.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.248+0000 7f555d050640 1 -- 172.21.15.40:0/3193191095 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5550030ca0 con 0x7f555809ee10 2024-04-22T08:12:56.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.248+0000 7f555d050640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:56.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.248+0000 7f55602b6640 4 set_mon_vals no callback set 2024-04-22T08:12:56.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.248+0000 7f555d050640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:12:56.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.248+0000 7f555d050640 10 monclient: dump: 2024-04-22T08:12:56.268 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:56.268 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:56.268 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:56.268 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:56.269 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:56.269 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:56.269 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:56.269 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:56.269 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:56.269 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:56.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.249+0000 7f55602b6640 10 monclient: shutdown 2024-04-22T08:12:56.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.249+0000 7f55602b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:56.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.249+0000 7f55602b6640 1 -- 172.21.15.40:0/3193191095 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f555809ee10 msgr2=0x7f55580a32b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:56.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.249+0000 7f55602b6640 1 --2- 172.21.15.40:0/3193191095 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f555809ee10 0x7f55580a32b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5558074690 tx=0x7f555002b710 comp rx=0 tx=0).stop 2024-04-22T08:12:56.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.249+0000 7f55602b6640 1 -- 172.21.15.40:0/3193191095 shutdown_connections 2024-04-22T08:12:56.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.249+0000 7f55602b6640 1 --2- 172.21.15.40:0/3193191095 >> v2:172.21.15.189:3300/0 conn(0x7f5558135d90 0x7f55581361c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:56.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.249+0000 7f55602b6640 1 --2- 172.21.15.40:0/3193191095 >> v2:172.21.15.40:3300/0 conn(0x7f55580a1d30 0x7f55580a2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:56.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.249+0000 7f55602b6640 1 --2- 172.21.15.40:0/3193191095 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f555809ee10 0x7f55580a32b0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:56.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.249+0000 7f55602b6640 1 -- 172.21.15.40:0/3193191095 >> 172.21.15.40:0/3193191095 conn(0x7f55580a4ef0 msgr2=0x7f55580a5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:56.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.249+0000 7f55602b6640 1 -- 172.21.15.40:0/3193191095 shutdown_connections 2024-04-22T08:12:56.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.249+0000 7f55602b6640 1 -- 172.21.15.40:0/3193191095 wait complete. 2024-04-22T08:12:56.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.249+0000 7f55602b6640 10 monclient: build_initial_monmap 2024-04-22T08:12:56.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.249+0000 7f55602b6640 10 monclient: monmap: 2024-04-22T08:12:56.271 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:56.272 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:12:56.272 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:12:56.272 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:12:56.272 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:56.272 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:56.272 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:12:56.272 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:12:56.272 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:12:56.272 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:56.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.250+0000 7f55602b6640 1 Processor -- start 2024-04-22T08:12:56.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.250+0000 7f55602b6640 1 -- start start 2024-04-22T08:12:56.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.250+0000 7f55602b6640 10 monclient: init 2024-04-22T08:12:56.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.250+0000 7f55602b6640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:56.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.250+0000 7f55602b6640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-22T08:12:56.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.250+0000 7f55602b6640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f555809ee10 0x7f55581cad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:56.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.250+0000 7f55602b6640 10 monclient(hunting): picked mon.noname-c con 0x7f555809ee10 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:56.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.250+0000 7f55602b6640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55580a1d30 0x7f55581cb260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:56.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.250+0000 7f55602b6640 10 monclient(hunting): picked mon.noname-b con 0x7f55580a1d30 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:56.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.250+0000 7f55602b6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5558135d90 0x7f555811f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:56.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.250+0000 7f55602b6640 10 monclient(hunting): picked mon.noname-a con 0x7f5558135d90 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:56.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.250+0000 7f55602b6640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:56.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.250+0000 7f55602b6640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55581211b0 con 0x7f5558135d90 2024-04-22T08:12:56.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.250+0000 7f55602b6640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:56.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.250+0000 7f55602b6640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5558122d70 con 0x7f55580a1d30 2024-04-22T08:12:56.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.250+0000 7f55602b6640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:56.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.250+0000 7f55602b6640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5558122fc0 con 0x7f555809ee10 2024-04-22T08:12:56.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.250+0000 7f55602b6640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:56.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.250+0000 7f55602b6640 10 monclient(hunting): authenticate will time out at 850.244404s 2024-04-22T08:12:56.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.250+0000 7f555e052640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f555809ee10 0x7f55581cad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:56.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.250+0000 7f555e052640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f555809ee10 0x7f55581cad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49930/0 (socket says 172.21.15.40:49930) 2024-04-22T08:12:56.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.250+0000 7f555e052640 1 -- 172.21.15.40:0/503536374 learned_addr learned my addr 172.21.15.40:0/503536374 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:56.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.250+0000 7f555e052640 10 monclient(hunting): get_auth_request con 0x7f555809ee10 auth_method 0 2024-04-22T08:12:56.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.250+0000 7f555e052640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:56.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.250+0000 7f555e052640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:56.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.250+0000 7f555e052640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:56.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.250+0000 7f555e853640 1 --2- 172.21.15.40:0/503536374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5558135d90 0x7f555811f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:56.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.250+0000 7f555e853640 10 monclient(hunting): get_auth_request con 0x7f5558135d90 auth_method 0 2024-04-22T08:12:56.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.250+0000 7f555e853640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:56.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.250+0000 7f555e853640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:56.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.250+0000 7f555e853640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:56.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.250+0000 7f555e052640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:56.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.250+0000 7f555e052640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:56.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.250+0000 7f555e052640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:56.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.250+0000 7f555e853640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:56.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.250+0000 7f555e853640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:56.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.250+0000 7f555e853640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:56.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.250+0000 7f555e052640 10 monclient(hunting): handle_auth_done global_id 4316 payload 824 2024-04-22T08:12:56.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.250+0000 7f555e052640 1 -- 172.21.15.40:0/503536374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55580a1d30 msgr2=0x7f55581cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:56.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.250+0000 7f555d851640 1 --2- 172.21.15.40:0/503536374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55580a1d30 0x7f55581cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:56.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.251+0000 7f555e052640 1 --2- 172.21.15.40:0/503536374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55580a1d30 0x7f55581cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:56.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.251+0000 7f555e052640 1 -- 172.21.15.40:0/503536374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5558135d90 msgr2=0x7f555811f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:56.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.251+0000 7f555e052640 1 --2- 172.21.15.40:0/503536374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5558135d90 0x7f555811f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:56.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.251+0000 7f555e052640 10 monclient: _finish_hunting 0 2024-04-22T08:12:56.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.251+0000 7f555e052640 1 monclient: found mon.noname-c 2024-04-22T08:12:56.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.251+0000 7f555e052640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:56.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.251+0000 7f555e052640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:12:56.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.251+0000 7f555e052640 1 -- 172.21.15.40:0/503536374 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5550009400 con 0x7f555809ee10 2024-04-22T08:12:56.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.251+0000 7f555d851640 10 monclient: get_auth_request con 0x7f55580a1d30 auth_method 0 2024-04-22T08:12:56.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.251+0000 7f555d851640 1 --2- 172.21.15.40:0/503536374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55580a1d30 0x7f55581cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T08:12:56.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.253+0000 7f555e052640 1 -- 172.21.15.40:0/503536374 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f555809ee10 msgr2=0x7f55581cad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:56.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.253+0000 7f555e052640 1 --2- 172.21.15.40:0/503536374 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f555809ee10 0x7f55581cad20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f555002bd20 tx=0x7f55500046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:56.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.253+0000 7f5556ffd640 1 -- 172.21.15.40:0/503536374 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555003bae0 con 0x7f555809ee10 2024-04-22T08:12:56.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.253+0000 7f5556ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:56.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.253+0000 7f5556ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:12:56.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.253+0000 7f5556ffd640 10 monclient: dump: 2024-04-22T08:12:56.280 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:56.281 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:56.281 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:56.281 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:56.281 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:56.281 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:56.281 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:56.281 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:56.281 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:56.282 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:56.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.253+0000 7f5556ffd640 10 monclient: _finish_auth 0 2024-04-22T08:12:56.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.253+0000 7f5556ffd640 10 monclient: _check_auth_tickets 2024-04-22T08:12:56.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.253+0000 7f5556ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:56.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.253+0000 7f5556ffd640 1 -- 172.21.15.40:0/503536374 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f555003bc40 con 0x7f555809ee10 2024-04-22T08:12:56.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.253+0000 7f55602b6640 5 monclient: authenticate success, global_id 4316 2024-04-22T08:12:56.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.253+0000 7f55602b6640 10 monclient: _renew_subs 2024-04-22T08:12:56.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.253+0000 7f55602b6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:56.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.253+0000 7f55602b6640 1 -- 172.21.15.40:0/503536374 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55581217c0 con 0x7f555809ee10 2024-04-22T08:12:56.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.253+0000 7f5556ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:56.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.253+0000 7f5556ffd640 1 -- 172.21.15.40:0/503536374 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5550032400 con 0x7f555809ee10 2024-04-22T08:12:56.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.253+0000 7f5556ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:56.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.253+0000 7f5556ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:12:56.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.253+0000 7f5556ffd640 10 monclient: dump: 2024-04-22T08:12:56.285 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:56.285 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:56.285 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:56.285 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:56.286 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:56.286 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:56.286 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:56.286 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:56.286 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:56.286 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:56.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.253+0000 7f55602b6640 10 monclient: _renew_subs 2024-04-22T08:12:56.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.253+0000 7f555c84f640 4 set_mon_vals no callback set 2024-04-22T08:12:56.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.253+0000 7f55602b6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:56.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.253+0000 7f55602b6640 1 -- 172.21.15.40:0/503536374 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5558120bf0 con 0x7f555809ee10 2024-04-22T08:12:56.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.254+0000 7f5556ffd640 1 -- 172.21.15.40:0/503536374 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f5550039040 con 0x7f555809ee10 2024-04-22T08:12:56.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.254+0000 7f55602b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:56.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.254+0000 7f55602b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:56.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.254+0000 7f55602b6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:56.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.254+0000 7f5556ffd640 1 --2- 172.21.15.40:0/503536374 >> v2:172.21.15.189:6800/30168 conn(0x7f5544037820 0x7f5544039ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:56.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.254+0000 7f55602b6640 1 -- 172.21.15.40:0/503536374 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5558122b20 con 0x7f555809ee10 2024-04-22T08:12:56.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.254+0000 7f5556ffd640 1 -- 172.21.15.40:0/503536374 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f555006bb00 con 0x7f555809ee10 2024-04-22T08:12:56.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.254+0000 7f555d851640 1 --2- 172.21.15.40:0/503536374 >> v2:172.21.15.189:6800/30168 conn(0x7f5544037820 0x7f5544039ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:56.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.254+0000 7f555d851640 10 monclient: get_auth_request con 0x7f5544037820 auth_method 0 2024-04-22T08:12:56.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.257+0000 7f555d851640 1 --2- 172.21.15.40:0/503536374 >> v2:172.21.15.189:6800/30168 conn(0x7f5544037820 0x7f5544039ce0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5558132b50 tx=0x7f554c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:56.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.259+0000 7f5556ffd640 1 -- 172.21.15.40:0/503536374 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f5550037070 con 0x7f555809ee10 2024-04-22T08:12:56.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.259+0000 7f5556ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:56.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.259+0000 7f5556ffd640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:12:56.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.475+0000 7f55602b6640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-04-22T08:12:56.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.475+0000 7f55602b6640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-04-22T08:12:56.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.475+0000 7f55602b6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:56.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.475+0000 7f55602b6640 1 -- 172.21.15.40:0/503536374 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5558123100 con 0x7f555809ee10 2024-04-22T08:12:56.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.475+0000 7f5556ffd640 1 -- 172.21.15.40:0/503536374 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f5558123100 con 0x7f555809ee10 2024-04-22T08:12:56.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.475+0000 7f5556ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-04-22T08:12:56.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.475+0000 7f5556ffd640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:12:56.476 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:12:56.476 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-22T08:12:56.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.477+0000 7f55602b6640 1 -- 172.21.15.40:0/503536374 >> v2:172.21.15.189:6800/30168 conn(0x7f5544037820 msgr2=0x7f5544039ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:56.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.477+0000 7f55602b6640 1 --2- 172.21.15.40:0/503536374 >> v2:172.21.15.189:6800/30168 conn(0x7f5544037820 0x7f5544039ce0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5558132b50 tx=0x7f554c008040 comp rx=0 tx=0).stop 2024-04-22T08:12:56.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.477+0000 7f55602b6640 10 monclient: shutdown 2024-04-22T08:12:56.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.477+0000 7f55602b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:56.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.477+0000 7f55602b6640 1 -- 172.21.15.40:0/503536374 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f555809ee10 msgr2=0x7f55581cad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:56.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.477+0000 7f55602b6640 1 --2- 172.21.15.40:0/503536374 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f555809ee10 0x7f55581cad20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f555002bd20 tx=0x7f55500046c0 comp rx=0 tx=0).stop 2024-04-22T08:12:56.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.477+0000 7f55602b6640 1 -- 172.21.15.40:0/503536374 shutdown_connections 2024-04-22T08:12:56.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.477+0000 7f55602b6640 1 --2- 172.21.15.40:0/503536374 >> v2:172.21.15.189:6800/30168 conn(0x7f5544037820 0x7f5544039ce0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:56.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.477+0000 7f55602b6640 1 --2- 172.21.15.40:0/503536374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55580a1d30 0x7f55581cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:56.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.477+0000 7f55602b6640 1 --2- 172.21.15.40:0/503536374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5558135d90 0x7f555811f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:56.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.477+0000 7f55602b6640 1 --2- 172.21.15.40:0/503536374 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f555809ee10 0x7f55581cad20 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:56.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.477+0000 7f55602b6640 1 -- 172.21.15.40:0/503536374 >> 172.21.15.40:0/503536374 conn(0x7f55580a4ef0 msgr2=0x7f55580a61f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:56.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.477+0000 7f55602b6640 1 -- 172.21.15.40:0/503536374 shutdown_connections 2024-04-22T08:12:56.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.477+0000 7f55602b6640 1 -- 172.21.15.40:0/503536374 wait complete. 2024-04-22T08:12:56.487 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-22T08:12:56.488 INFO:teuthology.run_tasks:Running task mon_thrash... 2024-04-22T08:12:56.514 INFO:tasks.mon_thrash:Beginning mon_thrash... 2024-04-22T08:12:56.514 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T08:12:56.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.614+0000 7f4992ce9640 10 monclient: get_monmap_and_config 2024-04-22T08:12:56.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.614+0000 7f4992ce9640 10 monclient: build_initial_monmap 2024-04-22T08:12:56.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.614+0000 7f4992ce9640 10 monclient: monmap: 2024-04-22T08:12:56.614 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:56.614 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:56.614 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:56.615183+0000 2024-04-22T08:12:56.614 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:56.615183+0000 2024-04-22T08:12:56.614 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:56.614 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:56.614 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:12:56.615 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:12:56.615 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:12:56.615 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:12:56.615 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:12:56.615 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:56.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.615+0000 7f4992ce9640 1 Processor -- start 2024-04-22T08:12:56.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.615+0000 7f4992ce9640 1 -- start start 2024-04-22T08:12:56.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.615+0000 7f4992ce9640 10 monclient: init 2024-04-22T08:12:56.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.615+0000 7f4992ce9640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:56.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.615+0000 7f4992ce9640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-04-22T08:12:56.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.615+0000 7f4992ce9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f498c09c150 addr v1:172.21.15.40:6789/0 2024-04-22T08:12:56.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.615+0000 7f4992ce9640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f498c0999f0 0x7f498c13ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:56.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.615+0000 7f4992ce9640 10 monclient(hunting): picked mon.noname-b con 0x7f498c0999f0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:56.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.615+0000 7f4992ce9640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f498c13b3a0 0x7f498c13d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:56.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.615+0000 7f4992ce9640 10 monclient(hunting): picked mon.noname-c con 0x7f498c13b3a0 addr v2:172.21.15.189:3300/0 2024-04-22T08:12:56.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.615+0000 7f4992ce9640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f498c09dca0 con 0x7f498c09c150 2024-04-22T08:12:56.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.615+0000 7f4992ce9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:56.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.615+0000 7f4992ce9640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f498c072270 con 0x7f498c13b3a0 2024-04-22T08:12:56.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.615+0000 7f4992ce9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:56.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.615+0000 7f4992ce9640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f498c09f580 con 0x7f498c0999f0 2024-04-22T08:12:56.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.615+0000 7f4992ce9640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:56.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.615+0000 7f4992ce9640 10 monclient(hunting): authenticate will time out at 850.609388s 2024-04-22T08:12:56.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.615+0000 7f498bfff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f498c0999f0 0x7f498c13ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:56.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.615+0000 7f4990a85640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f498c09c150 0x7f498c09c530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:38986/0 (socket says 172.21.15.40:38986) 2024-04-22T08:12:56.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.615+0000 7f4990a85640 1 -- 172.21.15.40:0/1007829730 learned_addr learned my addr 172.21.15.40:0/1007829730 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:12:56.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.615+0000 7f498bfff640 10 monclient(hunting): get_auth_request con 0x7f498c0999f0 auth_method 0 2024-04-22T08:12:56.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.615+0000 7f498bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:56.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.615+0000 7f498bfff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:56.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.615+0000 7f498bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:56.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.615+0000 7f498bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:56.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.615+0000 7f498bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:56.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.615+0000 7f498bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:56.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.615+0000 7f4991286640 1 --2- 172.21.15.40:0/1007829730 >> v2:172.21.15.189:3300/0 conn(0x7f498c13b3a0 0x7f498c13d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:56.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.616+0000 7f498bfff640 10 monclient(hunting): handle_auth_done global_id 4319 payload 274 2024-04-22T08:12:56.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.616+0000 7f498bfff640 1 -- 172.21.15.40:0/1007829730 >> v2:172.21.15.189:3300/0 conn(0x7f498c13b3a0 msgr2=0x7f498c13d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:56.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.616+0000 7f498bfff640 1 --2- 172.21.15.40:0/1007829730 >> v2:172.21.15.189:3300/0 conn(0x7f498c13b3a0 0x7f498c13d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:56.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.616+0000 7f498bfff640 1 -- 172.21.15.40:0/1007829730 >> v1:172.21.15.40:6789/0 conn(0x7f498c09c150 legacy=0x7f498c09c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:56.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.616+0000 7f498bfff640 10 monclient: _finish_hunting 0 2024-04-22T08:12:56.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.616+0000 7f498bfff640 1 monclient: found mon.noname-b 2024-04-22T08:12:56.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.616+0000 7f498bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:56.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.616+0000 7f498bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:12:56.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.616+0000 7f498bfff640 1 -- 172.21.15.40:0/1007829730 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f498c09efc0 con 0x7f498c0999f0 2024-04-22T08:12:56.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.616+0000 7f4991286640 10 monclient: get_auth_request con 0x7f498c13b3a0 auth_method 0 2024-04-22T08:12:56.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.616+0000 7f4991286640 1 --2- 172.21.15.40:0/1007829730 >> v2:172.21.15.189:3300/0 conn(0x7f498c13b3a0 0x7f498c13d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T08:12:56.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.618+0000 7f498bfff640 1 -- 172.21.15.40:0/1007829730 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f498c0999f0 msgr2=0x7f498c13ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:56.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.618+0000 7f498bfff640 1 --2- 172.21.15.40:0/1007829730 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f498c0999f0 0x7f498c13ae60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f49780097a0 tx=0x7f497802b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41902f763bf863f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:56.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.619+0000 7f498b7fe640 1 -- 172.21.15.40:0/1007829730 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f497802f050 con 0x7f498c0999f0 2024-04-22T08:12:56.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.619+0000 7f498b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:56.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.619+0000 7f498b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:12:56.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.619+0000 7f498b7fe640 10 monclient: dump: 2024-04-22T08:12:56.628 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:56.628 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:56.628 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:56.628 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:56.628 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:56.628 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:56.628 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:56.628 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:56.628 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:56.628 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:56.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.619+0000 7f498b7fe640 10 monclient: _finish_auth 0 2024-04-22T08:12:56.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.619+0000 7f498b7fe640 10 monclient: _check_auth_tickets 2024-04-22T08:12:56.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.619+0000 7f498b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:56.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.619+0000 7f498b7fe640 1 -- 172.21.15.40:0/1007829730 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49780319f0 con 0x7f498c0999f0 2024-04-22T08:12:56.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.619+0000 7f4992ce9640 5 monclient: authenticate success, global_id 4319 2024-04-22T08:12:56.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.619+0000 7f4992ce9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:12:56.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.619+0000 7f498b7fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:56.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.619+0000 7f4992ce9640 10 monclient: get_monmap_and_config success 2024-04-22T08:12:56.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.619+0000 7f4992ce9640 4 set_mon_vals no callback set 2024-04-22T08:12:56.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.619+0000 7f498b7fe640 1 -- 172.21.15.40:0/1007829730 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4978031d80 con 0x7f498c0999f0 2024-04-22T08:12:56.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.619+0000 7f498b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:56.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.619+0000 7f498b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:12:56.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.619+0000 7f498b7fe640 10 monclient: dump: 2024-04-22T08:12:56.630 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:56.630 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:56.630 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:56.630 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:56.630 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:56.630 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:56.630 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:56.630 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:56.630 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:56.630 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:56.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.619+0000 7f4992ce9640 10 monclient: shutdown 2024-04-22T08:12:56.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.619+0000 7f4992ce9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:56.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.619+0000 7f4992ce9640 1 -- 172.21.15.40:0/1007829730 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f498c0999f0 msgr2=0x7f498c13ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:56.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.619+0000 7f4992ce9640 1 --2- 172.21.15.40:0/1007829730 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f498c0999f0 0x7f498c13ae60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f49780097a0 tx=0x7f497802b7a0 comp rx=0 tx=0).stop 2024-04-22T08:12:56.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.619+0000 7f4992ce9640 1 -- 172.21.15.40:0/1007829730 shutdown_connections 2024-04-22T08:12:56.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.619+0000 7f4992ce9640 1 --2- 172.21.15.40:0/1007829730 >> v2:172.21.15.189:3300/0 conn(0x7f498c13b3a0 0x7f498c13d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:56.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.619+0000 7f4992ce9640 1 --2- 172.21.15.40:0/1007829730 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f498c0999f0 0x7f498c13ae60 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:56.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.619+0000 7f4992ce9640 1 -- 172.21.15.40:0/1007829730 >> 172.21.15.40:0/1007829730 conn(0x7f498c0a66a0 msgr2=0x7f498c0a2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:56.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.619+0000 7f4992ce9640 1 -- 172.21.15.40:0/1007829730 shutdown_connections 2024-04-22T08:12:56.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.619+0000 7f4992ce9640 1 -- 172.21.15.40:0/1007829730 wait complete. 2024-04-22T08:12:56.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.620+0000 7f4992ce9640 10 monclient: build_initial_monmap 2024-04-22T08:12:56.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.620+0000 7f4992ce9640 10 monclient: monmap: 2024-04-22T08:12:56.631 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:56.631 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:12:56.631 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:12:56.631 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:12:56.631 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:56.631 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:56.632 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:12:56.632 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:12:56.632 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:12:56.632 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:56.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.620+0000 7f4992ce9640 1 Processor -- start 2024-04-22T08:12:56.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.620+0000 7f4992ce9640 1 -- start start 2024-04-22T08:12:56.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.620+0000 7f4992ce9640 10 monclient: init 2024-04-22T08:12:56.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f4992ce9640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:56.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f4992ce9640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-22T08:12:56.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f4992ce9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f498c0999f0 0x7f498c0711f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:56.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f4992ce9640 10 monclient(hunting): picked mon.noname-b con 0x7f498c0999f0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:56.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f4992ce9640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f498c09c150 0x7f498c137430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:56.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f4992ce9640 10 monclient(hunting): picked mon.noname-c con 0x7f498c09c150 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:56.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f4992ce9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f498c13b3a0 0x7f498c137970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:56.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f4992ce9640 10 monclient(hunting): picked mon.noname-a con 0x7f498c13b3a0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:56.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f4992ce9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:56.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f4992ce9640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f498c09cce0 con 0x7f498c13b3a0 2024-04-22T08:12:56.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f4992ce9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:56.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f4992ce9640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f498c138de0 con 0x7f498c0999f0 2024-04-22T08:12:56.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f4992ce9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:56.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f4992ce9640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f498c123b00 con 0x7f498c09c150 2024-04-22T08:12:56.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f4992ce9640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:56.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f4992ce9640 10 monclient(hunting): authenticate will time out at 850.615136s 2024-04-22T08:12:56.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f498bfff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f498c09c150 0x7f498c137430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:56.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f4991286640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f498c13b3a0 0x7f498c137970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:56.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f498bfff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f498c09c150 0x7f498c137430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49946/0 (socket says 172.21.15.40:49946) 2024-04-22T08:12:56.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f498bfff640 1 -- 172.21.15.40:0/3409274998 learned_addr learned my addr 172.21.15.40:0/3409274998 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:56.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f498bfff640 10 monclient(hunting): get_auth_request con 0x7f498c09c150 auth_method 0 2024-04-22T08:12:56.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f498bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:56.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f498bfff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:56.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f498bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:56.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f4991286640 10 monclient(hunting): get_auth_request con 0x7f498c13b3a0 auth_method 0 2024-04-22T08:12:56.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f4991286640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:56.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f4991286640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:56.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f4991286640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:56.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f498bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:56.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f498bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:56.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f498bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:56.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f4991286640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:56.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f4991286640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:56.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f4991286640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:56.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f4990a85640 1 --2- 172.21.15.40:0/3409274998 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f498c0999f0 0x7f498c0711f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:56.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f4990a85640 10 monclient(hunting): get_auth_request con 0x7f498c0999f0 auth_method 0 2024-04-22T08:12:56.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f4990a85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:56.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f4990a85640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:56.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f4990a85640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:56.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f4990a85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:56.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f4990a85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:56.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f4990a85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:56.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f4991286640 10 monclient(hunting): handle_auth_done global_id 4338 payload 824 2024-04-22T08:12:56.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f4991286640 1 -- 172.21.15.40:0/3409274998 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f498c09c150 msgr2=0x7f498c137430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:56.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f4991286640 1 --2- 172.21.15.40:0/3409274998 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f498c09c150 0x7f498c137430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:56.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f4991286640 1 -- 172.21.15.40:0/3409274998 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f498c0999f0 msgr2=0x7f498c0711f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:56.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f4991286640 1 --2- 172.21.15.40:0/3409274998 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f498c0999f0 0x7f498c0711f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:56.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f4991286640 10 monclient: _finish_hunting 0 2024-04-22T08:12:56.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f4991286640 1 monclient: found mon.noname-a 2024-04-22T08:12:56.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f4991286640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:56.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f4991286640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:12:56.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f4991286640 1 -- 172.21.15.40:0/3409274998 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4978009400 con 0x7f498c13b3a0 2024-04-22T08:12:56.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.621+0000 7f498bfff640 1 --2- 172.21.15.40:0/3409274998 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f498c09c150 0x7f498c137430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:12:56.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.624+0000 7f4991286640 1 -- 172.21.15.40:0/3409274998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f498c13b3a0 msgr2=0x7f498c137970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:56.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.624+0000 7f4991286640 1 --2- 172.21.15.40:0/3409274998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f498c13b3a0 0x7f498c137970 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f498400cdf0 tx=0x7f498400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:56.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.624+0000 7f49897fa640 1 -- 172.21.15.40:0/3409274998 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498400d840 con 0x7f498c13b3a0 2024-04-22T08:12:56.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.624+0000 7f49897fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:56.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.624+0000 7f49897fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:12:56.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.624+0000 7f49897fa640 10 monclient: dump: 2024-04-22T08:12:56.636 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:56.636 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:56.636 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:56.636 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:56.636 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:56.637 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:56.637 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:56.637 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:56.637 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:56.637 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:56.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.624+0000 7f49897fa640 10 monclient: _finish_auth 0 2024-04-22T08:12:56.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.624+0000 7f49897fa640 10 monclient: _check_auth_tickets 2024-04-22T08:12:56.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.624+0000 7f49897fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:56.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.624+0000 7f49897fa640 1 -- 172.21.15.40:0/3409274998 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f498400d9e0 con 0x7f498c13b3a0 2024-04-22T08:12:56.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.624+0000 7f4992ce9640 5 monclient: authenticate success, global_id 4338 2024-04-22T08:12:56.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.624+0000 7f4992ce9640 10 monclient: _renew_subs 2024-04-22T08:12:56.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.624+0000 7f4992ce9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:56.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.624+0000 7f4992ce9640 1 -- 172.21.15.40:0/3409274998 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f498c123540 con 0x7f498c13b3a0 2024-04-22T08:12:56.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.624+0000 7f49897fa640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:56.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.624+0000 7f49897fa640 1 -- 172.21.15.40:0/3409274998 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498400dd90 con 0x7f498c13b3a0 2024-04-22T08:12:56.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.624+0000 7f49897fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:56.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.624+0000 7f49897fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:12:56.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.624+0000 7f49897fa640 10 monclient: dump: 2024-04-22T08:12:56.638 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:56.638 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:56.638 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:56.638 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:56.638 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:56.638 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:56.638 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:56.638 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:56.638 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:56.638 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:56.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.624+0000 7f4992ce9640 10 monclient: _renew_subs 2024-04-22T08:12:56.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.624+0000 7f4992ce9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:56.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.624+0000 7f498a7fc640 4 set_mon_vals no callback set 2024-04-22T08:12:56.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.624+0000 7f4992ce9640 1 -- 172.21.15.40:0/3409274998 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f498c120fb0 con 0x7f498c13b3a0 2024-04-22T08:12:56.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.625+0000 7f49897fa640 1 -- 172.21.15.40:0/3409274998 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f4984015440 con 0x7f498c13b3a0 2024-04-22T08:12:56.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.625+0000 7f4992ce9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:56.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.625+0000 7f4992ce9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:56.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.625+0000 7f4992ce9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:56.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.625+0000 7f4992ce9640 1 -- 172.21.15.40:0/3409274998 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f498c0a0370 con 0x7f498c13b3a0 2024-04-22T08:12:56.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.625+0000 7f49897fa640 1 --2- 172.21.15.40:0/3409274998 >> v2:172.21.15.189:6800/30168 conn(0x7f49700377d0 0x7f4970039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:56.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.625+0000 7f49897fa640 1 -- 172.21.15.40:0/3409274998 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f498404ab80 con 0x7f498c13b3a0 2024-04-22T08:12:56.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.625+0000 7f4990a85640 1 --2- 172.21.15.40:0/3409274998 >> v2:172.21.15.189:6800/30168 conn(0x7f49700377d0 0x7f4970039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:56.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.625+0000 7f4990a85640 10 monclient: get_auth_request con 0x7f49700377d0 auth_method 0 2024-04-22T08:12:56.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.628+0000 7f4990a85640 1 --2- 172.21.15.40:0/3409274998 >> v2:172.21.15.189:6800/30168 conn(0x7f49700377d0 0x7f4970039c90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4980007c90 tx=0x7f4980007be0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:56.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.629+0000 7f49897fa640 1 -- 172.21.15.40:0/3409274998 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f498c0a0370 con 0x7f498c13b3a0 2024-04-22T08:12:56.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.629+0000 7f49897fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:56.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.629+0000 7f49897fa640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:12:56.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.812+0000 7f4992ce9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-22T08:12:56.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.812+0000 7f4992ce9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-22T08:12:56.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.812+0000 7f4992ce9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:56.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.812+0000 7f4992ce9640 1 -- 172.21.15.40:0/3409274998 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f498c0a0370 con 0x7f498c13b3a0 2024-04-22T08:12:56.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.813+0000 7f49897fa640 1 -- 172.21.15.40:0/3409274998 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+10498 (secure 0 0 0) 0x7f498c0a0370 con 0x7f498c13b3a0 2024-04-22T08:12:56.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.813+0000 7f49897fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-22T08:12:56.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.813+0000 7f49897fa640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:12:56.813 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:12:56.813 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":15,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","created":"2024-04-22T08:12:36.003043+0000","modified":"2024-04-22T08:12:51.828738+0000","last_up_change":"2024-04-22T08:12:47.016043+0000","last_in_change":"2024-04-22T08:12:40.364359+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T08:12:48.855046+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-22T08:12:48.986472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b9a0210c-5881-4ca0-8318-a4a987109405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":30669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6805","nonce":30669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6807","nonce":30669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":30669}]},"public_addr":"172.21.15.40:6804/30669","cluster_addr":"172.21.15.40:6805/30669","heartbeat_back_addr":"172.21.15.40:6807/30669","heartbeat_front_addr":"172.21.15.40:6806/30669","state":["exists","up"]},{"osd":1,"uuid":"11370cc8-09a6-4c35-b92c-4d926c6c18d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":30670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6801","nonce":30670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6803","nonce":30670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":30670}]},"public_addr":"172.21.15.40:6800/30670","cluster_addr":"172.21.15.40:6801/30670","heartbeat_back_addr":"172.21.15.40:6803/30670","heartbeat_front_addr":"172.21.15.40:6802/30670","state":["exists","up"]},{"osd":2,"uuid":"d03db34e-f630-4a3e-86fb-7234bee0d0f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":30671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":30671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6811","nonce":30671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":30671}]},"public_addr":"172.21.15.40:6808/30671","cluster_addr":"172.21.15.40:6809/30671","heartbeat_back_addr":"172.21.15.40:6811/30671","heartbeat_front_addr":"172.21.15.40:6810/30671","state":["exists","up"]},{"osd":3,"uuid":"d94b2052-5847-4955-b1d4-b7e627572811","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6805","nonce":30673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6807","nonce":30673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6811","nonce":30673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6809","nonce":30673}]},"public_addr":"172.21.15.189:6805/30673","cluster_addr":"172.21.15.189:6807/30673","heartbeat_back_addr":"172.21.15.189:6811/30673","heartbeat_front_addr":"172.21.15.189:6809/30673","state":["exists","up"]},{"osd":4,"uuid":"cb626202-7411-4299-8d20-988bcc9103f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6801","nonce":30672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":30672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":30672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6803","nonce":30672}]},"public_addr":"172.21.15.189:6801/30672","cluster_addr":"172.21.15.189:6802/30672","heartbeat_back_addr":"172.21.15.189:6804/30672","heartbeat_front_addr":"172.21.15.189:6803/30672","state":["exists","up"]},{"osd":5,"uuid":"abbac246-54d6-47d7-a41c-a68df0b80ddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":30674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":30674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":30674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":30674}]},"public_addr":"172.21.15.189:6806/30674","cluster_addr":"172.21.15.189:6808/30674","heartbeat_back_addr":"172.21.15.189:6812/30674","heartbeat_front_addr":"172.21.15.189:6810/30674","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T08:12:45.488082+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T08:12:45.467994+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T08:12:45.503580+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T08:12:45.488169+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T08:12:45.465787+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T08:12:45.470729+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T08:12:56.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.815+0000 7f4992ce9640 1 -- 172.21.15.40:0/3409274998 >> v2:172.21.15.189:6800/30168 conn(0x7f49700377d0 msgr2=0x7f4970039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:56.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.815+0000 7f4992ce9640 1 --2- 172.21.15.40:0/3409274998 >> v2:172.21.15.189:6800/30168 conn(0x7f49700377d0 0x7f4970039c90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4980007c90 tx=0x7f4980007be0 comp rx=0 tx=0).stop 2024-04-22T08:12:56.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.815+0000 7f4992ce9640 10 monclient: shutdown 2024-04-22T08:12:56.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.815+0000 7f4992ce9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:56.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.815+0000 7f4992ce9640 1 -- 172.21.15.40:0/3409274998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f498c13b3a0 msgr2=0x7f498c137970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:56.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.815+0000 7f4992ce9640 1 --2- 172.21.15.40:0/3409274998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f498c13b3a0 0x7f498c137970 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f498400cdf0 tx=0x7f498400a710 comp rx=0 tx=0).stop 2024-04-22T08:12:56.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.815+0000 7f4992ce9640 1 -- 172.21.15.40:0/3409274998 shutdown_connections 2024-04-22T08:12:56.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.815+0000 7f4992ce9640 1 --2- 172.21.15.40:0/3409274998 >> v2:172.21.15.189:6800/30168 conn(0x7f49700377d0 0x7f4970039c90 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:56.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.815+0000 7f4992ce9640 1 --2- 172.21.15.40:0/3409274998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f498c13b3a0 0x7f498c137970 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:56.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.815+0000 7f4992ce9640 1 --2- 172.21.15.40:0/3409274998 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f498c09c150 0x7f498c137430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:56.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.815+0000 7f4992ce9640 1 --2- 172.21.15.40:0/3409274998 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f498c0999f0 0x7f498c0711f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:56.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.815+0000 7f4992ce9640 1 -- 172.21.15.40:0/3409274998 >> 172.21.15.40:0/3409274998 conn(0x7f498c0a66a0 msgr2=0x7f498c09a800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:56.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.815+0000 7f4992ce9640 1 -- 172.21.15.40:0/3409274998 shutdown_connections 2024-04-22T08:12:56.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.815+0000 7f4992ce9640 1 -- 172.21.15.40:0/3409274998 wait complete. 2024-04-22T08:12:56.825 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T08:12:48.855046+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-04-22T08:12:48.986472+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T08:12:56.826 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T08:12:56.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.928+0000 7f03f10ce640 10 monclient: get_monmap_and_config 2024-04-22T08:12:56.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.928+0000 7f03f10ce640 10 monclient: build_initial_monmap 2024-04-22T08:12:56.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.928+0000 7f03f10ce640 10 monclient: monmap: 2024-04-22T08:12:56.928 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:56.928 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:56.929 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:56.929690+0000 2024-04-22T08:12:56.929 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:56.929690+0000 2024-04-22T08:12:56.929 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:56.929 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:56.929 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:12:56.929 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:12:56.929 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:12:56.929 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:12:56.929 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:12:56.929 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:56.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.929+0000 7f03f10ce640 1 Processor -- start 2024-04-22T08:12:56.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.929+0000 7f03f10ce640 1 -- start start 2024-04-22T08:12:56.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.929+0000 7f03f10ce640 10 monclient: init 2024-04-22T08:12:56.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.930+0000 7f03f10ce640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:56.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.930+0000 7f03f10ce640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-04-22T08:12:56.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.930+0000 7f03f10ce640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f03ec135b40 addr v1:172.21.15.40:6789/0 2024-04-22T08:12:56.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.930+0000 7f03f10ce640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f03ec0a3d90 0x7f03ec13b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:56.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.930+0000 7f03f10ce640 10 monclient(hunting): picked mon.noname-c con 0x7f03ec0a3d90 addr v2:172.21.15.189:3300/0 2024-04-22T08:12:56.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.930+0000 7f03f10ce640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f03ec09eec0 addr v1:172.21.15.189:6789/0 2024-04-22T08:12:56.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.930+0000 7f03f10ce640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03ec09f9b0 con 0x7f03ec135b40 2024-04-22T08:12:56.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.930+0000 7f03f10ce640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03ec0a5c10 con 0x7f03ec09eec0 2024-04-22T08:12:56.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.930+0000 7f03f10ce640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:56.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.930+0000 7f03f10ce640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03ec09fd90 con 0x7f03ec0a3d90 2024-04-22T08:12:56.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.930+0000 7f03f10ce640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:56.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.930+0000 7f03f10ce640 10 monclient(hunting): authenticate will time out at 850.923913s 2024-04-22T08:12:56.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.930+0000 7f03ead9d640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f03ec135b40 0x7f03ec135f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:38990/0 (socket says 172.21.15.40:38990) 2024-04-22T08:12:56.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.930+0000 7f03ead9d640 1 -- 172.21.15.40:0/2361939804 learned_addr learned my addr 172.21.15.40:0/2361939804 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:12:56.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.930+0000 7f03e3fff640 1 --2- 172.21.15.40:0/2361939804 >> v2:172.21.15.189:3300/0 conn(0x7f03ec0a3d90 0x7f03ec13b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:56.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.930+0000 7f03e3fff640 10 monclient(hunting): get_auth_request con 0x7f03ec0a3d90 auth_method 0 2024-04-22T08:12:56.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.930+0000 7f03e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:56.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.930+0000 7f03e3fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:56.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.930+0000 7f03e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:56.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.930+0000 7f03e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:56.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.930+0000 7f03e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:56.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.930+0000 7f03e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:56.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.930+0000 7f03e3fff640 10 monclient(hunting): handle_auth_done global_id 4300 payload 274 2024-04-22T08:12:56.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.930+0000 7f03e3fff640 1 -- 172.21.15.40:0/2361939804 >> v1:172.21.15.189:6789/0 conn(0x7f03ec09eec0 legacy=0x7f03ec13d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:56.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.930+0000 7f03e3fff640 1 -- 172.21.15.40:0/2361939804 >> v1:172.21.15.40:6789/0 conn(0x7f03ec135b40 legacy=0x7f03ec135f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:56.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.930+0000 7f03e3fff640 10 monclient: _finish_hunting 0 2024-04-22T08:12:56.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.930+0000 7f03e3fff640 1 monclient: found mon.noname-c 2024-04-22T08:12:56.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.930+0000 7f03e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:56.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.930+0000 7f03e3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.189:3300/0 2024-04-22T08:12:56.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.930+0000 7f03e3fff640 1 -- 172.21.15.40:0/2361939804 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03ec0a5da0 con 0x7f03ec0a3d90 2024-04-22T08:12:56.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.934+0000 7f03e3fff640 1 -- 172.21.15.40:0/2361939804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03ec0a3d90 msgr2=0x7f03ec13b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:56.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.934+0000 7f03e3fff640 1 --2- 172.21.15.40:0/2361939804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03ec0a3d90 0x7f03ec13b000 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f03ec0725d0 tx=0x7f03d402b6e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d6a544f09e17db8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:56.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.934+0000 7f03ea59c640 1 -- 172.21.15.40:0/2361939804 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d4030a00 con 0x7f03ec0a3d90 2024-04-22T08:12:56.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.934+0000 7f03ea59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:56.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.934+0000 7f03ea59c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:12:56.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.934+0000 7f03ea59c640 10 monclient: dump: 2024-04-22T08:12:56.938 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:56.938 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:56.938 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:56.938 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:56.938 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:56.938 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:56.939 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:56.939 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:56.939 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:56.939 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:56.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.934+0000 7f03ea59c640 10 monclient: _finish_auth 0 2024-04-22T08:12:56.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.934+0000 7f03ea59c640 10 monclient: _check_auth_tickets 2024-04-22T08:12:56.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.934+0000 7f03ea59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:56.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.934+0000 7f03f10ce640 5 monclient: authenticate success, global_id 4300 2024-04-22T08:12:56.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.934+0000 7f03f10ce640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:12:56.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.934+0000 7f03ea59c640 1 -- 172.21.15.40:0/2361939804 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03d402e050 con 0x7f03ec0a3d90 2024-04-22T08:12:56.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.934+0000 7f03ea59c640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:56.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.934+0000 7f03ea59c640 1 -- 172.21.15.40:0/2361939804 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d4030d80 con 0x7f03ec0a3d90 2024-04-22T08:12:56.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.934+0000 7f03ea59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:56.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.934+0000 7f03ea59c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:12:56.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.934+0000 7f03ea59c640 10 monclient: dump: 2024-04-22T08:12:56.940 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:56.940 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:56.940 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:56.940 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:56.940 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:56.940 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:56.940 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:56.940 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:56.940 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:56.940 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:56.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.934+0000 7f03f10ce640 10 monclient: get_monmap_and_config success 2024-04-22T08:12:56.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.934+0000 7f03f10ce640 4 set_mon_vals no callback set 2024-04-22T08:12:56.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.934+0000 7f03f10ce640 10 monclient: shutdown 2024-04-22T08:12:56.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.934+0000 7f03f10ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:56.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.934+0000 7f03f10ce640 1 -- 172.21.15.40:0/2361939804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03ec0a3d90 msgr2=0x7f03ec13b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:56.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.934+0000 7f03f10ce640 1 --2- 172.21.15.40:0/2361939804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03ec0a3d90 0x7f03ec13b000 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f03ec0725d0 tx=0x7f03d402b6e0 comp rx=0 tx=0).stop 2024-04-22T08:12:56.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.934+0000 7f03f10ce640 1 -- 172.21.15.40:0/2361939804 shutdown_connections 2024-04-22T08:12:56.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.934+0000 7f03f10ce640 1 --2- 172.21.15.40:0/2361939804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03ec0a3d90 0x7f03ec13b000 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:56.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.934+0000 7f03f10ce640 1 -- 172.21.15.40:0/2361939804 >> 172.21.15.40:0/2361939804 conn(0x7f03ec0a4e80 msgr2=0x7f03ec0a5290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:56.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.934+0000 7f03f10ce640 1 -- 172.21.15.40:0/2361939804 shutdown_connections 2024-04-22T08:12:56.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.934+0000 7f03f10ce640 1 -- 172.21.15.40:0/2361939804 wait complete. 2024-04-22T08:12:56.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.935+0000 7f03f10ce640 10 monclient: build_initial_monmap 2024-04-22T08:12:56.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.935+0000 7f03f10ce640 10 monclient: monmap: 2024-04-22T08:12:56.942 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:56.942 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:12:56.942 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:12:56.942 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:12:56.942 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:56.942 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:56.942 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:12:56.942 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:12:56.942 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:12:56.942 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:56.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.935+0000 7f03f10ce640 1 Processor -- start 2024-04-22T08:12:56.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.935+0000 7f03f10ce640 1 -- start start 2024-04-22T08:12:56.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.935+0000 7f03f10ce640 10 monclient: init 2024-04-22T08:12:56.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03f10ce640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:56.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03f10ce640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-22T08:12:56.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03f10ce640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03ec09eec0 0x7f03ec06ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:56.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03f10ce640 10 monclient(hunting): picked mon.noname-a con 0x7f03ec09eec0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:56.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03f10ce640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03ec0a3d90 0x7f03ec06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:56.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03f10ce640 10 monclient(hunting): picked mon.noname-c con 0x7f03ec0a3d90 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:56.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03f10ce640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03ec135b40 0x7f03ec06d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:56.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03f10ce640 10 monclient(hunting): picked mon.noname-b con 0x7f03ec135b40 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:56.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03f10ce640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:56.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03f10ce640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03ec06e9f0 con 0x7f03ec09eec0 2024-04-22T08:12:56.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03f10ce640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:56.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03f10ce640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03ec0a4670 con 0x7f03ec135b40 2024-04-22T08:12:56.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03f10ce640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:56.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03f10ce640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03ec12ff30 con 0x7f03ec0a3d90 2024-04-22T08:12:56.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03f10ce640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:56.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03f10ce640 10 monclient(hunting): authenticate will time out at 850.930220s 2024-04-22T08:12:56.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03ead9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03ec09eec0 0x7f03ec06ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:56.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03e3fff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03ec0a3d90 0x7f03ec06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:56.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03ead9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03ec09eec0 0x7f03ec06ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58146/0 (socket says 172.21.15.40:58146) 2024-04-22T08:12:56.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03e3fff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03ec0a3d90 0x7f03ec06d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49950/0 (socket says 172.21.15.40:49950) 2024-04-22T08:12:56.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03ead9d640 1 -- 172.21.15.40:0/681732384 learned_addr learned my addr 172.21.15.40:0/681732384 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:56.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03ead9d640 10 monclient(hunting): get_auth_request con 0x7f03ec09eec0 auth_method 0 2024-04-22T08:12:56.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03ead9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:56.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03ead9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:56.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03ead9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:56.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03e3fff640 10 monclient(hunting): get_auth_request con 0x7f03ec0a3d90 auth_method 0 2024-04-22T08:12:56.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:56.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03e3fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:56.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:56.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03ead9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:56.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03ead9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:56.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03ead9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:56.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:56.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:56.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:56.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03eb59e640 1 --2- 172.21.15.40:0/681732384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03ec135b40 0x7f03ec06d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:56.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03eb59e640 10 monclient(hunting): get_auth_request con 0x7f03ec135b40 auth_method 0 2024-04-22T08:12:56.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03eb59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:56.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03eb59e640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:56.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03eb59e640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:56.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03eb59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:56.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03eb59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:56.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03eb59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:56.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03ead9d640 10 monclient(hunting): handle_auth_done global_id 4341 payload 824 2024-04-22T08:12:56.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03ead9d640 1 -- 172.21.15.40:0/681732384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03ec0a3d90 msgr2=0x7f03ec06d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:56.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03ead9d640 1 --2- 172.21.15.40:0/681732384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03ec0a3d90 0x7f03ec06d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:56.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03ead9d640 1 -- 172.21.15.40:0/681732384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03ec135b40 msgr2=0x7f03ec06d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:56.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03ead9d640 1 --2- 172.21.15.40:0/681732384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03ec135b40 0x7f03ec06d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:56.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03ead9d640 10 monclient: _finish_hunting 0 2024-04-22T08:12:56.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03ead9d640 1 monclient: found mon.noname-a 2024-04-22T08:12:56.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03ead9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:56.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03ead9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:12:56.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03ead9d640 1 -- 172.21.15.40:0/681732384 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d4009400 con 0x7f03ec09eec0 2024-04-22T08:12:56.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.936+0000 7f03e3fff640 1 --2- 172.21.15.40:0/681732384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03ec0a3d90 0x7f03ec06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:12:56.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.939+0000 7f03ead9d640 1 -- 172.21.15.40:0/681732384 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03ec09eec0 msgr2=0x7f03ec06ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:56.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.939+0000 7f03ead9d640 1 --2- 172.21.15.40:0/681732384 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03ec09eec0 0x7f03ec06ce50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f03dc00c460 tx=0x7f03dc00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:56.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.939+0000 7f03e37fe640 1 -- 172.21.15.40:0/681732384 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03dc007070 con 0x7f03ec09eec0 2024-04-22T08:12:56.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.939+0000 7f03e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:56.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.939+0000 7f03e37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:12:56.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.939+0000 7f03e37fe640 10 monclient: dump: 2024-04-22T08:12:56.949 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:56.949 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:56.949 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:56.949 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:56.949 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:56.949 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:56.949 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:56.949 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:56.950 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:56.950 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:56.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.939+0000 7f03e37fe640 10 monclient: _finish_auth 0 2024-04-22T08:12:56.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.939+0000 7f03e37fe640 10 monclient: _check_auth_tickets 2024-04-22T08:12:56.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.939+0000 7f03e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:56.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.939+0000 7f03e37fe640 1 -- 172.21.15.40:0/681732384 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03dc0071f0 con 0x7f03ec09eec0 2024-04-22T08:12:56.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.939+0000 7f03f10ce640 5 monclient: authenticate success, global_id 4341 2024-04-22T08:12:56.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.939+0000 7f03e37fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:56.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.939+0000 7f03e37fe640 1 -- 172.21.15.40:0/681732384 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03dc0075a0 con 0x7f03ec09eec0 2024-04-22T08:12:56.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.939+0000 7f03f10ce640 10 monclient: _renew_subs 2024-04-22T08:12:56.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.939+0000 7f03f10ce640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:56.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.939+0000 7f03f10ce640 1 -- 172.21.15.40:0/681732384 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03ec070820 con 0x7f03ec09eec0 2024-04-22T08:12:56.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.939+0000 7f03e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:56.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.939+0000 7f03e37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:12:56.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.939+0000 7f03e37fe640 10 monclient: dump: 2024-04-22T08:12:56.951 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:56.951 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:56.951 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:56.951 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:56.951 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:56.951 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:56.951 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:56.951 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:56.951 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:56.951 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:56.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.939+0000 7f03e9d9b640 4 set_mon_vals no callback set 2024-04-22T08:12:56.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.939+0000 7f03f10ce640 10 monclient: _renew_subs 2024-04-22T08:12:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.939+0000 7f03f10ce640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.939+0000 7f03f10ce640 1 -- 172.21.15.40:0/681732384 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03ec06e490 con 0x7f03ec09eec0 2024-04-22T08:12:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.940+0000 7f03e37fe640 1 -- 172.21.15.40:0/681732384 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f03dc01b400 con 0x7f03ec09eec0 2024-04-22T08:12:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.940+0000 7f03f10ce640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.940+0000 7f03f10ce640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.940+0000 7f03f10ce640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.940+0000 7f03f10ce640 1 -- 172.21.15.40:0/681732384 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03ec0704a0 con 0x7f03ec09eec0 2024-04-22T08:12:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.940+0000 7f03e37fe640 1 --2- 172.21.15.40:0/681732384 >> v2:172.21.15.189:6800/30168 conn(0x7f03cc0377d0 0x7f03cc039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.940+0000 7f03e37fe640 1 -- 172.21.15.40:0/681732384 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f03dc049e30 con 0x7f03ec09eec0 2024-04-22T08:12:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.940+0000 7f03e3fff640 1 --2- 172.21.15.40:0/681732384 >> v2:172.21.15.189:6800/30168 conn(0x7f03cc0377d0 0x7f03cc039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.940+0000 7f03e3fff640 10 monclient: get_auth_request con 0x7f03cc0377d0 auth_method 0 2024-04-22T08:12:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.943+0000 7f03e3fff640 1 --2- 172.21.15.40:0/681732384 >> v2:172.21.15.189:6800/30168 conn(0x7f03cc0377d0 0x7f03cc039c90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f03ec06ca20 tx=0x7f03d4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.945+0000 7f03e37fe640 1 -- 172.21.15.40:0/681732384 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f03dc01b6b0 con 0x7f03ec09eec0 2024-04-22T08:12:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.945+0000 7f03e37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:56.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:56.945+0000 7f03e37fe640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:12:57.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.122+0000 7f03f10ce640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-04-22T08:12:57.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.122+0000 7f03f10ce640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-04-22T08:12:57.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.122+0000 7f03f10ce640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:57.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.122+0000 7f03f10ce640 1 -- 172.21.15.40:0/681732384 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f03ec0702c0 con 0x7f03ec09eec0 2024-04-22T08:12:57.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.123+0000 7f03e37fe640 1 -- 172.21.15.40:0/681732384 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v15) v1 ==== 93+0+10 (secure 0 0 0) 0x7f03dc01a360 con 0x7f03ec09eec0 2024-04-22T08:12:57.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.123+0000 7f03e37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-04-22T08:12:57.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.123+0000 7f03e37fe640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:12:57.123 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-22T08:12:57.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.125+0000 7f03f10ce640 1 -- 172.21.15.40:0/681732384 >> v2:172.21.15.189:6800/30168 conn(0x7f03cc0377d0 msgr2=0x7f03cc039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:57.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.125+0000 7f03f10ce640 1 --2- 172.21.15.40:0/681732384 >> v2:172.21.15.189:6800/30168 conn(0x7f03cc0377d0 0x7f03cc039c90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f03ec06ca20 tx=0x7f03d4008030 comp rx=0 tx=0).stop 2024-04-22T08:12:57.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.125+0000 7f03f10ce640 10 monclient: shutdown 2024-04-22T08:12:57.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.125+0000 7f03f10ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:57.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.125+0000 7f03f10ce640 1 -- 172.21.15.40:0/681732384 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03ec09eec0 msgr2=0x7f03ec06ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:57.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.125+0000 7f03f10ce640 1 --2- 172.21.15.40:0/681732384 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03ec09eec0 0x7f03ec06ce50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f03dc00c460 tx=0x7f03dc00c920 comp rx=0 tx=0).stop 2024-04-22T08:12:57.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.125+0000 7f03f10ce640 1 -- 172.21.15.40:0/681732384 shutdown_connections 2024-04-22T08:12:57.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.125+0000 7f03f10ce640 1 --2- 172.21.15.40:0/681732384 >> v2:172.21.15.189:6800/30168 conn(0x7f03cc0377d0 0x7f03cc039c90 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:57.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.125+0000 7f03f10ce640 1 --2- 172.21.15.40:0/681732384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03ec135b40 0x7f03ec06d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:57.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.125+0000 7f03f10ce640 1 --2- 172.21.15.40:0/681732384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03ec0a3d90 0x7f03ec06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:57.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.125+0000 7f03f10ce640 1 --2- 172.21.15.40:0/681732384 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03ec09eec0 0x7f03ec06ce50 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:57.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.125+0000 7f03f10ce640 1 -- 172.21.15.40:0/681732384 >> 172.21.15.40:0/681732384 conn(0x7f03ec0a4e80 msgr2=0x7f03ec12ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:57.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.125+0000 7f03f10ce640 1 -- 172.21.15.40:0/681732384 shutdown_connections 2024-04-22T08:12:57.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.125+0000 7f03f10ce640 1 -- 172.21.15.40:0/681732384 wait complete. 2024-04-22T08:12:57.135 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-04-22T08:12:57.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.237+0000 7fe6431e4640 10 monclient: get_monmap_and_config 2024-04-22T08:12:57.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.237+0000 7fe6431e4640 10 monclient: build_initial_monmap 2024-04-22T08:12:57.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.237+0000 7fe6431e4640 10 monclient: monmap: 2024-04-22T08:12:57.237 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:57.237 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:57.237 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:57.238493+0000 2024-04-22T08:12:57.237 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:57.238493+0000 2024-04-22T08:12:57.237 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:57.238 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:57.238 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:12:57.238 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:12:57.238 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:12:57.238 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:12:57.238 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:12:57.238 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:57.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.238+0000 7fe6431e4640 1 Processor -- start 2024-04-22T08:12:57.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.238+0000 7fe6431e4640 1 -- start start 2024-04-22T08:12:57.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.238+0000 7fe6431e4640 10 monclient: init 2024-04-22T08:12:57.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.238+0000 7fe6431e4640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:57.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.238+0000 7fe6431e4640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-04-22T08:12:57.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.238+0000 7fe6431e4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe63c070d70 addr v1:172.21.15.189:6789/0 2024-04-22T08:12:57.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.238+0000 7fe6431e4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe63c135de0 addr v1:172.21.15.40:6789/0 2024-04-22T08:12:57.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.238+0000 7fe6431e4640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe63c136210 0x7fe63c13e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:57.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.238+0000 7fe6431e4640 10 monclient(hunting): picked mon.noname-a con 0x7fe63c136210 addr v2:172.21.15.40:3300/0 2024-04-22T08:12:57.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.238+0000 7fe6431e4640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe63c0a07a0 con 0x7fe63c135de0 2024-04-22T08:12:57.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.238+0000 7fe6431e4640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe63c06d310 con 0x7fe63c070d70 2024-04-22T08:12:57.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.238+0000 7fe6431e4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:57.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.238+0000 7fe6431e4640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe63c09fe30 con 0x7fe63c136210 2024-04-22T08:12:57.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.238+0000 7fe6431e4640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:57.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.238+0000 7fe6431e4640 10 monclient(hunting): authenticate will time out at 851.232742s 2024-04-22T08:12:57.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.239+0000 7fe641781640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe63c136210 0x7fe63c13e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:57.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.239+0000 7fe633fff640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fe63c135de0 0x7fe63c13ace0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:39000/0 (socket says 172.21.15.40:39000) 2024-04-22T08:12:57.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.239+0000 7fe641781640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe63c136210 0x7fe63c13e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58162/0 (socket says 172.21.15.40:58162) 2024-04-22T08:12:57.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.239+0000 7fe633fff640 1 -- 172.21.15.40:0/1807459837 learned_addr learned my addr 172.21.15.40:0/1807459837 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:12:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.239+0000 7fe641781640 10 monclient(hunting): get_auth_request con 0x7fe63c136210 auth_method 0 2024-04-22T08:12:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.239+0000 7fe641781640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.239+0000 7fe641781640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.239+0000 7fe641781640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.239+0000 7fe641781640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.239+0000 7fe641781640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.239+0000 7fe641781640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.239+0000 7fe641781640 10 monclient(hunting): handle_auth_done global_id 4344 payload 274 2024-04-22T08:12:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.239+0000 7fe641781640 1 -- 172.21.15.40:0/1807459837 >> v1:172.21.15.189:6789/0 conn(0x7fe63c070d70 legacy=0x7fe63c071150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.239+0000 7fe641781640 1 -- 172.21.15.40:0/1807459837 >> v1:172.21.15.40:6789/0 conn(0x7fe63c135de0 legacy=0x7fe63c13ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.239+0000 7fe641781640 10 monclient: _finish_hunting 0 2024-04-22T08:12:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.239+0000 7fe641781640 1 monclient: found mon.noname-a 2024-04-22T08:12:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.239+0000 7fe641781640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.239+0000 7fe641781640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:12:57.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.239+0000 7fe641781640 1 -- 172.21.15.40:0/1807459837 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe63c06d030 con 0x7fe63c136210 2024-04-22T08:12:57.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.242+0000 7fe641781640 1 -- 172.21.15.40:0/1807459837 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe63c136210 msgr2=0x7fe63c13e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.242+0000 7fe641781640 1 --2- 172.21.15.40:0/1807459837 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe63c136210 0x7fe63c13e410 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe634009d90 tx=0x7fe63402bd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24210699d7ed1d3a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.242+0000 7fe6337fe640 1 -- 172.21.15.40:0/1807459837 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe63402cda0 con 0x7fe63c136210 2024-04-22T08:12:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.242+0000 7fe6337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.242+0000 7fe6337fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:12:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.242+0000 7fe6337fe640 10 monclient: dump: 2024-04-22T08:12:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:57.250 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:57.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.242+0000 7fe6337fe640 10 monclient: _finish_auth 0 2024-04-22T08:12:57.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.242+0000 7fe6337fe640 10 monclient: _check_auth_tickets 2024-04-22T08:12:57.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.242+0000 7fe6337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:57.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.242+0000 7fe6337fe640 1 -- 172.21.15.40:0/1807459837 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe634034440 con 0x7fe63c136210 2024-04-22T08:12:57.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.242+0000 7fe6337fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:57.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.242+0000 7fe6337fe640 1 -- 172.21.15.40:0/1807459837 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6340347d0 con 0x7fe63c136210 2024-04-22T08:12:57.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.242+0000 7fe6431e4640 5 monclient: authenticate success, global_id 4344 2024-04-22T08:12:57.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.242+0000 7fe6431e4640 10 monclient: get_monmap_and_config success 2024-04-22T08:12:57.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.242+0000 7fe6431e4640 4 set_mon_vals no callback set 2024-04-22T08:12:57.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.242+0000 7fe6337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:57.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.242+0000 7fe6337fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:12:57.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.242+0000 7fe6337fe640 10 monclient: dump: 2024-04-22T08:12:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:57.251 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.242+0000 7fe6431e4640 10 monclient: shutdown 2024-04-22T08:12:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.242+0000 7fe6431e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.242+0000 7fe6431e4640 1 -- 172.21.15.40:0/1807459837 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe63c136210 msgr2=0x7fe63c13e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.242+0000 7fe6431e4640 1 --2- 172.21.15.40:0/1807459837 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe63c136210 0x7fe63c13e410 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe634009d90 tx=0x7fe63402bd00 comp rx=0 tx=0).stop 2024-04-22T08:12:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.242+0000 7fe6431e4640 1 -- 172.21.15.40:0/1807459837 shutdown_connections 2024-04-22T08:12:57.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.242+0000 7fe6431e4640 1 --2- 172.21.15.40:0/1807459837 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe63c136210 0x7fe63c13e410 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:57.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.242+0000 7fe6431e4640 1 -- 172.21.15.40:0/1807459837 >> 172.21.15.40:0/1807459837 conn(0x7fe63c0a16e0 msgr2=0x7fe63c0a1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:57.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.242+0000 7fe6431e4640 1 -- 172.21.15.40:0/1807459837 shutdown_connections 2024-04-22T08:12:57.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.242+0000 7fe6431e4640 1 -- 172.21.15.40:0/1807459837 wait complete. 2024-04-22T08:12:57.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.243+0000 7fe6431e4640 10 monclient: build_initial_monmap 2024-04-22T08:12:57.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.243+0000 7fe6431e4640 10 monclient: monmap: 2024-04-22T08:12:57.252 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:57.252 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:12:57.252 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:12:57.252 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:12:57.252 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:57.252 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:57.252 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:12:57.252 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:12:57.253 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:12:57.253 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:57.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.243+0000 7fe6431e4640 1 Processor -- start 2024-04-22T08:12:57.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.243+0000 7fe6431e4640 1 -- start start 2024-04-22T08:12:57.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.243+0000 7fe6431e4640 10 monclient: init 2024-04-22T08:12:57.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe6431e4640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:57.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe6431e4640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-22T08:12:57.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe6431e4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe63c070d70 0x7fe63c1caa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:57.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe6431e4640 10 monclient(hunting): picked mon.noname-b con 0x7fe63c070d70 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:57.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe6431e4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe63c135de0 0x7fe63c1cafa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:57.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe6431e4640 10 monclient(hunting): picked mon.noname-a con 0x7fe63c135de0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:57.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe6431e4640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe63c136210 0x7fe63c1cb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:57.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe6431e4640 10 monclient(hunting): picked mon.noname-c con 0x7fe63c136210 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:57.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe6431e4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:57.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe6431e4640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe63c12c1e0 con 0x7fe63c135de0 2024-04-22T08:12:57.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe6431e4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:57.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe6431e4640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe63c13dbb0 con 0x7fe63c070d70 2024-04-22T08:12:57.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe6431e4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:57.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe6431e4640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe63c132010 con 0x7fe63c136210 2024-04-22T08:12:57.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe6431e4640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:57.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe6431e4640 10 monclient(hunting): authenticate will time out at 851.238190s 2024-04-22T08:12:57.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe633fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe63c135de0 0x7fe63c1cafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:57.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe641781640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe63c136210 0x7fe63c1cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:57.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe641781640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe63c136210 0x7fe63c1cb4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49960/0 (socket says 172.21.15.40:49960) 2024-04-22T08:12:57.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe641781640 1 -- 172.21.15.40:0/2055861924 learned_addr learned my addr 172.21.15.40:0/2055861924 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:57.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe641781640 10 monclient(hunting): get_auth_request con 0x7fe63c136210 auth_method 0 2024-04-22T08:12:57.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe633fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe63c135de0 0x7fe63c1cafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58168/0 (socket says 172.21.15.40:58168) 2024-04-22T08:12:57.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe641781640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:57.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe641781640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:57.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe641781640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:57.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe633fff640 10 monclient(hunting): get_auth_request con 0x7fe63c135de0 auth_method 0 2024-04-22T08:12:57.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe633fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:57.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe633fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:57.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe633fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:57.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe633fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:57.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe633fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:57.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe633fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:57.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe641781640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:57.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe641781640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:57.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe641781640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:57.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe640f80640 1 --2- 172.21.15.40:0/2055861924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe63c070d70 0x7fe63c1caa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:57.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe640f80640 10 monclient(hunting): get_auth_request con 0x7fe63c070d70 auth_method 0 2024-04-22T08:12:57.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe640f80640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:57.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe640f80640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:57.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe640f80640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:57.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe633fff640 10 monclient(hunting): handle_auth_done global_id 4347 payload 824 2024-04-22T08:12:57.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe633fff640 1 -- 172.21.15.40:0/2055861924 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe63c136210 msgr2=0x7fe63c1cb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:57.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe633fff640 1 --2- 172.21.15.40:0/2055861924 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe63c136210 0x7fe63c1cb4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:57.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe633fff640 1 -- 172.21.15.40:0/2055861924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe63c070d70 msgr2=0x7fe63c1caa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:57.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe633fff640 1 --2- 172.21.15.40:0/2055861924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe63c070d70 0x7fe63c1caa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:57.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe633fff640 10 monclient: _finish_hunting 0 2024-04-22T08:12:57.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe633fff640 1 monclient: found mon.noname-a 2024-04-22T08:12:57.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe633fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:57.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe633fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:12:57.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe633fff640 1 -- 172.21.15.40:0/2055861924 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe634009a70 con 0x7fe63c135de0 2024-04-22T08:12:57.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.244+0000 7fe640f80640 1 --2- 172.21.15.40:0/2055861924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe63c070d70 0x7fe63c1caa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T08:12:57.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.247+0000 7fe633fff640 1 -- 172.21.15.40:0/2055861924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe63c135de0 msgr2=0x7fe63c1cafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:57.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.247+0000 7fe633fff640 1 --2- 172.21.15.40:0/2055861924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe63c135de0 0x7fe63c1cafa0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe62c00c410 tx=0x7fe62c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:57.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.247+0000 7fe6317fa640 1 -- 172.21.15.40:0/2055861924 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe62c0070d0 con 0x7fe63c135de0 2024-04-22T08:12:57.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.247+0000 7fe6317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:57.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.247+0000 7fe6317fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:12:57.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.247+0000 7fe6317fa640 10 monclient: dump: 2024-04-22T08:12:57.257 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:57.257 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:57.257 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:57.257 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:57.257 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:57.257 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:57.257 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:57.258 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:57.258 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:57.258 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:57.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.247+0000 7fe6317fa640 10 monclient: _finish_auth 0 2024-04-22T08:12:57.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.247+0000 7fe6317fa640 10 monclient: _check_auth_tickets 2024-04-22T08:12:57.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.247+0000 7fe6317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:57.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.247+0000 7fe6317fa640 1 -- 172.21.15.40:0/2055861924 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe62c007230 con 0x7fe63c135de0 2024-04-22T08:12:57.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.247+0000 7fe6317fa640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:57.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.247+0000 7fe6317fa640 1 -- 172.21.15.40:0/2055861924 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe62c0075e0 con 0x7fe63c135de0 2024-04-22T08:12:57.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.247+0000 7fe6431e4640 5 monclient: authenticate success, global_id 4347 2024-04-22T08:12:57.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.247+0000 7fe6431e4640 10 monclient: _renew_subs 2024-04-22T08:12:57.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.247+0000 7fe6431e4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:57.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.247+0000 7fe6431e4640 1 -- 172.21.15.40:0/2055861924 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe63c12b760 con 0x7fe63c135de0 2024-04-22T08:12:57.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.247+0000 7fe632ffd640 4 set_mon_vals no callback set 2024-04-22T08:12:57.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.247+0000 7fe6317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:57.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.247+0000 7fe6317fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:12:57.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.247+0000 7fe6317fa640 10 monclient: dump: 2024-04-22T08:12:57.259 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:57.259 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:57.259 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:57.259 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:57.259 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:57.259 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:57.259 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:57.259 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:57.259 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:57.259 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:57.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.247+0000 7fe6431e4640 10 monclient: _renew_subs 2024-04-22T08:12:57.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.247+0000 7fe6431e4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:57.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.247+0000 7fe6431e4640 1 -- 172.21.15.40:0/2055861924 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe63c12bf00 con 0x7fe63c135de0 2024-04-22T08:12:57.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.248+0000 7fe6317fa640 1 -- 172.21.15.40:0/2055861924 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fe62c01b3b0 con 0x7fe63c135de0 2024-04-22T08:12:57.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.248+0000 7fe6431e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:57.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.248+0000 7fe6431e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:57.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.248+0000 7fe6431e4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:57.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.248+0000 7fe6317fa640 1 --2- 172.21.15.40:0/2055861924 >> v2:172.21.15.189:6800/30168 conn(0x7fe6280377d0 0x7fe628039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:57.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.248+0000 7fe6431e4640 1 -- 172.21.15.40:0/2055861924 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe63c04e9b0 con 0x7fe63c135de0 2024-04-22T08:12:57.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.248+0000 7fe6317fa640 1 -- 172.21.15.40:0/2055861924 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3506+0+0 (secure 0 0 0) 0x7fe62c0498c0 con 0x7fe63c135de0 2024-04-22T08:12:57.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.248+0000 7fe640f80640 1 --2- 172.21.15.40:0/2055861924 >> v2:172.21.15.189:6800/30168 conn(0x7fe6280377d0 0x7fe628039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:57.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.248+0000 7fe640f80640 10 monclient: get_auth_request con 0x7fe6280377d0 auth_method 0 2024-04-22T08:12:57.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.251+0000 7fe640f80640 1 --2- 172.21.15.40:0/2055861924 >> v2:172.21.15.189:6800/30168 conn(0x7fe6280377d0 0x7fe628039c90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe638003eb0 tx=0x7fe638009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:57.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.252+0000 7fe6317fa640 1 -- 172.21.15.40:0/2055861924 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7fe62c01a360 con 0x7fe63c135de0 2024-04-22T08:12:57.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.252+0000 7fe6317fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:57.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.252+0000 7fe6317fa640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:12:57.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.432+0000 7fe6431e4640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-04-22T08:12:57.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.432+0000 7fe6431e4640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-04-22T08:12:57.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.432+0000 7fe6431e4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:57.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.432+0000 7fe6431e4640 1 -- 172.21.15.40:0/2055861924 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fe63c065bd0 con 0x7fe63c135de0 2024-04-22T08:12:57.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.432+0000 7fe6317fa640 1 -- 172.21.15.40:0/2055861924 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v15) v1 ==== 92+0+10 (secure 0 0 0) 0x7fe62c017030 con 0x7fe63c135de0 2024-04-22T08:12:57.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.432+0000 7fe6317fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-04-22T08:12:57.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.432+0000 7fe6317fa640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:12:57.433 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 8 2024-04-22T08:12:57.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.434+0000 7fe6431e4640 1 -- 172.21.15.40:0/2055861924 >> v2:172.21.15.189:6800/30168 conn(0x7fe6280377d0 msgr2=0x7fe628039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:57.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.434+0000 7fe6431e4640 1 --2- 172.21.15.40:0/2055861924 >> v2:172.21.15.189:6800/30168 conn(0x7fe6280377d0 0x7fe628039c90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe638003eb0 tx=0x7fe638009040 comp rx=0 tx=0).stop 2024-04-22T08:12:57.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.434+0000 7fe6431e4640 10 monclient: shutdown 2024-04-22T08:12:57.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.434+0000 7fe6431e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:57.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.434+0000 7fe6431e4640 1 -- 172.21.15.40:0/2055861924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe63c135de0 msgr2=0x7fe63c1cafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:57.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.434+0000 7fe6431e4640 1 --2- 172.21.15.40:0/2055861924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe63c135de0 0x7fe63c1cafa0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe62c00c410 tx=0x7fe62c00c8d0 comp rx=0 tx=0).stop 2024-04-22T08:12:57.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.434+0000 7fe6431e4640 1 -- 172.21.15.40:0/2055861924 shutdown_connections 2024-04-22T08:12:57.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.434+0000 7fe6431e4640 1 --2- 172.21.15.40:0/2055861924 >> v2:172.21.15.189:6800/30168 conn(0x7fe6280377d0 0x7fe628039c90 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:57.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.434+0000 7fe6431e4640 1 --2- 172.21.15.40:0/2055861924 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe63c136210 0x7fe63c1cb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:57.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.434+0000 7fe6431e4640 1 --2- 172.21.15.40:0/2055861924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe63c135de0 0x7fe63c1cafa0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:57.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.434+0000 7fe6431e4640 1 --2- 172.21.15.40:0/2055861924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe63c070d70 0x7fe63c1caa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:57.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.434+0000 7fe6431e4640 1 -- 172.21.15.40:0/2055861924 >> 172.21.15.40:0/2055861924 conn(0x7fe63c0a16e0 msgr2=0x7fe63c134580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:57.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.434+0000 7fe6431e4640 1 -- 172.21.15.40:0/2055861924 shutdown_connections 2024-04-22T08:12:57.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.434+0000 7fe6431e4640 1 -- 172.21.15.40:0/2055861924 wait complete. 2024-04-22T08:12:57.444 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2024-04-22T08:12:57.445 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:12:57.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.544+0000 7f2b2f900640 10 monclient: get_monmap_and_config 2024-04-22T08:12:57.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.544+0000 7f2b2f900640 10 monclient: build_initial_monmap 2024-04-22T08:12:57.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.544+0000 7f2b2f900640 10 monclient: monmap: 2024-04-22T08:12:57.544 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:57.544 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:57.544 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:57.544906+0000 2024-04-22T08:12:57.544 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:57.544906+0000 2024-04-22T08:12:57.544 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:57.545 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:57.545 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:12:57.545 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:12:57.545 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:12:57.545 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:12:57.545 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:12:57.545 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:57.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.544+0000 7f2b2f900640 1 Processor -- start 2024-04-22T08:12:57.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.544+0000 7f2b2f900640 1 -- start start 2024-04-22T08:12:57.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.544+0000 7f2b2f900640 10 monclient: init 2024-04-22T08:12:57.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.545+0000 7f2b2f900640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:57.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.545+0000 7f2b2f900640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-04-22T08:12:57.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.545+0000 7f2b2f900640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2b2809ee10 addr v1:172.21.15.189:6789/0 2024-04-22T08:12:57.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.545+0000 7f2b2f900640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2b281361e0 0x7f2b2813b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:57.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.545+0000 7f2b2f900640 10 monclient(hunting): picked mon.noname-c con 0x7f2b281361e0 addr v2:172.21.15.189:3300/0 2024-04-22T08:12:57.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.545+0000 7f2b2f900640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2b2813b5a0 0x7f2b2813d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:57.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.545+0000 7f2b2f900640 10 monclient(hunting): picked mon.noname-a con 0x7f2b2813b5a0 addr v2:172.21.15.40:3300/0 2024-04-22T08:12:57.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.545+0000 7f2b2f900640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b28136c80 con 0x7f2b2809ee10 2024-04-22T08:12:57.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.545+0000 7f2b2f900640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:57.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.545+0000 7f2b2f900640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b280a3f40 con 0x7f2b2813b5a0 2024-04-22T08:12:57.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.545+0000 7f2b2f900640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:57.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.545+0000 7f2b2f900640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b2809f1f0 con 0x7f2b281361e0 2024-04-22T08:12:57.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.545+0000 7f2b2f900640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:57.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.545+0000 7f2b2f900640 10 monclient(hunting): authenticate will time out at 851.539133s 2024-04-22T08:12:57.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.545+0000 7f2b2de9d640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2b2813b5a0 0x7f2b2813d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:57.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.545+0000 7f2b2de9d640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2b2813b5a0 0x7f2b2813d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58178/0 (socket says 172.21.15.40:58178) 2024-04-22T08:12:57.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.545+0000 7f2b2de9d640 1 -- 172.21.15.40:0/1296180074 learned_addr learned my addr 172.21.15.40:0/1296180074 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:57.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.545+0000 7f2b2de9d640 10 monclient(hunting): get_auth_request con 0x7f2b2813b5a0 auth_method 0 2024-04-22T08:12:57.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.545+0000 7f2b2de9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:57.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.545+0000 7f2b2de9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:57.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.545+0000 7f2b2de9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:57.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.545+0000 7f2b2ce9b640 1 --2- 172.21.15.40:0/1296180074 >> v2:172.21.15.189:3300/0 conn(0x7f2b281361e0 0x7f2b2813b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:57.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.545+0000 7f2b2de9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:57.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.545+0000 7f2b2de9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:57.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.545+0000 7f2b2de9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:57.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.545+0000 7f2b2ce9b640 10 monclient(hunting): get_auth_request con 0x7f2b281361e0 auth_method 0 2024-04-22T08:12:57.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.545+0000 7f2b2ce9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:57.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.545+0000 7f2b2ce9b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:57.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.545+0000 7f2b2ce9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:57.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.545+0000 7f2b2ce9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:57.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.545+0000 7f2b2ce9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:57.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.545+0000 7f2b2ce9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:57.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.545+0000 7f2b2de9d640 10 monclient(hunting): handle_auth_done global_id 4350 payload 274 2024-04-22T08:12:57.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.545+0000 7f2b2de9d640 1 -- 172.21.15.40:0/1296180074 >> v2:172.21.15.189:3300/0 conn(0x7f2b281361e0 msgr2=0x7f2b2813b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:57.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.545+0000 7f2b2de9d640 1 --2- 172.21.15.40:0/1296180074 >> v2:172.21.15.189:3300/0 conn(0x7f2b281361e0 0x7f2b2813b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:57.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.545+0000 7f2b2de9d640 1 -- 172.21.15.40:0/1296180074 >> v1:172.21.15.189:6789/0 conn(0x7f2b2809ee10 legacy=0x7f2b280a19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:57.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.545+0000 7f2b2de9d640 10 monclient: _finish_hunting 0 2024-04-22T08:12:57.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.545+0000 7f2b2de9d640 1 monclient: found mon.noname-a 2024-04-22T08:12:57.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.545+0000 7f2b2de9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:57.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.545+0000 7f2b2de9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:12:57.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.545+0000 7f2b2de9d640 1 -- 172.21.15.40:0/1296180074 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b280a20d0 con 0x7f2b2813b5a0 2024-04-22T08:12:57.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.548+0000 7f2b2de9d640 1 -- 172.21.15.40:0/1296180074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b2813b5a0 msgr2=0x7f2b2813d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:57.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.548+0000 7f2b2de9d640 1 --2- 172.21.15.40:0/1296180074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b2813b5a0 0x7f2b2813d990 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2b2809f930 tx=0x7f2b2402bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ce531d2e2c021c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:57.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.548+0000 7f2b1ffff640 1 -- 172.21.15.40:0/1296180074 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b2402cb70 con 0x7f2b2813b5a0 2024-04-22T08:12:57.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.548+0000 7f2b1ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:57.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.548+0000 7f2b1ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:12:57.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.548+0000 7f2b1ffff640 10 monclient: dump: 2024-04-22T08:12:57.560 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:57.561 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:57.561 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:57.561 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:57.561 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:57.561 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:57.561 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:57.561 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:57.561 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:57.561 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:57.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.548+0000 7f2b1ffff640 10 monclient: _finish_auth 0 2024-04-22T08:12:57.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.548+0000 7f2b1ffff640 10 monclient: _check_auth_tickets 2024-04-22T08:12:57.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.548+0000 7f2b1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:57.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.548+0000 7f2b1ffff640 1 -- 172.21.15.40:0/1296180074 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b2402ccd0 con 0x7f2b2813b5a0 2024-04-22T08:12:57.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.548+0000 7f2b1ffff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:57.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.548+0000 7f2b1ffff640 1 -- 172.21.15.40:0/1296180074 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b24034610 con 0x7f2b2813b5a0 2024-04-22T08:12:57.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.548+0000 7f2b2f900640 5 monclient: authenticate success, global_id 4350 2024-04-22T08:12:57.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.548+0000 7f2b2f900640 10 monclient: get_monmap_and_config success 2024-04-22T08:12:57.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.548+0000 7f2b2f900640 4 set_mon_vals no callback set 2024-04-22T08:12:57.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.548+0000 7f2b1ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:57.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.548+0000 7f2b1ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:12:57.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.548+0000 7f2b1ffff640 10 monclient: dump: 2024-04-22T08:12:57.562 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:57.562 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:57.562 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:57.562 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:57.562 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:57.562 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:57.562 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:57.563 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:57.563 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:57.563 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:57.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.549+0000 7f2b2f900640 10 monclient: shutdown 2024-04-22T08:12:57.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.549+0000 7f2b2f900640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:57.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.549+0000 7f2b2f900640 1 -- 172.21.15.40:0/1296180074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b2813b5a0 msgr2=0x7f2b2813d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:57.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.549+0000 7f2b2f900640 1 --2- 172.21.15.40:0/1296180074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b2813b5a0 0x7f2b2813d990 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2b2809f930 tx=0x7f2b2402bdc0 comp rx=0 tx=0).stop 2024-04-22T08:12:57.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.549+0000 7f2b2f900640 1 -- 172.21.15.40:0/1296180074 shutdown_connections 2024-04-22T08:12:57.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.549+0000 7f2b2f900640 1 --2- 172.21.15.40:0/1296180074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b2813b5a0 0x7f2b2813d990 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:57.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.549+0000 7f2b2f900640 1 --2- 172.21.15.40:0/1296180074 >> v2:172.21.15.189:3300/0 conn(0x7f2b281361e0 0x7f2b2813b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:57.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.549+0000 7f2b2f900640 1 -- 172.21.15.40:0/1296180074 >> 172.21.15.40:0/1296180074 conn(0x7f2b280a4e80 msgr2=0x7f2b280a5290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:57.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.549+0000 7f2b2f900640 1 -- 172.21.15.40:0/1296180074 shutdown_connections 2024-04-22T08:12:57.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.549+0000 7f2b2f900640 1 -- 172.21.15.40:0/1296180074 wait complete. 2024-04-22T08:12:57.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.550+0000 7f2b2f900640 10 monclient: build_initial_monmap 2024-04-22T08:12:57.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.550+0000 7f2b2f900640 10 monclient: monmap: 2024-04-22T08:12:57.564 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:57.564 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:12:57.564 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:12:57.564 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:12:57.564 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:57.564 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:57.564 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:12:57.564 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:12:57.564 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:12:57.564 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:57.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.550+0000 7f2b2f900640 1 Processor -- start 2024-04-22T08:12:57.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.550+0000 7f2b2f900640 1 -- start start 2024-04-22T08:12:57.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.550+0000 7f2b2f900640 10 monclient: init 2024-04-22T08:12:57.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.550+0000 7f2b2f900640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:57.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.550+0000 7f2b2f900640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:12:57.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.550+0000 7f2b2f900640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b2809ee10 0x7f2b2811ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:57.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.550+0000 7f2b2f900640 10 monclient(hunting): picked mon.noname-a con 0x7f2b2809ee10 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:57.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.550+0000 7f2b2f900640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b281361e0 0x7f2b28133a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:57.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.550+0000 7f2b2f900640 10 monclient(hunting): picked mon.noname-b con 0x7f2b281361e0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:57.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.550+0000 7f2b2f900640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b2813b5a0 0x7f2b281ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:57.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.550+0000 7f2b2f900640 10 monclient(hunting): picked mon.noname-c con 0x7f2b2813b5a0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:57.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.550+0000 7f2b2f900640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:57.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.550+0000 7f2b2f900640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b281200f0 con 0x7f2b2809ee10 2024-04-22T08:12:57.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.550+0000 7f2b2f900640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:57.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.550+0000 7f2b2f900640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b28071190 con 0x7f2b281361e0 2024-04-22T08:12:57.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.550+0000 7f2b2f900640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:57.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.550+0000 7f2b2f900640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b280a6bd0 con 0x7f2b2813b5a0 2024-04-22T08:12:57.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.550+0000 7f2b2f900640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:57.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.550+0000 7f2b2f900640 10 monclient(hunting): authenticate will time out at 851.544715s 2024-04-22T08:12:57.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.551+0000 7f2b2d69c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b2809ee10 0x7f2b2811ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:57.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.551+0000 7f2b2de9d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b2813b5a0 0x7f2b281ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:57.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.551+0000 7f2b2d69c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b2809ee10 0x7f2b2811ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58180/0 (socket says 172.21.15.40:58180) 2024-04-22T08:12:57.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.551+0000 7f2b2d69c640 1 -- 172.21.15.40:0/1989302939 learned_addr learned my addr 172.21.15.40:0/1989302939 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:57.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.551+0000 7f2b2d69c640 10 monclient(hunting): get_auth_request con 0x7f2b2809ee10 auth_method 0 2024-04-22T08:12:57.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.551+0000 7f2b2d69c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:57.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.551+0000 7f2b2d69c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:57.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.551+0000 7f2b2d69c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:57.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.551+0000 7f2b2de9d640 10 monclient(hunting): get_auth_request con 0x7f2b2813b5a0 auth_method 0 2024-04-22T08:12:57.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.551+0000 7f2b2de9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:57.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.551+0000 7f2b2de9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:57.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.551+0000 7f2b2de9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:57.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.551+0000 7f2b2d69c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:57.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.551+0000 7f2b2d69c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:57.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.551+0000 7f2b2d69c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:57.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.551+0000 7f2b2de9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:57.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.551+0000 7f2b2de9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:57.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.551+0000 7f2b2de9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:57.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.551+0000 7f2b2d69c640 10 monclient(hunting): handle_auth_done global_id 4353 payload 824 2024-04-22T08:12:57.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.551+0000 7f2b2ce9b640 1 --2- 172.21.15.40:0/1989302939 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b281361e0 0x7f2b28133a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:57.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.551+0000 7f2b2d69c640 1 -- 172.21.15.40:0/1989302939 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b2813b5a0 msgr2=0x7f2b281ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:57.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.551+0000 7f2b2d69c640 1 --2- 172.21.15.40:0/1989302939 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b2813b5a0 0x7f2b281ca7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:57.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.551+0000 7f2b2d69c640 1 -- 172.21.15.40:0/1989302939 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b281361e0 msgr2=0x7f2b28133a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:57.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.551+0000 7f2b2d69c640 1 --2- 172.21.15.40:0/1989302939 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b281361e0 0x7f2b28133a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:57.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.551+0000 7f2b2d69c640 10 monclient: _finish_hunting 0 2024-04-22T08:12:57.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.551+0000 7f2b2d69c640 1 monclient: found mon.noname-a 2024-04-22T08:12:57.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.551+0000 7f2b2d69c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:57.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.551+0000 7f2b2d69c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:12:57.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.551+0000 7f2b2d69c640 1 -- 172.21.15.40:0/1989302939 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b24009b30 con 0x7f2b2809ee10 2024-04-22T08:12:57.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.551+0000 7f2b2ce9b640 10 monclient: get_auth_request con 0x7f2b281361e0 auth_method 0 2024-04-22T08:12:57.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.551+0000 7f2b2de9d640 1 --2- 172.21.15.40:0/1989302939 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b2813b5a0 0x7f2b281ca7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:12:57.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.551+0000 7f2b2ce9b640 1 --2- 172.21.15.40:0/1989302939 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b281361e0 0x7f2b28133a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T08:12:57.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.553+0000 7f2b2d69c640 1 -- 172.21.15.40:0/1989302939 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b2809ee10 msgr2=0x7f2b2811ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:57.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.553+0000 7f2b2d69c640 1 --2- 172.21.15.40:0/1989302939 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b2809ee10 0x7f2b2811ebd0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2b2000c460 tx=0x7f2b2000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:57.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.553+0000 7f2b1dffb640 1 -- 172.21.15.40:0/1989302939 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b200119a0 con 0x7f2b2809ee10 2024-04-22T08:12:57.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.553+0000 7f2b1dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:57.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.553+0000 7f2b1dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:12:57.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.553+0000 7f2b1dffb640 10 monclient: dump: 2024-04-22T08:12:57.568 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:57.568 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:57.569 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:57.569 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:57.569 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:57.569 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:57.569 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:57.569 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:57.569 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:57.569 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:57.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.553+0000 7f2b1dffb640 10 monclient: _finish_auth 0 2024-04-22T08:12:57.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.553+0000 7f2b1dffb640 10 monclient: _check_auth_tickets 2024-04-22T08:12:57.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.553+0000 7f2b1dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:57.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.553+0000 7f2b1dffb640 1 -- 172.21.15.40:0/1989302939 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b20011b40 con 0x7f2b2809ee10 2024-04-22T08:12:57.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.553+0000 7f2b2f900640 5 monclient: authenticate success, global_id 4353 2024-04-22T08:12:57.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.553+0000 7f2b1dffb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:57.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.553+0000 7f2b2f900640 10 monclient: _renew_subs 2024-04-22T08:12:57.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.553+0000 7f2b1dffb640 1 -- 172.21.15.40:0/1989302939 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b20012420 con 0x7f2b2809ee10 2024-04-22T08:12:57.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.553+0000 7f2b2f900640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:57.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.553+0000 7f2b2f900640 1 -- 172.21.15.40:0/1989302939 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b2811fb30 con 0x7f2b2809ee10 2024-04-22T08:12:57.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.553+0000 7f2b1dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:57.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.553+0000 7f2b1dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:12:57.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.553+0000 7f2b1dffb640 10 monclient: dump: 2024-04-22T08:12:57.571 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:57.571 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:57.571 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:57.571 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:57.571 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:57.571 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:57.571 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:57.571 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:57.571 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:57.571 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:57.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.553+0000 7f2b1f7fe640 4 set_mon_vals no callback set 2024-04-22T08:12:57.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.554+0000 7f2b2f900640 10 monclient: _renew_subs 2024-04-22T08:12:57.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.554+0000 7f2b2f900640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:57.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.554+0000 7f2b2f900640 1 -- 172.21.15.40:0/1989302939 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b280717a0 con 0x7f2b2809ee10 2024-04-22T08:12:57.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.554+0000 7f2b1dffb640 1 -- 172.21.15.40:0/1989302939 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f2b200125c0 con 0x7f2b2809ee10 2024-04-22T08:12:57.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.554+0000 7f2b1dffb640 1 --2- 172.21.15.40:0/1989302939 >> v2:172.21.15.189:6800/30168 conn(0x7f2b0c037850 0x7f2b0c039d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:57.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.554+0000 7f2b1dffb640 1 -- 172.21.15.40:0/1989302939 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f2b2004a140 con 0x7f2b2809ee10 2024-04-22T08:12:57.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.554+0000 7f2b2f900640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:57.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.555+0000 7f2b2f900640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:57.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.555+0000 7f2b2f900640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:57.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.555+0000 7f2b2f900640 1 -- 172.21.15.40:0/1989302939 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b280a23c0 con 0x7f2b2809ee10 2024-04-22T08:12:57.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.555+0000 7f2b2ce9b640 1 --2- 172.21.15.40:0/1989302939 >> v2:172.21.15.189:6800/30168 conn(0x7f2b0c037850 0x7f2b0c039d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:57.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.555+0000 7f2b2ce9b640 10 monclient: get_auth_request con 0x7f2b0c037850 auth_method 0 2024-04-22T08:12:57.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.557+0000 7f2b2ce9b640 1 --2- 172.21.15.40:0/1989302939 >> v2:172.21.15.189:6800/30168 conn(0x7f2b0c037850 0x7f2b0c039d10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2b2811fa60 tx=0x7f2b18008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:57.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.559+0000 7f2b1dffb640 1 -- 172.21.15.40:0/1989302939 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f2b280a23c0 con 0x7f2b2809ee10 2024-04-22T08:12:57.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.559+0000 7f2b1dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:57.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.559+0000 7f2b1dffb640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:12:57.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.737+0000 7f2b2f900640 1 -- 172.21.15.40:0/1989302939 --> v2:172.21.15.189:6800/30168 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2b2809fef0 con 0x7f2b0c037850 2024-04-22T08:12:57.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.739+0000 7f2b1dffb640 1 -- 172.21.15.40:0/1989302939 <== mgr.4102 v2:172.21.15.189:6800/30168 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+40579 (secure 0 0 0) 0x7f2b18010900 con 0x7f2b0c037850 2024-04-22T08:12:57.739 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:12:57.739 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-22T08:12:57.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.741+0000 7f2b2f900640 1 -- 172.21.15.40:0/1989302939 >> v2:172.21.15.189:6800/30168 conn(0x7f2b0c037850 msgr2=0x7f2b0c039d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:57.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.741+0000 7f2b2f900640 1 --2- 172.21.15.40:0/1989302939 >> v2:172.21.15.189:6800/30168 conn(0x7f2b0c037850 0x7f2b0c039d10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2b2811fa60 tx=0x7f2b18008040 comp rx=0 tx=0).stop 2024-04-22T08:12:57.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.741+0000 7f2b2f900640 10 monclient: shutdown 2024-04-22T08:12:57.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.741+0000 7f2b2f900640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:57.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.741+0000 7f2b2f900640 1 -- 172.21.15.40:0/1989302939 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b2809ee10 msgr2=0x7f2b2811ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:57.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.741+0000 7f2b2f900640 1 --2- 172.21.15.40:0/1989302939 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b2809ee10 0x7f2b2811ebd0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2b2000c460 tx=0x7f2b2000c920 comp rx=0 tx=0).stop 2024-04-22T08:12:57.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.741+0000 7f2b2f900640 1 -- 172.21.15.40:0/1989302939 shutdown_connections 2024-04-22T08:12:57.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.741+0000 7f2b2f900640 1 --2- 172.21.15.40:0/1989302939 >> v2:172.21.15.189:6800/30168 conn(0x7f2b0c037850 0x7f2b0c039d10 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:57.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.741+0000 7f2b2f900640 1 --2- 172.21.15.40:0/1989302939 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b281361e0 0x7f2b28133a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:57.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.741+0000 7f2b2f900640 1 --2- 172.21.15.40:0/1989302939 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b2813b5a0 0x7f2b281ca7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:57.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.741+0000 7f2b2f900640 1 --2- 172.21.15.40:0/1989302939 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b2809ee10 0x7f2b2811ebd0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:57.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.741+0000 7f2b2f900640 1 -- 172.21.15.40:0/1989302939 >> 172.21.15.40:0/1989302939 conn(0x7f2b280a4e80 msgr2=0x7f2b28134dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:57.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.741+0000 7f2b2f900640 1 -- 172.21.15.40:0/1989302939 shutdown_connections 2024-04-22T08:12:57.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.741+0000 7f2b2f900640 1 -- 172.21.15.40:0/1989302939 wait complete. 2024-04-22T08:12:57.751 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2024-04-22T08:12:56.078017+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":8,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":164644,"kb_used_data":1744,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566066396,"statfs":{"total":579820584960,"available":579651989504,"internally_reserved":0,"allocated":1785856,"data_stored":1223245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9560,"internal_metadata":165927592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001014"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:12:52.005174+0000","last_change":"2024-04-22T08:12:51.443086+0000","last_active":"2024-04-22T08:12:52.005174+0000","last_peered":"2024-04-22T08:12:52.005174+0000","last_clean":"2024-04-22T08:12:52.005174+0000","last_became_active":"2024-04-22T08:12:51.442730+0000","last_became_peered":"2024-04-22T08:12:51.442730+0000","last_unstale":"2024-04-22T08:12:52.005174+0000","last_undegraded":"2024-04-22T08:12:52.005174+0000","last_fullsized":"2024-04-22T08:12:52.005174+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:06:13.481863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:12:52.009552+0000","last_change":"2024-04-22T08:12:50.841276+0000","last_active":"2024-04-22T08:12:52.009552+0000","last_peered":"2024-04-22T08:12:52.009552+0000","last_clean":"2024-04-22T08:12:52.009552+0000","last_became_active":"2024-04-22T08:12:50.841008+0000","last_became_peered":"2024-04-22T08:12:50.841008+0000","last_unstale":"2024-04-22T08:12:52.009552+0000","last_undegraded":"2024-04-22T08:12:52.009552+0000","last_fullsized":"2024-04-22T08:12:52.009552+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:47:00.224321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:12:52.005217+0000","last_change":"2024-04-22T08:12:51.403419+0000","last_active":"2024-04-22T08:12:52.005217+0000","last_peered":"2024-04-22T08:12:52.005217+0000","last_clean":"2024-04-22T08:12:52.005217+0000","last_became_active":"2024-04-22T08:12:51.403182+0000","last_became_peered":"2024-04-22T08:12:51.403182+0000","last_unstale":"2024-04-22T08:12:52.005217+0000","last_undegraded":"2024-04-22T08:12:52.005217+0000","last_fullsized":"2024-04-22T08:12:52.005217+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:52:39.730597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:12:52.009486+0000","last_change":"2024-04-22T08:12:51.403260+0000","last_active":"2024-04-22T08:12:52.009486+0000","last_peered":"2024-04-22T08:12:52.009486+0000","last_clean":"2024-04-22T08:12:52.009486+0000","last_became_active":"2024-04-22T08:12:51.402980+0000","last_became_peered":"2024-04-22T08:12:51.402980+0000","last_unstale":"2024-04-22T08:12:52.009486+0000","last_undegraded":"2024-04-22T08:12:52.009486+0000","last_fullsized":"2024-04-22T08:12:52.009486+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:08:04.625870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:12:52.041135+0000","last_change":"2024-04-22T08:12:50.836566+0000","last_active":"2024-04-22T08:12:52.041135+0000","last_peered":"2024-04-22T08:12:52.041135+0000","last_clean":"2024-04-22T08:12:52.041135+0000","last_became_active":"2024-04-22T08:12:50.836250+0000","last_became_peered":"2024-04-22T08:12:50.836250+0000","last_unstale":"2024-04-22T08:12:52.041135+0000","last_undegraded":"2024-04-22T08:12:52.041135+0000","last_fullsized":"2024-04-22T08:12:52.041135+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:42:13.693209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:12:52.042149+0000","last_change":"2024-04-22T08:12:50.836508+0000","last_active":"2024-04-22T08:12:52.042149+0000","last_peered":"2024-04-22T08:12:52.042149+0000","last_clean":"2024-04-22T08:12:52.042149+0000","last_became_active":"2024-04-22T08:12:50.836210+0000","last_became_peered":"2024-04-22T08:12:50.836210+0000","last_unstale":"2024-04-22T08:12:52.042149+0000","last_undegraded":"2024-04-22T08:12:52.042149+0000","last_fullsized":"2024-04-22T08:12:52.042149+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:24:53.345661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:12:52.005162+0000","last_change":"2024-04-22T08:12:50.842633+0000","last_active":"2024-04-22T08:12:52.005162+0000","last_peered":"2024-04-22T08:12:52.005162+0000","last_clean":"2024-04-22T08:12:52.005162+0000","last_became_active":"2024-04-22T08:12:50.842392+0000","last_became_peered":"2024-04-22T08:12:50.842392+0000","last_unstale":"2024-04-22T08:12:52.005162+0000","last_undegraded":"2024-04-22T08:12:52.005162+0000","last_fullsized":"2024-04-22T08:12:52.005162+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:53:14.717539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:12:52.041139+0000","last_change":"2024-04-22T08:12:50.835820+0000","last_active":"2024-04-22T08:12:52.041139+0000","last_peered":"2024-04-22T08:12:52.041139+0000","last_clean":"2024-04-22T08:12:52.041139+0000","last_became_active":"2024-04-22T08:12:50.835541+0000","last_became_peered":"2024-04-22T08:12:50.835541+0000","last_unstale":"2024-04-22T08:12:52.041139+0000","last_undegraded":"2024-04-22T08:12:52.041139+0000","last_fullsized":"2024-04-22T08:12:52.041139+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:52:21.798509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":160,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:12:53.045764+0000","last_change":"2024-04-22T08:12:51.404305+0000","last_active":"2024-04-22T08:12:53.045764+0000","last_peered":"2024-04-22T08:12:53.045764+0000","last_clean":"2024-04-22T08:12:53.045764+0000","last_became_active":"2024-04-22T08:12:51.403208+0000","last_became_peered":"2024-04-22T08:12:51.403208+0000","last_unstale":"2024-04-22T08:12:53.045764+0000","last_undegraded":"2024-04-22T08:12:53.045764+0000","last_fullsized":"2024-04-22T08:12:53.045764+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:04:43.253802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640262,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27152,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":131072,"data_stored":37965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.248}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]}]},{"osd":0,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28112,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343728,"statfs":{"total":96636764160,"available":96607977472,"internally_reserved":0,"allocated":663552,"data_stored":566893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]}]},{"osd":5,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27152,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":131072,"data_stored":37965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.528}]}]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27152,"kb_used_data":124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":126976,"data_stored":37004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30399999999999999}]}]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27924,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343916,"statfs":{"total":96636764160,"available":96608169984,"internally_reserved":0,"allocated":602112,"data_stored":505453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.877}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98399999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27152,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":131072,"data_stored":37965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2170000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.093}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:12:57.752 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:12:57.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.851+0000 7f1435329640 10 monclient: get_monmap_and_config 2024-04-22T08:12:57.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.851+0000 7f1435329640 10 monclient: build_initial_monmap 2024-04-22T08:12:57.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.851+0000 7f1435329640 10 monclient: monmap: 2024-04-22T08:12:57.851 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:57.851 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:57.851 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:57.851940+0000 2024-04-22T08:12:57.851 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:57.851940+0000 2024-04-22T08:12:57.851 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:57.851 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:57.851 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:12:57.851 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:12:57.851 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:12:57.851 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:12:57.851 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:12:57.851 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:57.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.851+0000 7f1435329640 1 Processor -- start 2024-04-22T08:12:57.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.851+0000 7f1435329640 1 -- start start 2024-04-22T08:12:57.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.851+0000 7f1435329640 10 monclient: init 2024-04-22T08:12:57.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f1435329640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:57.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f1435329640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-04-22T08:12:57.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f1435329640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f143009ee10 0x7f14301361e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:57.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f1435329640 10 monclient(hunting): picked mon.noname-b con 0x7f143009ee10 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:57.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f1435329640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f14300a19c0 0x7f14300a2d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:57.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f1435329640 10 monclient(hunting): picked mon.noname-a con 0x7f14300a19c0 addr v2:172.21.15.40:3300/0 2024-04-22T08:12:57.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f1435329640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f14300a1da0 0x7f14300a32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:57.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f1435329640 10 monclient(hunting): picked mon.noname-c con 0x7f14300a1da0 addr v2:172.21.15.189:3300/0 2024-04-22T08:12:57.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f1435329640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:57.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f1435329640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14300a3f40 con 0x7f14300a19c0 2024-04-22T08:12:57.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f1435329640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:57.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f1435329640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1430136c80 con 0x7f14300a1da0 2024-04-22T08:12:57.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f1435329640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:57.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f1435329640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f143009f1f0 con 0x7f143009ee10 2024-04-22T08:12:57.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f1435329640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:57.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f1435329640 10 monclient(hunting): authenticate will time out at 851.846016s 2024-04-22T08:12:57.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f142effd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f143009ee10 0x7f14301361e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:57.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f142e7fc640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f14300a19c0 0x7f14300a2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:57.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f142effd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f143009ee10 0x7f14301361e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49964/0 (socket says 172.21.15.40:49964) 2024-04-22T08:12:57.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f142e7fc640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f14300a19c0 0x7f14300a2d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58196/0 (socket says 172.21.15.40:58196) 2024-04-22T08:12:57.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f142effd640 1 -- 172.21.15.40:0/1040440815 learned_addr learned my addr 172.21.15.40:0/1040440815 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:57.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f142effd640 10 monclient(hunting): get_auth_request con 0x7f143009ee10 auth_method 0 2024-04-22T08:12:57.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f142effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:57.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f142effd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:57.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f142effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:57.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f142e7fc640 10 monclient(hunting): get_auth_request con 0x7f14300a19c0 auth_method 0 2024-04-22T08:12:57.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f142e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:57.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f142e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:57.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f142e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:57.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f142effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:57.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f142effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:57.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f142effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:57.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f142e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:57.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f142e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:57.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f142e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:57.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f142f7fe640 1 --2- 172.21.15.40:0/1040440815 >> v2:172.21.15.189:3300/0 conn(0x7f14300a1da0 0x7f14300a32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:57.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f142f7fe640 10 monclient(hunting): get_auth_request con 0x7f14300a1da0 auth_method 0 2024-04-22T08:12:57.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f142f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:57.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f142f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:57.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f142f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:57.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f142effd640 10 monclient(hunting): handle_auth_done global_id 4334 payload 274 2024-04-22T08:12:57.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f142effd640 1 -- 172.21.15.40:0/1040440815 >> v2:172.21.15.189:3300/0 conn(0x7f14300a1da0 msgr2=0x7f14300a32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:57.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f142effd640 1 --2- 172.21.15.40:0/1040440815 >> v2:172.21.15.189:3300/0 conn(0x7f14300a1da0 0x7f14300a32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:57.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f142effd640 1 -- 172.21.15.40:0/1040440815 >> v2:172.21.15.40:3300/0 conn(0x7f14300a19c0 msgr2=0x7f14300a2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:57.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f142effd640 1 --2- 172.21.15.40:0/1040440815 >> v2:172.21.15.40:3300/0 conn(0x7f14300a19c0 0x7f14300a2d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:57.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f142effd640 10 monclient: _finish_hunting 0 2024-04-22T08:12:57.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f142effd640 1 monclient: found mon.noname-b 2024-04-22T08:12:57.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f142effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:57.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f142effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:12:57.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f142effd640 1 -- 172.21.15.40:0/1040440815 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f143009fc10 con 0x7f143009ee10 2024-04-22T08:12:57.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.852+0000 7f142e7fc640 1 --2- 172.21.15.40:0/1040440815 >> v2:172.21.15.40:3300/0 conn(0x7f14300a19c0 0x7f14300a2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:12:57.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.855+0000 7f142effd640 1 -- 172.21.15.40:0/1040440815 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f143009ee10 msgr2=0x7f14301361e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:57.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.855+0000 7f142effd640 1 --2- 172.21.15.40:0/1040440815 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f143009ee10 0x7f14301361e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f14300660a0 tx=0x7f142802b7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a59680b7e16a490 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:57.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.855+0000 7f142dffb640 1 -- 172.21.15.40:0/1040440815 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1428031960 con 0x7f143009ee10 2024-04-22T08:12:57.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.855+0000 7f142dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:57.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.855+0000 7f142dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:12:57.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.855+0000 7f142dffb640 10 monclient: dump: 2024-04-22T08:12:57.864 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:57.864 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:57.865 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:57.865 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:57.865 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:57.865 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:57.865 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:57.865 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:57.865 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:57.865 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:57.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.855+0000 7f142dffb640 10 monclient: _finish_auth 0 2024-04-22T08:12:57.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.855+0000 7f142dffb640 10 monclient: _check_auth_tickets 2024-04-22T08:12:57.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.855+0000 7f142dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:57.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.855+0000 7f142dffb640 1 -- 172.21.15.40:0/1040440815 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1428031ac0 con 0x7f143009ee10 2024-04-22T08:12:57.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.855+0000 7f142dffb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:57.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.855+0000 7f1435329640 5 monclient: authenticate success, global_id 4334 2024-04-22T08:12:57.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.855+0000 7f1435329640 10 monclient: get_monmap_and_config success 2024-04-22T08:12:57.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.855+0000 7f142dffb640 1 -- 172.21.15.40:0/1040440815 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1428031e50 con 0x7f143009ee10 2024-04-22T08:12:57.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.855+0000 7f1435329640 4 set_mon_vals no callback set 2024-04-22T08:12:57.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.855+0000 7f142dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:57.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.855+0000 7f142dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:12:57.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.855+0000 7f142dffb640 10 monclient: dump: 2024-04-22T08:12:57.866 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:57.866 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:57.866 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:57.866 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:57.866 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:57.866 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:57.866 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:57.866 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:57.867 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:57.867 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:57.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.856+0000 7f1435329640 10 monclient: shutdown 2024-04-22T08:12:57.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.856+0000 7f1435329640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:57.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.856+0000 7f1435329640 1 -- 172.21.15.40:0/1040440815 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f143009ee10 msgr2=0x7f14301361e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:57.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.856+0000 7f1435329640 1 --2- 172.21.15.40:0/1040440815 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f143009ee10 0x7f14301361e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f14300660a0 tx=0x7f142802b7f0 comp rx=0 tx=0).stop 2024-04-22T08:12:57.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.856+0000 7f1435329640 1 -- 172.21.15.40:0/1040440815 shutdown_connections 2024-04-22T08:12:57.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.856+0000 7f1435329640 1 --2- 172.21.15.40:0/1040440815 >> v2:172.21.15.189:3300/0 conn(0x7f14300a1da0 0x7f14300a32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:57.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.856+0000 7f1435329640 1 --2- 172.21.15.40:0/1040440815 >> v2:172.21.15.40:3300/0 conn(0x7f14300a19c0 0x7f14300a2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:57.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.856+0000 7f1435329640 1 --2- 172.21.15.40:0/1040440815 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f143009ee10 0x7f14301361e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:57.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.856+0000 7f1435329640 1 -- 172.21.15.40:0/1040440815 >> 172.21.15.40:0/1040440815 conn(0x7f14300a4e80 msgr2=0x7f14300a5290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:57.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.856+0000 7f1435329640 1 -- 172.21.15.40:0/1040440815 shutdown_connections 2024-04-22T08:12:57.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.856+0000 7f1435329640 1 -- 172.21.15.40:0/1040440815 wait complete. 2024-04-22T08:12:57.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.856+0000 7f1435329640 10 monclient: build_initial_monmap 2024-04-22T08:12:57.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.856+0000 7f1435329640 10 monclient: monmap: 2024-04-22T08:12:57.868 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:57.868 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:12:57.868 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:12:57.869 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:12:57.869 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:57.869 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:57.869 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:12:57.869 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:12:57.869 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:12:57.870 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:57.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.857+0000 7f1435329640 1 Processor -- start 2024-04-22T08:12:57.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.857+0000 7f1435329640 1 -- start start 2024-04-22T08:12:57.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.857+0000 7f1435329640 10 monclient: init 2024-04-22T08:12:57.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.857+0000 7f1435329640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:57.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.857+0000 7f1435329640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-22T08:12:57.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.857+0000 7f1435329640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f143009ee10 0x7f14301ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:57.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.857+0000 7f1435329640 10 monclient(hunting): picked mon.noname-b con 0x7f143009ee10 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:57.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.857+0000 7f1435329640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f14300a19c0 0x7f1430135010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:57.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.857+0000 7f1435329640 10 monclient(hunting): picked mon.noname-c con 0x7f14300a19c0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:57.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.857+0000 7f1435329640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14300a1da0 0x7f1430135550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:57.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.857+0000 7f1435329640 10 monclient(hunting): picked mon.noname-a con 0x7f14300a1da0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:57.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.857+0000 7f1435329640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:57.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.857+0000 7f1435329640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14300a27c0 con 0x7f14300a1da0 2024-04-22T08:12:57.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.857+0000 7f1435329640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:57.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.857+0000 7f1435329640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14301cb590 con 0x7f143009ee10 2024-04-22T08:12:57.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.857+0000 7f1435329640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:57.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.857+0000 7f1435329640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14300716a0 con 0x7f14300a19c0 2024-04-22T08:12:57.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.857+0000 7f1435329640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:57.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.857+0000 7f1435329640 10 monclient(hunting): authenticate will time out at 851.851659s 2024-04-22T08:12:57.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.857+0000 7f142f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14300a1da0 0x7f1430135550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:57.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.858+0000 7f142e7fc640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f14300a19c0 0x7f1430135010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:57.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.858+0000 7f142f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14300a1da0 0x7f1430135550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58212/0 (socket says 172.21.15.40:58212) 2024-04-22T08:12:57.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.858+0000 7f142e7fc640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f14300a19c0 0x7f1430135010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49968/0 (socket says 172.21.15.40:49968) 2024-04-22T08:12:57.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.858+0000 7f142f7fe640 1 -- 172.21.15.40:0/1077633950 learned_addr learned my addr 172.21.15.40:0/1077633950 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:57.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.858+0000 7f142f7fe640 10 monclient(hunting): get_auth_request con 0x7f14300a1da0 auth_method 0 2024-04-22T08:12:57.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.858+0000 7f142f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:57.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.858+0000 7f142f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:57.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.858+0000 7f142f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:57.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.858+0000 7f142e7fc640 10 monclient(hunting): get_auth_request con 0x7f14300a19c0 auth_method 0 2024-04-22T08:12:57.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.858+0000 7f142e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:57.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.858+0000 7f142e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:57.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.858+0000 7f142e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:57.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.858+0000 7f142f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:57.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.858+0000 7f142f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:57.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.858+0000 7f142f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:57.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.858+0000 7f142effd640 1 --2- 172.21.15.40:0/1077633950 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f143009ee10 0x7f14301ca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:57.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.858+0000 7f142e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:57.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.858+0000 7f142e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:57.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.858+0000 7f142e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:57.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.858+0000 7f142effd640 10 monclient(hunting): get_auth_request con 0x7f143009ee10 auth_method 0 2024-04-22T08:12:57.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.858+0000 7f142effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:57.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.858+0000 7f142effd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:57.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.858+0000 7f142effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:57.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.858+0000 7f142effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:57.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.858+0000 7f142effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:57.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.858+0000 7f142effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:57.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.858+0000 7f142f7fe640 10 monclient(hunting): handle_auth_done global_id 4359 payload 824 2024-04-22T08:12:57.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.858+0000 7f142f7fe640 1 -- 172.21.15.40:0/1077633950 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f14300a19c0 msgr2=0x7f1430135010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:57.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.858+0000 7f142f7fe640 1 --2- 172.21.15.40:0/1077633950 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f14300a19c0 0x7f1430135010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:57.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.858+0000 7f142f7fe640 1 -- 172.21.15.40:0/1077633950 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f143009ee10 msgr2=0x7f14301ca8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:57.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.858+0000 7f142f7fe640 1 --2- 172.21.15.40:0/1077633950 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f143009ee10 0x7f14301ca8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:57.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.858+0000 7f142f7fe640 10 monclient: _finish_hunting 0 2024-04-22T08:12:57.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.858+0000 7f142f7fe640 1 monclient: found mon.noname-a 2024-04-22T08:12:57.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.858+0000 7f142f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:57.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.858+0000 7f142f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:12:57.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.858+0000 7f142f7fe640 1 -- 172.21.15.40:0/1077633950 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1418009400 con 0x7f14300a1da0 2024-04-22T08:12:57.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.858+0000 7f142e7fc640 1 --2- 172.21.15.40:0/1077633950 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f14300a19c0 0x7f1430135010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:12:57.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.860+0000 7f142f7fe640 1 -- 172.21.15.40:0/1077633950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14300a1da0 msgr2=0x7f1430135550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:57.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.860+0000 7f142f7fe640 1 --2- 172.21.15.40:0/1077633950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14300a1da0 0x7f1430135550 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1420007e40 tx=0x7f142000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:57.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.860+0000 7f14277fe640 1 -- 172.21.15.40:0/1077633950 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1420013790 con 0x7f14300a1da0 2024-04-22T08:12:57.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.860+0000 7f14277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:57.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.860+0000 7f14277fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:12:57.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.860+0000 7f14277fe640 10 monclient: dump: 2024-04-22T08:12:57.877 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:57.877 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:57.877 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:57.877 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:57.877 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:57.877 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:57.878 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:57.878 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:57.878 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:57.878 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:57.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.860+0000 7f14277fe640 10 monclient: _finish_auth 0 2024-04-22T08:12:57.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.860+0000 7f14277fe640 10 monclient: _check_auth_tickets 2024-04-22T08:12:57.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.860+0000 7f14277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:57.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.860+0000 7f1435329640 5 monclient: authenticate success, global_id 4359 2024-04-22T08:12:57.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.860+0000 7f14277fe640 1 -- 172.21.15.40:0/1077633950 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f142000ee90 con 0x7f14300a1da0 2024-04-22T08:12:57.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.860+0000 7f14277fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:57.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.860+0000 7f1435329640 10 monclient: _renew_subs 2024-04-22T08:12:57.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.860+0000 7f14277fe640 1 -- 172.21.15.40:0/1077633950 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1420013b50 con 0x7f14300a1da0 2024-04-22T08:12:57.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.860+0000 7f1435329640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:57.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.860+0000 7f1435329640 1 -- 172.21.15.40:0/1077633950 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14280094d0 con 0x7f14300a1da0 2024-04-22T08:12:57.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.860+0000 7f14277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:57.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.860+0000 7f142cff9640 4 set_mon_vals no callback set 2024-04-22T08:12:57.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.860+0000 7f14277fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:12:57.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.860+0000 7f14277fe640 10 monclient: dump: 2024-04-22T08:12:57.879 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:57.879 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:57.879 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:57.879 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:57.880 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:57.880 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:57.880 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:57.880 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:57.880 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:57.880 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:57.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.860+0000 7f1435329640 10 monclient: _renew_subs 2024-04-22T08:12:57.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.860+0000 7f1435329640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:57.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.860+0000 7f1435329640 1 -- 172.21.15.40:0/1077633950 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14301227a0 con 0x7f14300a1da0 2024-04-22T08:12:57.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.861+0000 7f142f7fe640 1 --2- 172.21.15.40:0/1077633950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14300a1da0 0x7f1430135550 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1420007e40 tx=0x7f142000e4b0 comp rx=0 tx=0).queue_received will delay after 1.71377e+09 on 0x7f1420013cb0 mgrmap(e 5) v1 2024-04-22T08:12:57.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.861+0000 7f14277fe640 1 -- 172.21.15.40:0/1077633950 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f1420013cb0 con 0x7f14300a1da0 2024-04-22T08:12:57.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.861+0000 7f1435329640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:57.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.861+0000 7f1435329640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:57.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.861+0000 7f1435329640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:57.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.861+0000 7f1435329640 1 -- 172.21.15.40:0/1077633950 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f143004e9b0 con 0x7f14300a1da0 2024-04-22T08:12:57.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.861+0000 7f14277fe640 1 --2- 172.21.15.40:0/1077633950 >> v2:172.21.15.189:6800/30168 conn(0x7f1410037820 0x7f1410039ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:57.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.862+0000 7f142effd640 1 --2- 172.21.15.40:0/1077633950 >> v2:172.21.15.189:6800/30168 conn(0x7f1410037820 0x7f1410039ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:57.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.862+0000 7f142effd640 10 monclient: get_auth_request con 0x7f1410037820 auth_method 0 2024-04-22T08:12:57.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.864+0000 7f142effd640 1 --2- 172.21.15.40:0/1077633950 >> v2:172.21.15.189:6800/30168 conn(0x7f1410037820 0x7f1410039ce0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f14300673f0 tx=0x7f1428008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:57.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:57.866+0000 7f14277fe640 1 -- 172.21.15.40:0/1077633950 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f142004ab50 con 0x7f14300a1da0 2024-04-22T08:12:58.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.689+0000 7f14277fe640 1 -- 172.21.15.40:0/1077633950 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f1420014420 con 0x7f14300a1da0 2024-04-22T08:12:58.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.689+0000 7f14277fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:58.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.689+0000 7f14277fe640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:12:58.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.871+0000 7f1435329640 1 -- 172.21.15.40:0/1077633950 --> v2:172.21.15.189:6800/30168 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f14300a0200 con 0x7f1410037820 2024-04-22T08:12:58.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.873+0000 7f14277fe640 1 -- 172.21.15.40:0/1077633950 <== mgr.4102 v2:172.21.15.189:6800/30168 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+40551 (secure 0 0 0) 0x7f14300a0200 con 0x7f1410037820 2024-04-22T08:12:58.874 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:12:58.874 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-22T08:12:58.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.875+0000 7f1435329640 1 -- 172.21.15.40:0/1077633950 >> v2:172.21.15.189:6800/30168 conn(0x7f1410037820 msgr2=0x7f1410039ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:58.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.875+0000 7f1435329640 1 --2- 172.21.15.40:0/1077633950 >> v2:172.21.15.189:6800/30168 conn(0x7f1410037820 0x7f1410039ce0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f14300673f0 tx=0x7f1428008030 comp rx=0 tx=0).stop 2024-04-22T08:12:58.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.875+0000 7f1435329640 10 monclient: shutdown 2024-04-22T08:12:58.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.875+0000 7f1435329640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:58.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.875+0000 7f1435329640 1 -- 172.21.15.40:0/1077633950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14300a1da0 msgr2=0x7f1430135550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:58.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.875+0000 7f1435329640 1 --2- 172.21.15.40:0/1077633950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14300a1da0 0x7f1430135550 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1420007e40 tx=0x7f142000e4b0 comp rx=0 tx=0).stop 2024-04-22T08:12:58.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.875+0000 7f1435329640 1 -- 172.21.15.40:0/1077633950 shutdown_connections 2024-04-22T08:12:58.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.875+0000 7f1435329640 1 --2- 172.21.15.40:0/1077633950 >> v2:172.21.15.189:6800/30168 conn(0x7f1410037820 0x7f1410039ce0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:58.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.875+0000 7f1435329640 1 --2- 172.21.15.40:0/1077633950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14300a1da0 0x7f1430135550 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:58.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.875+0000 7f1435329640 1 --2- 172.21.15.40:0/1077633950 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f14300a19c0 0x7f1430135010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:58.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.875+0000 7f1435329640 1 --2- 172.21.15.40:0/1077633950 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f143009ee10 0x7f14301ca8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:58.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.875+0000 7f1435329640 1 -- 172.21.15.40:0/1077633950 >> 172.21.15.40:0/1077633950 conn(0x7f14300a4e80 msgr2=0x7f14300a0990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:58.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.875+0000 7f1435329640 1 -- 172.21.15.40:0/1077633950 shutdown_connections 2024-04-22T08:12:58.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.875+0000 7f1435329640 1 -- 172.21.15.40:0/1077633950 wait complete. 2024-04-22T08:12:58.886 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-04-22T08:12:58.078344+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":8,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":164644,"kb_used_data":1744,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566066396,"statfs":{"total":579820584960,"available":579651989504,"internally_reserved":0,"allocated":1785856,"data_stored":1223245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9560,"internal_metadata":165927592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001341"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:12:52.005174+0000","last_change":"2024-04-22T08:12:51.443086+0000","last_active":"2024-04-22T08:12:52.005174+0000","last_peered":"2024-04-22T08:12:52.005174+0000","last_clean":"2024-04-22T08:12:52.005174+0000","last_became_active":"2024-04-22T08:12:51.442730+0000","last_became_peered":"2024-04-22T08:12:51.442730+0000","last_unstale":"2024-04-22T08:12:52.005174+0000","last_undegraded":"2024-04-22T08:12:52.005174+0000","last_fullsized":"2024-04-22T08:12:52.005174+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:06:13.481863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:12:52.009552+0000","last_change":"2024-04-22T08:12:50.841276+0000","last_active":"2024-04-22T08:12:52.009552+0000","last_peered":"2024-04-22T08:12:52.009552+0000","last_clean":"2024-04-22T08:12:52.009552+0000","last_became_active":"2024-04-22T08:12:50.841008+0000","last_became_peered":"2024-04-22T08:12:50.841008+0000","last_unstale":"2024-04-22T08:12:52.009552+0000","last_undegraded":"2024-04-22T08:12:52.009552+0000","last_fullsized":"2024-04-22T08:12:52.009552+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:47:00.224321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:12:52.005217+0000","last_change":"2024-04-22T08:12:51.403419+0000","last_active":"2024-04-22T08:12:52.005217+0000","last_peered":"2024-04-22T08:12:52.005217+0000","last_clean":"2024-04-22T08:12:52.005217+0000","last_became_active":"2024-04-22T08:12:51.403182+0000","last_became_peered":"2024-04-22T08:12:51.403182+0000","last_unstale":"2024-04-22T08:12:52.005217+0000","last_undegraded":"2024-04-22T08:12:52.005217+0000","last_fullsized":"2024-04-22T08:12:52.005217+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:52:39.730597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:12:52.009486+0000","last_change":"2024-04-22T08:12:51.403260+0000","last_active":"2024-04-22T08:12:52.009486+0000","last_peered":"2024-04-22T08:12:52.009486+0000","last_clean":"2024-04-22T08:12:52.009486+0000","last_became_active":"2024-04-22T08:12:51.402980+0000","last_became_peered":"2024-04-22T08:12:51.402980+0000","last_unstale":"2024-04-22T08:12:52.009486+0000","last_undegraded":"2024-04-22T08:12:52.009486+0000","last_fullsized":"2024-04-22T08:12:52.009486+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:08:04.625870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:12:52.041135+0000","last_change":"2024-04-22T08:12:50.836566+0000","last_active":"2024-04-22T08:12:52.041135+0000","last_peered":"2024-04-22T08:12:52.041135+0000","last_clean":"2024-04-22T08:12:52.041135+0000","last_became_active":"2024-04-22T08:12:50.836250+0000","last_became_peered":"2024-04-22T08:12:50.836250+0000","last_unstale":"2024-04-22T08:12:52.041135+0000","last_undegraded":"2024-04-22T08:12:52.041135+0000","last_fullsized":"2024-04-22T08:12:52.041135+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:42:13.693209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:12:52.042149+0000","last_change":"2024-04-22T08:12:50.836508+0000","last_active":"2024-04-22T08:12:52.042149+0000","last_peered":"2024-04-22T08:12:52.042149+0000","last_clean":"2024-04-22T08:12:52.042149+0000","last_became_active":"2024-04-22T08:12:50.836210+0000","last_became_peered":"2024-04-22T08:12:50.836210+0000","last_unstale":"2024-04-22T08:12:52.042149+0000","last_undegraded":"2024-04-22T08:12:52.042149+0000","last_fullsized":"2024-04-22T08:12:52.042149+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:24:53.345661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:12:52.005162+0000","last_change":"2024-04-22T08:12:50.842633+0000","last_active":"2024-04-22T08:12:52.005162+0000","last_peered":"2024-04-22T08:12:52.005162+0000","last_clean":"2024-04-22T08:12:52.005162+0000","last_became_active":"2024-04-22T08:12:50.842392+0000","last_became_peered":"2024-04-22T08:12:50.842392+0000","last_unstale":"2024-04-22T08:12:52.005162+0000","last_undegraded":"2024-04-22T08:12:52.005162+0000","last_fullsized":"2024-04-22T08:12:52.005162+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:53:14.717539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:12:52.041139+0000","last_change":"2024-04-22T08:12:50.835820+0000","last_active":"2024-04-22T08:12:52.041139+0000","last_peered":"2024-04-22T08:12:52.041139+0000","last_clean":"2024-04-22T08:12:52.041139+0000","last_became_active":"2024-04-22T08:12:50.835541+0000","last_became_peered":"2024-04-22T08:12:50.835541+0000","last_unstale":"2024-04-22T08:12:52.041139+0000","last_undegraded":"2024-04-22T08:12:52.041139+0000","last_fullsized":"2024-04-22T08:12:52.041139+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:52:21.798509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":160,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T08:12:53.045764+0000","last_change":"2024-04-22T08:12:51.404305+0000","last_active":"2024-04-22T08:12:53.045764+0000","last_peered":"2024-04-22T08:12:53.045764+0000","last_clean":"2024-04-22T08:12:53.045764+0000","last_became_active":"2024-04-22T08:12:51.403208+0000","last_became_peered":"2024-04-22T08:12:51.403208+0000","last_unstale":"2024-04-22T08:12:53.045764+0000","last_undegraded":"2024-04-22T08:12:53.045764+0000","last_fullsized":"2024-04-22T08:12:53.045764+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:04:43.253802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640262,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27152,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":131072,"data_stored":37965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.248}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]}]},{"osd":0,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28112,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343728,"statfs":{"total":96636764160,"available":96607977472,"internally_reserved":0,"allocated":663552,"data_stored":566893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]}]},{"osd":5,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27152,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":131072,"data_stored":37965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.528}]}]},{"osd":4,"up_from":11,"seq":47244640261,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27152,"kb_used_data":124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":126976,"data_stored":37004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.313}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]}]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27924,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343916,"statfs":{"total":96636764160,"available":96608169984,"internally_reserved":0,"allocated":602112,"data_stored":505453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.877}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98399999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27152,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":131072,"data_stored":37965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2170000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.093}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:12:58.887 INFO:tasks.mon_thrash.ceph_manager:clean! 2024-04-22T08:12:58.887 DEBUG:tasks.mon_thrash:Yielding 2024-04-22T08:12:58.888 INFO:teuthology.run_tasks:Running task exec... 2024-04-22T08:12:58.897 INFO:teuthology.task.exec:Executing custom commands... 2024-04-22T08:12:58.897 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi189.front.sepia.ceph.com 2024-04-22T08:12:58.897 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:12:58.897 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2024-04-22T08:12:58.898 INFO:tasks.mon_thrash.mon_thrasher:seed: 1713773578, revive delay: 20.0, thrash delay: 1.0 thrash many: False, maintain quorum: True store thrash: False, probability: 50 freeze mon: prob 10.0 duration 15.0 2024-04-22T08:12:58.898 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-22T08:12:58.898 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:12:58.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.997+0000 7f572e934640 10 monclient: get_monmap_and_config 2024-04-22T08:12:58.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.997+0000 7f572e934640 10 monclient: build_initial_monmap 2024-04-22T08:12:58.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.997+0000 7f572e934640 10 monclient: monmap: 2024-04-22T08:12:58.997 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:58.997 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:58.998 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:58.997913+0000 2024-04-22T08:12:58.998 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:58.997913+0000 2024-04-22T08:12:58.998 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:58.998 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:58.998 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:12:58.998 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:12:58.998 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:12:58.998 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:12:58.999 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:12:58.999 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:58.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.997+0000 7f572e934640 1 Processor -- start 2024-04-22T08:12:58.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.997+0000 7f572e934640 1 -- start start 2024-04-22T08:12:58.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.997+0000 7f572e934640 10 monclient: init 2024-04-22T08:12:59.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.998+0000 7f572e934640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:59.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.998+0000 7f572e934640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-04-22T08:12:59.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.998+0000 7f572e934640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f572809ee10 addr v1:172.21.15.189:6789/0 2024-04-22T08:12:59.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.998+0000 7f572e934640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f57280a1f30 addr v1:172.21.15.40:6789/0 2024-04-22T08:12:59.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.998+0000 7f572e934640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f57280a3240 0x7f572813e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:59.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.998+0000 7f572e934640 10 monclient(hunting): picked mon.noname-b con 0x7f57280a3240 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:59.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.998+0000 7f572e934640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57280a4470 con 0x7f57280a1f30 2024-04-22T08:12:59.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.998+0000 7f572e934640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f572809f1f0 con 0x7f572809ee10 2024-04-22T08:12:59.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.998+0000 7f572e934640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:59.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.998+0000 7f572e934640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57280a3660 con 0x7f57280a3240 2024-04-22T08:12:59.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.998+0000 7f572e934640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:59.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.998+0000 7f572e934640 10 monclient(hunting): authenticate will time out at 852.992169s 2024-04-22T08:12:59.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.998+0000 7f572ced1640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f57280a3240 0x7f572813e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:59.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.998+0000 7f57277fe640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f57280a1f30 0x7f572813b0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:58066/0 (socket says 172.21.15.40:58066) 2024-04-22T08:12:59.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.998+0000 7f57277fe640 1 -- 172.21.15.40:0/3068298560 learned_addr learned my addr 172.21.15.40:0/3068298560 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:12:59.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.998+0000 7f572ced1640 10 monclient(hunting): get_auth_request con 0x7f57280a3240 auth_method 0 2024-04-22T08:12:59.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.998+0000 7f572ced1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:59.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.998+0000 7f572ced1640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:59.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.998+0000 7f572ced1640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:59.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.998+0000 7f572ced1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:59.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.998+0000 7f572ced1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:59.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.998+0000 7f572ced1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:59.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.998+0000 7f572ced1640 10 monclient(hunting): handle_auth_done global_id 4349 payload 274 2024-04-22T08:12:59.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.998+0000 7f572ced1640 1 -- 172.21.15.40:0/3068298560 >> v1:172.21.15.189:6789/0 conn(0x7f572809ee10 legacy=0x7f57280a1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:59.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.998+0000 7f572ced1640 1 -- 172.21.15.40:0/3068298560 >> v1:172.21.15.40:6789/0 conn(0x7f57280a1f30 legacy=0x7f572813b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:59.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.998+0000 7f572ced1640 10 monclient: _finish_hunting 0 2024-04-22T08:12:59.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.998+0000 7f572ced1640 1 monclient: found mon.noname-b 2024-04-22T08:12:59.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.998+0000 7f572ced1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:59.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.998+0000 7f572ced1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:12:59.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:58.998+0000 7f572ced1640 1 -- 172.21.15.40:0/3068298560 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f572809fd30 con 0x7f57280a3240 2024-04-22T08:12:59.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.001+0000 7f572ced1640 1 -- 172.21.15.40:0/3068298560 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f57280a3240 msgr2=0x7f572813e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:59.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.001+0000 7f572ced1640 1 --2- 172.21.15.40:0/3068298560 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f57280a3240 0x7f572813e810 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5720009dc0 tx=0x7f572002bce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cfde1f80c05fb8c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:59.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.001+0000 7f5726ffd640 1 -- 172.21.15.40:0/3068298560 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f572002cd80 con 0x7f57280a3240 2024-04-22T08:12:59.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.001+0000 7f5726ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:59.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.001+0000 7f5726ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:12:59.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.001+0000 7f5726ffd640 10 monclient: dump: 2024-04-22T08:12:59.012 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:59.012 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:59.012 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:59.012 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:59.012 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:59.012 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:59.012 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:59.013 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:59.013 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:59.013 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:59.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.001+0000 7f5726ffd640 10 monclient: _finish_auth 0 2024-04-22T08:12:59.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.001+0000 7f5726ffd640 10 monclient: _check_auth_tickets 2024-04-22T08:12:59.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.001+0000 7f5726ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:59.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.002+0000 7f5726ffd640 1 -- 172.21.15.40:0/3068298560 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5720034440 con 0x7f57280a3240 2024-04-22T08:12:59.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.002+0000 7f5726ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:59.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.002+0000 7f5726ffd640 1 -- 172.21.15.40:0/3068298560 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57200347d0 con 0x7f57280a3240 2024-04-22T08:12:59.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.002+0000 7f572e934640 5 monclient: authenticate success, global_id 4349 2024-04-22T08:12:59.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.002+0000 7f572e934640 10 monclient: get_monmap_and_config success 2024-04-22T08:12:59.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.002+0000 7f572e934640 4 set_mon_vals no callback set 2024-04-22T08:12:59.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.002+0000 7f5726ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:59.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.002+0000 7f5726ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:12:59.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.002+0000 7f5726ffd640 10 monclient: dump: 2024-04-22T08:12:59.014 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:59.014 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:59.014 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:59.014 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:59.014 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:59.014 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:59.014 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:59.014 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:59.014 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:59.014 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:59.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.002+0000 7f572e934640 10 monclient: shutdown 2024-04-22T08:12:59.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.002+0000 7f572e934640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:59.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.002+0000 7f572e934640 1 -- 172.21.15.40:0/3068298560 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f57280a3240 msgr2=0x7f572813e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:59.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.002+0000 7f572e934640 1 --2- 172.21.15.40:0/3068298560 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f57280a3240 0x7f572813e810 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5720009dc0 tx=0x7f572002bce0 comp rx=0 tx=0).stop 2024-04-22T08:12:59.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.002+0000 7f572e934640 1 -- 172.21.15.40:0/3068298560 shutdown_connections 2024-04-22T08:12:59.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.002+0000 7f572e934640 1 --2- 172.21.15.40:0/3068298560 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f57280a3240 0x7f572813e810 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:59.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.002+0000 7f572e934640 1 -- 172.21.15.40:0/3068298560 >> 172.21.15.40:0/3068298560 conn(0x7f57280a4e80 msgr2=0x7f57280a5290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:59.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.002+0000 7f572e934640 1 -- 172.21.15.40:0/3068298560 shutdown_connections 2024-04-22T08:12:59.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.002+0000 7f572e934640 1 -- 172.21.15.40:0/3068298560 wait complete. 2024-04-22T08:12:59.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.003+0000 7f572e934640 10 monclient: build_initial_monmap 2024-04-22T08:12:59.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.003+0000 7f572e934640 10 monclient: monmap: 2024-04-22T08:12:59.015 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:59.015 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:12:59.015 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:12:59.015 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:12:59.016 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:59.016 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:59.016 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:12:59.016 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:12:59.016 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:12:59.016 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:59.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.003+0000 7f572e934640 1 Processor -- start 2024-04-22T08:12:59.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.003+0000 7f572e934640 1 -- start start 2024-04-22T08:12:59.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.003+0000 7f572e934640 10 monclient: init 2024-04-22T08:12:59.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.003+0000 7f572e934640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:59.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.003+0000 7f572e934640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-22T08:12:59.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f572e934640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f572809ee10 0x7f57281cadd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:59.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f572e934640 10 monclient(hunting): picked mon.noname-b con 0x7f572809ee10 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:59.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f572e934640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f57280a1f30 0x7f57281cb310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:59.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f572e934640 10 monclient(hunting): picked mon.noname-c con 0x7f57280a1f30 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:59.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f572e934640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57280a3240 0x7f572806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:59.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f572e934640 10 monclient(hunting): picked mon.noname-a con 0x7f57280a3240 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:59.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f572e934640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:59.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f572e934640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f572813e290 con 0x7f57280a3240 2024-04-22T08:12:59.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f572e934640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:59.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f572e934640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f572811f540 con 0x7f572809ee10 2024-04-22T08:12:59.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f572e934640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:59.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f572e934640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5728120200 con 0x7f57280a1f30 2024-04-22T08:12:59.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f572e934640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:59.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f572e934640 10 monclient(hunting): authenticate will time out at 852.997851s 2024-04-22T08:12:59.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f57277fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f57280a1f30 0x7f57281cb310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:59.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f572ced1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57280a3240 0x7f572806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:59.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f57277fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f57280a1f30 0x7f57281cb310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:51150/0 (socket says 172.21.15.40:51150) 2024-04-22T08:12:59.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f572ced1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57280a3240 0x7f572806ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40648/0 (socket says 172.21.15.40:40648) 2024-04-22T08:12:59.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f57277fe640 1 -- 172.21.15.40:0/3142419843 learned_addr learned my addr 172.21.15.40:0/3142419843 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:59.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f57277fe640 10 monclient(hunting): get_auth_request con 0x7f57280a1f30 auth_method 0 2024-04-22T08:12:59.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f57277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:59.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f57277fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:59.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f57277fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:59.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f572ced1640 10 monclient(hunting): get_auth_request con 0x7f57280a3240 auth_method 0 2024-04-22T08:12:59.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f572ced1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:59.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f572ced1640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:59.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f572ced1640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:59.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f57277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:59.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f57277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:59.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f57277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:59.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f5727fff640 1 --2- 172.21.15.40:0/3142419843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f572809ee10 0x7f57281cadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:59.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f572ced1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:59.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f572ced1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:59.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f572ced1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:59.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f5727fff640 10 monclient(hunting): get_auth_request con 0x7f572809ee10 auth_method 0 2024-04-22T08:12:59.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f5727fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:59.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f5727fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:59.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f5727fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:59.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f5727fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:59.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f5727fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:59.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f5727fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:59.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f57277fe640 10 monclient(hunting): handle_auth_done global_id 4352 payload 824 2024-04-22T08:12:59.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f57277fe640 1 -- 172.21.15.40:0/3142419843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f572809ee10 msgr2=0x7f57281cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:59.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f57277fe640 1 --2- 172.21.15.40:0/3142419843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f572809ee10 0x7f57281cadd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:59.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f57277fe640 1 -- 172.21.15.40:0/3142419843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57280a3240 msgr2=0x7f572806ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:59.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f57277fe640 1 --2- 172.21.15.40:0/3142419843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57280a3240 0x7f572806ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:59.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f57277fe640 10 monclient: _finish_hunting 0 2024-04-22T08:12:59.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f57277fe640 1 monclient: found mon.noname-c 2024-04-22T08:12:59.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f57277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:59.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f57277fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:12:59.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.004+0000 7f57277fe640 1 -- 172.21.15.40:0/3142419843 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5720009a70 con 0x7f57280a1f30 2024-04-22T08:12:59.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.006+0000 7f57277fe640 1 -- 172.21.15.40:0/3142419843 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f57280a1f30 msgr2=0x7f57281cb310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:59.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.006+0000 7f57277fe640 1 --2- 172.21.15.40:0/3142419843 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f57280a1f30 0x7f57281cb310 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f571c00c490 tx=0x7f571c00c950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:59.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.006+0000 7f5724ff9640 1 -- 172.21.15.40:0/3142419843 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f571c007070 con 0x7f57280a1f30 2024-04-22T08:12:59.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.006+0000 7f5724ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:59.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.006+0000 7f5724ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:12:59.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.006+0000 7f5724ff9640 10 monclient: dump: 2024-04-22T08:12:59.021 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:59.021 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:59.021 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:59.021 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:59.021 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:59.021 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:59.021 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:59.021 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:59.021 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:59.021 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:59.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.006+0000 7f5724ff9640 10 monclient: _finish_auth 0 2024-04-22T08:12:59.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.006+0000 7f5724ff9640 10 monclient: _check_auth_tickets 2024-04-22T08:12:59.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.006+0000 7f5724ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:59.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.006+0000 7f5724ff9640 1 -- 172.21.15.40:0/3142419843 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f571c0071f0 con 0x7f57280a1f30 2024-04-22T08:12:59.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.006+0000 7f572e934640 5 monclient: authenticate success, global_id 4352 2024-04-22T08:12:59.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.006+0000 7f572e934640 10 monclient: _renew_subs 2024-04-22T08:12:59.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.006+0000 7f572e934640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:59.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.006+0000 7f572e934640 1 -- 172.21.15.40:0/3142419843 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f572811ef80 con 0x7f57280a1f30 2024-04-22T08:12:59.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.006+0000 7f5724ff9640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:59.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.007+0000 7f5724ff9640 1 -- 172.21.15.40:0/3142419843 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f571c0075a0 con 0x7f57280a1f30 2024-04-22T08:12:59.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.007+0000 7f5724ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:59.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.007+0000 7f5724ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:12:59.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.007+0000 7f5724ff9640 10 monclient: dump: 2024-04-22T08:12:59.022 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:59.022 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:59.023 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:59.023 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:59.023 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:59.023 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:59.023 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:59.023 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:59.023 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:59.023 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:59.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.007+0000 7f572e934640 10 monclient: _renew_subs 2024-04-22T08:12:59.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.007+0000 7f57267fc640 4 set_mon_vals no callback set 2024-04-22T08:12:59.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.007+0000 7f572e934640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:59.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.007+0000 7f572e934640 1 -- 172.21.15.40:0/3142419843 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5728070b70 con 0x7f57280a1f30 2024-04-22T08:12:59.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.007+0000 7f5724ff9640 1 -- 172.21.15.40:0/3142419843 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f571c01b400 con 0x7f57280a1f30 2024-04-22T08:12:59.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.007+0000 7f572e934640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:59.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.007+0000 7f572e934640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:59.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.007+0000 7f572e934640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:59.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.007+0000 7f572e934640 1 -- 172.21.15.40:0/3142419843 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57280a3aa0 con 0x7f57280a1f30 2024-04-22T08:12:59.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.008+0000 7f5724ff9640 1 --2- 172.21.15.40:0/3142419843 >> v2:172.21.15.189:6800/30168 conn(0x7f57140377d0 0x7f5714039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:59.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.008+0000 7f5724ff9640 1 -- 172.21.15.40:0/3142419843 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f571c049de0 con 0x7f57280a1f30 2024-04-22T08:12:59.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.008+0000 7f5727fff640 1 --2- 172.21.15.40:0/3142419843 >> v2:172.21.15.189:6800/30168 conn(0x7f57140377d0 0x7f5714039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:59.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.008+0000 7f5727fff640 10 monclient: get_auth_request con 0x7f57140377d0 auth_method 0 2024-04-22T08:12:59.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.011+0000 7f5727fff640 1 --2- 172.21.15.40:0/3142419843 >> v2:172.21.15.189:6800/30168 conn(0x7f57140377d0 0x7f5714039c90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5718004f60 tx=0x7f5718007e30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:59.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.012+0000 7f5724ff9640 1 -- 172.21.15.40:0/3142419843 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f571c007740 con 0x7f57280a1f30 2024-04-22T08:12:59.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.012+0000 7f5724ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:59.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.012+0000 7f5724ff9640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:12:59.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.264+0000 7f572e934640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:12:59.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.264+0000 7f572e934640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:12:59.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.264+0000 7f572e934640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:59.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.264+0000 7f572e934640 1 -- 172.21.15.40:0/3142419843 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f572804e9b0 con 0x7f57280a1f30 2024-04-22T08:12:59.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.264+0000 7f5724ff9640 1 -- 172.21.15.40:0/3142419843 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f571c010300 con 0x7f57280a1f30 2024-04-22T08:12:59.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.264+0000 7f5724ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:12:59.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.264+0000 7f5724ff9640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:12:59.264 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":23,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:12:59.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.266+0000 7f572e934640 1 -- 172.21.15.40:0/3142419843 >> v2:172.21.15.189:6800/30168 conn(0x7f57140377d0 msgr2=0x7f5714039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:59.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.266+0000 7f572e934640 1 --2- 172.21.15.40:0/3142419843 >> v2:172.21.15.189:6800/30168 conn(0x7f57140377d0 0x7f5714039c90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5718004f60 tx=0x7f5718007e30 comp rx=0 tx=0).stop 2024-04-22T08:12:59.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.266+0000 7f572e934640 10 monclient: shutdown 2024-04-22T08:12:59.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.266+0000 7f572e934640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:59.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.266+0000 7f572e934640 1 -- 172.21.15.40:0/3142419843 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f57280a1f30 msgr2=0x7f57281cb310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:59.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.266+0000 7f572e934640 1 --2- 172.21.15.40:0/3142419843 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f57280a1f30 0x7f57281cb310 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f571c00c490 tx=0x7f571c00c950 comp rx=0 tx=0).stop 2024-04-22T08:12:59.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.266+0000 7f572e934640 1 -- 172.21.15.40:0/3142419843 shutdown_connections 2024-04-22T08:12:59.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.266+0000 7f572e934640 1 --2- 172.21.15.40:0/3142419843 >> v2:172.21.15.189:6800/30168 conn(0x7f57140377d0 0x7f5714039c90 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:59.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.266+0000 7f572e934640 1 --2- 172.21.15.40:0/3142419843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57280a3240 0x7f572806ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:59.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.266+0000 7f572e934640 1 --2- 172.21.15.40:0/3142419843 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f57280a1f30 0x7f57281cb310 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:59.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.266+0000 7f572e934640 1 --2- 172.21.15.40:0/3142419843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f572809ee10 0x7f57281cadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:59.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.266+0000 7f572e934640 1 -- 172.21.15.40:0/3142419843 >> 172.21.15.40:0/3142419843 conn(0x7f57280a4e80 msgr2=0x7f57280a6280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:59.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.266+0000 7f572e934640 1 -- 172.21.15.40:0/3142419843 shutdown_connections 2024-04-22T08:12:59.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.266+0000 7f572e934640 1 -- 172.21.15.40:0/3142419843 wait complete. 2024-04-22T08:12:59.276 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-22T08:12:59.277 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-22T08:12:59.277 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-22T08:12:59.277 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-22T08:12:59.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.377+0000 7fb78ff75640 10 monclient: get_monmap_and_config 2024-04-22T08:12:59.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.377+0000 7fb78ff75640 10 monclient: build_initial_monmap 2024-04-22T08:12:59.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.377+0000 7fb78ff75640 10 monclient: monmap: 2024-04-22T08:12:59.377 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:59.377 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:59.377 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:59.378116+0000 2024-04-22T08:12:59.377 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:59.378116+0000 2024-04-22T08:12:59.377 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:59.377 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:59.377 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:12:59.377 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:12:59.377 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:12:59.378 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:12:59.378 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:12:59.378 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:59.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.377+0000 7fb78ff75640 1 Processor -- start 2024-04-22T08:12:59.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.378+0000 7fb78ff75640 1 -- start start 2024-04-22T08:12:59.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.378+0000 7fb78ff75640 10 monclient: init 2024-04-22T08:12:59.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.378+0000 7fb78ff75640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:59.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.378+0000 7fb78ff75640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-04-22T08:12:59.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.378+0000 7fb78ff75640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb7880a1a80 addr v1:172.21.15.189:6789/0 2024-04-22T08:12:59.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.378+0000 7fb78ff75640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb7880a2570 0x7fb7880a01b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:59.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.378+0000 7fb78ff75640 10 monclient(hunting): picked mon.noname-c con 0x7fb7880a2570 addr v2:172.21.15.189:3300/0 2024-04-22T08:12:59.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.378+0000 7fb78ff75640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb78813b0e0 addr v1:172.21.15.40:6789/0 2024-04-22T08:12:59.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.378+0000 7fb78ff75640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb78809d7f0 con 0x7fb78813b0e0 2024-04-22T08:12:59.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.378+0000 7fb78ff75640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb78809dc30 con 0x7fb7880a1a80 2024-04-22T08:12:59.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.378+0000 7fb78ff75640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:59.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.378+0000 7fb78ff75640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb78809f1c0 con 0x7fb7880a2570 2024-04-22T08:12:59.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.378+0000 7fb78ff75640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:59.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.378+0000 7fb78ff75640 10 monclient(hunting): authenticate will time out at 853.372475s 2024-04-22T08:12:59.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.378+0000 7fb78e512640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fb78813b0e0 0x7fb78813d4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:58078/0 (socket says 172.21.15.40:58078) 2024-04-22T08:12:59.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.378+0000 7fb78e512640 1 -- 172.21.15.40:0/3015663186 learned_addr learned my addr 172.21.15.40:0/3015663186 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:12:59.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.379+0000 7fb78d510640 1 --2- 172.21.15.40:0/3015663186 >> v2:172.21.15.189:3300/0 conn(0x7fb7880a2570 0x7fb7880a01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:59.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.379+0000 7fb78d510640 10 monclient(hunting): get_auth_request con 0x7fb7880a2570 auth_method 0 2024-04-22T08:12:59.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.379+0000 7fb78d510640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:59.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.379+0000 7fb78d510640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:59.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.379+0000 7fb78d510640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:59.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.379+0000 7fb78d510640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:59.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.379+0000 7fb78d510640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:59.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.379+0000 7fb78d510640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:59.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.379+0000 7fb78d510640 10 monclient(hunting): handle_auth_done global_id 4330 payload 274 2024-04-22T08:12:59.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.379+0000 7fb78d510640 1 -- 172.21.15.40:0/3015663186 >> v1:172.21.15.189:6789/0 conn(0x7fb7880a1a80 legacy=0x7fb7880a1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:59.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.379+0000 7fb78d510640 1 -- 172.21.15.40:0/3015663186 >> v1:172.21.15.40:6789/0 conn(0x7fb78813b0e0 legacy=0x7fb78813d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:59.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.379+0000 7fb78d510640 10 monclient: _finish_hunting 0 2024-04-22T08:12:59.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.379+0000 7fb78d510640 1 monclient: found mon.noname-c 2024-04-22T08:12:59.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.379+0000 7fb78d510640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:59.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.379+0000 7fb78d510640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.189:3300/0 2024-04-22T08:12:59.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.379+0000 7fb78d510640 1 -- 172.21.15.40:0/3015663186 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb78809eee0 con 0x7fb7880a2570 2024-04-22T08:12:59.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.382+0000 7fb78d510640 1 -- 172.21.15.40:0/3015663186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7880a2570 msgr2=0x7fb7880a01b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:59.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.382+0000 7fb78d510640 1 --2- 172.21.15.40:0/3015663186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7880a2570 0x7fb7880a01b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb7780097a0 tx=0x7fb77802b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1cfebe1fbcedb4a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:59.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.382+0000 7fb78cd0f640 1 -- 172.21.15.40:0/3015663186 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb778030a00 con 0x7fb7880a2570 2024-04-22T08:12:59.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.382+0000 7fb78cd0f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:59.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.382+0000 7fb78cd0f640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:12:59.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.382+0000 7fb78cd0f640 10 monclient: dump: 2024-04-22T08:12:59.386 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:59.386 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:59.386 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:59.387 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:59.387 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:59.387 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:59.387 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:59.387 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:59.387 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:59.387 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:59.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.382+0000 7fb78cd0f640 10 monclient: _finish_auth 0 2024-04-22T08:12:59.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.382+0000 7fb78cd0f640 10 monclient: _check_auth_tickets 2024-04-22T08:12:59.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.382+0000 7fb78cd0f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:59.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.382+0000 7fb78cd0f640 1 -- 172.21.15.40:0/3015663186 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb77802e050 con 0x7fb7880a2570 2024-04-22T08:12:59.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.382+0000 7fb78cd0f640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:59.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.382+0000 7fb78cd0f640 1 -- 172.21.15.40:0/3015663186 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb778030d80 con 0x7fb7880a2570 2024-04-22T08:12:59.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.382+0000 7fb78ff75640 5 monclient: authenticate success, global_id 4330 2024-04-22T08:12:59.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.382+0000 7fb78ff75640 10 monclient: get_monmap_and_config success 2024-04-22T08:12:59.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.382+0000 7fb78ff75640 4 set_mon_vals no callback set 2024-04-22T08:12:59.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.382+0000 7fb78cd0f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:59.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.382+0000 7fb78cd0f640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:12:59.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.382+0000 7fb78cd0f640 10 monclient: dump: 2024-04-22T08:12:59.388 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:59.388 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:59.388 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:59.388 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:59.388 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:59.388 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:59.388 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:59.389 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:59.389 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:59.389 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:59.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.383+0000 7fb78ff75640 10 monclient: shutdown 2024-04-22T08:12:59.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.383+0000 7fb78ff75640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:59.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.383+0000 7fb78ff75640 1 -- 172.21.15.40:0/3015663186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7880a2570 msgr2=0x7fb7880a01b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:59.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.383+0000 7fb78ff75640 1 --2- 172.21.15.40:0/3015663186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7880a2570 0x7fb7880a01b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb7780097a0 tx=0x7fb77802b710 comp rx=0 tx=0).stop 2024-04-22T08:12:59.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.383+0000 7fb78ff75640 1 -- 172.21.15.40:0/3015663186 shutdown_connections 2024-04-22T08:12:59.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.383+0000 7fb78ff75640 1 --2- 172.21.15.40:0/3015663186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7880a2570 0x7fb7880a01b0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:59.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.383+0000 7fb78ff75640 1 -- 172.21.15.40:0/3015663186 >> 172.21.15.40:0/3015663186 conn(0x7fb7880a66a0 msgr2=0x7fb7880a2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:59.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.383+0000 7fb78ff75640 1 -- 172.21.15.40:0/3015663186 shutdown_connections 2024-04-22T08:12:59.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.383+0000 7fb78ff75640 1 -- 172.21.15.40:0/3015663186 wait complete. 2024-04-22T08:12:59.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.383+0000 7fb78ff75640 10 monclient: build_initial_monmap 2024-04-22T08:12:59.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.383+0000 7fb78ff75640 10 monclient: monmap: 2024-04-22T08:12:59.390 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:59.390 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:12:59.390 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:12:59.390 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:12:59.390 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:59.390 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:59.390 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:12:59.390 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:12:59.390 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:12:59.390 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:59.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.384+0000 7fb78ff75640 1 Processor -- start 2024-04-22T08:12:59.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.384+0000 7fb78ff75640 1 -- start start 2024-04-22T08:12:59.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.384+0000 7fb78ff75640 10 monclient: init 2024-04-22T08:12:59.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.384+0000 7fb78ff75640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:59.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.384+0000 7fb78ff75640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-22T08:12:59.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.384+0000 7fb78ff75640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb7880a1a80 0x7fb7880706f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:59.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.384+0000 7fb78ff75640 10 monclient(hunting): picked mon.noname-c con 0x7fb7880a1a80 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:59.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.384+0000 7fb78ff75640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7880a2570 0x7fb7881cac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:59.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.384+0000 7fb78ff75640 10 monclient(hunting): picked mon.noname-b con 0x7fb7880a2570 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:59.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.384+0000 7fb78ff75640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb78813b0e0 0x7fb7881cb150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:59.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.384+0000 7fb78ff75640 10 monclient(hunting): picked mon.noname-a con 0x7fb78813b0e0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:59.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.384+0000 7fb78ff75640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:59.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.384+0000 7fb78ff75640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb788122bd0 con 0x7fb78813b0e0 2024-04-22T08:12:59.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.384+0000 7fb78ff75640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:59.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.384+0000 7fb78ff75640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7880718b0 con 0x7fb7880a2570 2024-04-22T08:12:59.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.384+0000 7fb78ff75640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:59.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.384+0000 7fb78ff75640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb788121df0 con 0x7fb7880a1a80 2024-04-22T08:12:59.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.384+0000 7fb78ff75640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:59.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.384+0000 7fb78ff75640 10 monclient(hunting): authenticate will time out at 853.378637s 2024-04-22T08:12:59.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.384+0000 7fb78dd11640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb7880a1a80 0x7fb7880706f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:59.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.384+0000 7fb78e512640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb78813b0e0 0x7fb7881cb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:59.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.384+0000 7fb78dd11640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb7880a1a80 0x7fb7880706f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:51162/0 (socket says 172.21.15.40:51162) 2024-04-22T08:12:59.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.384+0000 7fb78dd11640 1 -- 172.21.15.40:0/3721969691 learned_addr learned my addr 172.21.15.40:0/3721969691 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:59.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.384+0000 7fb78dd11640 10 monclient(hunting): get_auth_request con 0x7fb7880a1a80 auth_method 0 2024-04-22T08:12:59.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.384+0000 7fb78dd11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:59.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.384+0000 7fb78dd11640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:59.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.384+0000 7fb78dd11640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:59.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.385+0000 7fb78e512640 10 monclient(hunting): get_auth_request con 0x7fb78813b0e0 auth_method 0 2024-04-22T08:12:59.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.385+0000 7fb78e512640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:59.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.385+0000 7fb78e512640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:59.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.385+0000 7fb78e512640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:59.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.385+0000 7fb78dd11640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:59.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.385+0000 7fb78dd11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:59.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.385+0000 7fb78dd11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:59.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.385+0000 7fb78e512640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:59.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.385+0000 7fb78e512640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:59.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.385+0000 7fb78e512640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:59.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.385+0000 7fb78d510640 1 --2- 172.21.15.40:0/3721969691 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7880a2570 0x7fb7881cac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:59.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.385+0000 7fb78d510640 10 monclient(hunting): get_auth_request con 0x7fb7880a2570 auth_method 0 2024-04-22T08:12:59.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.385+0000 7fb78d510640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:59.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.385+0000 7fb78d510640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:59.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.385+0000 7fb78d510640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:59.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.385+0000 7fb78dd11640 10 monclient(hunting): handle_auth_done global_id 4355 payload 824 2024-04-22T08:12:59.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.385+0000 7fb78dd11640 1 -- 172.21.15.40:0/3721969691 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7880a2570 msgr2=0x7fb7881cac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:59.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.385+0000 7fb78dd11640 1 --2- 172.21.15.40:0/3721969691 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7880a2570 0x7fb7881cac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:59.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.385+0000 7fb78dd11640 1 -- 172.21.15.40:0/3721969691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb78813b0e0 msgr2=0x7fb7881cb150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:59.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.385+0000 7fb78dd11640 1 --2- 172.21.15.40:0/3721969691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb78813b0e0 0x7fb7881cb150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:59.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.385+0000 7fb78dd11640 10 monclient: _finish_hunting 0 2024-04-22T08:12:59.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.385+0000 7fb78dd11640 1 monclient: found mon.noname-c 2024-04-22T08:12:59.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.385+0000 7fb78dd11640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:59.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.385+0000 7fb78dd11640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:12:59.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.385+0000 7fb78dd11640 1 -- 172.21.15.40:0/3721969691 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb778009400 con 0x7fb7880a1a80 2024-04-22T08:12:59.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.385+0000 7fb78e512640 1 --2- 172.21.15.40:0/3721969691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb78813b0e0 0x7fb7881cb150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:12:59.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.387+0000 7fb78dd11640 1 -- 172.21.15.40:0/3721969691 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb7880a1a80 msgr2=0x7fb7880706f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:59.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.387+0000 7fb78dd11640 1 --2- 172.21.15.40:0/3721969691 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb7880a1a80 0x7fb7880706f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb78400b420 tx=0x7fb78400b8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:59.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.387+0000 7fb77e7fc640 1 -- 172.21.15.40:0/3721969691 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb784011a40 con 0x7fb7880a1a80 2024-04-22T08:12:59.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.387+0000 7fb77e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:59.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.387+0000 7fb77e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:12:59.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.387+0000 7fb77e7fc640 10 monclient: dump: 2024-04-22T08:12:59.399 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:59.399 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:59.399 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:59.399 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:59.399 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:59.399 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:59.399 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:59.399 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:59.399 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:59.399 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:59.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.387+0000 7fb77e7fc640 10 monclient: _finish_auth 0 2024-04-22T08:12:59.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.387+0000 7fb77e7fc640 10 monclient: _check_auth_tickets 2024-04-22T08:12:59.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.387+0000 7fb77e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:59.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.387+0000 7fb78ff75640 5 monclient: authenticate success, global_id 4355 2024-04-22T08:12:59.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.387+0000 7fb77e7fc640 1 -- 172.21.15.40:0/3721969691 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb784011ba0 con 0x7fb7880a1a80 2024-04-22T08:12:59.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.387+0000 7fb78ff75640 10 monclient: _renew_subs 2024-04-22T08:12:59.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.387+0000 7fb78ff75640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:59.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.387+0000 7fb78ff75640 1 -- 172.21.15.40:0/3721969691 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb78809cc20 con 0x7fb7880a1a80 2024-04-22T08:12:59.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.387+0000 7fb77e7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:59.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.387+0000 7fb77e7fc640 1 -- 172.21.15.40:0/3721969691 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb784010380 con 0x7fb7880a1a80 2024-04-22T08:12:59.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.387+0000 7fb77e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:59.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.387+0000 7fb77e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:12:59.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.387+0000 7fb77e7fc640 10 monclient: dump: 2024-04-22T08:12:59.401 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:59.401 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:59.401 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:59.401 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:59.401 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:59.401 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:59.401 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:59.401 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:59.401 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:59.401 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:59.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.387+0000 7fb77ffff640 4 set_mon_vals no callback set 2024-04-22T08:12:59.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.387+0000 7fb78ff75640 10 monclient: _renew_subs 2024-04-22T08:12:59.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.387+0000 7fb78ff75640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:59.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.387+0000 7fb78ff75640 1 -- 172.21.15.40:0/3721969691 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb788123470 con 0x7fb7880a1a80 2024-04-22T08:12:59.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.388+0000 7fb77e7fc640 1 -- 172.21.15.40:0/3721969691 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fb784010560 con 0x7fb7880a1a80 2024-04-22T08:12:59.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.388+0000 7fb78ff75640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:59.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.388+0000 7fb78ff75640 1 --2- 172.21.15.40:0/3721969691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7881cb690 0x7fb788120ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:59.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.388+0000 7fb78ff75640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:59.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.388+0000 7fb78ff75640 1 -- 172.21.15.40:0/3721969691 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb788066770 con 0x7fb7881cb690 2024-04-22T08:12:59.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.388+0000 7fb77e7fc640 1 --2- 172.21.15.40:0/3721969691 >> v2:172.21.15.189:6800/30168 conn(0x7fb76c0377d0 0x7fb76c039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:59.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.388+0000 7fb78d510640 1 --2- 172.21.15.40:0/3721969691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7881cb690 0x7fb788120ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:59.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.388+0000 7fb78d510640 10 monclient: get_auth_request con 0x7fb7881cb690 auth_method 0 2024-04-22T08:12:59.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.388+0000 7fb78d510640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:59.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.388+0000 7fb78d510640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:59.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.388+0000 7fb78d510640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:59.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.388+0000 7fb77e7fc640 1 -- 172.21.15.40:0/3721969691 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3506+0+0 (secure 0 0 0) 0x7fb784049ff0 con 0x7fb7880a1a80 2024-04-22T08:12:59.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.388+0000 7fb78d510640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:59.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.388+0000 7fb78d510640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:59.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.388+0000 7fb78d510640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:59.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.389+0000 7fb78d510640 10 monclient(hunting): handle_auth_done global_id 4380 payload 824 2024-04-22T08:12:59.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.389+0000 7fb78d510640 1 -- 172.21.15.40:0/3721969691 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb78813ebf0 con 0x7fb7881cb690 2024-04-22T08:12:59.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.389+0000 7fb78e512640 1 --2- 172.21.15.40:0/3721969691 >> v2:172.21.15.189:6800/30168 conn(0x7fb76c0377d0 0x7fb76c039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:59.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.389+0000 7fb78e512640 10 monclient: get_auth_request con 0x7fb76c0377d0 auth_method 0 2024-04-22T08:12:59.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.391+0000 7fb78d510640 1 -- 172.21.15.40:0/3721969691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7881cb690 msgr2=0x7fb788120ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:59.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.391+0000 7fb78d510640 1 --2- 172.21.15.40:0/3721969691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7881cb690 0x7fb788120ad0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb78806ca60 tx=0x7fb7780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:59.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.391+0000 7fb77e7fc640 1 -- 172.21.15.40:0/3721969691 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb77803dc10 con 0x7fb7881cb690 2024-04-22T08:12:59.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.391+0000 7fb77e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:59.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.391+0000 7fb77e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:12:59.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.391+0000 7fb77e7fc640 10 monclient: dump: 2024-04-22T08:12:59.403 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:59.403 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:59.404 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:59.404 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:59.404 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:59.404 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:59.404 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:59.404 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:59.404 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:59.404 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:59.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.391+0000 7fb78e512640 1 --2- 172.21.15.40:0/3721969691 >> v2:172.21.15.189:6800/30168 conn(0x7fb76c0377d0 0x7fb76c039c90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb780005c00 tx=0x7fb780009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:59.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.391+0000 7fb77e7fc640 1 -- 172.21.15.40:0/3721969691 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7fb77803dd70 con 0x7fb7881cb690 2024-04-22T08:12:59.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.391+0000 7fb77e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:59.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.391+0000 7fb77e7fc640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:12:59.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.391+0000 7fb77e7fc640 1 -- 172.21.15.40:0/3721969691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7881cb690 msgr2=0x7fb788120ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:59.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.391+0000 7fb77e7fc640 1 --2- 172.21.15.40:0/3721969691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7881cb690 0x7fb788120ad0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb78806ca60 tx=0x7fb7780046c0 comp rx=0 tx=0).stop 2024-04-22T08:12:59.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.391+0000 7fb77e7fc640 1 -- 172.21.15.40:0/3721969691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7881cb690 msgr2=0x7fb788120ad0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:12:59.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.391+0000 7fb77e7fc640 1 --2- 172.21.15.40:0/3721969691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7881cb690 0x7fb788120ad0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:59.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.398+0000 7fb78ff75640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:12:59.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.398+0000 7fb78ff75640 1 --2- 172.21.15.40:0/3721969691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb788121010 0x7fb788121420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:59.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.398+0000 7fb78ff75640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:59.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.398+0000 7fb78ff75640 1 -- 172.21.15.40:0/3721969691 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7880713f0 con 0x7fb788121010 2024-04-22T08:12:59.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.398+0000 7fb78d510640 1 --2- 172.21.15.40:0/3721969691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb788121010 0x7fb788121420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:59.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.398+0000 7fb78d510640 10 monclient: get_auth_request con 0x7fb788121010 auth_method 0 2024-04-22T08:12:59.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.398+0000 7fb78d510640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:59.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.398+0000 7fb78d510640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:59.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.398+0000 7fb78d510640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:59.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.398+0000 7fb78d510640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:59.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.398+0000 7fb78d510640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:59.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.398+0000 7fb78d510640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:59.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.398+0000 7fb78d510640 10 monclient(hunting): handle_auth_done global_id 4383 payload 824 2024-04-22T08:12:59.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.398+0000 7fb78d510640 1 -- 172.21.15.40:0/3721969691 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb7881cba70 con 0x7fb788121010 2024-04-22T08:12:59.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.400+0000 7fb78d510640 1 -- 172.21.15.40:0/3721969691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb788121010 msgr2=0x7fb788121420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:59.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.400+0000 7fb78d510640 1 --2- 172.21.15.40:0/3721969691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb788121010 0x7fb788121420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb77802b750 tx=0x7fb778008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:59.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.400+0000 7fb77e7fc640 1 -- 172.21.15.40:0/3721969691 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb778050a40 con 0x7fb788121010 2024-04-22T08:12:59.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.400+0000 7fb77e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:59.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.400+0000 7fb77e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:12:59.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.400+0000 7fb77e7fc640 10 monclient: dump: 2024-04-22T08:12:59.407 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:59.407 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:59.407 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:59.407 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:59.407 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:59.407 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:59.407 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:59.408 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:59.408 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:59.408 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:59.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.400+0000 7fb77e7fc640 1 -- 172.21.15.40:0/3721969691 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4633 (secure 0 0 0) 0x7fb778050ba0 con 0x7fb788121010 2024-04-22T08:12:59.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.400+0000 7fb77e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:12:59.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.400+0000 7fb77e7fc640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:12:59.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.401+0000 7fb77e7fc640 1 -- 172.21.15.40:0/3721969691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb788121010 msgr2=0x7fb788121420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:59.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.401+0000 7fb77e7fc640 1 --2- 172.21.15.40:0/3721969691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb788121010 0x7fb788121420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb77802b750 tx=0x7fb778008030 comp rx=0 tx=0).stop 2024-04-22T08:12:59.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.401+0000 7fb77e7fc640 1 -- 172.21.15.40:0/3721969691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb788121010 msgr2=0x7fb788121420 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:12:59.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.401+0000 7fb77e7fc640 1 --2- 172.21.15.40:0/3721969691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb788121010 0x7fb788121420 secure :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb77802b750 tx=0x7fb778008030 comp rx=0 tx=0).stop 2024-04-22T08:12:59.408 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:12:59.409 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:12:59.409 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:12:59.409 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "leader", 2024-04-22T08:12:59.409 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 8, 2024-04-22T08:12:59.409 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:12:59.409 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:12:59.409 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:12:59.409 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:12:59.409 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:12:59.409 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 23, 2024-04-22T08:12:59.410 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:12:59.410 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:12:59.410 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:12:59.410 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:12:59.410 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:12:59.410 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:12:59.410 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:12:59.410 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:12:59.410 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:12:59.410 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:12:59.410 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:12:59.410 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:12:59.410 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:12:59.411 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:12:59.411 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:12:59.411 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:12:59.411 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:12:59.411 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:12:59.411 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:12:59.411 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:12:59.411 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:12:59.411 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:12:59.411 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:12:59.411 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:12:59.411 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:12:59.411 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:12:59.411 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:12:59.411 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:12:59.412 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:12:59.412 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:12:59.412 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:12:59.412 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:12:59.412 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:12:59.412 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:12:59.412 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:12:59.412 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:12:59.412 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:12:59.412 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:12:59.412 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:12:59.412 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:12:59.412 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:12:59.412 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:12:59.413 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:12:59.413 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:12:59.413 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:12:59.413 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:12:59.413 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:12:59.413 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:12:59.413 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:12:59.413 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:12:59.413 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:12:59.413 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:12:59.413 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:12:59.413 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:12:59.413 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:12:59.414 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:12:59.414 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:12:59.414 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:12:59.414 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:12:59.414 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:12:59.414 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:12:59.414 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:12:59.414 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:12:59.414 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:12:59.414 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:12:59.414 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:12:59.414 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:12:59.414 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:12:59.414 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:12:59.415 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:12:59.415 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:12:59.415 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:12:59.415 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:12:59.415 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:12:59.415 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:12:59.415 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:12:59.415 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:12:59.415 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:12:59.415 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:12:59.415 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:12:59.415 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:12:59.415 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:12:59.415 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:12:59.416 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:12:59.416 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:12:59.416 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:12:59.416 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:12:59.416 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:12:59.416 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:12:59.416 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:12:59.416 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:12:59.416 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:12:59.416 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:12:59.416 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:12:59.416 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:12:59.416 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:12:59.417 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:12:59.417 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:12:59.417 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:12:59.417 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:12:59.417 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:12:59.417 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:12:59.417 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:12:59.417 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:12:59.417 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:12:59.417 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:12:59.417 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:12:59.417 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:12:59.417 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:12:59.417 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:12:59.418 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:12:59.418 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:12:59.418 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:12:59.418 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:12:59.418 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:12:59.418 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:12:59.418 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:12:59.418 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:12:59.418 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:12:59.418 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:12:59.418 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:12:59.418 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:12:59.419 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:12:59.419 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:12:59.419 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:12:59.419 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:12:59.419 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:12:59.419 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:12:59.419 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:12:59.419 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:12:59.419 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:12:59.419 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:12:59.419 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:12:59.419 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:12:59.419 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:12:59.419 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:12:59.419 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:12:59.420 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:12:59.420 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:12:59.420 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:12:59.420 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:12:59.420 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:12:59.420 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 2 2024-04-22T08:12:59.420 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:12:59.420 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:12:59.420 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:12:59.420 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:12:59.420 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:12:59.420 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:12:59.420 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 3 2024-04-22T08:12:59.420 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:12:59.421 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:12:59.421 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:12:59.421 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:12:59.421 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:12:59.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.401+0000 7fb78ff75640 1 -- 172.21.15.40:0/3721969691 >> v2:172.21.15.189:6800/30168 conn(0x7fb76c0377d0 msgr2=0x7fb76c039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:59.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.401+0000 7fb78ff75640 1 --2- 172.21.15.40:0/3721969691 >> v2:172.21.15.189:6800/30168 conn(0x7fb76c0377d0 0x7fb76c039c90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb780005c00 tx=0x7fb780009040 comp rx=0 tx=0).stop 2024-04-22T08:12:59.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.401+0000 7fb78ff75640 10 monclient: shutdown 2024-04-22T08:12:59.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.401+0000 7fb78ff75640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:59.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.401+0000 7fb78ff75640 1 -- 172.21.15.40:0/3721969691 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb7880a1a80 msgr2=0x7fb7880706f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:59.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.401+0000 7fb78ff75640 1 --2- 172.21.15.40:0/3721969691 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb7880a1a80 0x7fb7880706f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb78400b420 tx=0x7fb78400b8e0 comp rx=0 tx=0).stop 2024-04-22T08:12:59.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.401+0000 7fb78e512640 1 -- 172.21.15.40:0/3721969691 reap_dead start 2024-04-22T08:12:59.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.401+0000 7fb78e512640 1 -- 172.21.15.40:0/3721969691 reap_dead start 2024-04-22T08:12:59.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.401+0000 7fb78ff75640 1 -- 172.21.15.40:0/3721969691 shutdown_connections 2024-04-22T08:12:59.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.401+0000 7fb78ff75640 1 -- 172.21.15.40:0/3721969691 >> 172.21.15.40:0/3721969691 conn(0x7fb7880a66a0 msgr2=0x7fb78809b910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:59.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.401+0000 7fb78ff75640 1 -- 172.21.15.40:0/3721969691 shutdown_connections 2024-04-22T08:12:59.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.401+0000 7fb78ff75640 1 -- 172.21.15.40:0/3721969691 wait complete. 2024-04-22T08:12:59.423 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:12:59.423 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-22T08:12:59.424 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-22T08:12:59.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.524+0000 7fc27e8d5640 10 monclient: get_monmap_and_config 2024-04-22T08:12:59.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.524+0000 7fc27e8d5640 10 monclient: build_initial_monmap 2024-04-22T08:12:59.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.524+0000 7fc27e8d5640 10 monclient: monmap: 2024-04-22T08:12:59.530 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:59.530 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:59.530 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:59.525471+0000 2024-04-22T08:12:59.530 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:59.525471+0000 2024-04-22T08:12:59.530 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:59.531 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:59.531 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:12:59.531 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:12:59.531 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:12:59.531 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:12:59.531 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:12:59.531 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:59.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.525+0000 7fc27e8d5640 1 Processor -- start 2024-04-22T08:12:59.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.525+0000 7fc27e8d5640 1 -- start start 2024-04-22T08:12:59.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.525+0000 7fc27e8d5640 10 monclient: init 2024-04-22T08:12:59.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.525+0000 7fc27e8d5640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:59.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.525+0000 7fc27e8d5640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-04-22T08:12:59.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.525+0000 7fc27e8d5640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc2780a1c60 0x7fc2780a31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:59.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.525+0000 7fc27e8d5640 10 monclient(hunting): picked mon.noname-a con 0x7fc2780a1c60 addr v2:172.21.15.40:3300/0 2024-04-22T08:12:59.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.525+0000 7fc27e8d5640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc2780a2040 0x7fc278135bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:59.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.525+0000 7fc27e8d5640 10 monclient(hunting): picked mon.noname-c con 0x7fc2780a2040 addr v2:172.21.15.189:3300/0 2024-04-22T08:12:59.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.525+0000 7fc27e8d5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc2780a24d0 addr v1:172.21.15.189:6789/0 2024-04-22T08:12:59.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.525+0000 7fc27e8d5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2780a41f0 con 0x7fc2780a24d0 2024-04-22T08:12:59.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.525+0000 7fc27e8d5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:59.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.525+0000 7fc27e8d5640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc27809fe90 con 0x7fc2780a1c60 2024-04-22T08:12:59.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.525+0000 7fc27e8d5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:59.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.525+0000 7fc27e8d5640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2780a0170 con 0x7fc2780a2040 2024-04-22T08:12:59.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.525+0000 7fc27e8d5640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:59.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.525+0000 7fc27e8d5640 10 monclient(hunting): authenticate will time out at 853.519717s 2024-04-22T08:12:59.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.526+0000 7fc277fff640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc2780a1c60 0x7fc2780a31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:59.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.526+0000 7fc277fff640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc2780a1c60 0x7fc2780a31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40688/0 (socket says 172.21.15.40:40688) 2024-04-22T08:12:59.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.526+0000 7fc277fff640 1 -- 172.21.15.40:0/1654945857 learned_addr learned my addr 172.21.15.40:0/1654945857 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:59.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.526+0000 7fc277fff640 10 monclient(hunting): get_auth_request con 0x7fc2780a1c60 auth_method 0 2024-04-22T08:12:59.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.526+0000 7fc277fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:59.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.526+0000 7fc277fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:59.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.526+0000 7fc277fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:59.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.526+0000 7fc277fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:59.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.526+0000 7fc277fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:59.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.526+0000 7fc277fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:59.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.526+0000 7fc2777fe640 1 --2- 172.21.15.40:0/1654945857 >> v2:172.21.15.189:3300/0 conn(0x7fc2780a2040 0x7fc278135bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:59.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.526+0000 7fc2777fe640 10 monclient(hunting): get_auth_request con 0x7fc2780a2040 auth_method 0 2024-04-22T08:12:59.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.526+0000 7fc2777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:59.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.526+0000 7fc2777fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:59.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.526+0000 7fc2777fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:59.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.526+0000 7fc277fff640 10 monclient(hunting): handle_auth_done global_id 4386 payload 274 2024-04-22T08:12:59.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.526+0000 7fc277fff640 1 -- 172.21.15.40:0/1654945857 >> v2:172.21.15.189:3300/0 conn(0x7fc2780a2040 msgr2=0x7fc278135bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:59.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.526+0000 7fc277fff640 1 --2- 172.21.15.40:0/1654945857 >> v2:172.21.15.189:3300/0 conn(0x7fc2780a2040 0x7fc278135bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:59.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.526+0000 7fc277fff640 1 -- 172.21.15.40:0/1654945857 >> v1:172.21.15.189:6789/0 conn(0x7fc2780a24d0 legacy=0x7fc2781360f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:59.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.526+0000 7fc277fff640 10 monclient: _finish_hunting 0 2024-04-22T08:12:59.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.526+0000 7fc277fff640 1 monclient: found mon.noname-a 2024-04-22T08:12:59.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.526+0000 7fc277fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:59.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.526+0000 7fc277fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:12:59.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.526+0000 7fc277fff640 1 -- 172.21.15.40:0/1654945857 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2780a0450 con 0x7fc2780a1c60 2024-04-22T08:12:59.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.529+0000 7fc277fff640 1 -- 172.21.15.40:0/1654945857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2780a1c60 msgr2=0x7fc2780a31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:59.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.529+0000 7fc277fff640 1 --2- 172.21.15.40:0/1654945857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2780a1c60 0x7fc2780a31d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc27809f4b0 tx=0x7fc26c02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14527ac814ac2c4f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:59.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.529+0000 7fc276ffd640 1 -- 172.21.15.40:0/1654945857 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc26c031960 con 0x7fc2780a1c60 2024-04-22T08:12:59.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.529+0000 7fc276ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:59.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.529+0000 7fc276ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:12:59.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.529+0000 7fc276ffd640 10 monclient: dump: 2024-04-22T08:12:59.535 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:59.535 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:59.535 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:59.535 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:59.535 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:59.535 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:59.535 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:59.535 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:59.535 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:59.535 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:59.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.529+0000 7fc276ffd640 10 monclient: _finish_auth 0 2024-04-22T08:12:59.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.529+0000 7fc276ffd640 10 monclient: _check_auth_tickets 2024-04-22T08:12:59.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.529+0000 7fc276ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:59.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.529+0000 7fc276ffd640 1 -- 172.21.15.40:0/1654945857 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc26c031ac0 con 0x7fc2780a1c60 2024-04-22T08:12:59.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.529+0000 7fc27e8d5640 5 monclient: authenticate success, global_id 4386 2024-04-22T08:12:59.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.529+0000 7fc27e8d5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:12:59.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.529+0000 7fc276ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:59.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.529+0000 7fc27e8d5640 10 monclient: get_monmap_and_config success 2024-04-22T08:12:59.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.529+0000 7fc276ffd640 1 -- 172.21.15.40:0/1654945857 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc26c031e50 con 0x7fc2780a1c60 2024-04-22T08:12:59.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.529+0000 7fc27e8d5640 4 set_mon_vals no callback set 2024-04-22T08:12:59.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.529+0000 7fc276ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:59.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.529+0000 7fc276ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:12:59.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.529+0000 7fc276ffd640 10 monclient: dump: 2024-04-22T08:12:59.536 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:59.537 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:59.537 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:59.537 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:59.537 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:59.537 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:59.537 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:59.537 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:59.537 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:59.537 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:59.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.529+0000 7fc27e8d5640 10 monclient: shutdown 2024-04-22T08:12:59.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.529+0000 7fc27e8d5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:59.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.529+0000 7fc27e8d5640 1 -- 172.21.15.40:0/1654945857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2780a1c60 msgr2=0x7fc2780a31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:59.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.529+0000 7fc27e8d5640 1 --2- 172.21.15.40:0/1654945857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2780a1c60 0x7fc2780a31d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc27809f4b0 tx=0x7fc26c02b7f0 comp rx=0 tx=0).stop 2024-04-22T08:12:59.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.529+0000 7fc27e8d5640 1 -- 172.21.15.40:0/1654945857 shutdown_connections 2024-04-22T08:12:59.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.529+0000 7fc27e8d5640 1 --2- 172.21.15.40:0/1654945857 >> v2:172.21.15.189:3300/0 conn(0x7fc2780a2040 0x7fc278135bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:59.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.529+0000 7fc27e8d5640 1 --2- 172.21.15.40:0/1654945857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2780a1c60 0x7fc2780a31d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:59.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.529+0000 7fc27e8d5640 1 -- 172.21.15.40:0/1654945857 >> 172.21.15.40:0/1654945857 conn(0x7fc2780a89a0 msgr2=0x7fc2780a4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:59.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.529+0000 7fc27e8d5640 1 -- 172.21.15.40:0/1654945857 shutdown_connections 2024-04-22T08:12:59.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.529+0000 7fc27e8d5640 1 -- 172.21.15.40:0/1654945857 wait complete. 2024-04-22T08:12:59.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.530+0000 7fc27e8d5640 10 monclient: build_initial_monmap 2024-04-22T08:12:59.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.530+0000 7fc27e8d5640 10 monclient: monmap: 2024-04-22T08:12:59.538 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:59.538 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:12:59.538 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:12:59.538 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:12:59.538 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:59.538 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:59.538 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:12:59.538 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:12:59.539 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:12:59.539 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:59.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.530+0000 7fc27e8d5640 1 Processor -- start 2024-04-22T08:12:59.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc27e8d5640 1 -- start start 2024-04-22T08:12:59.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc27e8d5640 10 monclient: init 2024-04-22T08:12:59.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc27e8d5640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:59.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc27e8d5640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-22T08:12:59.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc27e8d5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2780a1c60 0x7fc2781cb030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:59.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc27e8d5640 10 monclient(hunting): picked mon.noname-b con 0x7fc2780a1c60 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:59.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc27e8d5640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2780a2040 0x7fc278121300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:59.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc27e8d5640 10 monclient(hunting): picked mon.noname-c con 0x7fc2780a2040 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:59.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc27e8d5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2780a24d0 0x7fc278121840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:59.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc27e8d5640 10 monclient(hunting): picked mon.noname-a con 0x7fc2780a24d0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:59.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc27e8d5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:59.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc27e8d5640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2781249e0 con 0x7fc2780a24d0 2024-04-22T08:12:59.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc27e8d5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:59.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc27e8d5640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc278123af0 con 0x7fc2780a1c60 2024-04-22T08:12:59.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc27e8d5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:59.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc27e8d5640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc278134fd0 con 0x7fc2780a2040 2024-04-22T08:12:59.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc27e8d5640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:59.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc27e8d5640 10 monclient(hunting): authenticate will time out at 853.525292s 2024-04-22T08:12:59.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc2777fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2780a2040 0x7fc278121300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:59.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc27ce72640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2780a24d0 0x7fc278121840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:59.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc2777fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2780a2040 0x7fc278121300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:51174/0 (socket says 172.21.15.40:51174) 2024-04-22T08:12:59.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc2777fe640 1 -- 172.21.15.40:0/439338928 learned_addr learned my addr 172.21.15.40:0/439338928 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:59.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc27ce72640 10 monclient(hunting): get_auth_request con 0x7fc2780a24d0 auth_method 0 2024-04-22T08:12:59.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc27ce72640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:59.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc27ce72640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:59.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc27ce72640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:59.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc2777fe640 10 monclient(hunting): get_auth_request con 0x7fc2780a2040 auth_method 0 2024-04-22T08:12:59.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc2777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:59.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc2777fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:59.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc2777fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:59.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc27ce72640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:59.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc27ce72640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:59.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc27ce72640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:59.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc2777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:59.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc2777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:59.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc277fff640 1 --2- 172.21.15.40:0/439338928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2780a1c60 0x7fc2781cb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:59.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc2777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:59.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc277fff640 10 monclient(hunting): get_auth_request con 0x7fc2780a1c60 auth_method 0 2024-04-22T08:12:59.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc277fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:59.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc277fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:59.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc277fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:59.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc277fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:59.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc277fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:59.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc277fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:59.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc27ce72640 10 monclient(hunting): handle_auth_done global_id 4389 payload 824 2024-04-22T08:12:59.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc27ce72640 1 -- 172.21.15.40:0/439338928 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2780a2040 msgr2=0x7fc278121300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:59.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc27ce72640 1 --2- 172.21.15.40:0/439338928 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2780a2040 0x7fc278121300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:59.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc27ce72640 1 -- 172.21.15.40:0/439338928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2780a1c60 msgr2=0x7fc2781cb030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:59.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc27ce72640 1 --2- 172.21.15.40:0/439338928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2780a1c60 0x7fc2781cb030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:59.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc27ce72640 10 monclient: _finish_hunting 0 2024-04-22T08:12:59.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc27ce72640 1 monclient: found mon.noname-a 2024-04-22T08:12:59.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc27ce72640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:59.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc27ce72640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:12:59.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc27ce72640 1 -- 172.21.15.40:0/439338928 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc26c0094d0 con 0x7fc2780a24d0 2024-04-22T08:12:59.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.531+0000 7fc2777fe640 1 --2- 172.21.15.40:0/439338928 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2780a2040 0x7fc278121300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:12:59.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.534+0000 7fc27ce72640 1 -- 172.21.15.40:0/439338928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2780a24d0 msgr2=0x7fc278121840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:59.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.534+0000 7fc27ce72640 1 --2- 172.21.15.40:0/439338928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2780a24d0 0x7fc278121840 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc27000c780 tx=0x7fc27000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:59.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.534+0000 7fc274ff9640 1 -- 172.21.15.40:0/439338928 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc27000ac60 con 0x7fc2780a24d0 2024-04-22T08:12:59.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.534+0000 7fc274ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:59.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.534+0000 7fc274ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:12:59.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.534+0000 7fc274ff9640 10 monclient: dump: 2024-04-22T08:12:59.551 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:59.551 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:59.551 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:59.551 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:59.551 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:59.551 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:59.551 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:59.551 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:59.551 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:59.551 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:59.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.534+0000 7fc274ff9640 10 monclient: _finish_auth 0 2024-04-22T08:12:59.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.534+0000 7fc274ff9640 10 monclient: _check_auth_tickets 2024-04-22T08:12:59.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.534+0000 7fc274ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:59.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.534+0000 7fc27e8d5640 5 monclient: authenticate success, global_id 4389 2024-04-22T08:12:59.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.534+0000 7fc274ff9640 1 -- 172.21.15.40:0/439338928 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc27000adc0 con 0x7fc2780a24d0 2024-04-22T08:12:59.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.534+0000 7fc27e8d5640 10 monclient: _renew_subs 2024-04-22T08:12:59.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.534+0000 7fc27e8d5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:59.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.534+0000 7fc27e8d5640 1 -- 172.21.15.40:0/439338928 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2780a4790 con 0x7fc2780a24d0 2024-04-22T08:12:59.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.534+0000 7fc274ff9640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:59.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.534+0000 7fc274ff9640 1 -- 172.21.15.40:0/439338928 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc270012660 con 0x7fc2780a24d0 2024-04-22T08:12:59.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.534+0000 7fc274ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:59.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.534+0000 7fc274ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:12:59.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.534+0000 7fc274ff9640 10 monclient: dump: 2024-04-22T08:12:59.552 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:59.552 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:59.552 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:59.552 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:59.552 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:59.553 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:59.553 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:59.553 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:59.553 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:59.553 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:59.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.534+0000 7fc27e8d5640 10 monclient: _renew_subs 2024-04-22T08:12:59.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.534+0000 7fc27e8d5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:12:59.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.534+0000 7fc27e8d5640 1 -- 172.21.15.40:0/439338928 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc278122eb0 con 0x7fc2780a24d0 2024-04-22T08:12:59.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.534+0000 7fc275ffb640 4 set_mon_vals no callback set 2024-04-22T08:12:59.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.535+0000 7fc27e8d5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:59.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.535+0000 7fc27e8d5640 1 --2- 172.21.15.40:0/439338928 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc278125bd0 0x7fc278121d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:59.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.535+0000 7fc27e8d5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:59.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.535+0000 7fc27e8d5640 1 -- 172.21.15.40:0/439338928 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc278066650 con 0x7fc278125bd0 2024-04-22T08:12:59.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.535+0000 7fc277fff640 1 --2- 172.21.15.40:0/439338928 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc278125bd0 0x7fc278121d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:59.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.535+0000 7fc277fff640 10 monclient: get_auth_request con 0x7fc278125bd0 auth_method 0 2024-04-22T08:12:59.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.535+0000 7fc277fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:59.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.535+0000 7fc277fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:59.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.535+0000 7fc277fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:59.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.535+0000 7fc277fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:59.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.535+0000 7fc277fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:59.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.535+0000 7fc277fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:59.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.535+0000 7fc274ff9640 1 -- 172.21.15.40:0/439338928 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fc270012840 con 0x7fc2780a24d0 2024-04-22T08:12:59.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.535+0000 7fc277fff640 10 monclient(hunting): handle_auth_done global_id 4361 payload 824 2024-04-22T08:12:59.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.535+0000 7fc277fff640 1 -- 172.21.15.40:0/439338928 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc278136800 con 0x7fc278125bd0 2024-04-22T08:12:59.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.535+0000 7fc274ff9640 1 --2- 172.21.15.40:0/439338928 >> v2:172.21.15.189:6800/30168 conn(0x7fc264037850 0x7fc264039d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:59.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.535+0000 7fc274ff9640 1 -- 172.21.15.40:0/439338928 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3506+0+0 (secure 0 0 0) 0x7fc27004a2d0 con 0x7fc2780a24d0 2024-04-22T08:12:59.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.536+0000 7fc2777fe640 1 --2- 172.21.15.40:0/439338928 >> v2:172.21.15.189:6800/30168 conn(0x7fc264037850 0x7fc264039d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:59.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.536+0000 7fc2777fe640 10 monclient: get_auth_request con 0x7fc264037850 auth_method 0 2024-04-22T08:12:59.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.537+0000 7fc277fff640 1 -- 172.21.15.40:0/439338928 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc278125bd0 msgr2=0x7fc278121d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:59.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.537+0000 7fc277fff640 1 --2- 172.21.15.40:0/439338928 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc278125bd0 0x7fc278121d80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc2780a4be0 tx=0x7fc26c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:59.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.537+0000 7fc274ff9640 1 -- 172.21.15.40:0/439338928 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc26c03d640 con 0x7fc278125bd0 2024-04-22T08:12:59.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.537+0000 7fc274ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:59.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.537+0000 7fc274ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:12:59.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.537+0000 7fc274ff9640 10 monclient: dump: 2024-04-22T08:12:59.555 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:59.555 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:59.555 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:59.555 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:59.555 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:59.555 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:59.556 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:59.556 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:59.556 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:59.556 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:59.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.538+0000 7fc274ff9640 1 -- 172.21.15.40:0/439338928 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7fc26c041070 con 0x7fc278125bd0 2024-04-22T08:12:59.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.538+0000 7fc274ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:59.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.538+0000 7fc274ff9640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:12:59.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.538+0000 7fc274ff9640 1 -- 172.21.15.40:0/439338928 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc278125bd0 msgr2=0x7fc278121d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:59.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.538+0000 7fc274ff9640 1 --2- 172.21.15.40:0/439338928 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc278125bd0 0x7fc278121d80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc2780a4be0 tx=0x7fc26c0046c0 comp rx=0 tx=0).stop 2024-04-22T08:12:59.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.538+0000 7fc274ff9640 1 -- 172.21.15.40:0/439338928 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc278125bd0 msgr2=0x7fc278121d80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:12:59.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.538+0000 7fc274ff9640 1 --2- 172.21.15.40:0/439338928 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc278125bd0 0x7fc278121d80 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:59.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.538+0000 7fc2777fe640 1 --2- 172.21.15.40:0/439338928 >> v2:172.21.15.189:6800/30168 conn(0x7fc264037850 0x7fc264039d10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc2781351b0 tx=0x7fc268008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:59.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.544+0000 7fc27e8d5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:12:59.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.544+0000 7fc27e8d5640 1 --2- 172.21.15.40:0/439338928 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2781cb570 0x7fc2781222c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:59.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.544+0000 7fc27e8d5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:59.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.544+0000 7fc27e8d5640 1 -- 172.21.15.40:0/439338928 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc278136350 con 0x7fc2781cb570 2024-04-22T08:12:59.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.544+0000 7fc277fff640 1 --2- 172.21.15.40:0/439338928 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2781cb570 0x7fc2781222c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:59.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.544+0000 7fc277fff640 10 monclient: get_auth_request con 0x7fc2781cb570 auth_method 0 2024-04-22T08:12:59.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.544+0000 7fc277fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:59.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.544+0000 7fc277fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:59.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.544+0000 7fc277fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:59.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.544+0000 7fc277fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:59.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.544+0000 7fc277fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:59.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.544+0000 7fc277fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:59.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.544+0000 7fc277fff640 10 monclient(hunting): handle_auth_done global_id 4364 payload 824 2024-04-22T08:12:59.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.544+0000 7fc277fff640 1 -- 172.21.15.40:0/439338928 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc2780a64f0 con 0x7fc2781cb570 2024-04-22T08:12:59.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.547+0000 7fc277fff640 1 -- 172.21.15.40:0/439338928 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2781cb570 msgr2=0x7fc2781222c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:59.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.547+0000 7fc277fff640 1 --2- 172.21.15.40:0/439338928 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2781cb570 0x7fc2781222c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc26c032040 tx=0x7fc26c031000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:59.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.547+0000 7fc274ff9640 1 -- 172.21.15.40:0/439338928 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc26c047cd0 con 0x7fc2781cb570 2024-04-22T08:12:59.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.547+0000 7fc274ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:59.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.547+0000 7fc274ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:12:59.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.547+0000 7fc274ff9640 10 monclient: dump: 2024-04-22T08:12:59.559 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:59.559 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:59.559 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:59.559 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:59.559 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:59.559 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:59.559 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:59.559 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:59.559 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:59.559 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:59.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.547+0000 7fc274ff9640 1 -- 172.21.15.40:0/439338928 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4631 (secure 0 0 0) 0x7fc26c047e30 con 0x7fc2781cb570 2024-04-22T08:12:59.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.547+0000 7fc274ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:12:59.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.547+0000 7fc274ff9640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:12:59.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.547+0000 7fc274ff9640 1 -- 172.21.15.40:0/439338928 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2781cb570 msgr2=0x7fc2781222c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:59.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.547+0000 7fc274ff9640 1 --2- 172.21.15.40:0/439338928 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2781cb570 0x7fc2781222c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc26c032040 tx=0x7fc26c031000 comp rx=0 tx=0).stop 2024-04-22T08:12:59.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.547+0000 7fc274ff9640 1 -- 172.21.15.40:0/439338928 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2781cb570 msgr2=0x7fc2781222c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:12:59.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.547+0000 7fc274ff9640 1 --2- 172.21.15.40:0/439338928 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2781cb570 0x7fc2781222c0 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc26c032040 tx=0x7fc26c031000 comp rx=0 tx=0).stop 2024-04-22T08:12:59.560 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:12:59.560 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:12:59.560 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:12:59.560 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:12:59.561 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 8, 2024-04-22T08:12:59.561 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:12:59.561 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:12:59.561 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:12:59.561 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:12:59.561 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:12:59.561 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 23, 2024-04-22T08:12:59.561 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:12:59.561 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:12:59.561 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:12:59.561 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:12:59.561 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:12:59.561 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:12:59.561 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:12:59.562 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:12:59.562 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:12:59.562 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:12:59.562 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:12:59.562 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:12:59.562 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:12:59.562 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:12:59.562 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:12:59.562 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:12:59.562 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:12:59.562 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:12:59.562 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:12:59.562 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:12:59.562 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:12:59.563 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:12:59.563 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:12:59.563 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:12:59.563 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:12:59.563 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:12:59.563 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:12:59.563 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:12:59.563 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:12:59.563 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:12:59.563 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:12:59.563 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:12:59.563 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:12:59.563 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:12:59.563 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:12:59.564 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:12:59.564 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:12:59.564 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:12:59.564 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:12:59.564 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:12:59.564 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:12:59.564 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:12:59.564 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:12:59.564 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:12:59.564 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:12:59.564 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:12:59.564 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:12:59.564 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:12:59.564 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:12:59.564 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:12:59.565 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:12:59.565 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:12:59.565 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:12:59.565 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:12:59.565 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:12:59.565 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:12:59.565 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:12:59.565 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:12:59.565 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:12:59.565 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:12:59.565 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:12:59.565 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:12:59.565 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:12:59.565 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:12:59.565 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:12:59.566 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:12:59.566 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:12:59.566 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:12:59.566 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:12:59.566 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:12:59.566 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:12:59.566 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:12:59.566 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:12:59.566 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:12:59.566 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:12:59.566 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:12:59.566 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:12:59.566 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:12:59.566 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:12:59.567 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:12:59.567 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:12:59.567 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:12:59.567 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:12:59.567 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:12:59.567 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:12:59.567 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:12:59.567 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:12:59.567 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:12:59.567 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:12:59.567 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:12:59.567 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:12:59.567 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:12:59.567 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:12:59.567 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:12:59.568 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:12:59.568 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:12:59.568 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:12:59.568 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:12:59.568 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:12:59.568 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:12:59.568 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:12:59.568 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:12:59.568 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:12:59.568 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:12:59.568 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:12:59.568 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:12:59.568 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:12:59.568 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:12:59.568 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:12:59.569 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:12:59.569 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:12:59.569 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:12:59.569 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:12:59.569 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:12:59.569 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:12:59.569 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:12:59.569 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:12:59.569 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:12:59.569 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:12:59.569 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:12:59.569 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:12:59.569 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:12:59.570 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:12:59.570 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:12:59.570 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:12:59.570 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:12:59.570 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:12:59.570 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:12:59.570 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:12:59.570 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:12:59.570 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:12:59.570 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:12:59.570 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:12:59.570 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:12:59.570 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:12:59.571 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:12:59.571 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:12:59.571 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:12:59.571 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:12:59.571 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:12:59.571 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:12:59.571 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:12:59.571 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:12:59.571 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:12:59.571 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:12:59.571 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:12:59.571 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:12:59.571 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:12:59.571 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:12:59.571 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:12:59.572 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:12:59.572 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:12:59.572 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:12:59.572 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:12:59.572 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:12:59.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.547+0000 7fc27e8d5640 1 -- 172.21.15.40:0/439338928 >> v2:172.21.15.189:6800/30168 conn(0x7fc264037850 msgr2=0x7fc264039d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:59.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.547+0000 7fc27e8d5640 1 --2- 172.21.15.40:0/439338928 >> v2:172.21.15.189:6800/30168 conn(0x7fc264037850 0x7fc264039d10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc2781351b0 tx=0x7fc268008040 comp rx=0 tx=0).stop 2024-04-22T08:12:59.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.547+0000 7fc27e8d5640 10 monclient: shutdown 2024-04-22T08:12:59.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.547+0000 7fc27e8d5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:59.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.547+0000 7fc27e8d5640 1 -- 172.21.15.40:0/439338928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2780a24d0 msgr2=0x7fc278121840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:59.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.547+0000 7fc27ce72640 1 -- 172.21.15.40:0/439338928 reap_dead start 2024-04-22T08:12:59.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.547+0000 7fc27e8d5640 1 --2- 172.21.15.40:0/439338928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2780a24d0 0x7fc278121840 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc27000c780 tx=0x7fc27000cc40 comp rx=0 tx=0).stop 2024-04-22T08:12:59.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.547+0000 7fc27e8d5640 1 -- 172.21.15.40:0/439338928 shutdown_connections 2024-04-22T08:12:59.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.547+0000 7fc27e8d5640 1 --2- 172.21.15.40:0/439338928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2780a24d0 0x7fc278121840 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:59.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.547+0000 7fc27e8d5640 1 -- 172.21.15.40:0/439338928 >> 172.21.15.40:0/439338928 conn(0x7fc2780a89a0 msgr2=0x7fc278135ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:59.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.548+0000 7fc27e8d5640 1 -- 172.21.15.40:0/439338928 shutdown_connections 2024-04-22T08:12:59.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.548+0000 7fc27e8d5640 1 -- 172.21.15.40:0/439338928 wait complete. 2024-04-22T08:12:59.574 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:12:59.574 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-22T08:12:59.575 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-22T08:12:59.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.675+0000 7f1eb5ce0640 10 monclient: get_monmap_and_config 2024-04-22T08:12:59.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.675+0000 7f1eb5ce0640 10 monclient: build_initial_monmap 2024-04-22T08:12:59.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.675+0000 7f1eb5ce0640 10 monclient: monmap: 2024-04-22T08:12:59.675 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:59.675 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:59.675 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:59.675843+0000 2024-04-22T08:12:59.675 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:59.675843+0000 2024-04-22T08:12:59.675 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:59.675 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:59.675 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:12:59.675 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:12:59.675 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:12:59.676 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:12:59.676 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:12:59.676 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:59.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.675+0000 7f1eb5ce0640 1 Processor -- start 2024-04-22T08:12:59.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.675+0000 7f1eb5ce0640 1 -- start start 2024-04-22T08:12:59.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.675+0000 7f1eb5ce0640 10 monclient: init 2024-04-22T08:12:59.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.676+0000 7f1eb5ce0640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:59.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.676+0000 7f1eb5ce0640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-04-22T08:12:59.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.676+0000 7f1eb5ce0640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1eb00a1c60 0x7f1eb00a31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:59.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.676+0000 7f1eb5ce0640 10 monclient(hunting): picked mon.noname-c con 0x7f1eb00a1c60 addr v2:172.21.15.189:3300/0 2024-04-22T08:12:59.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.676+0000 7f1eb5ce0640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1eb00a2040 0x7f1eb0135bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:59.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.676+0000 7f1eb5ce0640 10 monclient(hunting): picked mon.noname-a con 0x7f1eb00a2040 addr v2:172.21.15.40:3300/0 2024-04-22T08:12:59.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.676+0000 7f1eb5ce0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1eb00a24d0 addr v1:172.21.15.189:6789/0 2024-04-22T08:12:59.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.676+0000 7f1eb5ce0640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb00a41f0 con 0x7f1eb00a24d0 2024-04-22T08:12:59.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.676+0000 7f1eb5ce0640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:59.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.676+0000 7f1eb5ce0640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eb009fe90 con 0x7f1eb00a2040 2024-04-22T08:12:59.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.676+0000 7f1eb5ce0640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:59.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.676+0000 7f1eb5ce0640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eb00a0170 con 0x7f1eb00a1c60 2024-04-22T08:12:59.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.676+0000 7f1eb5ce0640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:59.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.676+0000 7f1eb5ce0640 10 monclient(hunting): authenticate will time out at 853.670078s 2024-04-22T08:12:59.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.676+0000 7f1eaeffd640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1eb00a2040 0x7f1eb0135bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:59.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.676+0000 7f1eaeffd640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1eb00a2040 0x7f1eb0135bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40704/0 (socket says 172.21.15.40:40704) 2024-04-22T08:12:59.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.676+0000 7f1eaeffd640 1 -- 172.21.15.40:0/154810038 learned_addr learned my addr 172.21.15.40:0/154810038 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:59.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.676+0000 7f1eaeffd640 10 monclient(hunting): get_auth_request con 0x7f1eb00a2040 auth_method 0 2024-04-22T08:12:59.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.676+0000 7f1eaeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:59.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.676+0000 7f1eaeffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:59.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.676+0000 7f1eaeffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:59.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.676+0000 7f1eaeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:59.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.676+0000 7f1eaf7fe640 1 --2- 172.21.15.40:0/154810038 >> v2:172.21.15.189:3300/0 conn(0x7f1eb00a1c60 0x7f1eb00a31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:59.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.676+0000 7f1eaeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:59.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.676+0000 7f1eaeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:59.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.676+0000 7f1eaf7fe640 10 monclient(hunting): get_auth_request con 0x7f1eb00a1c60 auth_method 0 2024-04-22T08:12:59.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.676+0000 7f1eaf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:59.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.676+0000 7f1eaf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:59.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.676+0000 7f1eaf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:59.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.676+0000 7f1eaf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:59.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.676+0000 7f1eaf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:59.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.676+0000 7f1eaf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:59.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.676+0000 7f1eaeffd640 10 monclient(hunting): handle_auth_done global_id 4392 payload 274 2024-04-22T08:12:59.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.676+0000 7f1eaeffd640 1 -- 172.21.15.40:0/154810038 >> v2:172.21.15.189:3300/0 conn(0x7f1eb00a1c60 msgr2=0x7f1eb00a31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:59.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.676+0000 7f1eaeffd640 1 --2- 172.21.15.40:0/154810038 >> v2:172.21.15.189:3300/0 conn(0x7f1eb00a1c60 0x7f1eb00a31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:59.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.676+0000 7f1eaeffd640 1 -- 172.21.15.40:0/154810038 >> v1:172.21.15.189:6789/0 conn(0x7f1eb00a24d0 legacy=0x7f1eb01360f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:12:59.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.676+0000 7f1eaeffd640 10 monclient: _finish_hunting 0 2024-04-22T08:12:59.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.676+0000 7f1eaeffd640 1 monclient: found mon.noname-a 2024-04-22T08:12:59.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.676+0000 7f1eaeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:59.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.676+0000 7f1eaeffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:12:59.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.676+0000 7f1eaeffd640 1 -- 172.21.15.40:0/154810038 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb00a0450 con 0x7f1eb00a2040 2024-04-22T08:12:59.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.679+0000 7f1eaeffd640 1 -- 172.21.15.40:0/154810038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1eb00a2040 msgr2=0x7f1eb0135bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:59.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.679+0000 7f1eaeffd640 1 --2- 172.21.15.40:0/154810038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1eb00a2040 0x7f1eb0135bb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1eb00a5720 tx=0x7f1e9802b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c243cb7cd8bcdf3e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:59.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.679+0000 7f1eae7fc640 1 -- 172.21.15.40:0/154810038 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e98031960 con 0x7f1eb00a2040 2024-04-22T08:12:59.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.679+0000 7f1eae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:59.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.679+0000 7f1eae7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:12:59.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.679+0000 7f1eae7fc640 10 monclient: dump: 2024-04-22T08:12:59.689 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:59.689 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:59.690 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:59.690 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:59.690 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:59.690 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:59.690 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:59.690 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:59.690 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:59.690 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:59.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.679+0000 7f1eae7fc640 10 monclient: _finish_auth 0 2024-04-22T08:12:59.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.679+0000 7f1eae7fc640 10 monclient: _check_auth_tickets 2024-04-22T08:12:59.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.679+0000 7f1eae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:59.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.679+0000 7f1eae7fc640 1 -- 172.21.15.40:0/154810038 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e98031ac0 con 0x7f1eb00a2040 2024-04-22T08:12:59.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.679+0000 7f1eae7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:59.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.679+0000 7f1eae7fc640 1 -- 172.21.15.40:0/154810038 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e98031e50 con 0x7f1eb00a2040 2024-04-22T08:12:59.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.679+0000 7f1eb5ce0640 5 monclient: authenticate success, global_id 4392 2024-04-22T08:12:59.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.679+0000 7f1eb5ce0640 10 monclient: get_monmap_and_config success 2024-04-22T08:12:59.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.679+0000 7f1eb5ce0640 4 set_mon_vals no callback set 2024-04-22T08:12:59.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.679+0000 7f1eae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:59.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.679+0000 7f1eae7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:12:59.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.679+0000 7f1eae7fc640 10 monclient: dump: 2024-04-22T08:12:59.691 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:59.691 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:59.691 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:59.691 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:59.691 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:59.691 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:59.691 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:59.691 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:59.692 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:59.692 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:59.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.680+0000 7f1eb5ce0640 10 monclient: shutdown 2024-04-22T08:12:59.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.680+0000 7f1eb5ce0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:59.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.680+0000 7f1eb5ce0640 1 -- 172.21.15.40:0/154810038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1eb00a2040 msgr2=0x7f1eb0135bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:59.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.680+0000 7f1eb5ce0640 1 --2- 172.21.15.40:0/154810038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1eb00a2040 0x7f1eb0135bb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1eb00a5720 tx=0x7f1e9802b7f0 comp rx=0 tx=0).stop 2024-04-22T08:12:59.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.680+0000 7f1eb5ce0640 1 -- 172.21.15.40:0/154810038 shutdown_connections 2024-04-22T08:12:59.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.680+0000 7f1eb5ce0640 1 --2- 172.21.15.40:0/154810038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1eb00a2040 0x7f1eb0135bb0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:59.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.680+0000 7f1eb5ce0640 1 --2- 172.21.15.40:0/154810038 >> v2:172.21.15.189:3300/0 conn(0x7f1eb00a1c60 0x7f1eb00a31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:59.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.680+0000 7f1eb5ce0640 1 -- 172.21.15.40:0/154810038 >> 172.21.15.40:0/154810038 conn(0x7f1eb00a89a0 msgr2=0x7f1eb00a4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:59.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.680+0000 7f1eb5ce0640 1 -- 172.21.15.40:0/154810038 shutdown_connections 2024-04-22T08:12:59.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.680+0000 7f1eb5ce0640 1 -- 172.21.15.40:0/154810038 wait complete. 2024-04-22T08:12:59.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.681+0000 7f1eb5ce0640 10 monclient: build_initial_monmap 2024-04-22T08:12:59.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.681+0000 7f1eb5ce0640 10 monclient: monmap: 2024-04-22T08:12:59.693 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:12:59.693 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:12:59.693 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:12:59.693 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:12:59.693 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:12:59.693 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:59.693 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:12:59.693 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:12:59.693 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:12:59.693 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:59.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.681+0000 7f1eb5ce0640 1 Processor -- start 2024-04-22T08:12:59.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.681+0000 7f1eb5ce0640 1 -- start start 2024-04-22T08:12:59.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.681+0000 7f1eb5ce0640 10 monclient: init 2024-04-22T08:12:59.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eb5ce0640 10 monclient: _reopen_session rank -1 2024-04-22T08:12:59.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eb5ce0640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-22T08:12:59.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eb5ce0640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1eb00a1c60 0x7f1eb01cb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:59.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eb5ce0640 10 monclient(hunting): picked mon.noname-c con 0x7f1eb00a1c60 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:12:59.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eb5ce0640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1eb00a2040 0x7f1eb01204a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:59.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eb5ce0640 10 monclient(hunting): picked mon.noname-b con 0x7f1eb00a2040 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:12:59.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eb5ce0640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1eb00a24d0 0x7f1eb01209e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:59.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eb5ce0640 10 monclient(hunting): picked mon.noname-a con 0x7f1eb00a24d0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:12:59.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eb5ce0640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:59.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eb5ce0640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb0122ae0 con 0x7f1eb00a24d0 2024-04-22T08:12:59.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eb5ce0640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:59.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eb5ce0640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb00a4790 con 0x7f1eb00a2040 2024-04-22T08:12:59.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eb5ce0640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:59.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eb5ce0640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb0134f80 con 0x7f1eb00a1c60 2024-04-22T08:12:59.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eb5ce0640 10 monclient(hunting): _renew_subs 2024-04-22T08:12:59.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eb5ce0640 10 monclient(hunting): authenticate will time out at 853.675899s 2024-04-22T08:12:59.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eaf7fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1eb00a1c60 0x7f1eb01cb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:59.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eaf7fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1eb00a1c60 0x7f1eb01cb0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:51204/0 (socket says 172.21.15.40:51204) 2024-04-22T08:12:59.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eaf7fe640 1 -- 172.21.15.40:0/4002885319 learned_addr learned my addr 172.21.15.40:0/4002885319 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:12:59.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eaf7fe640 10 monclient(hunting): get_auth_request con 0x7f1eb00a1c60 auth_method 0 2024-04-22T08:12:59.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eaffff640 1 --2- 172.21.15.40:0/4002885319 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1eb00a24d0 0x7f1eb01209e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:59.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eaf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:59.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eaf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:59.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eaf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:59.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eaffff640 10 monclient(hunting): get_auth_request con 0x7f1eb00a24d0 auth_method 0 2024-04-22T08:12:59.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eaffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:59.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eaffff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:59.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eaffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:59.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eaf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:59.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eaf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:59.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eaf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:59.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eaffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:59.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eaffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:59.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eaeffd640 1 --2- 172.21.15.40:0/4002885319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1eb00a2040 0x7f1eb01204a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:59.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eaffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:59.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eaeffd640 10 monclient(hunting): get_auth_request con 0x7f1eb00a2040 auth_method 0 2024-04-22T08:12:59.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eaeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:59.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eaeffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:59.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eaeffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:59.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eaeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:59.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eaeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:59.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eaeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:59.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eaf7fe640 10 monclient(hunting): handle_auth_done global_id 4367 payload 824 2024-04-22T08:12:59.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eaf7fe640 1 -- 172.21.15.40:0/4002885319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1eb00a2040 msgr2=0x7f1eb01204a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:59.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eaf7fe640 1 --2- 172.21.15.40:0/4002885319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1eb00a2040 0x7f1eb01204a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:59.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eaf7fe640 1 -- 172.21.15.40:0/4002885319 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1eb00a24d0 msgr2=0x7f1eb01209e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:59.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eaf7fe640 1 --2- 172.21.15.40:0/4002885319 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1eb00a24d0 0x7f1eb01209e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:12:59.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eaf7fe640 10 monclient: _finish_hunting 0 2024-04-22T08:12:59.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eaf7fe640 1 monclient: found mon.noname-c 2024-04-22T08:12:59.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eaf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:12:59.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eaf7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:12:59.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eaf7fe640 1 -- 172.21.15.40:0/4002885319 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e980094d0 con 0x7f1eb00a1c60 2024-04-22T08:12:59.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.682+0000 7f1eaffff640 1 --2- 172.21.15.40:0/4002885319 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1eb00a24d0 0x7f1eb01209e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:12:59.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.684+0000 7f1eaf7fe640 1 -- 172.21.15.40:0/4002885319 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1eb00a1c60 msgr2=0x7f1eb01cb0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:59.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.684+0000 7f1eaf7fe640 1 --2- 172.21.15.40:0/4002885319 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1eb00a1c60 0x7f1eb01cb0a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1ea000e9b0 tx=0x7f1ea000ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:59.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.684+0000 7f1ea7fff640 1 -- 172.21.15.40:0/4002885319 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea0009670 con 0x7f1eb00a1c60 2024-04-22T08:12:59.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.684+0000 7f1ea7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:59.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.684+0000 7f1ea7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:12:59.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.684+0000 7f1ea7fff640 10 monclient: dump: 2024-04-22T08:12:59.698 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:59.698 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:59.698 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:59.698 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:59.698 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:59.699 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:59.699 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:59.699 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:59.699 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:59.699 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:59.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.684+0000 7f1ea7fff640 10 monclient: _finish_auth 0 2024-04-22T08:12:59.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.684+0000 7f1ea7fff640 10 monclient: _check_auth_tickets 2024-04-22T08:12:59.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.684+0000 7f1ea7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:12:59.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.684+0000 7f1ea7fff640 1 -- 172.21.15.40:0/4002885319 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ea000cbb0 con 0x7f1eb00a1c60 2024-04-22T08:12:59.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.684+0000 7f1eb5ce0640 5 monclient: authenticate success, global_id 4367 2024-04-22T08:12:59.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.684+0000 7f1ea7fff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:12:59.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.684+0000 7f1ea7fff640 1 -- 172.21.15.40:0/4002885319 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea0013470 con 0x7f1eb00a1c60 2024-04-22T08:12:59.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.684+0000 7f1eb5ce0640 10 monclient: _renew_subs 2024-04-22T08:12:59.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.684+0000 7f1eb5ce0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:59.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.684+0000 7f1eb5ce0640 1 -- 172.21.15.40:0/4002885319 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eb0124000 con 0x7f1eb00a1c60 2024-04-22T08:12:59.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.684+0000 7f1eadffb640 4 set_mon_vals no callback set 2024-04-22T08:12:59.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.685+0000 7f1ea7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:59.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.685+0000 7f1ea7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:12:59.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.685+0000 7f1ea7fff640 10 monclient: dump: 2024-04-22T08:12:59.700 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:59.700 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:59.700 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:59.700 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:59.700 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:59.700 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:59.700 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:59.700 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:59.701 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:59.701 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:59.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.685+0000 7f1eb5ce0640 10 monclient: _renew_subs 2024-04-22T08:12:59.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.685+0000 7f1eb5ce0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:12:59.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.685+0000 7f1eb5ce0640 1 -- 172.21.15.40:0/4002885319 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eb0124cc0 con 0x7f1eb00a1c60 2024-04-22T08:12:59.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.685+0000 7f1ea7fff640 1 -- 172.21.15.40:0/4002885319 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f1ea00135d0 con 0x7f1eb00a1c60 2024-04-22T08:12:59.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.685+0000 7f1eb5ce0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:59.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.685+0000 7f1eb5ce0640 1 --2- 172.21.15.40:0/4002885319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1eb01cb5e0 0x7f1eb002b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:59.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.685+0000 7f1eb5ce0640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:59.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.685+0000 7f1eb5ce0640 1 -- 172.21.15.40:0/4002885319 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb0066650 con 0x7f1eb01cb5e0 2024-04-22T08:12:59.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.685+0000 7f1ea7fff640 1 --2- 172.21.15.40:0/4002885319 >> v2:172.21.15.189:6800/30168 conn(0x7f1e900377d0 0x7f1e90039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:59.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.686+0000 7f1ea7fff640 1 -- 172.21.15.40:0/4002885319 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f1ea004a8f0 con 0x7f1eb00a1c60 2024-04-22T08:12:59.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.686+0000 7f1eaeffd640 1 --2- 172.21.15.40:0/4002885319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1eb01cb5e0 0x7f1eb002b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:59.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.686+0000 7f1eaffff640 1 --2- 172.21.15.40:0/4002885319 >> v2:172.21.15.189:6800/30168 conn(0x7f1e900377d0 0x7f1e90039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.686+0000 7f1eaeffd640 10 monclient: get_auth_request con 0x7f1eb01cb5e0 auth_method 0 2024-04-22T08:12:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.686+0000 7f1eaeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.686+0000 7f1eaeffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.686+0000 7f1eaeffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.686+0000 7f1eaffff640 10 monclient: get_auth_request con 0x7f1e900377d0 auth_method 0 2024-04-22T08:12:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.686+0000 7f1eaeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.686+0000 7f1eaeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.686+0000 7f1eaeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.686+0000 7f1eaeffd640 10 monclient(hunting): handle_auth_done global_id 4348 payload 824 2024-04-22T08:12:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.686+0000 7f1eaeffd640 1 -- 172.21.15.40:0/4002885319 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1eb0123a40 con 0x7f1eb01cb5e0 2024-04-22T08:12:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.688+0000 7f1eaffff640 1 --2- 172.21.15.40:0/4002885319 >> v2:172.21.15.189:6800/30168 conn(0x7f1e900377d0 0x7f1e90039c90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1ea800f1d0 tx=0x7f1ea8003d10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.688+0000 7f1eaeffd640 1 -- 172.21.15.40:0/4002885319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1eb01cb5e0 msgr2=0x7f1eb002b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.688+0000 7f1eaeffd640 1 --2- 172.21.15.40:0/4002885319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1eb01cb5e0 0x7f1eb002b200 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1e98035000 tx=0x7f1e980046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.689+0000 7f1ea7fff640 1 -- 172.21.15.40:0/4002885319 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e98031e30 con 0x7f1eb01cb5e0 2024-04-22T08:12:59.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.689+0000 7f1ea7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:59.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.689+0000 7f1ea7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:12:59.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.689+0000 7f1ea7fff640 10 monclient: dump: 2024-04-22T08:12:59.703 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:59.703 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:59.703 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:59.703 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:59.703 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:59.703 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:59.703 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:59.703 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:59.704 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:59.704 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:59.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.689+0000 7f1ea7fff640 1 -- 172.21.15.40:0/4002885319 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f1e98041070 con 0x7f1eb01cb5e0 2024-04-22T08:12:59.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.689+0000 7f1ea7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:12:59.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.689+0000 7f1ea7fff640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:12:59.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.689+0000 7f1ea7fff640 1 -- 172.21.15.40:0/4002885319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1eb01cb5e0 msgr2=0x7f1eb002b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:59.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.689+0000 7f1ea7fff640 1 --2- 172.21.15.40:0/4002885319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1eb01cb5e0 0x7f1eb002b200 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1e98035000 tx=0x7f1e980046c0 comp rx=0 tx=0).stop 2024-04-22T08:12:59.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.689+0000 7f1ea7fff640 1 -- 172.21.15.40:0/4002885319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1eb01cb5e0 msgr2=0x7f1eb002b200 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:12:59.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.689+0000 7f1ea7fff640 1 --2- 172.21.15.40:0/4002885319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1eb01cb5e0 0x7f1eb002b200 secure :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1e98035000 tx=0x7f1e980046c0 comp rx=0 tx=0).stop 2024-04-22T08:12:59.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.695+0000 7f1eb5ce0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:12:59.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.695+0000 7f1eb5ce0640 1 --2- 172.21.15.40:0/4002885319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1eb002b740 0x7f1eb002bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:12:59.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.695+0000 7f1eb5ce0640 10 monclient(hunting): start opening mon connection 2024-04-22T08:12:59.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.695+0000 7f1eb5ce0640 1 -- 172.21.15.40:0/4002885319 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb0136800 con 0x7f1eb002b740 2024-04-22T08:12:59.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.696+0000 7f1eaeffd640 1 --2- 172.21.15.40:0/4002885319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1eb002b740 0x7f1eb002bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:12:59.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.696+0000 7f1eaeffd640 10 monclient: get_auth_request con 0x7f1eb002b740 auth_method 0 2024-04-22T08:12:59.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.696+0000 7f1eaeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:12:59.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.696+0000 7f1eaeffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:12:59.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.696+0000 7f1eaeffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:12:59.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.696+0000 7f1eaeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:12:59.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.696+0000 7f1eaeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:12:59.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.696+0000 7f1eaeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:12:59.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.696+0000 7f1eaeffd640 10 monclient(hunting): handle_auth_done global_id 4351 payload 824 2024-04-22T08:12:59.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.696+0000 7f1eaeffd640 1 -- 172.21.15.40:0/4002885319 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1eb004e9b0 con 0x7f1eb002b740 2024-04-22T08:12:59.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.698+0000 7f1eaeffd640 1 -- 172.21.15.40:0/4002885319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1eb002b740 msgr2=0x7f1eb002bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:12:59.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.698+0000 7f1eaeffd640 1 --2- 172.21.15.40:0/4002885319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1eb002b740 0x7f1eb002bb50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1e98032040 tx=0x7f1e98009240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:12:59.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.699+0000 7f1ea7fff640 1 -- 172.21.15.40:0/4002885319 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e98004220 con 0x7f1eb002b740 2024-04-22T08:12:59.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.699+0000 7f1ea7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:12:59.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.699+0000 7f1ea7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:12:59.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.699+0000 7f1ea7fff640 10 monclient: dump: 2024-04-22T08:12:59.708 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:12:59.708 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:12:59.708 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:59.708 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:12:59.708 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:12:59.708 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:12:59.708 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:12:59.708 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:12:59.708 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:12:59.708 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:12:59.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.699+0000 7f1ea7fff640 1 -- 172.21.15.40:0/4002885319 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4801 (secure 0 0 0) 0x7f1e98004380 con 0x7f1eb002b740 2024-04-22T08:12:59.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.699+0000 7f1ea7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:12:59.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.699+0000 7f1ea7fff640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:12:59.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.699+0000 7f1ea7fff640 1 -- 172.21.15.40:0/4002885319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1eb002b740 msgr2=0x7f1eb002bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:59.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.699+0000 7f1ea7fff640 1 --2- 172.21.15.40:0/4002885319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1eb002b740 0x7f1eb002bb50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1e98032040 tx=0x7f1e98009240 comp rx=0 tx=0).stop 2024-04-22T08:12:59.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.699+0000 7f1ea7fff640 1 -- 172.21.15.40:0/4002885319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1eb002b740 msgr2=0x7f1eb002bb50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:12:59.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.699+0000 7f1ea7fff640 1 --2- 172.21.15.40:0/4002885319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1eb002b740 0x7f1eb002bb50 secure :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1e98032040 tx=0x7f1e98009240 comp rx=0 tx=0).stop 2024-04-22T08:12:59.709 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:12:59.709 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:12:59.709 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:12:59.709 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:12:59.709 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 8, 2024-04-22T08:12:59.709 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:12:59.709 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:12:59.710 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:12:59.710 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:12:59.710 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:12:59.710 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 23, 2024-04-22T08:12:59.710 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:12:59.710 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:12:59.710 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:12:59.710 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:12:59.710 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:12:59.710 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:12:59.710 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:12:59.710 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:12:59.710 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:12:59.710 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:12:59.711 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:12:59.711 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:12:59.711 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:12:59.711 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:12:59.711 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:12:59.711 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:12:59.711 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:12:59.711 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:12:59.711 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:12:59.711 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:12:59.711 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:12:59.711 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:12:59.711 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:12:59.712 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:12:59.712 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:12:59.712 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:12:59.712 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:12:59.712 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:12:59.712 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:12:59.712 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:12:59.712 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:12:59.712 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:12:59.712 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:12:59.712 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:12:59.712 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:12:59.712 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:12:59.712 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:12:59.713 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:12:59.713 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:12:59.713 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:12:59.713 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:12:59.713 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:12:59.713 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:12:59.713 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:12:59.713 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:12:59.713 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:12:59.713 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:12:59.713 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:12:59.713 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:12:59.713 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:12:59.713 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:12:59.714 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:12:59.714 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:12:59.714 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:12:59.714 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:12:59.714 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:12:59.714 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:12:59.714 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:12:59.714 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:12:59.714 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:12:59.714 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:12:59.714 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:12:59.714 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:12:59.714 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:12:59.714 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:12:59.715 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:12:59.715 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:12:59.715 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:12:59.715 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:12:59.715 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:12:59.715 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:12:59.715 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:12:59.715 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:12:59.715 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:12:59.715 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:12:59.715 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:12:59.715 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:12:59.715 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:12:59.715 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:12:59.715 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:12:59.716 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:12:59.716 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:12:59.716 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:12:59.716 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:12:59.716 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:12:59.716 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:12:59.716 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:12:59.716 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:12:59.716 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:12:59.716 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:12:59.716 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:12:59.716 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:12:59.716 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:12:59.716 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:12:59.717 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:12:59.717 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:12:59.717 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:12:59.717 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:12:59.717 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:12:59.717 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:12:59.717 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:12:59.717 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:12:59.717 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:12:59.717 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:12:59.717 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:12:59.717 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:12:59.717 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:12:59.717 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:12:59.717 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:12:59.718 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:12:59.718 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:12:59.718 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:12:59.718 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:12:59.718 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:12:59.718 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:12:59.718 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:12:59.718 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:12:59.718 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:12:59.718 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:12:59.718 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:12:59.718 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:12:59.718 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:12:59.718 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:12:59.719 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:12:59.719 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:12:59.719 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:12:59.719 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:12:59.719 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:12:59.719 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:12:59.719 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:12:59.719 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:12:59.719 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:12:59.719 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:12:59.719 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:12:59.719 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:12:59.719 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:12:59.719 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:12:59.719 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:12:59.720 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:12:59.720 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:12:59.720 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:12:59.720 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:12:59.720 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 3 2024-04-22T08:12:59.720 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:12:59.720 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:12:59.720 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:12:59.720 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:12:59.720 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:12:59.720 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:12:59.720 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 5 2024-04-22T08:12:59.720 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:12:59.720 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:12:59.720 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": [ 2024-04-22T08:12:59.721 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:12:59.721 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:12:59.721 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:12:59.721 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:12:59.721 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:12:59.721 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:12:59.721 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:12:59.721 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:12:59.721 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:12:59.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.700+0000 7f1eb5ce0640 1 -- 172.21.15.40:0/4002885319 >> v2:172.21.15.189:6800/30168 conn(0x7f1e900377d0 msgr2=0x7f1e90039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:59.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.700+0000 7f1eb5ce0640 1 --2- 172.21.15.40:0/4002885319 >> v2:172.21.15.189:6800/30168 conn(0x7f1e900377d0 0x7f1e90039c90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1ea800f1d0 tx=0x7f1ea8003d10 comp rx=0 tx=0).stop 2024-04-22T08:12:59.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.700+0000 7f1eb5ce0640 10 monclient: shutdown 2024-04-22T08:12:59.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.700+0000 7f1eb5ce0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:12:59.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.700+0000 7f1eb5ce0640 1 -- 172.21.15.40:0/4002885319 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1eb00a1c60 msgr2=0x7f1eb01cb0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:12:59.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.700+0000 7f1eb5ce0640 1 --2- 172.21.15.40:0/4002885319 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1eb00a1c60 0x7f1eb01cb0a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1ea000e9b0 tx=0x7f1ea000ee70 comp rx=0 tx=0).stop 2024-04-22T08:12:59.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.700+0000 7f1eaffff640 1 -- 172.21.15.40:0/4002885319 reap_dead start 2024-04-22T08:12:59.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.700+0000 7f1eaffff640 1 -- 172.21.15.40:0/4002885319 reap_dead start 2024-04-22T08:12:59.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.700+0000 7f1eb5ce0640 1 -- 172.21.15.40:0/4002885319 shutdown_connections 2024-04-22T08:12:59.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.700+0000 7f1eb5ce0640 1 -- 172.21.15.40:0/4002885319 >> 172.21.15.40:0/4002885319 conn(0x7f1eb00a89a0 msgr2=0x7f1eb0135780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:12:59.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.700+0000 7f1eb5ce0640 1 -- 172.21.15.40:0/4002885319 shutdown_connections 2024-04-22T08:12:59.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:12:59.700+0000 7f1eb5ce0640 1 -- 172.21.15.40:0/4002885319 wait complete. 2024-04-22T08:12:59.724 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:12:59.724 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-04-22T08:12:59.724 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2024-04-22T08:12:59.724 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-04-22T08:12:59.724 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-04-22T08:12:59.725 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-22T08:12:59.725 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T08:12:59.739 INFO:tasks.ceph.mon.b:Stopped 2024-04-22T08:12:59.739 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-04-22T08:12:59.740 INFO:tasks.ceph.mon.a:Sent signal 19 2024-04-22T08:12:59.740 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-04-22T08:13:14.741 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-04-22T08:13:14.741 INFO:tasks.ceph.mon.a:Sent signal 18 2024-04-22T08:13:14.741 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-22T08:13:14.741 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:13:14.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:14.853+0000 7fd54c11a640 10 monclient: get_monmap_and_config 2024-04-22T08:13:14.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:14.853+0000 7fd54c11a640 10 monclient: build_initial_monmap 2024-04-22T08:13:14.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:14.853+0000 7fd54c11a640 10 monclient: monmap: 2024-04-22T08:13:14.853 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:13:14.853 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:14.853 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:13:14.854344+0000 2024-04-22T08:13:14.853 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:13:14.854344+0000 2024-04-22T08:13:14.853 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:13:14.853 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:14.853 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:13:14.854 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:13:14.854 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:13:14.854 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:13:14.854 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:13:14.854 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:14.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:14.854+0000 7fd54c11a640 1 Processor -- start 2024-04-22T08:13:14.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:14.854+0000 7fd54c11a640 1 -- start start 2024-04-22T08:13:14.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:14.854+0000 7fd54c11a640 10 monclient: init 2024-04-22T08:13:14.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:14.854+0000 7fd54c11a640 10 monclient: _reopen_session rank -1 2024-04-22T08:13:14.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:14.854+0000 7fd54c11a640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-04-22T08:13:14.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:14.854+0000 7fd54c11a640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd544133e80 0x7fd544134260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:14.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:14.854+0000 7fd54c11a640 10 monclient(hunting): picked mon.noname-c con 0x7fd544133e80 addr v2:172.21.15.189:3300/0 2024-04-22T08:13:14.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:14.854+0000 7fd54c11a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd54409cb40 addr v1:172.21.15.40:6789/0 2024-04-22T08:13:14.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:14.854+0000 7fd54c11a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd5440a20b0 addr v1:172.21.15.189:6789/0 2024-04-22T08:13:14.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:14.854+0000 7fd54c11a640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd54409de80 con 0x7fd54409cb40 2024-04-22T08:13:14.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:14.854+0000 7fd54c11a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd54409f130 con 0x7fd5440a20b0 2024-04-22T08:13:14.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:14.854+0000 7fd54c11a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:14.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:14.854+0000 7fd54c11a640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54409f410 con 0x7fd544133e80 2024-04-22T08:13:14.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:14.854+0000 7fd54c11a640 10 monclient(hunting): _renew_subs 2024-04-22T08:13:14.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:14.854+0000 7fd54c11a640 10 monclient(hunting): authenticate will time out at 868.848678s 2024-04-22T08:13:14.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:14.855+0000 7fd549eb6640 1 -- >> v2:172.21.15.189:3300/0 conn(0x7fd544133e80 msgr2=0x7fd544134260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.189:3300/0 2024-04-22T08:13:14.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:14.855+0000 7fd54a6b7640 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fd5440a20b0 legacy=0x7fd54413bd10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-04-22T08:13:14.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:14.855+0000 7fd549eb6640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd544133e80 0x7fd544134260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:13:14.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:14.855+0000 7fd5496b5640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fd54409cb40 0x7fd5440a19a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41624/0 (socket says 172.21.15.40:41624) 2024-04-22T08:13:14.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:14.855+0000 7fd5496b5640 1 -- 172.21.15.40:0/2775500780 learned_addr learned my addr 172.21.15.40:0/2775500780 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:13:14.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:14.857+0000 7fd5496b5640 1 -- 172.21.15.40:0/2775500780 >> v1:172.21.15.40:6789/0 conn(0x7fd54409cb40 legacy=0x7fd5440a19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:13:15.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:15.056+0000 7fd549eb6640 1 -- 172.21.15.40:0/2775500780 >> v2:172.21.15.189:3300/0 conn(0x7fd544133e80 msgr2=0x7fd544134260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.189:3300/0 2024-04-22T08:13:15.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:15.056+0000 7fd54a6b7640 1 -- 172.21.15.40:0/2775500780 >> v1:172.21.15.189:6789/0 conn(0x7fd5440a20b0 legacy=0x7fd54413bd10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-04-22T08:13:15.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:15.056+0000 7fd549eb6640 1 --2- 172.21.15.40:0/2775500780 >> v2:172.21.15.189:3300/0 conn(0x7fd544133e80 0x7fd544134260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:13:15.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:15.457+0000 7fd549eb6640 1 -- 172.21.15.40:0/2775500780 >> v2:172.21.15.189:3300/0 conn(0x7fd544133e80 msgr2=0x7fd544134260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.189:3300/0 2024-04-22T08:13:15.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:15.457+0000 7fd54a6b7640 1 -- 172.21.15.40:0/2775500780 >> v1:172.21.15.189:6789/0 conn(0x7fd5440a20b0 legacy=0x7fd54413bd10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-04-22T08:13:15.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:15.457+0000 7fd549eb6640 1 --2- 172.21.15.40:0/2775500780 >> v2:172.21.15.189:3300/0 conn(0x7fd544133e80 0x7fd544134260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:13:16.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:16.258+0000 7fd54a6b7640 1 -- 172.21.15.40:0/2775500780 >> v1:172.21.15.189:6789/0 conn(0x7fd5440a20b0 legacy=0x7fd54413bd10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-04-22T08:13:16.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:16.258+0000 7fd549eb6640 1 -- 172.21.15.40:0/2775500780 >> v2:172.21.15.189:3300/0 conn(0x7fd544133e80 msgr2=0x7fd544134260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.189:3300/0 2024-04-22T08:13:16.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:16.258+0000 7fd549eb6640 1 --2- 172.21.15.40:0/2775500780 >> v2:172.21.15.189:3300/0 conn(0x7fd544133e80 0x7fd544134260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T08:13:17.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:17.855+0000 7fd53b7fe640 10 monclient(hunting): tick 2024-04-22T08:13:17.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:17.855+0000 7fd53b7fe640 10 monclient(hunting): _check_auth_tickets 2024-04-22T08:13:17.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:17.855+0000 7fd53b7fe640 1 monclient(hunting): continuing hunt 2024-04-22T08:13:17.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:17.855+0000 7fd53b7fe640 10 monclient(hunting): _reopen_session rank -1 2024-04-22T08:13:17.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:17.855+0000 7fd53b7fe640 1 -- 172.21.15.40:0/2775500780 >> v2:172.21.15.189:3300/0 conn(0x7fd544133e80 msgr2=0x7fd544134260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:13:17.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:17.855+0000 7fd53b7fe640 1 --2- 172.21.15.40:0/2775500780 >> v2:172.21.15.189:3300/0 conn(0x7fd544133e80 0x7fd544134260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:17.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:17.855+0000 7fd53b7fe640 1 -- 172.21.15.40:0/2775500780 >> v1:172.21.15.189:6789/0 conn(0x7fd5440a20b0 legacy=0x7fd54413bd10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:13:17.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:17.855+0000 7fd53b7fe640 1 -- 172.21.15.40:0/2775500780 >> v1:172.21.15.40:6789/0 conn(0x7fd54409cb40 legacy=0x7fd5440a19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:17.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:17.855+0000 7fd53b7fe640 10 monclient: _add_conns ranks=[3,2] 2024-04-22T08:13:17.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:17.855+0000 7fd53b7fe640 1 --2- 172.21.15.40:0/2775500780 >> v2:172.21.15.40:3300/0 conn(0x7fd528003740 0x7fd528005bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:17.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:17.855+0000 7fd53b7fe640 10 monclient(hunting): picked mon.noname-a con 0x7fd528003740 addr v2:172.21.15.40:3300/0 2024-04-22T08:13:17.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:17.855+0000 7fd53b7fe640 1 --2- 172.21.15.40:0/2775500780 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd528006250 0x7fd528008710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:17.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:17.855+0000 7fd53b7fe640 10 monclient(hunting): picked mon.noname-b con 0x7fd528006250 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:13:17.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:17.855+0000 7fd53b7fe640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:17.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:17.855+0000 7fd53b7fe640 1 -- 172.21.15.40:0/2775500780 --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54409f410 con 0x7fd528003740 2024-04-22T08:13:17.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:17.855+0000 7fd53b7fe640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:17.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:17.855+0000 7fd53b7fe640 1 -- 172.21.15.40:0/2775500780 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd528008e40 con 0x7fd528006250 2024-04-22T08:13:17.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:17.855+0000 7fd53b7fe640 10 monclient(hunting): _renew_subs 2024-04-22T08:13:17.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:17.855+0000 7fd5496b5640 1 --2- 172.21.15.40:0/2775500780 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd528006250 0x7fd528008710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:17.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:17.855+0000 7fd5496b5640 10 monclient(hunting): get_auth_request con 0x7fd528006250 auth_method 0 2024-04-22T08:13:17.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:17.855+0000 7fd5496b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:17.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:17.855+0000 7fd5496b5640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:17.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:17.855+0000 7fd5496b5640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:17.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:17.855+0000 7fd549eb6640 1 --2- 172.21.15.40:0/2775500780 >> v2:172.21.15.40:3300/0 conn(0x7fd528003740 0x7fd528005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:17.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:17.855+0000 7fd549eb6640 10 monclient(hunting): get_auth_request con 0x7fd528003740 auth_method 0 2024-04-22T08:13:17.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:17.855+0000 7fd549eb6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:17.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:17.855+0000 7fd549eb6640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:17.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:17.855+0000 7fd549eb6640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:17.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:17.855+0000 7fd5496b5640 1 -- 172.21.15.40:0/2775500780 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd528006250 msgr2=0x7fd528008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:13:17.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:17.855+0000 7fd5496b5640 1 -- 172.21.15.40:0/2775500780 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd528006250 msgr2=0x7fd528008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:13:17.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:17.855+0000 7fd5496b5640 1 --2- 172.21.15.40:0/2775500780 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd528006250 0x7fd528008710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:13:17.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:17.855+0000 7fd5496b5640 1 --2- 172.21.15.40:0/2775500780 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd528006250 0x7fd528008710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:13:17.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:17.855+0000 7fd549eb6640 1 -- 172.21.15.40:0/2775500780 >> v2:172.21.15.40:3300/0 conn(0x7fd528003740 msgr2=0x7fd528005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:13:17.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:17.855+0000 7fd549eb6640 1 -- 172.21.15.40:0/2775500780 >> v2:172.21.15.40:3300/0 conn(0x7fd528003740 msgr2=0x7fd528005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:13:17.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:17.855+0000 7fd549eb6640 1 --2- 172.21.15.40:0/2775500780 >> v2:172.21.15.40:3300/0 conn(0x7fd528003740 0x7fd528005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:13:17.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:17.855+0000 7fd549eb6640 1 --2- 172.21.15.40:0/2775500780 >> v2:172.21.15.40:3300/0 conn(0x7fd528003740 0x7fd528005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:13:18.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:18.056+0000 7fd5496b5640 1 --2- 172.21.15.40:0/2775500780 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd528006250 0x7fd528008710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:18.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:18.056+0000 7fd549eb6640 1 --2- 172.21.15.40:0/2775500780 >> v2:172.21.15.40:3300/0 conn(0x7fd528003740 0x7fd528005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:18.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:18.056+0000 7fd5496b5640 10 monclient(hunting): get_auth_request con 0x7fd528006250 auth_method 0 2024-04-22T08:13:18.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:18.056+0000 7fd5496b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:18.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:18.056+0000 7fd5496b5640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:18.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:18.056+0000 7fd5496b5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:13:18.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:18.056+0000 7fd549eb6640 10 monclient(hunting): get_auth_request con 0x7fd528003740 auth_method 0 2024-04-22T08:13:18.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:18.056+0000 7fd549eb6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:18.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:18.056+0000 7fd549eb6640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:18.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:18.056+0000 7fd549eb6640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:13:18.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:18.056+0000 7fd5496b5640 1 -- 172.21.15.40:0/2775500780 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd528006250 msgr2=0x7fd528008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:13:18.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:18.056+0000 7fd5496b5640 1 -- 172.21.15.40:0/2775500780 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd528006250 msgr2=0x7fd528008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:13:18.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:18.056+0000 7fd5496b5640 1 --2- 172.21.15.40:0/2775500780 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd528006250 0x7fd528008710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:13:18.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:18.056+0000 7fd5496b5640 1 --2- 172.21.15.40:0/2775500780 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd528006250 0x7fd528008710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:13:18.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:18.056+0000 7fd549eb6640 1 -- 172.21.15.40:0/2775500780 >> v2:172.21.15.40:3300/0 conn(0x7fd528003740 msgr2=0x7fd528005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:13:18.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:18.056+0000 7fd549eb6640 1 -- 172.21.15.40:0/2775500780 >> v2:172.21.15.40:3300/0 conn(0x7fd528003740 msgr2=0x7fd528005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:13:18.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:18.056+0000 7fd549eb6640 1 --2- 172.21.15.40:0/2775500780 >> v2:172.21.15.40:3300/0 conn(0x7fd528003740 0x7fd528005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:13:18.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:18.056+0000 7fd549eb6640 1 --2- 172.21.15.40:0/2775500780 >> v2:172.21.15.40:3300/0 conn(0x7fd528003740 0x7fd528005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:13:18.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:18.457+0000 7fd549eb6640 1 --2- 172.21.15.40:0/2775500780 >> v2:172.21.15.40:3300/0 conn(0x7fd528003740 0x7fd528005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:18.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:18.457+0000 7fd5496b5640 1 --2- 172.21.15.40:0/2775500780 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd528006250 0x7fd528008710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:18.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:18.457+0000 7fd549eb6640 10 monclient(hunting): get_auth_request con 0x7fd528003740 auth_method 0 2024-04-22T08:13:18.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:18.457+0000 7fd549eb6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:18.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:18.457+0000 7fd549eb6640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:18.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:18.457+0000 7fd549eb6640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:13:18.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:18.457+0000 7fd5496b5640 10 monclient(hunting): get_auth_request con 0x7fd528006250 auth_method 0 2024-04-22T08:13:18.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:18.457+0000 7fd5496b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:18.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:18.457+0000 7fd5496b5640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:18.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:18.457+0000 7fd5496b5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:13:18.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:18.457+0000 7fd549eb6640 1 -- 172.21.15.40:0/2775500780 >> v2:172.21.15.40:3300/0 conn(0x7fd528003740 msgr2=0x7fd528005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:13:18.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:18.457+0000 7fd549eb6640 1 -- 172.21.15.40:0/2775500780 >> v2:172.21.15.40:3300/0 conn(0x7fd528003740 msgr2=0x7fd528005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:13:18.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:18.457+0000 7fd549eb6640 1 --2- 172.21.15.40:0/2775500780 >> v2:172.21.15.40:3300/0 conn(0x7fd528003740 0x7fd528005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:13:18.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:18.457+0000 7fd5496b5640 1 -- 172.21.15.40:0/2775500780 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd528006250 msgr2=0x7fd528008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:13:18.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:18.457+0000 7fd5496b5640 1 -- 172.21.15.40:0/2775500780 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd528006250 msgr2=0x7fd528008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:13:18.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:18.457+0000 7fd549eb6640 1 --2- 172.21.15.40:0/2775500780 >> v2:172.21.15.40:3300/0 conn(0x7fd528003740 0x7fd528005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:13:18.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:18.457+0000 7fd5496b5640 1 --2- 172.21.15.40:0/2775500780 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd528006250 0x7fd528008710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:13:18.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:18.457+0000 7fd5496b5640 1 --2- 172.21.15.40:0/2775500780 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd528006250 0x7fd528008710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:13:19.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:19.258+0000 7fd549eb6640 1 --2- 172.21.15.40:0/2775500780 >> v2:172.21.15.40:3300/0 conn(0x7fd528003740 0x7fd528005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:19.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:19.258+0000 7fd5496b5640 1 --2- 172.21.15.40:0/2775500780 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd528006250 0x7fd528008710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:19.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:19.258+0000 7fd549eb6640 10 monclient(hunting): get_auth_request con 0x7fd528003740 auth_method 0 2024-04-22T08:13:19.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:19.258+0000 7fd549eb6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:19.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:19.258+0000 7fd549eb6640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:19.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:19.258+0000 7fd549eb6640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:13:19.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:19.258+0000 7fd5496b5640 10 monclient(hunting): get_auth_request con 0x7fd528006250 auth_method 0 2024-04-22T08:13:19.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:19.258+0000 7fd5496b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:19.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:19.258+0000 7fd5496b5640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:19.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:19.258+0000 7fd5496b5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:13:19.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:19.258+0000 7fd549eb6640 1 -- 172.21.15.40:0/2775500780 >> v2:172.21.15.40:3300/0 conn(0x7fd528003740 msgr2=0x7fd528005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:13:19.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:19.258+0000 7fd549eb6640 1 -- 172.21.15.40:0/2775500780 >> v2:172.21.15.40:3300/0 conn(0x7fd528003740 msgr2=0x7fd528005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:13:19.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:19.258+0000 7fd549eb6640 1 --2- 172.21.15.40:0/2775500780 >> v2:172.21.15.40:3300/0 conn(0x7fd528003740 0x7fd528005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:13:19.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:19.258+0000 7fd5496b5640 1 -- 172.21.15.40:0/2775500780 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd528006250 msgr2=0x7fd528008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:13:19.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:19.258+0000 7fd5496b5640 1 -- 172.21.15.40:0/2775500780 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd528006250 msgr2=0x7fd528008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:13:19.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:19.258+0000 7fd5496b5640 1 --2- 172.21.15.40:0/2775500780 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd528006250 0x7fd528008710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:13:19.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:19.258+0000 7fd549eb6640 1 --2- 172.21.15.40:0/2775500780 >> v2:172.21.15.40:3300/0 conn(0x7fd528003740 0x7fd528005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T08:13:19.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:19.258+0000 7fd5496b5640 1 --2- 172.21.15.40:0/2775500780 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd528006250 0x7fd528008710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T08:13:20.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd53b7fe640 10 monclient(hunting): tick 2024-04-22T08:13:20.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd53b7fe640 10 monclient(hunting): _check_auth_tickets 2024-04-22T08:13:20.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd53b7fe640 1 monclient(hunting): continuing hunt 2024-04-22T08:13:20.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd53b7fe640 10 monclient(hunting): _reopen_session rank -1 2024-04-22T08:13:20.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd53b7fe640 1 -- 172.21.15.40:0/2775500780 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd528006250 msgr2=0x7fd528008710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:13:20.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd53b7fe640 1 --2- 172.21.15.40:0/2775500780 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd528006250 0x7fd528008710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:20.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd53b7fe640 1 -- 172.21.15.40:0/2775500780 >> v2:172.21.15.40:3300/0 conn(0x7fd528003740 msgr2=0x7fd528005bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:13:20.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd53b7fe640 1 --2- 172.21.15.40:0/2775500780 >> v2:172.21.15.40:3300/0 conn(0x7fd528003740 0x7fd528005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:20.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd53b7fe640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-04-22T08:13:20.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd54a6b7640 1 -- 172.21.15.40:0/2775500780 reap_dead start 2024-04-22T08:13:20.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd53b7fe640 1 --2- 172.21.15.40:0/2775500780 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd528006250 0x7fd5280086f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:20.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd53b7fe640 10 monclient(hunting): picked mon.noname-b con 0x7fd528006250 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:13:20.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd53b7fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd528003b50 addr v1:172.21.15.189:6789/0 2024-04-22T08:13:20.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd53b7fe640 1 --2- 172.21.15.40:0/2775500780 >> v2:172.21.15.40:3300/0 conn(0x7fd52800a920 0x7fd52800cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:20.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd53b7fe640 10 monclient(hunting): picked mon.noname-a con 0x7fd52800a920 addr v2:172.21.15.40:3300/0 2024-04-22T08:13:20.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd53b7fe640 1 -- 172.21.15.40:0/2775500780 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd54409f6f0 con 0x7fd528003b50 2024-04-22T08:13:20.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd53b7fe640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:20.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd53b7fe640 1 -- 172.21.15.40:0/2775500780 --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54409f410 con 0x7fd52800a920 2024-04-22T08:13:20.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd53b7fe640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:20.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd53b7fe640 1 -- 172.21.15.40:0/2775500780 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd528008e40 con 0x7fd528006250 2024-04-22T08:13:20.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd53b7fe640 10 monclient(hunting): _renew_subs 2024-04-22T08:13:20.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd5496b5640 1 -- 172.21.15.40:0/2775500780 >> v1:172.21.15.189:6789/0 conn(0x7fd528003b50 legacy=0x7fd528009200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-04-22T08:13:20.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd549eb6640 1 --2- 172.21.15.40:0/2775500780 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd528006250 0x7fd5280086f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:20.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd549eb6640 10 monclient(hunting): get_auth_request con 0x7fd528006250 auth_method 0 2024-04-22T08:13:20.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd549eb6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:20.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd549eb6640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:20.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd549eb6640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:20.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd54a6b7640 1 --2- 172.21.15.40:0/2775500780 >> v2:172.21.15.40:3300/0 conn(0x7fd52800a920 0x7fd52800cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:20.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd54a6b7640 10 monclient(hunting): get_auth_request con 0x7fd52800a920 auth_method 0 2024-04-22T08:13:20.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd54a6b7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:20.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd54a6b7640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:20.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd54a6b7640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:20.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd549eb6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:20.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd549eb6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:20.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd549eb6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:20.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd54a6b7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:20.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd54a6b7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:20.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd54a6b7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:20.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd549eb6640 10 monclient(hunting): handle_auth_done global_id 4370 payload 274 2024-04-22T08:13:20.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd549eb6640 1 -- 172.21.15.40:0/2775500780 >> v2:172.21.15.40:3300/0 conn(0x7fd52800a920 msgr2=0x7fd52800cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:13:20.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd549eb6640 1 --2- 172.21.15.40:0/2775500780 >> v2:172.21.15.40:3300/0 conn(0x7fd52800a920 0x7fd52800cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:20.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd549eb6640 1 -- 172.21.15.40:0/2775500780 >> v1:172.21.15.189:6789/0 conn(0x7fd528003b50 legacy=0x7fd528009200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:13:20.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd549eb6640 10 monclient: _finish_hunting 0 2024-04-22T08:13:20.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd549eb6640 1 monclient: found mon.noname-b 2024-04-22T08:13:20.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd549eb6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:13:20.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd549eb6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:13:20.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd549eb6640 1 -- 172.21.15.40:0/2775500780 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd54409f130 con 0x7fd528006250 2024-04-22T08:13:20.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.855+0000 7fd54a6b7640 1 --2- 172.21.15.40:0/2775500780 >> v2:172.21.15.40:3300/0 conn(0x7fd52800a920 0x7fd52800cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:13:20.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.858+0000 7fd549eb6640 1 -- 172.21.15.40:0/2775500780 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd528006250 msgr2=0x7fd5280086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:13:20.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.858+0000 7fd549eb6640 1 --2- 172.21.15.40:0/2775500780 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd528006250 0x7fd5280086f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd540003930 tx=0x7fd54002d0e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bea3284298c6bd1b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:20.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.859+0000 7fd548eb4640 1 -- 172.21.15.40:0/2775500780 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd54002dea0 con 0x7fd528006250 2024-04-22T08:13:20.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.859+0000 7fd548eb4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:20.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.859+0000 7fd548eb4640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:13:20.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.859+0000 7fd548eb4640 10 monclient: dump: 2024-04-22T08:13:20.867 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:20.867 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:20.867 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:20.867 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:20.867 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:20.867 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:20.867 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:20.867 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:20.868 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:20.868 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:20.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.859+0000 7fd548eb4640 10 monclient: _finish_auth 0 2024-04-22T08:13:20.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.859+0000 7fd548eb4640 10 monclient: _check_auth_tickets 2024-04-22T08:13:20.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.859+0000 7fd548eb4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:13:20.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.859+0000 7fd548eb4640 1 -- 172.21.15.40:0/2775500780 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd540031070 con 0x7fd528006250 2024-04-22T08:13:20.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.859+0000 7fd54c11a640 5 monclient: authenticate success, global_id 4370 2024-04-22T08:13:20.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.859+0000 7fd54c11a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:13:20.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.859+0000 7fd548eb4640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:13:20.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.859+0000 7fd548eb4640 1 -- 172.21.15.40:0/2775500780 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd540037640 con 0x7fd528006250 2024-04-22T08:13:20.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.859+0000 7fd548eb4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:20.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.859+0000 7fd548eb4640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:13:20.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.859+0000 7fd548eb4640 10 monclient: dump: 2024-04-22T08:13:20.869 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:20.869 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:20.869 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:20.869 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:20.869 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:20.869 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:20.869 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:20.869 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:20.869 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:20.869 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:20.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.859+0000 7fd54c11a640 10 monclient: get_monmap_and_config success 2024-04-22T08:13:20.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.859+0000 7fd54c11a640 4 set_mon_vals no callback set 2024-04-22T08:13:20.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.860+0000 7fd54c11a640 10 monclient: shutdown 2024-04-22T08:13:20.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.860+0000 7fd54c11a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:13:20.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.860+0000 7fd54c11a640 1 -- 172.21.15.40:0/2775500780 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd528006250 msgr2=0x7fd5280086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:20.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.860+0000 7fd54c11a640 1 --2- 172.21.15.40:0/2775500780 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd528006250 0x7fd5280086f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd540003930 tx=0x7fd54002d0e0 comp rx=0 tx=0).stop 2024-04-22T08:13:20.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.860+0000 7fd54c11a640 1 -- 172.21.15.40:0/2775500780 shutdown_connections 2024-04-22T08:13:20.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.860+0000 7fd54c11a640 1 --2- 172.21.15.40:0/2775500780 >> v2:172.21.15.40:3300/0 conn(0x7fd52800a920 0x7fd52800cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:20.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.860+0000 7fd54c11a640 1 --2- 172.21.15.40:0/2775500780 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd528006250 0x7fd5280086f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:20.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.860+0000 7fd54c11a640 1 -- 172.21.15.40:0/2775500780 >> 172.21.15.40:0/2775500780 conn(0x7fd5440a65c0 msgr2=0x7fd5440a2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:13:20.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.860+0000 7fd54c11a640 1 -- 172.21.15.40:0/2775500780 shutdown_connections 2024-04-22T08:13:20.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.860+0000 7fd54c11a640 1 -- 172.21.15.40:0/2775500780 wait complete. 2024-04-22T08:13:20.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.861+0000 7fd54c11a640 10 monclient: build_initial_monmap 2024-04-22T08:13:20.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.861+0000 7fd54c11a640 10 monclient: monmap: 2024-04-22T08:13:20.870 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:13:20.870 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:13:20.870 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:13:20.870 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:13:20.871 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:13:20.871 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:20.871 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:13:20.871 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:13:20.871 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:13:20.871 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:20.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.861+0000 7fd54c11a640 1 Processor -- start 2024-04-22T08:13:20.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.861+0000 7fd54c11a640 1 -- start start 2024-04-22T08:13:20.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.861+0000 7fd54c11a640 10 monclient: init 2024-04-22T08:13:20.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.861+0000 7fd54c11a640 10 monclient: _reopen_session rank -1 2024-04-22T08:13:20.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.861+0000 7fd54c11a640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-22T08:13:20.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.862+0000 7fd54c11a640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd528003b50 0x7fd544131ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:20.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.862+0000 7fd54c11a640 10 monclient(hunting): picked mon.noname-c con 0x7fd528003b50 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:13:20.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.862+0000 7fd54c11a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd528006250 0x7fd5441320e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:20.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.862+0000 7fd54c11a640 10 monclient(hunting): picked mon.noname-a con 0x7fd528006250 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:13:20.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.862+0000 7fd54c11a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd52800a920 0x7fd54411c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:20.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.862+0000 7fd54c11a640 10 monclient(hunting): picked mon.noname-b con 0x7fd52800a920 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:13:20.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.862+0000 7fd54c11a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:20.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.862+0000 7fd54c11a640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd54411efa0 con 0x7fd528006250 2024-04-22T08:13:20.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.862+0000 7fd54c11a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:20.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.862+0000 7fd54c11a640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd544132ff0 con 0x7fd52800a920 2024-04-22T08:13:20.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.862+0000 7fd54c11a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:20.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.862+0000 7fd54c11a640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54411ecc0 con 0x7fd528003b50 2024-04-22T08:13:20.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.862+0000 7fd54c11a640 10 monclient(hunting): _renew_subs 2024-04-22T08:13:20.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.862+0000 7fd54c11a640 10 monclient(hunting): authenticate will time out at 874.855865s 2024-04-22T08:13:20.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.862+0000 7fd54a6b7640 1 -- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd52800a920 msgr2=0x7fd54411c5e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:3300/0 2024-04-22T08:13:20.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.862+0000 7fd549eb6640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd528003b50 0x7fd544131ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:20.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.862+0000 7fd5496b5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd528006250 0x7fd5441320e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:20.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.862+0000 7fd54a6b7640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd52800a920 0x7fd54411c5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:13:20.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.862+0000 7fd5496b5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd528006250 0x7fd5441320e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57928/0 (socket says 172.21.15.40:57928) 2024-04-22T08:13:20.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.862+0000 7fd549eb6640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd528003b50 0x7fd544131ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:34294/0 (socket says 172.21.15.40:34294) 2024-04-22T08:13:20.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.862+0000 7fd5496b5640 1 -- 172.21.15.40:0/3844825093 learned_addr learned my addr 172.21.15.40:0/3844825093 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:13:20.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.862+0000 7fd5496b5640 10 monclient(hunting): get_auth_request con 0x7fd528006250 auth_method 0 2024-04-22T08:13:20.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.862+0000 7fd5496b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:20.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.862+0000 7fd5496b5640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:20.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.862+0000 7fd5496b5640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:20.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.862+0000 7fd549eb6640 10 monclient(hunting): get_auth_request con 0x7fd528003b50 auth_method 0 2024-04-22T08:13:20.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.862+0000 7fd549eb6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:20.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.862+0000 7fd549eb6640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:20.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.862+0000 7fd549eb6640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:20.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.862+0000 7fd5496b5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:20.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.862+0000 7fd5496b5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:20.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.862+0000 7fd5496b5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:20.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.862+0000 7fd549eb6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:20.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.862+0000 7fd549eb6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:20.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.862+0000 7fd549eb6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:20.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.862+0000 7fd549eb6640 10 monclient(hunting): handle_auth_done global_id 4373 payload 824 2024-04-22T08:13:20.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.862+0000 7fd549eb6640 1 -- 172.21.15.40:0/3844825093 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd52800a920 msgr2=0x7fd54411c5e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:13:20.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.862+0000 7fd549eb6640 1 --2- 172.21.15.40:0/3844825093 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd52800a920 0x7fd54411c5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:20.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.862+0000 7fd549eb6640 1 -- 172.21.15.40:0/3844825093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd528006250 msgr2=0x7fd5441320e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:20.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.862+0000 7fd549eb6640 1 --2- 172.21.15.40:0/3844825093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd528006250 0x7fd5441320e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:20.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.862+0000 7fd549eb6640 10 monclient: _finish_hunting 0 2024-04-22T08:13:20.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.862+0000 7fd549eb6640 1 monclient: found mon.noname-c 2024-04-22T08:13:20.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.862+0000 7fd549eb6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:13:20.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.862+0000 7fd549eb6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:13:20.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.862+0000 7fd549eb6640 1 -- 172.21.15.40:0/3844825093 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd53c0078a0 con 0x7fd528003b50 2024-04-22T08:13:20.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.862+0000 7fd5496b5640 1 --2- 172.21.15.40:0/3844825093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd528006250 0x7fd5441320e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:13:20.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.864+0000 7fd549eb6640 1 -- 172.21.15.40:0/3844825093 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd528003b50 msgr2=0x7fd544131ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:13:20.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.864+0000 7fd549eb6640 1 --2- 172.21.15.40:0/3844825093 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd528003b50 0x7fd544131ba0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd540035000 tx=0x7fd540005c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:20.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.865+0000 7fd53a7fc640 1 -- 172.21.15.40:0/3844825093 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd54002dea0 con 0x7fd528003b50 2024-04-22T08:13:20.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.865+0000 7fd53a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:20.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.865+0000 7fd53a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:13:20.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.865+0000 7fd53a7fc640 10 monclient: dump: 2024-04-22T08:13:20.875 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:20.875 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:20.875 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:20.875 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:20.875 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:20.875 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:20.875 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:20.875 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:20.875 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:20.876 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:20.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.865+0000 7fd53a7fc640 10 monclient: _finish_auth 0 2024-04-22T08:13:20.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.865+0000 7fd53a7fc640 10 monclient: _check_auth_tickets 2024-04-22T08:13:20.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.865+0000 7fd53a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:13:20.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.865+0000 7fd54c11a640 5 monclient: authenticate success, global_id 4373 2024-04-22T08:13:20.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.865+0000 7fd54c11a640 10 monclient: _renew_subs 2024-04-22T08:13:20.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.865+0000 7fd53a7fc640 1 -- 172.21.15.40:0/3844825093 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd540042640 con 0x7fd528003b50 2024-04-22T08:13:20.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.865+0000 7fd54c11a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:13:20.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.865+0000 7fd54c11a640 1 -- 172.21.15.40:0/3844825093 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd540009ae0 con 0x7fd528003b50 2024-04-22T08:13:20.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.865+0000 7fd53a7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:13:20.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.865+0000 7fd53a7fc640 1 -- 172.21.15.40:0/3844825093 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5400429f0 con 0x7fd528003b50 2024-04-22T08:13:20.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.865+0000 7fd53a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:20.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.865+0000 7fd53a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:13:20.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.865+0000 7fd53a7fc640 10 monclient: dump: 2024-04-22T08:13:20.877 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:20.877 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:20.877 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:20.877 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:20.877 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:20.877 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:20.877 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:20.877 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:20.877 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:20.877 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:20.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.865+0000 7fd53bfff640 4 set_mon_vals no callback set 2024-04-22T08:13:20.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.865+0000 7fd54c11a640 10 monclient: _renew_subs 2024-04-22T08:13:20.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.865+0000 7fd54c11a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:13:20.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.865+0000 7fd54c11a640 1 -- 172.21.15.40:0/3844825093 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd54409e400 con 0x7fd528003b50 2024-04-22T08:13:20.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.866+0000 7fd53a7fc640 1 -- 172.21.15.40:0/3844825093 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fd540042b50 con 0x7fd528003b50 2024-04-22T08:13:20.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.866+0000 7fd53a7fc640 1 --2- 172.21.15.40:0/3844825093 >> v2:172.21.15.189:6800/30168 conn(0x7fd528038ec0 0x7fd52803b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:20.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.866+0000 7fd53a7fc640 1 -- 172.21.15.40:0/3844825093 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3898+0+0 (secure 0 0 0) 0x7fd54006dba0 con 0x7fd528003b50 2024-04-22T08:13:20.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.866+0000 7fd5496b5640 1 --2- 172.21.15.40:0/3844825093 >> v2:172.21.15.189:6800/30168 conn(0x7fd528038ec0 0x7fd52803b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:20.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.866+0000 7fd54c11a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:13:20.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.866+0000 7fd54c11a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:13:20.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.866+0000 7fd54c11a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:13:20.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.866+0000 7fd54c11a640 1 -- 172.21.15.40:0/3844825093 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5441bcbb0 con 0x7fd528003b50 2024-04-22T08:13:20.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.866+0000 7fd5496b5640 10 monclient: get_auth_request con 0x7fd528038ec0 auth_method 0 2024-04-22T08:13:20.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.869+0000 7fd5496b5640 1 --2- 172.21.15.40:0/3844825093 >> v2:172.21.15.189:6800/30168 conn(0x7fd528038ec0 0x7fd52803b380 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd534007fd0 tx=0x7fd534009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:20.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.871+0000 7fd53a7fc640 1 -- 172.21.15.40:0/3844825093 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7fd5441bcbb0 con 0x7fd528003b50 2024-04-22T08:13:20.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.871+0000 7fd53a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:13:20.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:20.871+0000 7fd53a7fc640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:13:20.883 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[34366]: starting. 2024-04-22T08:13:20.883 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[34366]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-34365 2024-04-22T08:13:20.883 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[34366]: created object 0... 2024-04-22T08:13:20.883 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[34366]: created object 25... 2024-04-22T08:13:20.883 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[34366]: created object 49... 2024-04-22T08:13:20.883 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[34366]: finishing. 2024-04-22T08:13:20.884 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[34366]: shutting down. 2024-04-22T08:13:21.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.129+0000 7fd54c11a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:13:21.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.129+0000 7fd54c11a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:13:21.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.129+0000 7fd54c11a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:13:21.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.129+0000 7fd54c11a640 1 -- 172.21.15.40:0/3844825093 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd54411f1c0 con 0x7fd528003b50 2024-04-22T08:13:21.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.129+0000 7fd53a7fc640 1 -- 172.21.15.40:0/3844825093 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1537 (secure 0 0 0) 0x7fd54411f1c0 con 0x7fd528003b50 2024-04-22T08:13:21.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.129+0000 7fd53a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:13:21.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.129+0000 7fd53a7fc640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:13:21.129 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":12,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:13:21.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.131+0000 7fd54c11a640 1 -- 172.21.15.40:0/3844825093 >> v2:172.21.15.189:6800/30168 conn(0x7fd528038ec0 msgr2=0x7fd52803b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:21.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.131+0000 7fd54c11a640 1 --2- 172.21.15.40:0/3844825093 >> v2:172.21.15.189:6800/30168 conn(0x7fd528038ec0 0x7fd52803b380 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd534007fd0 tx=0x7fd534009040 comp rx=0 tx=0).stop 2024-04-22T08:13:21.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.131+0000 7fd54c11a640 10 monclient: shutdown 2024-04-22T08:13:21.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.131+0000 7fd54c11a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:13:21.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.131+0000 7fd54c11a640 1 -- 172.21.15.40:0/3844825093 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd528003b50 msgr2=0x7fd544131ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:21.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.131+0000 7fd54c11a640 1 --2- 172.21.15.40:0/3844825093 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd528003b50 0x7fd544131ba0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd540035000 tx=0x7fd540005c30 comp rx=0 tx=0).stop 2024-04-22T08:13:21.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.131+0000 7fd54c11a640 1 -- 172.21.15.40:0/3844825093 shutdown_connections 2024-04-22T08:13:21.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.131+0000 7fd54c11a640 1 --2- 172.21.15.40:0/3844825093 >> v2:172.21.15.189:6800/30168 conn(0x7fd528038ec0 0x7fd52803b380 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:21.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.131+0000 7fd54c11a640 1 --2- 172.21.15.40:0/3844825093 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd52800a920 0x7fd54411c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:21.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.131+0000 7fd54c11a640 1 --2- 172.21.15.40:0/3844825093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd528006250 0x7fd5441320e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:21.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.131+0000 7fd54c11a640 1 --2- 172.21.15.40:0/3844825093 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd528003b50 0x7fd544131ba0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:21.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.131+0000 7fd54c11a640 1 -- 172.21.15.40:0/3844825093 >> 172.21.15.40:0/3844825093 conn(0x7fd5440a65c0 msgr2=0x7fd54409ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:13:21.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.131+0000 7fd54c11a640 1 -- 172.21.15.40:0/3844825093 shutdown_connections 2024-04-22T08:13:21.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.131+0000 7fd54c11a640 1 -- 172.21.15.40:0/3844825093 wait complete. 2024-04-22T08:13:21.142 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-22T08:13:21.142 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-04-22T08:13:21.142 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-22T08:13:21.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.243+0000 7f7f0b62d640 10 monclient: get_monmap_and_config 2024-04-22T08:13:21.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.243+0000 7f7f0b62d640 10 monclient: build_initial_monmap 2024-04-22T08:13:21.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.243+0000 7f7f0b62d640 10 monclient: monmap: 2024-04-22T08:13:21.243 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:13:21.243 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:21.243 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:13:21.244017+0000 2024-04-22T08:13:21.243 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:13:21.244017+0000 2024-04-22T08:13:21.243 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:13:21.243 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:21.244 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:13:21.244 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:13:21.244 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:13:21.244 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:13:21.244 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:13:21.244 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:21.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.243+0000 7f7f0b62d640 1 Processor -- start 2024-04-22T08:13:21.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.243+0000 7f7f0b62d640 1 -- start start 2024-04-22T08:13:21.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.243+0000 7f7f0b62d640 10 monclient: init 2024-04-22T08:13:21.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.244+0000 7f7f0b62d640 10 monclient: _reopen_session rank -1 2024-04-22T08:13:21.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.244+0000 7f7f0b62d640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-04-22T08:13:21.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.244+0000 7f7f0b62d640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7f040a1c60 0x7f7f040a31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:21.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.244+0000 7f7f0b62d640 10 monclient(hunting): picked mon.noname-c con 0x7f7f040a1c60 addr v2:172.21.15.189:3300/0 2024-04-22T08:13:21.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.244+0000 7f7f0b62d640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7f040a2040 0x7f7f04135bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:21.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.244+0000 7f7f0b62d640 10 monclient(hunting): picked mon.noname-a con 0x7f7f040a2040 addr v2:172.21.15.40:3300/0 2024-04-22T08:13:21.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.244+0000 7f7f0b62d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7f040a24d0 addr v1:172.21.15.40:6789/0 2024-04-22T08:13:21.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.244+0000 7f7f0b62d640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f040a41f0 con 0x7f7f040a24d0 2024-04-22T08:13:21.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.244+0000 7f7f0b62d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:21.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.244+0000 7f7f0b62d640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f0409fe90 con 0x7f7f040a2040 2024-04-22T08:13:21.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.244+0000 7f7f0b62d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:21.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.244+0000 7f7f0b62d640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f040a0170 con 0x7f7f040a1c60 2024-04-22T08:13:21.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.244+0000 7f7f0b62d640 10 monclient(hunting): _renew_subs 2024-04-22T08:13:21.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.244+0000 7f7f0b62d640 10 monclient(hunting): authenticate will time out at 875.238248s 2024-04-22T08:13:21.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.244+0000 7f7f08bc8640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7f040a2040 0x7f7f04135bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:21.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.244+0000 7f7f093c9640 1 -- >> v2:172.21.15.189:3300/0 conn(0x7f7f040a1c60 msgr2=0x7f7f040a31d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.189:3300/0 2024-04-22T08:13:21.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.244+0000 7f7f08bc8640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7f040a2040 0x7f7f04135bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57930/0 (socket says 172.21.15.40:57930) 2024-04-22T08:13:21.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.244+0000 7f7f08bc8640 1 -- 172.21.15.40:0/2149665191 learned_addr learned my addr 172.21.15.40:0/2149665191 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:13:21.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.244+0000 7f7f09bca640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f7f040a24d0 0x7f7f041360f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:38944/0 (socket says 172.21.15.40:38944) 2024-04-22T08:13:21.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.244+0000 7f7f08bc8640 10 monclient(hunting): get_auth_request con 0x7f7f040a2040 auth_method 0 2024-04-22T08:13:21.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.244+0000 7f7f08bc8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:21.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.244+0000 7f7f08bc8640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:21.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.244+0000 7f7f08bc8640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:21.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.244+0000 7f7f093c9640 1 --2- 172.21.15.40:0/2149665191 >> v2:172.21.15.189:3300/0 conn(0x7f7f040a1c60 0x7f7f040a31d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:13:21.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.244+0000 7f7f08bc8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:21.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.244+0000 7f7f08bc8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:21.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.244+0000 7f7f08bc8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:21.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.244+0000 7f7f08bc8640 10 monclient(hunting): handle_auth_done global_id 4404 payload 274 2024-04-22T08:13:21.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.244+0000 7f7f08bc8640 1 -- 172.21.15.40:0/2149665191 >> v2:172.21.15.189:3300/0 conn(0x7f7f040a1c60 msgr2=0x7f7f040a31d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:13:21.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.244+0000 7f7f08bc8640 1 --2- 172.21.15.40:0/2149665191 >> v2:172.21.15.189:3300/0 conn(0x7f7f040a1c60 0x7f7f040a31d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:21.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.244+0000 7f7f08bc8640 1 -- 172.21.15.40:0/2149665191 >> v1:172.21.15.40:6789/0 conn(0x7f7f040a24d0 legacy=0x7f7f041360f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:13:21.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.244+0000 7f7f08bc8640 10 monclient: _finish_hunting 0 2024-04-22T08:13:21.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.244+0000 7f7f08bc8640 1 monclient: found mon.noname-a 2024-04-22T08:13:21.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.244+0000 7f7f08bc8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:13:21.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.244+0000 7f7f08bc8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:13:21.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.244+0000 7f7f08bc8640 1 -- 172.21.15.40:0/2149665191 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f040a0450 con 0x7f7f040a2040 2024-04-22T08:13:21.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.247+0000 7f7f08bc8640 1 -- 172.21.15.40:0/2149665191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f040a2040 msgr2=0x7f7f04135bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:13:21.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.247+0000 7f7f08bc8640 1 --2- 172.21.15.40:0/2149665191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f040a2040 0x7f7f04135bb0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7f040a5720 tx=0x7f7ef802b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=141477ee474a0cb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:21.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.248+0000 7f7f03fff640 1 -- 172.21.15.40:0/2149665191 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef8031a60 con 0x7f7f040a2040 2024-04-22T08:13:21.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.248+0000 7f7f03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:21.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.248+0000 7f7f03fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:13:21.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.248+0000 7f7f03fff640 10 monclient: dump: 2024-04-22T08:13:21.255 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:21.255 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:21.256 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:21.256 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:21.256 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:21.256 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:21.256 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:21.256 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:21.256 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:21.256 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:21.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.248+0000 7f7f03fff640 10 monclient: _finish_auth 0 2024-04-22T08:13:21.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.248+0000 7f7f03fff640 10 monclient: _check_auth_tickets 2024-04-22T08:13:21.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.248+0000 7f7f03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:13:21.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.248+0000 7f7f0b62d640 5 monclient: authenticate success, global_id 4404 2024-04-22T08:13:21.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.248+0000 7f7f0b62d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:13:21.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.248+0000 7f7f03fff640 1 -- 172.21.15.40:0/2149665191 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ef8031bc0 con 0x7f7f040a2040 2024-04-22T08:13:21.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.248+0000 7f7f03fff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:13:21.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.248+0000 7f7f0b62d640 10 monclient: get_monmap_and_config success 2024-04-22T08:13:21.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.248+0000 7f7f0b62d640 4 set_mon_vals no callback set 2024-04-22T08:13:21.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.248+0000 7f7f03fff640 1 -- 172.21.15.40:0/2149665191 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef8032400 con 0x7f7f040a2040 2024-04-22T08:13:21.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.248+0000 7f7f03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:21.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.248+0000 7f7f03fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:13:21.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.248+0000 7f7f03fff640 10 monclient: dump: 2024-04-22T08:13:21.257 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:21.257 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:21.257 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:21.257 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:21.257 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:21.257 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:21.257 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:21.257 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:21.258 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:21.258 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:21.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.248+0000 7f7f0b62d640 10 monclient: shutdown 2024-04-22T08:13:21.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.248+0000 7f7f0b62d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:13:21.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.248+0000 7f7f0b62d640 1 -- 172.21.15.40:0/2149665191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f040a2040 msgr2=0x7f7f04135bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:21.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.248+0000 7f7f0b62d640 1 --2- 172.21.15.40:0/2149665191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f040a2040 0x7f7f04135bb0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7f040a5720 tx=0x7f7ef802b810 comp rx=0 tx=0).stop 2024-04-22T08:13:21.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.248+0000 7f7f0b62d640 1 -- 172.21.15.40:0/2149665191 shutdown_connections 2024-04-22T08:13:21.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.248+0000 7f7f0b62d640 1 --2- 172.21.15.40:0/2149665191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f040a2040 0x7f7f04135bb0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:21.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.248+0000 7f7f0b62d640 1 --2- 172.21.15.40:0/2149665191 >> v2:172.21.15.189:3300/0 conn(0x7f7f040a1c60 0x7f7f040a31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:21.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.248+0000 7f7f0b62d640 1 -- 172.21.15.40:0/2149665191 >> 172.21.15.40:0/2149665191 conn(0x7f7f040a89a0 msgr2=0x7f7f040a4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:13:21.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.248+0000 7f7f0b62d640 1 -- 172.21.15.40:0/2149665191 shutdown_connections 2024-04-22T08:13:21.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.248+0000 7f7f0b62d640 1 -- 172.21.15.40:0/2149665191 wait complete. 2024-04-22T08:13:21.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.249+0000 7f7f0b62d640 10 monclient: build_initial_monmap 2024-04-22T08:13:21.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.249+0000 7f7f0b62d640 10 monclient: monmap: 2024-04-22T08:13:21.258 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:13:21.259 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:13:21.259 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:13:21.259 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:13:21.259 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:13:21.259 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:21.259 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:13:21.259 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:13:21.259 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:13:21.259 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:21.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.249+0000 7f7f0b62d640 1 Processor -- start 2024-04-22T08:13:21.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.249+0000 7f7f0b62d640 1 -- start start 2024-04-22T08:13:21.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.249+0000 7f7f0b62d640 10 monclient: init 2024-04-22T08:13:21.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.249+0000 7f7f0b62d640 10 monclient: _reopen_session rank -1 2024-04-22T08:13:21.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.249+0000 7f7f0b62d640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-22T08:13:21.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.250+0000 7f7f0b62d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f040a1c60 0x7f7f041cace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:21.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.250+0000 7f7f0b62d640 10 monclient(hunting): picked mon.noname-b con 0x7f7f040a1c60 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:13:21.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.250+0000 7f7f0b62d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f040a2040 0x7f7f041cb220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:21.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.250+0000 7f7f0b62d640 10 monclient(hunting): picked mon.noname-a con 0x7f7f040a2040 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:13:21.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.250+0000 7f7f0b62d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7f040a24d0 0x7f7f04122010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:21.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.250+0000 7f7f0b62d640 10 monclient(hunting): picked mon.noname-c con 0x7f7f040a24d0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:13:21.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.250+0000 7f7f0b62d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:21.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.250+0000 7f7f0b62d640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f041256d0 con 0x7f7f040a2040 2024-04-22T08:13:21.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.250+0000 7f7f0b62d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:21.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.250+0000 7f7f0b62d640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f041247e0 con 0x7f7f040a1c60 2024-04-22T08:13:21.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.250+0000 7f7f0b62d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:21.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.250+0000 7f7f0b62d640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f040a4790 con 0x7f7f040a24d0 2024-04-22T08:13:21.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.250+0000 7f7f0b62d640 10 monclient(hunting): _renew_subs 2024-04-22T08:13:21.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.250+0000 7f7f0b62d640 10 monclient(hunting): authenticate will time out at 875.243823s 2024-04-22T08:13:21.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.250+0000 7f7f08bc8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f040a2040 0x7f7f041cb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:21.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.250+0000 7f7f093c9640 1 -- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f040a1c60 msgr2=0x7f7f041cace0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:3300/0 2024-04-22T08:13:21.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.250+0000 7f7f093c9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f040a1c60 0x7f7f041cace0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:13:21.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.250+0000 7f7f09bca640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7f040a24d0 0x7f7f04122010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:21.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.250+0000 7f7f08bc8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f040a2040 0x7f7f041cb220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57938/0 (socket says 172.21.15.40:57938) 2024-04-22T08:13:21.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.250+0000 7f7f08bc8640 1 -- 172.21.15.40:0/3777640992 learned_addr learned my addr 172.21.15.40:0/3777640992 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:13:21.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.250+0000 7f7f08bc8640 10 monclient(hunting): get_auth_request con 0x7f7f040a2040 auth_method 0 2024-04-22T08:13:21.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.250+0000 7f7f08bc8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:21.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.250+0000 7f7f08bc8640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:21.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.250+0000 7f7f08bc8640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:21.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.250+0000 7f7f09bca640 10 monclient(hunting): get_auth_request con 0x7f7f040a24d0 auth_method 0 2024-04-22T08:13:21.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.250+0000 7f7f09bca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:21.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.250+0000 7f7f09bca640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:21.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.250+0000 7f7f09bca640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:21.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.250+0000 7f7f08bc8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:21.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.250+0000 7f7f08bc8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:21.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.250+0000 7f7f08bc8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:21.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.250+0000 7f7f09bca640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:21.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.250+0000 7f7f09bca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:21.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.250+0000 7f7f09bca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:21.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.250+0000 7f7f08bc8640 10 monclient(hunting): handle_auth_done global_id 4407 payload 824 2024-04-22T08:13:21.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.250+0000 7f7f08bc8640 1 -- 172.21.15.40:0/3777640992 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7f040a24d0 msgr2=0x7f7f04122010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:21.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.250+0000 7f7f08bc8640 1 --2- 172.21.15.40:0/3777640992 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7f040a24d0 0x7f7f04122010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:21.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.250+0000 7f7f08bc8640 1 -- 172.21.15.40:0/3777640992 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f040a1c60 msgr2=0x7f7f041cace0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:13:21.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.250+0000 7f7f08bc8640 1 --2- 172.21.15.40:0/3777640992 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f040a1c60 0x7f7f041cace0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:21.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.250+0000 7f7f08bc8640 10 monclient: _finish_hunting 0 2024-04-22T08:13:21.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.250+0000 7f7f08bc8640 1 monclient: found mon.noname-a 2024-04-22T08:13:21.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.250+0000 7f7f08bc8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:13:21.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.250+0000 7f7f08bc8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:13:21.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.250+0000 7f7f08bc8640 1 -- 172.21.15.40:0/3777640992 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ef80094d0 con 0x7f7f040a2040 2024-04-22T08:13:21.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.252+0000 7f7f08bc8640 1 -- 172.21.15.40:0/3777640992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f040a2040 msgr2=0x7f7f041cb220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:13:21.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.252+0000 7f7f08bc8640 1 --2- 172.21.15.40:0/3777640992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f040a2040 0x7f7f041cb220 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7ef80090b0 tx=0x7f7ef80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:21.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.252+0000 7f7f01ffb640 1 -- 172.21.15.40:0/3777640992 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef8032c90 con 0x7f7f040a2040 2024-04-22T08:13:21.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.252+0000 7f7f01ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:21.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.252+0000 7f7f01ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:13:21.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.252+0000 7f7f01ffb640 10 monclient: dump: 2024-04-22T08:13:21.265 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:21.265 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:21.265 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:21.265 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:21.265 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:21.265 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:21.265 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:21.265 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:21.265 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:21.265 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:21.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.252+0000 7f7f01ffb640 10 monclient: _finish_auth 0 2024-04-22T08:13:21.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.252+0000 7f7f01ffb640 10 monclient: _check_auth_tickets 2024-04-22T08:13:21.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.252+0000 7f7f01ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:13:21.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.252+0000 7f7f01ffb640 1 -- 172.21.15.40:0/3777640992 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ef8032e10 con 0x7f7f040a2040 2024-04-22T08:13:21.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.252+0000 7f7f0b62d640 5 monclient: authenticate success, global_id 4407 2024-04-22T08:13:21.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.252+0000 7f7f0b62d640 10 monclient: _renew_subs 2024-04-22T08:13:21.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.252+0000 7f7f0b62d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:13:21.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.252+0000 7f7f0b62d640 1 -- 172.21.15.40:0/3777640992 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f04070bc0 con 0x7f7f040a2040 2024-04-22T08:13:21.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.252+0000 7f7f01ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:13:21.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.252+0000 7f7f01ffb640 1 -- 172.21.15.40:0/3777640992 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef8045630 con 0x7f7f040a2040 2024-04-22T08:13:21.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.252+0000 7f7f01ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:21.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.252+0000 7f7f01ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:13:21.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.252+0000 7f7f01ffb640 10 monclient: dump: 2024-04-22T08:13:21.266 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:21.266 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:21.266 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:21.266 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:21.267 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:21.267 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:21.267 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:21.267 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:21.267 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:21.267 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:21.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.252+0000 7f7f037fe640 4 set_mon_vals no callback set 2024-04-22T08:13:21.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.253+0000 7f7f0b62d640 10 monclient: _renew_subs 2024-04-22T08:13:21.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.253+0000 7f7f0b62d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:13:21.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.253+0000 7f7f0b62d640 1 -- 172.21.15.40:0/3777640992 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f04123c10 con 0x7f7f040a2040 2024-04-22T08:13:21.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.254+0000 7f7f0b62d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:13:21.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.254+0000 7f7f0b62d640 1 --2- 172.21.15.40:0/3777640992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f04126390 0x7f7f04122550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:21.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.254+0000 7f7f0b62d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:21.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.254+0000 7f7f0b62d640 1 -- 172.21.15.40:0/3777640992 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f04066650 con 0x7f7f04126390 2024-04-22T08:13:21.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.254+0000 7f7f01ffb640 1 -- 172.21.15.40:0/3777640992 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f7ef804d3f0 con 0x7f7f040a2040 2024-04-22T08:13:21.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.254+0000 7f7f093c9640 1 --2- 172.21.15.40:0/3777640992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f04126390 0x7f7f04122550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:21.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.254+0000 7f7f093c9640 10 monclient: get_auth_request con 0x7f7f04126390 auth_method 0 2024-04-22T08:13:21.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.254+0000 7f7f093c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:21.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.254+0000 7f7f093c9640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:21.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.254+0000 7f7f093c9640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:21.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.254+0000 7f7f093c9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:21.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.254+0000 7f7f093c9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:21.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.254+0000 7f7f01ffb640 1 --2- 172.21.15.40:0/3777640992 >> v2:172.21.15.189:6800/30168 conn(0x7f7ef00377d0 0x7f7ef0039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:21.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.254+0000 7f7f093c9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:21.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.254+0000 7f7f01ffb640 1 -- 172.21.15.40:0/3777640992 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f7ef806c180 con 0x7f7f040a2040 2024-04-22T08:13:21.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.254+0000 7f7f093c9640 10 monclient(hunting): handle_auth_done global_id 4410 payload 824 2024-04-22T08:13:21.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.254+0000 7f7f093c9640 1 -- 172.21.15.40:0/3777640992 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7f04136800 con 0x7f7f04126390 2024-04-22T08:13:21.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.254+0000 7f7f09bca640 1 --2- 172.21.15.40:0/3777640992 >> v2:172.21.15.189:6800/30168 conn(0x7f7ef00377d0 0x7f7ef0039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:21.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.254+0000 7f7f09bca640 10 monclient: get_auth_request con 0x7f7ef00377d0 auth_method 0 2024-04-22T08:13:21.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.256+0000 7f7f093c9640 1 -- 172.21.15.40:0/3777640992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f04126390 msgr2=0x7f7f04122550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:13:21.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.256+0000 7f7f093c9640 1 --2- 172.21.15.40:0/3777640992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f04126390 0x7f7f04122550 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7efc003930 tx=0x7f7efc00b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:21.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.256+0000 7f7f01ffb640 1 -- 172.21.15.40:0/3777640992 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7efc005270 con 0x7f7f04126390 2024-04-22T08:13:21.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.256+0000 7f7f01ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:21.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.256+0000 7f7f01ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:13:21.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.256+0000 7f7f01ffb640 10 monclient: dump: 2024-04-22T08:13:21.271 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:21.271 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:21.272 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:21.272 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:21.272 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:21.272 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:21.272 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:21.272 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:21.272 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:21.272 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:21.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.257+0000 7f7f01ffb640 1 -- 172.21.15.40:0/3777640992 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f7efc0053d0 con 0x7f7f04126390 2024-04-22T08:13:21.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.257+0000 7f7f01ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:13:21.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.257+0000 7f7f01ffb640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:13:21.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.257+0000 7f7f01ffb640 1 -- 172.21.15.40:0/3777640992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f04126390 msgr2=0x7f7f04122550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:21.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.257+0000 7f7f01ffb640 1 --2- 172.21.15.40:0/3777640992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f04126390 0x7f7f04122550 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7efc003930 tx=0x7f7efc00b8c0 comp rx=0 tx=0).stop 2024-04-22T08:13:21.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.257+0000 7f7f01ffb640 1 -- 172.21.15.40:0/3777640992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f04126390 msgr2=0x7f7f04122550 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:13:21.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.257+0000 7f7f01ffb640 1 --2- 172.21.15.40:0/3777640992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f04126390 0x7f7f04122550 secure :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7efc003930 tx=0x7f7efc00b8c0 comp rx=0 tx=0).stop 2024-04-22T08:13:21.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.257+0000 7f7f09bca640 1 --2- 172.21.15.40:0/3777640992 >> v2:172.21.15.189:6800/30168 conn(0x7f7ef00377d0 0x7f7ef0039c90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7ef40078a0 tx=0x7f7ef4004ee0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:21.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.263+0000 7f7f0b62d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:13:21.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.263+0000 7f7f0b62d640 1 --2- 172.21.15.40:0/3777640992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f04122a90 0x7f7f04122ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:21.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.263+0000 7f7f0b62d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:21.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.263+0000 7f7f0b62d640 1 -- 172.21.15.40:0/3777640992 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f04125a70 con 0x7f7f04122a90 2024-04-22T08:13:21.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.263+0000 7f7f093c9640 1 --2- 172.21.15.40:0/3777640992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f04122a90 0x7f7f04122ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:21.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.263+0000 7f7f093c9640 10 monclient: get_auth_request con 0x7f7f04122a90 auth_method 0 2024-04-22T08:13:21.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.263+0000 7f7f093c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:21.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.263+0000 7f7f093c9640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:21.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.263+0000 7f7f093c9640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:21.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.263+0000 7f7f093c9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:21.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.263+0000 7f7f093c9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:21.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.263+0000 7f7f093c9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:21.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.263+0000 7f7f093c9640 10 monclient(hunting): handle_auth_done global_id 4413 payload 824 2024-04-22T08:13:21.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.263+0000 7f7f093c9640 1 -- 172.21.15.40:0/3777640992 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7f041233e0 con 0x7f7f04122a90 2024-04-22T08:13:21.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.266+0000 7f7f093c9640 1 -- 172.21.15.40:0/3777640992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f04122a90 msgr2=0x7f7f04122ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:13:21.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.266+0000 7f7f093c9640 1 --2- 172.21.15.40:0/3777640992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f04122a90 0x7f7f04122ea0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7efc003930 tx=0x7f7efc005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:21.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.266+0000 7f7f01ffb640 1 -- 172.21.15.40:0/3777640992 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7efc020070 con 0x7f7f04122a90 2024-04-22T08:13:21.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.266+0000 7f7f01ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:21.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.266+0000 7f7f01ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:13:21.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.266+0000 7f7f01ffb640 10 monclient: dump: 2024-04-22T08:13:21.274 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:21.274 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:21.274 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:21.274 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:21.274 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:21.275 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:21.275 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:21.275 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:21.275 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:21.275 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:21.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.266+0000 7f7f01ffb640 1 -- 172.21.15.40:0/3777640992 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4792 (secure 0 0 0) 0x7f7efc010400 con 0x7f7f04122a90 2024-04-22T08:13:21.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.266+0000 7f7f01ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:13:21.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.266+0000 7f7f01ffb640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:13:21.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.266+0000 7f7f01ffb640 1 -- 172.21.15.40:0/3777640992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f04122a90 msgr2=0x7f7f04122ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:21.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.266+0000 7f7f01ffb640 1 --2- 172.21.15.40:0/3777640992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f04122a90 0x7f7f04122ea0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7efc003930 tx=0x7f7efc005a20 comp rx=0 tx=0).stop 2024-04-22T08:13:21.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.266+0000 7f7f01ffb640 1 -- 172.21.15.40:0/3777640992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f04122a90 msgr2=0x7f7f04122ea0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:13:21.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.266+0000 7f7f01ffb640 1 --2- 172.21.15.40:0/3777640992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f04122a90 0x7f7f04122ea0 secure :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7efc003930 tx=0x7f7efc005a20 comp rx=0 tx=0).stop 2024-04-22T08:13:21.276 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:13:21.276 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:13:21.276 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:13:21.276 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "leader", 2024-04-22T08:13:21.276 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 12, 2024-04-22T08:13:21.276 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:13:21.276 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:13:21.276 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:13:21.276 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:13:21.276 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 1, 2024-04-22T08:13:21.276 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:13:21.276 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:13:21.276 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:13:21.277 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:13:21.277 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:13:21.277 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:13:21.277 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:13:21.277 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:13:21.277 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:13:21.277 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:13:21.277 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:13:21.277 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:13:21.277 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:13:21.277 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:13:21.277 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:13:21.277 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:13:21.277 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:13:21.277 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:13:21.278 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:13:21.278 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:13:21.278 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:13:21.278 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:13:21.278 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:13:21.278 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:13:21.278 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:13:21.278 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:13:21.278 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:21.278 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:21.278 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:13:21.278 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:13:21.278 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:13:21.278 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:13:21.279 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:13:21.279 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:13:21.279 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:13:21.279 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:13:21.279 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:13:21.279 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:13:21.279 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:13:21.279 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:13:21.279 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:13:21.279 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:13:21.279 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:13:21.279 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:13:21.279 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:13:21.279 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:13:21.279 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:13:21.280 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:13:21.280 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:13:21.280 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:13:21.280 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:13:21.280 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:13:21.280 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:13:21.280 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:13:21.280 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:13:21.280 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:13:21.280 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:13:21.280 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:21.280 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:13:21.280 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:21.280 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:13:21.281 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:13:21.281 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:13:21.281 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:13:21.281 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:21.281 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:13:21.281 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:13:21.281 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:21.281 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:21.281 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:21.281 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:13:21.281 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:13:21.281 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:21.281 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:21.281 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:21.281 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:21.282 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:13:21.282 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:13:21.282 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:13:21.282 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:13:21.282 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:13:21.282 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:21.282 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:21.282 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:13:21.282 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:13:21.282 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:13:21.282 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:13:21.282 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:21.282 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:13:21.282 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:13:21.283 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:21.283 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:21.283 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:21.283 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:13:21.283 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:13:21.283 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:21.283 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:21.283 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:21.283 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:21.283 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:13:21.283 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:13:21.283 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:13:21.283 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:13:21.283 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:13:21.283 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:21.284 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:21.284 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:13:21.284 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:13:21.284 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:13:21.284 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:13:21.284 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:21.284 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:13:21.284 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:13:21.284 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:21.284 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:21.284 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:21.284 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:13:21.284 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:13:21.284 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:21.285 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:21.285 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:21.285 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:21.285 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:13:21.285 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:13:21.285 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:13:21.285 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:13:21.285 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:13:21.285 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:21.285 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:21.285 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:21.285 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:13:21.285 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:13:21.286 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:21.286 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:13:21.286 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:13:21.286 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:13:21.286 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:21.286 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:13:21.286 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:13:21.286 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:21.286 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:13:21.286 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:13:21.286 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 2 2024-04-22T08:13:21.286 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:21.286 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:13:21.287 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:13:21.287 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:21.287 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:13:21.287 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:13:21.287 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 6 2024-04-22T08:13:21.287 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:21.287 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:13:21.287 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": [ 2024-04-22T08:13:21.287 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:21.287 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:13:21.287 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:13:21.287 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:13:21.287 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:21.288 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:21.288 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:21.288 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:13:21.288 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:13:21.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.266+0000 7f7f0b62d640 1 -- 172.21.15.40:0/3777640992 >> v2:172.21.15.189:6800/30168 conn(0x7f7ef00377d0 msgr2=0x7f7ef0039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:21.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.266+0000 7f7f0b62d640 1 --2- 172.21.15.40:0/3777640992 >> v2:172.21.15.189:6800/30168 conn(0x7f7ef00377d0 0x7f7ef0039c90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7ef40078a0 tx=0x7f7ef4004ee0 comp rx=0 tx=0).stop 2024-04-22T08:13:21.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.266+0000 7f7f0b62d640 10 monclient: shutdown 2024-04-22T08:13:21.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.266+0000 7f7f0b62d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:13:21.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.266+0000 7f7f0b62d640 1 -- 172.21.15.40:0/3777640992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f040a2040 msgr2=0x7f7f041cb220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:21.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.266+0000 7f7f0b62d640 1 --2- 172.21.15.40:0/3777640992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f040a2040 0x7f7f041cb220 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7ef80090b0 tx=0x7f7ef80046c0 comp rx=0 tx=0).stop 2024-04-22T08:13:21.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.266+0000 7f7f09bca640 1 -- 172.21.15.40:0/3777640992 reap_dead start 2024-04-22T08:13:21.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.266+0000 7f7f0b62d640 1 -- 172.21.15.40:0/3777640992 shutdown_connections 2024-04-22T08:13:21.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.266+0000 7f7f0b62d640 1 --2- 172.21.15.40:0/3777640992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f040a2040 0x7f7f041cb220 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:21.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.266+0000 7f7f0b62d640 1 -- 172.21.15.40:0/3777640992 >> 172.21.15.40:0/3777640992 conn(0x7f7f040a89a0 msgr2=0x7f7f040a6d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:13:21.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.266+0000 7f7f0b62d640 1 -- 172.21.15.40:0/3777640992 shutdown_connections 2024-04-22T08:13:21.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.266+0000 7f7f0b62d640 1 -- 172.21.15.40:0/3777640992 wait complete. 2024-04-22T08:13:21.291 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-22T08:13:21.291 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-04-22T08:13:21.291 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-22T08:13:21.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.391+0000 7f0f509eb640 10 monclient: get_monmap_and_config 2024-04-22T08:13:21.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.391+0000 7f0f509eb640 10 monclient: build_initial_monmap 2024-04-22T08:13:21.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.391+0000 7f0f509eb640 10 monclient: monmap: 2024-04-22T08:13:21.391 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:13:21.391 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:21.391 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:13:21.392560+0000 2024-04-22T08:13:21.391 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:13:21.392560+0000 2024-04-22T08:13:21.392 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:13:21.392 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:21.392 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:13:21.392 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:13:21.392 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:13:21.392 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:13:21.392 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:13:21.392 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:21.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.392+0000 7f0f509eb640 1 Processor -- start 2024-04-22T08:13:21.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.392+0000 7f0f509eb640 1 -- start start 2024-04-22T08:13:21.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.392+0000 7f0f509eb640 10 monclient: init 2024-04-22T08:13:21.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.392+0000 7f0f509eb640 10 monclient: _reopen_session rank -1 2024-04-22T08:13:21.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.392+0000 7f0f509eb640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-04-22T08:13:21.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.393+0000 7f0f509eb640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0f4c0a5a10 0x7f0f4c0a32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:21.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.393+0000 7f0f509eb640 10 monclient(hunting): picked mon.noname-c con 0x7f0f4c0a5a10 addr v2:172.21.15.189:3300/0 2024-04-22T08:13:21.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.393+0000 7f0f509eb640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0f4c0a1970 0x7f0f4c0a1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:21.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.393+0000 7f0f509eb640 10 monclient(hunting): picked mon.noname-b con 0x7f0f4c0a1970 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:13:21.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.393+0000 7f0f509eb640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0f4c135c20 0x7f0f4c1360b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:21.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.393+0000 7f0f509eb640 10 monclient(hunting): picked mon.noname-a con 0x7f0f4c135c20 addr v2:172.21.15.40:3300/0 2024-04-22T08:13:21.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.393+0000 7f0f509eb640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:21.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.393+0000 7f0f509eb640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f4c09fe80 con 0x7f0f4c135c20 2024-04-22T08:13:21.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.393+0000 7f0f509eb640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:21.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.393+0000 7f0f509eb640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f4c0a37f0 con 0x7f0f4c0a5a10 2024-04-22T08:13:21.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.393+0000 7f0f509eb640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:21.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.393+0000 7f0f509eb640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4c0a25b0 con 0x7f0f4c0a1970 2024-04-22T08:13:21.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.393+0000 7f0f509eb640 10 monclient(hunting): _renew_subs 2024-04-22T08:13:21.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.393+0000 7f0f509eb640 10 monclient(hunting): authenticate will time out at 875.386868s 2024-04-22T08:13:21.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.393+0000 7f0f49d9b640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0f4c0a1970 0x7f0f4c0a1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:21.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.393+0000 7f0f4a59c640 1 -- >> v2:172.21.15.189:3300/0 conn(0x7f0f4c0a5a10 msgr2=0x7f0f4c0a32b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.189:3300/0 2024-04-22T08:13:21.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.393+0000 7f0f4ad9d640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0f4c135c20 0x7f0f4c1360b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:21.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.393+0000 7f0f4a59c640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0f4c0a5a10 0x7f0f4c0a32b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:13:21.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.393+0000 7f0f49d9b640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0f4c0a1970 0x7f0f4c0a1d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:34318/0 (socket says 172.21.15.40:34318) 2024-04-22T08:13:21.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.393+0000 7f0f49d9b640 1 -- 172.21.15.40:0/1680335240 learned_addr learned my addr 172.21.15.40:0/1680335240 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:13:21.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.393+0000 7f0f49d9b640 10 monclient(hunting): get_auth_request con 0x7f0f4c0a1970 auth_method 0 2024-04-22T08:13:21.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.393+0000 7f0f49d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:21.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.393+0000 7f0f49d9b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:21.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.393+0000 7f0f49d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:21.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.393+0000 7f0f4ad9d640 10 monclient(hunting): get_auth_request con 0x7f0f4c135c20 auth_method 0 2024-04-22T08:13:21.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.393+0000 7f0f4ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:21.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.393+0000 7f0f4ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:21.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.393+0000 7f0f4ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:21.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.393+0000 7f0f49d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:21.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.393+0000 7f0f49d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:21.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.393+0000 7f0f49d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:21.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.393+0000 7f0f4ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:21.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.393+0000 7f0f4ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:21.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.393+0000 7f0f4ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:21.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.393+0000 7f0f49d9b640 10 monclient(hunting): handle_auth_done global_id 4379 payload 274 2024-04-22T08:13:21.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.393+0000 7f0f49d9b640 1 -- 172.21.15.40:0/1680335240 >> v2:172.21.15.189:3300/0 conn(0x7f0f4c0a5a10 msgr2=0x7f0f4c0a32b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:13:21.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.393+0000 7f0f49d9b640 1 --2- 172.21.15.40:0/1680335240 >> v2:172.21.15.189:3300/0 conn(0x7f0f4c0a5a10 0x7f0f4c0a32b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:21.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.393+0000 7f0f49d9b640 1 -- 172.21.15.40:0/1680335240 >> v2:172.21.15.40:3300/0 conn(0x7f0f4c135c20 msgr2=0x7f0f4c1360b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:13:21.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.393+0000 7f0f49d9b640 1 --2- 172.21.15.40:0/1680335240 >> v2:172.21.15.40:3300/0 conn(0x7f0f4c135c20 0x7f0f4c1360b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:21.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.393+0000 7f0f49d9b640 10 monclient: _finish_hunting 0 2024-04-22T08:13:21.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.393+0000 7f0f49d9b640 1 monclient: found mon.noname-b 2024-04-22T08:13:21.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.393+0000 7f0f49d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:13:21.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.393+0000 7f0f49d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:13:21.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.393+0000 7f0f49d9b640 1 -- 172.21.15.40:0/1680335240 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f4c09fba0 con 0x7f0f4c0a1970 2024-04-22T08:13:21.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.393+0000 7f0f4ad9d640 1 --2- 172.21.15.40:0/1680335240 >> v2:172.21.15.40:3300/0 conn(0x7f0f4c135c20 0x7f0f4c1360b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:13:21.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.396+0000 7f0f49d9b640 1 -- 172.21.15.40:0/1680335240 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0f4c0a1970 msgr2=0x7f0f4c0a1d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:13:21.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.396+0000 7f0f49d9b640 1 --2- 172.21.15.40:0/1680335240 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0f4c0a1970 0x7f0f4c0a1d90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c06c3a0 tx=0x7f0f4002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d818f1ed3b1a279 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:21.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.396+0000 7f0f4959a640 1 -- 172.21.15.40:0/1680335240 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f40031a10 con 0x7f0f4c0a1970 2024-04-22T08:13:21.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.396+0000 7f0f4959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:21.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.396+0000 7f0f4959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:13:21.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.396+0000 7f0f4959a640 10 monclient: dump: 2024-04-22T08:13:21.404 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:21.404 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:21.404 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:21.404 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:21.404 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:21.404 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:21.404 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:21.404 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:21.404 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:21.404 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:21.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.396+0000 7f0f4959a640 10 monclient: _finish_auth 0 2024-04-22T08:13:21.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.396+0000 7f0f4959a640 10 monclient: _check_auth_tickets 2024-04-22T08:13:21.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.396+0000 7f0f4959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:13:21.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.396+0000 7f0f509eb640 5 monclient: authenticate success, global_id 4379 2024-04-22T08:13:21.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.396+0000 7f0f509eb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:13:21.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.396+0000 7f0f4959a640 1 -- 172.21.15.40:0/1680335240 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f40031b70 con 0x7f0f4c0a1970 2024-04-22T08:13:21.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.396+0000 7f0f4959a640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:13:21.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.396+0000 7f0f509eb640 10 monclient: get_monmap_and_config success 2024-04-22T08:13:21.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.396+0000 7f0f4959a640 1 -- 172.21.15.40:0/1680335240 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f40030350 con 0x7f0f4c0a1970 2024-04-22T08:13:21.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.396+0000 7f0f4959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:21.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.396+0000 7f0f509eb640 4 set_mon_vals no callback set 2024-04-22T08:13:21.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.396+0000 7f0f4959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:13:21.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.396+0000 7f0f4959a640 10 monclient: dump: 2024-04-22T08:13:21.405 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:21.405 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:21.406 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:21.406 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:21.406 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:21.406 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:21.406 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:21.406 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:21.406 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:21.406 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:21.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.397+0000 7f0f509eb640 10 monclient: shutdown 2024-04-22T08:13:21.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.397+0000 7f0f509eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:13:21.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.397+0000 7f0f509eb640 1 -- 172.21.15.40:0/1680335240 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0f4c0a1970 msgr2=0x7f0f4c0a1d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:21.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.397+0000 7f0f509eb640 1 --2- 172.21.15.40:0/1680335240 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0f4c0a1970 0x7f0f4c0a1d90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c06c3a0 tx=0x7f0f4002b8a0 comp rx=0 tx=0).stop 2024-04-22T08:13:21.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.397+0000 7f0f509eb640 1 -- 172.21.15.40:0/1680335240 shutdown_connections 2024-04-22T08:13:21.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.397+0000 7f0f509eb640 1 --2- 172.21.15.40:0/1680335240 >> v2:172.21.15.40:3300/0 conn(0x7f0f4c135c20 0x7f0f4c1360b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:21.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.397+0000 7f0f509eb640 1 --2- 172.21.15.40:0/1680335240 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0f4c0a1970 0x7f0f4c0a1d90 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:21.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.397+0000 7f0f509eb640 1 --2- 172.21.15.40:0/1680335240 >> v2:172.21.15.189:3300/0 conn(0x7f0f4c0a5a10 0x7f0f4c0a32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:21.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.397+0000 7f0f509eb640 1 -- 172.21.15.40:0/1680335240 >> 172.21.15.40:0/1680335240 conn(0x7f0f4c0a4ef0 msgr2=0x7f0f4c0a5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:13:21.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.397+0000 7f0f509eb640 1 -- 172.21.15.40:0/1680335240 shutdown_connections 2024-04-22T08:13:21.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.397+0000 7f0f509eb640 1 -- 172.21.15.40:0/1680335240 wait complete. 2024-04-22T08:13:21.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.397+0000 7f0f509eb640 10 monclient: build_initial_monmap 2024-04-22T08:13:21.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.397+0000 7f0f509eb640 10 monclient: monmap: 2024-04-22T08:13:21.407 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:13:21.407 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:13:21.407 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:13:21.407 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:13:21.407 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:13:21.407 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:21.407 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:13:21.407 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:13:21.408 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:13:21.408 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:21.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.398+0000 7f0f509eb640 1 Processor -- start 2024-04-22T08:13:21.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.398+0000 7f0f509eb640 1 -- start start 2024-04-22T08:13:21.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.398+0000 7f0f509eb640 10 monclient: init 2024-04-22T08:13:21.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.398+0000 7f0f509eb640 10 monclient: _reopen_session rank -1 2024-04-22T08:13:21.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.398+0000 7f0f509eb640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-22T08:13:21.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.398+0000 7f0f509eb640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0f4c0a1970 0x7f0f4c1cad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:21.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.398+0000 7f0f509eb640 10 monclient(hunting): picked mon.noname-c con 0x7f0f4c0a1970 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:13:21.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.398+0000 7f0f509eb640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f4c0a5a10 0x7f0f4c1cb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:21.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.398+0000 7f0f509eb640 10 monclient(hunting): picked mon.noname-b con 0x7f0f4c0a5a10 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:13:21.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.398+0000 7f0f509eb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f4c135c20 0x7f0f4c120270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:21.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.398+0000 7f0f509eb640 10 monclient(hunting): picked mon.noname-a con 0x7f0f4c135c20 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:13:21.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.398+0000 7f0f509eb640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:21.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.398+0000 7f0f509eb640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4c122410 con 0x7f0f4c135c20 2024-04-22T08:13:21.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.398+0000 7f0f509eb640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:21.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.398+0000 7f0f509eb640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4c123f80 con 0x7f0f4c0a5a10 2024-04-22T08:13:21.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.398+0000 7f0f509eb640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:21.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.398+0000 7f0f509eb640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4c1241d0 con 0x7f0f4c0a1970 2024-04-22T08:13:21.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.398+0000 7f0f509eb640 10 monclient(hunting): _renew_subs 2024-04-22T08:13:21.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.398+0000 7f0f509eb640 10 monclient(hunting): authenticate will time out at 875.392494s 2024-04-22T08:13:21.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.398+0000 7f0f4a59c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0f4c0a1970 0x7f0f4c1cad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:21.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.398+0000 7f0f4ad9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f4c135c20 0x7f0f4c120270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:21.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.398+0000 7f0f4a59c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0f4c0a1970 0x7f0f4c1cad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:34320/0 (socket says 172.21.15.40:34320) 2024-04-22T08:13:21.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.398+0000 7f0f4ad9d640 10 monclient(hunting): get_auth_request con 0x7f0f4c135c20 auth_method 0 2024-04-22T08:13:21.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.398+0000 7f0f4a59c640 1 -- 172.21.15.40:0/271938495 learned_addr learned my addr 172.21.15.40:0/271938495 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:13:21.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.398+0000 7f0f4ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:21.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.398+0000 7f0f4ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:21.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.398+0000 7f0f4ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:21.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.398+0000 7f0f4a59c640 10 monclient(hunting): get_auth_request con 0x7f0f4c0a1970 auth_method 0 2024-04-22T08:13:21.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.398+0000 7f0f4a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:21.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.398+0000 7f0f4a59c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:21.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.398+0000 7f0f4a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:21.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.398+0000 7f0f49d9b640 1 -- 172.21.15.40:0/271938495 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f4c0a5a10 msgr2=0x7f0f4c1cb2a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:3300/0 2024-04-22T08:13:21.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.398+0000 7f0f49d9b640 1 --2- 172.21.15.40:0/271938495 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f4c0a5a10 0x7f0f4c1cb2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:13:21.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.398+0000 7f0f4ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:21.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.398+0000 7f0f4ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:21.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.398+0000 7f0f4ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:21.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.398+0000 7f0f4a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:21.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.398+0000 7f0f4a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:21.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.398+0000 7f0f4a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:21.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.399+0000 7f0f4ad9d640 10 monclient(hunting): handle_auth_done global_id 4419 payload 824 2024-04-22T08:13:21.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.399+0000 7f0f4ad9d640 1 -- 172.21.15.40:0/271938495 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0f4c0a1970 msgr2=0x7f0f4c1cad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:21.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.399+0000 7f0f4ad9d640 1 --2- 172.21.15.40:0/271938495 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0f4c0a1970 0x7f0f4c1cad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:21.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.399+0000 7f0f4ad9d640 1 -- 172.21.15.40:0/271938495 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f4c0a5a10 msgr2=0x7f0f4c1cb2a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:13:21.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.399+0000 7f0f4ad9d640 1 --2- 172.21.15.40:0/271938495 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f4c0a5a10 0x7f0f4c1cb2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:21.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.399+0000 7f0f4ad9d640 10 monclient: _finish_hunting 0 2024-04-22T08:13:21.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.399+0000 7f0f4ad9d640 1 monclient: found mon.noname-a 2024-04-22T08:13:21.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.399+0000 7f0f4ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:13:21.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.399+0000 7f0f4ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:13:21.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.399+0000 7f0f4ad9d640 1 -- 172.21.15.40:0/271938495 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f44009a70 con 0x7f0f4c135c20 2024-04-22T08:13:21.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.399+0000 7f0f4a59c640 1 --2- 172.21.15.40:0/271938495 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0f4c0a1970 0x7f0f4c1cad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:13:21.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.401+0000 7f0f4ad9d640 1 -- 172.21.15.40:0/271938495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f4c135c20 msgr2=0x7f0f4c120270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:13:21.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.401+0000 7f0f4ad9d640 1 --2- 172.21.15.40:0/271938495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f4c135c20 0x7f0f4c120270 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0f4400efc0 tx=0x7f0f4400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:21.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.401+0000 7f0f3affd640 1 -- 172.21.15.40:0/271938495 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f44003990 con 0x7f0f4c135c20 2024-04-22T08:13:21.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.401+0000 7f0f3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:21.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.401+0000 7f0f3affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:13:21.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.401+0000 7f0f3affd640 10 monclient: dump: 2024-04-22T08:13:21.412 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:21.412 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:21.412 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:21.412 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:21.412 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:21.412 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:21.412 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:21.412 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:21.412 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:21.412 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:21.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.401+0000 7f0f3affd640 10 monclient: _finish_auth 0 2024-04-22T08:13:21.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.401+0000 7f0f3affd640 10 monclient: _check_auth_tickets 2024-04-22T08:13:21.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.401+0000 7f0f3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:13:21.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.401+0000 7f0f509eb640 5 monclient: authenticate success, global_id 4419 2024-04-22T08:13:21.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.401+0000 7f0f3affd640 1 -- 172.21.15.40:0/271938495 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f4400f040 con 0x7f0f4c135c20 2024-04-22T08:13:21.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.401+0000 7f0f3affd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:13:21.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.401+0000 7f0f3affd640 1 -- 172.21.15.40:0/271938495 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f44003d50 con 0x7f0f4c135c20 2024-04-22T08:13:21.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.401+0000 7f0f509eb640 10 monclient: _renew_subs 2024-04-22T08:13:21.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.401+0000 7f0f509eb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:13:21.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.401+0000 7f0f509eb640 1 -- 172.21.15.40:0/271938495 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f400094d0 con 0x7f0f4c135c20 2024-04-22T08:13:21.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.401+0000 7f0f3bfff640 4 set_mon_vals no callback set 2024-04-22T08:13:21.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.401+0000 7f0f3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:21.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.401+0000 7f0f3affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:13:21.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.401+0000 7f0f3affd640 10 monclient: dump: 2024-04-22T08:13:21.413 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:21.413 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:21.413 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:21.413 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:21.414 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:21.414 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:21.414 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:21.414 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:21.414 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:21.414 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:21.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.401+0000 7f0f509eb640 10 monclient: _renew_subs 2024-04-22T08:13:21.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.401+0000 7f0f509eb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:13:21.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.401+0000 7f0f509eb640 1 -- 172.21.15.40:0/271938495 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f4c135280 con 0x7f0f4c135c20 2024-04-22T08:13:21.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.402+0000 7f0f509eb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:13:21.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.402+0000 7f0f509eb640 1 --2- 172.21.15.40:0/271938495 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0f4c1207b0 0x7f0f4c120b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:21.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.402+0000 7f0f509eb640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:21.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.402+0000 7f0f509eb640 1 -- 172.21.15.40:0/271938495 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4c124360 con 0x7f0f4c1207b0 2024-04-22T08:13:21.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.402+0000 7f0f3affd640 1 -- 172.21.15.40:0/271938495 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f0f44003eb0 con 0x7f0f4c135c20 2024-04-22T08:13:21.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.402+0000 7f0f4a59c640 1 --2- 172.21.15.40:0/271938495 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0f4c1207b0 0x7f0f4c120b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:21.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.402+0000 7f0f4a59c640 10 monclient: get_auth_request con 0x7f0f4c1207b0 auth_method 0 2024-04-22T08:13:21.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.402+0000 7f0f4a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:21.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.402+0000 7f0f4a59c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:21.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.402+0000 7f0f4a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:21.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.402+0000 7f0f4a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:21.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.402+0000 7f0f4a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:21.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.402+0000 7f0f4a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:21.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.402+0000 7f0f3affd640 1 --2- 172.21.15.40:0/271938495 >> v2:172.21.15.189:6800/30168 conn(0x7f0f2c037780 0x7f0f2c039c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:21.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.403+0000 7f0f3affd640 1 -- 172.21.15.40:0/271938495 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f0f4404af30 con 0x7f0f4c135c20 2024-04-22T08:13:21.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.403+0000 7f0f4a59c640 10 monclient(hunting): handle_auth_done global_id 4385 payload 824 2024-04-22T08:13:21.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.403+0000 7f0f4a59c640 1 -- 172.21.15.40:0/271938495 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0f4c09ee10 con 0x7f0f4c1207b0 2024-04-22T08:13:21.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.403+0000 7f0f49d9b640 1 --2- 172.21.15.40:0/271938495 >> v2:172.21.15.189:6800/30168 conn(0x7f0f2c037780 0x7f0f2c039c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:21.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.403+0000 7f0f49d9b640 10 monclient: get_auth_request con 0x7f0f2c037780 auth_method 0 2024-04-22T08:13:21.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.405+0000 7f0f4a59c640 1 -- 172.21.15.40:0/271938495 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0f4c1207b0 msgr2=0x7f0f4c120b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:13:21.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.405+0000 7f0f4a59c640 1 --2- 172.21.15.40:0/271938495 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0f4c1207b0 0x7f0f4c120b90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c0053f0 tx=0x7f0f3c0058b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:21.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.405+0000 7f0f3affd640 1 -- 172.21.15.40:0/271938495 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f3c00ee20 con 0x7f0f4c1207b0 2024-04-22T08:13:21.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.405+0000 7f0f3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:21.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.405+0000 7f0f3affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:13:21.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.405+0000 7f0f3affd640 10 monclient: dump: 2024-04-22T08:13:21.419 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:21.419 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:21.419 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:21.419 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:21.419 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:21.419 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:21.419 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:21.419 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:21.419 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:21.420 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:21.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.405+0000 7f0f3affd640 1 -- 172.21.15.40:0/271938495 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f0f3c01b450 con 0x7f0f4c1207b0 2024-04-22T08:13:21.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.405+0000 7f0f3affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:13:21.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.405+0000 7f0f3affd640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:13:21.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.405+0000 7f0f3affd640 1 -- 172.21.15.40:0/271938495 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0f4c1207b0 msgr2=0x7f0f4c120b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:21.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.405+0000 7f0f3affd640 1 --2- 172.21.15.40:0/271938495 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0f4c1207b0 0x7f0f4c120b90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c0053f0 tx=0x7f0f3c0058b0 comp rx=0 tx=0).stop 2024-04-22T08:13:21.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.405+0000 7f0f3affd640 1 -- 172.21.15.40:0/271938495 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0f4c1207b0 msgr2=0x7f0f4c120b90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:13:21.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.405+0000 7f0f3affd640 1 --2- 172.21.15.40:0/271938495 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0f4c1207b0 0x7f0f4c120b90 secure :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c0053f0 tx=0x7f0f3c0058b0 comp rx=0 tx=0).stop 2024-04-22T08:13:21.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.405+0000 7f0f49d9b640 1 --2- 172.21.15.40:0/271938495 >> v2:172.21.15.189:6800/30168 conn(0x7f0f2c037780 0x7f0f2c039c40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c0a5ee0 tx=0x7f0f40008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:21.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.411+0000 7f0f509eb640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:13:21.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.411+0000 7f0f509eb640 1 --2- 172.21.15.40:0/271938495 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0f4c1210d0 0x7f0f4c06ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:21.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.412+0000 7f0f509eb640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:21.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.412+0000 7f0f509eb640 1 -- 172.21.15.40:0/271938495 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4c0667a0 con 0x7f0f4c1210d0 2024-04-22T08:13:21.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.412+0000 7f0f4a59c640 1 --2- 172.21.15.40:0/271938495 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0f4c1210d0 0x7f0f4c06ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:21.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.412+0000 7f0f4a59c640 10 monclient: get_auth_request con 0x7f0f4c1210d0 auth_method 0 2024-04-22T08:13:21.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.412+0000 7f0f4a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:21.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.412+0000 7f0f4a59c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:21.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.412+0000 7f0f4a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:21.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.412+0000 7f0f4a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:21.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.412+0000 7f0f4a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:21.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.412+0000 7f0f4a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:21.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.412+0000 7f0f4a59c640 10 monclient(hunting): handle_auth_done global_id 4388 payload 824 2024-04-22T08:13:21.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.412+0000 7f0f4a59c640 1 -- 172.21.15.40:0/271938495 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0f4c123de0 con 0x7f0f4c1210d0 2024-04-22T08:13:21.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.414+0000 7f0f4a59c640 1 -- 172.21.15.40:0/271938495 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0f4c1210d0 msgr2=0x7f0f4c06ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:13:21.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.414+0000 7f0f4a59c640 1 --2- 172.21.15.40:0/271938495 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0f4c1210d0 0x7f0f4c06ff40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c018000 tx=0x7f0f3c0059d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:21.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.414+0000 7f0f3affd640 1 -- 172.21.15.40:0/271938495 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f3c006230 con 0x7f0f4c1210d0 2024-04-22T08:13:21.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.414+0000 7f0f3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:21.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.414+0000 7f0f3affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:13:21.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.414+0000 7f0f3affd640 10 monclient: dump: 2024-04-22T08:13:21.422 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:21.422 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:21.422 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:21.422 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:21.422 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:21.422 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:21.422 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:21.422 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:21.422 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:21.422 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:21.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.415+0000 7f0f3affd640 1 -- 172.21.15.40:0/271938495 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4620 (secure 0 0 0) 0x7f0f3c006390 con 0x7f0f4c1210d0 2024-04-22T08:13:21.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.415+0000 7f0f3affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:13:21.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.415+0000 7f0f3affd640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:13:21.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.415+0000 7f0f3affd640 1 -- 172.21.15.40:0/271938495 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0f4c1210d0 msgr2=0x7f0f4c06ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:21.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.415+0000 7f0f3affd640 1 --2- 172.21.15.40:0/271938495 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0f4c1210d0 0x7f0f4c06ff40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c018000 tx=0x7f0f3c0059d0 comp rx=0 tx=0).stop 2024-04-22T08:13:21.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.415+0000 7f0f3affd640 1 -- 172.21.15.40:0/271938495 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0f4c1210d0 msgr2=0x7f0f4c06ff40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:13:21.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.415+0000 7f0f3affd640 1 --2- 172.21.15.40:0/271938495 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0f4c1210d0 0x7f0f4c06ff40 secure :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c018000 tx=0x7f0f3c0059d0 comp rx=0 tx=0).stop 2024-04-22T08:13:21.423 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:13:21.423 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:13:21.423 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:13:21.423 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:13:21.423 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 12, 2024-04-22T08:13:21.424 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:13:21.424 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:13:21.424 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:13:21.424 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:13:21.424 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 1, 2024-04-22T08:13:21.424 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:13:21.424 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:13:21.424 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:13:21.424 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:13:21.424 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:13:21.424 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:13:21.424 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:13:21.424 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:13:21.424 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:13:21.425 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:13:21.425 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:13:21.425 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:13:21.425 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:13:21.425 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:13:21.425 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:13:21.425 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:13:21.425 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:13:21.425 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:13:21.425 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:13:21.425 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:13:21.425 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:13:21.425 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:13:21.425 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:13:21.425 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:13:21.426 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:13:21.426 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:13:21.426 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:21.426 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:21.426 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:13:21.426 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:13:21.426 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:13:21.426 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:13:21.426 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:13:21.426 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:13:21.426 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:13:21.426 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:13:21.427 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:13:21.427 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:13:21.427 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:13:21.427 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:13:21.427 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:13:21.427 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:13:21.427 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:13:21.427 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:13:21.427 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:13:21.427 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:13:21.427 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:13:21.427 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:13:21.427 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:13:21.427 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:13:21.428 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:13:21.428 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:13:21.428 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:13:21.428 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:13:21.428 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:13:21.428 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:13:21.428 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:13:21.428 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:21.428 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:13:21.428 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:21.428 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:13:21.428 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:13:21.428 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:13:21.428 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:13:21.429 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:21.429 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:13:21.429 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:13:21.429 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:21.429 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:21.429 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:21.429 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:13:21.429 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:13:21.429 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:21.429 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:21.429 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:21.429 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:21.429 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:13:21.429 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:13:21.430 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:13:21.430 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:13:21.430 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:13:21.430 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:21.430 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:21.430 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:13:21.430 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:13:21.430 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:13:21.430 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:13:21.430 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:21.430 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:13:21.430 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:13:21.430 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:21.431 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:21.431 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:21.431 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:13:21.431 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:13:21.431 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:21.431 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:21.431 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:21.431 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:21.431 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:13:21.431 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:13:21.431 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:13:21.431 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:13:21.431 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:13:21.432 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:21.432 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:21.432 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:13:21.432 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:13:21.432 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:13:21.432 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:13:21.432 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:21.432 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:13:21.432 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:13:21.432 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:21.432 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:21.432 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:21.432 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:13:21.432 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:13:21.432 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:21.433 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:21.433 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:21.433 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:21.433 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:13:21.433 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:13:21.433 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:13:21.433 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:13:21.433 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:13:21.433 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:21.433 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:21.433 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:21.433 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:13:21.433 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:13:21.433 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:21.433 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:13:21.434 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:13:21.434 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:13:21.434 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:21.434 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:13:21.434 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:13:21.434 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:21.434 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:13:21.434 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:13:21.434 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 4 2024-04-22T08:13:21.434 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:21.434 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:13:21.434 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:13:21.434 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:21.434 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:13:21.435 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:13:21.435 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 2 2024-04-22T08:13:21.435 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:21.435 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:21.435 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:21.435 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:13:21.435 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:13:21.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.415+0000 7f0f509eb640 1 -- 172.21.15.40:0/271938495 >> v2:172.21.15.189:6800/30168 conn(0x7f0f2c037780 msgr2=0x7f0f2c039c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:21.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.415+0000 7f0f509eb640 1 --2- 172.21.15.40:0/271938495 >> v2:172.21.15.189:6800/30168 conn(0x7f0f2c037780 0x7f0f2c039c40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c0a5ee0 tx=0x7f0f40008030 comp rx=0 tx=0).stop 2024-04-22T08:13:21.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.415+0000 7f0f509eb640 10 monclient: shutdown 2024-04-22T08:13:21.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.415+0000 7f0f509eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:13:21.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.415+0000 7f0f509eb640 1 -- 172.21.15.40:0/271938495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f4c135c20 msgr2=0x7f0f4c120270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:21.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.415+0000 7f0f4ad9d640 1 -- 172.21.15.40:0/271938495 reap_dead start 2024-04-22T08:13:21.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.415+0000 7f0f509eb640 1 --2- 172.21.15.40:0/271938495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f4c135c20 0x7f0f4c120270 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0f4400efc0 tx=0x7f0f4400c490 comp rx=0 tx=0).stop 2024-04-22T08:13:21.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.415+0000 7f0f509eb640 1 -- 172.21.15.40:0/271938495 shutdown_connections 2024-04-22T08:13:21.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.415+0000 7f0f509eb640 1 --2- 172.21.15.40:0/271938495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f4c135c20 0x7f0f4c120270 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:21.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.415+0000 7f0f509eb640 1 -- 172.21.15.40:0/271938495 >> 172.21.15.40:0/271938495 conn(0x7f0f4c0a4ef0 msgr2=0x7f0f4c0a6280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:13:21.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.415+0000 7f0f509eb640 1 -- 172.21.15.40:0/271938495 shutdown_connections 2024-04-22T08:13:21.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:21.415+0000 7f0f509eb640 1 -- 172.21.15.40:0/271938495 wait complete. 2024-04-22T08:13:21.438 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-22T08:13:21.438 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-04-22T08:13:21.767 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[34367]: starting. 2024-04-22T08:13:21.768 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[34367]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-34365 2024-04-22T08:13:21.768 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[34367]: shutting down. 2024-04-22T08:13:23.889 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:13:23.890 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[34412]: starting. 2024-04-22T08:13:23.890 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[34412]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-34365 2024-04-22T08:13:23.890 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[34412]: created object 0... 2024-04-22T08:13:23.890 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[34412]: created object 25... 2024-04-22T08:13:23.890 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[34412]: created object 49... 2024-04-22T08:13:23.890 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[34412]: finishing. 2024-04-22T08:13:23.890 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[34412]: shutting down. 2024-04-22T08:13:23.930 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:13:23.930 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[34414]: starting. 2024-04-22T08:13:23.930 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[34414]: listing objects. 2024-04-22T08:13:23.930 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[34414]: listed object 0... 2024-04-22T08:13:23.930 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[34414]: listed object 25... 2024-04-22T08:13:23.930 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[34414]: saw 50 objects 2024-04-22T08:13:23.930 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[34414]: shutting down. 2024-04-22T08:13:24.780 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:13:24.780 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[34413]: starting. 2024-04-22T08:13:24.780 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[34413]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-34365 2024-04-22T08:13:24.780 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[34413]: shutting down. 2024-04-22T08:13:24.781 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:13:24.781 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:13:24.782 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:13:24.784 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:13:26.912 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[34481]: starting. 2024-04-22T08:13:26.913 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[34481]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-34480 2024-04-22T08:13:26.913 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[34481]: created object 0... 2024-04-22T08:13:26.913 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[34481]: created object 25... 2024-04-22T08:13:26.913 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[34481]: created object 49... 2024-04-22T08:13:26.913 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[34481]: finishing. 2024-04-22T08:13:26.913 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[34481]: shutting down. 2024-04-22T08:13:27.792 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[34482]: starting. 2024-04-22T08:13:27.793 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[34482]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-34480 2024-04-22T08:13:27.793 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[34482]: shutting down. 2024-04-22T08:13:29.921 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:13:29.922 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[34515]: starting. 2024-04-22T08:13:29.922 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[34515]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-34480 2024-04-22T08:13:29.922 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[34515]: created object 0... 2024-04-22T08:13:29.922 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[34515]: created object 25... 2024-04-22T08:13:29.922 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[34515]: created object 49... 2024-04-22T08:13:29.922 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[34515]: finishing. 2024-04-22T08:13:29.922 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[34515]: shutting down. 2024-04-22T08:13:29.960 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:13:29.960 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[34517]: starting. 2024-04-22T08:13:29.960 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[34517]: listing objects. 2024-04-22T08:13:29.960 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[34517]: listed object 0... 2024-04-22T08:13:29.960 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[34517]: listed object 25... 2024-04-22T08:13:29.960 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[34517]: saw 50 objects 2024-04-22T08:13:29.960 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[34517]: shutting down. 2024-04-22T08:13:30.816 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:13:30.817 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[34516]: starting. 2024-04-22T08:13:30.817 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[34516]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-34480 2024-04-22T08:13:30.817 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[34516]: shutting down. 2024-04-22T08:13:30.819 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:13:30.819 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:13:30.819 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:13:30.823 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:13:33.351 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[34584]: starting. 2024-04-22T08:13:33.351 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[34584]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-34583 2024-04-22T08:13:33.351 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[34584]: created object 0... 2024-04-22T08:13:33.351 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[34584]: created object 25... 2024-04-22T08:13:33.351 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[34584]: created object 49... 2024-04-22T08:13:33.351 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[34584]: finishing. 2024-04-22T08:13:33.351 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[34584]: shutting down. 2024-04-22T08:13:34.235 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[34585]: starting. 2024-04-22T08:13:34.235 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[34585]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-34583 2024-04-22T08:13:34.235 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[34585]: shutting down. 2024-04-22T08:13:36.358 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:13:36.358 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[34618]: starting. 2024-04-22T08:13:36.358 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[34618]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-34583 2024-04-22T08:13:36.358 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[34618]: created object 0... 2024-04-22T08:13:36.358 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[34618]: created object 25... 2024-04-22T08:13:36.358 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[34618]: created object 49... 2024-04-22T08:13:36.358 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[34618]: finishing. 2024-04-22T08:13:36.358 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[34618]: shutting down. 2024-04-22T08:13:36.392 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:13:36.392 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[34620]: starting. 2024-04-22T08:13:36.393 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[34620]: listing objects. 2024-04-22T08:13:36.393 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[34620]: listed object 0... 2024-04-22T08:13:36.393 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[34620]: listed object 25... 2024-04-22T08:13:36.393 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[34620]: saw 50 objects 2024-04-22T08:13:36.393 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[34620]: shutting down. 2024-04-22T08:13:37.246 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:13:37.246 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[34619]: starting. 2024-04-22T08:13:37.246 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[34619]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-34583 2024-04-22T08:13:37.246 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[34619]: shutting down. 2024-04-22T08:13:37.247 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:13:37.248 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:13:37.248 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:13:37.251 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:13:39.392 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[34687]: starting. 2024-04-22T08:13:39.392 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[34687]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-34686 2024-04-22T08:13:39.392 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[34687]: created object 0... 2024-04-22T08:13:39.393 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[34687]: created object 25... 2024-04-22T08:13:39.393 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[34687]: created object 49... 2024-04-22T08:13:39.393 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[34687]: finishing. 2024-04-22T08:13:39.393 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[34687]: shutting down. 2024-04-22T08:13:40.276 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[34688]: starting. 2024-04-22T08:13:40.276 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[34688]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-34686 2024-04-22T08:13:40.276 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[34688]: shutting down. 2024-04-22T08:13:41.438 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-04-22T08:13:41.439 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-04-22T08:13:41.439 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-22T08:13:41.439 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-22T08:13:41.442 INFO:tasks.ceph.mon.b:Started 2024-04-22T08:13:41.442 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-04-22T08:13:41.442 INFO:tasks.ceph.mon.a:Sent signal 19 2024-04-22T08:13:41.442 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-04-22T08:13:56.443 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-04-22T08:13:56.444 INFO:tasks.ceph.mon.a:Sent signal 18 2024-04-22T08:13:56.444 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-22T08:13:56.444 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:13:56.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.556+0000 7fdbe414c640 10 monclient: get_monmap_and_config 2024-04-22T08:13:56.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.556+0000 7fdbe414c640 10 monclient: build_initial_monmap 2024-04-22T08:13:56.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.556+0000 7fdbe414c640 10 monclient: monmap: 2024-04-22T08:13:56.556 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:13:56.556 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:56.556 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:13:56.557365+0000 2024-04-22T08:13:56.556 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:13:56.557365+0000 2024-04-22T08:13:56.556 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:13:56.556 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:56.556 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:13:56.557 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:13:56.557 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:13:56.557 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:13:56.557 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:13:56.557 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:56.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.557+0000 7fdbe414c640 1 Processor -- start 2024-04-22T08:13:56.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.557+0000 7fdbe414c640 1 -- start start 2024-04-22T08:13:56.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.557+0000 7fdbe414c640 10 monclient: init 2024-04-22T08:13:56.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.557+0000 7fdbe414c640 10 monclient: _reopen_session rank -1 2024-04-22T08:13:56.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.557+0000 7fdbe414c640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-04-22T08:13:56.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.557+0000 7fdbe414c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdbdc133e80 addr v1:172.21.15.189:6789/0 2024-04-22T08:13:56.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.557+0000 7fdbe414c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdbdc09cb40 addr v1:172.21.15.40:6789/0 2024-04-22T08:13:56.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.557+0000 7fdbe414c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbdc13a410 0x7fdbdc13c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:56.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.557+0000 7fdbe414c640 10 monclient(hunting): picked mon.noname-b con 0x7fdbdc13a410 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:13:56.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.557+0000 7fdbe414c640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbdc09de80 con 0x7fdbdc09cb40 2024-04-22T08:13:56.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.557+0000 7fdbe414c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbdc09f130 con 0x7fdbdc133e80 2024-04-22T08:13:56.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.557+0000 7fdbe414c640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:56.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.557+0000 7fdbe414c640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbdc134970 con 0x7fdbdc13a410 2024-04-22T08:13:56.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.557+0000 7fdbe414c640 10 monclient(hunting): _renew_subs 2024-04-22T08:13:56.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.557+0000 7fdbe414c640 10 monclient(hunting): authenticate will time out at 910.551757s 2024-04-22T08:13:56.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.558+0000 7fdbe26e9640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbdc13a410 0x7fdbdc13c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:56.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.558+0000 7fdbe16e7640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fdbdc09cb40 0x7fdbdc138cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:48460/0 (socket says 172.21.15.40:48460) 2024-04-22T08:13:56.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.558+0000 7fdbe16e7640 1 -- 172.21.15.40:0/2517373301 learned_addr learned my addr 172.21.15.40:0/2517373301 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:13:56.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.558+0000 7fdbe26e9640 10 monclient(hunting): get_auth_request con 0x7fdbdc13a410 auth_method 0 2024-04-22T08:13:56.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.558+0000 7fdbe26e9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:56.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.558+0000 7fdbe26e9640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:56.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.558+0000 7fdbe26e9640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:56.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.558+0000 7fdbe26e9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:56.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.558+0000 7fdbe26e9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:56.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.558+0000 7fdbe26e9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:56.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.558+0000 7fdbe26e9640 10 monclient(hunting): handle_auth_done global_id 4478 payload 274 2024-04-22T08:13:56.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.558+0000 7fdbe26e9640 1 -- 172.21.15.40:0/2517373301 >> v1:172.21.15.189:6789/0 conn(0x7fdbdc133e80 legacy=0x7fdbdc134260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:13:56.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.558+0000 7fdbe26e9640 1 -- 172.21.15.40:0/2517373301 >> v1:172.21.15.40:6789/0 conn(0x7fdbdc09cb40 legacy=0x7fdbdc138cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:13:56.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.558+0000 7fdbe26e9640 10 monclient: _finish_hunting 0 2024-04-22T08:13:56.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.558+0000 7fdbe26e9640 1 monclient: found mon.noname-b 2024-04-22T08:13:56.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.558+0000 7fdbe26e9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:13:56.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.558+0000 7fdbe26e9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:13:56.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.558+0000 7fdbe26e9640 1 -- 172.21.15.40:0/2517373301 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbdc09f410 con 0x7fdbdc13a410 2024-04-22T08:13:56.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.561+0000 7fdbe26e9640 1 -- 172.21.15.40:0/2517373301 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbdc13a410 msgr2=0x7fdbdc13c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:13:56.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.561+0000 7fdbe26e9640 1 --2- 172.21.15.40:0/2517373301 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbdc13a410 0x7fdbdc13c800 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fdbd4009dc0 tx=0x7fdbd402bce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1579822c62bcbe0b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:56.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.561+0000 7fdbe0ee6640 1 -- 172.21.15.40:0/2517373301 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd402cd80 con 0x7fdbdc13a410 2024-04-22T08:13:56.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.561+0000 7fdbe0ee6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:56.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.561+0000 7fdbe0ee6640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:13:56.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.561+0000 7fdbe0ee6640 10 monclient: dump: 2024-04-22T08:13:56.568 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:56.568 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:56.568 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:56.568 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:56.568 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:56.568 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:56.568 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:56.568 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:56.569 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:56.569 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:56.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.561+0000 7fdbe0ee6640 10 monclient: _finish_auth 0 2024-04-22T08:13:56.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.561+0000 7fdbe0ee6640 10 monclient: _check_auth_tickets 2024-04-22T08:13:56.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.561+0000 7fdbe0ee6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:13:56.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.561+0000 7fdbe0ee6640 1 -- 172.21.15.40:0/2517373301 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbd4034440 con 0x7fdbdc13a410 2024-04-22T08:13:56.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.561+0000 7fdbe414c640 5 monclient: authenticate success, global_id 4478 2024-04-22T08:13:56.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.561+0000 7fdbe414c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:13:56.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.561+0000 7fdbe0ee6640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:13:56.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.561+0000 7fdbe414c640 10 monclient: get_monmap_and_config success 2024-04-22T08:13:56.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.561+0000 7fdbe414c640 4 set_mon_vals no callback set 2024-04-22T08:13:56.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.561+0000 7fdbe0ee6640 1 -- 172.21.15.40:0/2517373301 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd40347d0 con 0x7fdbdc13a410 2024-04-22T08:13:56.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.561+0000 7fdbe0ee6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:56.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.561+0000 7fdbe0ee6640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:13:56.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.561+0000 7fdbe0ee6640 10 monclient: dump: 2024-04-22T08:13:56.570 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:56.570 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:56.570 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:56.570 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:56.570 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:56.570 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:56.570 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:56.570 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:56.570 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:56.570 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:56.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.562+0000 7fdbe414c640 10 monclient: shutdown 2024-04-22T08:13:56.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.562+0000 7fdbe414c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:13:56.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.562+0000 7fdbe414c640 1 -- 172.21.15.40:0/2517373301 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbdc13a410 msgr2=0x7fdbdc13c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:56.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.562+0000 7fdbe414c640 1 --2- 172.21.15.40:0/2517373301 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbdc13a410 0x7fdbdc13c800 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fdbd4009dc0 tx=0x7fdbd402bce0 comp rx=0 tx=0).stop 2024-04-22T08:13:56.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.562+0000 7fdbe414c640 1 -- 172.21.15.40:0/2517373301 shutdown_connections 2024-04-22T08:13:56.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.562+0000 7fdbe414c640 1 --2- 172.21.15.40:0/2517373301 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbdc13a410 0x7fdbdc13c800 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:56.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.562+0000 7fdbe414c640 1 -- 172.21.15.40:0/2517373301 >> 172.21.15.40:0/2517373301 conn(0x7fdbdc0a65c0 msgr2=0x7fdbdc0a2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:13:56.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.562+0000 7fdbe414c640 1 -- 172.21.15.40:0/2517373301 shutdown_connections 2024-04-22T08:13:56.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.562+0000 7fdbe414c640 1 -- 172.21.15.40:0/2517373301 wait complete. 2024-04-22T08:13:56.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.562+0000 7fdbe414c640 10 monclient: build_initial_monmap 2024-04-22T08:13:56.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.562+0000 7fdbe414c640 10 monclient: monmap: 2024-04-22T08:13:56.571 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:13:56.571 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:13:56.571 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:13:56.571 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:13:56.572 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:13:56.572 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:56.572 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:13:56.572 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:13:56.572 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:13:56.572 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:56.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.563+0000 7fdbe414c640 1 Processor -- start 2024-04-22T08:13:56.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.563+0000 7fdbe414c640 1 -- start start 2024-04-22T08:13:56.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.563+0000 7fdbe414c640 10 monclient: init 2024-04-22T08:13:56.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.563+0000 7fdbe414c640 10 monclient: _reopen_session rank -1 2024-04-22T08:13:56.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.563+0000 7fdbe414c640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-22T08:13:56.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.563+0000 7fdbe414c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbdc09cb40 0x7fdbdc070250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:56.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.563+0000 7fdbe414c640 10 monclient(hunting): picked mon.noname-b con 0x7fdbdc09cb40 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:13:56.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.563+0000 7fdbe414c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbdc133e80 0x7fdbdc070790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:56.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.563+0000 7fdbe414c640 10 monclient(hunting): picked mon.noname-c con 0x7fdbdc133e80 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:13:56.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.563+0000 7fdbe414c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbdc13a410 0x7fdbdc070cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:56.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.563+0000 7fdbe414c640 10 monclient(hunting): picked mon.noname-a con 0x7fdbdc13a410 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:13:56.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.563+0000 7fdbe414c640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:56.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.563+0000 7fdbe414c640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbdc09b650 con 0x7fdbdc13a410 2024-04-22T08:13:56.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.563+0000 7fdbe414c640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:56.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.563+0000 7fdbe414c640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbdc09b8a0 con 0x7fdbdc09cb40 2024-04-22T08:13:56.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.563+0000 7fdbe414c640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:56.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.563+0000 7fdbe414c640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbdc13a160 con 0x7fdbdc133e80 2024-04-22T08:13:56.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.563+0000 7fdbe414c640 10 monclient(hunting): _renew_subs 2024-04-22T08:13:56.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.563+0000 7fdbe414c640 10 monclient(hunting): authenticate will time out at 910.557372s 2024-04-22T08:13:56.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.563+0000 7fdbe16e7640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbdc133e80 0x7fdbdc070790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:56.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.563+0000 7fdbe26e9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbdc13a410 0x7fdbdc070cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:56.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.563+0000 7fdbe16e7640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbdc133e80 0x7fdbdc070790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:54352/0 (socket says 172.21.15.40:54352) 2024-04-22T08:13:56.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.563+0000 7fdbe16e7640 1 -- 172.21.15.40:0/4076014390 learned_addr learned my addr 172.21.15.40:0/4076014390 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:13:56.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.563+0000 7fdbe26e9640 10 monclient(hunting): get_auth_request con 0x7fdbdc13a410 auth_method 0 2024-04-22T08:13:56.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.563+0000 7fdbe26e9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:56.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.563+0000 7fdbe26e9640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:56.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.563+0000 7fdbe26e9640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:56.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.563+0000 7fdbe16e7640 10 monclient(hunting): get_auth_request con 0x7fdbdc133e80 auth_method 0 2024-04-22T08:13:56.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.563+0000 7fdbe16e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:56.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.563+0000 7fdbe16e7640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:56.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.563+0000 7fdbe16e7640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:56.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.563+0000 7fdbe26e9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:56.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.563+0000 7fdbe26e9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:56.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.563+0000 7fdbe26e9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:56.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.563+0000 7fdbe16e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:56.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.563+0000 7fdbe16e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:56.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.563+0000 7fdbe16e7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:56.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.563+0000 7fdbe1ee8640 1 --2- 172.21.15.40:0/4076014390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbdc09cb40 0x7fdbdc070250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:56.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.563+0000 7fdbe1ee8640 10 monclient(hunting): get_auth_request con 0x7fdbdc09cb40 auth_method 0 2024-04-22T08:13:56.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.563+0000 7fdbe1ee8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:56.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.563+0000 7fdbe1ee8640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:56.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.563+0000 7fdbe1ee8640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:56.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.564+0000 7fdbe1ee8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:56.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.564+0000 7fdbe1ee8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:56.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.564+0000 7fdbe1ee8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:56.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.564+0000 7fdbe16e7640 10 monclient(hunting): handle_auth_done global_id 4481 payload 824 2024-04-22T08:13:56.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.564+0000 7fdbe16e7640 1 -- 172.21.15.40:0/4076014390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbdc09cb40 msgr2=0x7fdbdc070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:56.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.564+0000 7fdbe16e7640 1 --2- 172.21.15.40:0/4076014390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbdc09cb40 0x7fdbdc070250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:56.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.564+0000 7fdbe16e7640 1 -- 172.21.15.40:0/4076014390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbdc13a410 msgr2=0x7fdbdc070cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:56.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.564+0000 7fdbe16e7640 1 --2- 172.21.15.40:0/4076014390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbdc13a410 0x7fdbdc070cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:56.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.564+0000 7fdbe16e7640 10 monclient: _finish_hunting 0 2024-04-22T08:13:56.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.564+0000 7fdbe16e7640 1 monclient: found mon.noname-c 2024-04-22T08:13:56.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.564+0000 7fdbe16e7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:13:56.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.564+0000 7fdbe16e7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:13:56.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.564+0000 7fdbe16e7640 1 -- 172.21.15.40:0/4076014390 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbd4009a70 con 0x7fdbdc133e80 2024-04-22T08:13:56.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.564+0000 7fdbe26e9640 1 --2- 172.21.15.40:0/4076014390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbdc13a410 0x7fdbdc070cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:13:56.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.566+0000 7fdbe16e7640 1 -- 172.21.15.40:0/4076014390 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbdc133e80 msgr2=0x7fdbdc070790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:13:56.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.566+0000 7fdbe16e7640 1 --2- 172.21.15.40:0/4076014390 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbdc133e80 0x7fdbdc070790 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc00b440 tx=0x7fdbcc00b900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:56.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.566+0000 7fdbd27fc640 1 -- 172.21.15.40:0/4076014390 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbcc011aa0 con 0x7fdbdc133e80 2024-04-22T08:13:56.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.566+0000 7fdbd27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:56.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.566+0000 7fdbd27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:13:56.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.566+0000 7fdbd27fc640 10 monclient: dump: 2024-04-22T08:13:56.577 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:56.577 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:56.577 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:56.577 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:56.577 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:56.577 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:56.577 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:56.577 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:56.577 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:56.577 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:56.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.566+0000 7fdbd27fc640 10 monclient: _finish_auth 0 2024-04-22T08:13:56.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.566+0000 7fdbd27fc640 10 monclient: _check_auth_tickets 2024-04-22T08:13:56.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.566+0000 7fdbd27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:13:56.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.566+0000 7fdbd27fc640 1 -- 172.21.15.40:0/4076014390 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbcc011c20 con 0x7fdbdc133e80 2024-04-22T08:13:56.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.566+0000 7fdbd27fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:13:56.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.566+0000 7fdbd27fc640 1 -- 172.21.15.40:0/4076014390 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbcc0103e0 con 0x7fdbdc133e80 2024-04-22T08:13:56.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.566+0000 7fdbe414c640 5 monclient: authenticate success, global_id 4481 2024-04-22T08:13:56.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.566+0000 7fdbe414c640 10 monclient: _renew_subs 2024-04-22T08:13:56.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.566+0000 7fdbe414c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:13:56.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.566+0000 7fdbe414c640 1 -- 172.21.15.40:0/4076014390 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbdc09e3b0 con 0x7fdbdc133e80 2024-04-22T08:13:56.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.566+0000 7fdbd3fff640 4 set_mon_vals no callback set 2024-04-22T08:13:56.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.566+0000 7fdbd27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:56.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.566+0000 7fdbd27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:13:56.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.566+0000 7fdbd27fc640 10 monclient: dump: 2024-04-22T08:13:56.579 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:56.579 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:56.579 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:56.579 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:56.579 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:56.579 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:56.579 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:56.579 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:56.579 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:56.579 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:56.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.566+0000 7fdbe414c640 10 monclient: _renew_subs 2024-04-22T08:13:56.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.566+0000 7fdbe414c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:13:56.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.566+0000 7fdbe414c640 1 -- 172.21.15.40:0/4076014390 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbdc09a5f0 con 0x7fdbdc133e80 2024-04-22T08:13:56.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.567+0000 7fdbd27fc640 1 -- 172.21.15.40:0/4076014390 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fdbcc010580 con 0x7fdbdc133e80 2024-04-22T08:13:56.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.567+0000 7fdbe414c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:13:56.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.567+0000 7fdbe414c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:13:56.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.567+0000 7fdbe414c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:13:56.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.567+0000 7fdbe414c640 1 -- 172.21.15.40:0/4076014390 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbdc09acb0 con 0x7fdbdc133e80 2024-04-22T08:13:56.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.567+0000 7fdbd27fc640 1 --2- 172.21.15.40:0/4076014390 >> v2:172.21.15.189:6800/30168 conn(0x7fdbc0037820 0x7fdbc0039ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:56.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.567+0000 7fdbd27fc640 1 -- 172.21.15.40:0/4076014390 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 3506+0+0 (secure 0 0 0) 0x7fdbcc049fd0 con 0x7fdbdc133e80 2024-04-22T08:13:56.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.568+0000 7fdbe1ee8640 1 --2- 172.21.15.40:0/4076014390 >> v2:172.21.15.189:6800/30168 conn(0x7fdbc0037820 0x7fdbc0039ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:56.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.568+0000 7fdbe1ee8640 10 monclient: get_auth_request con 0x7fdbc0037820 auth_method 0 2024-04-22T08:13:56.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.570+0000 7fdbe1ee8640 1 --2- 172.21.15.40:0/4076014390 >> v2:172.21.15.189:6800/30168 conn(0x7fdbc0037820 0x7fdbc0039ce0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8004f60 tx=0x7fdbd8007e30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:56.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.572+0000 7fdbd27fc640 1 -- 172.21.15.40:0/4076014390 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7fdbcc00e8b0 con 0x7fdbdc133e80 2024-04-22T08:13:56.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.572+0000 7fdbd27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:13:56.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.572+0000 7fdbd27fc640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:13:56.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.825+0000 7fdbe414c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:13:56.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.825+0000 7fdbe414c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:13:56.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.825+0000 7fdbe414c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:13:56.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.825+0000 7fdbe414c640 1 -- 172.21.15.40:0/4076014390 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fdbdc09b9e0 con 0x7fdbdc133e80 2024-04-22T08:13:56.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.825+0000 7fdbd27fc640 1 -- 172.21.15.40:0/4076014390 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7fdbdc09b9e0 con 0x7fdbdc133e80 2024-04-22T08:13:56.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.825+0000 7fdbd27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:13:56.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.825+0000 7fdbd27fc640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:13:56.825 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:13:56.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.827+0000 7fdbe414c640 1 -- 172.21.15.40:0/4076014390 >> v2:172.21.15.189:6800/30168 conn(0x7fdbc0037820 msgr2=0x7fdbc0039ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:56.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.827+0000 7fdbe414c640 1 --2- 172.21.15.40:0/4076014390 >> v2:172.21.15.189:6800/30168 conn(0x7fdbc0037820 0x7fdbc0039ce0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8004f60 tx=0x7fdbd8007e30 comp rx=0 tx=0).stop 2024-04-22T08:13:56.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.827+0000 7fdbe414c640 10 monclient: shutdown 2024-04-22T08:13:56.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.827+0000 7fdbe414c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:13:56.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.827+0000 7fdbe414c640 1 -- 172.21.15.40:0/4076014390 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbdc133e80 msgr2=0x7fdbdc070790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:56.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.827+0000 7fdbe414c640 1 --2- 172.21.15.40:0/4076014390 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbdc133e80 0x7fdbdc070790 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc00b440 tx=0x7fdbcc00b900 comp rx=0 tx=0).stop 2024-04-22T08:13:56.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.827+0000 7fdbe414c640 1 -- 172.21.15.40:0/4076014390 shutdown_connections 2024-04-22T08:13:56.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.827+0000 7fdbe414c640 1 --2- 172.21.15.40:0/4076014390 >> v2:172.21.15.189:6800/30168 conn(0x7fdbc0037820 0x7fdbc0039ce0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:56.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.827+0000 7fdbe414c640 1 --2- 172.21.15.40:0/4076014390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbdc13a410 0x7fdbdc070cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:56.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.827+0000 7fdbe414c640 1 --2- 172.21.15.40:0/4076014390 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbdc133e80 0x7fdbdc070790 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:56.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.827+0000 7fdbe414c640 1 --2- 172.21.15.40:0/4076014390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbdc09cb40 0x7fdbdc070250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:56.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.827+0000 7fdbe414c640 1 -- 172.21.15.40:0/4076014390 >> 172.21.15.40:0/4076014390 conn(0x7fdbdc0a65c0 msgr2=0x7fdbdc12ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:13:56.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.827+0000 7fdbe414c640 1 -- 172.21.15.40:0/4076014390 shutdown_connections 2024-04-22T08:13:56.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.827+0000 7fdbe414c640 1 -- 172.21.15.40:0/4076014390 wait complete. 2024-04-22T08:13:56.837 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-22T08:13:56.837 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-22T08:13:56.837 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-22T08:13:56.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.937+0000 7ff26a4a1640 10 monclient: get_monmap_and_config 2024-04-22T08:13:56.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.937+0000 7ff26a4a1640 10 monclient: build_initial_monmap 2024-04-22T08:13:56.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.937+0000 7ff26a4a1640 10 monclient: monmap: 2024-04-22T08:13:56.937 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:13:56.937 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:56.937 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:13:56.937874+0000 2024-04-22T08:13:56.937 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:13:56.937874+0000 2024-04-22T08:13:56.937 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:13:56.937 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:56.937 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:13:56.937 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:13:56.937 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:13:56.937 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:13:56.937 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:13:56.937 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:56.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.937+0000 7ff26a4a1640 1 Processor -- start 2024-04-22T08:13:56.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.937+0000 7ff26a4a1640 1 -- start start 2024-04-22T08:13:56.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.937+0000 7ff26a4a1640 10 monclient: init 2024-04-22T08:13:56.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.938+0000 7ff26a4a1640 10 monclient: _reopen_session rank -1 2024-04-22T08:13:56.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.938+0000 7ff26a4a1640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-04-22T08:13:56.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.938+0000 7ff26a4a1640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff2640a1ba0 0x7ff2640a1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:56.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.938+0000 7ff26a4a1640 10 monclient(hunting): picked mon.noname-b con 0x7ff2640a1ba0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:13:56.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.938+0000 7ff26a4a1640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff264136230 0x7ff2640a2d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:56.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.938+0000 7ff26a4a1640 10 monclient(hunting): picked mon.noname-a con 0x7ff264136230 addr v2:172.21.15.40:3300/0 2024-04-22T08:13:56.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.938+0000 7ff26a4a1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff264136670 addr v1:172.21.15.189:6789/0 2024-04-22T08:13:56.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.938+0000 7ff26a4a1640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff26409fe80 con 0x7ff264136670 2024-04-22T08:13:56.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.938+0000 7ff26a4a1640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:56.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.938+0000 7ff26a4a1640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2640722a0 con 0x7ff264136230 2024-04-22T08:13:56.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.938+0000 7ff26a4a1640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:56.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.938+0000 7ff26a4a1640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2640a4290 con 0x7ff2640a1ba0 2024-04-22T08:13:56.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.938+0000 7ff26a4a1640 10 monclient(hunting): _renew_subs 2024-04-22T08:13:56.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.938+0000 7ff26a4a1640 10 monclient(hunting): authenticate will time out at 910.932082s 2024-04-22T08:13:56.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.938+0000 7ff2637fe640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff264136230 0x7ff2640a2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:56.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.938+0000 7ff263fff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff2640a1ba0 0x7ff2640a1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:56.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.938+0000 7ff2637fe640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff264136230 0x7ff2640a2d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36998/0 (socket says 172.21.15.40:36998) 2024-04-22T08:13:56.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.938+0000 7ff2637fe640 1 -- 172.21.15.40:0/833791832 learned_addr learned my addr 172.21.15.40:0/833791832 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:13:56.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.938+0000 7ff2637fe640 10 monclient(hunting): get_auth_request con 0x7ff264136230 auth_method 0 2024-04-22T08:13:56.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.938+0000 7ff2637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:56.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.938+0000 7ff2637fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:56.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.938+0000 7ff2637fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:56.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.938+0000 7ff263fff640 10 monclient(hunting): get_auth_request con 0x7ff2640a1ba0 auth_method 0 2024-04-22T08:13:56.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.938+0000 7ff263fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:56.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.938+0000 7ff263fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:56.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.938+0000 7ff263fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:56.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.938+0000 7ff263fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:56.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.938+0000 7ff263fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:56.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.938+0000 7ff263fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:56.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.938+0000 7ff2637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:56.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.938+0000 7ff2637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:56.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.938+0000 7ff2637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:56.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.938+0000 7ff263fff640 10 monclient(hunting): handle_auth_done global_id 4484 payload 274 2024-04-22T08:13:56.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.938+0000 7ff263fff640 1 -- 172.21.15.40:0/833791832 >> v2:172.21.15.40:3300/0 conn(0x7ff264136230 msgr2=0x7ff2640a2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:13:56.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.938+0000 7ff263fff640 1 --2- 172.21.15.40:0/833791832 >> v2:172.21.15.40:3300/0 conn(0x7ff264136230 0x7ff2640a2d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:56.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.938+0000 7ff263fff640 1 -- 172.21.15.40:0/833791832 >> v1:172.21.15.189:6789/0 conn(0x7ff264136670 legacy=0x7ff2640a32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:13:56.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.938+0000 7ff263fff640 10 monclient: _finish_hunting 0 2024-04-22T08:13:56.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.938+0000 7ff263fff640 1 monclient: found mon.noname-b 2024-04-22T08:13:56.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.938+0000 7ff263fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:13:56.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.938+0000 7ff263fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:13:56.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.938+0000 7ff263fff640 1 -- 172.21.15.40:0/833791832 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2640a0160 con 0x7ff2640a1ba0 2024-04-22T08:13:56.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.938+0000 7ff2637fe640 1 --2- 172.21.15.40:0/833791832 >> v2:172.21.15.40:3300/0 conn(0x7ff264136230 0x7ff2640a2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:13:56.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.941+0000 7ff263fff640 1 -- 172.21.15.40:0/833791832 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff2640a1ba0 msgr2=0x7ff2640a1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:13:56.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.941+0000 7ff263fff640 1 --2- 172.21.15.40:0/833791832 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff2640a1ba0 0x7ff2640a1f80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff24c0097a0 tx=0x7ff24c02b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6096fda52f726c04 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.941+0000 7ff262ffd640 1 -- 172.21.15.40:0/833791832 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24c02f050 con 0x7ff2640a1ba0 2024-04-22T08:13:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.941+0000 7ff262ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.941+0000 7ff262ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:13:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.941+0000 7ff262ffd640 10 monclient: dump: 2024-04-22T08:13:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:56.953 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:56.953 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:56.953 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:56.953 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:56.953 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:56.953 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:56.953 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:56.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.941+0000 7ff262ffd640 10 monclient: _finish_auth 0 2024-04-22T08:13:56.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.941+0000 7ff262ffd640 10 monclient: _check_auth_tickets 2024-04-22T08:13:56.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.941+0000 7ff262ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:13:56.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.941+0000 7ff262ffd640 1 -- 172.21.15.40:0/833791832 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff24c031910 con 0x7ff2640a1ba0 2024-04-22T08:13:56.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.941+0000 7ff26a4a1640 5 monclient: authenticate success, global_id 4484 2024-04-22T08:13:56.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.941+0000 7ff26a4a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:13:56.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.941+0000 7ff262ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:13:56.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.941+0000 7ff26a4a1640 10 monclient: get_monmap_and_config success 2024-04-22T08:13:56.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.941+0000 7ff262ffd640 1 -- 172.21.15.40:0/833791832 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24c031ca0 con 0x7ff2640a1ba0 2024-04-22T08:13:56.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.941+0000 7ff262ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:56.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.941+0000 7ff26a4a1640 4 set_mon_vals no callback set 2024-04-22T08:13:56.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.941+0000 7ff262ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:13:56.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.941+0000 7ff262ffd640 10 monclient: dump: 2024-04-22T08:13:56.954 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:56.954 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:56.954 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:56.954 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:56.954 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:56.954 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:56.954 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:56.954 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:56.955 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:56.955 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:56.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.942+0000 7ff26a4a1640 10 monclient: shutdown 2024-04-22T08:13:56.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.942+0000 7ff26a4a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:13:56.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.942+0000 7ff26a4a1640 1 -- 172.21.15.40:0/833791832 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff2640a1ba0 msgr2=0x7ff2640a1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:56.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.942+0000 7ff26a4a1640 1 --2- 172.21.15.40:0/833791832 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff2640a1ba0 0x7ff2640a1f80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff24c0097a0 tx=0x7ff24c02b7a0 comp rx=0 tx=0).stop 2024-04-22T08:13:56.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.942+0000 7ff26a4a1640 1 -- 172.21.15.40:0/833791832 shutdown_connections 2024-04-22T08:13:56.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.942+0000 7ff26a4a1640 1 --2- 172.21.15.40:0/833791832 >> v2:172.21.15.40:3300/0 conn(0x7ff264136230 0x7ff2640a2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:56.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.942+0000 7ff26a4a1640 1 --2- 172.21.15.40:0/833791832 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff2640a1ba0 0x7ff2640a1f80 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:56.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.942+0000 7ff26a4a1640 1 -- 172.21.15.40:0/833791832 >> 172.21.15.40:0/833791832 conn(0x7ff2640a4e80 msgr2=0x7ff2640a5290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:13:56.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.942+0000 7ff26a4a1640 1 -- 172.21.15.40:0/833791832 shutdown_connections 2024-04-22T08:13:56.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.942+0000 7ff26a4a1640 1 -- 172.21.15.40:0/833791832 wait complete. 2024-04-22T08:13:56.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.942+0000 7ff26a4a1640 10 monclient: build_initial_monmap 2024-04-22T08:13:56.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.942+0000 7ff26a4a1640 10 monclient: monmap: 2024-04-22T08:13:56.956 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:13:56.956 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:13:56.956 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:13:56.956 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:13:56.956 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:13:56.956 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:56.956 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:13:56.956 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:13:56.956 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:13:56.956 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:56.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.943+0000 7ff26a4a1640 1 Processor -- start 2024-04-22T08:13:56.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.943+0000 7ff26a4a1640 1 -- start start 2024-04-22T08:13:56.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.943+0000 7ff26a4a1640 10 monclient: init 2024-04-22T08:13:56.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.943+0000 7ff26a4a1640 10 monclient: _reopen_session rank -1 2024-04-22T08:13:56.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.943+0000 7ff26a4a1640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:13:56.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.943+0000 7ff26a4a1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2640a1ba0 0x7ff2641cab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:56.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.943+0000 7ff26a4a1640 10 monclient(hunting): picked mon.noname-a con 0x7ff2640a1ba0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:13:56.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.943+0000 7ff26a4a1640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff264136230 0x7ff2641cb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:56.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.943+0000 7ff26a4a1640 10 monclient(hunting): picked mon.noname-b con 0x7ff264136230 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:13:56.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.943+0000 7ff26a4a1640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff264136670 0x7ff264122230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:56.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.943+0000 7ff26a4a1640 10 monclient(hunting): picked mon.noname-c con 0x7ff264136670 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:13:56.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.943+0000 7ff26a4a1640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:56.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.943+0000 7ff26a4a1640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff264125eb0 con 0x7ff2640a1ba0 2024-04-22T08:13:56.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.943+0000 7ff26a4a1640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:56.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.943+0000 7ff26a4a1640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff264126120 con 0x7ff264136230 2024-04-22T08:13:56.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.943+0000 7ff26a4a1640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:56.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.943+0000 7ff26a4a1640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2640a2710 con 0x7ff264136670 2024-04-22T08:13:56.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.943+0000 7ff26a4a1640 10 monclient(hunting): _renew_subs 2024-04-22T08:13:56.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.943+0000 7ff26a4a1640 10 monclient(hunting): authenticate will time out at 910.937737s 2024-04-22T08:13:56.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.944+0000 7ff263fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2640a1ba0 0x7ff2641cab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:56.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.944+0000 7ff268a3e640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff264136670 0x7ff264122230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:56.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.944+0000 7ff263fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2640a1ba0 0x7ff2641cab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37014/0 (socket says 172.21.15.40:37014) 2024-04-22T08:13:56.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.944+0000 7ff263fff640 1 -- 172.21.15.40:0/1133586158 learned_addr learned my addr 172.21.15.40:0/1133586158 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:13:56.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.944+0000 7ff263fff640 10 monclient(hunting): get_auth_request con 0x7ff2640a1ba0 auth_method 0 2024-04-22T08:13:56.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.944+0000 7ff263fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:56.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.944+0000 7ff263fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:56.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.944+0000 7ff263fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:56.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.944+0000 7ff268a3e640 10 monclient(hunting): get_auth_request con 0x7ff264136670 auth_method 0 2024-04-22T08:13:56.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.944+0000 7ff268a3e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:56.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.944+0000 7ff268a3e640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:56.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.944+0000 7ff268a3e640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:56.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.944+0000 7ff263fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:56.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.944+0000 7ff263fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:56.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.944+0000 7ff263fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:56.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.944+0000 7ff268a3e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:56.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.944+0000 7ff268a3e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:56.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.944+0000 7ff268a3e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:56.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.944+0000 7ff268a3e640 10 monclient(hunting): handle_auth_done global_id 4487 payload 824 2024-04-22T08:13:56.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.944+0000 7ff2637fe640 1 --2- 172.21.15.40:0/1133586158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff264136230 0x7ff2641cb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:56.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.944+0000 7ff268a3e640 1 -- 172.21.15.40:0/1133586158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff264136230 msgr2=0x7ff2641cb0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:56.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.944+0000 7ff268a3e640 1 --2- 172.21.15.40:0/1133586158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff264136230 0x7ff2641cb0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:56.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.944+0000 7ff268a3e640 1 -- 172.21.15.40:0/1133586158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2640a1ba0 msgr2=0x7ff2641cab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:56.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.944+0000 7ff268a3e640 1 --2- 172.21.15.40:0/1133586158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2640a1ba0 0x7ff2641cab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:56.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.944+0000 7ff268a3e640 10 monclient: _finish_hunting 0 2024-04-22T08:13:56.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.944+0000 7ff268a3e640 1 monclient: found mon.noname-c 2024-04-22T08:13:56.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.944+0000 7ff268a3e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:13:56.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.944+0000 7ff268a3e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:13:56.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.944+0000 7ff268a3e640 1 -- 172.21.15.40:0/1133586158 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2540094d0 con 0x7ff264136670 2024-04-22T08:13:56.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.944+0000 7ff263fff640 1 --2- 172.21.15.40:0/1133586158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2640a1ba0 0x7ff2641cab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:13:56.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.946+0000 7ff268a3e640 1 -- 172.21.15.40:0/1133586158 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff264136670 msgr2=0x7ff264122230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:13:56.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.946+0000 7ff268a3e640 1 --2- 172.21.15.40:0/1133586158 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff264136670 0x7ff264122230 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff25c00c7b0 tx=0x7ff25c00cc70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:56.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.946+0000 7ff260ff9640 1 -- 172.21.15.40:0/1133586158 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25c00ac60 con 0x7ff264136670 2024-04-22T08:13:56.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.946+0000 7ff260ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:56.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.946+0000 7ff260ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:13:56.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.946+0000 7ff260ff9640 10 monclient: dump: 2024-04-22T08:13:56.961 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:56.962 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:56.962 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:56.962 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:56.962 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:56.962 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:56.962 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:56.963 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:56.963 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:56.963 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:56.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.946+0000 7ff260ff9640 10 monclient: _finish_auth 0 2024-04-22T08:13:56.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.946+0000 7ff260ff9640 10 monclient: _check_auth_tickets 2024-04-22T08:13:56.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.946+0000 7ff260ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:13:56.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.946+0000 7ff26a4a1640 5 monclient: authenticate success, global_id 4487 2024-04-22T08:13:56.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.946+0000 7ff260ff9640 1 -- 172.21.15.40:0/1133586158 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff25c00adc0 con 0x7ff264136670 2024-04-22T08:13:56.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.946+0000 7ff260ff9640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:13:56.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.946+0000 7ff260ff9640 1 -- 172.21.15.40:0/1133586158 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25c012660 con 0x7ff264136670 2024-04-22T08:13:56.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.946+0000 7ff26a4a1640 10 monclient: _renew_subs 2024-04-22T08:13:56.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.946+0000 7ff26a4a1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:13:56.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.946+0000 7ff26a4a1640 1 -- 172.21.15.40:0/1133586158 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff24c009400 con 0x7ff264136670 2024-04-22T08:13:56.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.946+0000 7ff261ffb640 4 set_mon_vals no callback set 2024-04-22T08:13:56.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.946+0000 7ff260ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:56.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.946+0000 7ff260ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:13:56.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.946+0000 7ff260ff9640 10 monclient: dump: 2024-04-22T08:13:56.964 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:56.964 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:56.964 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:56.964 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:56.965 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:56.965 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:56.965 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:56.965 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:56.965 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:56.965 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:56.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.946+0000 7ff26a4a1640 10 monclient: _renew_subs 2024-04-22T08:13:56.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.946+0000 7ff26a4a1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:13:56.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.946+0000 7ff26a4a1640 1 -- 172.21.15.40:0/1133586158 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff264123d50 con 0x7ff264136670 2024-04-22T08:13:56.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.947+0000 7ff26a4a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:13:56.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.947+0000 7ff26a4a1640 1 --2- 172.21.15.40:0/1133586158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff264122770 0x7ff264122b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:56.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.947+0000 7ff26a4a1640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:56.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.947+0000 7ff26a4a1640 1 -- 172.21.15.40:0/1133586158 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2641262b0 con 0x7ff264122770 2024-04-22T08:13:56.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.947+0000 7ff260ff9640 1 -- 172.21.15.40:0/1133586158 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7ff25c012800 con 0x7ff264136670 2024-04-22T08:13:56.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.947+0000 7ff263fff640 1 --2- 172.21.15.40:0/1133586158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff264122770 0x7ff264122b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:56.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.947+0000 7ff263fff640 10 monclient: get_auth_request con 0x7ff264122770 auth_method 0 2024-04-22T08:13:56.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.947+0000 7ff263fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:56.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.947+0000 7ff263fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:56.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.947+0000 7ff263fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:56.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.947+0000 7ff263fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:56.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.947+0000 7ff263fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:56.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.947+0000 7ff263fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:56.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.948+0000 7ff260ff9640 1 --2- 172.21.15.40:0/1133586158 >> v2:172.21.15.189:6800/30168 conn(0x7ff244037850 0x7ff244039d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:56.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.948+0000 7ff263fff640 10 monclient(hunting): handle_auth_done global_id 4518 payload 824 2024-04-22T08:13:56.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.948+0000 7ff260ff9640 1 -- 172.21.15.40:0/1133586158 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 3898+0+0 (secure 0 0 0) 0x7ff25c04a420 con 0x7ff264136670 2024-04-22T08:13:56.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.948+0000 7ff263fff640 1 -- 172.21.15.40:0/1133586158 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff264125d10 con 0x7ff264122770 2024-04-22T08:13:56.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.948+0000 7ff2637fe640 1 --2- 172.21.15.40:0/1133586158 >> v2:172.21.15.189:6800/30168 conn(0x7ff244037850 0x7ff244039d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:56.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.948+0000 7ff2637fe640 10 monclient: get_auth_request con 0x7ff244037850 auth_method 0 2024-04-22T08:13:56.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.950+0000 7ff263fff640 1 -- 172.21.15.40:0/1133586158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff264122770 msgr2=0x7ff264122b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:13:56.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.950+0000 7ff263fff640 1 --2- 172.21.15.40:0/1133586158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff264122770 0x7ff264122b50 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff24c0097a0 tx=0x7ff24c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:56.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.950+0000 7ff260ff9640 1 -- 172.21.15.40:0/1133586158 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24c03d640 con 0x7ff264122770 2024-04-22T08:13:56.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.950+0000 7ff260ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:56.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.950+0000 7ff260ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:13:56.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.950+0000 7ff260ff9640 10 monclient: dump: 2024-04-22T08:13:56.968 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:56.968 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:56.968 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:56.968 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:56.969 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:56.969 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:56.969 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:56.969 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:56.969 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:56.969 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:56.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.950+0000 7ff260ff9640 1 -- 172.21.15.40:0/1133586158 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7ff24c041070 con 0x7ff264122770 2024-04-22T08:13:56.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.950+0000 7ff260ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:13:56.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.950+0000 7ff260ff9640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:13:56.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.950+0000 7ff260ff9640 1 -- 172.21.15.40:0/1133586158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff264122770 msgr2=0x7ff264122b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:56.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.950+0000 7ff260ff9640 1 --2- 172.21.15.40:0/1133586158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff264122770 0x7ff264122b50 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff24c0097a0 tx=0x7ff24c0046c0 comp rx=0 tx=0).stop 2024-04-22T08:13:56.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.950+0000 7ff260ff9640 1 -- 172.21.15.40:0/1133586158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff264122770 msgr2=0x7ff264122b50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:13:56.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.950+0000 7ff260ff9640 1 --2- 172.21.15.40:0/1133586158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff264122770 0x7ff264122b50 secure :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff24c0097a0 tx=0x7ff24c0046c0 comp rx=0 tx=0).stop 2024-04-22T08:13:56.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.950+0000 7ff2637fe640 1 --2- 172.21.15.40:0/1133586158 >> v2:172.21.15.189:6800/30168 conn(0x7ff244037850 0x7ff244039d10 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff2640a81f0 tx=0x7ff254008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:56.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.957+0000 7ff26a4a1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:13:56.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.957+0000 7ff26a4a1640 1 --2- 172.21.15.40:0/1133586158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff264123090 0x7ff26411e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:56.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.957+0000 7ff26a4a1640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:56.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.957+0000 7ff26a4a1640 1 -- 172.21.15.40:0/1133586158 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff264066650 con 0x7ff264123090 2024-04-22T08:13:56.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.957+0000 7ff263fff640 1 --2- 172.21.15.40:0/1133586158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff264123090 0x7ff26411e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:56.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.957+0000 7ff263fff640 10 monclient: get_auth_request con 0x7ff264123090 auth_method 0 2024-04-22T08:13:56.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.957+0000 7ff263fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:56.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.957+0000 7ff263fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:56.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.957+0000 7ff263fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:56.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.957+0000 7ff263fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:56.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.957+0000 7ff263fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:56.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.957+0000 7ff263fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:56.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.957+0000 7ff263fff640 10 monclient(hunting): handle_auth_done global_id 4521 payload 824 2024-04-22T08:13:56.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.957+0000 7ff263fff640 1 -- 172.21.15.40:0/1133586158 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff2640a2440 con 0x7ff264123090 2024-04-22T08:13:56.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.959+0000 7ff263fff640 1 -- 172.21.15.40:0/1133586158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff264123090 msgr2=0x7ff26411e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:13:56.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.959+0000 7ff263fff640 1 --2- 172.21.15.40:0/1133586158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff264123090 0x7ff26411e9d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff24c034000 tx=0x7ff24c031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:56.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.959+0000 7ff260ff9640 1 -- 172.21.15.40:0/1133586158 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24c02f040 con 0x7ff264123090 2024-04-22T08:13:56.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.959+0000 7ff260ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:56.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.959+0000 7ff260ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:13:56.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.959+0000 7ff260ff9640 10 monclient: dump: 2024-04-22T08:13:56.974 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:56.974 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:56.974 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:56.974 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:56.974 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:56.974 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:56.974 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:56.974 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:56.974 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:56.975 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:56.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.960+0000 7ff260ff9640 1 -- 172.21.15.40:0/1133586158 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4803 (secure 0 0 0) 0x7ff24c047c90 con 0x7ff264123090 2024-04-22T08:13:56.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.960+0000 7ff260ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:13:56.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.960+0000 7ff260ff9640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:13:56.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.960+0000 7ff260ff9640 1 -- 172.21.15.40:0/1133586158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff264123090 msgr2=0x7ff26411e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:56.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.960+0000 7ff260ff9640 1 --2- 172.21.15.40:0/1133586158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff264123090 0x7ff26411e9d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff24c034000 tx=0x7ff24c031000 comp rx=0 tx=0).stop 2024-04-22T08:13:56.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.960+0000 7ff260ff9640 1 -- 172.21.15.40:0/1133586158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff264123090 msgr2=0x7ff26411e9d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:13:56.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.960+0000 7ff260ff9640 1 --2- 172.21.15.40:0/1133586158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff264123090 0x7ff26411e9d0 secure :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff24c034000 tx=0x7ff24c031000 comp rx=0 tx=0).stop 2024-04-22T08:13:56.976 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:13:56.976 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:13:56.976 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:13:56.976 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "leader", 2024-04-22T08:13:56.976 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 20, 2024-04-22T08:13:56.976 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:13:56.976 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:13:56.976 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:13:56.976 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:13:56.976 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:13:56.976 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 0, 2024-04-22T08:13:56.976 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:13:56.977 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:13:56.977 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:13:56.977 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:13:56.977 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:13:56.977 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:13:56.977 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:13:56.977 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:13:56.977 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:13:56.977 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:13:56.977 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:13:56.977 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:13:56.977 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:13:56.978 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:13:56.978 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:13:56.978 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:13:56.978 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:13:56.978 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:13:56.978 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:13:56.978 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:13:56.978 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:13:56.978 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:13:56.978 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:13:56.978 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:13:56.978 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:13:56.978 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:13:56.978 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:56.979 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:56.979 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:13:56.979 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:13:56.979 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:13:56.979 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:13:56.979 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:13:56.979 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:13:56.979 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:13:56.979 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:13:56.979 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:13:56.979 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:13:56.979 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:13:56.979 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:13:56.979 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:13:56.980 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:13:56.980 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:13:56.980 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:13:56.980 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:13:56.980 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:13:56.980 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:13:56.980 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:13:56.980 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:13:56.980 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:13:56.980 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:13:56.980 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:13:56.980 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:13:56.980 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:13:56.980 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:13:56.981 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:13:56.981 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:13:56.981 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:56.981 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:13:56.981 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:56.981 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:13:56.981 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:13:56.981 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:13:56.981 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:13:56.981 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:56.981 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:13:56.981 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:13:56.981 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:56.981 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:56.982 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:56.982 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:13:56.982 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:13:56.982 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:56.982 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:56.982 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:56.982 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:56.982 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:13:56.982 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:13:56.982 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:13:56.982 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:13:56.982 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:13:56.982 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:56.982 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:56.983 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:13:56.983 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:13:56.983 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:13:56.983 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:13:56.983 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:56.983 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:13:56.983 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:13:56.983 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:56.983 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:56.983 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:56.983 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:13:56.983 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:13:56.983 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:56.983 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:56.983 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:56.984 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:56.984 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:13:56.984 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:13:56.984 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:13:56.984 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:13:56.984 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:13:56.984 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:56.984 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:56.984 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:13:56.984 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:13:56.984 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:13:56.984 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:13:56.984 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:56.984 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:13:56.984 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:13:56.985 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:56.985 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:56.985 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:56.985 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:13:56.985 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:13:56.985 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:56.985 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:56.985 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:56.985 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:56.985 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:13:56.985 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:13:56.985 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:13:56.985 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:13:56.985 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:13:56.986 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:56.986 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:56.986 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:56.986 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:13:56.986 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:13:56.986 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:56.986 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:13:56.986 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:13:56.986 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:13:56.986 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:56.986 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:13:56.986 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:13:56.986 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:56.986 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:13:56.987 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:13:56.987 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:13:56.987 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:56.987 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:13:56.987 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:13:56.987 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:56.987 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:13:56.987 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:13:56.987 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 5 2024-04-22T08:13:56.987 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:56.987 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:13:56.987 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": [ 2024-04-22T08:13:56.987 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:56.987 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:13:56.987 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:13:56.988 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:13:56.988 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:56.988 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:56.988 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:56.988 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:13:56.988 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:13:56.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.960+0000 7ff26a4a1640 1 -- 172.21.15.40:0/1133586158 >> v2:172.21.15.189:6800/30168 conn(0x7ff244037850 msgr2=0x7ff244039d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:56.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.960+0000 7ff26a4a1640 1 --2- 172.21.15.40:0/1133586158 >> v2:172.21.15.189:6800/30168 conn(0x7ff244037850 0x7ff244039d10 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff2640a81f0 tx=0x7ff254008040 comp rx=0 tx=0).stop 2024-04-22T08:13:56.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.960+0000 7ff26a4a1640 10 monclient: shutdown 2024-04-22T08:13:56.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.960+0000 7ff26a4a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:13:56.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.960+0000 7ff26a4a1640 1 -- 172.21.15.40:0/1133586158 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff264136670 msgr2=0x7ff264122230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:56.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.960+0000 7ff26a4a1640 1 --2- 172.21.15.40:0/1133586158 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff264136670 0x7ff264122230 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff25c00c7b0 tx=0x7ff25c00cc70 comp rx=0 tx=0).stop 2024-04-22T08:13:56.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.960+0000 7ff268a3e640 1 -- 172.21.15.40:0/1133586158 reap_dead start 2024-04-22T08:13:56.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.960+0000 7ff268a3e640 1 -- 172.21.15.40:0/1133586158 reap_dead start 2024-04-22T08:13:56.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.960+0000 7ff26a4a1640 1 -- 172.21.15.40:0/1133586158 shutdown_connections 2024-04-22T08:13:56.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.960+0000 7ff26a4a1640 1 -- 172.21.15.40:0/1133586158 >> 172.21.15.40:0/1133586158 conn(0x7ff2640a4e80 msgr2=0x7ff2640a08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:13:56.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.960+0000 7ff26a4a1640 1 -- 172.21.15.40:0/1133586158 shutdown_connections 2024-04-22T08:13:56.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:56.960+0000 7ff26a4a1640 1 -- 172.21.15.40:0/1133586158 wait complete. 2024-04-22T08:13:56.990 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:13:56.990 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-22T08:13:56.991 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-22T08:13:57.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.090+0000 7fd410b0a640 10 monclient: get_monmap_and_config 2024-04-22T08:13:57.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.090+0000 7fd410b0a640 10 monclient: build_initial_monmap 2024-04-22T08:13:57.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.090+0000 7fd410b0a640 10 monclient: monmap: 2024-04-22T08:13:57.090 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:13:57.090 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:57.090 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:13:57.091626+0000 2024-04-22T08:13:57.091 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:13:57.091626+0000 2024-04-22T08:13:57.091 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:13:57.091 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:57.091 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:13:57.091 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:13:57.091 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:13:57.091 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:13:57.091 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:13:57.091 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:57.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.091+0000 7fd410b0a640 1 Processor -- start 2024-04-22T08:13:57.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.091+0000 7fd410b0a640 1 -- start start 2024-04-22T08:13:57.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.091+0000 7fd410b0a640 10 monclient: init 2024-04-22T08:13:57.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.091+0000 7fd410b0a640 10 monclient: _reopen_session rank -1 2024-04-22T08:13:57.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.091+0000 7fd410b0a640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-04-22T08:13:57.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.092+0000 7fd410b0a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd40c0a1a80 addr v1:172.21.15.40:6789/0 2024-04-22T08:13:57.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.092+0000 7fd410b0a640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd40c0a2570 0x7fd40c0a01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:57.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.092+0000 7fd410b0a640 10 monclient(hunting): picked mon.noname-a con 0x7fd40c0a2570 addr v2:172.21.15.40:3300/0 2024-04-22T08:13:57.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.092+0000 7fd410b0a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd40c13b0e0 addr v1:172.21.15.189:6789/0 2024-04-22T08:13:57.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.092+0000 7fd410b0a640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd40c09d860 con 0x7fd40c0a1a80 2024-04-22T08:13:57.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.092+0000 7fd410b0a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd40c09ef50 con 0x7fd40c13b0e0 2024-04-22T08:13:57.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.092+0000 7fd410b0a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:57.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.092+0000 7fd410b0a640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd40c09f880 con 0x7fd40c0a2570 2024-04-22T08:13:57.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.092+0000 7fd410b0a640 10 monclient(hunting): _renew_subs 2024-04-22T08:13:57.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.092+0000 7fd410b0a640 10 monclient(hunting): authenticate will time out at 911.085838s 2024-04-22T08:13:57.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.092+0000 7fd409d9b640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd40c0a2570 0x7fd40c0a01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:57.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.092+0000 7fd409d9b640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd40c0a2570 0x7fd40c0a01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37042/0 (socket says 172.21.15.40:37042) 2024-04-22T08:13:57.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.092+0000 7fd409d9b640 1 -- 172.21.15.40:0/2135532962 learned_addr learned my addr 172.21.15.40:0/2135532962 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:13:57.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.092+0000 7fd409d9b640 10 monclient(hunting): get_auth_request con 0x7fd40c0a2570 auth_method 0 2024-04-22T08:13:57.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.092+0000 7fd409d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:57.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.092+0000 7fd409d9b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:57.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.092+0000 7fd409d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:57.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.094+0000 7fd40a59c640 1 -- 172.21.15.40:0/2135532962 >> v1:172.21.15.40:6789/0 conn(0x7fd40c0a1a80 legacy=0x7fd40c0a1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:13:57.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.094+0000 7fd409d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:57.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.094+0000 7fd409d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:57.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.094+0000 7fd409d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:57.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.094+0000 7fd40ad9d640 1 -- 172.21.15.40:0/2135532962 >> v1:172.21.15.189:6789/0 conn(0x7fd40c13b0e0 legacy=0x7fd40c13d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:13:57.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.094+0000 7fd409d9b640 10 monclient(hunting): handle_auth_done global_id 4524 payload 274 2024-04-22T08:13:57.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.094+0000 7fd409d9b640 1 -- 172.21.15.40:0/2135532962 >> v1:172.21.15.189:6789/0 conn(0x7fd40c13b0e0 legacy=0x7fd40c13d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:57.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.094+0000 7fd409d9b640 1 -- 172.21.15.40:0/2135532962 >> v1:172.21.15.40:6789/0 conn(0x7fd40c0a1a80 legacy=0x7fd40c0a1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:57.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.094+0000 7fd409d9b640 10 monclient: _finish_hunting 0 2024-04-22T08:13:57.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.094+0000 7fd409d9b640 1 monclient: found mon.noname-a 2024-04-22T08:13:57.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.094+0000 7fd409d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:13:57.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.094+0000 7fd409d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:13:57.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.094+0000 7fd409d9b640 1 -- 172.21.15.40:0/2135532962 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd40c072160 con 0x7fd40c0a2570 2024-04-22T08:13:57.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.097+0000 7fd409d9b640 1 -- 172.21.15.40:0/2135532962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd40c0a2570 msgr2=0x7fd40c0a01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:13:57.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.097+0000 7fd409d9b640 1 --2- 172.21.15.40:0/2135532962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd40c0a2570 0x7fd40c0a01d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd3f4009970 tx=0x7fd3f402ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7edf4054eaad87b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:57.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.097+0000 7fd40959a640 1 -- 172.21.15.40:0/2135532962 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f402fb20 con 0x7fd40c0a2570 2024-04-22T08:13:57.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.097+0000 7fd40959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:57.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.097+0000 7fd40959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:13:57.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.097+0000 7fd40959a640 10 monclient: dump: 2024-04-22T08:13:57.102 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:57.102 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:57.102 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:57.102 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:57.102 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:57.102 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:57.102 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:57.102 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:57.102 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:57.102 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:57.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.097+0000 7fd40959a640 10 monclient: _finish_auth 0 2024-04-22T08:13:57.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.097+0000 7fd40959a640 10 monclient: _check_auth_tickets 2024-04-22T08:13:57.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.097+0000 7fd40959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:13:57.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.097+0000 7fd410b0a640 5 monclient: authenticate success, global_id 4524 2024-04-22T08:13:57.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.097+0000 7fd410b0a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:13:57.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.097+0000 7fd40959a640 1 -- 172.21.15.40:0/2135532962 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3f402fc80 con 0x7fd40c0a2570 2024-04-22T08:13:57.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.097+0000 7fd40959a640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:13:57.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.097+0000 7fd410b0a640 10 monclient: get_monmap_and_config success 2024-04-22T08:13:57.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.097+0000 7fd410b0a640 4 set_mon_vals no callback set 2024-04-22T08:13:57.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.097+0000 7fd40959a640 1 -- 172.21.15.40:0/2135532962 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f4037400 con 0x7fd40c0a2570 2024-04-22T08:13:57.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.097+0000 7fd40959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:57.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.097+0000 7fd40959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:13:57.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.097+0000 7fd40959a640 10 monclient: dump: 2024-04-22T08:13:57.103 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:57.104 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:57.104 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:57.104 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:57.104 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:57.104 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:57.104 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:57.104 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:57.104 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:57.104 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:57.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.098+0000 7fd410b0a640 10 monclient: shutdown 2024-04-22T08:13:57.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.098+0000 7fd410b0a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:13:57.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.098+0000 7fd410b0a640 1 -- 172.21.15.40:0/2135532962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd40c0a2570 msgr2=0x7fd40c0a01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:57.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.098+0000 7fd410b0a640 1 --2- 172.21.15.40:0/2135532962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd40c0a2570 0x7fd40c0a01d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd3f4009970 tx=0x7fd3f402ba10 comp rx=0 tx=0).stop 2024-04-22T08:13:57.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.098+0000 7fd410b0a640 1 -- 172.21.15.40:0/2135532962 shutdown_connections 2024-04-22T08:13:57.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.098+0000 7fd410b0a640 1 --2- 172.21.15.40:0/2135532962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd40c0a2570 0x7fd40c0a01d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:57.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.098+0000 7fd410b0a640 1 -- 172.21.15.40:0/2135532962 >> 172.21.15.40:0/2135532962 conn(0x7fd40c0a66a0 msgr2=0x7fd40c0a2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:13:57.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.098+0000 7fd410b0a640 1 -- 172.21.15.40:0/2135532962 shutdown_connections 2024-04-22T08:13:57.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.098+0000 7fd410b0a640 1 -- 172.21.15.40:0/2135532962 wait complete. 2024-04-22T08:13:57.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.099+0000 7fd410b0a640 10 monclient: build_initial_monmap 2024-04-22T08:13:57.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.099+0000 7fd410b0a640 10 monclient: monmap: 2024-04-22T08:13:57.105 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:13:57.105 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:13:57.105 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:13:57.105 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:13:57.105 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:13:57.105 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:57.105 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:13:57.105 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:13:57.106 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:13:57.106 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:57.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.099+0000 7fd410b0a640 1 Processor -- start 2024-04-22T08:13:57.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.099+0000 7fd410b0a640 1 -- start start 2024-04-22T08:13:57.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.099+0000 7fd410b0a640 10 monclient: init 2024-04-22T08:13:57.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.099+0000 7fd410b0a640 10 monclient: _reopen_session rank -1 2024-04-22T08:13:57.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd410b0a640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-22T08:13:57.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd410b0a640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd40c0a1a80 0x7fd40c0706d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:57.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd410b0a640 10 monclient(hunting): picked mon.noname-c con 0x7fd40c0a1a80 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:13:57.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd410b0a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd40c0a2570 0x7fd40c1cac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:57.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd410b0a640 10 monclient(hunting): picked mon.noname-b con 0x7fd40c0a2570 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:13:57.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd410b0a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd40c13b0e0 0x7fd40c1cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:57.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd410b0a640 10 monclient(hunting): picked mon.noname-a con 0x7fd40c13b0e0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:13:57.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd410b0a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:57.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd410b0a640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd40c122bd0 con 0x7fd40c13b0e0 2024-04-22T08:13:57.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd410b0a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:57.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd410b0a640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd40c071890 con 0x7fd40c0a2570 2024-04-22T08:13:57.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd410b0a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:57.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd410b0a640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd40c121df0 con 0x7fd40c0a1a80 2024-04-22T08:13:57.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd410b0a640 10 monclient(hunting): _renew_subs 2024-04-22T08:13:57.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd410b0a640 10 monclient(hunting): authenticate will time out at 911.093836s 2024-04-22T08:13:57.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd40ad9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd40c13b0e0 0x7fd40c1cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:57.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd40a59c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd40c0a1a80 0x7fd40c0706d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:57.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd40ad9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd40c13b0e0 0x7fd40c1cb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37054/0 (socket says 172.21.15.40:37054) 2024-04-22T08:13:57.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd40ad9d640 1 -- 172.21.15.40:0/1022296325 learned_addr learned my addr 172.21.15.40:0/1022296325 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:13:57.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd40ad9d640 10 monclient(hunting): get_auth_request con 0x7fd40c13b0e0 auth_method 0 2024-04-22T08:13:57.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd40ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:57.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd40ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:57.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd40ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:57.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd40a59c640 10 monclient(hunting): get_auth_request con 0x7fd40c0a1a80 auth_method 0 2024-04-22T08:13:57.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd40a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:57.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd40a59c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:57.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd40a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:57.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd40ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:57.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd40ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:57.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd40ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:57.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd40a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:57.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd40a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:57.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd40a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:57.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd409d9b640 1 --2- 172.21.15.40:0/1022296325 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd40c0a2570 0x7fd40c1cac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:57.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd409d9b640 10 monclient(hunting): get_auth_request con 0x7fd40c0a2570 auth_method 0 2024-04-22T08:13:57.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd409d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:57.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd409d9b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:57.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd409d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:57.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd40ad9d640 10 monclient(hunting): handle_auth_done global_id 4530 payload 824 2024-04-22T08:13:57.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd40ad9d640 1 -- 172.21.15.40:0/1022296325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd40c0a1a80 msgr2=0x7fd40c0706d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:57.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd40ad9d640 1 --2- 172.21.15.40:0/1022296325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd40c0a1a80 0x7fd40c0706d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:57.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd40ad9d640 1 -- 172.21.15.40:0/1022296325 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd40c0a2570 msgr2=0x7fd40c1cac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:57.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd40ad9d640 1 --2- 172.21.15.40:0/1022296325 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd40c0a2570 0x7fd40c1cac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:57.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd40ad9d640 10 monclient: _finish_hunting 0 2024-04-22T08:13:57.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd40ad9d640 1 monclient: found mon.noname-a 2024-04-22T08:13:57.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd40ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:13:57.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd40ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:13:57.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd40ad9d640 1 -- 172.21.15.40:0/1022296325 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f40094d0 con 0x7fd40c13b0e0 2024-04-22T08:13:57.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd409d9b640 1 --2- 172.21.15.40:0/1022296325 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd40c0a2570 0x7fd40c1cac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T08:13:57.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.100+0000 7fd40a59c640 1 --2- 172.21.15.40:0/1022296325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd40c0a1a80 0x7fd40c0706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:13:57.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.102+0000 7fd40ad9d640 1 -- 172.21.15.40:0/1022296325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd40c13b0e0 msgr2=0x7fd40c1cb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:13:57.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.102+0000 7fd40ad9d640 1 --2- 172.21.15.40:0/1022296325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd40c13b0e0 0x7fd40c1cb1b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd40400c3d0 tx=0x7fd40400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:57.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.102+0000 7fd402ffd640 1 -- 172.21.15.40:0/1022296325 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4040057c0 con 0x7fd40c13b0e0 2024-04-22T08:13:57.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.102+0000 7fd402ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:57.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.102+0000 7fd402ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:13:57.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.102+0000 7fd402ffd640 10 monclient: dump: 2024-04-22T08:13:57.114 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:57.114 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:57.114 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:57.114 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:57.114 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:57.114 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:57.114 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:57.114 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:57.114 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:57.114 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:57.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.102+0000 7fd402ffd640 10 monclient: _finish_auth 0 2024-04-22T08:13:57.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.102+0000 7fd402ffd640 10 monclient: _check_auth_tickets 2024-04-22T08:13:57.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.102+0000 7fd402ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:13:57.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.102+0000 7fd410b0a640 5 monclient: authenticate success, global_id 4530 2024-04-22T08:13:57.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.102+0000 7fd410b0a640 10 monclient: _renew_subs 2024-04-22T08:13:57.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.102+0000 7fd410b0a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:13:57.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.102+0000 7fd402ffd640 1 -- 172.21.15.40:0/1022296325 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd404005920 con 0x7fd40c13b0e0 2024-04-22T08:13:57.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.102+0000 7fd410b0a640 1 -- 172.21.15.40:0/1022296325 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd40c09cc20 con 0x7fd40c13b0e0 2024-04-22T08:13:57.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.102+0000 7fd402ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:13:57.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.102+0000 7fd402ffd640 1 -- 172.21.15.40:0/1022296325 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd404005cd0 con 0x7fd40c13b0e0 2024-04-22T08:13:57.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.102+0000 7fd402ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:57.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.103+0000 7fd402ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:13:57.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.103+0000 7fd402ffd640 10 monclient: dump: 2024-04-22T08:13:57.115 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:57.116 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:57.116 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:57.116 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:57.116 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:57.116 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:57.116 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:57.116 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:57.116 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:57.116 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:57.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.102+0000 7fd408d99640 4 set_mon_vals no callback set 2024-04-22T08:13:57.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.103+0000 7fd410b0a640 10 monclient: _renew_subs 2024-04-22T08:13:57.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.103+0000 7fd410b0a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:13:57.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.103+0000 7fd410b0a640 1 -- 172.21.15.40:0/1022296325 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd40c123470 con 0x7fd40c13b0e0 2024-04-22T08:13:57.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.103+0000 7fd410b0a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:13:57.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.103+0000 7fd410b0a640 1 --2- 172.21.15.40:0/1022296325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd40c13eae0 0x7fd40c1cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:57.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.103+0000 7fd402ffd640 1 -- 172.21.15.40:0/1022296325 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fd40401b3b0 con 0x7fd40c13b0e0 2024-04-22T08:13:57.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.103+0000 7fd410b0a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:57.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.103+0000 7fd410b0a640 1 -- 172.21.15.40:0/1022296325 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd40c066770 con 0x7fd40c13eae0 2024-04-22T08:13:57.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.103+0000 7fd40a59c640 1 --2- 172.21.15.40:0/1022296325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd40c13eae0 0x7fd40c1cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:57.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.103+0000 7fd40a59c640 10 monclient: get_auth_request con 0x7fd40c13eae0 auth_method 0 2024-04-22T08:13:57.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.103+0000 7fd40a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:57.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.103+0000 7fd40a59c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:57.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.103+0000 7fd40a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:57.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.104+0000 7fd40a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:57.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.104+0000 7fd40a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:57.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.104+0000 7fd40a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:57.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.104+0000 7fd402ffd640 1 --2- 172.21.15.40:0/1022296325 >> v2:172.21.15.189:6800/30168 conn(0x7fd3ec0377d0 0x7fd3ec039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:57.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.104+0000 7fd402ffd640 1 -- 172.21.15.40:0/1022296325 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 3898+0+0 (secure 0 0 0) 0x7fd404049940 con 0x7fd40c13b0e0 2024-04-22T08:13:57.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.104+0000 7fd40a59c640 10 monclient(hunting): handle_auth_done global_id 4493 payload 824 2024-04-22T08:13:57.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.104+0000 7fd40a59c640 1 -- 172.21.15.40:0/1022296325 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd40c09dc10 con 0x7fd40c13eae0 2024-04-22T08:13:57.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.104+0000 7fd409d9b640 1 --2- 172.21.15.40:0/1022296325 >> v2:172.21.15.189:6800/30168 conn(0x7fd3ec0377d0 0x7fd3ec039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:57.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.104+0000 7fd409d9b640 10 monclient: get_auth_request con 0x7fd3ec0377d0 auth_method 0 2024-04-22T08:13:57.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.106+0000 7fd40a59c640 1 -- 172.21.15.40:0/1022296325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd40c13eae0 msgr2=0x7fd40c1cb6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:13:57.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.106+0000 7fd40a59c640 1 --2- 172.21.15.40:0/1022296325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd40c13eae0 0x7fd40c1cb6f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc00fdc0 tx=0x7fd3fc00d520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:57.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.106+0000 7fd402ffd640 1 -- 172.21.15.40:0/1022296325 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3fc00de10 con 0x7fd40c13eae0 2024-04-22T08:13:57.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.106+0000 7fd402ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:57.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.106+0000 7fd402ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:13:57.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.106+0000 7fd402ffd640 10 monclient: dump: 2024-04-22T08:13:57.118 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:57.118 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:57.118 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:57.118 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:57.118 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:57.119 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:57.119 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:57.119 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:57.119 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:57.119 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:57.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.106+0000 7fd402ffd640 1 -- 172.21.15.40:0/1022296325 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7fd3fc005d50 con 0x7fd40c13eae0 2024-04-22T08:13:57.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.106+0000 7fd402ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:13:57.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.106+0000 7fd402ffd640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:13:57.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.106+0000 7fd402ffd640 1 -- 172.21.15.40:0/1022296325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd40c13eae0 msgr2=0x7fd40c1cb6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:57.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.106+0000 7fd402ffd640 1 --2- 172.21.15.40:0/1022296325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd40c13eae0 0x7fd40c1cb6f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc00fdc0 tx=0x7fd3fc00d520 comp rx=0 tx=0).stop 2024-04-22T08:13:57.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.106+0000 7fd402ffd640 1 -- 172.21.15.40:0/1022296325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd40c13eae0 msgr2=0x7fd40c1cb6f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:13:57.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.106+0000 7fd402ffd640 1 --2- 172.21.15.40:0/1022296325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd40c13eae0 0x7fd40c1cb6f0 secure :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc00fdc0 tx=0x7fd3fc00d520 comp rx=0 tx=0).stop 2024-04-22T08:13:57.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.106+0000 7fd409d9b640 1 --2- 172.21.15.40:0/1022296325 >> v2:172.21.15.189:6800/30168 conn(0x7fd3ec0377d0 0x7fd3ec039c90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd40c0a0730 tx=0x7fd3f4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:57.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.112+0000 7fd410b0a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:13:57.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.113+0000 7fd410b0a640 1 --2- 172.21.15.40:0/1022296325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd40c120ad0 0x7fd40c120ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:57.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.113+0000 7fd410b0a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:57.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.113+0000 7fd410b0a640 1 -- 172.21.15.40:0/1022296325 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd40c04e9b0 con 0x7fd40c120ad0 2024-04-22T08:13:57.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.113+0000 7fd40a59c640 1 --2- 172.21.15.40:0/1022296325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd40c120ad0 0x7fd40c120ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:57.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.113+0000 7fd40a59c640 10 monclient: get_auth_request con 0x7fd40c120ad0 auth_method 0 2024-04-22T08:13:57.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.113+0000 7fd40a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:57.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.113+0000 7fd40a59c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:57.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.113+0000 7fd40a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:57.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.113+0000 7fd40a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:57.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.113+0000 7fd40a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:57.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.113+0000 7fd40a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:57.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.113+0000 7fd40a59c640 10 monclient(hunting): handle_auth_done global_id 4496 payload 824 2024-04-22T08:13:57.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.113+0000 7fd40a59c640 1 -- 172.21.15.40:0/1022296325 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd40c071310 con 0x7fd40c120ad0 2024-04-22T08:13:57.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.115+0000 7fd40a59c640 1 -- 172.21.15.40:0/1022296325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd40c120ad0 msgr2=0x7fd40c120ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:13:57.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.115+0000 7fd40a59c640 1 --2- 172.21.15.40:0/1022296325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd40c120ad0 0x7fd40c120ee0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc00fdc0 tx=0x7fd3fc001830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:57.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.115+0000 7fd402ffd640 1 -- 172.21.15.40:0/1022296325 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3fc024070 con 0x7fd40c120ad0 2024-04-22T08:13:57.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.115+0000 7fd402ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:57.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.115+0000 7fd402ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:13:57.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.115+0000 7fd402ffd640 10 monclient: dump: 2024-04-22T08:13:57.122 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:57.122 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:57.122 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:57.122 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:57.122 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:57.122 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:57.122 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:57.122 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:57.122 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:57.122 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:57.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.116+0000 7fd402ffd640 1 -- 172.21.15.40:0/1022296325 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4631 (secure 0 0 0) 0x7fd3fc01d970 con 0x7fd40c120ad0 2024-04-22T08:13:57.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.116+0000 7fd402ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:13:57.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.116+0000 7fd402ffd640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:13:57.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.116+0000 7fd402ffd640 1 -- 172.21.15.40:0/1022296325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd40c120ad0 msgr2=0x7fd40c120ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:57.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.116+0000 7fd402ffd640 1 --2- 172.21.15.40:0/1022296325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd40c120ad0 0x7fd40c120ee0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc00fdc0 tx=0x7fd3fc001830 comp rx=0 tx=0).stop 2024-04-22T08:13:57.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.116+0000 7fd402ffd640 1 -- 172.21.15.40:0/1022296325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd40c120ad0 msgr2=0x7fd40c120ee0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:13:57.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.116+0000 7fd402ffd640 1 --2- 172.21.15.40:0/1022296325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd40c120ad0 0x7fd40c120ee0 secure :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc00fdc0 tx=0x7fd3fc001830 comp rx=0 tx=0).stop 2024-04-22T08:13:57.123 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:13:57.123 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:13:57.123 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:13:57.123 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:13:57.124 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 20, 2024-04-22T08:13:57.124 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:13:57.124 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:13:57.124 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:13:57.124 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:13:57.124 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:13:57.124 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 0, 2024-04-22T08:13:57.124 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:13:57.124 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:13:57.124 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:13:57.124 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:13:57.124 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:13:57.124 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:13:57.124 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:13:57.125 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:13:57.125 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:13:57.125 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:13:57.125 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:13:57.125 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:13:57.125 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:13:57.125 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:13:57.125 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:13:57.125 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:13:57.125 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:13:57.125 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:13:57.125 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:13:57.125 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:13:57.125 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:13:57.126 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:13:57.126 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:13:57.126 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:13:57.126 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:13:57.126 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:13:57.126 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:57.126 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:57.126 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:13:57.126 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:13:57.126 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:13:57.126 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:13:57.126 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:13:57.126 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:13:57.127 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:13:57.127 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:13:57.127 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:13:57.127 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:13:57.127 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:13:57.127 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:13:57.127 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:13:57.127 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:13:57.127 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:13:57.127 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:13:57.127 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:13:57.127 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:13:57.127 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:13:57.127 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:13:57.127 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:13:57.128 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:13:57.128 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:13:57.128 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:13:57.128 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:13:57.128 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:13:57.128 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:13:57.128 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:13:57.128 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:13:57.128 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:57.128 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:13:57.128 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:57.128 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:13:57.128 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:13:57.128 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:13:57.129 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:13:57.129 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:57.129 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:13:57.129 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:13:57.129 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:57.129 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:57.129 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:57.129 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:13:57.129 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:13:57.129 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:57.129 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:57.129 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:57.129 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:57.129 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:13:57.129 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:13:57.130 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:13:57.130 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:13:57.130 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:13:57.130 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:57.130 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:57.130 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:13:57.130 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:13:57.130 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:13:57.130 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:13:57.130 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:57.130 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:13:57.130 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:13:57.130 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:57.130 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:57.131 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:57.131 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:13:57.131 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:13:57.131 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:57.131 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:57.131 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:57.131 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:57.131 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:13:57.131 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:13:57.131 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:13:57.131 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:13:57.131 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:13:57.131 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:57.131 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:57.132 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:13:57.132 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:13:57.132 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:13:57.132 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:13:57.132 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:57.132 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:13:57.132 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:13:57.132 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:57.132 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:57.132 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:57.132 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:13:57.132 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:13:57.132 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:57.132 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:57.133 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:57.133 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:57.133 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:13:57.133 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:13:57.133 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:13:57.133 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:13:57.133 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:13:57.133 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:57.133 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:57.133 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:57.133 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:13:57.133 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:13:57.133 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:57.133 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:13:57.133 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:13:57.134 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:13:57.134 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:57.134 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:13:57.134 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:13:57.134 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:57.134 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:13:57.134 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:13:57.134 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 5 2024-04-22T08:13:57.134 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:57.134 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:13:57.134 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:13:57.134 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:57.134 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:13:57.134 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:13:57.134 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 3 2024-04-22T08:13:57.135 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:57.135 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:57.135 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:57.135 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:13:57.135 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:13:57.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.116+0000 7fd410b0a640 1 -- 172.21.15.40:0/1022296325 >> v2:172.21.15.189:6800/30168 conn(0x7fd3ec0377d0 msgr2=0x7fd3ec039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:57.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.116+0000 7fd410b0a640 1 --2- 172.21.15.40:0/1022296325 >> v2:172.21.15.189:6800/30168 conn(0x7fd3ec0377d0 0x7fd3ec039c90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd40c0a0730 tx=0x7fd3f4008030 comp rx=0 tx=0).stop 2024-04-22T08:13:57.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.116+0000 7fd410b0a640 10 monclient: shutdown 2024-04-22T08:13:57.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.116+0000 7fd410b0a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:13:57.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.116+0000 7fd410b0a640 1 -- 172.21.15.40:0/1022296325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd40c13b0e0 msgr2=0x7fd40c1cb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:57.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.116+0000 7fd40ad9d640 1 -- 172.21.15.40:0/1022296325 reap_dead start 2024-04-22T08:13:57.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.116+0000 7fd410b0a640 1 --2- 172.21.15.40:0/1022296325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd40c13b0e0 0x7fd40c1cb1b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd40400c3d0 tx=0x7fd40400c890 comp rx=0 tx=0).stop 2024-04-22T08:13:57.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.116+0000 7fd410b0a640 1 -- 172.21.15.40:0/1022296325 shutdown_connections 2024-04-22T08:13:57.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.116+0000 7fd410b0a640 1 --2- 172.21.15.40:0/1022296325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd40c13b0e0 0x7fd40c1cb1b0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:57.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.116+0000 7fd410b0a640 1 -- 172.21.15.40:0/1022296325 >> 172.21.15.40:0/1022296325 conn(0x7fd40c0a66a0 msgr2=0x7fd40c09b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:13:57.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.116+0000 7fd410b0a640 1 -- 172.21.15.40:0/1022296325 shutdown_connections 2024-04-22T08:13:57.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.116+0000 7fd410b0a640 1 -- 172.21.15.40:0/1022296325 wait complete. 2024-04-22T08:13:57.137 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:13:57.138 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-22T08:13:57.138 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-22T08:13:57.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.237+0000 7ff06f50c640 10 monclient: get_monmap_and_config 2024-04-22T08:13:57.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.237+0000 7ff06f50c640 10 monclient: build_initial_monmap 2024-04-22T08:13:57.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.237+0000 7ff06f50c640 10 monclient: monmap: 2024-04-22T08:13:57.237 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:13:57.237 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:57.238 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:13:57.238702+0000 2024-04-22T08:13:57.238 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:13:57.238702+0000 2024-04-22T08:13:57.238 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:13:57.238 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:57.238 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:13:57.238 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:13:57.238 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:13:57.238 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:13:57.238 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:13:57.238 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:57.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.238+0000 7ff06f50c640 1 Processor -- start 2024-04-22T08:13:57.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.238+0000 7ff06f50c640 1 -- start start 2024-04-22T08:13:57.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.238+0000 7ff06f50c640 10 monclient: init 2024-04-22T08:13:57.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.239+0000 7ff06f50c640 10 monclient: _reopen_session rank -1 2024-04-22T08:13:57.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.239+0000 7ff06f50c640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-04-22T08:13:57.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.239+0000 7ff06f50c640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff0680a1a80 0x7ff0680a01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:57.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.239+0000 7ff06f50c640 10 monclient(hunting): picked mon.noname-c con 0x7ff0680a1a80 addr v2:172.21.15.189:3300/0 2024-04-22T08:13:57.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.239+0000 7ff06f50c640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff0680a1e60 0x7ff0680a2270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:57.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.239+0000 7ff06f50c640 10 monclient(hunting): picked mon.noname-a con 0x7ff0680a1e60 addr v2:172.21.15.40:3300/0 2024-04-22T08:13:57.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.239+0000 7ff06f50c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff068135c20 addr v1:172.21.15.189:6789/0 2024-04-22T08:13:57.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.239+0000 7ff06f50c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0680721e0 con 0x7ff068135c20 2024-04-22T08:13:57.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.239+0000 7ff06f50c640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:57.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.239+0000 7ff06f50c640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff06809f230 con 0x7ff0680a1e60 2024-04-22T08:13:57.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.239+0000 7ff06f50c640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:57.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.239+0000 7ff06f50c640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff06809f7f0 con 0x7ff0680a1a80 2024-04-22T08:13:57.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.239+0000 7ff06f50c640 10 monclient(hunting): _renew_subs 2024-04-22T08:13:57.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.239+0000 7ff06f50c640 10 monclient(hunting): authenticate will time out at 911.232948s 2024-04-22T08:13:57.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.239+0000 7ff06caa7640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff0680a1e60 0x7ff0680a2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:57.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.239+0000 7ff06caa7640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff0680a1e60 0x7ff0680a2270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37070/0 (socket says 172.21.15.40:37070) 2024-04-22T08:13:57.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.239+0000 7ff06caa7640 1 -- 172.21.15.40:0/1998144691 learned_addr learned my addr 172.21.15.40:0/1998144691 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:13:57.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.239+0000 7ff06caa7640 10 monclient(hunting): get_auth_request con 0x7ff0680a1e60 auth_method 0 2024-04-22T08:13:57.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.239+0000 7ff06caa7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.239+0000 7ff06caa7640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.239+0000 7ff06caa7640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.239+0000 7ff06caa7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.239+0000 7ff06caa7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.239+0000 7ff06caa7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.239+0000 7ff06caa7640 10 monclient(hunting): handle_auth_done global_id 4533 payload 274 2024-04-22T08:13:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.239+0000 7ff06caa7640 1 -- 172.21.15.40:0/1998144691 >> v2:172.21.15.189:3300/0 conn(0x7ff0680a1a80 msgr2=0x7ff0680a01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:13:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.239+0000 7ff06caa7640 1 --2- 172.21.15.40:0/1998144691 >> v2:172.21.15.189:3300/0 conn(0x7ff0680a1a80 0x7ff0680a01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.239+0000 7ff06caa7640 1 -- 172.21.15.40:0/1998144691 >> v1:172.21.15.189:6789/0 conn(0x7ff068135c20 legacy=0x7ff068136070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:13:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.239+0000 7ff06caa7640 10 monclient: _finish_hunting 0 2024-04-22T08:13:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.239+0000 7ff06caa7640 1 monclient: found mon.noname-a 2024-04-22T08:13:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.239+0000 7ff06caa7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:13:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.239+0000 7ff06caa7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:13:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.239+0000 7ff06caa7640 1 -- 172.21.15.40:0/1998144691 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff06809d860 con 0x7ff0680a1e60 2024-04-22T08:13:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.242+0000 7ff06caa7640 1 -- 172.21.15.40:0/1998144691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0680a1e60 msgr2=0x7ff0680a2270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:13:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.242+0000 7ff06caa7640 1 --2- 172.21.15.40:0/1998144691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0680a1e60 0x7ff0680a2270 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff0680a3c30 tx=0x7ff05c02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=869b1cf8c8abdef2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.242+0000 7ff067fff640 1 -- 172.21.15.40:0/1998144691 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff05c031a40 con 0x7ff0680a1e60 2024-04-22T08:13:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.242+0000 7ff067fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.242+0000 7ff067fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:13:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.242+0000 7ff067fff640 10 monclient: dump: 2024-04-22T08:13:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:57.250 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:57.250 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:57.250 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:57.250 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:57.250 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:57.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.242+0000 7ff067fff640 10 monclient: _finish_auth 0 2024-04-22T08:13:57.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.242+0000 7ff067fff640 10 monclient: _check_auth_tickets 2024-04-22T08:13:57.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.242+0000 7ff067fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:13:57.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.242+0000 7ff067fff640 1 -- 172.21.15.40:0/1998144691 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff05c031ba0 con 0x7ff0680a1e60 2024-04-22T08:13:57.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.242+0000 7ff067fff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:13:57.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.242+0000 7ff067fff640 1 -- 172.21.15.40:0/1998144691 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff05c033440 con 0x7ff0680a1e60 2024-04-22T08:13:57.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.242+0000 7ff06f50c640 5 monclient: authenticate success, global_id 4533 2024-04-22T08:13:57.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.242+0000 7ff06f50c640 10 monclient: get_monmap_and_config success 2024-04-22T08:13:57.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.242+0000 7ff06f50c640 4 set_mon_vals no callback set 2024-04-22T08:13:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.242+0000 7ff067fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.242+0000 7ff067fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:13:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.242+0000 7ff067fff640 10 monclient: dump: 2024-04-22T08:13:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:57.251 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.242+0000 7ff06f50c640 10 monclient: shutdown 2024-04-22T08:13:57.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.242+0000 7ff06f50c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:13:57.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.242+0000 7ff06f50c640 1 -- 172.21.15.40:0/1998144691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0680a1e60 msgr2=0x7ff0680a2270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:57.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.242+0000 7ff06f50c640 1 --2- 172.21.15.40:0/1998144691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0680a1e60 0x7ff0680a2270 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff0680a3c30 tx=0x7ff05c02b7f0 comp rx=0 tx=0).stop 2024-04-22T08:13:57.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.243+0000 7ff06f50c640 1 -- 172.21.15.40:0/1998144691 shutdown_connections 2024-04-22T08:13:57.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.243+0000 7ff06f50c640 1 --2- 172.21.15.40:0/1998144691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0680a1e60 0x7ff0680a2270 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:57.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.243+0000 7ff06f50c640 1 --2- 172.21.15.40:0/1998144691 >> v2:172.21.15.189:3300/0 conn(0x7ff0680a1a80 0x7ff0680a01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:57.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.243+0000 7ff06f50c640 1 -- 172.21.15.40:0/1998144691 >> 172.21.15.40:0/1998144691 conn(0x7ff0680a66a0 msgr2=0x7ff0680a2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:13:57.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.243+0000 7ff06f50c640 1 -- 172.21.15.40:0/1998144691 shutdown_connections 2024-04-22T08:13:57.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.243+0000 7ff06f50c640 1 -- 172.21.15.40:0/1998144691 wait complete. 2024-04-22T08:13:57.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.243+0000 7ff06f50c640 10 monclient: build_initial_monmap 2024-04-22T08:13:57.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.243+0000 7ff06f50c640 10 monclient: monmap: 2024-04-22T08:13:57.252 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:13:57.252 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:13:57.252 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:13:57.252 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:13:57.253 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:13:57.253 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:57.253 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:13:57.253 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:13:57.253 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:13:57.253 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:57.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.243+0000 7ff06f50c640 1 Processor -- start 2024-04-22T08:13:57.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06f50c640 1 -- start start 2024-04-22T08:13:57.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06f50c640 10 monclient: init 2024-04-22T08:13:57.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06f50c640 10 monclient: _reopen_session rank -1 2024-04-22T08:13:57.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06f50c640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-22T08:13:57.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06f50c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0680a1a80 0x7ff0681ca660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:57.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06f50c640 10 monclient(hunting): picked mon.noname-b con 0x7ff0680a1a80 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:13:57.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06f50c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff0680a1e60 0x7ff068134d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:57.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06f50c640 10 monclient(hunting): picked mon.noname-c con 0x7ff0680a1e60 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:13:57.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06f50c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff068135c20 0x7ff06806f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:57.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06f50c640 10 monclient(hunting): picked mon.noname-a con 0x7ff068135c20 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:13:57.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06f50c640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:57.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06f50c640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff068071490 con 0x7ff068135c20 2024-04-22T08:13:57.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06f50c640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:57.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06f50c640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0681bdd90 con 0x7ff0680a1a80 2024-04-22T08:13:57.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06f50c640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:57.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06f50c640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff06809cce0 con 0x7ff0680a1e60 2024-04-22T08:13:57.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06f50c640 10 monclient(hunting): _renew_subs 2024-04-22T08:13:57.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06f50c640 10 monclient(hunting): authenticate will time out at 911.238279s 2024-04-22T08:13:57.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06daa9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff068135c20 0x7ff06806f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:57.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06daa9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff068135c20 0x7ff06806f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37082/0 (socket says 172.21.15.40:37082) 2024-04-22T08:13:57.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06caa7640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff0680a1e60 0x7ff068134d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:57.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06daa9640 1 -- 172.21.15.40:0/3253697976 learned_addr learned my addr 172.21.15.40:0/3253697976 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:13:57.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06daa9640 10 monclient(hunting): get_auth_request con 0x7ff068135c20 auth_method 0 2024-04-22T08:13:57.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06daa9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:57.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06daa9640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:57.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06daa9640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:57.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06caa7640 10 monclient(hunting): get_auth_request con 0x7ff0680a1e60 auth_method 0 2024-04-22T08:13:57.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06caa7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:57.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06caa7640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:57.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06caa7640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:57.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06daa9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:57.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06daa9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:57.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06daa9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:57.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06d2a8640 1 --2- 172.21.15.40:0/3253697976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0680a1a80 0x7ff0681ca660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:57.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06caa7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:57.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06caa7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:57.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06caa7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:57.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06d2a8640 10 monclient(hunting): get_auth_request con 0x7ff0680a1a80 auth_method 0 2024-04-22T08:13:57.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06d2a8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:57.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06d2a8640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:57.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06d2a8640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:57.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06daa9640 10 monclient(hunting): handle_auth_done global_id 4536 payload 824 2024-04-22T08:13:57.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06daa9640 1 -- 172.21.15.40:0/3253697976 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff0680a1e60 msgr2=0x7ff068134d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:57.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06daa9640 1 --2- 172.21.15.40:0/3253697976 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff0680a1e60 0x7ff068134d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:57.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06daa9640 1 -- 172.21.15.40:0/3253697976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0680a1a80 msgr2=0x7ff0681ca660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:57.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06daa9640 1 --2- 172.21.15.40:0/3253697976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0680a1a80 0x7ff0681ca660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:57.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06daa9640 10 monclient: _finish_hunting 0 2024-04-22T08:13:57.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06daa9640 1 monclient: found mon.noname-a 2024-04-22T08:13:57.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06daa9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:13:57.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06daa9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:13:57.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06daa9640 1 -- 172.21.15.40:0/3253697976 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff05c0094d0 con 0x7ff068135c20 2024-04-22T08:13:57.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.244+0000 7ff06d2a8640 1 --2- 172.21.15.40:0/3253697976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0680a1a80 0x7ff0681ca660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T08:13:57.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.247+0000 7ff06daa9640 1 -- 172.21.15.40:0/3253697976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff068135c20 msgr2=0x7ff06806f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:13:57.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.247+0000 7ff06daa9640 1 --2- 172.21.15.40:0/3253697976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff068135c20 0x7ff06806f930 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff05800c780 tx=0x7ff05800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:57.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.247+0000 7ff065ffb640 1 -- 172.21.15.40:0/3253697976 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff05800ce90 con 0x7ff068135c20 2024-04-22T08:13:57.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.247+0000 7ff065ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:57.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.247+0000 7ff065ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:13:57.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.247+0000 7ff065ffb640 10 monclient: dump: 2024-04-22T08:13:57.258 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:57.258 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:57.258 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:57.258 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:57.258 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:57.258 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:57.258 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:57.258 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:57.258 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:57.258 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:57.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.247+0000 7ff065ffb640 10 monclient: _finish_auth 0 2024-04-22T08:13:57.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.247+0000 7ff065ffb640 10 monclient: _check_auth_tickets 2024-04-22T08:13:57.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.247+0000 7ff065ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:13:57.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.247+0000 7ff065ffb640 1 -- 172.21.15.40:0/3253697976 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff058005070 con 0x7ff068135c20 2024-04-22T08:13:57.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.247+0000 7ff06f50c640 5 monclient: authenticate success, global_id 4536 2024-04-22T08:13:57.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.247+0000 7ff06f50c640 10 monclient: _renew_subs 2024-04-22T08:13:57.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.247+0000 7ff06f50c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:13:57.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.247+0000 7ff06f50c640 1 -- 172.21.15.40:0/3253697976 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff068121e90 con 0x7ff068135c20 2024-04-22T08:13:57.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.247+0000 7ff065ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:13:57.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.247+0000 7ff065ffb640 1 -- 172.21.15.40:0/3253697976 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff058005400 con 0x7ff068135c20 2024-04-22T08:13:57.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.247+0000 7ff065ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:57.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.247+0000 7ff065ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:13:57.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.247+0000 7ff065ffb640 10 monclient: dump: 2024-04-22T08:13:57.260 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:57.260 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:57.260 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:57.260 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:57.260 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:57.260 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:57.260 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:57.260 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:57.260 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:57.260 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:57.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.247+0000 7ff06f50c640 10 monclient: _renew_subs 2024-04-22T08:13:57.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.247+0000 7ff06f50c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:13:57.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.247+0000 7ff06f50c640 1 -- 172.21.15.40:0/3253697976 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0681356e0 con 0x7ff068135c20 2024-04-22T08:13:57.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.247+0000 7ff066ffd640 4 set_mon_vals no callback set 2024-04-22T08:13:57.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.248+0000 7ff065ffb640 1 -- 172.21.15.40:0/3253697976 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7ff058005660 con 0x7ff068135c20 2024-04-22T08:13:57.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.248+0000 7ff06f50c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:13:57.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.248+0000 7ff06f50c640 1 --2- 172.21.15.40:0/3253697976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0681352b0 0x7ff06806fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:57.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.248+0000 7ff06f50c640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:57.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.248+0000 7ff06f50c640 1 -- 172.21.15.40:0/3253697976 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff068066770 con 0x7ff0681352b0 2024-04-22T08:13:57.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.248+0000 7ff065ffb640 1 --2- 172.21.15.40:0/3253697976 >> v2:172.21.15.189:6800/30168 conn(0x7ff054037820 0x7ff054039ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:57.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.248+0000 7ff065ffb640 1 -- 172.21.15.40:0/3253697976 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 3898+0+0 (secure 0 0 0) 0x7ff058049c60 con 0x7ff068135c20 2024-04-22T08:13:57.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.248+0000 7ff06d2a8640 1 --2- 172.21.15.40:0/3253697976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0681352b0 0x7ff06806fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:57.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.248+0000 7ff06d2a8640 10 monclient: get_auth_request con 0x7ff0681352b0 auth_method 0 2024-04-22T08:13:57.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.248+0000 7ff06d2a8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:57.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.248+0000 7ff06d2a8640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:57.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.248+0000 7ff06d2a8640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:57.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.248+0000 7ff06d2a8640 1 -- 172.21.15.40:0/3253697976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0681352b0 msgr2=0x7ff06806fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-22T08:13:57.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.248+0000 7ff06d2a8640 1 -- 172.21.15.40:0/3253697976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0681352b0 msgr2=0x7ff06806fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:13:57.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.248+0000 7ff06d2a8640 1 --2- 172.21.15.40:0/3253697976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0681352b0 0x7ff06806fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:13:57.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.248+0000 7ff06d2a8640 1 --2- 172.21.15.40:0/3253697976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0681352b0 0x7ff06806fe70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:13:57.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.248+0000 7ff06caa7640 1 --2- 172.21.15.40:0/3253697976 >> v2:172.21.15.189:6800/30168 conn(0x7ff054037820 0x7ff054039ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:57.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.249+0000 7ff06caa7640 10 monclient: get_auth_request con 0x7ff054037820 auth_method 0 2024-04-22T08:13:57.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.251+0000 7ff06caa7640 1 --2- 172.21.15.40:0/3253697976 >> v2:172.21.15.189:6800/30168 conn(0x7ff054037820 0x7ff054039ce0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff06809fc70 tx=0x7ff05c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:57.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.449+0000 7ff06d2a8640 1 --2- 172.21.15.40:0/3253697976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0681352b0 0x7ff06806fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:57.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.449+0000 7ff06d2a8640 10 monclient: get_auth_request con 0x7ff0681352b0 auth_method 0 2024-04-22T08:13:57.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.449+0000 7ff06d2a8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:57.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.449+0000 7ff06d2a8640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:57.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.449+0000 7ff06d2a8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:13:57.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.449+0000 7ff06d2a8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:57.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.449+0000 7ff06d2a8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:57.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.449+0000 7ff06d2a8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:57.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.449+0000 7ff06d2a8640 10 monclient(hunting): handle_auth_done global_id 24112 payload 824 2024-04-22T08:13:57.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.449+0000 7ff06d2a8640 1 -- 172.21.15.40:0/3253697976 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff06809dbf0 con 0x7ff0681352b0 2024-04-22T08:13:57.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.452+0000 7ff06d2a8640 1 -- 172.21.15.40:0/3253697976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0681352b0 msgr2=0x7ff06806fe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:13:57.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.452+0000 7ff06d2a8640 1 --2- 172.21.15.40:0/3253697976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0681352b0 0x7ff06806fe70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff06000dfd0 tx=0x7ff06000f8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:57.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.452+0000 7ff065ffb640 1 -- 172.21.15.40:0/3253697976 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff060014880 con 0x7ff0681352b0 2024-04-22T08:13:57.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.452+0000 7ff065ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:57.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.452+0000 7ff065ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:13:57.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.452+0000 7ff065ffb640 10 monclient: dump: 2024-04-22T08:13:57.452 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:57.453 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:57.453 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:57.453 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:57.453 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:57.453 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:57.453 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:57.453 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:57.453 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:57.453 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:57.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.453+0000 7ff065ffb640 1 -- 172.21.15.40:0/3253697976 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7ff0600149e0 con 0x7ff0681352b0 2024-04-22T08:13:57.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.453+0000 7ff065ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:13:57.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.453+0000 7ff065ffb640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:13:57.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.453+0000 7ff065ffb640 1 -- 172.21.15.40:0/3253697976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0681352b0 msgr2=0x7ff06806fe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:57.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.453+0000 7ff065ffb640 1 --2- 172.21.15.40:0/3253697976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0681352b0 0x7ff06806fe70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff06000dfd0 tx=0x7ff06000f8a0 comp rx=0 tx=0).stop 2024-04-22T08:13:57.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.453+0000 7ff065ffb640 1 -- 172.21.15.40:0/3253697976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0681352b0 msgr2=0x7ff06806fe70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:13:57.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.453+0000 7ff065ffb640 1 --2- 172.21.15.40:0/3253697976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0681352b0 0x7ff06806fe70 secure :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff06000dfd0 tx=0x7ff06000f8a0 comp rx=0 tx=0).stop 2024-04-22T08:13:57.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.459+0000 7ff06f50c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:13:57.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.459+0000 7ff06f50c640 1 --2- 172.21.15.40:0/3253697976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0680703b0 0x7ff06806cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:57.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.459+0000 7ff06f50c640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:57.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.459+0000 7ff06f50c640 1 -- 172.21.15.40:0/3253697976 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0680a34f0 con 0x7ff0680703b0 2024-04-22T08:13:57.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.459+0000 7ff06d2a8640 1 --2- 172.21.15.40:0/3253697976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0680703b0 0x7ff06806cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:57.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.459+0000 7ff06d2a8640 10 monclient: get_auth_request con 0x7ff0680703b0 auth_method 0 2024-04-22T08:13:57.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.459+0000 7ff06d2a8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:57.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.459+0000 7ff06d2a8640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:57.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.459+0000 7ff06d2a8640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:57.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.459+0000 7ff06d2a8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:57.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.459+0000 7ff06d2a8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:57.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.459+0000 7ff06d2a8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:57.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.460+0000 7ff06d2a8640 10 monclient(hunting): handle_auth_done global_id 24115 payload 824 2024-04-22T08:13:57.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.460+0000 7ff06d2a8640 1 -- 172.21.15.40:0/3253697976 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff0680707c0 con 0x7ff0680703b0 2024-04-22T08:13:57.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.462+0000 7ff06d2a8640 1 -- 172.21.15.40:0/3253697976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0680703b0 msgr2=0x7ff06806cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:13:57.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.462+0000 7ff06d2a8640 1 --2- 172.21.15.40:0/3253697976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0680703b0 0x7ff06806cb70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff060019000 tx=0x7ff0600046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:57.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.463+0000 7ff065ffb640 1 -- 172.21.15.40:0/3253697976 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff060024070 con 0x7ff0680703b0 2024-04-22T08:13:57.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.463+0000 7ff065ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:57.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.463+0000 7ff065ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:13:57.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.463+0000 7ff065ffb640 10 monclient: dump: 2024-04-22T08:13:57.463 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:57.463 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:57.463 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:57.463 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:57.463 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:57.463 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:57.463 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:57.463 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:57.463 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:57.463 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:57.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.463+0000 7ff065ffb640 1 -- 172.21.15.40:0/3253697976 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4461 (secure 0 0 0) 0x7ff060015410 con 0x7ff0680703b0 2024-04-22T08:13:57.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.463+0000 7ff065ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:13:57.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.463+0000 7ff065ffb640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:13:57.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.463+0000 7ff065ffb640 1 -- 172.21.15.40:0/3253697976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0680703b0 msgr2=0x7ff06806cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:57.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.463+0000 7ff065ffb640 1 --2- 172.21.15.40:0/3253697976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0680703b0 0x7ff06806cb70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff060019000 tx=0x7ff0600046c0 comp rx=0 tx=0).stop 2024-04-22T08:13:57.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.463+0000 7ff065ffb640 1 -- 172.21.15.40:0/3253697976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0680703b0 msgr2=0x7ff06806cb70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:13:57.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.463+0000 7ff065ffb640 1 --2- 172.21.15.40:0/3253697976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0680703b0 0x7ff06806cb70 secure :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff060019000 tx=0x7ff0600046c0 comp rx=0 tx=0).stop 2024-04-22T08:13:57.464 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:13:57.464 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:13:57.465 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:13:57.465 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:13:57.465 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 20, 2024-04-22T08:13:57.465 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:13:57.465 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:13:57.465 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:13:57.465 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:13:57.465 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:13:57.465 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 1, 2024-04-22T08:13:57.465 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:13:57.465 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:13:57.465 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:13:57.465 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:13:57.466 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:13:57.466 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:13:57.466 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:13:57.466 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:13:57.466 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:13:57.466 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:13:57.466 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:13:57.466 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:13:57.466 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:13:57.466 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:13:57.466 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:13:57.466 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:13:57.466 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:13:57.466 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:13:57.467 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:13:57.467 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:13:57.467 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:13:57.467 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:13:57.467 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:13:57.467 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:13:57.467 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:13:57.467 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:13:57.467 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:57.467 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:57.467 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:13:57.467 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:13:57.467 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:13:57.467 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:13:57.467 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:13:57.468 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:13:57.468 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:13:57.468 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:13:57.468 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:13:57.468 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:13:57.468 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:13:57.468 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:13:57.468 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:13:57.468 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:13:57.468 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:13:57.468 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:13:57.468 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:13:57.468 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:13:57.468 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:13:57.469 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:13:57.469 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:13:57.469 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:13:57.469 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:13:57.469 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:13:57.469 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:13:57.469 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:13:57.469 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:13:57.469 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:13:57.469 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:13:57.469 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:57.469 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:13:57.469 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:57.469 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:13:57.469 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:13:57.470 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:13:57.470 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:13:57.470 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:57.470 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:13:57.470 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:13:57.470 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:57.470 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:57.470 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:57.470 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:13:57.470 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:13:57.470 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:57.470 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:57.470 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:57.470 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:57.470 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:13:57.471 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:13:57.471 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:13:57.471 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:13:57.471 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:13:57.471 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:57.471 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:57.471 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:13:57.471 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:13:57.471 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:13:57.471 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:13:57.471 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:57.471 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:13:57.471 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:13:57.471 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:57.471 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:57.472 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:57.472 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:13:57.472 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:13:57.472 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:57.472 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:57.472 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:57.472 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:57.472 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:13:57.472 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:13:57.472 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:13:57.472 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:13:57.472 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:13:57.472 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:57.472 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:57.472 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:13:57.473 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:13:57.473 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:13:57.473 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:13:57.473 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:57.473 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:13:57.473 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:13:57.473 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:57.473 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:57.473 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:57.473 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:13:57.473 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:13:57.473 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:57.473 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:57.474 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:57.474 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:57.474 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:13:57.474 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:13:57.474 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:13:57.474 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:13:57.474 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:13:57.474 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:57.474 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:57.474 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:57.474 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:13:57.474 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:13:57.474 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:57.474 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:13:57.474 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:13:57.475 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:13:57.475 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:57.475 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:13:57.475 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:13:57.475 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:57.475 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:13:57.475 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:13:57.475 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 2 2024-04-22T08:13:57.475 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:57.475 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:57.475 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:57.475 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:13:57.475 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:13:57.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.463+0000 7ff06f50c640 1 -- 172.21.15.40:0/3253697976 >> v2:172.21.15.189:6800/30168 conn(0x7ff054037820 msgr2=0x7ff054039ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:57.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.463+0000 7ff06f50c640 1 --2- 172.21.15.40:0/3253697976 >> v2:172.21.15.189:6800/30168 conn(0x7ff054037820 0x7ff054039ce0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff06809fc70 tx=0x7ff05c008030 comp rx=0 tx=0).stop 2024-04-22T08:13:57.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.463+0000 7ff06f50c640 10 monclient: shutdown 2024-04-22T08:13:57.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.463+0000 7ff06f50c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:13:57.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.463+0000 7ff06f50c640 1 -- 172.21.15.40:0/3253697976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff068135c20 msgr2=0x7ff06806f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:57.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.463+0000 7ff06f50c640 1 --2- 172.21.15.40:0/3253697976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff068135c20 0x7ff06806f930 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff05800c780 tx=0x7ff05800cc40 comp rx=0 tx=0).stop 2024-04-22T08:13:57.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.463+0000 7ff06daa9640 1 -- 172.21.15.40:0/3253697976 reap_dead start 2024-04-22T08:13:57.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.463+0000 7ff06daa9640 1 -- 172.21.15.40:0/3253697976 reap_dead start 2024-04-22T08:13:57.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.463+0000 7ff06f50c640 1 -- 172.21.15.40:0/3253697976 shutdown_connections 2024-04-22T08:13:57.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.463+0000 7ff06f50c640 1 -- 172.21.15.40:0/3253697976 >> 172.21.15.40:0/3253697976 conn(0x7ff0680a66a0 msgr2=0x7ff068099f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:13:57.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.463+0000 7ff06f50c640 1 -- 172.21.15.40:0/3253697976 shutdown_connections 2024-04-22T08:13:57.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.463+0000 7ff06f50c640 1 -- 172.21.15.40:0/3253697976 wait complete. 2024-04-22T08:13:57.478 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:13:57.478 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-22T08:13:57.478 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-22T08:13:57.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.579+0000 7f6b0df8d640 10 monclient: get_monmap_and_config 2024-04-22T08:13:57.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.579+0000 7f6b0df8d640 10 monclient: build_initial_monmap 2024-04-22T08:13:57.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.579+0000 7f6b0df8d640 10 monclient: monmap: 2024-04-22T08:13:57.579 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:13:57.579 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:57.579 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:13:57.579943+0000 2024-04-22T08:13:57.579 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:13:57.579943+0000 2024-04-22T08:13:57.579 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:13:57.579 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:57.579 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:13:57.579 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:13:57.579 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:13:57.580 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:13:57.580 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:13:57.580 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:57.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.579+0000 7f6b0df8d640 1 Processor -- start 2024-04-22T08:13:57.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.579+0000 7f6b0df8d640 1 -- start start 2024-04-22T08:13:57.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.579+0000 7f6b0df8d640 10 monclient: init 2024-04-22T08:13:57.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.580+0000 7f6b0df8d640 10 monclient: _reopen_session rank -1 2024-04-22T08:13:57.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.580+0000 7f6b0df8d640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-04-22T08:13:57.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.580+0000 7f6b0df8d640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6b080a20d0 0x7f6b080a24b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:57.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.580+0000 7f6b0df8d640 10 monclient(hunting): picked mon.noname-a con 0x7f6b080a20d0 addr v2:172.21.15.40:3300/0 2024-04-22T08:13:57.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.580+0000 7f6b0df8d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6b080a2d10 addr v1:172.21.15.40:6789/0 2024-04-22T08:13:57.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.580+0000 7f6b0df8d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b080a38d0 0x7f6b0813db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:57.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.580+0000 7f6b0df8d640 10 monclient(hunting): picked mon.noname-b con 0x7f6b080a38d0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:13:57.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.580+0000 7f6b0df8d640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b080a45f0 con 0x7f6b080a2d10 2024-04-22T08:13:57.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.580+0000 7f6b0df8d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:57.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.580+0000 7f6b0df8d640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b080a0360 con 0x7f6b080a20d0 2024-04-22T08:13:57.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.580+0000 7f6b0df8d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:57.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.580+0000 7f6b0df8d640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b080a0640 con 0x7f6b080a38d0 2024-04-22T08:13:57.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.580+0000 7f6b0df8d640 10 monclient(hunting): _renew_subs 2024-04-22T08:13:57.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.580+0000 7f6b0df8d640 10 monclient(hunting): authenticate will time out at 911.574249s 2024-04-22T08:13:57.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.580+0000 7f6b077fe640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6b080a20d0 0x7f6b080a24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:57.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.580+0000 7f6b07fff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b080a38d0 0x7f6b0813db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:57.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.580+0000 7f6b077fe640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6b080a20d0 0x7f6b080a24b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37090/0 (socket says 172.21.15.40:37090) 2024-04-22T08:13:57.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.580+0000 7f6b07fff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b080a38d0 0x7f6b0813db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:54430/0 (socket says 172.21.15.40:54430) 2024-04-22T08:13:57.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.580+0000 7f6b06ffd640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f6b080a2d10 0x7f6b080a31c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:48482/0 (socket says 172.21.15.40:48482) 2024-04-22T08:13:57.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.580+0000 7f6b077fe640 1 -- 172.21.15.40:0/4159549030 learned_addr learned my addr 172.21.15.40:0/4159549030 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:13:57.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.580+0000 7f6b077fe640 10 monclient(hunting): get_auth_request con 0x7f6b080a20d0 auth_method 0 2024-04-22T08:13:57.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.580+0000 7f6b077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:57.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.580+0000 7f6b077fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:57.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.580+0000 7f6b077fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:57.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.580+0000 7f6b07fff640 10 monclient(hunting): get_auth_request con 0x7f6b080a38d0 auth_method 0 2024-04-22T08:13:57.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.580+0000 7f6b07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:57.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.580+0000 7f6b07fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:57.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.580+0000 7f6b07fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:57.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.580+0000 7f6b07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:57.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.580+0000 7f6b07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:57.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.580+0000 7f6b07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:57.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.580+0000 7f6b07fff640 10 monclient(hunting): handle_auth_done global_id 4502 payload 274 2024-04-22T08:13:57.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.580+0000 7f6b07fff640 1 -- 172.21.15.40:0/4159549030 >> v2:172.21.15.40:3300/0 conn(0x7f6b080a20d0 msgr2=0x7f6b080a24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:13:57.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.580+0000 7f6b07fff640 1 --2- 172.21.15.40:0/4159549030 >> v2:172.21.15.40:3300/0 conn(0x7f6b080a20d0 0x7f6b080a24b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:57.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.580+0000 7f6b07fff640 1 -- 172.21.15.40:0/4159549030 >> v1:172.21.15.40:6789/0 conn(0x7f6b080a2d10 legacy=0x7f6b080a31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:13:57.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.580+0000 7f6b07fff640 10 monclient: _finish_hunting 0 2024-04-22T08:13:57.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.580+0000 7f6b07fff640 1 monclient: found mon.noname-b 2024-04-22T08:13:57.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.580+0000 7f6b07fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:13:57.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.580+0000 7f6b07fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:13:57.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.580+0000 7f6b07fff640 1 -- 172.21.15.40:0/4159549030 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b080a0920 con 0x7f6b080a38d0 2024-04-22T08:13:57.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.583+0000 7f6b07fff640 1 -- 172.21.15.40:0/4159549030 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b080a38d0 msgr2=0x7f6b0813db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:13:57.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.583+0000 7f6b07fff640 1 --2- 172.21.15.40:0/4159549030 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b080a38d0 0x7f6b0813db20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6b00009d90 tx=0x7f6b0002bcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27c5452f716dfa53 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:57.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.583+0000 7f6b067fc640 1 -- 172.21.15.40:0/4159549030 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b0002ccb0 con 0x7f6b080a38d0 2024-04-22T08:13:57.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.583+0000 7f6b067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:57.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.583+0000 7f6b067fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:13:57.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.583+0000 7f6b067fc640 10 monclient: dump: 2024-04-22T08:13:57.591 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:57.591 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:57.591 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:57.592 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:57.592 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:57.592 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:57.592 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:57.592 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:57.592 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:57.592 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:57.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.584+0000 7f6b067fc640 10 monclient: _finish_auth 0 2024-04-22T08:13:57.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.584+0000 7f6b067fc640 10 monclient: _check_auth_tickets 2024-04-22T08:13:57.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.584+0000 7f6b067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:13:57.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.584+0000 7f6b067fc640 1 -- 172.21.15.40:0/4159549030 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b0002ce10 con 0x7f6b080a38d0 2024-04-22T08:13:57.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.584+0000 7f6b0df8d640 5 monclient: authenticate success, global_id 4502 2024-04-22T08:13:57.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.584+0000 7f6b0df8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:13:57.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.584+0000 7f6b067fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:13:57.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.584+0000 7f6b0df8d640 10 monclient: get_monmap_and_config success 2024-04-22T08:13:57.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.584+0000 7f6b067fc640 1 -- 172.21.15.40:0/4159549030 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b00034660 con 0x7f6b080a38d0 2024-04-22T08:13:57.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.584+0000 7f6b067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:57.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.584+0000 7f6b0df8d640 4 set_mon_vals no callback set 2024-04-22T08:13:57.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.584+0000 7f6b067fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:13:57.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.584+0000 7f6b067fc640 10 monclient: dump: 2024-04-22T08:13:57.593 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:57.593 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:57.593 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:57.593 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:57.593 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:57.594 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:57.594 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:57.594 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:57.594 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:57.594 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:57.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.584+0000 7f6b0df8d640 10 monclient: shutdown 2024-04-22T08:13:57.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.584+0000 7f6b0df8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:13:57.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.584+0000 7f6b0df8d640 1 -- 172.21.15.40:0/4159549030 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b080a38d0 msgr2=0x7f6b0813db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:57.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.584+0000 7f6b0df8d640 1 --2- 172.21.15.40:0/4159549030 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b080a38d0 0x7f6b0813db20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6b00009d90 tx=0x7f6b0002bcf0 comp rx=0 tx=0).stop 2024-04-22T08:13:57.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.584+0000 7f6b0df8d640 1 -- 172.21.15.40:0/4159549030 shutdown_connections 2024-04-22T08:13:57.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.584+0000 7f6b0df8d640 1 --2- 172.21.15.40:0/4159549030 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b080a38d0 0x7f6b0813db20 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:57.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.584+0000 7f6b0df8d640 1 --2- 172.21.15.40:0/4159549030 >> v2:172.21.15.40:3300/0 conn(0x7f6b080a20d0 0x7f6b080a24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:57.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.584+0000 7f6b0df8d640 1 -- 172.21.15.40:0/4159549030 >> 172.21.15.40:0/4159549030 conn(0x7f6b080a8e60 msgr2=0x7f6b080a4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:13:57.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.584+0000 7f6b0df8d640 1 -- 172.21.15.40:0/4159549030 shutdown_connections 2024-04-22T08:13:57.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.584+0000 7f6b0df8d640 1 -- 172.21.15.40:0/4159549030 wait complete. 2024-04-22T08:13:57.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.585+0000 7f6b0df8d640 10 monclient: build_initial_monmap 2024-04-22T08:13:57.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.585+0000 7f6b0df8d640 10 monclient: monmap: 2024-04-22T08:13:57.596 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:13:57.596 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:13:57.596 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:13:57.596 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:13:57.596 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:13:57.596 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:57.596 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:13:57.596 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:13:57.596 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:13:57.596 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:57.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.585+0000 7f6b0df8d640 1 Processor -- start 2024-04-22T08:13:57.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.585+0000 7f6b0df8d640 1 -- start start 2024-04-22T08:13:57.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.585+0000 7f6b0df8d640 10 monclient: init 2024-04-22T08:13:57.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.586+0000 7f6b0df8d640 10 monclient: _reopen_session rank -1 2024-04-22T08:13:57.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.586+0000 7f6b0df8d640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-22T08:13:57.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.586+0000 7f6b0df8d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b080a20d0 0x7f6b081c4540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:57.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.586+0000 7f6b0df8d640 10 monclient(hunting): picked mon.noname-a con 0x7f6b080a20d0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:13:57.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.586+0000 7f6b0df8d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b080a2d10 0x7f6b081c71c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:57.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.586+0000 7f6b0df8d640 10 monclient(hunting): picked mon.noname-c con 0x7f6b080a2d10 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:13:57.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.586+0000 7f6b0df8d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b080a38d0 0x7f6b081c7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:57.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.586+0000 7f6b0df8d640 10 monclient(hunting): picked mon.noname-b con 0x7f6b080a38d0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:13:57.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.586+0000 7f6b0df8d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:57.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.586+0000 7f6b0df8d640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b08070d80 con 0x7f6b080a20d0 2024-04-22T08:13:57.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.586+0000 7f6b0df8d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:57.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.586+0000 7f6b0df8d640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b08071750 con 0x7f6b080a38d0 2024-04-22T08:13:57.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.586+0000 7f6b0df8d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:57.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.586+0000 7f6b0df8d640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b081a93c0 con 0x7f6b080a2d10 2024-04-22T08:13:57.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.586+0000 7f6b0df8d640 10 monclient(hunting): _renew_subs 2024-04-22T08:13:57.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.586+0000 7f6b0df8d640 10 monclient(hunting): authenticate will time out at 911.579880s 2024-04-22T08:13:57.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.586+0000 7f6b077fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b080a20d0 0x7f6b081c4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:57.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.586+0000 7f6b06ffd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b080a2d10 0x7f6b081c71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:57.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.586+0000 7f6b077fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b080a20d0 0x7f6b081c4540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37098/0 (socket says 172.21.15.40:37098) 2024-04-22T08:13:57.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.586+0000 7f6b077fe640 1 -- 172.21.15.40:0/3720240479 learned_addr learned my addr 172.21.15.40:0/3720240479 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:13:57.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.586+0000 7f6b077fe640 10 monclient(hunting): get_auth_request con 0x7f6b080a20d0 auth_method 0 2024-04-22T08:13:57.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.586+0000 7f6b077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:57.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.586+0000 7f6b077fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:57.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.586+0000 7f6b077fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:57.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.586+0000 7f6b06ffd640 10 monclient(hunting): get_auth_request con 0x7f6b080a2d10 auth_method 0 2024-04-22T08:13:57.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.586+0000 7f6b06ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:57.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.586+0000 7f6b06ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:57.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.586+0000 7f6b06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:57.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.586+0000 7f6b06ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:57.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.586+0000 7f6b06ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:57.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.586+0000 7f6b06ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:57.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.586+0000 7f6b077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:57.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.586+0000 7f6b077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:57.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.586+0000 7f6b077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:57.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.586+0000 7f6b07fff640 1 --2- 172.21.15.40:0/3720240479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b080a38d0 0x7f6b081c7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:57.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.586+0000 7f6b06ffd640 10 monclient(hunting): handle_auth_done global_id 4505 payload 824 2024-04-22T08:13:57.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.586+0000 7f6b06ffd640 1 -- 172.21.15.40:0/3720240479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b080a38d0 msgr2=0x7f6b081c7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:57.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.586+0000 7f6b06ffd640 1 --2- 172.21.15.40:0/3720240479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b080a38d0 0x7f6b081c7700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:57.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.586+0000 7f6b06ffd640 1 -- 172.21.15.40:0/3720240479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b080a20d0 msgr2=0x7f6b081c4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:57.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.586+0000 7f6b06ffd640 1 --2- 172.21.15.40:0/3720240479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b080a20d0 0x7f6b081c4540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:57.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.586+0000 7f6b06ffd640 10 monclient: _finish_hunting 0 2024-04-22T08:13:57.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.586+0000 7f6b06ffd640 1 monclient: found mon.noname-c 2024-04-22T08:13:57.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.586+0000 7f6b06ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:13:57.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.586+0000 7f6b06ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:13:57.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.586+0000 7f6b06ffd640 1 -- 172.21.15.40:0/3720240479 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b00009a70 con 0x7f6b080a2d10 2024-04-22T08:13:57.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.586+0000 7f6b07fff640 10 monclient: get_auth_request con 0x7f6b080a38d0 auth_method 0 2024-04-22T08:13:57.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.586+0000 7f6b07fff640 1 --2- 172.21.15.40:0/3720240479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b080a38d0 0x7f6b081c7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T08:13:57.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.588+0000 7f6b06ffd640 1 -- 172.21.15.40:0/3720240479 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b080a2d10 msgr2=0x7f6b081c71c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:13:57.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.588+0000 7f6b06ffd640 1 --2- 172.21.15.40:0/3720240479 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b080a2d10 0x7f6b081c71c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6afc00c390 tx=0x7f6afc00c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:57.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.590+0000 7f6aeffff640 1 -- 172.21.15.40:0/3720240479 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6afc005810 con 0x7f6b080a2d10 2024-04-22T08:13:57.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.591+0000 7f6aeffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:57.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.591+0000 7f6aeffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:13:57.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.591+0000 7f6aeffff640 10 monclient: dump: 2024-04-22T08:13:57.604 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:57.604 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:57.604 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:57.604 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:57.604 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:57.604 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:57.605 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:57.605 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:57.605 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:57.605 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:57.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.591+0000 7f6aeffff640 10 monclient: _finish_auth 0 2024-04-22T08:13:57.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.591+0000 7f6aeffff640 10 monclient: _check_auth_tickets 2024-04-22T08:13:57.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.591+0000 7f6aeffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:13:57.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.591+0000 7f6aeffff640 1 -- 172.21.15.40:0/3720240479 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6afc005990 con 0x7f6b080a2d10 2024-04-22T08:13:57.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.591+0000 7f6b0df8d640 5 monclient: authenticate success, global_id 4505 2024-04-22T08:13:57.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.591+0000 7f6b0df8d640 10 monclient: _renew_subs 2024-04-22T08:13:57.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.591+0000 7f6b0df8d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:13:57.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.591+0000 7f6b0df8d640 1 -- 172.21.15.40:0/3720240479 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b081a9b60 con 0x7f6b080a2d10 2024-04-22T08:13:57.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.591+0000 7f6aeffff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:13:57.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.591+0000 7f6aeffff640 1 -- 172.21.15.40:0/3720240479 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6afc005d40 con 0x7f6b080a2d10 2024-04-22T08:13:57.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.591+0000 7f6aeffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:57.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.591+0000 7f6aeffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:13:57.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.591+0000 7f6aeffff640 10 monclient: dump: 2024-04-22T08:13:57.606 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:57.606 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:57.606 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:57.606 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:57.606 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:57.606 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:57.606 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:57.606 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:57.606 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:57.606 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:57.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.591+0000 7f6b0df8d640 10 monclient: _renew_subs 2024-04-22T08:13:57.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.591+0000 7f6b05ffb640 4 set_mon_vals no callback set 2024-04-22T08:13:57.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.591+0000 7f6b0df8d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:13:57.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.591+0000 7f6b0df8d640 1 -- 172.21.15.40:0/3720240479 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b081ab8e0 con 0x7f6b080a2d10 2024-04-22T08:13:57.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.591+0000 7f6aeffff640 1 -- 172.21.15.40:0/3720240479 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f6afc01b3b0 con 0x7f6b080a2d10 2024-04-22T08:13:57.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.591+0000 7f6b0df8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:13:57.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.591+0000 7f6b0df8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:13:57.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.591+0000 7f6b0df8d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:13:57.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.591+0000 7f6b0df8d640 1 -- 172.21.15.40:0/3720240479 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b080695f0 con 0x7f6b080a2d10 2024-04-22T08:13:57.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.592+0000 7f6aeffff640 1 --2- 172.21.15.40:0/3720240479 >> v2:172.21.15.189:6800/30168 conn(0x7f6af4037820 0x7f6af4039ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:57.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.592+0000 7f6aeffff640 1 -- 172.21.15.40:0/3720240479 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f6afc049a70 con 0x7f6b080a2d10 2024-04-22T08:13:57.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.592+0000 7f6b077fe640 1 --2- 172.21.15.40:0/3720240479 >> v2:172.21.15.189:6800/30168 conn(0x7f6af4037820 0x7f6af4039ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:57.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.592+0000 7f6b077fe640 10 monclient: get_auth_request con 0x7f6af4037820 auth_method 0 2024-04-22T08:13:57.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.594+0000 7f6b077fe640 1 --2- 172.21.15.40:0/3720240479 >> v2:172.21.15.189:6800/30168 conn(0x7f6af4037820 0x7f6af4039ce0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6b080a8220 tx=0x7f6af8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:57.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.596+0000 7f6aeffff640 1 -- 172.21.15.40:0/3720240479 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f6afc01a360 con 0x7f6b080a2d10 2024-04-22T08:13:57.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.596+0000 7f6aeffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:13:57.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.596+0000 7f6aeffff640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:13:57.774 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:13:57.774 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[34721]: starting. 2024-04-22T08:13:57.774 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[34721]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-34686 2024-04-22T08:13:57.774 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[34721]: created object 0... 2024-04-22T08:13:57.775 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[34721]: created object 25... 2024-04-22T08:13:57.775 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[34721]: created object 49... 2024-04-22T08:13:57.775 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[34721]: finishing. 2024-04-22T08:13:57.775 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[34721]: shutting down. 2024-04-22T08:13:57.812 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:13:57.812 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[34723]: starting. 2024-04-22T08:13:57.813 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[34723]: listing objects. 2024-04-22T08:13:57.813 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[34723]: listed object 0... 2024-04-22T08:13:57.813 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[34723]: listed object 25... 2024-04-22T08:13:57.813 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[34723]: saw 50 objects 2024-04-22T08:13:57.813 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[34723]: shutting down. 2024-04-22T08:13:57.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.859+0000 7f6b0df8d640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-22T08:13:57.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.859+0000 7f6b0df8d640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-22T08:13:57.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.859+0000 7f6b0df8d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:13:57.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.859+0000 7f6b0df8d640 1 -- 172.21.15.40:0/3720240479 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f6b0804e9b0 con 0x7f6b080a2d10 2024-04-22T08:13:57.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.865+0000 7f6aeffff640 1 -- 172.21.15.40:0/3720240479 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f6afc011360 con 0x7f6b080a2d10 2024-04-22T08:13:57.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.865+0000 7f6aeffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-22T08:13:57.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.865+0000 7f6aeffff640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:13:57.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.867+0000 7f6b0df8d640 1 -- 172.21.15.40:0/3720240479 >> v2:172.21.15.189:6800/30168 conn(0x7f6af4037820 msgr2=0x7f6af4039ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:57.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.867+0000 7f6b0df8d640 1 --2- 172.21.15.40:0/3720240479 >> v2:172.21.15.189:6800/30168 conn(0x7f6af4037820 0x7f6af4039ce0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6b080a8220 tx=0x7f6af8008040 comp rx=0 tx=0).stop 2024-04-22T08:13:57.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.867+0000 7f6b0df8d640 10 monclient: shutdown 2024-04-22T08:13:57.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.867+0000 7f6b0df8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:13:57.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.867+0000 7f6b0df8d640 1 -- 172.21.15.40:0/3720240479 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b080a2d10 msgr2=0x7f6b081c71c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:57.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.867+0000 7f6b0df8d640 1 --2- 172.21.15.40:0/3720240479 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b080a2d10 0x7f6b081c71c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6afc00c390 tx=0x7f6afc00c850 comp rx=0 tx=0).stop 2024-04-22T08:13:57.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.867+0000 7f6b0df8d640 1 -- 172.21.15.40:0/3720240479 shutdown_connections 2024-04-22T08:13:57.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.867+0000 7f6b0df8d640 1 --2- 172.21.15.40:0/3720240479 >> v2:172.21.15.189:6800/30168 conn(0x7f6af4037820 0x7f6af4039ce0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:57.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.867+0000 7f6b0df8d640 1 --2- 172.21.15.40:0/3720240479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b080a38d0 0x7f6b081c7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:57.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.867+0000 7f6b0df8d640 1 --2- 172.21.15.40:0/3720240479 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b080a2d10 0x7f6b081c71c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:57.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.867+0000 7f6b0df8d640 1 --2- 172.21.15.40:0/3720240479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b080a20d0 0x7f6b081c4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:57.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.867+0000 7f6b0df8d640 1 -- 172.21.15.40:0/3720240479 >> 172.21.15.40:0/3720240479 conn(0x7f6b080a8e60 msgr2=0x7f6b0806ef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:13:57.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.867+0000 7f6b0df8d640 1 -- 172.21.15.40:0/3720240479 shutdown_connections 2024-04-22T08:13:57.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:57.867+0000 7f6b0df8d640 1 -- 172.21.15.40:0/3720240479 wait complete. 2024-04-22T08:13:57.877 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-22T08:13:57.921 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:13:57.922+0000 7fe725c5d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:13:57.922 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:13:57.922+0000 7fe725c5d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=8,config=2,health=12,kv=9,logm=69} crc {auth=2146927704,config=1618278320,health=861809967,kv=1350343662,logm=408021472}) 2024-04-22T08:13:57.922 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:13:57.922+0000 7fe725c5d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=9,config=2,health=11,kv=7,logm=71} crc {auth=1696912916,config=1618278320,health=305525474,kv=251385012,logm=3538681491}) 2024-04-22T08:13:57.922 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:13:57.922+0000 7fe725c5d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:13:57.922 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:13:57.922+0000 7fe725c5d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=8,config=2,health=12,kv=9,logm=69} crc {auth=2146927704,config=1618278320,health=861809967,kv=1350343662,logm=408021472}) 2024-04-22T08:13:57.922 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:13:57.922+0000 7fe725c5d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=9,config=2,health=12,kv=9,logm=68} crc {auth=1696912916,config=1618278320,health=861809967,kv=1350343662,logm=1986270515}) 2024-04-22T08:13:58.447 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:13:58.447+0000 7fe725c5d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:13:58.447 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:13:58.447+0000 7fe725c5d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4195790171}) 2024-04-22T08:13:58.447 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:13:58.447+0000 7fe725c5d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3675865838}) 2024-04-22T08:13:58.448 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:13:58.447+0000 7fe725c5d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:13:58.448 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:13:58.447+0000 7fe725c5d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4195790171}) 2024-04-22T08:13:58.448 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:13:58.447+0000 7fe725c5d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1404114855}) 2024-04-22T08:13:58.652 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:13:58.652+0000 7fe725c5d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:13:58.652 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:13:58.652+0000 7fe725c5d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=162147813}) 2024-04-22T08:13:58.652 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:13:58.652+0000 7fe725c5d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1775760891}) 2024-04-22T08:13:58.652 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:13:58.652+0000 7fe725c5d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:13:58.652 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:13:58.652+0000 7fe725c5d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=162147813}) 2024-04-22T08:13:58.652 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:13:58.652+0000 7fe725c5d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1444905598}) 2024-04-22T08:13:58.655 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:13:58.656+0000 7fe725c5d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:13:58.656 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:13:58.656+0000 7fe725c5d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=90,mdsmap=3,mgr=7} crc {logm=1333071766,mdsmap=1531151339,mgr=1219260174}) 2024-04-22T08:13:58.656 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:13:58.656+0000 7fe725c5d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=90,mdsmap=3,mgr=7} crc {logm=4240689465,mdsmap=2147943554,mgr=1219260174}) 2024-04-22T08:13:58.656 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:13:58.656+0000 7fe725c5d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:13:58.656 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:13:58.656+0000 7fe725c5d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=90,mdsmap=3,mgr=7} crc {logm=1333071766,mdsmap=1531151339,mgr=1219260174}) 2024-04-22T08:13:58.656 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:13:58.656+0000 7fe725c5d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=90,mdsmap=3,mgr=7} crc {logm=3047466143,mdsmap=1531151339,mgr=1475285388}) 2024-04-22T08:13:58.658 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:13:58.658+0000 7fe725c5d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:13:58.658 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:13:58.658+0000 7fe725c5d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=1,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=8,monmap=2,osd_metadata=6,osd_pg_creating=1,osdmap=70} crc {mgr=1747058506,mgr_command_descs=470045235,mgr_metadata=4222303120,mgrstat=510721047,mon_config_key=4255067121,monmap=2343331355,osd_metadata=4151711904,osd_pg_creating=1267774760,osdmap=3888696507}) 2024-04-22T08:13:58.658 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:13:58.658+0000 7fe725c5d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr=1,mgr_metadata=1,mgrstat=8,mon_config_key=8,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=72} crc {mgr=1747058506,mgr_metadata=4222303120,mgrstat=3441439591,mon_config_key=4255067121,monmap=3469086636,osd_metadata=4151711904,osd_pg_creating=1267774760,osdmap=3318833662}) 2024-04-22T08:13:58.658 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:13:58.658+0000 7fe725c5d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:13:58.658 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:13:58.658+0000 7fe725c5d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=1,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=8,monmap=2,osd_metadata=6,osd_pg_creating=1,osdmap=70} crc {mgr=1747058506,mgr_command_descs=470045235,mgr_metadata=4222303120,mgrstat=510721047,mon_config_key=4255067121,monmap=2343331355,osd_metadata=4151711904,osd_pg_creating=1267774760,osdmap=3888696507}) 2024-04-22T08:13:58.658 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:13:58.658+0000 7fe725c5d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=1,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=8,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=70} crc {mgr=1747058506,mgr_command_descs=470045235,mgr_metadata=4222303120,mgrstat=3920853130,mon_config_key=4255067121,monmap=3469086636,osd_metadata=4151711904,osd_pg_creating=1267774760,osdmap=671727140}) 2024-04-22T08:13:58.659 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:13:58.660+0000 7fe725c5d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:13:58.660 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:13:58.660+0000 7fe725c5d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=6} crc {osdmap=1355694421}) 2024-04-22T08:13:58.660 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:13:58.660+0000 7fe725c5d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=6} crc {osdmap=2649776845}) 2024-04-22T08:13:58.660 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:13:58.660+0000 7fe725c5d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:13:58.660 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:13:58.660+0000 7fe725c5d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=6} crc {osdmap=1355694421}) 2024-04-22T08:13:58.660 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:13:58.660+0000 7fe725c5d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=6} crc {osdmap=1878814515}) 2024-04-22T08:13:58.661 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:13:58.661 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[34722]: starting. 2024-04-22T08:13:58.661 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[34722]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-34686 2024-04-22T08:13:58.661 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[34722]: shutting down. 2024-04-22T08:13:58.662 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:13:58.663 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:13:58.663 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:13:58.665 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:13:58.878 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-22T08:13:58.879 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:13:58.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.986+0000 7f743c83f640 10 monclient: get_monmap_and_config 2024-04-22T08:13:58.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.986+0000 7f743c83f640 10 monclient: build_initial_monmap 2024-04-22T08:13:58.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.986+0000 7f743c83f640 10 monclient: monmap: 2024-04-22T08:13:58.986 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:13:58.986 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:58.986 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:13:58.987004+0000 2024-04-22T08:13:58.986 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:13:58.987004+0000 2024-04-22T08:13:58.986 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:13:58.986 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:58.986 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:13:58.986 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:13:58.986 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:13:58.986 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:13:58.986 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:13:58.987 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:58.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.986+0000 7f743c83f640 1 Processor -- start 2024-04-22T08:13:58.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.987+0000 7f743c83f640 1 -- start start 2024-04-22T08:13:58.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.987+0000 7f743c83f640 10 monclient: init 2024-04-22T08:13:58.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.987+0000 7f743c83f640 10 monclient: _reopen_session rank -1 2024-04-22T08:13:58.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.987+0000 7f743c83f640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-04-22T08:13:58.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.987+0000 7f743c83f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f743809ee10 addr v1:172.21.15.189:6789/0 2024-04-22T08:13:58.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.987+0000 7f743c83f640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74380a32b0 0x7f743813b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:58.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.987+0000 7f743c83f640 10 monclient(hunting): picked mon.noname-b con 0x7f74380a32b0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:13:58.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.987+0000 7f743c83f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f743813b5f0 addr v1:172.21.15.40:6789/0 2024-04-22T08:13:58.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.987+0000 7f743c83f640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f743809f1f0 con 0x7f743813b5f0 2024-04-22T08:13:58.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.987+0000 7f743c83f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74380a39c0 con 0x7f743809ee10 2024-04-22T08:13:58.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.987+0000 7f743c83f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:58.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.987+0000 7f743c83f640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f743809fd30 con 0x7f74380a32b0 2024-04-22T08:13:58.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.987+0000 7f743c83f640 10 monclient(hunting): _renew_subs 2024-04-22T08:13:58.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.987+0000 7f743c83f640 10 monclient(hunting): authenticate will time out at 912.981282s 2024-04-22T08:13:58.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.987+0000 7f742ffff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74380a32b0 0x7f743813b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:58.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.987+0000 7f7436d9d640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f743813b5f0 0x7f743813d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:51162/0 (socket says 172.21.15.40:51162) 2024-04-22T08:13:58.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.987+0000 7f742ffff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74380a32b0 0x7f743813b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:51492/0 (socket says 172.21.15.40:51492) 2024-04-22T08:13:58.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.987+0000 7f7436d9d640 1 -- 172.21.15.40:0/2224980103 learned_addr learned my addr 172.21.15.40:0/2224980103 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:13:58.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.987+0000 7f742ffff640 10 monclient(hunting): get_auth_request con 0x7f74380a32b0 auth_method 0 2024-04-22T08:13:58.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.987+0000 7f742ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:58.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.987+0000 7f742ffff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:58.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.987+0000 7f742ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:58.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.987+0000 7f742ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:58.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.987+0000 7f742ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:58.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.987+0000 7f742ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:58.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.987+0000 7f742ffff640 10 monclient(hunting): handle_auth_done global_id 4517 payload 274 2024-04-22T08:13:58.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.987+0000 7f742ffff640 1 -- 172.21.15.40:0/2224980103 >> v1:172.21.15.189:6789/0 conn(0x7f743809ee10 legacy=0x7f74380a1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:13:58.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.987+0000 7f742ffff640 1 -- 172.21.15.40:0/2224980103 >> v1:172.21.15.40:6789/0 conn(0x7f743813b5f0 legacy=0x7f743813d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:13:58.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.987+0000 7f742ffff640 10 monclient: _finish_hunting 0 2024-04-22T08:13:58.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.987+0000 7f742ffff640 1 monclient: found mon.noname-b 2024-04-22T08:13:58.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.987+0000 7f742ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:13:58.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.987+0000 7f742ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:13:58.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.987+0000 7f742ffff640 1 -- 172.21.15.40:0/2224980103 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74380a02f0 con 0x7f74380a32b0 2024-04-22T08:13:58.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.990+0000 7f742ffff640 1 -- 172.21.15.40:0/2224980103 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74380a32b0 msgr2=0x7f743813b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:13:58.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.990+0000 7f742ffff640 1 --2- 172.21.15.40:0/2224980103 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74380a32b0 0x7f743813b0b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f74380a43e0 tx=0x7f742002b6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b32ce726c5468b41 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:58.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.991+0000 7f7435d9b640 1 -- 172.21.15.40:0/2224980103 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7420030a00 con 0x7f74380a32b0 2024-04-22T08:13:58.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.991+0000 7f7435d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:58.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.991+0000 7f7435d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:13:58.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.991+0000 7f7435d9b640 10 monclient: dump: 2024-04-22T08:13:58.995 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:58.995 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:58.995 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:58.995 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:58.995 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:58.995 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:58.996 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:58.996 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:58.996 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:58.996 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:58.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.991+0000 7f7435d9b640 10 monclient: _finish_auth 0 2024-04-22T08:13:58.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.991+0000 7f7435d9b640 10 monclient: _check_auth_tickets 2024-04-22T08:13:58.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.991+0000 7f7435d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:13:58.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.991+0000 7f7435d9b640 1 -- 172.21.15.40:0/2224980103 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f742002e050 con 0x7f74380a32b0 2024-04-22T08:13:58.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.991+0000 7f7435d9b640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:13:58.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.991+0000 7f7435d9b640 1 -- 172.21.15.40:0/2224980103 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7420030d80 con 0x7f74380a32b0 2024-04-22T08:13:58.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.991+0000 7f743c83f640 5 monclient: authenticate success, global_id 4517 2024-04-22T08:13:58.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.991+0000 7f743c83f640 10 monclient: get_monmap_and_config success 2024-04-22T08:13:58.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.991+0000 7f743c83f640 4 set_mon_vals no callback set 2024-04-22T08:13:58.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.991+0000 7f7435d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:58.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.991+0000 7f7435d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:13:58.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.991+0000 7f7435d9b640 10 monclient: dump: 2024-04-22T08:13:58.997 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:58.997 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:58.997 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:58.997 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:58.997 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:58.997 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:58.997 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:58.997 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:58.997 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:58.997 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:58.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.991+0000 7f743c83f640 10 monclient: shutdown 2024-04-22T08:13:58.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.991+0000 7f743c83f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:13:58.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.991+0000 7f743c83f640 1 -- 172.21.15.40:0/2224980103 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74380a32b0 msgr2=0x7f743813b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:58.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.991+0000 7f743c83f640 1 --2- 172.21.15.40:0/2224980103 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74380a32b0 0x7f743813b0b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f74380a43e0 tx=0x7f742002b6d0 comp rx=0 tx=0).stop 2024-04-22T08:13:58.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.991+0000 7f743c83f640 1 -- 172.21.15.40:0/2224980103 shutdown_connections 2024-04-22T08:13:58.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.991+0000 7f743c83f640 1 --2- 172.21.15.40:0/2224980103 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74380a32b0 0x7f743813b0b0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:58.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.991+0000 7f743c83f640 1 -- 172.21.15.40:0/2224980103 >> 172.21.15.40:0/2224980103 conn(0x7f74380a4ef0 msgr2=0x7f74380a5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:13:58.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.991+0000 7f743c83f640 1 -- 172.21.15.40:0/2224980103 shutdown_connections 2024-04-22T08:13:58.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.991+0000 7f743c83f640 1 -- 172.21.15.40:0/2224980103 wait complete. 2024-04-22T08:13:58.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.992+0000 7f743c83f640 10 monclient: build_initial_monmap 2024-04-22T08:13:58.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.992+0000 7f743c83f640 10 monclient: monmap: 2024-04-22T08:13:58.998 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:13:58.998 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:13:58.998 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:13:58.998 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:13:58.998 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:13:58.999 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:58.999 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:13:58.999 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:13:58.999 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:13:58.999 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:58.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.992+0000 7f743c83f640 1 Processor -- start 2024-04-22T08:13:58.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.992+0000 7f743c83f640 1 -- start start 2024-04-22T08:13:58.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.992+0000 7f743c83f640 10 monclient: init 2024-04-22T08:13:58.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f743c83f640 10 monclient: _reopen_session rank -1 2024-04-22T08:13:58.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f743c83f640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-22T08:13:58.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f743c83f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f743809ee10 0x7f74381cb180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:58.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f743c83f640 10 monclient(hunting): picked mon.noname-b con 0x7f743809ee10 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:13:58.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f743c83f640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74380a32b0 0x7f74381cb6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:58.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f743c83f640 10 monclient(hunting): picked mon.noname-c con 0x7f74380a32b0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:13:58.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f743c83f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f743813b5f0 0x7f743806d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:59.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f743c83f640 10 monclient(hunting): picked mon.noname-a con 0x7f743813b5f0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:13:59.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f743c83f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:59.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f743c83f640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f743806fc10 con 0x7f743813b5f0 2024-04-22T08:13:59.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f743c83f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:59.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f743c83f640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74380a2730 con 0x7f743809ee10 2024-04-22T08:13:59.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f743c83f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:59.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f743c83f640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7438070d40 con 0x7f74380a32b0 2024-04-22T08:13:59.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f743c83f640 10 monclient(hunting): _renew_subs 2024-04-22T08:13:59.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f743c83f640 10 monclient(hunting): authenticate will time out at 912.987116s 2024-04-22T08:13:59.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f742ffff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74380a32b0 0x7f74381cb6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:59.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f742ffff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74380a32b0 0x7f74381cb6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:51500/0 (socket says 172.21.15.40:51500) 2024-04-22T08:13:59.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f7436d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f743813b5f0 0x7f743806d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:59.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f742ffff640 1 -- 172.21.15.40:0/677780119 learned_addr learned my addr 172.21.15.40:0/677780119 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:13:59.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f742ffff640 10 monclient(hunting): get_auth_request con 0x7f74380a32b0 auth_method 0 2024-04-22T08:13:59.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f742ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:59.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f742ffff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:59.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f742ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:59.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f7436d9d640 10 monclient(hunting): get_auth_request con 0x7f743813b5f0 auth_method 0 2024-04-22T08:13:59.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f7436d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:59.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f7436d9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:59.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f7436d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:59.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f743659c640 1 --2- 172.21.15.40:0/677780119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f743809ee10 0x7f74381cb180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:59.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f743659c640 10 monclient(hunting): get_auth_request con 0x7f743809ee10 auth_method 0 2024-04-22T08:13:59.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f743659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:59.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f743659c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:59.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f743659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:59.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f742ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:59.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f742ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:59.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f742ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:59.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f7436d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:59.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f7436d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:59.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f7436d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:59.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f743659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:59.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f743659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:59.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f743659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:59.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f742ffff640 10 monclient(hunting): handle_auth_done global_id 4520 payload 824 2024-04-22T08:13:59.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f742ffff640 1 -- 172.21.15.40:0/677780119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f743809ee10 msgr2=0x7f74381cb180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:59.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f742ffff640 1 --2- 172.21.15.40:0/677780119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f743809ee10 0x7f74381cb180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:59.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f742ffff640 1 -- 172.21.15.40:0/677780119 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f743813b5f0 msgr2=0x7f743806d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:59.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f742ffff640 1 --2- 172.21.15.40:0/677780119 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f743813b5f0 0x7f743806d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:59.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f742ffff640 10 monclient: _finish_hunting 0 2024-04-22T08:13:59.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f742ffff640 1 monclient: found mon.noname-c 2024-04-22T08:13:59.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f742ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:13:59.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f742ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:13:59.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f742ffff640 1 -- 172.21.15.40:0/677780119 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7420009400 con 0x7f74380a32b0 2024-04-22T08:13:59.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.993+0000 7f7436d9d640 1 --2- 172.21.15.40:0/677780119 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f743813b5f0 0x7f743806d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:13:59.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.996+0000 7f742ffff640 1 -- 172.21.15.40:0/677780119 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74380a32b0 msgr2=0x7f74381cb6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:13:59.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.996+0000 7f742ffff640 1 --2- 172.21.15.40:0/677780119 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74380a32b0 0x7f74381cb6c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f74200098c0 tx=0x7f74200046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:59.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.996+0000 7f742effd640 1 -- 172.21.15.40:0/677780119 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f742003db60 con 0x7f74380a32b0 2024-04-22T08:13:59.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.996+0000 7f742effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:59.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.996+0000 7f742effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:13:59.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.996+0000 7f742effd640 10 monclient: dump: 2024-04-22T08:13:59.006 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:59.006 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:59.006 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.006 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.006 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:59.006 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:59.006 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:59.006 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:59.006 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:59.006 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:59.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.996+0000 7f742effd640 10 monclient: _finish_auth 0 2024-04-22T08:13:59.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.996+0000 7f742effd640 10 monclient: _check_auth_tickets 2024-04-22T08:13:59.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.996+0000 7f742effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:13:59.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.996+0000 7f742effd640 1 -- 172.21.15.40:0/677780119 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f742003dcc0 con 0x7f74380a32b0 2024-04-22T08:13:59.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.996+0000 7f743c83f640 5 monclient: authenticate success, global_id 4520 2024-04-22T08:13:59.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.996+0000 7f743c83f640 10 monclient: _renew_subs 2024-04-22T08:13:59.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.996+0000 7f743c83f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:13:59.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.996+0000 7f743c83f640 1 -- 172.21.15.40:0/677780119 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7438071020 con 0x7f74380a32b0 2024-04-22T08:13:59.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.996+0000 7f742effd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:13:59.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.996+0000 7f742effd640 1 -- 172.21.15.40:0/677780119 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74200326c0 con 0x7f74380a32b0 2024-04-22T08:13:59.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.996+0000 7f742effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:59.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.996+0000 7f742effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:13:59.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.996+0000 7f742effd640 10 monclient: dump: 2024-04-22T08:13:59.007 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:59.007 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:59.007 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.008 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.008 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:59.008 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:59.008 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:59.008 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:59.008 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:59.008 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:59.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.996+0000 7f743559a640 4 set_mon_vals no callback set 2024-04-22T08:13:59.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.996+0000 7f743c83f640 10 monclient: _renew_subs 2024-04-22T08:13:59.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.996+0000 7f743c83f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:13:59.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.996+0000 7f743c83f640 1 -- 172.21.15.40:0/677780119 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f743806f210 con 0x7f74380a32b0 2024-04-22T08:13:59.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.997+0000 7f742effd640 1 -- 172.21.15.40:0/677780119 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f7420039040 con 0x7f74380a32b0 2024-04-22T08:13:59.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.997+0000 7f743c83f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:13:59.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.997+0000 7f743c83f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:13:59.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.997+0000 7f743c83f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:13:59.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.997+0000 7f743c83f640 1 -- 172.21.15.40:0/677780119 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f743804e9b0 con 0x7f74380a32b0 2024-04-22T08:13:59.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.997+0000 7f742effd640 1 --2- 172.21.15.40:0/677780119 >> v2:172.21.15.189:6800/30168 conn(0x7f74180377d0 0x7f7418039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:59.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.997+0000 7f742effd640 1 -- 172.21.15.40:0/677780119 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f742006c3a0 con 0x7f74380a32b0 2024-04-22T08:13:59.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.997+0000 7f743659c640 1 --2- 172.21.15.40:0/677780119 >> v2:172.21.15.189:6800/30168 conn(0x7f74180377d0 0x7f7418039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:59.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:58.997+0000 7f743659c640 10 monclient: get_auth_request con 0x7f74180377d0 auth_method 0 2024-04-22T08:13:59.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.000+0000 7f743659c640 1 --2- 172.21.15.40:0/677780119 >> v2:172.21.15.189:6800/30168 conn(0x7f74180377d0 0x7f7418039c90 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7428006fd0 tx=0x7f7428009060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:59.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.002+0000 7f742effd640 1 -- 172.21.15.40:0/677780119 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f7420050350 con 0x7f74380a32b0 2024-04-22T08:13:59.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.002+0000 7f742effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:13:59.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.002+0000 7f742effd640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:13:59.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.251+0000 7f743c83f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:13:59.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.251+0000 7f743c83f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:13:59.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.251+0000 7f743c83f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:13:59.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.251+0000 7f743c83f640 1 -- 172.21.15.40:0/677780119 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7438070e80 con 0x7f74380a32b0 2024-04-22T08:13:59.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.252+0000 7f742effd640 1 -- 172.21.15.40:0/677780119 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f7438070e80 con 0x7f74380a32b0 2024-04-22T08:13:59.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.252+0000 7f742effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:13:59.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.252+0000 7f742effd640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:13:59.252 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:13:59.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.253+0000 7f743c83f640 1 -- 172.21.15.40:0/677780119 >> v2:172.21.15.189:6800/30168 conn(0x7f74180377d0 msgr2=0x7f7418039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:59.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.253+0000 7f743c83f640 1 --2- 172.21.15.40:0/677780119 >> v2:172.21.15.189:6800/30168 conn(0x7f74180377d0 0x7f7418039c90 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7428006fd0 tx=0x7f7428009060 comp rx=0 tx=0).stop 2024-04-22T08:13:59.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.253+0000 7f743c83f640 10 monclient: shutdown 2024-04-22T08:13:59.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.253+0000 7f743c83f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:13:59.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.253+0000 7f743c83f640 1 -- 172.21.15.40:0/677780119 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74380a32b0 msgr2=0x7f74381cb6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:59.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.253+0000 7f743c83f640 1 --2- 172.21.15.40:0/677780119 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74380a32b0 0x7f74381cb6c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f74200098c0 tx=0x7f74200046c0 comp rx=0 tx=0).stop 2024-04-22T08:13:59.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.253+0000 7f743c83f640 1 -- 172.21.15.40:0/677780119 shutdown_connections 2024-04-22T08:13:59.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.253+0000 7f743c83f640 1 --2- 172.21.15.40:0/677780119 >> v2:172.21.15.189:6800/30168 conn(0x7f74180377d0 0x7f7418039c90 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:59.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.253+0000 7f743c83f640 1 --2- 172.21.15.40:0/677780119 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f743813b5f0 0x7f743806d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:59.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.253+0000 7f743c83f640 1 --2- 172.21.15.40:0/677780119 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74380a32b0 0x7f74381cb6c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:59.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.253+0000 7f743c83f640 1 --2- 172.21.15.40:0/677780119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f743809ee10 0x7f74381cb180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:59.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.253+0000 7f743c83f640 1 -- 172.21.15.40:0/677780119 >> 172.21.15.40:0/677780119 conn(0x7f74380a4ef0 msgr2=0x7f74380a1980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:13:59.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.253+0000 7f743c83f640 1 -- 172.21.15.40:0/677780119 shutdown_connections 2024-04-22T08:13:59.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.254+0000 7f743c83f640 1 -- 172.21.15.40:0/677780119 wait complete. 2024-04-22T08:13:59.263 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-22T08:13:59.263 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-22T08:13:59.264 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-22T08:13:59.264 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-22T08:13:59.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.365+0000 7f6dc091d640 10 monclient: get_monmap_and_config 2024-04-22T08:13:59.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.365+0000 7f6dc091d640 10 monclient: build_initial_monmap 2024-04-22T08:13:59.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.365+0000 7f6dc091d640 10 monclient: monmap: 2024-04-22T08:13:59.365 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:13:59.365 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:59.365 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:13:59.365812+0000 2024-04-22T08:13:59.365 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:13:59.365812+0000 2024-04-22T08:13:59.365 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:13:59.365 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:59.365 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:13:59.365 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:13:59.365 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:13:59.365 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:13:59.365 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:13:59.365 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:59.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.365+0000 7f6dc091d640 1 Processor -- start 2024-04-22T08:13:59.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.365+0000 7f6dc091d640 1 -- start start 2024-04-22T08:13:59.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.365+0000 7f6dc091d640 10 monclient: init 2024-04-22T08:13:59.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.366+0000 7f6dc091d640 10 monclient: _reopen_session rank -1 2024-04-22T08:13:59.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.366+0000 7f6dc091d640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-04-22T08:13:59.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.366+0000 7f6dc091d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6dbc0a1ba0 0x7f6dbc0a1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:59.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.366+0000 7f6dc091d640 10 monclient(hunting): picked mon.noname-b con 0x7f6dbc0a1ba0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:13:59.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.366+0000 7f6dc091d640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6dbc136230 0x7f6dbc0a2d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:59.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.366+0000 7f6dc091d640 10 monclient(hunting): picked mon.noname-c con 0x7f6dbc136230 addr v2:172.21.15.189:3300/0 2024-04-22T08:13:59.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.366+0000 7f6dc091d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6dbc136670 addr v1:172.21.15.40:6789/0 2024-04-22T08:13:59.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.366+0000 7f6dc091d640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dbc09fe80 con 0x7f6dbc136670 2024-04-22T08:13:59.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.366+0000 7f6dc091d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:59.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.366+0000 7f6dc091d640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6dbc0722a0 con 0x7f6dbc136230 2024-04-22T08:13:59.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.366+0000 7f6dc091d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:59.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.366+0000 7f6dc091d640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dbc0a4290 con 0x7f6dbc0a1ba0 2024-04-22T08:13:59.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.366+0000 7f6dc091d640 10 monclient(hunting): _renew_subs 2024-04-22T08:13:59.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.366+0000 7f6dc091d640 10 monclient(hunting): authenticate will time out at 913.360174s 2024-04-22T08:13:59.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.366+0000 7f6dbad9d640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f6dbc136670 0x7f6dbc0a32c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:51178/0 (socket says 172.21.15.40:51178) 2024-04-22T08:13:59.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.366+0000 7f6dbad9d640 1 -- 172.21.15.40:0/2149539824 learned_addr learned my addr 172.21.15.40:0/2149539824 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:13:59.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.366+0000 7f6dba59c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6dbc0a1ba0 0x7f6dbc0a1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:59.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.366+0000 7f6dba59c640 10 monclient(hunting): get_auth_request con 0x7f6dbc0a1ba0 auth_method 0 2024-04-22T08:13:59.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.366+0000 7f6dba59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:59.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.366+0000 7f6dba59c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:59.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.366+0000 7f6dba59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:59.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.366+0000 7f6dba59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:59.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.366+0000 7f6dba59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:59.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.366+0000 7f6dba59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:59.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.366+0000 7f6db9d9b640 1 --2- 172.21.15.40:0/2149539824 >> v2:172.21.15.189:3300/0 conn(0x7f6dbc136230 0x7f6dbc0a2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:59.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.366+0000 7f6dba59c640 10 monclient(hunting): handle_auth_done global_id 4523 payload 274 2024-04-22T08:13:59.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.366+0000 7f6dba59c640 1 -- 172.21.15.40:0/2149539824 >> v2:172.21.15.189:3300/0 conn(0x7f6dbc136230 msgr2=0x7f6dbc0a2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:13:59.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.366+0000 7f6dba59c640 1 --2- 172.21.15.40:0/2149539824 >> v2:172.21.15.189:3300/0 conn(0x7f6dbc136230 0x7f6dbc0a2d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:59.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.366+0000 7f6dba59c640 1 -- 172.21.15.40:0/2149539824 >> v1:172.21.15.40:6789/0 conn(0x7f6dbc136670 legacy=0x7f6dbc0a32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:13:59.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.366+0000 7f6dba59c640 10 monclient: _finish_hunting 0 2024-04-22T08:13:59.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.366+0000 7f6dba59c640 1 monclient: found mon.noname-b 2024-04-22T08:13:59.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.366+0000 7f6dba59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:13:59.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.366+0000 7f6dba59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:13:59.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.366+0000 7f6dba59c640 1 -- 172.21.15.40:0/2149539824 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dbc0a0160 con 0x7f6dbc0a1ba0 2024-04-22T08:13:59.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.366+0000 7f6db9d9b640 10 monclient: get_auth_request con 0x7f6dbc136230 auth_method 0 2024-04-22T08:13:59.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.366+0000 7f6db9d9b640 1 --2- 172.21.15.40:0/2149539824 >> v2:172.21.15.189:3300/0 conn(0x7f6dbc136230 0x7f6dbc0a2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T08:13:59.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.369+0000 7f6dba59c640 1 -- 172.21.15.40:0/2149539824 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6dbc0a1ba0 msgr2=0x7f6dbc0a1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:13:59.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.369+0000 7f6dba59c640 1 --2- 172.21.15.40:0/2149539824 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6dbc0a1ba0 0x7f6dbc0a1f80 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6db40097a0 tx=0x7f6db402b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c0bcd6ae588e96a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:59.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.369+0000 7f6db959a640 1 -- 172.21.15.40:0/2149539824 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db40319e0 con 0x7f6dbc0a1ba0 2024-04-22T08:13:59.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.370+0000 7f6db959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:59.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.370+0000 7f6db959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:13:59.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.370+0000 7f6db959a640 10 monclient: dump: 2024-04-22T08:13:59.378 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:59.378 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:59.378 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.378 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.378 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:59.378 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:59.378 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:59.379 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:59.379 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:59.379 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:59.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.370+0000 7f6db959a640 10 monclient: _finish_auth 0 2024-04-22T08:13:59.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.370+0000 7f6db959a640 10 monclient: _check_auth_tickets 2024-04-22T08:13:59.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.370+0000 7f6db959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:13:59.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.370+0000 7f6db959a640 1 -- 172.21.15.40:0/2149539824 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6db4031b40 con 0x7f6dbc0a1ba0 2024-04-22T08:13:59.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.370+0000 7f6dc091d640 5 monclient: authenticate success, global_id 4523 2024-04-22T08:13:59.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.370+0000 7f6dc091d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:13:59.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.370+0000 7f6db959a640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:13:59.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.370+0000 7f6db959a640 1 -- 172.21.15.40:0/2149539824 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db4034480 con 0x7f6dbc0a1ba0 2024-04-22T08:13:59.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.370+0000 7f6dc091d640 10 monclient: get_monmap_and_config success 2024-04-22T08:13:59.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.370+0000 7f6dc091d640 4 set_mon_vals no callback set 2024-04-22T08:13:59.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.370+0000 7f6db959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:59.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.370+0000 7f6db959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:13:59.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.370+0000 7f6db959a640 10 monclient: dump: 2024-04-22T08:13:59.380 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:59.380 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:59.380 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.380 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.380 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:59.380 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:59.380 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:59.380 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:59.380 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:59.380 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:59.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.370+0000 7f6dc091d640 10 monclient: shutdown 2024-04-22T08:13:59.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.370+0000 7f6dc091d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:13:59.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.370+0000 7f6dc091d640 1 -- 172.21.15.40:0/2149539824 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6dbc0a1ba0 msgr2=0x7f6dbc0a1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:59.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.370+0000 7f6dc091d640 1 --2- 172.21.15.40:0/2149539824 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6dbc0a1ba0 0x7f6dbc0a1f80 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6db40097a0 tx=0x7f6db402b790 comp rx=0 tx=0).stop 2024-04-22T08:13:59.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.370+0000 7f6dc091d640 1 -- 172.21.15.40:0/2149539824 shutdown_connections 2024-04-22T08:13:59.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.370+0000 7f6dc091d640 1 --2- 172.21.15.40:0/2149539824 >> v2:172.21.15.189:3300/0 conn(0x7f6dbc136230 0x7f6dbc0a2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:59.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.370+0000 7f6dc091d640 1 --2- 172.21.15.40:0/2149539824 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6dbc0a1ba0 0x7f6dbc0a1f80 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:59.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.370+0000 7f6dc091d640 1 -- 172.21.15.40:0/2149539824 >> 172.21.15.40:0/2149539824 conn(0x7f6dbc0a4e80 msgr2=0x7f6dbc0a5290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:13:59.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.370+0000 7f6dc091d640 1 -- 172.21.15.40:0/2149539824 shutdown_connections 2024-04-22T08:13:59.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.370+0000 7f6dc091d640 1 -- 172.21.15.40:0/2149539824 wait complete. 2024-04-22T08:13:59.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.371+0000 7f6dc091d640 10 monclient: build_initial_monmap 2024-04-22T08:13:59.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.371+0000 7f6dc091d640 10 monclient: monmap: 2024-04-22T08:13:59.381 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:13:59.381 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:13:59.381 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:13:59.381 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:13:59.382 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:13:59.382 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:59.382 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:13:59.382 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:13:59.382 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:13:59.382 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:59.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.371+0000 7f6dc091d640 1 Processor -- start 2024-04-22T08:13:59.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.371+0000 7f6dc091d640 1 -- start start 2024-04-22T08:13:59.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.371+0000 7f6dc091d640 10 monclient: init 2024-04-22T08:13:59.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6dc091d640 10 monclient: _reopen_session rank -1 2024-04-22T08:13:59.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6dc091d640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-22T08:13:59.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6dc091d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6dbc0a1ba0 0x7f6dbc1cac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:59.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6dc091d640 10 monclient(hunting): picked mon.noname-b con 0x7f6dbc0a1ba0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:13:59.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6dc091d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dbc136230 0x7f6dbc1cb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:59.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6dc091d640 10 monclient(hunting): picked mon.noname-a con 0x7f6dbc136230 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:13:59.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6dc091d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6dbc136670 0x7f6dbc1221d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:59.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6dc091d640 10 monclient(hunting): picked mon.noname-c con 0x7f6dbc136670 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:13:59.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6dc091d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:59.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6dc091d640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dbc125810 con 0x7f6dbc136230 2024-04-22T08:13:59.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6dc091d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:59.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6dc091d640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dbc124920 con 0x7f6dbc0a1ba0 2024-04-22T08:13:59.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6dc091d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:59.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6dc091d640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dbc0a2710 con 0x7f6dbc136670 2024-04-22T08:13:59.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6dc091d640 10 monclient(hunting): _renew_subs 2024-04-22T08:13:59.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6dc091d640 10 monclient(hunting): authenticate will time out at 913.366089s 2024-04-22T08:13:59.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6db9d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dbc136230 0x7f6dbc1cb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:59.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6dbad9d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6dbc136670 0x7f6dbc1221d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:59.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6db9d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dbc136230 0x7f6dbc1cb1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56020/0 (socket says 172.21.15.40:56020) 2024-04-22T08:13:59.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6dbad9d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6dbc136670 0x7f6dbc1221d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:51518/0 (socket says 172.21.15.40:51518) 2024-04-22T08:13:59.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6db9d9b640 1 -- 172.21.15.40:0/1660787356 learned_addr learned my addr 172.21.15.40:0/1660787356 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:13:59.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6db9d9b640 10 monclient(hunting): get_auth_request con 0x7f6dbc136230 auth_method 0 2024-04-22T08:13:59.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6dba59c640 1 --2- 172.21.15.40:0/1660787356 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6dbc0a1ba0 0x7f6dbc1cac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:59.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6db9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:59.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6db9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:59.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6db9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:59.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6dba59c640 10 monclient(hunting): get_auth_request con 0x7f6dbc0a1ba0 auth_method 0 2024-04-22T08:13:59.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6dba59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:59.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6dba59c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:59.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6dba59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:59.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6dbad9d640 10 monclient(hunting): get_auth_request con 0x7f6dbc136670 auth_method 0 2024-04-22T08:13:59.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6dbad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:59.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6dbad9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:59.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6dbad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:59.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6db9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:59.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6db9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:59.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6db9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:59.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6dbad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:59.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6dbad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:59.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6dbad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:59.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6dba59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:59.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6dba59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:59.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6dba59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:59.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6db9d9b640 10 monclient(hunting): handle_auth_done global_id 4557 payload 824 2024-04-22T08:13:59.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6db9d9b640 1 -- 172.21.15.40:0/1660787356 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6dbc136670 msgr2=0x7f6dbc1221d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:59.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6db9d9b640 1 --2- 172.21.15.40:0/1660787356 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6dbc136670 0x7f6dbc1221d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:59.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6db9d9b640 1 -- 172.21.15.40:0/1660787356 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6dbc0a1ba0 msgr2=0x7f6dbc1cac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:59.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6db9d9b640 1 --2- 172.21.15.40:0/1660787356 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6dbc0a1ba0 0x7f6dbc1cac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:59.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6db9d9b640 10 monclient: _finish_hunting 0 2024-04-22T08:13:59.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6db9d9b640 1 monclient: found mon.noname-a 2024-04-22T08:13:59.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6db9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:13:59.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6db9d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:13:59.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6db9d9b640 1 -- 172.21.15.40:0/1660787356 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db4009400 con 0x7f6dbc136230 2024-04-22T08:13:59.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6dbad9d640 1 --2- 172.21.15.40:0/1660787356 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6dbc136670 0x7f6dbc1221d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:13:59.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.372+0000 7f6dba59c640 1 --2- 172.21.15.40:0/1660787356 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6dbc0a1ba0 0x7f6dbc1cac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:13:59.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.375+0000 7f6db9d9b640 1 -- 172.21.15.40:0/1660787356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dbc136230 msgr2=0x7f6dbc1cb1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:13:59.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.375+0000 7f6db9d9b640 1 --2- 172.21.15.40:0/1660787356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dbc136230 0x7f6dbc1cb1a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6db000c820 tx=0x7f6db000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:59.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.375+0000 7f6daaffd640 1 -- 172.21.15.40:0/1660787356 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db0007c40 con 0x7f6dbc136230 2024-04-22T08:13:59.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.375+0000 7f6daaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:59.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.375+0000 7f6daaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:13:59.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.375+0000 7f6daaffd640 10 monclient: dump: 2024-04-22T08:13:59.386 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:59.387 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:59.387 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.387 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.387 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:59.387 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:59.387 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:59.387 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:59.387 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:59.387 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:59.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.375+0000 7f6daaffd640 10 monclient: _finish_auth 0 2024-04-22T08:13:59.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.375+0000 7f6daaffd640 10 monclient: _check_auth_tickets 2024-04-22T08:13:59.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.375+0000 7f6daaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:13:59.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.375+0000 7f6daaffd640 1 -- 172.21.15.40:0/1660787356 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6db0007da0 con 0x7f6dbc136230 2024-04-22T08:13:59.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.375+0000 7f6dc091d640 5 monclient: authenticate success, global_id 4557 2024-04-22T08:13:59.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.375+0000 7f6dc091d640 10 monclient: _renew_subs 2024-04-22T08:13:59.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.375+0000 7f6dc091d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:13:59.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.375+0000 7f6dc091d640 1 -- 172.21.15.40:0/1660787356 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dbc0716c0 con 0x7f6dbc136230 2024-04-22T08:13:59.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.375+0000 7f6daaffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:13:59.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.375+0000 7f6daaffd640 1 -- 172.21.15.40:0/1660787356 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db0011610 con 0x7f6dbc136230 2024-04-22T08:13:59.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.375+0000 7f6daaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:59.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.375+0000 7f6daaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:13:59.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.375+0000 7f6daaffd640 10 monclient: dump: 2024-04-22T08:13:59.388 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:59.388 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:59.388 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.388 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.388 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:59.388 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:59.388 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:59.389 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:59.389 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:59.389 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:59.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.375+0000 7f6db8d99640 4 set_mon_vals no callback set 2024-04-22T08:13:59.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.375+0000 7f6dc091d640 10 monclient: _renew_subs 2024-04-22T08:13:59.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.375+0000 7f6dc091d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:13:59.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.375+0000 7f6dc091d640 1 -- 172.21.15.40:0/1660787356 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dbc123ce0 con 0x7f6dbc136230 2024-04-22T08:13:59.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.376+0000 7f6dc091d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:13:59.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.376+0000 7f6dc091d640 1 --2- 172.21.15.40:0/1660787356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dbc1264d0 0x7f6dbc122710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:59.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.376+0000 7f6dc091d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:59.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.376+0000 7f6dc091d640 1 -- 172.21.15.40:0/1660787356 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dbc066650 con 0x7f6dbc1264d0 2024-04-22T08:13:59.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.376+0000 7f6daaffd640 1 -- 172.21.15.40:0/1660787356 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f6db0011770 con 0x7f6dbc136230 2024-04-22T08:13:59.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.376+0000 7f6dba59c640 1 --2- 172.21.15.40:0/1660787356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dbc1264d0 0x7f6dbc122710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:59.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.376+0000 7f6dba59c640 10 monclient: get_auth_request con 0x7f6dbc1264d0 auth_method 0 2024-04-22T08:13:59.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.376+0000 7f6dba59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:59.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.376+0000 7f6dba59c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:59.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.376+0000 7f6dba59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:59.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.376+0000 7f6dba59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:59.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.376+0000 7f6dba59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:59.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.376+0000 7f6dba59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:59.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.376+0000 7f6daaffd640 1 --2- 172.21.15.40:0/1660787356 >> v2:172.21.15.189:6800/30168 conn(0x7f6d9c037820 0x7f6d9c039ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:59.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.376+0000 7f6daaffd640 1 -- 172.21.15.40:0/1660787356 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f6db004a4d0 con 0x7f6dbc136230 2024-04-22T08:13:59.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.376+0000 7f6dba59c640 10 monclient(hunting): handle_auth_done global_id 4560 payload 824 2024-04-22T08:13:59.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.376+0000 7f6dba59c640 1 -- 172.21.15.40:0/1660787356 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6dbc0a2440 con 0x7f6dbc1264d0 2024-04-22T08:13:59.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.376+0000 7f6dbad9d640 1 --2- 172.21.15.40:0/1660787356 >> v2:172.21.15.189:6800/30168 conn(0x7f6d9c037820 0x7f6d9c039ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:59.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.376+0000 7f6dbad9d640 10 monclient: get_auth_request con 0x7f6d9c037820 auth_method 0 2024-04-22T08:13:59.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.378+0000 7f6dba59c640 1 -- 172.21.15.40:0/1660787356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dbc1264d0 msgr2=0x7f6dbc122710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:13:59.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.378+0000 7f6dba59c640 1 --2- 172.21.15.40:0/1660787356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dbc1264d0 0x7f6dbc122710 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6db4036000 tx=0x7f6db40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:59.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.378+0000 7f6daaffd640 1 -- 172.21.15.40:0/1660787356 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db4048650 con 0x7f6dbc1264d0 2024-04-22T08:13:59.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.378+0000 7f6daaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:59.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.378+0000 7f6daaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:13:59.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.378+0000 7f6daaffd640 10 monclient: dump: 2024-04-22T08:13:59.393 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:59.393 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:59.393 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.393 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.393 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:59.393 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:59.393 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:59.393 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:59.394 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:59.394 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:59.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.379+0000 7f6dbad9d640 1 --2- 172.21.15.40:0/1660787356 >> v2:172.21.15.189:6800/30168 conn(0x7f6d9c037820 0x7f6d9c039ce0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6dac005c00 tx=0x7f6dac009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:59.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.379+0000 7f6daaffd640 1 -- 172.21.15.40:0/1660787356 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f6db4042070 con 0x7f6dbc1264d0 2024-04-22T08:13:59.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.379+0000 7f6daaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:13:59.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.379+0000 7f6daaffd640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:13:59.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.379+0000 7f6daaffd640 1 -- 172.21.15.40:0/1660787356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dbc1264d0 msgr2=0x7f6dbc122710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:59.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.379+0000 7f6daaffd640 1 --2- 172.21.15.40:0/1660787356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dbc1264d0 0x7f6dbc122710 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6db4036000 tx=0x7f6db40046c0 comp rx=0 tx=0).stop 2024-04-22T08:13:59.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.379+0000 7f6daaffd640 1 -- 172.21.15.40:0/1660787356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dbc1264d0 msgr2=0x7f6dbc122710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:13:59.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.379+0000 7f6daaffd640 1 --2- 172.21.15.40:0/1660787356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dbc1264d0 0x7f6dbc122710 secure :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6db4036000 tx=0x7f6db40046c0 comp rx=0 tx=0).stop 2024-04-22T08:13:59.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.385+0000 7f6dc091d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:13:59.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.385+0000 7f6dc091d640 1 --2- 172.21.15.40:0/1660787356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dbc125a30 0x7f6dbc122c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:59.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.385+0000 7f6dc091d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:59.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.385+0000 7f6dc091d640 1 -- 172.21.15.40:0/1660787356 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dbc04e9b0 con 0x7f6dbc125a30 2024-04-22T08:13:59.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.385+0000 7f6dba59c640 1 --2- 172.21.15.40:0/1660787356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dbc125a30 0x7f6dbc122c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:59.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.385+0000 7f6dba59c640 10 monclient: get_auth_request con 0x7f6dbc125a30 auth_method 0 2024-04-22T08:13:59.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.385+0000 7f6dba59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:59.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.385+0000 7f6dba59c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:59.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.385+0000 7f6dba59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:59.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.385+0000 7f6dba59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:59.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.385+0000 7f6dba59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:59.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.385+0000 7f6dba59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:59.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.385+0000 7f6dba59c640 10 monclient(hunting): handle_auth_done global_id 4563 payload 824 2024-04-22T08:13:59.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.385+0000 7f6dba59c640 1 -- 172.21.15.40:0/1660787356 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6dbc09f080 con 0x7f6dbc125a30 2024-04-22T08:13:59.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.388+0000 7f6dba59c640 1 -- 172.21.15.40:0/1660787356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dbc125a30 msgr2=0x7f6dbc122c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:13:59.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.388+0000 7f6dba59c640 1 --2- 172.21.15.40:0/1660787356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dbc125a30 0x7f6dbc122c50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6db402b7d0 tx=0x7f6db4031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:59.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.388+0000 7f6daaffd640 1 -- 172.21.15.40:0/1660787356 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db4003820 con 0x7f6dbc125a30 2024-04-22T08:13:59.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.388+0000 7f6daaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:59.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.388+0000 7f6daaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:13:59.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.388+0000 7f6daaffd640 10 monclient: dump: 2024-04-22T08:13:59.396 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:59.396 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:59.396 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.396 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.396 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:59.396 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:59.396 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:59.396 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:59.396 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:59.396 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:59.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.388+0000 7f6daaffd640 1 -- 172.21.15.40:0/1660787356 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4803 (secure 0 0 0) 0x7f6db4003980 con 0x7f6dbc125a30 2024-04-22T08:13:59.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.388+0000 7f6daaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:13:59.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.388+0000 7f6daaffd640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:13:59.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.388+0000 7f6daaffd640 1 -- 172.21.15.40:0/1660787356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dbc125a30 msgr2=0x7f6dbc122c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:59.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.388+0000 7f6daaffd640 1 --2- 172.21.15.40:0/1660787356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dbc125a30 0x7f6dbc122c50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6db402b7d0 tx=0x7f6db4031000 comp rx=0 tx=0).stop 2024-04-22T08:13:59.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.388+0000 7f6daaffd640 1 -- 172.21.15.40:0/1660787356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dbc125a30 msgr2=0x7f6dbc122c50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:13:59.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.388+0000 7f6daaffd640 1 --2- 172.21.15.40:0/1660787356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dbc125a30 0x7f6dbc122c50 secure :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6db402b7d0 tx=0x7f6db4031000 comp rx=0 tx=0).stop 2024-04-22T08:13:59.397 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:13:59.397 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:13:59.397 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:13:59.397 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "leader", 2024-04-22T08:13:59.397 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 20, 2024-04-22T08:13:59.397 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:13:59.398 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:13:59.398 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:13:59.398 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:13:59.398 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:13:59.398 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 2, 2024-04-22T08:13:59.398 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:13:59.398 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:13:59.398 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:13:59.398 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:13:59.398 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:13:59.398 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:13:59.398 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:13:59.398 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:13:59.399 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:13:59.399 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:13:59.399 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:13:59.399 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:13:59.399 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:13:59.399 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:13:59.399 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:13:59.399 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:13:59.399 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:13:59.399 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:13:59.399 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:13:59.399 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:13:59.399 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:13:59.399 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:13:59.400 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:13:59.400 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:13:59.400 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:13:59.400 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:13:59.400 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:59.400 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:59.400 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:13:59.400 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:13:59.400 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:13:59.400 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:13:59.400 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:13:59.400 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:13:59.400 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:13:59.400 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:13:59.401 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:13:59.401 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:13:59.401 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:13:59.401 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:13:59.401 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:13:59.401 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:13:59.401 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:13:59.401 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:13:59.401 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:13:59.401 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:13:59.401 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:13:59.401 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:13:59.401 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:13:59.401 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:13:59.402 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:13:59.402 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:13:59.402 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:13:59.402 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:13:59.402 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:13:59.402 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:13:59.402 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:13:59.402 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:59.402 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:13:59.402 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:59.402 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:13:59.402 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:13:59.402 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:13:59.403 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:13:59.403 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:59.403 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:13:59.403 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:13:59.403 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:59.403 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:59.403 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:59.403 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:13:59.403 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:13:59.403 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:59.403 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:59.403 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:59.403 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:59.403 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:13:59.403 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:13:59.404 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:13:59.404 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:13:59.404 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:13:59.404 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:59.404 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:59.404 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:13:59.404 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:13:59.404 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:13:59.404 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:13:59.404 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:59.405 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:13:59.405 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:13:59.405 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:59.405 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:59.405 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:59.405 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:13:59.405 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:13:59.405 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:59.405 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:59.405 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:59.405 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:59.405 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:13:59.405 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:13:59.405 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:13:59.406 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:13:59.406 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:13:59.406 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:59.406 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:59.406 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:13:59.406 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:13:59.406 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:13:59.406 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:13:59.406 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:59.406 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:13:59.406 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:13:59.406 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:59.406 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:59.406 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:59.407 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:13:59.407 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:13:59.407 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:59.407 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:59.407 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:59.407 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:59.407 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:13:59.407 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:13:59.407 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:13:59.407 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:13:59.407 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:13:59.407 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:59.407 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:59.407 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:59.407 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:13:59.408 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:13:59.408 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:59.408 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:13:59.408 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:13:59.408 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:13:59.408 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:59.408 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:13:59.408 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:13:59.408 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:59.408 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:13:59.408 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:13:59.408 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:13:59.408 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:59.408 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:13:59.408 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:13:59.409 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:59.409 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:13:59.409 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:13:59.409 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 4 2024-04-22T08:13:59.409 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:59.409 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:13:59.409 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": [ 2024-04-22T08:13:59.409 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:59.409 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:13:59.409 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:13:59.409 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:13:59.409 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:59.409 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:59.409 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:59.409 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:13:59.410 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:13:59.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.388+0000 7f6dc091d640 1 -- 172.21.15.40:0/1660787356 >> v2:172.21.15.189:6800/30168 conn(0x7f6d9c037820 msgr2=0x7f6d9c039ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:59.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.388+0000 7f6dc091d640 1 --2- 172.21.15.40:0/1660787356 >> v2:172.21.15.189:6800/30168 conn(0x7f6d9c037820 0x7f6d9c039ce0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6dac005c00 tx=0x7f6dac009040 comp rx=0 tx=0).stop 2024-04-22T08:13:59.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.388+0000 7f6dc091d640 10 monclient: shutdown 2024-04-22T08:13:59.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.388+0000 7f6dc091d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:13:59.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.388+0000 7f6dc091d640 1 -- 172.21.15.40:0/1660787356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dbc136230 msgr2=0x7f6dbc1cb1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:59.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.388+0000 7f6dc091d640 1 --2- 172.21.15.40:0/1660787356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dbc136230 0x7f6dbc1cb1a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6db000c820 tx=0x7f6db000cc20 comp rx=0 tx=0).stop 2024-04-22T08:13:59.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.388+0000 7f6dbad9d640 1 -- 172.21.15.40:0/1660787356 reap_dead start 2024-04-22T08:13:59.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.388+0000 7f6dc091d640 1 -- 172.21.15.40:0/1660787356 shutdown_connections 2024-04-22T08:13:59.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.388+0000 7f6dc091d640 1 --2- 172.21.15.40:0/1660787356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dbc136230 0x7f6dbc1cb1a0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:59.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.388+0000 7f6dc091d640 1 -- 172.21.15.40:0/1660787356 >> 172.21.15.40:0/1660787356 conn(0x7f6dbc0a4e80 msgr2=0x7f6dbc0a08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:13:59.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.388+0000 7f6dc091d640 1 -- 172.21.15.40:0/1660787356 shutdown_connections 2024-04-22T08:13:59.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.388+0000 7f6dc091d640 1 -- 172.21.15.40:0/1660787356 wait complete. 2024-04-22T08:13:59.412 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:13:59.412 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-22T08:13:59.412 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-22T08:13:59.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.513+0000 7f30ce84a640 10 monclient: get_monmap_and_config 2024-04-22T08:13:59.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.513+0000 7f30ce84a640 10 monclient: build_initial_monmap 2024-04-22T08:13:59.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.513+0000 7f30ce84a640 10 monclient: monmap: 2024-04-22T08:13:59.513 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:13:59.513 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:59.513 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:13:59.513927+0000 2024-04-22T08:13:59.513 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:13:59.513927+0000 2024-04-22T08:13:59.513 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:13:59.513 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:59.513 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:13:59.513 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:13:59.513 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:13:59.513 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:13:59.513 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:13:59.513 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:59.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.513+0000 7f30ce84a640 1 Processor -- start 2024-04-22T08:13:59.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.513+0000 7f30ce84a640 1 -- start start 2024-04-22T08:13:59.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.513+0000 7f30ce84a640 10 monclient: init 2024-04-22T08:13:59.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.514+0000 7f30ce84a640 10 monclient: _reopen_session rank -1 2024-04-22T08:13:59.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.514+0000 7f30ce84a640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-04-22T08:13:59.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.514+0000 7f30ce84a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f30c80a1c60 addr v1:172.21.15.40:6789/0 2024-04-22T08:13:59.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.514+0000 7f30ce84a640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f30c80a31d0 0x7f30c813aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:59.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.514+0000 7f30ce84a640 10 monclient(hunting): picked mon.noname-c con 0x7f30c80a31d0 addr v2:172.21.15.189:3300/0 2024-04-22T08:13:59.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.514+0000 7f30ce84a640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30c813b410 0x7f30c813d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:59.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.514+0000 7f30ce84a640 10 monclient(hunting): picked mon.noname-b con 0x7f30c813b410 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:13:59.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.514+0000 7f30ce84a640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c80a41f0 con 0x7f30c80a1c60 2024-04-22T08:13:59.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.514+0000 7f30ce84a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:59.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.514+0000 7f30ccde7640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30c813b410 0x7f30c813d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:59.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.514+0000 7f30c7fff640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f30c80a1c60 0x7f30c80a2040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:51182/0 (socket says 172.21.15.40:51182) 2024-04-22T08:13:59.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.514+0000 7f30ce84a640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30c80a3660 con 0x7f30c80a31d0 2024-04-22T08:13:59.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.514+0000 7f30ce84a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:59.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.514+0000 7f30c7fff640 1 -- 172.21.15.40:0/443222796 learned_addr learned my addr 172.21.15.40:0/443222796 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:13:59.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.514+0000 7f30ce84a640 1 -- 172.21.15.40:0/443222796 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30c809fe90 con 0x7f30c813b410 2024-04-22T08:13:59.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.514+0000 7f30ce84a640 10 monclient(hunting): _renew_subs 2024-04-22T08:13:59.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.514+0000 7f30ce84a640 10 monclient(hunting): authenticate will time out at 913.508390s 2024-04-22T08:13:59.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.514+0000 7f30ccde7640 10 monclient(hunting): get_auth_request con 0x7f30c813b410 auth_method 0 2024-04-22T08:13:59.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.514+0000 7f30ccde7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:59.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.514+0000 7f30ccde7640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:59.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.514+0000 7f30ccde7640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:59.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.514+0000 7f30c77fe640 1 --2- 172.21.15.40:0/443222796 >> v2:172.21.15.189:3300/0 conn(0x7f30c80a31d0 0x7f30c813aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:59.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.514+0000 7f30c77fe640 10 monclient(hunting): get_auth_request con 0x7f30c80a31d0 auth_method 0 2024-04-22T08:13:59.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.514+0000 7f30c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:59.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.514+0000 7f30c77fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:59.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.514+0000 7f30c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:59.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.514+0000 7f30ccde7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:59.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.514+0000 7f30ccde7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:59.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.514+0000 7f30ccde7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:59.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.514+0000 7f30c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:59.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.514+0000 7f30c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:59.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.514+0000 7f30c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:59.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.514+0000 7f30ccde7640 10 monclient(hunting): handle_auth_done global_id 4529 payload 274 2024-04-22T08:13:59.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.514+0000 7f30ccde7640 1 -- 172.21.15.40:0/443222796 >> v2:172.21.15.189:3300/0 conn(0x7f30c80a31d0 msgr2=0x7f30c813aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:13:59.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.514+0000 7f30ccde7640 1 --2- 172.21.15.40:0/443222796 >> v2:172.21.15.189:3300/0 conn(0x7f30c80a31d0 0x7f30c813aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:59.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.514+0000 7f30ccde7640 1 -- 172.21.15.40:0/443222796 >> v1:172.21.15.40:6789/0 conn(0x7f30c80a1c60 legacy=0x7f30c80a2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:13:59.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.514+0000 7f30ccde7640 10 monclient: _finish_hunting 0 2024-04-22T08:13:59.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.514+0000 7f30ccde7640 1 monclient: found mon.noname-b 2024-04-22T08:13:59.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.514+0000 7f30ccde7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:13:59.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.514+0000 7f30ccde7640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:13:59.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.514+0000 7f30ccde7640 1 -- 172.21.15.40:0/443222796 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c80a2750 con 0x7f30c813b410 2024-04-22T08:13:59.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.517+0000 7f30ccde7640 1 -- 172.21.15.40:0/443222796 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30c813b410 msgr2=0x7f30c813d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:13:59.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.517+0000 7f30ccde7640 1 --2- 172.21.15.40:0/443222796 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30c813b410 0x7f30c813d800 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f30c0009d90 tx=0x7f30c002bcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79e2c39545a3cefc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:59.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.518+0000 7f30c6ffd640 1 -- 172.21.15.40:0/443222796 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c002ccb0 con 0x7f30c813b410 2024-04-22T08:13:59.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.518+0000 7f30c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:59.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.518+0000 7f30c6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:13:59.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.518+0000 7f30c6ffd640 10 monclient: dump: 2024-04-22T08:13:59.546 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:59.546 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:59.546 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.546 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.546 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:59.546 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:59.546 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:59.546 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:59.546 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:59.546 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:59.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.518+0000 7f30c6ffd640 10 monclient: _finish_auth 0 2024-04-22T08:13:59.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.518+0000 7f30c6ffd640 10 monclient: _check_auth_tickets 2024-04-22T08:13:59.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.518+0000 7f30c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:13:59.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.518+0000 7f30c6ffd640 1 -- 172.21.15.40:0/443222796 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30c002ce10 con 0x7f30c813b410 2024-04-22T08:13:59.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.518+0000 7f30ce84a640 5 monclient: authenticate success, global_id 4529 2024-04-22T08:13:59.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.518+0000 7f30ce84a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:13:59.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.518+0000 7f30c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:13:59.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.518+0000 7f30ce84a640 10 monclient: get_monmap_and_config success 2024-04-22T08:13:59.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.518+0000 7f30c6ffd640 1 -- 172.21.15.40:0/443222796 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c0034660 con 0x7f30c813b410 2024-04-22T08:13:59.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.518+0000 7f30ce84a640 4 set_mon_vals no callback set 2024-04-22T08:13:59.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.518+0000 7f30c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:59.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.518+0000 7f30c6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:13:59.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.518+0000 7f30c6ffd640 10 monclient: dump: 2024-04-22T08:13:59.547 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:59.547 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:59.547 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.548 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.548 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:59.548 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:59.548 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:59.548 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:59.548 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:59.548 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:59.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.518+0000 7f30ce84a640 10 monclient: shutdown 2024-04-22T08:13:59.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.518+0000 7f30ce84a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:13:59.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.518+0000 7f30ce84a640 1 -- 172.21.15.40:0/443222796 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30c813b410 msgr2=0x7f30c813d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:59.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.518+0000 7f30ce84a640 1 --2- 172.21.15.40:0/443222796 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30c813b410 0x7f30c813d800 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f30c0009d90 tx=0x7f30c002bcf0 comp rx=0 tx=0).stop 2024-04-22T08:13:59.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.518+0000 7f30ce84a640 1 -- 172.21.15.40:0/443222796 shutdown_connections 2024-04-22T08:13:59.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.518+0000 7f30ce84a640 1 --2- 172.21.15.40:0/443222796 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30c813b410 0x7f30c813d800 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:59.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.518+0000 7f30ce84a640 1 --2- 172.21.15.40:0/443222796 >> v2:172.21.15.189:3300/0 conn(0x7f30c80a31d0 0x7f30c813aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:59.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.518+0000 7f30ce84a640 1 -- 172.21.15.40:0/443222796 >> 172.21.15.40:0/443222796 conn(0x7f30c80a89a0 msgr2=0x7f30c80a4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:13:59.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.518+0000 7f30ce84a640 1 -- 172.21.15.40:0/443222796 shutdown_connections 2024-04-22T08:13:59.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.518+0000 7f30ce84a640 1 -- 172.21.15.40:0/443222796 wait complete. 2024-04-22T08:13:59.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.519+0000 7f30ce84a640 10 monclient: build_initial_monmap 2024-04-22T08:13:59.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.519+0000 7f30ce84a640 10 monclient: monmap: 2024-04-22T08:13:59.549 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:13:59.549 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:13:59.549 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:13:59.549 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:13:59.549 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:13:59.549 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:59.549 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:13:59.550 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:13:59.550 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:13:59.550 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:59.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.519+0000 7f30ce84a640 1 Processor -- start 2024-04-22T08:13:59.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.519+0000 7f30ce84a640 1 -- start start 2024-04-22T08:13:59.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.519+0000 7f30ce84a640 10 monclient: init 2024-04-22T08:13:59.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30ce84a640 10 monclient: _reopen_session rank -1 2024-04-22T08:13:59.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30ce84a640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:13:59.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30ce84a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30c80a1c60 0x7f30c81cae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:59.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30ce84a640 10 monclient(hunting): picked mon.noname-a con 0x7f30c80a1c60 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:13:59.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30ce84a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30c80a31d0 0x7f30c81cb3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:59.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30ce84a640 10 monclient(hunting): picked mon.noname-b con 0x7f30c80a31d0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:13:59.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30ce84a640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30c813b410 0x7f30c81336f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:59.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30ce84a640 10 monclient(hunting): picked mon.noname-c con 0x7f30c813b410 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:13:59.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30ce84a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:59.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30ce84a640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c8137f00 con 0x7f30c80a1c60 2024-04-22T08:13:59.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30ce84a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:59.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30ce84a640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c80a4790 con 0x7f30c80a31d0 2024-04-22T08:13:59.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30ce84a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:59.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30ce84a640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30c80a6110 con 0x7f30c813b410 2024-04-22T08:13:59.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30ce84a640 10 monclient(hunting): _renew_subs 2024-04-22T08:13:59.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30ce84a640 10 monclient(hunting): authenticate will time out at 913.514052s 2024-04-22T08:13:59.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30c7fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30c80a1c60 0x7f30c81cae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:59.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30c7fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30c80a1c60 0x7f30c81cae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56054/0 (socket says 172.21.15.40:56054) 2024-04-22T08:13:59.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30c7fff640 1 -- 172.21.15.40:0/2695833017 learned_addr learned my addr 172.21.15.40:0/2695833017 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:13:59.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30c7fff640 10 monclient(hunting): get_auth_request con 0x7f30c80a1c60 auth_method 0 2024-04-22T08:13:59.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:59.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30c7fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:59.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:59.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30ccde7640 1 --2- 172.21.15.40:0/2695833017 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30c813b410 0x7f30c81336f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:59.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30ccde7640 10 monclient(hunting): get_auth_request con 0x7f30c813b410 auth_method 0 2024-04-22T08:13:59.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30ccde7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:59.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30ccde7640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:59.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30ccde7640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:59.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:59.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:59.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:59.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30c77fe640 1 --2- 172.21.15.40:0/2695833017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30c80a31d0 0x7f30c81cb3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:59.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30ccde7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:59.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30ccde7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:59.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30ccde7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:59.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30c77fe640 10 monclient(hunting): get_auth_request con 0x7f30c80a31d0 auth_method 0 2024-04-22T08:13:59.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:59.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30c77fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:59.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:59.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:59.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:59.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:59.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30c7fff640 10 monclient(hunting): handle_auth_done global_id 4566 payload 824 2024-04-22T08:13:59.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30c7fff640 1 -- 172.21.15.40:0/2695833017 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30c813b410 msgr2=0x7f30c81336f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:59.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30c7fff640 1 --2- 172.21.15.40:0/2695833017 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30c813b410 0x7f30c81336f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:59.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30c7fff640 1 -- 172.21.15.40:0/2695833017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30c80a31d0 msgr2=0x7f30c81cb3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:59.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30c7fff640 1 --2- 172.21.15.40:0/2695833017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30c80a31d0 0x7f30c81cb3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:59.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30c7fff640 10 monclient: _finish_hunting 0 2024-04-22T08:13:59.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30c7fff640 1 monclient: found mon.noname-a 2024-04-22T08:13:59.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:13:59.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30c7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:13:59.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30c7fff640 1 -- 172.21.15.40:0/2695833017 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c0009a70 con 0x7f30c80a1c60 2024-04-22T08:13:59.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.520+0000 7f30ccde7640 1 --2- 172.21.15.40:0/2695833017 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30c813b410 0x7f30c81336f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:13:59.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.522+0000 7f30c7fff640 1 -- 172.21.15.40:0/2695833017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30c80a1c60 msgr2=0x7f30c81cae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:13:59.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.522+0000 7f30c7fff640 1 --2- 172.21.15.40:0/2695833017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30c80a1c60 0x7f30c81cae60 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f30bc00c460 tx=0x7f30bc00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:59.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.523+0000 7f30c4ff9640 1 -- 172.21.15.40:0/2695833017 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30bc007070 con 0x7f30c80a1c60 2024-04-22T08:13:59.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.523+0000 7f30c4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:59.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.523+0000 7f30c4ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:13:59.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.523+0000 7f30c4ff9640 10 monclient: dump: 2024-04-22T08:13:59.555 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:59.555 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:59.555 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.555 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.555 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:59.555 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:59.555 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:59.555 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:59.556 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:59.556 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:59.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.523+0000 7f30c4ff9640 10 monclient: _finish_auth 0 2024-04-22T08:13:59.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.523+0000 7f30c4ff9640 10 monclient: _check_auth_tickets 2024-04-22T08:13:59.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.523+0000 7f30c4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:13:59.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.523+0000 7f30c4ff9640 1 -- 172.21.15.40:0/2695833017 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30bc0071f0 con 0x7f30c80a1c60 2024-04-22T08:13:59.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.523+0000 7f30ce84a640 5 monclient: authenticate success, global_id 4566 2024-04-22T08:13:59.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.523+0000 7f30ce84a640 10 monclient: _renew_subs 2024-04-22T08:13:59.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.523+0000 7f30ce84a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:13:59.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.523+0000 7f30ce84a640 1 -- 172.21.15.40:0/2695833017 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30c80a0220 con 0x7f30c80a1c60 2024-04-22T08:13:59.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.523+0000 7f30c4ff9640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:13:59.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.523+0000 7f30c4ff9640 1 -- 172.21.15.40:0/2695833017 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30bc0075a0 con 0x7f30c80a1c60 2024-04-22T08:13:59.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.523+0000 7f30c4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:59.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.523+0000 7f30c4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:13:59.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.523+0000 7f30c4ff9640 10 monclient: dump: 2024-04-22T08:13:59.557 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:59.557 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:59.557 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.557 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.557 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:59.557 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:59.557 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:59.557 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:59.557 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:59.557 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:59.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.523+0000 7f30c67fc640 4 set_mon_vals no callback set 2024-04-22T08:13:59.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.523+0000 7f30ce84a640 10 monclient: _renew_subs 2024-04-22T08:13:59.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.523+0000 7f30ce84a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:13:59.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.523+0000 7f30ce84a640 1 -- 172.21.15.40:0/2695833017 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30c8137590 con 0x7f30c80a1c60 2024-04-22T08:13:59.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.524+0000 7f30ce84a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:13:59.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.524+0000 7f30ce84a640 1 --2- 172.21.15.40:0/2695833017 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30c8133c30 0x7f30c8134010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:59.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.524+0000 7f30ce84a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:59.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.524+0000 7f30ce84a640 1 -- 172.21.15.40:0/2695833017 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30c8066650 con 0x7f30c8133c30 2024-04-22T08:13:59.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.524+0000 7f30c4ff9640 1 -- 172.21.15.40:0/2695833017 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f30bc01b400 con 0x7f30c80a1c60 2024-04-22T08:13:59.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.524+0000 7f30c77fe640 1 --2- 172.21.15.40:0/2695833017 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30c8133c30 0x7f30c8134010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:59.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.524+0000 7f30c77fe640 10 monclient: get_auth_request con 0x7f30c8133c30 auth_method 0 2024-04-22T08:13:59.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.524+0000 7f30c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:59.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.524+0000 7f30c77fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:59.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.524+0000 7f30c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:59.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.524+0000 7f30c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:59.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.524+0000 7f30c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:59.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.524+0000 7f30c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:59.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.524+0000 7f30c4ff9640 1 --2- 172.21.15.40:0/2695833017 >> v2:172.21.15.189:6800/30168 conn(0x7f30b40377d0 0x7f30b4039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:59.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.524+0000 7f30c77fe640 10 monclient(hunting): handle_auth_done global_id 4535 payload 824 2024-04-22T08:13:59.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.524+0000 7f30c77fe640 1 -- 172.21.15.40:0/2695833017 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f30c809fac0 con 0x7f30c8133c30 2024-04-22T08:13:59.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.524+0000 7f30c4ff9640 1 -- 172.21.15.40:0/2695833017 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f30bc049e30 con 0x7f30c80a1c60 2024-04-22T08:13:59.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.524+0000 7f30ccde7640 1 --2- 172.21.15.40:0/2695833017 >> v2:172.21.15.189:6800/30168 conn(0x7f30b40377d0 0x7f30b4039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:59.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.524+0000 7f30ccde7640 10 monclient: get_auth_request con 0x7f30b40377d0 auth_method 0 2024-04-22T08:13:59.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.526+0000 7f30c77fe640 1 -- 172.21.15.40:0/2695833017 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30c8133c30 msgr2=0x7f30c8134010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:13:59.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.526+0000 7f30c77fe640 1 --2- 172.21.15.40:0/2695833017 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30c8133c30 0x7f30c8134010 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f30b80118d0 tx=0x7f30b8011d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:59.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.526+0000 7f30c4ff9640 1 -- 172.21.15.40:0/2695833017 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b800f830 con 0x7f30c8133c30 2024-04-22T08:13:59.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.526+0000 7f30c4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:59.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.526+0000 7f30c4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:13:59.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.526+0000 7f30c4ff9640 10 monclient: dump: 2024-04-22T08:13:59.560 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:59.560 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:59.560 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.560 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.560 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:59.560 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:59.560 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:59.560 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:59.560 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:59.560 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:59.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.527+0000 7f30c4ff9640 1 -- 172.21.15.40:0/2695833017 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f30b800f990 con 0x7f30c8133c30 2024-04-22T08:13:59.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.527+0000 7f30c4ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:13:59.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.527+0000 7f30c4ff9640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:13:59.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.527+0000 7f30c4ff9640 1 -- 172.21.15.40:0/2695833017 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30c8133c30 msgr2=0x7f30c8134010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:59.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.527+0000 7f30c4ff9640 1 --2- 172.21.15.40:0/2695833017 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30c8133c30 0x7f30c8134010 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f30b80118d0 tx=0x7f30b8011d90 comp rx=0 tx=0).stop 2024-04-22T08:13:59.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.527+0000 7f30c4ff9640 1 -- 172.21.15.40:0/2695833017 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30c8133c30 msgr2=0x7f30c8134010 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:13:59.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.527+0000 7f30c4ff9640 1 --2- 172.21.15.40:0/2695833017 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30c8133c30 0x7f30c8134010 secure :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f30b80118d0 tx=0x7f30b8011d90 comp rx=0 tx=0).stop 2024-04-22T08:13:59.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.527+0000 7f30ccde7640 1 --2- 172.21.15.40:0/2695833017 >> v2:172.21.15.189:6800/30168 conn(0x7f30b40377d0 0x7f30b4039c90 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f30c80a5720 tx=0x7f30c0030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:59.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.533+0000 7f30ce84a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:13:59.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.533+0000 7f30ce84a640 1 --2- 172.21.15.40:0/2695833017 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30c8134550 0x7f30c8130030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:59.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.533+0000 7f30ce84a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:59.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.533+0000 7f30ce84a640 1 -- 172.21.15.40:0/2695833017 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30c804e9b0 con 0x7f30c8134550 2024-04-22T08:13:59.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.533+0000 7f30c77fe640 1 --2- 172.21.15.40:0/2695833017 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30c8134550 0x7f30c8130030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:59.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.533+0000 7f30c77fe640 10 monclient: get_auth_request con 0x7f30c8134550 auth_method 0 2024-04-22T08:13:59.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.533+0000 7f30c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:59.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.533+0000 7f30c77fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:59.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.533+0000 7f30c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:59.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.533+0000 7f30c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:59.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.533+0000 7f30c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:59.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.533+0000 7f30c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:59.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.533+0000 7f30c77fe640 10 monclient(hunting): handle_auth_done global_id 4538 payload 824 2024-04-22T08:13:59.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.533+0000 7f30c77fe640 1 -- 172.21.15.40:0/2695833017 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f30c8136930 con 0x7f30c8134550 2024-04-22T08:13:59.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.535+0000 7f30c77fe640 1 -- 172.21.15.40:0/2695833017 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30c8134550 msgr2=0x7f30c8130030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:13:59.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.536+0000 7f30c77fe640 1 --2- 172.21.15.40:0/2695833017 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30c8134550 0x7f30c8130030 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f30b801b000 tx=0x7f30b80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:59.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.536+0000 7f30c4ff9640 1 -- 172.21.15.40:0/2695833017 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b801e660 con 0x7f30c8134550 2024-04-22T08:13:59.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.536+0000 7f30c4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:59.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.536+0000 7f30c4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:13:59.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.536+0000 7f30c4ff9640 10 monclient: dump: 2024-04-22T08:13:59.562 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:59.562 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:59.562 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.563 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.563 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:59.563 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:59.563 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:59.563 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:59.563 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:59.563 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:59.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.536+0000 7f30c4ff9640 1 -- 172.21.15.40:0/2695833017 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4631 (secure 0 0 0) 0x7f30b8024040 con 0x7f30c8134550 2024-04-22T08:13:59.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.536+0000 7f30c4ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:13:59.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.536+0000 7f30c4ff9640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:13:59.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.536+0000 7f30c4ff9640 1 -- 172.21.15.40:0/2695833017 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30c8134550 msgr2=0x7f30c8130030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:59.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.536+0000 7f30c4ff9640 1 --2- 172.21.15.40:0/2695833017 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30c8134550 0x7f30c8130030 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f30b801b000 tx=0x7f30b80046c0 comp rx=0 tx=0).stop 2024-04-22T08:13:59.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.536+0000 7f30c4ff9640 1 -- 172.21.15.40:0/2695833017 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30c8134550 msgr2=0x7f30c8130030 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:13:59.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.536+0000 7f30c4ff9640 1 --2- 172.21.15.40:0/2695833017 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30c8134550 0x7f30c8130030 secure :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f30b801b000 tx=0x7f30b80046c0 comp rx=0 tx=0).stop 2024-04-22T08:13:59.564 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:13:59.564 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:13:59.564 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:13:59.564 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:13:59.564 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 20, 2024-04-22T08:13:59.564 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:13:59.564 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:13:59.565 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:13:59.565 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:13:59.565 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:13:59.565 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 3, 2024-04-22T08:13:59.565 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:13:59.565 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:13:59.565 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:13:59.565 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:13:59.565 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:13:59.565 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:13:59.565 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:13:59.565 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:13:59.565 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:13:59.565 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:13:59.566 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:13:59.566 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:13:59.566 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:13:59.566 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:13:59.566 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:13:59.566 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:13:59.566 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:13:59.566 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:13:59.566 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:13:59.566 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:13:59.566 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:13:59.566 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:13:59.567 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:13:59.567 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:13:59.567 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:13:59.567 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:13:59.567 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:59.567 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:59.567 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:13:59.567 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:13:59.567 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:13:59.567 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:13:59.567 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:13:59.567 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:13:59.567 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:13:59.567 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:13:59.568 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:13:59.568 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:13:59.568 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:13:59.568 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:13:59.568 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:13:59.568 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:13:59.568 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:13:59.568 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:13:59.568 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:13:59.568 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:13:59.568 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:13:59.568 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:13:59.568 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:13:59.568 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:13:59.569 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:13:59.569 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:13:59.569 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:13:59.569 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:13:59.569 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:13:59.569 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:13:59.569 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:13:59.569 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:59.569 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:13:59.569 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:59.569 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:13:59.569 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:13:59.569 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:13:59.569 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:13:59.570 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:59.570 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:13:59.570 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:13:59.570 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:59.570 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:59.570 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:59.570 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:13:59.570 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:13:59.570 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:59.570 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:59.570 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:59.570 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:59.570 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:13:59.570 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:13:59.571 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:13:59.571 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:13:59.571 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:13:59.571 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:59.571 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:59.571 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:13:59.571 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:13:59.571 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:13:59.571 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:13:59.571 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:59.571 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:13:59.571 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:13:59.571 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:59.571 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:59.572 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:59.572 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:13:59.572 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:13:59.572 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:59.572 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:59.572 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:59.572 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:59.572 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:13:59.572 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:13:59.572 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:13:59.572 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:13:59.572 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:13:59.572 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:59.572 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:59.573 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:13:59.573 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:13:59.573 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:13:59.573 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:13:59.573 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:59.573 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:13:59.573 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:13:59.573 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:59.573 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:59.573 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:59.573 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:13:59.573 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:13:59.573 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:59.573 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:59.574 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:59.574 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:59.574 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:13:59.574 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:13:59.574 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:13:59.574 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:13:59.574 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:13:59.574 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:59.574 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:59.574 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:59.574 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:13:59.574 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:13:59.574 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:59.575 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:13:59.575 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:13:59.575 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:13:59.575 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:59.575 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:13:59.575 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:13:59.575 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:59.575 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:13:59.575 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:13:59.575 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 5 2024-04-22T08:13:59.575 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:59.575 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:13:59.575 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:13:59.575 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:59.576 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:13:59.576 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:13:59.576 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 2 2024-04-22T08:13:59.576 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:59.576 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:59.576 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:59.576 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:13:59.576 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:13:59.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.536+0000 7f30ce84a640 1 -- 172.21.15.40:0/2695833017 >> v2:172.21.15.189:6800/30168 conn(0x7f30b40377d0 msgr2=0x7f30b4039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:59.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.536+0000 7f30ce84a640 1 --2- 172.21.15.40:0/2695833017 >> v2:172.21.15.189:6800/30168 conn(0x7f30b40377d0 0x7f30b4039c90 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f30c80a5720 tx=0x7f30c0030000 comp rx=0 tx=0).stop 2024-04-22T08:13:59.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.536+0000 7f30ce84a640 10 monclient: shutdown 2024-04-22T08:13:59.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.536+0000 7f30ce84a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:13:59.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.536+0000 7f30ce84a640 1 -- 172.21.15.40:0/2695833017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30c80a1c60 msgr2=0x7f30c81cae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:59.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.536+0000 7f30ce84a640 1 --2- 172.21.15.40:0/2695833017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30c80a1c60 0x7f30c81cae60 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f30bc00c460 tx=0x7f30bc00c920 comp rx=0 tx=0).stop 2024-04-22T08:13:59.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.536+0000 7f30ccde7640 1 -- 172.21.15.40:0/2695833017 reap_dead start 2024-04-22T08:13:59.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.536+0000 7f30ccde7640 1 -- 172.21.15.40:0/2695833017 reap_dead start 2024-04-22T08:13:59.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.536+0000 7f30ce84a640 1 -- 172.21.15.40:0/2695833017 shutdown_connections 2024-04-22T08:13:59.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.536+0000 7f30ce84a640 1 -- 172.21.15.40:0/2695833017 >> 172.21.15.40:0/2695833017 conn(0x7f30c80a89a0 msgr2=0x7f30c8132150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:13:59.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.536+0000 7f30ce84a640 1 -- 172.21.15.40:0/2695833017 shutdown_connections 2024-04-22T08:13:59.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.536+0000 7f30ce84a640 1 -- 172.21.15.40:0/2695833017 wait complete. 2024-04-22T08:13:59.578 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:13:59.579 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-22T08:13:59.579 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-22T08:13:59.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.684+0000 7fcc87597640 10 monclient: get_monmap_and_config 2024-04-22T08:13:59.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.684+0000 7fcc87597640 10 monclient: build_initial_monmap 2024-04-22T08:13:59.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.684+0000 7fcc87597640 10 monclient: monmap: 2024-04-22T08:13:59.685 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:13:59.685 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:59.685 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:13:59.685673+0000 2024-04-22T08:13:59.685 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:13:59.685673+0000 2024-04-22T08:13:59.685 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:13:59.685 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:59.685 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:13:59.685 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:13:59.685 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:13:59.686 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:13:59.686 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:13:59.686 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:59.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.685+0000 7fcc87597640 1 Processor -- start 2024-04-22T08:13:59.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.685+0000 7fcc87597640 1 -- start start 2024-04-22T08:13:59.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.685+0000 7fcc87597640 10 monclient: init 2024-04-22T08:13:59.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.686+0000 7fcc87597640 10 monclient: _reopen_session rank -1 2024-04-22T08:13:59.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.686+0000 7fcc87597640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-04-22T08:13:59.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.686+0000 7fcc87597640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fcc800a5a10 0x7fcc800a32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:59.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.686+0000 7fcc87597640 10 monclient(hunting): picked mon.noname-c con 0x7fcc800a5a10 addr v2:172.21.15.189:3300/0 2024-04-22T08:13:59.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.686+0000 7fcc87597640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcc800a1970 0x7fcc800a1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:59.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.686+0000 7fcc87597640 10 monclient(hunting): picked mon.noname-a con 0x7fcc800a1970 addr v2:172.21.15.40:3300/0 2024-04-22T08:13:59.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.686+0000 7fcc87597640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcc80135c20 addr v1:172.21.15.189:6789/0 2024-04-22T08:13:59.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.686+0000 7fcc87597640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc8009fe80 con 0x7fcc80135c20 2024-04-22T08:13:59.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.686+0000 7fcc87597640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:59.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.686+0000 7fcc87597640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc800a37f0 con 0x7fcc800a1970 2024-04-22T08:13:59.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.686+0000 7fcc87597640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:59.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.686+0000 7fcc87597640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc800a25b0 con 0x7fcc800a5a10 2024-04-22T08:13:59.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.686+0000 7fcc87597640 10 monclient(hunting): _renew_subs 2024-04-22T08:13:59.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.686+0000 7fcc87597640 10 monclient(hunting): authenticate will time out at 913.679900s 2024-04-22T08:13:59.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.686+0000 7fcc84b32640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcc800a1970 0x7fcc800a1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:59.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.686+0000 7fcc84b32640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcc800a1970 0x7fcc800a1dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56056/0 (socket says 172.21.15.40:56056) 2024-04-22T08:13:59.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.686+0000 7fcc84b32640 1 -- 172.21.15.40:0/1710353662 learned_addr learned my addr 172.21.15.40:0/1710353662 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:13:59.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.686+0000 7fcc84b32640 10 monclient(hunting): get_auth_request con 0x7fcc800a1970 auth_method 0 2024-04-22T08:13:59.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.686+0000 7fcc84b32640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:59.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.686+0000 7fcc84b32640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:59.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.686+0000 7fcc84b32640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:59.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.686+0000 7fcc84b32640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:59.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.686+0000 7fcc84b32640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:59.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.686+0000 7fcc84b32640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:59.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.686+0000 7fcc85333640 1 --2- 172.21.15.40:0/1710353662 >> v2:172.21.15.189:3300/0 conn(0x7fcc800a5a10 0x7fcc800a32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:59.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.686+0000 7fcc84b32640 10 monclient(hunting): handle_auth_done global_id 4569 payload 274 2024-04-22T08:13:59.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.686+0000 7fcc84b32640 1 -- 172.21.15.40:0/1710353662 >> v2:172.21.15.189:3300/0 conn(0x7fcc800a5a10 msgr2=0x7fcc800a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:13:59.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.686+0000 7fcc84b32640 1 --2- 172.21.15.40:0/1710353662 >> v2:172.21.15.189:3300/0 conn(0x7fcc800a5a10 0x7fcc800a32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:59.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.686+0000 7fcc84b32640 1 -- 172.21.15.40:0/1710353662 >> v1:172.21.15.189:6789/0 conn(0x7fcc80135c20 legacy=0x7fcc80136090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:13:59.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.686+0000 7fcc84b32640 10 monclient: _finish_hunting 0 2024-04-22T08:13:59.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.686+0000 7fcc84b32640 1 monclient: found mon.noname-a 2024-04-22T08:13:59.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.686+0000 7fcc84b32640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:13:59.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.686+0000 7fcc84b32640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:13:59.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.686+0000 7fcc84b32640 1 -- 172.21.15.40:0/1710353662 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc8009fba0 con 0x7fcc800a1970 2024-04-22T08:13:59.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.686+0000 7fcc85333640 10 monclient: get_auth_request con 0x7fcc800a5a10 auth_method 0 2024-04-22T08:13:59.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.686+0000 7fcc85333640 1 --2- 172.21.15.40:0/1710353662 >> v2:172.21.15.189:3300/0 conn(0x7fcc800a5a10 0x7fcc800a32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T08:13:59.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.689+0000 7fcc84b32640 1 -- 172.21.15.40:0/1710353662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc800a1970 msgr2=0x7fcc800a1dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:13:59.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.689+0000 7fcc84b32640 1 --2- 172.21.15.40:0/1710353662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc800a1970 0x7fcc800a1dc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fcc8006c3a0 tx=0x7fcc7402b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=659a51d0d67b083b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:59.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.689+0000 7fcc7ffff640 1 -- 172.21.15.40:0/1710353662 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc74031a40 con 0x7fcc800a1970 2024-04-22T08:13:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.689+0000 7fcc7ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.689+0000 7fcc7ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:13:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.689+0000 7fcc7ffff640 10 monclient: dump: 2024-04-22T08:13:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:59.702 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:59.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.689+0000 7fcc7ffff640 10 monclient: _finish_auth 0 2024-04-22T08:13:59.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.689+0000 7fcc7ffff640 10 monclient: _check_auth_tickets 2024-04-22T08:13:59.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.689+0000 7fcc7ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:13:59.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.689+0000 7fcc7ffff640 1 -- 172.21.15.40:0/1710353662 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc74031ba0 con 0x7fcc800a1970 2024-04-22T08:13:59.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.689+0000 7fcc87597640 5 monclient: authenticate success, global_id 4569 2024-04-22T08:13:59.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.689+0000 7fcc87597640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:13:59.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.689+0000 7fcc7ffff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:13:59.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.689+0000 7fcc87597640 10 monclient: get_monmap_and_config success 2024-04-22T08:13:59.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.689+0000 7fcc7ffff640 1 -- 172.21.15.40:0/1710353662 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc74033440 con 0x7fcc800a1970 2024-04-22T08:13:59.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.689+0000 7fcc87597640 4 set_mon_vals no callback set 2024-04-22T08:13:59.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.689+0000 7fcc7ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:59.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.689+0000 7fcc7ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:13:59.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.689+0000 7fcc7ffff640 10 monclient: dump: 2024-04-22T08:13:59.704 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:59.704 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:59.704 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.704 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.704 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:59.704 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:59.704 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:59.704 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:59.704 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:59.704 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:59.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.689+0000 7fcc87597640 10 monclient: shutdown 2024-04-22T08:13:59.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.689+0000 7fcc87597640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:13:59.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.689+0000 7fcc87597640 1 -- 172.21.15.40:0/1710353662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc800a1970 msgr2=0x7fcc800a1dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:59.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.689+0000 7fcc87597640 1 --2- 172.21.15.40:0/1710353662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc800a1970 0x7fcc800a1dc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fcc8006c3a0 tx=0x7fcc7402b7f0 comp rx=0 tx=0).stop 2024-04-22T08:13:59.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.690+0000 7fcc87597640 1 -- 172.21.15.40:0/1710353662 shutdown_connections 2024-04-22T08:13:59.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.690+0000 7fcc87597640 1 --2- 172.21.15.40:0/1710353662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc800a1970 0x7fcc800a1dc0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:59.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.690+0000 7fcc87597640 1 --2- 172.21.15.40:0/1710353662 >> v2:172.21.15.189:3300/0 conn(0x7fcc800a5a10 0x7fcc800a32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:59.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.690+0000 7fcc87597640 1 -- 172.21.15.40:0/1710353662 >> 172.21.15.40:0/1710353662 conn(0x7fcc800a4ef0 msgr2=0x7fcc800a5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:13:59.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.690+0000 7fcc87597640 1 -- 172.21.15.40:0/1710353662 shutdown_connections 2024-04-22T08:13:59.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.690+0000 7fcc87597640 1 -- 172.21.15.40:0/1710353662 wait complete. 2024-04-22T08:13:59.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.690+0000 7fcc87597640 10 monclient: build_initial_monmap 2024-04-22T08:13:59.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.690+0000 7fcc87597640 10 monclient: monmap: 2024-04-22T08:13:59.705 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:13:59.706 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:13:59.706 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:13:59.706 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:13:59.706 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:13:59.706 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:59.706 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:13:59.706 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:13:59.706 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:13:59.706 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:59.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.690+0000 7fcc87597640 1 Processor -- start 2024-04-22T08:13:59.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc87597640 1 -- start start 2024-04-22T08:13:59.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc87597640 10 monclient: init 2024-04-22T08:13:59.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc87597640 10 monclient: _reopen_session rank -1 2024-04-22T08:13:59.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc87597640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-22T08:13:59.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc87597640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc800a1970 0x7fcc8011ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:59.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc87597640 10 monclient(hunting): picked mon.noname-c con 0x7fcc800a1970 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:13:59.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc87597640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc800a5a10 0x7fcc8011ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:59.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc87597640 10 monclient(hunting): picked mon.noname-b con 0x7fcc800a5a10 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:13:59.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc87597640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc80135c20 0x7fcc8011f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:59.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc87597640 10 monclient(hunting): picked mon.noname-a con 0x7fcc80135c20 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:13:59.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc87597640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:59.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc87597640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc801222c0 con 0x7fcc80135c20 2024-04-22T08:13:59.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc87597640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:59.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc87597640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc80121ea0 con 0x7fcc800a5a10 2024-04-22T08:13:59.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc87597640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:59.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc87597640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc80120710 con 0x7fcc800a1970 2024-04-22T08:13:59.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc87597640 10 monclient(hunting): _renew_subs 2024-04-22T08:13:59.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc87597640 10 monclient(hunting): authenticate will time out at 913.685214s 2024-04-22T08:13:59.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc85333640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc800a1970 0x7fcc8011ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:59.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc85b34640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc80135c20 0x7fcc8011f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:59.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc85333640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc800a1970 0x7fcc8011ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:51556/0 (socket says 172.21.15.40:51556) 2024-04-22T08:13:59.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc85333640 1 -- 172.21.15.40:0/1237270540 learned_addr learned my addr 172.21.15.40:0/1237270540 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:13:59.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc85333640 10 monclient(hunting): get_auth_request con 0x7fcc800a1970 auth_method 0 2024-04-22T08:13:59.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc85333640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:59.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc85333640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:59.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc85333640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:59.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc85b34640 10 monclient(hunting): get_auth_request con 0x7fcc80135c20 auth_method 0 2024-04-22T08:13:59.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc85b34640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:59.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc85b34640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:59.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc85b34640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:59.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc85333640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:59.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc85333640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:59.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc84b32640 1 --2- 172.21.15.40:0/1237270540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc800a5a10 0x7fcc8011ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:59.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc85333640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:59.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc85b34640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:59.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc85b34640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:59.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc85b34640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:59.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc84b32640 10 monclient(hunting): get_auth_request con 0x7fcc800a5a10 auth_method 0 2024-04-22T08:13:59.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc84b32640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:59.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc84b32640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:59.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc84b32640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:59.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc84b32640 1 -- 172.21.15.40:0/1237270540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc800a5a10 msgr2=0x7fcc8011ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-22T08:13:59.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc84b32640 1 -- 172.21.15.40:0/1237270540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc800a5a10 msgr2=0x7fcc8011ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:13:59.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc84b32640 1 --2- 172.21.15.40:0/1237270540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc800a5a10 0x7fcc8011ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:13:59.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc84b32640 1 --2- 172.21.15.40:0/1237270540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc800a5a10 0x7fcc8011ef80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:13:59.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc85b34640 10 monclient(hunting): handle_auth_done global_id 4572 payload 824 2024-04-22T08:13:59.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc85b34640 1 -- 172.21.15.40:0/1237270540 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc800a1970 msgr2=0x7fcc8011ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:59.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc85b34640 1 --2- 172.21.15.40:0/1237270540 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc800a1970 0x7fcc8011ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:59.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc85b34640 1 -- 172.21.15.40:0/1237270540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc800a5a10 msgr2=0x7fcc8011ef80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:13:59.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc85b34640 1 --2- 172.21.15.40:0/1237270540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc800a5a10 0x7fcc8011ef80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:59.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc85b34640 10 monclient: _finish_hunting 0 2024-04-22T08:13:59.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc85b34640 1 monclient: found mon.noname-a 2024-04-22T08:13:59.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc85b34640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:13:59.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc85b34640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:13:59.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc85b34640 1 -- 172.21.15.40:0/1237270540 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc740094d0 con 0x7fcc80135c20 2024-04-22T08:13:59.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.691+0000 7fcc85333640 1 --2- 172.21.15.40:0/1237270540 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc800a1970 0x7fcc8011ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:13:59.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.694+0000 7fcc85b34640 1 -- 172.21.15.40:0/1237270540 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc80135c20 msgr2=0x7fcc8011f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:13:59.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.694+0000 7fcc85b34640 1 --2- 172.21.15.40:0/1237270540 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc80135c20 0x7fcc8011f4c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fcc7800c410 tx=0x7fcc7800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:59.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.694+0000 7fcc7dffb640 1 -- 172.21.15.40:0/1237270540 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc780119a0 con 0x7fcc80135c20 2024-04-22T08:13:59.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.694+0000 7fcc7dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:59.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.694+0000 7fcc7dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:13:59.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.694+0000 7fcc7dffb640 10 monclient: dump: 2024-04-22T08:13:59.711 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:59.711 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:59.711 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.711 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.711 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:59.711 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:59.711 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:59.711 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:59.711 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:59.711 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:59.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.694+0000 7fcc7dffb640 10 monclient: _finish_auth 0 2024-04-22T08:13:59.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.694+0000 7fcc7dffb640 10 monclient: _check_auth_tickets 2024-04-22T08:13:59.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.694+0000 7fcc7dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:13:59.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.694+0000 7fcc7dffb640 1 -- 172.21.15.40:0/1237270540 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc78011b00 con 0x7fcc80135c20 2024-04-22T08:13:59.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.694+0000 7fcc87597640 5 monclient: authenticate success, global_id 4572 2024-04-22T08:13:59.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.694+0000 7fcc87597640 10 monclient: _renew_subs 2024-04-22T08:13:59.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.694+0000 7fcc87597640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:13:59.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.694+0000 7fcc87597640 1 -- 172.21.15.40:0/1237270540 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc801350d0 con 0x7fcc80135c20 2024-04-22T08:13:59.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.694+0000 7fcc7dffb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:13:59.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.694+0000 7fcc7dffb640 1 -- 172.21.15.40:0/1237270540 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc78012420 con 0x7fcc80135c20 2024-04-22T08:13:59.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.694+0000 7fcc7dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:59.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.694+0000 7fcc7dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:13:59.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.694+0000 7fcc7dffb640 10 monclient: dump: 2024-04-22T08:13:59.713 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:59.713 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:59.713 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.713 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.713 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:59.713 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:59.713 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:59.713 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:59.713 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:59.713 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:59.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.694+0000 7fcc7f7fe640 4 set_mon_vals no callback set 2024-04-22T08:13:59.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.694+0000 7fcc87597640 10 monclient: _renew_subs 2024-04-22T08:13:59.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.694+0000 7fcc87597640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:13:59.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.694+0000 7fcc87597640 1 -- 172.21.15.40:0/1237270540 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc80131930 con 0x7fcc80135c20 2024-04-22T08:13:59.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.695+0000 7fcc7dffb640 1 -- 172.21.15.40:0/1237270540 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fcc78012640 con 0x7fcc80135c20 2024-04-22T08:13:59.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.695+0000 7fcc87597640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:13:59.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.695+0000 7fcc87597640 1 --2- 172.21.15.40:0/1237270540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc801cab10 0x7fcc801caef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:59.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.695+0000 7fcc87597640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:59.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.695+0000 7fcc87597640 1 -- 172.21.15.40:0/1237270540 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc800667a0 con 0x7fcc801cab10 2024-04-22T08:13:59.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.695+0000 7fcc7dffb640 1 --2- 172.21.15.40:0/1237270540 >> v2:172.21.15.189:6800/30168 conn(0x7fcc6c037820 0x7fcc6c039ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:59.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.695+0000 7fcc7dffb640 1 -- 172.21.15.40:0/1237270540 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 3898+0+0 (secure 0 0 0) 0x7fcc7804a030 con 0x7fcc80135c20 2024-04-22T08:13:59.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.695+0000 7fcc85333640 1 --2- 172.21.15.40:0/1237270540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc801cab10 0x7fcc801caef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:59.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.695+0000 7fcc85333640 10 monclient: get_auth_request con 0x7fcc801cab10 auth_method 0 2024-04-22T08:13:59.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.695+0000 7fcc85333640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:59.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.695+0000 7fcc85333640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:59.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.695+0000 7fcc85333640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:59.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.695+0000 7fcc85333640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:59.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.695+0000 7fcc85333640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:59.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.695+0000 7fcc85333640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:59.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.695+0000 7fcc84b32640 1 --2- 172.21.15.40:0/1237270540 >> v2:172.21.15.189:6800/30168 conn(0x7fcc6c037820 0x7fcc6c039ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:59.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.695+0000 7fcc84b32640 10 monclient: get_auth_request con 0x7fcc6c037820 auth_method 0 2024-04-22T08:13:59.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.696+0000 7fcc85333640 10 monclient(hunting): handle_auth_done global_id 24142 payload 824 2024-04-22T08:13:59.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.696+0000 7fcc85333640 1 -- 172.21.15.40:0/1237270540 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcc8009ee10 con 0x7fcc801cab10 2024-04-22T08:13:59.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.698+0000 7fcc85333640 1 -- 172.21.15.40:0/1237270540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc801cab10 msgr2=0x7fcc801caef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:13:59.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.698+0000 7fcc85333640 1 --2- 172.21.15.40:0/1237270540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc801cab10 0x7fcc801caef0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fcc7000c9f0 tx=0x7fcc7000ceb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:59.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.698+0000 7fcc84b32640 1 --2- 172.21.15.40:0/1237270540 >> v2:172.21.15.189:6800/30168 conn(0x7fcc6c037820 0x7fcc6c039ce0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fcc74033000 tx=0x7fcc74008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:59.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.698+0000 7fcc7dffb640 1 -- 172.21.15.40:0/1237270540 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc70012070 con 0x7fcc801cab10 2024-04-22T08:13:59.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.698+0000 7fcc7dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:59.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.698+0000 7fcc7dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:13:59.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.698+0000 7fcc7dffb640 10 monclient: dump: 2024-04-22T08:13:59.718 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:59.718 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:59.718 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.718 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.718 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:59.718 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:59.719 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:59.719 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:59.719 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:59.719 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:59.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.698+0000 7fcc7dffb640 1 -- 172.21.15.40:0/1237270540 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7fcc70017420 con 0x7fcc801cab10 2024-04-22T08:13:59.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.698+0000 7fcc7dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:13:59.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.698+0000 7fcc7dffb640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:13:59.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.698+0000 7fcc7dffb640 1 -- 172.21.15.40:0/1237270540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc801cab10 msgr2=0x7fcc801caef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:59.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.698+0000 7fcc7dffb640 1 --2- 172.21.15.40:0/1237270540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc801cab10 0x7fcc801caef0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fcc7000c9f0 tx=0x7fcc7000ceb0 comp rx=0 tx=0).stop 2024-04-22T08:13:59.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.698+0000 7fcc7dffb640 1 -- 172.21.15.40:0/1237270540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc801cab10 msgr2=0x7fcc801caef0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:13:59.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.698+0000 7fcc7dffb640 1 --2- 172.21.15.40:0/1237270540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc801cab10 0x7fcc801caef0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:59.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.705+0000 7fcc87597640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:13:59.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.705+0000 7fcc87597640 1 --2- 172.21.15.40:0/1237270540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc801cb430 0x7fcc8006df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:59.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.705+0000 7fcc87597640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:59.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.705+0000 7fcc87597640 1 -- 172.21.15.40:0/1237270540 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8004e9b0 con 0x7fcc801cb430 2024-04-22T08:13:59.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.705+0000 7fcc85333640 1 --2- 172.21.15.40:0/1237270540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc801cb430 0x7fcc8006df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:59.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.705+0000 7fcc85333640 10 monclient: get_auth_request con 0x7fcc801cb430 auth_method 0 2024-04-22T08:13:59.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.705+0000 7fcc85333640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:59.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.705+0000 7fcc85333640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:59.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.705+0000 7fcc85333640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:59.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.705+0000 7fcc85333640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:59.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.705+0000 7fcc85333640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:59.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.705+0000 7fcc85333640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:59.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.705+0000 7fcc85333640 10 monclient(hunting): handle_auth_done global_id 24145 payload 824 2024-04-22T08:13:59.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.705+0000 7fcc85333640 1 -- 172.21.15.40:0/1237270540 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fcc8011fb40 con 0x7fcc801cb430 2024-04-22T08:13:59.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.708+0000 7fcc85333640 1 -- 172.21.15.40:0/1237270540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc801cb430 msgr2=0x7fcc8006df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:13:59.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.708+0000 7fcc85333640 1 --2- 172.21.15.40:0/1237270540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc801cb430 0x7fcc8006df30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fcc70017000 tx=0x7fcc70004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:59.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.708+0000 7fcc7dffb640 1 -- 172.21.15.40:0/1237270540 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc7000ab60 con 0x7fcc801cb430 2024-04-22T08:13:59.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.708+0000 7fcc7dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:59.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.708+0000 7fcc7dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:13:59.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.708+0000 7fcc7dffb640 10 monclient: dump: 2024-04-22T08:13:59.721 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:59.721 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:59.721 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.721 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.721 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:59.721 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:59.721 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:59.721 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:59.721 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:59.721 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:59.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.708+0000 7fcc7dffb640 1 -- 172.21.15.40:0/1237270540 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4461 (secure 0 0 0) 0x7fcc7000acc0 con 0x7fcc801cb430 2024-04-22T08:13:59.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.708+0000 7fcc7dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:13:59.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.708+0000 7fcc7dffb640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:13:59.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.708+0000 7fcc7dffb640 1 -- 172.21.15.40:0/1237270540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc801cb430 msgr2=0x7fcc8006df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:59.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.708+0000 7fcc7dffb640 1 --2- 172.21.15.40:0/1237270540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc801cb430 0x7fcc8006df30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fcc70017000 tx=0x7fcc70004a40 comp rx=0 tx=0).stop 2024-04-22T08:13:59.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.708+0000 7fcc7dffb640 1 -- 172.21.15.40:0/1237270540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc801cb430 msgr2=0x7fcc8006df30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:13:59.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.708+0000 7fcc7dffb640 1 --2- 172.21.15.40:0/1237270540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc801cb430 0x7fcc8006df30 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:59.722 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:13:59.722 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:13:59.722 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:13:59.723 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:13:59.723 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 20, 2024-04-22T08:13:59.723 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:13:59.723 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:13:59.723 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:13:59.723 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:13:59.723 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:13:59.723 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 3, 2024-04-22T08:13:59.723 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:13:59.723 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:13:59.723 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:13:59.723 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:13:59.724 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:13:59.724 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:13:59.724 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:13:59.725 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:13:59.726 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:13:59.726 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:13:59.726 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:13:59.726 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:13:59.726 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:13:59.726 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:13:59.726 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:13:59.726 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:13:59.726 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:13:59.726 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:13:59.726 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:13:59.726 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:13:59.727 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:13:59.727 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:13:59.727 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:13:59.727 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:13:59.727 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:13:59.727 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:13:59.727 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:59.727 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:59.727 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:13:59.727 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:13:59.727 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:13:59.727 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:13:59.727 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:13:59.728 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:13:59.728 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:13:59.728 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:13:59.728 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:13:59.728 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:13:59.728 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:13:59.728 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:13:59.728 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:13:59.728 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:13:59.728 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:13:59.729 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:13:59.729 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:13:59.729 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:13:59.729 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:13:59.729 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:13:59.729 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:13:59.729 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:13:59.729 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:13:59.729 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:13:59.729 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:13:59.729 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:13:59.730 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:13:59.730 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:13:59.730 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:13:59.730 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:59.730 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:13:59.730 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:59.730 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:13:59.730 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:13:59.730 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:13:59.730 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:13:59.730 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:59.730 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:13:59.730 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:13:59.730 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:59.731 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:59.731 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:59.731 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:13:59.731 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:13:59.731 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:59.731 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:59.731 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:59.731 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:59.731 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:13:59.731 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:13:59.731 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:13:59.731 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:13:59.731 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:13:59.731 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:59.731 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:59.731 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:13:59.732 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:13:59.732 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:13:59.732 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:13:59.732 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:59.732 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:13:59.732 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:13:59.732 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:59.732 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:59.732 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:59.732 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:13:59.732 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:13:59.732 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:59.732 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:59.732 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:59.732 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:59.733 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:13:59.733 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:13:59.733 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:13:59.733 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:13:59.733 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:13:59.733 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:59.733 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:59.733 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:13:59.733 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:13:59.733 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:13:59.733 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:13:59.733 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:59.733 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:13:59.733 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:13:59.734 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:59.734 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:59.734 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:59.734 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:13:59.734 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:13:59.734 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:13:59.734 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:59.734 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:59.734 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:59.734 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:13:59.734 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:13:59.734 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:13:59.734 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:13:59.734 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:13:59.735 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:59.735 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:59.735 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:59.735 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:13:59.735 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:13:59.735 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:59.735 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:13:59.735 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:13:59.735 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:13:59.735 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:59.735 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:13:59.735 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:13:59.735 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:13:59.735 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:13:59.735 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:13:59.736 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 4 2024-04-22T08:13:59.736 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:13:59.736 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:13:59.736 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:13:59.736 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:13:59.736 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:13:59.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.709+0000 7fcc87597640 1 -- 172.21.15.40:0/1237270540 >> v2:172.21.15.189:6800/30168 conn(0x7fcc6c037820 msgr2=0x7fcc6c039ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:59.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.709+0000 7fcc87597640 1 --2- 172.21.15.40:0/1237270540 >> v2:172.21.15.189:6800/30168 conn(0x7fcc6c037820 0x7fcc6c039ce0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fcc74033000 tx=0x7fcc74008030 comp rx=0 tx=0).stop 2024-04-22T08:13:59.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.709+0000 7fcc87597640 10 monclient: shutdown 2024-04-22T08:13:59.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.709+0000 7fcc87597640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:13:59.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.709+0000 7fcc87597640 1 -- 172.21.15.40:0/1237270540 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc80135c20 msgr2=0x7fcc8011f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:59.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.709+0000 7fcc87597640 1 --2- 172.21.15.40:0/1237270540 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc80135c20 0x7fcc8011f4c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fcc7800c410 tx=0x7fcc7800c8d0 comp rx=0 tx=0).stop 2024-04-22T08:13:59.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.709+0000 7fcc85b34640 1 -- 172.21.15.40:0/1237270540 reap_dead start 2024-04-22T08:13:59.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.709+0000 7fcc85b34640 1 -- 172.21.15.40:0/1237270540 reap_dead start 2024-04-22T08:13:59.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.709+0000 7fcc87597640 1 -- 172.21.15.40:0/1237270540 shutdown_connections 2024-04-22T08:13:59.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.709+0000 7fcc87597640 1 -- 172.21.15.40:0/1237270540 >> 172.21.15.40:0/1237270540 conn(0x7fcc800a4ef0 msgr2=0x7fcc8009de00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:13:59.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.709+0000 7fcc87597640 1 -- 172.21.15.40:0/1237270540 shutdown_connections 2024-04-22T08:13:59.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.709+0000 7fcc87597640 1 -- 172.21.15.40:0/1237270540 wait complete. 2024-04-22T08:13:59.739 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:13:59.739 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-04-22T08:13:59.739 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-04-22T08:13:59.739 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-04-22T08:13:59.740 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-04-22T08:13:59.740 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-22T08:13:59.740 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T08:13:59.793 INFO:tasks.ceph.mon.a:Stopped 2024-04-22T08:13:59.793 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-22T08:13:59.793 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:13:59.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.935+0000 7f85ac8af640 10 monclient: get_monmap_and_config 2024-04-22T08:13:59.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.935+0000 7f85ac8af640 10 monclient: build_initial_monmap 2024-04-22T08:13:59.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.935+0000 7f85ac8af640 10 monclient: monmap: 2024-04-22T08:13:59.935 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:13:59.935 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:59.936 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:13:59.936263+0000 2024-04-22T08:13:59.936 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:13:59.936263+0000 2024-04-22T08:13:59.936 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:13:59.936 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:59.936 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:13:59.936 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:13:59.936 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:13:59.936 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:13:59.936 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:13:59.936 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:59.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.936+0000 7f85ac8af640 1 Processor -- start 2024-04-22T08:13:59.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.936+0000 7f85ac8af640 1 -- start start 2024-04-22T08:13:59.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.936+0000 7f85ac8af640 10 monclient: init 2024-04-22T08:13:59.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.936+0000 7f85ac8af640 10 monclient: _reopen_session rank -1 2024-04-22T08:13:59.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.936+0000 7f85ac8af640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-04-22T08:13:59.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.936+0000 7f85ac8af640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f85a812fc40 addr v1:172.21.15.40:6789/0 2024-04-22T08:13:59.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.936+0000 7f85ac8af640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f85a809e210 addr v1:172.21.15.189:6789/0 2024-04-22T08:13:59.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.936+0000 7f85ac8af640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f85a81393d0 0x7f85a813b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:59.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.936+0000 7f85ac8af640 10 monclient(hunting): picked mon.noname-b con 0x7f85a81393d0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:13:59.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.936+0000 7f85ac8af640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85a809ea50 con 0x7f85a812fc40 2024-04-22T08:13:59.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.936+0000 7f85ac8af640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85a809cf90 con 0x7f85a809e210 2024-04-22T08:13:59.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.936+0000 7f85ac8af640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:59.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.936+0000 7f85ac8af640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85a809d270 con 0x7f85a81393d0 2024-04-22T08:13:59.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.936+0000 7f85ac8af640 10 monclient(hunting): _renew_subs 2024-04-22T08:13:59.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.936+0000 7f85ac8af640 10 monclient(hunting): authenticate will time out at 913.930475s 2024-04-22T08:13:59.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.936+0000 7f85a659c640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f85a812fc40 legacy=0x7f85a8130020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-04-22T08:13:59.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.936+0000 7f85a6d9d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f85a81393d0 0x7f85a813b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:59.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.936+0000 7f85a6d9d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f85a81393d0 0x7f85a813b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:51574/0 (socket says 172.21.15.40:51574) 2024-04-22T08:13:59.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.936+0000 7f85a6d9d640 1 -- 172.21.15.40:0/2732546468 learned_addr learned my addr 172.21.15.40:0/2732546468 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:13:59.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.936+0000 7f85a6d9d640 10 monclient(hunting): get_auth_request con 0x7f85a81393d0 auth_method 0 2024-04-22T08:13:59.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.936+0000 7f85a6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:59.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.936+0000 7f85a6d9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:59.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.936+0000 7f85a6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:59.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.936+0000 7f85a6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:59.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.936+0000 7f85a6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:59.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.936+0000 7f85a6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:59.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.937+0000 7f85a6d9d640 10 monclient(hunting): handle_auth_done global_id 4544 payload 274 2024-04-22T08:13:59.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.937+0000 7f85a6d9d640 1 -- 172.21.15.40:0/2732546468 >> v1:172.21.15.189:6789/0 conn(0x7f85a809e210 legacy=0x7f85a8137cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:13:59.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.937+0000 7f85a6d9d640 1 -- 172.21.15.40:0/2732546468 >> v1:172.21.15.40:6789/0 conn(0x7f85a812fc40 legacy=0x7f85a8130020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:13:59.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.937+0000 7f85a6d9d640 10 monclient: _finish_hunting 0 2024-04-22T08:13:59.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.937+0000 7f85a6d9d640 1 monclient: found mon.noname-b 2024-04-22T08:13:59.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.937+0000 7f85a6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:13:59.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.937+0000 7f85a6d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:13:59.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.937+0000 7f85a6d9d640 1 -- 172.21.15.40:0/2732546468 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85a809d550 con 0x7f85a81393d0 2024-04-22T08:13:59.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.939+0000 7f85a6d9d640 1 -- 172.21.15.40:0/2732546468 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f85a81393d0 msgr2=0x7f85a813b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:13:59.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.939+0000 7f85a6d9d640 1 --2- 172.21.15.40:0/2732546468 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f85a81393d0 0x7f85a813b7c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f85a0009e80 tx=0x7f85a002bd90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=575e5d5a8b0dabe0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:59.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.940+0000 7f85a5d9b640 1 -- 172.21.15.40:0/2732546468 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a002cb40 con 0x7f85a81393d0 2024-04-22T08:13:59.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.940+0000 7f85a5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:59.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.940+0000 7f85a5d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:13:59.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.940+0000 7f85a5d9b640 10 monclient: dump: 2024-04-22T08:13:59.946 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:59.946 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:59.946 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.947 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.947 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:59.947 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:59.947 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:59.947 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:59.947 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:59.947 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:59.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.940+0000 7f85a5d9b640 10 monclient: _finish_auth 0 2024-04-22T08:13:59.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.940+0000 7f85a5d9b640 10 monclient: _check_auth_tickets 2024-04-22T08:13:59.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.940+0000 7f85a5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:13:59.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.940+0000 7f85a5d9b640 1 -- 172.21.15.40:0/2732546468 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85a002cca0 con 0x7f85a81393d0 2024-04-22T08:13:59.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.940+0000 7f85ac8af640 5 monclient: authenticate success, global_id 4544 2024-04-22T08:13:59.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.940+0000 7f85ac8af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:13:59.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.940+0000 7f85a5d9b640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:13:59.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.940+0000 7f85ac8af640 10 monclient: get_monmap_and_config success 2024-04-22T08:13:59.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.940+0000 7f85a5d9b640 1 -- 172.21.15.40:0/2732546468 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a0034610 con 0x7f85a81393d0 2024-04-22T08:13:59.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.940+0000 7f85ac8af640 4 set_mon_vals no callback set 2024-04-22T08:13:59.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.940+0000 7f85a5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:59.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.940+0000 7f85a5d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:13:59.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.940+0000 7f85a5d9b640 10 monclient: dump: 2024-04-22T08:13:59.948 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:59.948 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:59.948 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.948 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.948 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:59.948 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:59.948 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:59.948 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:59.948 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:59.948 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:59.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.940+0000 7f85ac8af640 10 monclient: shutdown 2024-04-22T08:13:59.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.940+0000 7f85ac8af640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:13:59.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.940+0000 7f85ac8af640 1 -- 172.21.15.40:0/2732546468 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f85a81393d0 msgr2=0x7f85a813b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:59.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.940+0000 7f85ac8af640 1 --2- 172.21.15.40:0/2732546468 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f85a81393d0 0x7f85a813b7c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f85a0009e80 tx=0x7f85a002bd90 comp rx=0 tx=0).stop 2024-04-22T08:13:59.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.940+0000 7f85ac8af640 1 -- 172.21.15.40:0/2732546468 shutdown_connections 2024-04-22T08:13:59.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.940+0000 7f85ac8af640 1 --2- 172.21.15.40:0/2732546468 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f85a81393d0 0x7f85a813b7c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:59.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.940+0000 7f85ac8af640 1 -- 172.21.15.40:0/2732546468 >> 172.21.15.40:0/2732546468 conn(0x7f85a80a0b50 msgr2=0x7f85a80a0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:13:59.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.940+0000 7f85ac8af640 1 -- 172.21.15.40:0/2732546468 shutdown_connections 2024-04-22T08:13:59.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.940+0000 7f85ac8af640 1 -- 172.21.15.40:0/2732546468 wait complete. 2024-04-22T08:13:59.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.941+0000 7f85ac8af640 10 monclient: build_initial_monmap 2024-04-22T08:13:59.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.941+0000 7f85ac8af640 10 monclient: monmap: 2024-04-22T08:13:59.949 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:13:59.949 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:13:59.949 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:13:59.949 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:13:59.950 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:13:59.950 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:59.950 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:13:59.950 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:13:59.950 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:13:59.950 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:59.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.941+0000 7f85ac8af640 1 Processor -- start 2024-04-22T08:13:59.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.941+0000 7f85ac8af640 1 -- start start 2024-04-22T08:13:59.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.941+0000 7f85ac8af640 10 monclient: init 2024-04-22T08:13:59.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.942+0000 7f85ac8af640 10 monclient: _reopen_session rank -1 2024-04-22T08:13:59.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.942+0000 7f85ac8af640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:13:59.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.942+0000 7f85ac8af640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85a809e210 0x7f85a8070340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:59.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.942+0000 7f85ac8af640 10 monclient(hunting): picked mon.noname-a con 0x7f85a809e210 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:13:59.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.942+0000 7f85ac8af640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85a812fc40 0x7f85a813a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:59.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.942+0000 7f85ac8af640 10 monclient(hunting): picked mon.noname-b con 0x7f85a812fc40 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:13:59.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.942+0000 7f85ac8af640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f85a81393d0 0x7f85a813ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:59.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.942+0000 7f85ac8af640 10 monclient(hunting): picked mon.noname-c con 0x7f85a81393d0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:13:59.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.942+0000 7f85ac8af640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:59.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.942+0000 7f85ac8af640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85a813bf10 con 0x7f85a809e210 2024-04-22T08:13:59.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.942+0000 7f85ac8af640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:59.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.942+0000 7f85ac8af640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85a8070880 con 0x7f85a812fc40 2024-04-22T08:13:59.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.942+0000 7f85ac8af640 10 monclient(hunting): start opening mon connection 2024-04-22T08:13:59.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.942+0000 7f85ac8af640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85a813da20 con 0x7f85a81393d0 2024-04-22T08:13:59.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.942+0000 7f85ac8af640 10 monclient(hunting): _renew_subs 2024-04-22T08:13:59.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.942+0000 7f85ac8af640 10 monclient(hunting): authenticate will time out at 913.936048s 2024-04-22T08:13:59.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.942+0000 7f85a659c640 1 -- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85a809e210 msgr2=0x7f85a8070340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:13:59.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.942+0000 7f85a659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85a809e210 0x7f85a8070340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:13:59.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.942+0000 7f85a6d9d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f85a81393d0 0x7f85a813ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:59.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.942+0000 7f85a6d9d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f85a81393d0 0x7f85a813ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:51582/0 (socket says 172.21.15.40:51582) 2024-04-22T08:13:59.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.942+0000 7f85a6d9d640 1 -- 172.21.15.40:0/1739433482 learned_addr learned my addr 172.21.15.40:0/1739433482 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:13:59.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.942+0000 7f85a6d9d640 10 monclient(hunting): get_auth_request con 0x7f85a81393d0 auth_method 0 2024-04-22T08:13:59.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.942+0000 7f85a6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:59.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.942+0000 7f85a6d9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:59.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.942+0000 7f85a6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:59.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.942+0000 7f85a6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:59.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.942+0000 7f85a6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:59.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.942+0000 7f859ffff640 1 --2- 172.21.15.40:0/1739433482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85a812fc40 0x7f85a813a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:59.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.942+0000 7f85a6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:59.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.942+0000 7f859ffff640 10 monclient(hunting): get_auth_request con 0x7f85a812fc40 auth_method 0 2024-04-22T08:13:59.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.942+0000 7f859ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:13:59.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.942+0000 7f859ffff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:13:59.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.942+0000 7f859ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:13:59.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.942+0000 7f859ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:13:59.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.942+0000 7f859ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:13:59.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.942+0000 7f859ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:13:59.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.942+0000 7f85a6d9d640 10 monclient(hunting): handle_auth_done global_id 4547 payload 824 2024-04-22T08:13:59.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.942+0000 7f85a6d9d640 1 -- 172.21.15.40:0/1739433482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85a812fc40 msgr2=0x7f85a813a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:13:59.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.942+0000 7f85a6d9d640 1 --2- 172.21.15.40:0/1739433482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85a812fc40 0x7f85a813a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:59.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.942+0000 7f85a6d9d640 1 -- 172.21.15.40:0/1739433482 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85a809e210 msgr2=0x7f85a8070340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:13:59.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.942+0000 7f85a6d9d640 1 --2- 172.21.15.40:0/1739433482 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85a809e210 0x7f85a8070340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:13:59.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.942+0000 7f85a6d9d640 10 monclient: _finish_hunting 0 2024-04-22T08:13:59.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.942+0000 7f85a6d9d640 1 monclient: found mon.noname-c 2024-04-22T08:13:59.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.942+0000 7f85a6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:13:59.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.942+0000 7f85a6d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:13:59.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.942+0000 7f85a6d9d640 1 -- 172.21.15.40:0/1739433482 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85a0009b30 con 0x7f85a81393d0 2024-04-22T08:13:59.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.944+0000 7f85a6d9d640 1 -- 172.21.15.40:0/1739433482 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f85a81393d0 msgr2=0x7f85a813ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:13:59.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.944+0000 7f85a6d9d640 1 --2- 172.21.15.40:0/1739433482 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f85a81393d0 0x7f85a813ac50 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f85a0009fa0 tx=0x7f85a0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:59.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.945+0000 7f859effd640 1 -- 172.21.15.40:0/1739433482 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a003e450 con 0x7f85a81393d0 2024-04-22T08:13:59.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.945+0000 7f859effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:59.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.945+0000 7f859effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:13:59.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.945+0000 7f859effd640 10 monclient: dump: 2024-04-22T08:13:59.954 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:59.954 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:59.955 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.955 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.955 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:59.955 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:59.955 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:59.955 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:59.955 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:59.955 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:59.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.945+0000 7f859effd640 10 monclient: _finish_auth 0 2024-04-22T08:13:59.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.945+0000 7f859effd640 10 monclient: _check_auth_tickets 2024-04-22T08:13:59.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.945+0000 7f859effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:13:59.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.945+0000 7f85ac8af640 5 monclient: authenticate success, global_id 4547 2024-04-22T08:13:59.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.945+0000 7f859effd640 1 -- 172.21.15.40:0/1739433482 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85a003e5b0 con 0x7f85a81393d0 2024-04-22T08:13:59.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.945+0000 7f859effd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:13:59.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.945+0000 7f859effd640 1 -- 172.21.15.40:0/1739433482 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a003e960 con 0x7f85a81393d0 2024-04-22T08:13:59.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.945+0000 7f859effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:13:59.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.945+0000 7f859effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:13:59.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.945+0000 7f859effd640 10 monclient: dump: 2024-04-22T08:13:59.956 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:13:59.956 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:13:59.956 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.956 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:13:59.956 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:13:59.956 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:13:59.956 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:13:59.956 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:13:59.956 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:13:59.956 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:13:59.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.945+0000 7f85a559a640 4 set_mon_vals no callback set 2024-04-22T08:13:59.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.945+0000 7f85ac8af640 10 monclient: _renew_subs 2024-04-22T08:13:59.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.945+0000 7f85ac8af640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:13:59.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.945+0000 7f85ac8af640 1 -- 172.21.15.40:0/1739433482 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85a806c8e0 con 0x7f85a81393d0 2024-04-22T08:13:59.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.945+0000 7f85ac8af640 10 monclient: _renew_subs 2024-04-22T08:13:59.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.945+0000 7f85ac8af640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:13:59.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.945+0000 7f85ac8af640 1 -- 172.21.15.40:0/1739433482 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85a813c640 con 0x7f85a81393d0 2024-04-22T08:13:59.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.945+0000 7f859effd640 1 -- 172.21.15.40:0/1739433482 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f85a003eb40 con 0x7f85a81393d0 2024-04-22T08:13:59.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.946+0000 7f859effd640 1 --2- 172.21.15.40:0/1739433482 >> v2:172.21.15.189:6800/30168 conn(0x7f8588037780 0x7f8588039c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:13:59.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.946+0000 7f859effd640 1 -- 172.21.15.40:0/1739433482 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f85a006d130 con 0x7f85a81393d0 2024-04-22T08:13:59.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.946+0000 7f85ac8af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:13:59.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.946+0000 7f85ac8af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:13:59.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.946+0000 7f85ac8af640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:13:59.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.946+0000 7f85ac8af640 1 -- 172.21.15.40:0/1739433482 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85a80a0030 con 0x7f85a81393d0 2024-04-22T08:13:59.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.946+0000 7f85a659c640 1 --2- 172.21.15.40:0/1739433482 >> v2:172.21.15.189:6800/30168 conn(0x7f8588037780 0x7f8588039c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:13:59.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.946+0000 7f85a659c640 10 monclient: get_auth_request con 0x7f8588037780 auth_method 0 2024-04-22T08:13:59.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.949+0000 7f85a659c640 1 --2- 172.21.15.40:0/1739433482 >> v2:172.21.15.189:6800/30168 conn(0x7f8588037780 0x7f8588039c40 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8598006fd0 tx=0x7f8598006d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:13:59.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.950+0000 7f859effd640 1 -- 172.21.15.40:0/1739433482 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f85a0043070 con 0x7f85a81393d0 2024-04-22T08:13:59.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.950+0000 7f859effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:13:59.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:13:59.950+0000 7f859effd640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:14:00.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:00.199+0000 7f85ac8af640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:14:00.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:00.199+0000 7f85ac8af640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:14:00.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:00.199+0000 7f85ac8af640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:14:00.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:00.199+0000 7f85ac8af640 1 -- 172.21.15.40:0/1739433482 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f85a80a0030 con 0x7f85a81393d0 2024-04-22T08:14:00.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:00.199+0000 7f859effd640 1 -- 172.21.15.40:0/1739433482 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f85a003c350 con 0x7f85a81393d0 2024-04-22T08:14:00.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:00.199+0000 7f859effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:14:00.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:00.199+0000 7f859effd640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:14:00.205 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:14:00.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:00.201+0000 7f85ac8af640 1 -- 172.21.15.40:0/1739433482 >> v2:172.21.15.189:6800/30168 conn(0x7f8588037780 msgr2=0x7f8588039c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:00.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:00.201+0000 7f85ac8af640 1 --2- 172.21.15.40:0/1739433482 >> v2:172.21.15.189:6800/30168 conn(0x7f8588037780 0x7f8588039c40 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8598006fd0 tx=0x7f8598006d80 comp rx=0 tx=0).stop 2024-04-22T08:14:00.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:00.201+0000 7f85ac8af640 10 monclient: shutdown 2024-04-22T08:14:00.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:00.201+0000 7f85ac8af640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:14:00.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:00.201+0000 7f85ac8af640 1 -- 172.21.15.40:0/1739433482 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f85a81393d0 msgr2=0x7f85a813ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:00.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:00.201+0000 7f85ac8af640 1 --2- 172.21.15.40:0/1739433482 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f85a81393d0 0x7f85a813ac50 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f85a0009fa0 tx=0x7f85a0004a40 comp rx=0 tx=0).stop 2024-04-22T08:14:00.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:00.201+0000 7f85ac8af640 1 -- 172.21.15.40:0/1739433482 shutdown_connections 2024-04-22T08:14:00.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:00.201+0000 7f85ac8af640 1 --2- 172.21.15.40:0/1739433482 >> v2:172.21.15.189:6800/30168 conn(0x7f8588037780 0x7f8588039c40 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:00.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:00.201+0000 7f85ac8af640 1 --2- 172.21.15.40:0/1739433482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85a812fc40 0x7f85a813a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:00.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:00.201+0000 7f85ac8af640 1 --2- 172.21.15.40:0/1739433482 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f85a81393d0 0x7f85a813ac50 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:00.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:00.201+0000 7f85ac8af640 1 --2- 172.21.15.40:0/1739433482 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85a809e210 0x7f85a8070340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:00.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:00.201+0000 7f85ac8af640 1 -- 172.21.15.40:0/1739433482 >> 172.21.15.40:0/1739433482 conn(0x7f85a80a0b50 msgr2=0x7f85a809b450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:14:00.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:00.201+0000 7f85ac8af640 1 -- 172.21.15.40:0/1739433482 shutdown_connections 2024-04-22T08:14:00.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:00.201+0000 7f85ac8af640 1 -- 172.21.15.40:0/1739433482 wait complete. 2024-04-22T08:14:03.214 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:14:03.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.324+0000 7f03f4f86640 10 monclient: get_monmap_and_config 2024-04-22T08:14:03.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.324+0000 7f03f4f86640 10 monclient: build_initial_monmap 2024-04-22T08:14:03.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.324+0000 7f03f4f86640 10 monclient: monmap: 2024-04-22T08:14:03.324 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:14:03.324 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:03.324 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:14:03.325673+0000 2024-04-22T08:14:03.325 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:14:03.325673+0000 2024-04-22T08:14:03.325 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:14:03.325 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:03.325 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:14:03.325 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:14:03.325 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:14:03.325 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:14:03.325 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:14:03.325 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:03.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.325+0000 7f03f4f86640 1 Processor -- start 2024-04-22T08:14:03.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.325+0000 7f03f4f86640 1 -- start start 2024-04-22T08:14:03.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.325+0000 7f03f4f86640 10 monclient: init 2024-04-22T08:14:03.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.326+0000 7f03f4f86640 10 monclient: _reopen_session rank -1 2024-04-22T08:14:03.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.326+0000 7f03f4f86640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-04-22T08:14:03.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.326+0000 7f03f4f86640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f03f00736d0 0x7f03f0133af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:03.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.326+0000 7f03f4f86640 10 monclient(hunting): picked mon.noname-c con 0x7f03f00736d0 addr v2:172.21.15.189:3300/0 2024-04-22T08:14:03.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.326+0000 7f03f4f86640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03f0134030 0x7f03f00a01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:03.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.326+0000 7f03f4f86640 10 monclient(hunting): picked mon.noname-b con 0x7f03f0134030 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:14:03.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.326+0000 7f03f4f86640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f03f0134490 0x7f03f00a0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:03.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.326+0000 7f03f4f86640 10 monclient(hunting): picked mon.noname-a con 0x7f03f0134490 addr v2:172.21.15.40:3300/0 2024-04-22T08:14:03.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.326+0000 7f03f4f86640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:03.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.326+0000 7f03f4f86640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03f01338c0 con 0x7f03f0134490 2024-04-22T08:14:03.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.326+0000 7f03f4f86640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:03.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.326+0000 7f03f4f86640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03f006d2c0 con 0x7f03f00736d0 2024-04-22T08:14:03.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.326+0000 7f03f4f86640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:03.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.326+0000 7f03f4f86640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03f0073ab0 con 0x7f03f0134030 2024-04-22T08:14:03.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.326+0000 7f03f4f86640 10 monclient(hunting): _renew_subs 2024-04-22T08:14:03.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.326+0000 7f03f4f86640 10 monclient(hunting): authenticate will time out at 917.320099s 2024-04-22T08:14:03.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.326+0000 7f03eed9d640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f03f0134490 msgr2=0x7f03f00a0710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:14:03.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.326+0000 7f03eed9d640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f03f0134490 0x7f03f00a0710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:14:03.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.326+0000 7f03e7fff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03f0134030 0x7f03f00a01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:03.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.326+0000 7f03e7fff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03f0134030 0x7f03f00a01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:51584/0 (socket says 172.21.15.40:51584) 2024-04-22T08:14:03.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.326+0000 7f03e7fff640 1 -- 172.21.15.40:0/2322902808 learned_addr learned my addr 172.21.15.40:0/2322902808 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:14:03.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.326+0000 7f03e7fff640 10 monclient(hunting): get_auth_request con 0x7f03f0134030 auth_method 0 2024-04-22T08:14:03.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.326+0000 7f03e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:03.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.326+0000 7f03e7fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:03.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.326+0000 7f03e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:03.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.326+0000 7f03ee59c640 1 --2- 172.21.15.40:0/2322902808 >> v2:172.21.15.189:3300/0 conn(0x7f03f00736d0 0x7f03f0133af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:03.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.326+0000 7f03ee59c640 10 monclient(hunting): get_auth_request con 0x7f03f00736d0 auth_method 0 2024-04-22T08:14:03.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.326+0000 7f03ee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:03.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.326+0000 7f03ee59c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:03.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.326+0000 7f03ee59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:03.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.326+0000 7f03e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:03.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.326+0000 7f03e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:03.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.326+0000 7f03e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:03.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.326+0000 7f03ee59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:03.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.326+0000 7f03ee59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:03.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.326+0000 7f03ee59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:03.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.326+0000 7f03e7fff640 10 monclient(hunting): handle_auth_done global_id 4550 payload 274 2024-04-22T08:14:03.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.326+0000 7f03e7fff640 1 -- 172.21.15.40:0/2322902808 >> v2:172.21.15.189:3300/0 conn(0x7f03f00736d0 msgr2=0x7f03f0133af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:14:03.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.326+0000 7f03e7fff640 1 --2- 172.21.15.40:0/2322902808 >> v2:172.21.15.189:3300/0 conn(0x7f03f00736d0 0x7f03f0133af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:03.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.326+0000 7f03e7fff640 1 -- 172.21.15.40:0/2322902808 >> v2:172.21.15.40:3300/0 conn(0x7f03f0134490 msgr2=0x7f03f00a0710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:14:03.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.326+0000 7f03e7fff640 1 --2- 172.21.15.40:0/2322902808 >> v2:172.21.15.40:3300/0 conn(0x7f03f0134490 0x7f03f00a0710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:03.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.326+0000 7f03e7fff640 10 monclient: _finish_hunting 0 2024-04-22T08:14:03.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.326+0000 7f03e7fff640 1 monclient: found mon.noname-b 2024-04-22T08:14:03.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.326+0000 7f03e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:14:03.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.326+0000 7f03e7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:14:03.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.326+0000 7f03e7fff640 1 -- 172.21.15.40:0/2322902808 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03f0070320 con 0x7f03f0134030 2024-04-22T08:14:03.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.329+0000 7f03e7fff640 1 -- 172.21.15.40:0/2322902808 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03f0134030 msgr2=0x7f03f00a01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:03.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.329+0000 7f03e7fff640 1 --2- 172.21.15.40:0/2322902808 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03f0134030 0x7f03f00a01d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f03f00a2310 tx=0x7f03d802b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28a183f66537f5bb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:03.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.330+0000 7f03edd9b640 1 -- 172.21.15.40:0/2322902808 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d8031a10 con 0x7f03f0134030 2024-04-22T08:14:03.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.330+0000 7f03edd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:03.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.330+0000 7f03edd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:14:03.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.330+0000 7f03edd9b640 10 monclient: dump: 2024-04-22T08:14:03.337 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:03.337 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:03.337 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:03.337 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:03.337 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:03.337 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:03.337 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:03.337 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:03.337 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:03.337 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:03.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.330+0000 7f03edd9b640 10 monclient: _finish_auth 0 2024-04-22T08:14:03.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.330+0000 7f03edd9b640 10 monclient: _check_auth_tickets 2024-04-22T08:14:03.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.330+0000 7f03edd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:14:03.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.330+0000 7f03edd9b640 1 -- 172.21.15.40:0/2322902808 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03d8031b70 con 0x7f03f0134030 2024-04-22T08:14:03.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.330+0000 7f03f4f86640 5 monclient: authenticate success, global_id 4550 2024-04-22T08:14:03.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.330+0000 7f03f4f86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:14:03.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.330+0000 7f03edd9b640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:14:03.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.330+0000 7f03f4f86640 10 monclient: get_monmap_and_config success 2024-04-22T08:14:03.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.330+0000 7f03edd9b640 1 -- 172.21.15.40:0/2322902808 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d8030350 con 0x7f03f0134030 2024-04-22T08:14:03.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.330+0000 7f03edd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:03.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.330+0000 7f03f4f86640 4 set_mon_vals no callback set 2024-04-22T08:14:03.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.330+0000 7f03edd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:14:03.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.330+0000 7f03edd9b640 10 monclient: dump: 2024-04-22T08:14:03.338 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:03.338 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:03.339 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:03.339 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:03.339 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:03.339 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:03.339 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:03.339 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:03.339 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:03.339 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:03.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.330+0000 7f03f4f86640 10 monclient: shutdown 2024-04-22T08:14:03.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.330+0000 7f03f4f86640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:14:03.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.330+0000 7f03f4f86640 1 -- 172.21.15.40:0/2322902808 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03f0134030 msgr2=0x7f03f00a01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:03.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.330+0000 7f03f4f86640 1 --2- 172.21.15.40:0/2322902808 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03f0134030 0x7f03f00a01d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f03f00a2310 tx=0x7f03d802b8a0 comp rx=0 tx=0).stop 2024-04-22T08:14:03.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.330+0000 7f03f4f86640 1 -- 172.21.15.40:0/2322902808 shutdown_connections 2024-04-22T08:14:03.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.330+0000 7f03f4f86640 1 --2- 172.21.15.40:0/2322902808 >> v2:172.21.15.40:3300/0 conn(0x7f03f0134490 0x7f03f00a0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:03.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.330+0000 7f03f4f86640 1 --2- 172.21.15.40:0/2322902808 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03f0134030 0x7f03f00a01d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:03.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.330+0000 7f03f4f86640 1 --2- 172.21.15.40:0/2322902808 >> v2:172.21.15.189:3300/0 conn(0x7f03f00736d0 0x7f03f0133af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:03.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.330+0000 7f03f4f86640 1 -- 172.21.15.40:0/2322902808 >> 172.21.15.40:0/2322902808 conn(0x7f03f00a18f0 msgr2=0x7f03f006ba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:14:03.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.330+0000 7f03f4f86640 1 -- 172.21.15.40:0/2322902808 shutdown_connections 2024-04-22T08:14:03.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.330+0000 7f03f4f86640 1 -- 172.21.15.40:0/2322902808 wait complete. 2024-04-22T08:14:03.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.331+0000 7f03f4f86640 10 monclient: build_initial_monmap 2024-04-22T08:14:03.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.331+0000 7f03f4f86640 10 monclient: monmap: 2024-04-22T08:14:03.340 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:14:03.340 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:14:03.340 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:14:03.340 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:14:03.340 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:14:03.340 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:03.340 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:14:03.340 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:14:03.341 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:14:03.341 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:03.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.331+0000 7f03f4f86640 1 Processor -- start 2024-04-22T08:14:03.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.331+0000 7f03f4f86640 1 -- start start 2024-04-22T08:14:03.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.331+0000 7f03f4f86640 10 monclient: init 2024-04-22T08:14:03.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.332+0000 7f03f4f86640 10 monclient: _reopen_session rank -1 2024-04-22T08:14:03.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.332+0000 7f03f4f86640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-22T08:14:03.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.332+0000 7f03f4f86640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03f00736d0 0x7f03f01c8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:03.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.332+0000 7f03f4f86640 10 monclient(hunting): picked mon.noname-b con 0x7f03f00736d0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:14:03.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.332+0000 7f03f4f86640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03f0134030 0x7f03f01c8f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:03.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.332+0000 7f03f4f86640 10 monclient(hunting): picked mon.noname-c con 0x7f03f0134030 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:14:03.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.332+0000 7f03f4f86640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03f0134490 0x7f03f009b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:03.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.332+0000 7f03f4f86640 10 monclient(hunting): picked mon.noname-a con 0x7f03f0134490 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:14:03.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.332+0000 7f03f4f86640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:03.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.332+0000 7f03f4f86640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f011b180 con 0x7f03f0134490 2024-04-22T08:14:03.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.332+0000 7f03f4f86640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:03.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.332+0000 7f03f4f86640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f009d2a0 con 0x7f03f00736d0 2024-04-22T08:14:03.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.332+0000 7f03f4f86640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:03.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.332+0000 7f03f4f86640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03f006d940 con 0x7f03f0134030 2024-04-22T08:14:03.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.332+0000 7f03f4f86640 10 monclient(hunting): _renew_subs 2024-04-22T08:14:03.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.332+0000 7f03f4f86640 10 monclient(hunting): authenticate will time out at 917.326256s 2024-04-22T08:14:03.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.332+0000 7f03eed9d640 1 -- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03f0134490 msgr2=0x7f03f009b230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:14:03.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.332+0000 7f03eed9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03f0134490 0x7f03f009b230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:14:03.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.332+0000 7f03e7fff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03f0134030 0x7f03f01c8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:03.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.332+0000 7f03e7fff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03f0134030 0x7f03f01c8f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:51588/0 (socket says 172.21.15.40:51588) 2024-04-22T08:14:03.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.332+0000 7f03e7fff640 1 -- 172.21.15.40:0/2384854017 learned_addr learned my addr 172.21.15.40:0/2384854017 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:14:03.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.332+0000 7f03e7fff640 10 monclient(hunting): get_auth_request con 0x7f03f0134030 auth_method 0 2024-04-22T08:14:03.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.332+0000 7f03e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:03.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.332+0000 7f03e7fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:03.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.332+0000 7f03e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:03.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.332+0000 7f03e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:03.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.332+0000 7f03e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:03.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.332+0000 7f03e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:03.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.332+0000 7f03ee59c640 1 --2- 172.21.15.40:0/2384854017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03f00736d0 0x7f03f01c8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:03.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.332+0000 7f03ee59c640 10 monclient(hunting): get_auth_request con 0x7f03f00736d0 auth_method 0 2024-04-22T08:14:03.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.332+0000 7f03ee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:03.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.332+0000 7f03ee59c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:03.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.332+0000 7f03ee59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:03.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.332+0000 7f03e7fff640 10 monclient(hunting): handle_auth_done global_id 4553 payload 824 2024-04-22T08:14:03.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.332+0000 7f03e7fff640 1 -- 172.21.15.40:0/2384854017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03f00736d0 msgr2=0x7f03f01c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:03.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.332+0000 7f03e7fff640 1 --2- 172.21.15.40:0/2384854017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03f00736d0 0x7f03f01c8a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:03.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.332+0000 7f03e7fff640 1 -- 172.21.15.40:0/2384854017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03f0134490 msgr2=0x7f03f009b230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:14:03.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.332+0000 7f03e7fff640 1 --2- 172.21.15.40:0/2384854017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03f0134490 0x7f03f009b230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:03.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.332+0000 7f03e7fff640 10 monclient: _finish_hunting 0 2024-04-22T08:14:03.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.332+0000 7f03e7fff640 1 monclient: found mon.noname-c 2024-04-22T08:14:03.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.332+0000 7f03e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:14:03.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.332+0000 7f03e7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:14:03.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.332+0000 7f03e7fff640 1 -- 172.21.15.40:0/2384854017 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d80094d0 con 0x7f03f0134030 2024-04-22T08:14:03.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.335+0000 7f03e7fff640 1 -- 172.21.15.40:0/2384854017 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03f0134030 msgr2=0x7f03f01c8f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:03.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.335+0000 7f03e7fff640 1 --2- 172.21.15.40:0/2384854017 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03f0134030 0x7f03f01c8f60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f03d8009960 tx=0x7f03d80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:03.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.335+0000 7f03e6ffd640 1 -- 172.21.15.40:0/2384854017 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d803a620 con 0x7f03f0134030 2024-04-22T08:14:03.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.335+0000 7f03e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:03.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.335+0000 7f03e6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:14:03.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.335+0000 7f03e6ffd640 10 monclient: dump: 2024-04-22T08:14:03.344 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:03.344 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:03.345 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:03.345 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:03.345 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:03.345 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:03.345 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:03.345 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:03.345 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:03.345 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:03.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.335+0000 7f03e6ffd640 10 monclient: _finish_auth 0 2024-04-22T08:14:03.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.335+0000 7f03e6ffd640 10 monclient: _check_auth_tickets 2024-04-22T08:14:03.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.335+0000 7f03e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:14:03.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.335+0000 7f03e6ffd640 1 -- 172.21.15.40:0/2384854017 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03d803a7a0 con 0x7f03f0134030 2024-04-22T08:14:03.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.335+0000 7f03e6ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:14:03.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.335+0000 7f03e6ffd640 1 -- 172.21.15.40:0/2384854017 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d803ab50 con 0x7f03f0134030 2024-04-22T08:14:03.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.335+0000 7f03e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:03.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.335+0000 7f03e6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:14:03.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.335+0000 7f03e6ffd640 10 monclient: dump: 2024-04-22T08:14:03.346 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:03.346 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:03.346 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:03.346 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:03.346 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:03.346 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:03.346 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:03.346 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:03.346 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:03.346 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:03.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.335+0000 7f03f4f86640 5 monclient: authenticate success, global_id 4553 2024-04-22T08:14:03.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.335+0000 7f03f4f86640 10 monclient: _renew_subs 2024-04-22T08:14:03.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.335+0000 7f03f4f86640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:14:03.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.335+0000 7f03f4f86640 1 -- 172.21.15.40:0/2384854017 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03f006d670 con 0x7f03f0134030 2024-04-22T08:14:03.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.335+0000 7f03ed59a640 4 set_mon_vals no callback set 2024-04-22T08:14:03.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.335+0000 7f03f4f86640 10 monclient: _renew_subs 2024-04-22T08:14:03.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.335+0000 7f03f4f86640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:14:03.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.335+0000 7f03f4f86640 1 -- 172.21.15.40:0/2384854017 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03f00a3da0 con 0x7f03f0134030 2024-04-22T08:14:03.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.336+0000 7f03e6ffd640 1 -- 172.21.15.40:0/2384854017 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f03d803acf0 con 0x7f03f0134030 2024-04-22T08:14:03.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.336+0000 7f03f4f86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:14:03.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.336+0000 7f03f4f86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:14:03.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.336+0000 7f03f4f86640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:14:03.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.336+0000 7f03e6ffd640 1 --2- 172.21.15.40:0/2384854017 >> v2:172.21.15.189:6800/30168 conn(0x7f03d0037780 0x7f03d0039c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:03.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.336+0000 7f03f4f86640 1 -- 172.21.15.40:0/2384854017 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03f004e9b0 con 0x7f03f0134030 2024-04-22T08:14:03.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.336+0000 7f03e6ffd640 1 -- 172.21.15.40:0/2384854017 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f03d802e070 con 0x7f03f0134030 2024-04-22T08:14:03.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.336+0000 7f03ee59c640 1 --2- 172.21.15.40:0/2384854017 >> v2:172.21.15.189:6800/30168 conn(0x7f03d0037780 0x7f03d0039c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:03.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.336+0000 7f03ee59c640 10 monclient: get_auth_request con 0x7f03d0037780 auth_method 0 2024-04-22T08:14:03.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.339+0000 7f03ee59c640 1 --2- 172.21.15.40:0/2384854017 >> v2:172.21.15.189:6800/30168 conn(0x7f03d0037780 0x7f03d0039c40 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f03e0009a30 tx=0x7f03e0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:03.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.341+0000 7f03e6ffd640 1 -- 172.21.15.40:0/2384854017 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f03d8035070 con 0x7f03f0134030 2024-04-22T08:14:03.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.341+0000 7f03e6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:14:03.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.341+0000 7f03e6ffd640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:14:03.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.590+0000 7f03f4f86640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:14:03.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.590+0000 7f03f4f86640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:14:03.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.590+0000 7f03f4f86640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:14:03.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.590+0000 7f03f4f86640 1 -- 172.21.15.40:0/2384854017 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f03f009e980 con 0x7f03f0134030 2024-04-22T08:14:03.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.590+0000 7f03e6ffd640 1 -- 172.21.15.40:0/2384854017 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f03d804bd70 con 0x7f03f0134030 2024-04-22T08:14:03.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.590+0000 7f03e6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:14:03.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.590+0000 7f03e6ffd640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:14:03.591 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:14:03.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.592+0000 7f03f4f86640 1 -- 172.21.15.40:0/2384854017 >> v2:172.21.15.189:6800/30168 conn(0x7f03d0037780 msgr2=0x7f03d0039c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:03.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.592+0000 7f03f4f86640 1 --2- 172.21.15.40:0/2384854017 >> v2:172.21.15.189:6800/30168 conn(0x7f03d0037780 0x7f03d0039c40 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f03e0009a30 tx=0x7f03e0008040 comp rx=0 tx=0).stop 2024-04-22T08:14:03.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.592+0000 7f03f4f86640 10 monclient: shutdown 2024-04-22T08:14:03.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.592+0000 7f03f4f86640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:14:03.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.592+0000 7f03f4f86640 1 -- 172.21.15.40:0/2384854017 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03f0134030 msgr2=0x7f03f01c8f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:03.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.592+0000 7f03f4f86640 1 --2- 172.21.15.40:0/2384854017 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03f0134030 0x7f03f01c8f60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f03d8009960 tx=0x7f03d80046c0 comp rx=0 tx=0).stop 2024-04-22T08:14:03.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.592+0000 7f03f4f86640 1 -- 172.21.15.40:0/2384854017 shutdown_connections 2024-04-22T08:14:03.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.592+0000 7f03f4f86640 1 --2- 172.21.15.40:0/2384854017 >> v2:172.21.15.189:6800/30168 conn(0x7f03d0037780 0x7f03d0039c40 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:03.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.592+0000 7f03f4f86640 1 --2- 172.21.15.40:0/2384854017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03f0134490 0x7f03f009b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:03.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.592+0000 7f03f4f86640 1 --2- 172.21.15.40:0/2384854017 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03f0134030 0x7f03f01c8f60 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:03.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.592+0000 7f03f4f86640 1 --2- 172.21.15.40:0/2384854017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03f00736d0 0x7f03f01c8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:03.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.592+0000 7f03f4f86640 1 -- 172.21.15.40:0/2384854017 >> 172.21.15.40:0/2384854017 conn(0x7f03f00a18f0 msgr2=0x7f03f012d970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:14:03.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.592+0000 7f03f4f86640 1 -- 172.21.15.40:0/2384854017 shutdown_connections 2024-04-22T08:14:03.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:03.592+0000 7f03f4f86640 1 -- 172.21.15.40:0/2384854017 wait complete. 2024-04-22T08:14:06.607 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:14:06.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.718+0000 7f40a5084640 10 monclient: get_monmap_and_config 2024-04-22T08:14:06.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.718+0000 7f40a5084640 10 monclient: build_initial_monmap 2024-04-22T08:14:06.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.718+0000 7f40a5084640 10 monclient: monmap: 2024-04-22T08:14:06.718 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:14:06.718 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:06.718 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:14:06.719067+0000 2024-04-22T08:14:06.718 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:14:06.719067+0000 2024-04-22T08:14:06.718 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:14:06.719 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:06.719 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:14:06.719 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:14:06.719 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:14:06.719 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:14:06.719 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:14:06.719 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:06.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.718+0000 7f40a5084640 1 Processor -- start 2024-04-22T08:14:06.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.718+0000 7f40a5084640 1 -- start start 2024-04-22T08:14:06.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.718+0000 7f40a5084640 10 monclient: init 2024-04-22T08:14:06.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.719+0000 7f40a5084640 10 monclient: _reopen_session rank -1 2024-04-22T08:14:06.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.719+0000 7f40a5084640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-04-22T08:14:06.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.719+0000 7f40a5084640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f40a009ee10 0x7f40a00a32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:06.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.719+0000 7f40a5084640 10 monclient(hunting): picked mon.noname-b con 0x7f40a009ee10 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:14:06.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.719+0000 7f40a5084640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f40a00a1d30 addr v1:172.21.15.40:6789/0 2024-04-22T08:14:06.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.719+0000 7f40a5084640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f40a00a2150 0x7f40a01364a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:06.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.719+0000 7f40a5084640 10 monclient(hunting): picked mon.noname-a con 0x7f40a00a2150 addr v2:172.21.15.40:3300/0 2024-04-22T08:14:06.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.719+0000 7f40a5084640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40a00a2580 con 0x7f40a00a1d30 2024-04-22T08:14:06.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.719+0000 7f40a5084640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:06.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.719+0000 7f40a5084640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40a009f1f0 con 0x7f40a00a2150 2024-04-22T08:14:06.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.719+0000 7f40a5084640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:06.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.719+0000 7f40a5084640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40a00a39c0 con 0x7f40a009ee10 2024-04-22T08:14:06.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.719+0000 7f40a5084640 10 monclient(hunting): _renew_subs 2024-04-22T08:14:06.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.719+0000 7f40a5084640 10 monclient(hunting): authenticate will time out at 920.713189s 2024-04-22T08:14:06.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.719+0000 7f409f59e640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f40a00a2150 msgr2=0x7f40a01364a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:14:06.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.719+0000 7f409e59c640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f40a00a1d30 legacy=0x7f40a0135d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-04-22T08:14:06.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.719+0000 7f409f59e640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f40a00a2150 0x7f40a01364a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:14:06.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.719+0000 7f409ed9d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f40a009ee10 0x7f40a00a32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:06.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.719+0000 7f409ed9d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f40a009ee10 0x7f40a00a32b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:51594/0 (socket says 172.21.15.40:51594) 2024-04-22T08:14:06.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.719+0000 7f409ed9d640 1 -- 172.21.15.40:0/3550986779 learned_addr learned my addr 172.21.15.40:0/3550986779 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:14:06.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.719+0000 7f409ed9d640 10 monclient(hunting): get_auth_request con 0x7f40a009ee10 auth_method 0 2024-04-22T08:14:06.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.719+0000 7f409ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:06.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.719+0000 7f409ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:06.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.719+0000 7f409ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:06.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.719+0000 7f409ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:06.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.719+0000 7f409ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:06.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.719+0000 7f409ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:06.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.719+0000 7f409ed9d640 10 monclient(hunting): handle_auth_done global_id 4556 payload 274 2024-04-22T08:14:06.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.719+0000 7f409ed9d640 1 -- 172.21.15.40:0/3550986779 >> v2:172.21.15.40:3300/0 conn(0x7f40a00a2150 msgr2=0x7f40a01364a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:14:06.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.719+0000 7f409ed9d640 1 --2- 172.21.15.40:0/3550986779 >> v2:172.21.15.40:3300/0 conn(0x7f40a00a2150 0x7f40a01364a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:06.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.719+0000 7f409ed9d640 1 -- 172.21.15.40:0/3550986779 >> v1:172.21.15.40:6789/0 conn(0x7f40a00a1d30 legacy=0x7f40a0135d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:14:06.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.719+0000 7f409ed9d640 10 monclient: _finish_hunting 0 2024-04-22T08:14:06.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.719+0000 7f409ed9d640 1 monclient: found mon.noname-b 2024-04-22T08:14:06.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.719+0000 7f409ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:14:06.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.719+0000 7f409ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:14:06.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.719+0000 7f409ed9d640 1 -- 172.21.15.40:0/3550986779 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40a009fd30 con 0x7f40a009ee10 2024-04-22T08:14:06.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.722+0000 7f409ed9d640 1 -- 172.21.15.40:0/3550986779 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f40a009ee10 msgr2=0x7f40a00a32b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:06.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.722+0000 7f409ed9d640 1 --2- 172.21.15.40:0/3550986779 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f40a009ee10 0x7f40a00a32b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4090009870 tx=0x7f409002b7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=241e9ae91a1eaab3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:06.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.722+0000 7f409dd9b640 1 -- 172.21.15.40:0/3550986779 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4090031a40 con 0x7f40a009ee10 2024-04-22T08:14:06.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.722+0000 7f409dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:06.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.722+0000 7f409dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:14:06.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.722+0000 7f409dd9b640 10 monclient: dump: 2024-04-22T08:14:06.730 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:06.730 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:06.730 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:06.730 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:06.730 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:06.731 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:06.731 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:06.731 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:06.731 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:06.731 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:06.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.722+0000 7f409dd9b640 10 monclient: _finish_auth 0 2024-04-22T08:14:06.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.722+0000 7f409dd9b640 10 monclient: _check_auth_tickets 2024-04-22T08:14:06.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.722+0000 7f409dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:14:06.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.722+0000 7f409dd9b640 1 -- 172.21.15.40:0/3550986779 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4090031ba0 con 0x7f40a009ee10 2024-04-22T08:14:06.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.722+0000 7f409dd9b640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:14:06.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.723+0000 7f409dd9b640 1 -- 172.21.15.40:0/3550986779 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4090033440 con 0x7f40a009ee10 2024-04-22T08:14:06.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.723+0000 7f409dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:06.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.723+0000 7f409dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:14:06.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.723+0000 7f409dd9b640 10 monclient: dump: 2024-04-22T08:14:06.732 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:06.732 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:06.732 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:06.732 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:06.732 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:06.732 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:06.732 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:06.732 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:06.732 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:06.732 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:06.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.723+0000 7f40a5084640 5 monclient: authenticate success, global_id 4556 2024-04-22T08:14:06.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.723+0000 7f40a5084640 10 monclient: get_monmap_and_config success 2024-04-22T08:14:06.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.723+0000 7f40a5084640 4 set_mon_vals no callback set 2024-04-22T08:14:06.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.723+0000 7f40a5084640 10 monclient: shutdown 2024-04-22T08:14:06.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.723+0000 7f40a5084640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:14:06.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.723+0000 7f40a5084640 1 -- 172.21.15.40:0/3550986779 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f40a009ee10 msgr2=0x7f40a00a32b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:06.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.723+0000 7f40a5084640 1 --2- 172.21.15.40:0/3550986779 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f40a009ee10 0x7f40a00a32b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4090009870 tx=0x7f409002b7f0 comp rx=0 tx=0).stop 2024-04-22T08:14:06.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.723+0000 7f40a5084640 1 -- 172.21.15.40:0/3550986779 shutdown_connections 2024-04-22T08:14:06.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.723+0000 7f40a5084640 1 --2- 172.21.15.40:0/3550986779 >> v2:172.21.15.40:3300/0 conn(0x7f40a00a2150 0x7f40a01364a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:06.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.723+0000 7f40a5084640 1 --2- 172.21.15.40:0/3550986779 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f40a009ee10 0x7f40a00a32b0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:06.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.723+0000 7f40a5084640 1 -- 172.21.15.40:0/3550986779 >> 172.21.15.40:0/3550986779 conn(0x7f40a00a4ef0 msgr2=0x7f40a00a5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:14:06.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.723+0000 7f40a5084640 1 -- 172.21.15.40:0/3550986779 shutdown_connections 2024-04-22T08:14:06.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.723+0000 7f40a5084640 1 -- 172.21.15.40:0/3550986779 wait complete. 2024-04-22T08:14:06.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.724+0000 7f40a5084640 10 monclient: build_initial_monmap 2024-04-22T08:14:06.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.724+0000 7f40a5084640 10 monclient: monmap: 2024-04-22T08:14:06.733 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:14:06.733 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:14:06.734 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:14:06.734 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:14:06.734 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:14:06.734 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:06.734 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:14:06.734 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:14:06.734 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:14:06.734 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:06.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.724+0000 7f40a5084640 1 Processor -- start 2024-04-22T08:14:06.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.724+0000 7f40a5084640 1 -- start start 2024-04-22T08:14:06.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.724+0000 7f40a5084640 10 monclient: init 2024-04-22T08:14:06.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.725+0000 7f40a5084640 10 monclient: _reopen_session rank -1 2024-04-22T08:14:06.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.725+0000 7f40a5084640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-22T08:14:06.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.725+0000 7f40a5084640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40a009ee10 0x7f40a01cb160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:06.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.725+0000 7f40a5084640 10 monclient(hunting): picked mon.noname-b con 0x7f40a009ee10 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:14:06.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.725+0000 7f40a5084640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f40a00a1d30 0x7f40a0122270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:06.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.725+0000 7f40a5084640 10 monclient(hunting): picked mon.noname-c con 0x7f40a00a1d30 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:14:06.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.725+0000 7f40a5084640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40a00a2150 0x7f40a01227b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:06.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.725+0000 7f40a5084640 10 monclient(hunting): picked mon.noname-a con 0x7f40a00a2150 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:14:06.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.725+0000 7f40a5084640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:06.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.725+0000 7f40a5084640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40a0126280 con 0x7f40a00a2150 2024-04-22T08:14:06.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.725+0000 7f40a5084640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:06.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.725+0000 7f40a5084640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40a00a2730 con 0x7f40a009ee10 2024-04-22T08:14:06.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.725+0000 7f40a5084640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:06.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.725+0000 7f40a5084640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40a0136190 con 0x7f40a00a1d30 2024-04-22T08:14:06.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.725+0000 7f40a5084640 10 monclient(hunting): _renew_subs 2024-04-22T08:14:06.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.725+0000 7f40a5084640 10 monclient(hunting): authenticate will time out at 920.719162s 2024-04-22T08:14:06.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.725+0000 7f409f59e640 1 -- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40a00a2150 msgr2=0x7f40a01227b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:14:06.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.725+0000 7f409f59e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40a00a2150 0x7f40a01227b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:14:06.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.725+0000 7f409e59c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f40a00a1d30 0x7f40a0122270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:06.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.725+0000 7f409e59c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f40a00a1d30 0x7f40a0122270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:51596/0 (socket says 172.21.15.40:51596) 2024-04-22T08:14:06.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.725+0000 7f409e59c640 1 -- 172.21.15.40:0/2077391918 learned_addr learned my addr 172.21.15.40:0/2077391918 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:14:06.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.725+0000 7f409e59c640 10 monclient(hunting): get_auth_request con 0x7f40a00a1d30 auth_method 0 2024-04-22T08:14:06.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.725+0000 7f409e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:06.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.725+0000 7f409e59c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:06.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.725+0000 7f409e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:06.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.725+0000 7f409e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:06.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.725+0000 7f409e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:06.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.725+0000 7f409e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:06.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.725+0000 7f409e59c640 10 monclient(hunting): handle_auth_done global_id 4559 payload 824 2024-04-22T08:14:06.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.725+0000 7f409e59c640 1 -- 172.21.15.40:0/2077391918 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40a009ee10 msgr2=0x7f40a01cb160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:06.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.725+0000 7f409ed9d640 1 --2- 172.21.15.40:0/2077391918 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40a009ee10 0x7f40a01cb160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:06.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.725+0000 7f409e59c640 1 --2- 172.21.15.40:0/2077391918 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40a009ee10 0x7f40a01cb160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:06.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.725+0000 7f409e59c640 1 -- 172.21.15.40:0/2077391918 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40a00a2150 msgr2=0x7f40a01227b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:14:06.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.725+0000 7f409e59c640 1 --2- 172.21.15.40:0/2077391918 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40a00a2150 0x7f40a01227b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:06.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.725+0000 7f409e59c640 10 monclient: _finish_hunting 0 2024-04-22T08:14:06.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.725+0000 7f409e59c640 1 monclient: found mon.noname-c 2024-04-22T08:14:06.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.725+0000 7f409e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:14:06.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.725+0000 7f409e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:14:06.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.725+0000 7f409e59c640 1 -- 172.21.15.40:0/2077391918 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40900094d0 con 0x7f40a00a1d30 2024-04-22T08:14:06.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.728+0000 7f409e59c640 1 -- 172.21.15.40:0/2077391918 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f40a00a1d30 msgr2=0x7f40a0122270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:06.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.728+0000 7f409e59c640 1 --2- 172.21.15.40:0/2077391918 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f40a00a1d30 0x7f40a0122270 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f409400b580 tx=0x7f409400ba40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:06.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.728+0000 7f40877fe640 1 -- 172.21.15.40:0/2077391918 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40940050f0 con 0x7f40a00a1d30 2024-04-22T08:14:06.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.728+0000 7f40877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:06.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.728+0000 7f40877fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:14:06.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.728+0000 7f40877fe640 10 monclient: dump: 2024-04-22T08:14:06.738 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:06.738 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:06.738 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:06.738 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:06.738 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:06.738 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:06.738 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:06.738 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:06.738 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:06.738 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:06.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.728+0000 7f40877fe640 10 monclient: _finish_auth 0 2024-04-22T08:14:06.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.728+0000 7f40877fe640 10 monclient: _check_auth_tickets 2024-04-22T08:14:06.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.728+0000 7f40877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:14:06.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.728+0000 7f40877fe640 1 -- 172.21.15.40:0/2077391918 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4094005270 con 0x7f40a00a1d30 2024-04-22T08:14:06.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.728+0000 7f40a5084640 5 monclient: authenticate success, global_id 4559 2024-04-22T08:14:06.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.728+0000 7f40a5084640 10 monclient: _renew_subs 2024-04-22T08:14:06.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.728+0000 7f40a5084640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:14:06.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.728+0000 7f40a5084640 1 -- 172.21.15.40:0/2077391918 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40a0123e20 con 0x7f40a00a1d30 2024-04-22T08:14:06.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.728+0000 7f40877fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:14:06.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.728+0000 7f40877fe640 1 -- 172.21.15.40:0/2077391918 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4094005620 con 0x7f40a00a1d30 2024-04-22T08:14:06.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.728+0000 7f40877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:06.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.728+0000 7f40877fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:14:06.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.728+0000 7f40877fe640 10 monclient: dump: 2024-04-22T08:14:06.739 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:06.739 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:06.739 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:06.739 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:06.739 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:06.740 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:06.740 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:06.740 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:06.740 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:06.740 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:06.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.728+0000 7f409d59a640 4 set_mon_vals no callback set 2024-04-22T08:14:06.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.728+0000 7f40a5084640 10 monclient: _renew_subs 2024-04-22T08:14:06.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.728+0000 7f40a5084640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:14:06.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.728+0000 7f40a5084640 1 -- 172.21.15.40:0/2077391918 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40a0125640 con 0x7f40a00a1d30 2024-04-22T08:14:06.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.729+0000 7f40877fe640 1 -- 172.21.15.40:0/2077391918 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f4094010480 con 0x7f40a00a1d30 2024-04-22T08:14:06.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.729+0000 7f40a5084640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:14:06.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.729+0000 7f40a5084640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:14:06.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.729+0000 7f40a5084640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:14:06.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.729+0000 7f40a5084640 1 -- 172.21.15.40:0/2077391918 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40a0125b90 con 0x7f40a00a1d30 2024-04-22T08:14:06.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.729+0000 7f40877fe640 1 --2- 172.21.15.40:0/2077391918 >> v2:172.21.15.189:6800/30168 conn(0x7f408c0377b0 0x7f408c039c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:06.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.729+0000 7f40877fe640 1 -- 172.21.15.40:0/2077391918 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f409404a3f0 con 0x7f40a00a1d30 2024-04-22T08:14:06.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.729+0000 7f409ed9d640 1 --2- 172.21.15.40:0/2077391918 >> v2:172.21.15.189:6800/30168 conn(0x7f408c0377b0 0x7f408c039c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:06.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.729+0000 7f409ed9d640 10 monclient: get_auth_request con 0x7f408c0377b0 auth_method 0 2024-04-22T08:14:06.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.732+0000 7f409ed9d640 1 --2- 172.21.15.40:0/2077391918 >> v2:172.21.15.189:6800/30168 conn(0x7f408c0377b0 0x7f408c039c70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f40a006c020 tx=0x7f4090008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:06.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.734+0000 7f40877fe640 1 -- 172.21.15.40:0/2077391918 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f409400d720 con 0x7f40a00a1d30 2024-04-22T08:14:06.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.734+0000 7f40877fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:14:06.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.734+0000 7f40877fe640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:14:06.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.988+0000 7f40a5084640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:14:06.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.988+0000 7f40a5084640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:14:06.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.988+0000 7f40a5084640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:14:06.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.988+0000 7f40a5084640 1 -- 172.21.15.40:0/2077391918 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f40a004e9b0 con 0x7f40a00a1d30 2024-04-22T08:14:06.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.988+0000 7f40877fe640 1 -- 172.21.15.40:0/2077391918 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7f409400d360 con 0x7f40a00a1d30 2024-04-22T08:14:06.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.988+0000 7f40877fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:14:06.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.988+0000 7f40877fe640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:14:06.989 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:14:06.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.990+0000 7f40a5084640 1 -- 172.21.15.40:0/2077391918 >> v2:172.21.15.189:6800/30168 conn(0x7f408c0377b0 msgr2=0x7f408c039c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:06.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.990+0000 7f40a5084640 1 --2- 172.21.15.40:0/2077391918 >> v2:172.21.15.189:6800/30168 conn(0x7f408c0377b0 0x7f408c039c70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f40a006c020 tx=0x7f4090008030 comp rx=0 tx=0).stop 2024-04-22T08:14:06.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.990+0000 7f40a5084640 10 monclient: shutdown 2024-04-22T08:14:06.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.990+0000 7f40a5084640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:14:06.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.990+0000 7f40a5084640 1 -- 172.21.15.40:0/2077391918 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f40a00a1d30 msgr2=0x7f40a0122270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:06.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.990+0000 7f40a5084640 1 --2- 172.21.15.40:0/2077391918 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f40a00a1d30 0x7f40a0122270 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f409400b580 tx=0x7f409400ba40 comp rx=0 tx=0).stop 2024-04-22T08:14:06.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.990+0000 7f40a5084640 1 -- 172.21.15.40:0/2077391918 shutdown_connections 2024-04-22T08:14:06.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.990+0000 7f40a5084640 1 --2- 172.21.15.40:0/2077391918 >> v2:172.21.15.189:6800/30168 conn(0x7f408c0377b0 0x7f408c039c70 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:06.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.990+0000 7f40a5084640 1 --2- 172.21.15.40:0/2077391918 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40a00a2150 0x7f40a01227b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:06.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.990+0000 7f40a5084640 1 --2- 172.21.15.40:0/2077391918 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f40a00a1d30 0x7f40a0122270 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:06.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.990+0000 7f40a5084640 1 --2- 172.21.15.40:0/2077391918 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40a009ee10 0x7f40a01cb160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:06.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.990+0000 7f40a5084640 1 -- 172.21.15.40:0/2077391918 >> 172.21.15.40:0/2077391918 conn(0x7f40a00a4ef0 msgr2=0x7f40a006f8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:14:06.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.990+0000 7f40a5084640 1 -- 172.21.15.40:0/2077391918 shutdown_connections 2024-04-22T08:14:06.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:06.990+0000 7f40a5084640 1 -- 172.21.15.40:0/2077391918 wait complete. 2024-04-22T08:14:10.002 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:14:10.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.112+0000 7fb6aa575640 10 monclient: get_monmap_and_config 2024-04-22T08:14:10.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.112+0000 7fb6aa575640 10 monclient: build_initial_monmap 2024-04-22T08:14:10.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.112+0000 7fb6aa575640 10 monclient: monmap: 2024-04-22T08:14:10.112 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:14:10.112 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:10.113 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:14:10.113617+0000 2024-04-22T08:14:10.113 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:14:10.113617+0000 2024-04-22T08:14:10.113 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:14:10.113 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:10.113 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:14:10.113 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:14:10.113 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:14:10.113 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:14:10.113 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:14:10.113 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:10.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.113+0000 7fb6aa575640 1 Processor -- start 2024-04-22T08:14:10.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.113+0000 7fb6aa575640 1 -- start start 2024-04-22T08:14:10.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.113+0000 7fb6aa575640 10 monclient: init 2024-04-22T08:14:10.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.114+0000 7fb6aa575640 10 monclient: _reopen_session rank -1 2024-04-22T08:14:10.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.114+0000 7fb6aa575640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-04-22T08:14:10.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.114+0000 7fb6aa575640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb6a409ee10 addr v1:172.21.15.40:6789/0 2024-04-22T08:14:10.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.114+0000 7fb6aa575640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6a40a32b0 0x7fb6a413b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:10.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.114+0000 7fb6aa575640 10 monclient(hunting): picked mon.noname-b con 0x7fb6a40a32b0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:14:10.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.114+0000 7fb6aa575640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb6a413b5f0 addr v1:172.21.15.189:6789/0 2024-04-22T08:14:10.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.114+0000 7fb6aa575640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6a409f1f0 con 0x7fb6a409ee10 2024-04-22T08:14:10.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.114+0000 7fb6aa575640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6a40a39c0 con 0x7fb6a413b5f0 2024-04-22T08:14:10.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.114+0000 7fb6aa575640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:10.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.114+0000 7fb6aa575640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a409fd30 con 0x7fb6a40a32b0 2024-04-22T08:14:10.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.114+0000 7fb6aa575640 10 monclient(hunting): _renew_subs 2024-04-22T08:14:10.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.114+0000 7fb6aa575640 10 monclient(hunting): authenticate will time out at 924.107918s 2024-04-22T08:14:10.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.114+0000 7fb6a3fff640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7fb6a409ee10 legacy=0x7fb6a40a1d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-04-22T08:14:10.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.114+0000 7fb6a37fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6a40a32b0 0x7fb6a413b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:10.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.114+0000 7fb6a37fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6a40a32b0 0x7fb6a413b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:60908/0 (socket says 172.21.15.40:60908) 2024-04-22T08:14:10.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.114+0000 7fb6a37fe640 1 -- 172.21.15.40:0/2180936420 learned_addr learned my addr 172.21.15.40:0/2180936420 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:14:10.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.114+0000 7fb6a37fe640 10 monclient(hunting): get_auth_request con 0x7fb6a40a32b0 auth_method 0 2024-04-22T08:14:10.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.114+0000 7fb6a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:10.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.114+0000 7fb6a37fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:10.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.114+0000 7fb6a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:10.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.114+0000 7fb6a37fe640 1 -- 172.21.15.40:0/2180936420 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6a40a32b0 msgr2=0x7fb6a413b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:14:10.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.114+0000 7fb6a37fe640 1 -- 172.21.15.40:0/2180936420 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6a40a32b0 msgr2=0x7fb6a413b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:14:10.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.114+0000 7fb6a37fe640 1 --2- 172.21.15.40:0/2180936420 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6a40a32b0 0x7fb6a413b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:14:10.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.114+0000 7fb6a37fe640 1 --2- 172.21.15.40:0/2180936420 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6a40a32b0 0x7fb6a413b0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:14:10.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.116+0000 7fb6a8b12640 1 -- 172.21.15.40:0/2180936420 >> v1:172.21.15.189:6789/0 conn(0x7fb6a413b5f0 legacy=0x7fb6a413d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:10.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.315+0000 7fb6a3fff640 1 -- 172.21.15.40:0/2180936420 >> v1:172.21.15.40:6789/0 conn(0x7fb6a409ee10 legacy=0x7fb6a40a1d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-04-22T08:14:10.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.315+0000 7fb6a37fe640 1 --2- 172.21.15.40:0/2180936420 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6a40a32b0 0x7fb6a413b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:10.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.315+0000 7fb6a37fe640 10 monclient(hunting): get_auth_request con 0x7fb6a40a32b0 auth_method 0 2024-04-22T08:14:10.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.315+0000 7fb6a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:10.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.315+0000 7fb6a37fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:10.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.315+0000 7fb6a37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:14:10.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.315+0000 7fb6a37fe640 1 -- 172.21.15.40:0/2180936420 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6a40a32b0 msgr2=0x7fb6a413b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:14:10.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.315+0000 7fb6a37fe640 1 -- 172.21.15.40:0/2180936420 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6a40a32b0 msgr2=0x7fb6a413b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:14:10.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.315+0000 7fb6a37fe640 1 --2- 172.21.15.40:0/2180936420 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6a40a32b0 0x7fb6a413b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:14:10.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.315+0000 7fb6a37fe640 1 --2- 172.21.15.40:0/2180936420 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6a40a32b0 0x7fb6a413b0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:14:10.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.716+0000 7fb6a3fff640 1 -- 172.21.15.40:0/2180936420 >> v1:172.21.15.40:6789/0 conn(0x7fb6a409ee10 legacy=0x7fb6a40a1d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-04-22T08:14:10.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.716+0000 7fb6a37fe640 1 --2- 172.21.15.40:0/2180936420 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6a40a32b0 0x7fb6a413b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:10.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.716+0000 7fb6a37fe640 10 monclient(hunting): get_auth_request con 0x7fb6a40a32b0 auth_method 0 2024-04-22T08:14:10.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.716+0000 7fb6a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:10.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.716+0000 7fb6a37fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:10.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.716+0000 7fb6a37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:14:10.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.716+0000 7fb6a37fe640 1 -- 172.21.15.40:0/2180936420 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6a40a32b0 msgr2=0x7fb6a413b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:14:10.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.716+0000 7fb6a37fe640 1 -- 172.21.15.40:0/2180936420 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6a40a32b0 msgr2=0x7fb6a413b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:14:10.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.716+0000 7fb6a37fe640 1 --2- 172.21.15.40:0/2180936420 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6a40a32b0 0x7fb6a413b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:14:10.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:10.716+0000 7fb6a37fe640 1 --2- 172.21.15.40:0/2180936420 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6a40a32b0 0x7fb6a413b0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:14:11.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:11.517+0000 7fb6a3fff640 1 -- 172.21.15.40:0/2180936420 >> v1:172.21.15.40:6789/0 conn(0x7fb6a409ee10 legacy=0x7fb6a40a1d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-04-22T08:14:11.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:11.517+0000 7fb6a37fe640 1 --2- 172.21.15.40:0/2180936420 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6a40a32b0 0x7fb6a413b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:11.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:11.517+0000 7fb6a37fe640 10 monclient(hunting): get_auth_request con 0x7fb6a40a32b0 auth_method 0 2024-04-22T08:14:11.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:11.517+0000 7fb6a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:11.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:11.517+0000 7fb6a37fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:11.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:11.517+0000 7fb6a37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:14:11.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:11.517+0000 7fb6a37fe640 1 -- 172.21.15.40:0/2180936420 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6a40a32b0 msgr2=0x7fb6a413b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:14:11.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:11.517+0000 7fb6a37fe640 1 -- 172.21.15.40:0/2180936420 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6a40a32b0 msgr2=0x7fb6a413b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:14:11.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:11.517+0000 7fb6a37fe640 1 --2- 172.21.15.40:0/2180936420 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6a40a32b0 0x7fb6a413b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:14:11.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:11.517+0000 7fb6a37fe640 1 --2- 172.21.15.40:0/2180936420 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6a40a32b0 0x7fb6a413b0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T08:14:13.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.114+0000 7fb6a1ffb640 10 monclient(hunting): tick 2024-04-22T08:14:13.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.114+0000 7fb6a1ffb640 10 monclient(hunting): _check_auth_tickets 2024-04-22T08:14:13.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.114+0000 7fb6a1ffb640 1 monclient(hunting): continuing hunt 2024-04-22T08:14:13.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.114+0000 7fb6a1ffb640 10 monclient(hunting): _reopen_session rank -1 2024-04-22T08:14:13.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.114+0000 7fb6a1ffb640 1 -- 172.21.15.40:0/2180936420 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6a40a32b0 msgr2=0x7fb6a413b0b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:14:13.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.114+0000 7fb6a1ffb640 1 --2- 172.21.15.40:0/2180936420 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6a40a32b0 0x7fb6a413b0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:13.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.114+0000 7fb6a1ffb640 1 -- 172.21.15.40:0/2180936420 >> v1:172.21.15.189:6789/0 conn(0x7fb6a413b5f0 legacy=0x7fb6a413d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:13.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.114+0000 7fb6a1ffb640 1 -- 172.21.15.40:0/2180936420 >> v1:172.21.15.40:6789/0 conn(0x7fb6a409ee10 legacy=0x7fb6a40a1d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:14:13.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.114+0000 7fb6a1ffb640 10 monclient: _add_conns ranks=[4,3] 2024-04-22T08:14:13.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.114+0000 7fb6a1ffb640 1 --2- 172.21.15.40:0/2180936420 >> v2:172.21.15.189:3300/0 conn(0x7fb684003740 0x7fb684005bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:13.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.114+0000 7fb6a1ffb640 10 monclient(hunting): picked mon.noname-c con 0x7fb684003740 addr v2:172.21.15.189:3300/0 2024-04-22T08:14:13.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.114+0000 7fb6a1ffb640 1 --2- 172.21.15.40:0/2180936420 >> v2:172.21.15.40:3300/0 conn(0x7fb6840062b0 0x7fb684008770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:13.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.114+0000 7fb6a1ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fb6840062b0 addr v2:172.21.15.40:3300/0 2024-04-22T08:14:13.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.114+0000 7fb6a1ffb640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:13.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.114+0000 7fb6a1ffb640 1 -- 172.21.15.40:0/2180936420 --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6a409fd30 con 0x7fb6840062b0 2024-04-22T08:14:13.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.114+0000 7fb6a1ffb640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:13.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.114+0000 7fb6a1ffb640 1 -- 172.21.15.40:0/2180936420 --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb684008e30 con 0x7fb684003740 2024-04-22T08:14:13.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.114+0000 7fb6a1ffb640 10 monclient(hunting): _renew_subs 2024-04-22T08:14:13.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.114+0000 7fb6a37fe640 1 -- 172.21.15.40:0/2180936420 >> v2:172.21.15.40:3300/0 conn(0x7fb6840062b0 msgr2=0x7fb684008770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:14:13.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.114+0000 7fb6a37fe640 1 --2- 172.21.15.40:0/2180936420 >> v2:172.21.15.40:3300/0 conn(0x7fb6840062b0 0x7fb684008770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:14:13.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.114+0000 7fb6a3fff640 1 --2- 172.21.15.40:0/2180936420 >> v2:172.21.15.189:3300/0 conn(0x7fb684003740 0x7fb684005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:13.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.114+0000 7fb6a3fff640 10 monclient(hunting): get_auth_request con 0x7fb684003740 auth_method 0 2024-04-22T08:14:13.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.114+0000 7fb6a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:13.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.114+0000 7fb6a3fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:13.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.114+0000 7fb6a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:13.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.114+0000 7fb6a3fff640 1 -- 172.21.15.40:0/2180936420 >> v2:172.21.15.189:3300/0 conn(0x7fb684003740 msgr2=0x7fb684005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:14:13.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.114+0000 7fb6a3fff640 1 -- 172.21.15.40:0/2180936420 >> v2:172.21.15.189:3300/0 conn(0x7fb684003740 msgr2=0x7fb684005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:14:13.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.114+0000 7fb6a3fff640 1 --2- 172.21.15.40:0/2180936420 >> v2:172.21.15.189:3300/0 conn(0x7fb684003740 0x7fb684005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:14:13.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.114+0000 7fb6a3fff640 1 --2- 172.21.15.40:0/2180936420 >> v2:172.21.15.189:3300/0 conn(0x7fb684003740 0x7fb684005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:14:13.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.315+0000 7fb6a37fe640 1 -- 172.21.15.40:0/2180936420 >> v2:172.21.15.40:3300/0 conn(0x7fb6840062b0 msgr2=0x7fb684008770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:14:13.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.315+0000 7fb6a37fe640 1 --2- 172.21.15.40:0/2180936420 >> v2:172.21.15.40:3300/0 conn(0x7fb6840062b0 0x7fb684008770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:14:13.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.315+0000 7fb6a3fff640 1 --2- 172.21.15.40:0/2180936420 >> v2:172.21.15.189:3300/0 conn(0x7fb684003740 0x7fb684005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:13.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.315+0000 7fb6a3fff640 10 monclient(hunting): get_auth_request con 0x7fb684003740 auth_method 0 2024-04-22T08:14:13.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.315+0000 7fb6a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:13.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.315+0000 7fb6a3fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:13.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.315+0000 7fb6a3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:14:13.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.315+0000 7fb6a3fff640 1 -- 172.21.15.40:0/2180936420 >> v2:172.21.15.189:3300/0 conn(0x7fb684003740 msgr2=0x7fb684005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:14:13.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.315+0000 7fb6a3fff640 1 -- 172.21.15.40:0/2180936420 >> v2:172.21.15.189:3300/0 conn(0x7fb684003740 msgr2=0x7fb684005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:14:13.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.315+0000 7fb6a3fff640 1 --2- 172.21.15.40:0/2180936420 >> v2:172.21.15.189:3300/0 conn(0x7fb684003740 0x7fb684005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:14:13.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.315+0000 7fb6a3fff640 1 --2- 172.21.15.40:0/2180936420 >> v2:172.21.15.189:3300/0 conn(0x7fb684003740 0x7fb684005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:14:13.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.716+0000 7fb6a37fe640 1 -- 172.21.15.40:0/2180936420 >> v2:172.21.15.40:3300/0 conn(0x7fb6840062b0 msgr2=0x7fb684008770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:14:13.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.716+0000 7fb6a37fe640 1 --2- 172.21.15.40:0/2180936420 >> v2:172.21.15.40:3300/0 conn(0x7fb6840062b0 0x7fb684008770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:14:13.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.716+0000 7fb6a3fff640 1 --2- 172.21.15.40:0/2180936420 >> v2:172.21.15.189:3300/0 conn(0x7fb684003740 0x7fb684005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:13.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.716+0000 7fb6a3fff640 10 monclient(hunting): get_auth_request con 0x7fb684003740 auth_method 0 2024-04-22T08:14:13.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.716+0000 7fb6a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:13.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.716+0000 7fb6a3fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:13.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.716+0000 7fb6a3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:14:13.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.716+0000 7fb6a3fff640 1 -- 172.21.15.40:0/2180936420 >> v2:172.21.15.189:3300/0 conn(0x7fb684003740 msgr2=0x7fb684005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:14:13.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.716+0000 7fb6a3fff640 1 -- 172.21.15.40:0/2180936420 >> v2:172.21.15.189:3300/0 conn(0x7fb684003740 msgr2=0x7fb684005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:14:13.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.716+0000 7fb6a3fff640 1 --2- 172.21.15.40:0/2180936420 >> v2:172.21.15.189:3300/0 conn(0x7fb684003740 0x7fb684005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:14:13.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:13.716+0000 7fb6a3fff640 1 --2- 172.21.15.40:0/2180936420 >> v2:172.21.15.189:3300/0 conn(0x7fb684003740 0x7fb684005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:14:14.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:14.517+0000 7fb6a37fe640 1 -- 172.21.15.40:0/2180936420 >> v2:172.21.15.40:3300/0 conn(0x7fb6840062b0 msgr2=0x7fb684008770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:14:14.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:14.517+0000 7fb6a37fe640 1 --2- 172.21.15.40:0/2180936420 >> v2:172.21.15.40:3300/0 conn(0x7fb6840062b0 0x7fb684008770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T08:14:14.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:14.517+0000 7fb6a3fff640 1 --2- 172.21.15.40:0/2180936420 >> v2:172.21.15.189:3300/0 conn(0x7fb684003740 0x7fb684005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:14.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:14.517+0000 7fb6a3fff640 10 monclient(hunting): get_auth_request con 0x7fb684003740 auth_method 0 2024-04-22T08:14:14.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:14.517+0000 7fb6a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:14.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:14.517+0000 7fb6a3fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:14.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:14.517+0000 7fb6a3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:14:14.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:14.517+0000 7fb6a3fff640 1 -- 172.21.15.40:0/2180936420 >> v2:172.21.15.189:3300/0 conn(0x7fb684003740 msgr2=0x7fb684005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:14:14.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:14.517+0000 7fb6a3fff640 1 -- 172.21.15.40:0/2180936420 >> v2:172.21.15.189:3300/0 conn(0x7fb684003740 msgr2=0x7fb684005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:14:14.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:14.517+0000 7fb6a3fff640 1 --2- 172.21.15.40:0/2180936420 >> v2:172.21.15.189:3300/0 conn(0x7fb684003740 0x7fb684005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:14:14.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:14.517+0000 7fb6a3fff640 1 --2- 172.21.15.40:0/2180936420 >> v2:172.21.15.189:3300/0 conn(0x7fb684003740 0x7fb684005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T08:14:14.853 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[34849]: starting. 2024-04-22T08:14:14.854 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[34849]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-34848 2024-04-22T08:14:14.854 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[34849]: created object 0... 2024-04-22T08:14:14.854 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[34849]: created object 25... 2024-04-22T08:14:14.854 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[34849]: created object 49... 2024-04-22T08:14:14.854 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[34849]: finishing. 2024-04-22T08:14:14.854 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[34849]: shutting down. 2024-04-22T08:14:15.733 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[34850]: starting. 2024-04-22T08:14:15.733 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[34850]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-34848 2024-04-22T08:14:15.733 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[34850]: shutting down. 2024-04-22T08:14:16.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.114+0000 7fb6a1ffb640 10 monclient(hunting): tick 2024-04-22T08:14:16.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.114+0000 7fb6a1ffb640 10 monclient(hunting): _check_auth_tickets 2024-04-22T08:14:16.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.114+0000 7fb6a1ffb640 1 monclient(hunting): continuing hunt 2024-04-22T08:14:16.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.114+0000 7fb6a1ffb640 10 monclient(hunting): _reopen_session rank -1 2024-04-22T08:14:16.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.114+0000 7fb6a1ffb640 1 -- 172.21.15.40:0/2180936420 >> v2:172.21.15.189:3300/0 conn(0x7fb684003740 msgr2=0x7fb684005bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:14:16.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.114+0000 7fb6a1ffb640 1 --2- 172.21.15.40:0/2180936420 >> v2:172.21.15.189:3300/0 conn(0x7fb684003740 0x7fb684005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:16.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.114+0000 7fb6a1ffb640 1 -- 172.21.15.40:0/2180936420 >> v2:172.21.15.40:3300/0 conn(0x7fb6840062b0 msgr2=0x7fb684008770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:14:16.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.114+0000 7fb6a1ffb640 1 --2- 172.21.15.40:0/2180936420 >> v2:172.21.15.40:3300/0 conn(0x7fb6840062b0 0x7fb684008770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:16.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.114+0000 7fb6a1ffb640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-04-22T08:14:16.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.114+0000 7fb6a8b12640 1 -- 172.21.15.40:0/2180936420 reap_dead start 2024-04-22T08:14:16.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.114+0000 7fb6a1ffb640 1 --2- 172.21.15.40:0/2180936420 >> v2:172.21.15.189:3300/0 conn(0x7fb684009200 0x7fb68400b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:16.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.114+0000 7fb6a1ffb640 10 monclient(hunting): picked mon.noname-c con 0x7fb684009200 addr v2:172.21.15.189:3300/0 2024-04-22T08:14:16.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.114+0000 7fb6a1ffb640 1 --2- 172.21.15.40:0/2180936420 >> v2:172.21.15.40:3300/0 conn(0x7fb6840062b0 0x7fb684008770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:16.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.114+0000 7fb6a1ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fb6840062b0 addr v2:172.21.15.40:3300/0 2024-04-22T08:14:16.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.114+0000 7fb6a1ffb640 1 --2- 172.21.15.40:0/2180936420 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb684003b50 0x7fb68400bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:16.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.114+0000 7fb6a1ffb640 10 monclient(hunting): picked mon.noname-b con 0x7fb684003b50 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:14:16.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.114+0000 7fb6a1ffb640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:16.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.114+0000 7fb6a1ffb640 1 -- 172.21.15.40:0/2180936420 --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6a409fd30 con 0x7fb6840062b0 2024-04-22T08:14:16.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.114+0000 7fb6a1ffb640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:16.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.114+0000 7fb6a1ffb640 1 -- 172.21.15.40:0/2180936420 --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb684008e30 con 0x7fb684009200 2024-04-22T08:14:16.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.114+0000 7fb6a1ffb640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:16.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.114+0000 7fb6a1ffb640 1 -- 172.21.15.40:0/2180936420 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb68400c240 con 0x7fb684003b50 2024-04-22T08:14:16.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.114+0000 7fb6a1ffb640 10 monclient(hunting): _renew_subs 2024-04-22T08:14:16.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.114+0000 7fb6a37fe640 1 -- 172.21.15.40:0/2180936420 >> v2:172.21.15.40:3300/0 conn(0x7fb6840062b0 msgr2=0x7fb684008770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:14:16.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.114+0000 7fb6a37fe640 1 --2- 172.21.15.40:0/2180936420 >> v2:172.21.15.40:3300/0 conn(0x7fb6840062b0 0x7fb684008770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:14:16.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.114+0000 7fb6a8b12640 1 --2- 172.21.15.40:0/2180936420 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb684003b50 0x7fb68400bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:16.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.114+0000 7fb6a8b12640 10 monclient(hunting): get_auth_request con 0x7fb684003b50 auth_method 0 2024-04-22T08:14:16.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.114+0000 7fb6a8b12640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:16.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.114+0000 7fb6a8b12640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:16.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.114+0000 7fb6a8b12640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:16.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.114+0000 7fb6a3fff640 1 --2- 172.21.15.40:0/2180936420 >> v2:172.21.15.189:3300/0 conn(0x7fb684009200 0x7fb68400b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:16.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.115+0000 7fb6a3fff640 10 monclient(hunting): get_auth_request con 0x7fb684009200 auth_method 0 2024-04-22T08:14:16.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.115+0000 7fb6a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:16.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.115+0000 7fb6a3fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:16.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.115+0000 7fb6a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:16.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.115+0000 7fb6a8b12640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:16.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.115+0000 7fb6a8b12640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:16.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.115+0000 7fb6a8b12640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:16.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.115+0000 7fb6a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:16.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.115+0000 7fb6a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:16.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.115+0000 7fb6a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:16.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.115+0000 7fb6a8b12640 10 monclient(hunting): handle_auth_done global_id 4577 payload 274 2024-04-22T08:14:16.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.115+0000 7fb6a8b12640 1 -- 172.21.15.40:0/2180936420 >> v2:172.21.15.189:3300/0 conn(0x7fb684009200 msgr2=0x7fb68400b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:14:16.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.115+0000 7fb6a8b12640 1 --2- 172.21.15.40:0/2180936420 >> v2:172.21.15.189:3300/0 conn(0x7fb684009200 0x7fb68400b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:16.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.115+0000 7fb6a8b12640 1 -- 172.21.15.40:0/2180936420 >> v2:172.21.15.40:3300/0 conn(0x7fb6840062b0 msgr2=0x7fb684008770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:14:16.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.115+0000 7fb6a8b12640 1 --2- 172.21.15.40:0/2180936420 >> v2:172.21.15.40:3300/0 conn(0x7fb6840062b0 0x7fb684008770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:16.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.115+0000 7fb6a8b12640 10 monclient: _finish_hunting 0 2024-04-22T08:14:16.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.115+0000 7fb6a8b12640 1 monclient: found mon.noname-b 2024-04-22T08:14:16.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.115+0000 7fb6a8b12640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:14:16.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.115+0000 7fb6a8b12640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:14:16.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.115+0000 7fb6a8b12640 1 -- 172.21.15.40:0/2180936420 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a40a02f0 con 0x7fb684003b50 2024-04-22T08:14:16.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.118+0000 7fb6a8b12640 1 -- 172.21.15.40:0/2180936420 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb684003b50 msgr2=0x7fb68400bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:16.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.118+0000 7fb6a8b12640 1 --2- 172.21.15.40:0/2180936420 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb684003b50 0x7fb68400bc30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb6a40a4650 tx=0x7fb69c02b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9fc075d9aa2bbac server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:16.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.118+0000 7fb6a2ffd640 1 -- 172.21.15.40:0/2180936420 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb69c007070 con 0x7fb684003b50 2024-04-22T08:14:16.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.118+0000 7fb6a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:16.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.118+0000 7fb6a2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:14:16.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.118+0000 7fb6a2ffd640 10 monclient: dump: 2024-04-22T08:14:16.120 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:16.121 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:16.121 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:16.121 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:16.121 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:16.121 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:16.121 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:16.121 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:16.121 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:16.121 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:16.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.118+0000 7fb6a2ffd640 10 monclient: _finish_auth 0 2024-04-22T08:14:16.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.118+0000 7fb6a2ffd640 10 monclient: _check_auth_tickets 2024-04-22T08:14:16.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.118+0000 7fb6a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:14:16.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.118+0000 7fb6a2ffd640 1 -- 172.21.15.40:0/2180936420 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb69c0071d0 con 0x7fb684003b50 2024-04-22T08:14:16.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.118+0000 7fb6aa575640 5 monclient: authenticate success, global_id 4577 2024-04-22T08:14:16.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.118+0000 7fb6aa575640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:14:16.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.118+0000 7fb6a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:14:16.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.118+0000 7fb6aa575640 10 monclient: get_monmap_and_config success 2024-04-22T08:14:16.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.118+0000 7fb6aa575640 4 set_mon_vals no callback set 2024-04-22T08:14:16.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.118+0000 7fb6a2ffd640 1 -- 172.21.15.40:0/2180936420 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb69c007540 con 0x7fb684003b50 2024-04-22T08:14:16.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.118+0000 7fb6a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:16.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.118+0000 7fb6a2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:14:16.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.118+0000 7fb6a2ffd640 10 monclient: dump: 2024-04-22T08:14:16.122 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:16.122 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:16.122 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:16.122 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:16.122 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:16.122 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:16.122 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:16.122 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:16.123 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:16.123 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:16.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.119+0000 7fb6aa575640 10 monclient: shutdown 2024-04-22T08:14:16.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.119+0000 7fb6aa575640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:14:16.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.119+0000 7fb6aa575640 1 -- 172.21.15.40:0/2180936420 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb684003b50 msgr2=0x7fb68400bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:16.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.119+0000 7fb6aa575640 1 --2- 172.21.15.40:0/2180936420 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb684003b50 0x7fb68400bc30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb6a40a4650 tx=0x7fb69c02b820 comp rx=0 tx=0).stop 2024-04-22T08:14:16.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.119+0000 7fb6aa575640 1 -- 172.21.15.40:0/2180936420 shutdown_connections 2024-04-22T08:14:16.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.119+0000 7fb6aa575640 1 --2- 172.21.15.40:0/2180936420 >> v2:172.21.15.189:3300/0 conn(0x7fb684009200 0x7fb68400b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:16.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.119+0000 7fb6aa575640 1 --2- 172.21.15.40:0/2180936420 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb684003b50 0x7fb68400bc30 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:16.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.119+0000 7fb6aa575640 1 --2- 172.21.15.40:0/2180936420 >> v2:172.21.15.40:3300/0 conn(0x7fb6840062b0 0x7fb684008770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:16.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.119+0000 7fb6aa575640 1 -- 172.21.15.40:0/2180936420 >> 172.21.15.40:0/2180936420 conn(0x7fb6a40a4ef0 msgr2=0x7fb6a40a5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:14:16.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.120+0000 7fb6aa575640 1 -- 172.21.15.40:0/2180936420 shutdown_connections 2024-04-22T08:14:16.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.120+0000 7fb6aa575640 1 -- 172.21.15.40:0/2180936420 wait complete. 2024-04-22T08:14:16.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.121+0000 7fb6aa575640 10 monclient: build_initial_monmap 2024-04-22T08:14:16.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.121+0000 7fb6aa575640 10 monclient: monmap: 2024-04-22T08:14:16.128 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:14:16.128 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:14:16.128 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:14:16.128 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:14:16.128 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:14:16.128 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:16.129 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:14:16.129 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:14:16.129 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:14:16.129 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:16.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.121+0000 7fb6aa575640 1 Processor -- start 2024-04-22T08:14:16.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.122+0000 7fb6aa575640 1 -- start start 2024-04-22T08:14:16.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.122+0000 7fb6aa575640 10 monclient: init 2024-04-22T08:14:16.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.122+0000 7fb6aa575640 10 monclient: _reopen_session rank -1 2024-04-22T08:14:16.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.122+0000 7fb6aa575640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-22T08:14:16.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.123+0000 7fb6aa575640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb684003b50 0x7fb6a41379c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:16.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.123+0000 7fb6aa575640 10 monclient(hunting): picked mon.noname-b con 0x7fb684003b50 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:14:16.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.123+0000 7fb6aa575640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6840062b0 0x7fb6a4120410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:16.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.123+0000 7fb6aa575640 10 monclient(hunting): picked mon.noname-a con 0x7fb6840062b0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:14:16.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.123+0000 7fb6aa575640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb684009200 0x7fb6a4120950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:16.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.123+0000 7fb6aa575640 10 monclient(hunting): picked mon.noname-c con 0x7fb684009200 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:14:16.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.123+0000 7fb6aa575640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:16.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.123+0000 7fb6aa575640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a41cb630 con 0x7fb6840062b0 2024-04-22T08:14:16.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.123+0000 7fb6aa575640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:16.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.123+0000 7fb6aa575640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a4123ad0 con 0x7fb684003b50 2024-04-22T08:14:16.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.123+0000 7fb6aa575640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:16.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.123+0000 7fb6aa575640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a4122c50 con 0x7fb684009200 2024-04-22T08:14:16.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.123+0000 7fb6aa575640 10 monclient(hunting): _renew_subs 2024-04-22T08:14:16.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.123+0000 7fb6aa575640 10 monclient(hunting): authenticate will time out at 930.116853s 2024-04-22T08:14:16.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.123+0000 7fb6a37fe640 1 -- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6840062b0 msgr2=0x7fb6a4120410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:14:16.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.123+0000 7fb6a37fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6840062b0 0x7fb6a4120410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:14:16.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.123+0000 7fb6a8b12640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb684009200 0x7fb6a4120950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:16.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.123+0000 7fb6a8b12640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb684009200 0x7fb6a4120950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:60948/0 (socket says 172.21.15.40:60948) 2024-04-22T08:14:16.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.123+0000 7fb6a8b12640 1 -- 172.21.15.40:0/2263804986 learned_addr learned my addr 172.21.15.40:0/2263804986 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:14:16.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.123+0000 7fb6a8b12640 10 monclient(hunting): get_auth_request con 0x7fb684009200 auth_method 0 2024-04-22T08:14:16.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.123+0000 7fb6a8b12640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:16.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.123+0000 7fb6a8b12640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:16.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.123+0000 7fb6a8b12640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:16.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.123+0000 7fb6a8b12640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:16.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.123+0000 7fb6a8b12640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:16.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.123+0000 7fb6a8b12640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:16.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.123+0000 7fb6a3fff640 1 --2- 172.21.15.40:0/2263804986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb684003b50 0x7fb6a41379c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:16.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.123+0000 7fb6a3fff640 10 monclient(hunting): get_auth_request con 0x7fb684003b50 auth_method 0 2024-04-22T08:14:16.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.123+0000 7fb6a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:16.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.123+0000 7fb6a3fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:16.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.123+0000 7fb6a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:16.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.123+0000 7fb6a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:16.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.123+0000 7fb6a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:16.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.123+0000 7fb6a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:16.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.123+0000 7fb6a8b12640 10 monclient(hunting): handle_auth_done global_id 4580 payload 824 2024-04-22T08:14:16.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.123+0000 7fb6a8b12640 1 -- 172.21.15.40:0/2263804986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb684003b50 msgr2=0x7fb6a41379c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:16.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.123+0000 7fb6a8b12640 1 --2- 172.21.15.40:0/2263804986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb684003b50 0x7fb6a41379c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:16.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.123+0000 7fb6a8b12640 1 -- 172.21.15.40:0/2263804986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6840062b0 msgr2=0x7fb6a4120410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:14:16.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.123+0000 7fb6a8b12640 1 --2- 172.21.15.40:0/2263804986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6840062b0 0x7fb6a4120410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:16.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.123+0000 7fb6a8b12640 10 monclient: _finish_hunting 0 2024-04-22T08:14:16.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.123+0000 7fb6a8b12640 1 monclient: found mon.noname-c 2024-04-22T08:14:16.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.123+0000 7fb6a8b12640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:14:16.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.123+0000 7fb6a8b12640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:14:16.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.123+0000 7fb6a8b12640 1 -- 172.21.15.40:0/2263804986 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a40a39c0 con 0x7fb684009200 2024-04-22T08:14:16.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.126+0000 7fb6a8b12640 1 -- 172.21.15.40:0/2263804986 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb684009200 msgr2=0x7fb6a4120950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:16.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.126+0000 7fb6a8b12640 1 --2- 172.21.15.40:0/2263804986 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb684009200 0x7fb6a4120950 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb6a409fa50 tx=0x7fb69c001830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:16.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.126+0000 7fb6a0ff9640 1 -- 172.21.15.40:0/2263804986 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb69c03e840 con 0x7fb684009200 2024-04-22T08:14:16.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.126+0000 7fb6a0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:16.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.126+0000 7fb6a0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:14:16.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.126+0000 7fb6a0ff9640 10 monclient: dump: 2024-04-22T08:14:16.133 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:16.133 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:16.133 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:16.133 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:16.133 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:16.133 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:16.133 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:16.133 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:16.133 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:16.133 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:16.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.126+0000 7fb6a0ff9640 10 monclient: _finish_auth 0 2024-04-22T08:14:16.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.126+0000 7fb6a0ff9640 10 monclient: _check_auth_tickets 2024-04-22T08:14:16.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.126+0000 7fb6a0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:14:16.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.126+0000 7fb6aa575640 5 monclient: authenticate success, global_id 4580 2024-04-22T08:14:16.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.126+0000 7fb6a0ff9640 1 -- 172.21.15.40:0/2263804986 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb69c03e9a0 con 0x7fb684009200 2024-04-22T08:14:16.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.126+0000 7fb6aa575640 10 monclient: _renew_subs 2024-04-22T08:14:16.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.126+0000 7fb6aa575640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:14:16.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.126+0000 7fb6aa575640 1 -- 172.21.15.40:0/2263804986 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6a40a2730 con 0x7fb684009200 2024-04-22T08:14:16.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.126+0000 7fb6a0ff9640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:14:16.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.126+0000 7fb6a0ff9640 1 -- 172.21.15.40:0/2263804986 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb69c03ed50 con 0x7fb684009200 2024-04-22T08:14:16.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.126+0000 7fb6a0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:16.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.126+0000 7fb6a0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:14:16.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.126+0000 7fb6a0ff9640 10 monclient: dump: 2024-04-22T08:14:16.134 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:16.135 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:16.135 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:16.135 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:16.135 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:16.135 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:16.135 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:16.135 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:16.135 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:16.135 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:16.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.126+0000 7fb6aa575640 10 monclient: _renew_subs 2024-04-22T08:14:16.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.126+0000 7fb6a27fc640 4 set_mon_vals no callback set 2024-04-22T08:14:16.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.126+0000 7fb6aa575640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:14:16.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.126+0000 7fb6aa575640 1 -- 172.21.15.40:0/2263804986 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6a4138cd0 con 0x7fb684009200 2024-04-22T08:14:16.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.127+0000 7fb6a0ff9640 1 -- 172.21.15.40:0/2263804986 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fb69c0323f0 con 0x7fb684009200 2024-04-22T08:14:16.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.127+0000 7fb6aa575640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:14:16.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.127+0000 7fb6aa575640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:14:16.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.127+0000 7fb6aa575640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:14:16.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.127+0000 7fb6aa575640 1 -- 172.21.15.40:0/2263804986 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6a41220d0 con 0x7fb684009200 2024-04-22T08:14:16.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.127+0000 7fb6a0ff9640 1 --2- 172.21.15.40:0/2263804986 >> v2:172.21.15.189:6800/30168 conn(0x7fb684039070 0x7fb68403b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:16.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.128+0000 7fb6a0ff9640 1 -- 172.21.15.40:0/2263804986 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 3506+0+0 (secure 0 0 0) 0x7fb69c06c150 con 0x7fb684009200 2024-04-22T08:14:16.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.128+0000 7fb6a3fff640 1 --2- 172.21.15.40:0/2263804986 >> v2:172.21.15.189:6800/30168 conn(0x7fb684039070 0x7fb68403b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:16.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.128+0000 7fb6a3fff640 10 monclient: get_auth_request con 0x7fb684039070 auth_method 0 2024-04-22T08:14:16.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.131+0000 7fb6a3fff640 1 --2- 172.21.15.40:0/2263804986 >> v2:172.21.15.189:6800/30168 conn(0x7fb684039070 0x7fb68403b530 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb694003950 tx=0x7fb694005040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:16.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.133+0000 7fb6a0ff9640 1 -- 172.21.15.40:0/2263804986 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7fb6a41220d0 con 0x7fb684009200 2024-04-22T08:14:16.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.133+0000 7fb6a0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:14:16.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.133+0000 7fb6a0ff9640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:14:16.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.387+0000 7fb6aa575640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:14:16.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.387+0000 7fb6aa575640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:14:16.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.387+0000 7fb6aa575640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:14:16.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.387+0000 7fb6aa575640 1 -- 172.21.15.40:0/2263804986 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb6a41220d0 con 0x7fb684009200 2024-04-22T08:14:16.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.388+0000 7fb6a0ff9640 1 -- 172.21.15.40:0/2263804986 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1537 (secure 0 0 0) 0x7fb6a41220d0 con 0x7fb684009200 2024-04-22T08:14:16.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.388+0000 7fb6a0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:14:16.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.388+0000 7fb6a0ff9640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:14:16.388 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":22,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:14:16.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.389+0000 7fb6aa575640 1 -- 172.21.15.40:0/2263804986 >> v2:172.21.15.189:6800/30168 conn(0x7fb684039070 msgr2=0x7fb68403b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:16.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.390+0000 7fb6aa575640 1 --2- 172.21.15.40:0/2263804986 >> v2:172.21.15.189:6800/30168 conn(0x7fb684039070 0x7fb68403b530 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb694003950 tx=0x7fb694005040 comp rx=0 tx=0).stop 2024-04-22T08:14:16.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.390+0000 7fb6aa575640 10 monclient: shutdown 2024-04-22T08:14:16.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.390+0000 7fb6aa575640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:14:16.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.390+0000 7fb6aa575640 1 -- 172.21.15.40:0/2263804986 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb684009200 msgr2=0x7fb6a4120950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:16.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.390+0000 7fb6aa575640 1 --2- 172.21.15.40:0/2263804986 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb684009200 0x7fb6a4120950 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb6a409fa50 tx=0x7fb69c001830 comp rx=0 tx=0).stop 2024-04-22T08:14:16.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.390+0000 7fb6aa575640 1 -- 172.21.15.40:0/2263804986 shutdown_connections 2024-04-22T08:14:16.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.390+0000 7fb6aa575640 1 --2- 172.21.15.40:0/2263804986 >> v2:172.21.15.189:6800/30168 conn(0x7fb684039070 0x7fb68403b530 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:16.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.390+0000 7fb6aa575640 1 --2- 172.21.15.40:0/2263804986 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb684009200 0x7fb6a4120950 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:16.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.390+0000 7fb6aa575640 1 --2- 172.21.15.40:0/2263804986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6840062b0 0x7fb6a4120410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:16.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.390+0000 7fb6aa575640 1 --2- 172.21.15.40:0/2263804986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb684003b50 0x7fb6a41379c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:16.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.390+0000 7fb6aa575640 1 -- 172.21.15.40:0/2263804986 >> 172.21.15.40:0/2263804986 conn(0x7fb6a40a4ef0 msgr2=0x7fb6a406d300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:14:16.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.390+0000 7fb6aa575640 1 -- 172.21.15.40:0/2263804986 shutdown_connections 2024-04-22T08:14:16.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.390+0000 7fb6aa575640 1 -- 172.21.15.40:0/2263804986 wait complete. 2024-04-22T08:14:16.400 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-22T08:14:16.400 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-04-22T08:14:16.401 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-22T08:14:16.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.502+0000 7f9d7fdb4640 10 monclient: get_monmap_and_config 2024-04-22T08:14:16.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.502+0000 7f9d7fdb4640 10 monclient: build_initial_monmap 2024-04-22T08:14:16.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.502+0000 7f9d7fdb4640 10 monclient: monmap: 2024-04-22T08:14:16.503 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:14:16.503 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:16.503 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:14:16.503025+0000 2024-04-22T08:14:16.503 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:14:16.503025+0000 2024-04-22T08:14:16.504 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:14:16.504 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:16.504 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:14:16.504 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:14:16.504 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:14:16.504 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:14:16.505 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:14:16.505 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:16.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.502+0000 7f9d7fdb4640 1 Processor -- start 2024-04-22T08:14:16.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.503+0000 7f9d7fdb4640 1 -- start start 2024-04-22T08:14:16.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.503+0000 7f9d7fdb4640 10 monclient: init 2024-04-22T08:14:16.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.503+0000 7f9d7fdb4640 10 monclient: _reopen_session rank -1 2024-04-22T08:14:16.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.503+0000 7f9d7fdb4640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-04-22T08:14:16.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.503+0000 7f9d7fdb4640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9d780a1c60 0x7f9d780a31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:16.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.503+0000 7f9d7fdb4640 10 monclient(hunting): picked mon.noname-c con 0x7f9d780a1c60 addr v2:172.21.15.189:3300/0 2024-04-22T08:14:16.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.503+0000 7f9d7fdb4640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9d780a2040 0x7f9d78135bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:16.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.503+0000 7f9d7fdb4640 10 monclient(hunting): picked mon.noname-a con 0x7f9d780a2040 addr v2:172.21.15.40:3300/0 2024-04-22T08:14:16.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.503+0000 7f9d7fdb4640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d780a24d0 0x7f9d781360f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:16.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.503+0000 7f9d7fdb4640 10 monclient(hunting): picked mon.noname-b con 0x7f9d780a24d0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:14:16.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.503+0000 7f9d7fdb4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:16.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.503+0000 7f9d7fdb4640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d780a41f0 con 0x7f9d780a2040 2024-04-22T08:14:16.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.503+0000 7f9d7fdb4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:16.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.503+0000 7f9d7fdb4640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d7809fe90 con 0x7f9d780a1c60 2024-04-22T08:14:16.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.503+0000 7f9d7fdb4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:16.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.503+0000 7f9d7fdb4640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d780a0170 con 0x7f9d780a24d0 2024-04-22T08:14:16.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.503+0000 7f9d7fdb4640 10 monclient(hunting): _renew_subs 2024-04-22T08:14:16.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.503+0000 7f9d7fdb4640 10 monclient(hunting): authenticate will time out at 930.497520s 2024-04-22T08:14:16.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.503+0000 7f9d7d34f640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f9d780a2040 msgr2=0x7f9d78135bb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:14:16.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.503+0000 7f9d7d34f640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9d780a2040 0x7f9d78135bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:14:16.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.503+0000 7f9d7e351640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d780a24d0 0x7f9d781360f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:16.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.503+0000 7f9d7e351640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d780a24d0 0x7f9d781360f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:60956/0 (socket says 172.21.15.40:60956) 2024-04-22T08:14:16.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.503+0000 7f9d7e351640 1 -- 172.21.15.40:0/3129835015 learned_addr learned my addr 172.21.15.40:0/3129835015 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:14:16.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.503+0000 7f9d7e351640 10 monclient(hunting): get_auth_request con 0x7f9d780a24d0 auth_method 0 2024-04-22T08:14:16.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.503+0000 7f9d7e351640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:16.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.503+0000 7f9d7e351640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:16.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.503+0000 7f9d7e351640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:16.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.504+0000 7f9d7db50640 1 --2- 172.21.15.40:0/3129835015 >> v2:172.21.15.189:3300/0 conn(0x7f9d780a1c60 0x7f9d780a31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:16.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.504+0000 7f9d7e351640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:16.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.504+0000 7f9d7e351640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:16.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.504+0000 7f9d7e351640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:16.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.504+0000 7f9d7db50640 10 monclient(hunting): get_auth_request con 0x7f9d780a1c60 auth_method 0 2024-04-22T08:14:16.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.504+0000 7f9d7db50640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:16.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.504+0000 7f9d7db50640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:16.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.504+0000 7f9d7db50640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:16.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.504+0000 7f9d7db50640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:16.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.504+0000 7f9d7db50640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:16.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.504+0000 7f9d7db50640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:16.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.504+0000 7f9d7e351640 10 monclient(hunting): handle_auth_done global_id 4583 payload 274 2024-04-22T08:14:16.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.504+0000 7f9d7e351640 1 -- 172.21.15.40:0/3129835015 >> v2:172.21.15.189:3300/0 conn(0x7f9d780a1c60 msgr2=0x7f9d780a31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:14:16.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.504+0000 7f9d7e351640 1 --2- 172.21.15.40:0/3129835015 >> v2:172.21.15.189:3300/0 conn(0x7f9d780a1c60 0x7f9d780a31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:16.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.504+0000 7f9d7e351640 1 -- 172.21.15.40:0/3129835015 >> v2:172.21.15.40:3300/0 conn(0x7f9d780a2040 msgr2=0x7f9d78135bb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:14:16.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.504+0000 7f9d7e351640 1 --2- 172.21.15.40:0/3129835015 >> v2:172.21.15.40:3300/0 conn(0x7f9d780a2040 0x7f9d78135bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:16.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.504+0000 7f9d7e351640 10 monclient: _finish_hunting 0 2024-04-22T08:14:16.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.504+0000 7f9d7e351640 1 monclient: found mon.noname-b 2024-04-22T08:14:16.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.504+0000 7f9d7e351640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:14:16.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.504+0000 7f9d7e351640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:14:16.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.504+0000 7f9d7e351640 1 -- 172.21.15.40:0/3129835015 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d780a0450 con 0x7f9d780a24d0 2024-04-22T08:14:16.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.507+0000 7f9d7e351640 1 -- 172.21.15.40:0/3129835015 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d780a24d0 msgr2=0x7f9d781360f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:16.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.507+0000 7f9d7e351640 1 --2- 172.21.15.40:0/3129835015 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d780a24d0 0x7f9d781360f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9d780a5720 tx=0x7f9d7002bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e63e862e8ce43a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:16.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.507+0000 7f9d7cb4e640 1 -- 172.21.15.40:0/3129835015 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d7002cb90 con 0x7f9d780a24d0 2024-04-22T08:14:16.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.507+0000 7f9d7cb4e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:16.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.507+0000 7f9d7cb4e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:14:16.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.507+0000 7f9d7cb4e640 10 monclient: dump: 2024-04-22T08:14:16.540 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:16.540 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:16.541 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:16.541 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:16.541 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:16.541 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:16.541 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:16.541 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:16.541 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:16.541 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:16.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.507+0000 7f9d7cb4e640 10 monclient: _finish_auth 0 2024-04-22T08:14:16.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.507+0000 7f9d7cb4e640 10 monclient: _check_auth_tickets 2024-04-22T08:14:16.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.507+0000 7f9d7cb4e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:14:16.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.507+0000 7f9d7fdb4640 5 monclient: authenticate success, global_id 4583 2024-04-22T08:14:16.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.507+0000 7f9d7fdb4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:14:16.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.507+0000 7f9d7cb4e640 1 -- 172.21.15.40:0/3129835015 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d7002ccf0 con 0x7f9d780a24d0 2024-04-22T08:14:16.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.507+0000 7f9d7cb4e640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:14:16.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.507+0000 7f9d7fdb4640 10 monclient: get_monmap_and_config success 2024-04-22T08:14:16.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.507+0000 7f9d7fdb4640 4 set_mon_vals no callback set 2024-04-22T08:14:16.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.507+0000 7f9d7cb4e640 1 -- 172.21.15.40:0/3129835015 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d70034610 con 0x7f9d780a24d0 2024-04-22T08:14:16.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.507+0000 7f9d7cb4e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:16.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.507+0000 7f9d7cb4e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:14:16.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.507+0000 7f9d7cb4e640 10 monclient: dump: 2024-04-22T08:14:16.544 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:16.544 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:16.544 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:16.544 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:16.545 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:16.545 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:16.545 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:16.545 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:16.545 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:16.545 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:16.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.508+0000 7f9d7fdb4640 10 monclient: shutdown 2024-04-22T08:14:16.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.508+0000 7f9d7fdb4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:14:16.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.508+0000 7f9d7fdb4640 1 -- 172.21.15.40:0/3129835015 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d780a24d0 msgr2=0x7f9d781360f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:16.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.508+0000 7f9d7fdb4640 1 --2- 172.21.15.40:0/3129835015 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d780a24d0 0x7f9d781360f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9d780a5720 tx=0x7f9d7002bde0 comp rx=0 tx=0).stop 2024-04-22T08:14:16.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.508+0000 7f9d7fdb4640 1 -- 172.21.15.40:0/3129835015 shutdown_connections 2024-04-22T08:14:16.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.508+0000 7f9d7fdb4640 1 --2- 172.21.15.40:0/3129835015 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d780a24d0 0x7f9d781360f0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:16.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.508+0000 7f9d7fdb4640 1 --2- 172.21.15.40:0/3129835015 >> v2:172.21.15.40:3300/0 conn(0x7f9d780a2040 0x7f9d78135bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:16.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.508+0000 7f9d7fdb4640 1 --2- 172.21.15.40:0/3129835015 >> v2:172.21.15.189:3300/0 conn(0x7f9d780a1c60 0x7f9d780a31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:16.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.508+0000 7f9d7fdb4640 1 -- 172.21.15.40:0/3129835015 >> 172.21.15.40:0/3129835015 conn(0x7f9d780a89a0 msgr2=0x7f9d780a4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:14:16.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.508+0000 7f9d7fdb4640 1 -- 172.21.15.40:0/3129835015 shutdown_connections 2024-04-22T08:14:16.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.508+0000 7f9d7fdb4640 1 -- 172.21.15.40:0/3129835015 wait complete. 2024-04-22T08:14:16.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.509+0000 7f9d7fdb4640 10 monclient: build_initial_monmap 2024-04-22T08:14:16.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.509+0000 7f9d7fdb4640 10 monclient: monmap: 2024-04-22T08:14:16.547 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:14:16.548 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:14:16.548 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:14:16.548 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:14:16.548 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:14:16.548 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:16.548 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:14:16.548 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:14:16.548 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:14:16.548 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:16.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.509+0000 7f9d7fdb4640 1 Processor -- start 2024-04-22T08:14:16.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.509+0000 7f9d7fdb4640 1 -- start start 2024-04-22T08:14:16.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.509+0000 7f9d7fdb4640 10 monclient: init 2024-04-22T08:14:16.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.510+0000 7f9d7fdb4640 10 monclient: _reopen_session rank -1 2024-04-22T08:14:16.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.510+0000 7f9d7fdb4640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-22T08:14:16.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.510+0000 7f9d7fdb4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d780a1c60 0x7f9d781cae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:16.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.510+0000 7f9d7fdb4640 10 monclient(hunting): picked mon.noname-a con 0x7f9d780a1c60 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:14:16.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.510+0000 7f9d7fdb4640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d780a2040 0x7f9d781222d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:16.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.510+0000 7f9d7fdb4640 10 monclient(hunting): picked mon.noname-c con 0x7f9d780a2040 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:14:16.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.510+0000 7f9d7fdb4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d780a24d0 0x7f9d78122810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:16.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.510+0000 7f9d7fdb4640 10 monclient(hunting): picked mon.noname-b con 0x7f9d780a24d0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:14:16.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.510+0000 7f9d7fdb4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:16.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.510+0000 7f9d7fdb4640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d78124aa0 con 0x7f9d780a1c60 2024-04-22T08:14:16.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.510+0000 7f9d7fdb4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:16.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.510+0000 7f9d7fdb4640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d780a4790 con 0x7f9d780a24d0 2024-04-22T08:14:16.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.510+0000 7f9d7fdb4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:16.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.510+0000 7f9d7db50640 1 -- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d780a1c60 msgr2=0x7f9d781cae40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:14:16.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.510+0000 7f9d7fdb4640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d78070c30 con 0x7f9d780a2040 2024-04-22T08:14:16.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.510+0000 7f9d7fdb4640 10 monclient(hunting): _renew_subs 2024-04-22T08:14:16.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.510+0000 7f9d7fdb4640 10 monclient(hunting): authenticate will time out at 930.504224s 2024-04-22T08:14:16.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.510+0000 7f9d7db50640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d780a1c60 0x7f9d781cae40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:14:16.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.510+0000 7f9d7d34f640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d780a2040 0x7f9d781222d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:16.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.510+0000 7f9d7d34f640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d780a2040 0x7f9d781222d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:60970/0 (socket says 172.21.15.40:60970) 2024-04-22T08:14:16.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.510+0000 7f9d7d34f640 1 -- 172.21.15.40:0/3992403585 learned_addr learned my addr 172.21.15.40:0/3992403585 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:14:16.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.510+0000 7f9d7d34f640 10 monclient(hunting): get_auth_request con 0x7f9d780a2040 auth_method 0 2024-04-22T08:14:16.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.510+0000 7f9d7d34f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:16.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.510+0000 7f9d7d34f640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:16.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.510+0000 7f9d7d34f640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:16.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.510+0000 7f9d7d34f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:16.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.510+0000 7f9d7d34f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:16.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.510+0000 7f9d7d34f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:16.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.510+0000 7f9d7e351640 1 --2- 172.21.15.40:0/3992403585 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d780a24d0 0x7f9d78122810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:16.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.510+0000 7f9d7e351640 10 monclient(hunting): get_auth_request con 0x7f9d780a24d0 auth_method 0 2024-04-22T08:14:16.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.510+0000 7f9d7e351640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:16.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.510+0000 7f9d7e351640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:16.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.510+0000 7f9d7e351640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:16.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.510+0000 7f9d7e351640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:16.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.510+0000 7f9d7e351640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:16.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.510+0000 7f9d7e351640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:16.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.510+0000 7f9d7d34f640 10 monclient(hunting): handle_auth_done global_id 4586 payload 824 2024-04-22T08:14:16.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.510+0000 7f9d7d34f640 1 -- 172.21.15.40:0/3992403585 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d780a24d0 msgr2=0x7f9d78122810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:16.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.510+0000 7f9d7d34f640 1 --2- 172.21.15.40:0/3992403585 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d780a24d0 0x7f9d78122810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:16.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.510+0000 7f9d7d34f640 1 -- 172.21.15.40:0/3992403585 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d780a1c60 msgr2=0x7f9d781cae40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:14:16.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.510+0000 7f9d7d34f640 1 --2- 172.21.15.40:0/3992403585 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d780a1c60 0x7f9d781cae40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:16.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.510+0000 7f9d7d34f640 10 monclient: _finish_hunting 0 2024-04-22T08:14:16.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.510+0000 7f9d7d34f640 1 monclient: found mon.noname-c 2024-04-22T08:14:16.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.510+0000 7f9d7d34f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:14:16.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.510+0000 7f9d7d34f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:14:16.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.510+0000 7f9d7d34f640 1 -- 172.21.15.40:0/3992403585 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d70009b30 con 0x7f9d780a2040 2024-04-22T08:14:16.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.513+0000 7f9d7d34f640 1 -- 172.21.15.40:0/3992403585 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d780a2040 msgr2=0x7f9d781222d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:16.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.513+0000 7f9d7d34f640 1 --2- 172.21.15.40:0/3992403585 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d780a2040 0x7f9d781222d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9d6800b550 tx=0x7f9d6800ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:16.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.513+0000 7f9d6e7fc640 1 -- 172.21.15.40:0/3992403585 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d68005070 con 0x7f9d780a2040 2024-04-22T08:14:16.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.513+0000 7f9d6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:16.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.513+0000 7f9d6e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:14:16.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.513+0000 7f9d6e7fc640 10 monclient: dump: 2024-04-22T08:14:16.556 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:16.556 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:16.556 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:16.556 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:16.556 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:16.556 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:16.556 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:16.556 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:16.557 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:16.557 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:16.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.513+0000 7f9d6e7fc640 10 monclient: _finish_auth 0 2024-04-22T08:14:16.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.513+0000 7f9d6e7fc640 10 monclient: _check_auth_tickets 2024-04-22T08:14:16.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.513+0000 7f9d6e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:14:16.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.513+0000 7f9d6e7fc640 1 -- 172.21.15.40:0/3992403585 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d680051f0 con 0x7f9d780a2040 2024-04-22T08:14:16.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.513+0000 7f9d7fdb4640 5 monclient: authenticate success, global_id 4586 2024-04-22T08:14:16.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.513+0000 7f9d7fdb4640 10 monclient: _renew_subs 2024-04-22T08:14:16.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.513+0000 7f9d7fdb4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:14:16.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.513+0000 7f9d7fdb4640 1 -- 172.21.15.40:0/3992403585 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d78126230 con 0x7f9d780a2040 2024-04-22T08:14:16.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.513+0000 7f9d6e7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:14:16.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.513+0000 7f9d6e7fc640 1 -- 172.21.15.40:0/3992403585 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d680055a0 con 0x7f9d780a2040 2024-04-22T08:14:16.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.513+0000 7f9d6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:16.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.513+0000 7f9d6e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:14:16.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.513+0000 7f9d6e7fc640 10 monclient: dump: 2024-04-22T08:14:16.558 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:16.558 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:16.558 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:16.558 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:16.558 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:16.559 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:16.559 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:16.559 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:16.559 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:16.559 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:16.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.513+0000 7f9d6ffff640 4 set_mon_vals no callback set 2024-04-22T08:14:16.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.513+0000 7f9d7fdb4640 10 monclient: _renew_subs 2024-04-22T08:14:16.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.513+0000 7f9d7fdb4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:14:16.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.513+0000 7f9d7fdb4640 1 -- 172.21.15.40:0/3992403585 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d78123ed0 con 0x7f9d780a2040 2024-04-22T08:14:16.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.514+0000 7f9d7fdb4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:14:16.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.514+0000 7f9d7fdb4640 1 --2- 172.21.15.40:0/3992403585 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d78126ba0 0x7f9d78122d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:16.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.514+0000 7f9d7fdb4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:16.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.514+0000 7f9d7fdb4640 1 -- 172.21.15.40:0/3992403585 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d78066650 con 0x7f9d78126ba0 2024-04-22T08:14:16.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.514+0000 7f9d6e7fc640 1 -- 172.21.15.40:0/3992403585 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f9d68005700 con 0x7f9d780a2040 2024-04-22T08:14:16.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.514+0000 7f9d7db50640 1 --2- 172.21.15.40:0/3992403585 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d78126ba0 0x7f9d78122d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:16.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.514+0000 7f9d7db50640 10 monclient: get_auth_request con 0x7f9d78126ba0 auth_method 0 2024-04-22T08:14:16.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.514+0000 7f9d7db50640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:16.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.514+0000 7f9d7db50640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:16.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.514+0000 7f9d7db50640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:16.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.514+0000 7f9d7db50640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:16.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.514+0000 7f9d7db50640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:16.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.514+0000 7f9d7db50640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:16.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.514+0000 7f9d6e7fc640 1 --2- 172.21.15.40:0/3992403585 >> v2:172.21.15.189:6800/30168 conn(0x7f9d5c037780 0x7f9d5c039c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:16.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.514+0000 7f9d6e7fc640 1 -- 172.21.15.40:0/3992403585 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f9d6804a1b0 con 0x7f9d780a2040 2024-04-22T08:14:16.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.514+0000 7f9d7db50640 10 monclient(hunting): handle_auth_done global_id 4589 payload 824 2024-04-22T08:14:16.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.514+0000 7f9d7db50640 1 -- 172.21.15.40:0/3992403585 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9d78136630 con 0x7f9d78126ba0 2024-04-22T08:14:16.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.515+0000 7f9d7e351640 1 --2- 172.21.15.40:0/3992403585 >> v2:172.21.15.189:6800/30168 conn(0x7f9d5c037780 0x7f9d5c039c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:16.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.515+0000 7f9d7e351640 10 monclient: get_auth_request con 0x7f9d5c037780 auth_method 0 2024-04-22T08:14:16.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.517+0000 7f9d7db50640 1 -- 172.21.15.40:0/3992403585 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d78126ba0 msgr2=0x7f9d78122d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:16.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.517+0000 7f9d7db50640 1 --2- 172.21.15.40:0/3992403585 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d78126ba0 0x7f9d78122d50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9d74009fd0 tx=0x7f9d7400ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:16.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.517+0000 7f9d6e7fc640 1 -- 172.21.15.40:0/3992403585 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d74009610 con 0x7f9d78126ba0 2024-04-22T08:14:16.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.517+0000 7f9d6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:16.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.517+0000 7f9d6e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:14:16.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.517+0000 7f9d6e7fc640 10 monclient: dump: 2024-04-22T08:14:16.562 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:16.562 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:16.562 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:16.562 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:16.562 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:16.562 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:16.562 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:16.562 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:16.562 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:16.562 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:16.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.517+0000 7f9d6e7fc640 1 -- 172.21.15.40:0/3992403585 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f9d7400cbc0 con 0x7f9d78126ba0 2024-04-22T08:14:16.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.517+0000 7f9d6e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:14:16.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.517+0000 7f9d6e7fc640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:14:16.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.517+0000 7f9d6e7fc640 1 -- 172.21.15.40:0/3992403585 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d78126ba0 msgr2=0x7f9d78122d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:16.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.517+0000 7f9d6e7fc640 1 --2- 172.21.15.40:0/3992403585 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d78126ba0 0x7f9d78122d50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9d74009fd0 tx=0x7f9d7400ec70 comp rx=0 tx=0).stop 2024-04-22T08:14:16.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.517+0000 7f9d6e7fc640 1 -- 172.21.15.40:0/3992403585 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d78126ba0 msgr2=0x7f9d78122d50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:14:16.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.517+0000 7f9d6e7fc640 1 --2- 172.21.15.40:0/3992403585 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d78126ba0 0x7f9d78122d50 secure :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9d74009fd0 tx=0x7f9d7400ec70 comp rx=0 tx=0).stop 2024-04-22T08:14:16.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.517+0000 7f9d7e351640 1 --2- 172.21.15.40:0/3992403585 >> v2:172.21.15.189:6800/30168 conn(0x7f9d5c037780 0x7f9d5c039c40 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9d7809f4b0 tx=0x7f9d70030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:16.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.523+0000 7f9d7fdb4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:14:16.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.523+0000 7f9d7fdb4640 1 --2- 172.21.15.40:0/3992403585 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d781cb380 0x7f9d78123290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:16.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.523+0000 7f9d7fdb4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:16.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.523+0000 7f9d7fdb4640 1 -- 172.21.15.40:0/3992403585 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d780a44d0 con 0x7f9d781cb380 2024-04-22T08:14:16.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.523+0000 7f9d7db50640 1 --2- 172.21.15.40:0/3992403585 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d781cb380 0x7f9d78123290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:16.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.523+0000 7f9d7db50640 10 monclient: get_auth_request con 0x7f9d781cb380 auth_method 0 2024-04-22T08:14:16.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.523+0000 7f9d7db50640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:16.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.523+0000 7f9d7db50640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:16.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.523+0000 7f9d7db50640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:16.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.523+0000 7f9d7db50640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:16.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.523+0000 7f9d7db50640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:16.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.523+0000 7f9d7db50640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:16.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.523+0000 7f9d7db50640 10 monclient(hunting): handle_auth_done global_id 4592 payload 824 2024-04-22T08:14:16.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.524+0000 7f9d7db50640 1 -- 172.21.15.40:0/3992403585 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9d7804e9b0 con 0x7f9d781cb380 2024-04-22T08:14:16.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.526+0000 7f9d7db50640 1 -- 172.21.15.40:0/3992403585 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d781cb380 msgr2=0x7f9d78123290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:16.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.526+0000 7f9d7db50640 1 --2- 172.21.15.40:0/3992403585 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d781cb380 0x7f9d78123290 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9d74018000 tx=0x7f9d74004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:16.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.526+0000 7f9d6e7fc640 1 -- 172.21.15.40:0/3992403585 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d74022070 con 0x7f9d781cb380 2024-04-22T08:14:16.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.526+0000 7f9d6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:16.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.526+0000 7f9d6e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:14:16.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.526+0000 7f9d6e7fc640 10 monclient: dump: 2024-04-22T08:14:16.564 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:16.564 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:16.565 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:16.565 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:16.565 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:16.565 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:16.565 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:16.565 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:16.565 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:16.565 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:16.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.526+0000 7f9d6e7fc640 1 -- 172.21.15.40:0/3992403585 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4620 (secure 0 0 0) 0x7f9d74005e90 con 0x7f9d781cb380 2024-04-22T08:14:16.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.526+0000 7f9d6e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:14:16.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.526+0000 7f9d6e7fc640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:14:16.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.526+0000 7f9d6e7fc640 1 -- 172.21.15.40:0/3992403585 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d781cb380 msgr2=0x7f9d78123290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:16.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.526+0000 7f9d6e7fc640 1 --2- 172.21.15.40:0/3992403585 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d781cb380 0x7f9d78123290 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9d74018000 tx=0x7f9d74004730 comp rx=0 tx=0).stop 2024-04-22T08:14:16.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.526+0000 7f9d6e7fc640 1 -- 172.21.15.40:0/3992403585 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d781cb380 msgr2=0x7f9d78123290 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:14:16.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.526+0000 7f9d6e7fc640 1 --2- 172.21.15.40:0/3992403585 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d781cb380 0x7f9d78123290 secure :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9d74018000 tx=0x7f9d74004730 comp rx=0 tx=0).stop 2024-04-22T08:14:16.566 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:14:16.567 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:14:16.567 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:14:16.567 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:14:16.567 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 22, 2024-04-22T08:14:16.567 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:14:16.567 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:14:16.568 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:14:16.568 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:14:16.568 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 1, 2024-04-22T08:14:16.568 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:14:16.568 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:14:16.568 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:14:16.569 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:14:16.569 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:14:16.569 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:14:16.569 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:14:16.569 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:14:16.569 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:14:16.569 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:14:16.569 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:14:16.569 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:14:16.570 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:14:16.570 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:14:16.570 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:14:16.570 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:14:16.570 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:14:16.570 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:14:16.570 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:14:16.570 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:14:16.571 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:14:16.571 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:14:16.571 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:14:16.571 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:14:16.571 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:14:16.571 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:14:16.571 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:16.571 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:16.571 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:14:16.572 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:14:16.572 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:14:16.572 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:14:16.572 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:14:16.572 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:14:16.572 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:14:16.572 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:14:16.572 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:14:16.572 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:14:16.572 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:14:16.572 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:14:16.572 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:14:16.572 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:14:16.572 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:14:16.573 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:14:16.573 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:14:16.573 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:14:16.573 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:14:16.573 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:14:16.573 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:14:16.573 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:14:16.573 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:14:16.573 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:14:16.573 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:14:16.573 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:14:16.573 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:14:16.573 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:14:16.573 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:14:16.573 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:16.574 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:14:16.574 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:16.574 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:14:16.574 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:14:16.574 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:14:16.574 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:14:16.574 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:16.574 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:14:16.574 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:14:16.574 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:16.574 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:16.574 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:16.574 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:14:16.575 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:14:16.575 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:16.575 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:16.575 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:16.575 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:16.575 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:14:16.575 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:14:16.575 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:14:16.575 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:14:16.575 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:14:16.575 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:16.575 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:16.575 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:14:16.575 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:14:16.576 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:14:16.576 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:14:16.576 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:16.576 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:14:16.576 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:14:16.576 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:16.576 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:16.576 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:16.576 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:14:16.576 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:14:16.576 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:16.576 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:16.576 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:16.576 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:16.576 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:14:16.577 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:14:16.577 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:14:16.577 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:14:16.577 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:14:16.577 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:16.577 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:16.577 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:14:16.577 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:14:16.577 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:14:16.577 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:14:16.577 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:16.577 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:14:16.577 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:14:16.577 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:16.578 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:16.578 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:16.578 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:14:16.578 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:14:16.578 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:16.578 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:16.578 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:16.578 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:16.578 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:14:16.578 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:14:16.578 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:14:16.578 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:14:16.578 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:14:16.579 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:16.579 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:16.579 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:16.579 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:14:16.579 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:14:16.579 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:16.579 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:14:16.579 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:14:16.579 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:14:16.579 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:16.579 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:14:16.579 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:14:16.579 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:16.579 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:14:16.579 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:14:16.580 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 6 2024-04-22T08:14:16.580 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:16.580 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:14:16.580 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:14:16.580 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:16.580 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:14:16.580 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:14:16.580 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 4 2024-04-22T08:14:16.580 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:16.580 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:16.580 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:16.580 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:14:16.580 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:14:16.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.526+0000 7f9d7fdb4640 1 -- 172.21.15.40:0/3992403585 >> v2:172.21.15.189:6800/30168 conn(0x7f9d5c037780 msgr2=0x7f9d5c039c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:16.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.526+0000 7f9d7fdb4640 1 --2- 172.21.15.40:0/3992403585 >> v2:172.21.15.189:6800/30168 conn(0x7f9d5c037780 0x7f9d5c039c40 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9d7809f4b0 tx=0x7f9d70030000 comp rx=0 tx=0).stop 2024-04-22T08:14:16.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.526+0000 7f9d7fdb4640 10 monclient: shutdown 2024-04-22T08:14:16.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.526+0000 7f9d7fdb4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:14:16.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.526+0000 7f9d7fdb4640 1 -- 172.21.15.40:0/3992403585 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d780a2040 msgr2=0x7f9d781222d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:16.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.526+0000 7f9d7fdb4640 1 --2- 172.21.15.40:0/3992403585 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d780a2040 0x7f9d781222d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9d6800b550 tx=0x7f9d6800ba10 comp rx=0 tx=0).stop 2024-04-22T08:14:16.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.526+0000 7f9d7e351640 1 -- 172.21.15.40:0/3992403585 reap_dead start 2024-04-22T08:14:16.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.526+0000 7f9d7fdb4640 1 -- 172.21.15.40:0/3992403585 shutdown_connections 2024-04-22T08:14:16.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.526+0000 7f9d7fdb4640 1 --2- 172.21.15.40:0/3992403585 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d780a2040 0x7f9d781222d0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:16.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.526+0000 7f9d7fdb4640 1 -- 172.21.15.40:0/3992403585 >> 172.21.15.40:0/3992403585 conn(0x7f9d780a89a0 msgr2=0x7f9d780a0b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:14:16.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.527+0000 7f9d7fdb4640 1 -- 172.21.15.40:0/3992403585 shutdown_connections 2024-04-22T08:14:16.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.527+0000 7f9d7fdb4640 1 -- 172.21.15.40:0/3992403585 wait complete. 2024-04-22T08:14:16.583 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-22T08:14:16.584 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-04-22T08:14:16.584 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-22T08:14:16.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.685+0000 7fdeed5ca640 10 monclient: get_monmap_and_config 2024-04-22T08:14:16.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.685+0000 7fdeed5ca640 10 monclient: build_initial_monmap 2024-04-22T08:14:16.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.685+0000 7fdeed5ca640 10 monclient: monmap: 2024-04-22T08:14:16.685 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:14:16.685 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:16.686 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:14:16.686418+0000 2024-04-22T08:14:16.686 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:14:16.686418+0000 2024-04-22T08:14:16.686 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:14:16.686 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:16.686 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:14:16.686 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:14:16.686 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:14:16.687 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:14:16.687 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:14:16.687 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:16.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.686+0000 7fdeed5ca640 1 Processor -- start 2024-04-22T08:14:16.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.686+0000 7fdeed5ca640 1 -- start start 2024-04-22T08:14:16.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.686+0000 7fdeed5ca640 10 monclient: init 2024-04-22T08:14:16.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.686+0000 7fdeed5ca640 10 monclient: _reopen_session rank -1 2024-04-22T08:14:16.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.686+0000 7fdeed5ca640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-04-22T08:14:16.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.686+0000 7fdeed5ca640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdee812fad0 0x7fdee812feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:16.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.686+0000 7fdeed5ca640 10 monclient(hunting): picked mon.noname-b con 0x7fdee812fad0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:14:16.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.686+0000 7fdeed5ca640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdee8132920 addr v1:172.21.15.40:6789/0 2024-04-22T08:14:16.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.686+0000 7fdeed5ca640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdee8133430 addr v1:172.21.15.189:6789/0 2024-04-22T08:14:16.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.686+0000 7fdeed5ca640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdee809ae50 con 0x7fdee8132920 2024-04-22T08:14:16.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.686+0000 7fdeed5ca640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdee809efc0 con 0x7fdee8133430 2024-04-22T08:14:16.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.686+0000 7fdeed5ca640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:16.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.686+0000 7fdeed5ca640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdee809d070 con 0x7fdee812fad0 2024-04-22T08:14:16.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.686+0000 7fdeed5ca640 10 monclient(hunting): _renew_subs 2024-04-22T08:14:16.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.686+0000 7fdeed5ca640 10 monclient(hunting): authenticate will time out at 930.680675s 2024-04-22T08:14:16.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.686+0000 7fdedffff640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7fdee8132920 legacy=0x7fdee8132d20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-04-22T08:14:16.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.686+0000 7fdee6ffd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdee812fad0 0x7fdee812feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:16.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.687+0000 7fdee6ffd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdee812fad0 0x7fdee812feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:60992/0 (socket says 172.21.15.40:60992) 2024-04-22T08:14:16.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.687+0000 7fdee6ffd640 1 -- 172.21.15.40:0/3697573238 learned_addr learned my addr 172.21.15.40:0/3697573238 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:14:16.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.687+0000 7fdee6ffd640 10 monclient(hunting): get_auth_request con 0x7fdee812fad0 auth_method 0 2024-04-22T08:14:16.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.687+0000 7fdee6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:16.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.687+0000 7fdee6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:16.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.687+0000 7fdee6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:16.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.687+0000 7fdee6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:16.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.687+0000 7fdee6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:16.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.687+0000 7fdee6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:16.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.687+0000 7fdee6ffd640 10 monclient(hunting): handle_auth_done global_id 4595 payload 274 2024-04-22T08:14:16.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.687+0000 7fdee6ffd640 1 -- 172.21.15.40:0/3697573238 >> v1:172.21.15.189:6789/0 conn(0x7fdee8133430 legacy=0x7fdee813ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:14:16.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.687+0000 7fdee6ffd640 1 -- 172.21.15.40:0/3697573238 >> v1:172.21.15.40:6789/0 conn(0x7fdee8132920 legacy=0x7fdee8132d20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:14:16.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.687+0000 7fdee6ffd640 10 monclient: _finish_hunting 0 2024-04-22T08:14:16.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.687+0000 7fdee6ffd640 1 monclient: found mon.noname-b 2024-04-22T08:14:16.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.687+0000 7fdee6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:14:16.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.687+0000 7fdee6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:14:16.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.687+0000 7fdee6ffd640 1 -- 172.21.15.40:0/3697573238 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee809d350 con 0x7fdee812fad0 2024-04-22T08:14:16.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.690+0000 7fdee6ffd640 1 -- 172.21.15.40:0/3697573238 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdee812fad0 msgr2=0x7fdee812feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:16.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.690+0000 7fdee6ffd640 1 --2- 172.21.15.40:0/3697573238 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdee812fad0 0x7fdee812feb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fded80098a0 tx=0x7fded802b7c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6ea061a80c91228 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:16.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.690+0000 7fdee67fc640 1 -- 172.21.15.40:0/3697573238 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded8031730 con 0x7fdee812fad0 2024-04-22T08:14:16.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.690+0000 7fdee67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:16.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.690+0000 7fdee67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:14:16.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.690+0000 7fdee67fc640 10 monclient: dump: 2024-04-22T08:14:16.695 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:16.695 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:16.695 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:16.695 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:16.695 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:16.695 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:16.696 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:16.696 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:16.696 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:16.696 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:16.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.690+0000 7fdee67fc640 10 monclient: _finish_auth 0 2024-04-22T08:14:16.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.690+0000 7fdee67fc640 10 monclient: _check_auth_tickets 2024-04-22T08:14:16.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.690+0000 7fdee67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:14:16.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.690+0000 7fdee67fc640 1 -- 172.21.15.40:0/3697573238 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fded8031890 con 0x7fdee812fad0 2024-04-22T08:14:16.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.690+0000 7fdeed5ca640 5 monclient: authenticate success, global_id 4595 2024-04-22T08:14:16.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.690+0000 7fdeed5ca640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:14:16.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.690+0000 7fdee67fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:14:16.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.690+0000 7fdeed5ca640 10 monclient: get_monmap_and_config success 2024-04-22T08:14:16.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.690+0000 7fdee67fc640 1 -- 172.21.15.40:0/3697573238 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded8031c20 con 0x7fdee812fad0 2024-04-22T08:14:16.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.690+0000 7fdee67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:16.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.690+0000 7fdeed5ca640 4 set_mon_vals no callback set 2024-04-22T08:14:16.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.690+0000 7fdee67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:14:16.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.690+0000 7fdee67fc640 10 monclient: dump: 2024-04-22T08:14:16.698 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:16.698 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:16.698 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:16.698 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:16.698 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:16.698 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:16.699 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:16.699 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:16.699 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:16.699 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:16.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.690+0000 7fdeed5ca640 10 monclient: shutdown 2024-04-22T08:14:16.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.690+0000 7fdeed5ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:14:16.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.690+0000 7fdeed5ca640 1 -- 172.21.15.40:0/3697573238 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdee812fad0 msgr2=0x7fdee812feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:16.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.690+0000 7fdeed5ca640 1 --2- 172.21.15.40:0/3697573238 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdee812fad0 0x7fdee812feb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fded80098a0 tx=0x7fded802b7c0 comp rx=0 tx=0).stop 2024-04-22T08:14:16.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.690+0000 7fdeed5ca640 1 -- 172.21.15.40:0/3697573238 shutdown_connections 2024-04-22T08:14:16.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.690+0000 7fdeed5ca640 1 --2- 172.21.15.40:0/3697573238 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdee812fad0 0x7fdee812feb0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:16.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.690+0000 7fdeed5ca640 1 -- 172.21.15.40:0/3697573238 >> 172.21.15.40:0/3697573238 conn(0x7fdee80a0ae0 msgr2=0x7fdee80a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:14:16.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.690+0000 7fdeed5ca640 1 -- 172.21.15.40:0/3697573238 shutdown_connections 2024-04-22T08:14:16.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.690+0000 7fdeed5ca640 1 -- 172.21.15.40:0/3697573238 wait complete. 2024-04-22T08:14:16.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.691+0000 7fdeed5ca640 10 monclient: build_initial_monmap 2024-04-22T08:14:16.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.691+0000 7fdeed5ca640 10 monclient: monmap: 2024-04-22T08:14:16.700 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:14:16.700 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:14:16.700 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:14:16.700 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:14:16.700 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:14:16.700 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:16.700 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:14:16.700 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:14:16.700 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:14:16.700 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:16.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.691+0000 7fdeed5ca640 1 Processor -- start 2024-04-22T08:14:16.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.691+0000 7fdeed5ca640 1 -- start start 2024-04-22T08:14:16.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.691+0000 7fdeed5ca640 10 monclient: init 2024-04-22T08:14:16.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.692+0000 7fdeed5ca640 10 monclient: _reopen_session rank -1 2024-04-22T08:14:16.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.692+0000 7fdeed5ca640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-22T08:14:16.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.692+0000 7fdeed5ca640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdee812fad0 0x7fdee81cab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:16.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.692+0000 7fdeed5ca640 10 monclient(hunting): picked mon.noname-c con 0x7fdee812fad0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:14:16.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.692+0000 7fdeed5ca640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdee8132920 0x7fdee812eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:16.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.692+0000 7fdeed5ca640 10 monclient(hunting): picked mon.noname-b con 0x7fdee8132920 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:14:16.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.692+0000 7fdeed5ca640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdee8133430 0x7fdee812f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:16.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.692+0000 7fdeed5ca640 10 monclient(hunting): picked mon.noname-a con 0x7fdee8133430 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:14:16.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.692+0000 7fdeed5ca640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:16.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.692+0000 7fdeed5ca640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee81cb810 con 0x7fdee8133430 2024-04-22T08:14:16.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.692+0000 7fdeed5ca640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:16.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.692+0000 7fdeed5ca640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee8071870 con 0x7fdee8132920 2024-04-22T08:14:16.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.692+0000 7fdeed5ca640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:16.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.692+0000 7fdeed5ca640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdee8132390 con 0x7fdee812fad0 2024-04-22T08:14:16.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.692+0000 7fdeed5ca640 10 monclient(hunting): _renew_subs 2024-04-22T08:14:16.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.692+0000 7fdeed5ca640 10 monclient(hunting): authenticate will time out at 930.685958s 2024-04-22T08:14:16.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.692+0000 7fdee77fe640 1 -- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdee8133430 msgr2=0x7fdee812f050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:14:16.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.692+0000 7fdee77fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdee8133430 0x7fdee812f050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:14:16.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.692+0000 7fdee6ffd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdee812fad0 0x7fdee81cab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:16.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.692+0000 7fdee6ffd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdee812fad0 0x7fdee81cab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:60994/0 (socket says 172.21.15.40:60994) 2024-04-22T08:14:16.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.692+0000 7fdee6ffd640 1 -- 172.21.15.40:0/2802660183 learned_addr learned my addr 172.21.15.40:0/2802660183 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:14:16.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.692+0000 7fdee6ffd640 10 monclient(hunting): get_auth_request con 0x7fdee812fad0 auth_method 0 2024-04-22T08:14:16.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.692+0000 7fdee6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:16.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.692+0000 7fdee6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:16.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.692+0000 7fdee6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:16.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.692+0000 7fdee6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:16.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.692+0000 7fdee6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:16.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.692+0000 7fdee6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:16.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.692+0000 7fdedffff640 1 --2- 172.21.15.40:0/2802660183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdee8132920 0x7fdee812eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:16.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.692+0000 7fdedffff640 10 monclient(hunting): get_auth_request con 0x7fdee8132920 auth_method 0 2024-04-22T08:14:16.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.692+0000 7fdedffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:16.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.692+0000 7fdedffff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:16.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.692+0000 7fdedffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:16.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.692+0000 7fdee6ffd640 10 monclient(hunting): handle_auth_done global_id 4598 payload 824 2024-04-22T08:14:16.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.692+0000 7fdee6ffd640 1 -- 172.21.15.40:0/2802660183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdee8132920 msgr2=0x7fdee812eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:16.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.692+0000 7fdee6ffd640 1 --2- 172.21.15.40:0/2802660183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdee8132920 0x7fdee812eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:16.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.692+0000 7fdee6ffd640 1 -- 172.21.15.40:0/2802660183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdee8133430 msgr2=0x7fdee812f050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:14:16.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.692+0000 7fdee6ffd640 1 --2- 172.21.15.40:0/2802660183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdee8133430 0x7fdee812f050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:16.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.692+0000 7fdee6ffd640 10 monclient: _finish_hunting 0 2024-04-22T08:14:16.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.692+0000 7fdee6ffd640 1 monclient: found mon.noname-c 2024-04-22T08:14:16.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.692+0000 7fdee6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:14:16.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.692+0000 7fdee6ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:14:16.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.692+0000 7fdee6ffd640 1 -- 172.21.15.40:0/2802660183 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fded80094d0 con 0x7fdee812fad0 2024-04-22T08:14:16.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.694+0000 7fdee6ffd640 1 -- 172.21.15.40:0/2802660183 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdee812fad0 msgr2=0x7fdee81cab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:16.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.694+0000 7fdee6ffd640 1 --2- 172.21.15.40:0/2802660183 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdee812fad0 0x7fdee81cab40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fded80097f0 tx=0x7fded80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:16.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.694+0000 7fdedf7fe640 1 -- 172.21.15.40:0/2802660183 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded8031730 con 0x7fdee812fad0 2024-04-22T08:14:16.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.694+0000 7fdedf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:16.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.694+0000 7fdedf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:14:16.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.694+0000 7fdedf7fe640 10 monclient: dump: 2024-04-22T08:14:16.710 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:16.710 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:16.710 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:16.710 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:16.710 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:16.710 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:16.710 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:16.710 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:16.710 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:16.710 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:16.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.694+0000 7fdedf7fe640 10 monclient: _finish_auth 0 2024-04-22T08:14:16.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.694+0000 7fdedf7fe640 10 monclient: _check_auth_tickets 2024-04-22T08:14:16.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.694+0000 7fdedf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:14:16.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.695+0000 7fdedf7fe640 1 -- 172.21.15.40:0/2802660183 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fded8033de0 con 0x7fdee812fad0 2024-04-22T08:14:16.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.695+0000 7fdeed5ca640 5 monclient: authenticate success, global_id 4598 2024-04-22T08:14:16.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.695+0000 7fdeed5ca640 10 monclient: _renew_subs 2024-04-22T08:14:16.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.695+0000 7fdeed5ca640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:14:16.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.695+0000 7fdeed5ca640 1 -- 172.21.15.40:0/2802660183 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdee8123330 con 0x7fdee812fad0 2024-04-22T08:14:16.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.695+0000 7fdedf7fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:14:16.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.695+0000 7fdedf7fe640 1 -- 172.21.15.40:0/2802660183 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded803d6b0 con 0x7fdee812fad0 2024-04-22T08:14:16.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.695+0000 7fdedf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:16.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.695+0000 7fdedf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:14:16.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.695+0000 7fdedf7fe640 10 monclient: dump: 2024-04-22T08:14:16.711 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:16.711 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:16.711 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:16.712 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:16.712 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:16.712 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:16.712 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:16.712 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:16.712 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:16.712 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:16.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.695+0000 7fdee57fa640 4 set_mon_vals no callback set 2024-04-22T08:14:16.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.695+0000 7fdeed5ca640 10 monclient: _renew_subs 2024-04-22T08:14:16.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.695+0000 7fdeed5ca640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:14:16.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.695+0000 7fdeed5ca640 1 -- 172.21.15.40:0/2802660183 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdee8120f60 con 0x7fdee812fad0 2024-04-22T08:14:16.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.695+0000 7fdeed5ca640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:14:16.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.695+0000 7fdeed5ca640 1 --2- 172.21.15.40:0/2802660183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdee811f4a0 0x7fdee811f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:16.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.695+0000 7fdeed5ca640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:16.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.695+0000 7fdeed5ca640 1 -- 172.21.15.40:0/2802660183 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee80667a0 con 0x7fdee811f4a0 2024-04-22T08:14:16.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.695+0000 7fdedf7fe640 1 -- 172.21.15.40:0/2802660183 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fded803a040 con 0x7fdee812fad0 2024-04-22T08:14:16.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.696+0000 7fdedf7fe640 1 --2- 172.21.15.40:0/2802660183 >> v2:172.21.15.189:6800/30168 conn(0x7fdec80377b0 0x7fdec8039c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:16.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.696+0000 7fdedf7fe640 1 -- 172.21.15.40:0/2802660183 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 3898+0+0 (secure 0 0 0) 0x7fded806bcf0 con 0x7fdee812fad0 2024-04-22T08:14:16.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.696+0000 7fdedffff640 1 --2- 172.21.15.40:0/2802660183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdee811f4a0 0x7fdee811f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:16.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.696+0000 7fdedffff640 10 monclient: get_auth_request con 0x7fdee811f4a0 auth_method 0 2024-04-22T08:14:16.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.696+0000 7fdedffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:16.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.696+0000 7fdedffff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:16.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.696+0000 7fdedffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:16.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.696+0000 7fdee77fe640 1 --2- 172.21.15.40:0/2802660183 >> v2:172.21.15.189:6800/30168 conn(0x7fdec80377b0 0x7fdec8039c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:16.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.696+0000 7fdee77fe640 10 monclient: get_auth_request con 0x7fdec80377b0 auth_method 0 2024-04-22T08:14:16.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.696+0000 7fdedffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:16.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.696+0000 7fdedffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:16.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.696+0000 7fdedffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:16.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.696+0000 7fdedffff640 10 monclient(hunting): handle_auth_done global_id 24187 payload 824 2024-04-22T08:14:16.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.696+0000 7fdedffff640 1 -- 172.21.15.40:0/2802660183 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdee809e2f0 con 0x7fdee811f4a0 2024-04-22T08:14:16.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.698+0000 7fdee77fe640 1 --2- 172.21.15.40:0/2802660183 >> v2:172.21.15.189:6800/30168 conn(0x7fdec80377b0 0x7fdec8039c70 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fdee812f850 tx=0x7fdee0007d10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:16.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.698+0000 7fdedffff640 1 -- 172.21.15.40:0/2802660183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdee811f4a0 msgr2=0x7fdee811f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:16.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.698+0000 7fdedffff640 1 --2- 172.21.15.40:0/2802660183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdee811f4a0 0x7fdee811f880 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdee8133050 tx=0x7fded000db90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:16.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.699+0000 7fdedf7fe640 1 -- 172.21.15.40:0/2802660183 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded0005c50 con 0x7fdee811f4a0 2024-04-22T08:14:16.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.699+0000 7fdedf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:16.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.699+0000 7fdedf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:14:16.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.699+0000 7fdedf7fe640 10 monclient: dump: 2024-04-22T08:14:16.714 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:16.714 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:16.714 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:16.714 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:16.714 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:16.715 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:16.715 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:16.715 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:16.715 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:16.715 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:16.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.699+0000 7fdedf7fe640 1 -- 172.21.15.40:0/2802660183 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7fded0005db0 con 0x7fdee811f4a0 2024-04-22T08:14:16.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.699+0000 7fdedf7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:14:16.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.699+0000 7fdedf7fe640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:14:16.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.699+0000 7fdedf7fe640 1 -- 172.21.15.40:0/2802660183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdee811f4a0 msgr2=0x7fdee811f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:16.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.699+0000 7fdedf7fe640 1 --2- 172.21.15.40:0/2802660183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdee811f4a0 0x7fdee811f880 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdee8133050 tx=0x7fded000db90 comp rx=0 tx=0).stop 2024-04-22T08:14:16.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.699+0000 7fdedf7fe640 1 -- 172.21.15.40:0/2802660183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdee811f4a0 msgr2=0x7fdee811f880 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:14:16.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.699+0000 7fdedf7fe640 1 --2- 172.21.15.40:0/2802660183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdee811f4a0 0x7fdee811f880 secure :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdee8133050 tx=0x7fded000db90 comp rx=0 tx=0).stop 2024-04-22T08:14:16.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.706+0000 7fdeed5ca640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:14:16.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.706+0000 7fdeed5ca640 1 --2- 172.21.15.40:0/2802660183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdee811fdc0 0x7fdee811b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:16.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.706+0000 7fdeed5ca640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:16.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.706+0000 7fdeed5ca640 1 -- 172.21.15.40:0/2802660183 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee8122e20 con 0x7fdee811fdc0 2024-04-22T08:14:16.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.706+0000 7fdedffff640 1 --2- 172.21.15.40:0/2802660183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdee811fdc0 0x7fdee811b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:16.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.706+0000 7fdedffff640 10 monclient: get_auth_request con 0x7fdee811fdc0 auth_method 0 2024-04-22T08:14:16.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.706+0000 7fdedffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:16.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.706+0000 7fdedffff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:16.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.706+0000 7fdedffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:16.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.706+0000 7fdedffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:16.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.706+0000 7fdedffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:16.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.706+0000 7fdedffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:16.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.706+0000 7fdedffff640 10 monclient(hunting): handle_auth_done global_id 24190 payload 824 2024-04-22T08:14:16.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.706+0000 7fdedffff640 1 -- 172.21.15.40:0/2802660183 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fdee804e9b0 con 0x7fdee811fdc0 2024-04-22T08:14:16.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.709+0000 7fdedffff640 1 -- 172.21.15.40:0/2802660183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdee811fdc0 msgr2=0x7fdee811b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:16.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.709+0000 7fdedffff640 1 --2- 172.21.15.40:0/2802660183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdee811fdc0 0x7fdee811b640 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdee8133050 tx=0x7fded00059d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:16.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.709+0000 7fdedf7fe640 1 -- 172.21.15.40:0/2802660183 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded0021070 con 0x7fdee811fdc0 2024-04-22T08:14:16.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.709+0000 7fdedf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:16.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.709+0000 7fdedf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:14:16.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.709+0000 7fdedf7fe640 10 monclient: dump: 2024-04-22T08:14:16.718 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:16.718 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:16.718 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:16.718 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:16.718 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:16.718 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:16.718 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:16.718 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:16.718 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:16.718 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:16.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.709+0000 7fdedf7fe640 1 -- 172.21.15.40:0/2802660183 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4622 (secure 0 0 0) 0x7fded00096e0 con 0x7fdee811fdc0 2024-04-22T08:14:16.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.709+0000 7fdedf7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:14:16.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.709+0000 7fdedf7fe640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:14:16.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.709+0000 7fdedf7fe640 1 -- 172.21.15.40:0/2802660183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdee811fdc0 msgr2=0x7fdee811b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:16.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.709+0000 7fdedf7fe640 1 --2- 172.21.15.40:0/2802660183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdee811fdc0 0x7fdee811b640 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdee8133050 tx=0x7fded00059d0 comp rx=0 tx=0).stop 2024-04-22T08:14:16.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.709+0000 7fdedf7fe640 1 -- 172.21.15.40:0/2802660183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdee811fdc0 msgr2=0x7fdee811b640 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:14:16.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.709+0000 7fdedf7fe640 1 --2- 172.21.15.40:0/2802660183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdee811fdc0 0x7fdee811b640 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdee8133050 tx=0x7fded00059d0 comp rx=0 tx=0).stop 2024-04-22T08:14:16.722 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:14:16.722 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:14:16.722 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:14:16.722 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "leader", 2024-04-22T08:14:16.722 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 22, 2024-04-22T08:14:16.722 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:14:16.723 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:14:16.723 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:14:16.723 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:14:16.723 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 2, 2024-04-22T08:14:16.723 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:14:16.723 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:14:16.723 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:14:16.723 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:14:16.723 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:14:16.723 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:14:16.723 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:14:16.723 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:14:16.723 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:14:16.723 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:14:16.723 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:14:16.724 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:14:16.724 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:14:16.724 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:14:16.724 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:14:16.724 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:14:16.724 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:14:16.724 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:14:16.724 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:14:16.724 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:14:16.724 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:14:16.724 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:14:16.724 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:14:16.724 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:14:16.724 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:14:16.725 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:14:16.725 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:16.725 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:16.725 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:14:16.725 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:14:16.725 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:14:16.725 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:14:16.725 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:14:16.725 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:14:16.725 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:14:16.725 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:14:16.725 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:14:16.725 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:14:16.725 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:14:16.725 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:14:16.726 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:14:16.726 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:14:16.726 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:14:16.726 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:14:16.726 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:14:16.726 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:14:16.726 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:14:16.726 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:14:16.726 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:14:16.726 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:14:16.726 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:14:16.726 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:14:16.726 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:14:16.726 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:14:16.726 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:14:16.727 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:14:16.727 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:14:16.727 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:16.727 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:14:16.727 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:16.727 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:14:16.727 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:14:16.727 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:14:16.727 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:14:16.727 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:16.727 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:14:16.727 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:14:16.727 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:16.727 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:16.728 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:16.728 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:14:16.728 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:14:16.728 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:16.728 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:16.728 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:16.728 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:16.728 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:14:16.728 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:14:16.728 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:14:16.728 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:14:16.728 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:14:16.728 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:16.728 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:16.728 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:14:16.728 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:14:16.729 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:14:16.729 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:14:16.729 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:16.729 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:14:16.729 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:14:16.729 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:16.729 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:16.729 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:16.729 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:14:16.729 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:14:16.729 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:16.729 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:16.729 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:16.729 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:16.730 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:14:16.730 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:14:16.730 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:14:16.730 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:14:16.730 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:14:16.730 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:16.730 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:16.730 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:14:16.730 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:14:16.730 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:14:16.730 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:14:16.730 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:16.730 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:14:16.730 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:14:16.730 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:16.731 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:16.731 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:16.731 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:14:16.731 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:14:16.731 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:16.731 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:16.731 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:16.731 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:16.731 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:14:16.731 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:14:16.731 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:14:16.731 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:14:16.731 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:14:16.731 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:16.732 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:16.732 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:16.732 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:14:16.732 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:14:16.732 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:16.732 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:14:16.732 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:14:16.732 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:14:16.732 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:16.732 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:14:16.732 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:14:16.732 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:16.732 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:14:16.732 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:14:16.733 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 6 2024-04-22T08:14:16.733 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:16.733 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:14:16.733 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": [ 2024-04-22T08:14:16.733 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:16.733 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:14:16.733 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:14:16.733 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:14:16.733 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:16.733 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:16.733 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:16.733 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:14:16.733 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:14:16.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.709+0000 7fdeed5ca640 1 -- 172.21.15.40:0/2802660183 >> v2:172.21.15.189:6800/30168 conn(0x7fdec80377b0 msgr2=0x7fdec8039c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:16.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.709+0000 7fdeed5ca640 1 --2- 172.21.15.40:0/2802660183 >> v2:172.21.15.189:6800/30168 conn(0x7fdec80377b0 0x7fdec8039c70 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fdee812f850 tx=0x7fdee0007d10 comp rx=0 tx=0).stop 2024-04-22T08:14:16.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.709+0000 7fdeed5ca640 10 monclient: shutdown 2024-04-22T08:14:16.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.709+0000 7fdeed5ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:14:16.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.709+0000 7fdeed5ca640 1 -- 172.21.15.40:0/2802660183 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdee812fad0 msgr2=0x7fdee81cab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:16.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.709+0000 7fdeed5ca640 1 --2- 172.21.15.40:0/2802660183 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdee812fad0 0x7fdee81cab40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fded80097f0 tx=0x7fded80046c0 comp rx=0 tx=0).stop 2024-04-22T08:14:16.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.709+0000 7fdee77fe640 1 -- 172.21.15.40:0/2802660183 reap_dead start 2024-04-22T08:14:16.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.709+0000 7fdee77fe640 1 -- 172.21.15.40:0/2802660183 reap_dead start 2024-04-22T08:14:16.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.709+0000 7fdeed5ca640 1 -- 172.21.15.40:0/2802660183 shutdown_connections 2024-04-22T08:14:16.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.709+0000 7fdeed5ca640 1 -- 172.21.15.40:0/2802660183 >> 172.21.15.40:0/2802660183 conn(0x7fdee80a0ae0 msgr2=0x7fdee80a2770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:14:16.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.710+0000 7fdeed5ca640 1 -- 172.21.15.40:0/2802660183 shutdown_connections 2024-04-22T08:14:16.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:16.710+0000 7fdeed5ca640 1 -- 172.21.15.40:0/2802660183 wait complete. 2024-04-22T08:14:16.736 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-22T08:14:16.736 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-04-22T08:14:17.861 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:14:17.861 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[34883]: starting. 2024-04-22T08:14:17.861 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[34883]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-34848 2024-04-22T08:14:17.861 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[34883]: created object 0... 2024-04-22T08:14:17.861 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[34883]: created object 25... 2024-04-22T08:14:17.862 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[34883]: created object 49... 2024-04-22T08:14:17.862 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[34883]: finishing. 2024-04-22T08:14:17.862 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[34883]: shutting down. 2024-04-22T08:14:17.901 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:14:17.902 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[34885]: starting. 2024-04-22T08:14:17.902 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[34885]: listing objects. 2024-04-22T08:14:17.902 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[34885]: listed object 0... 2024-04-22T08:14:17.902 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[34885]: listed object 25... 2024-04-22T08:14:17.902 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[34885]: saw 50 objects 2024-04-22T08:14:17.902 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[34885]: shutting down. 2024-04-22T08:14:18.548 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:14:18.548 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[34884]: starting. 2024-04-22T08:14:18.548 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[34884]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-34848 2024-04-22T08:14:18.548 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[34884]: shutting down. 2024-04-22T08:14:18.550 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:14:18.550 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:14:18.550 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:14:18.552 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:14:21.222 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[34954]: starting. 2024-04-22T08:14:21.222 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[34954]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-34953 2024-04-22T08:14:21.222 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[34954]: created object 0... 2024-04-22T08:14:21.222 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[34954]: created object 25... 2024-04-22T08:14:21.222 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[34954]: created object 49... 2024-04-22T08:14:21.222 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[34954]: finishing. 2024-04-22T08:14:21.223 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[34954]: shutting down. 2024-04-22T08:14:21.565 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[34955]: starting. 2024-04-22T08:14:21.565 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[34955]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-34953 2024-04-22T08:14:21.565 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[34955]: shutting down. 2024-04-22T08:14:23.905 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:14:23.906 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[34990]: starting. 2024-04-22T08:14:23.906 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[34990]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-34953 2024-04-22T08:14:23.906 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[34990]: created object 0... 2024-04-22T08:14:23.906 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[34990]: created object 25... 2024-04-22T08:14:23.906 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[34990]: created object 49... 2024-04-22T08:14:23.906 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[34990]: finishing. 2024-04-22T08:14:23.906 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[34990]: shutting down. 2024-04-22T08:14:23.947 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:14:23.947 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[34992]: starting. 2024-04-22T08:14:23.947 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[34992]: listing objects. 2024-04-22T08:14:23.947 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[34992]: listed object 0... 2024-04-22T08:14:23.947 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[34992]: listed object 25... 2024-04-22T08:14:23.947 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[34992]: saw 50 objects 2024-04-22T08:14:23.947 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[34992]: shutting down. 2024-04-22T08:14:24.580 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:14:24.580 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[34991]: starting. 2024-04-22T08:14:24.580 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[34991]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-34953 2024-04-22T08:14:24.580 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[34991]: shutting down. 2024-04-22T08:14:24.582 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:14:24.582 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:14:24.582 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:14:24.585 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:14:26.780 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35061]: starting. 2024-04-22T08:14:26.780 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35061]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-35060 2024-04-22T08:14:26.780 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35061]: created object 0... 2024-04-22T08:14:26.780 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35061]: created object 25... 2024-04-22T08:14:26.781 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35061]: created object 49... 2024-04-22T08:14:26.781 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35061]: finishing. 2024-04-22T08:14:26.781 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35061]: shutting down. 2024-04-22T08:14:27.656 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[35062]: starting. 2024-04-22T08:14:27.656 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[35062]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-35060 2024-04-22T08:14:27.656 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[35062]: shutting down. 2024-04-22T08:14:29.801 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:14:29.801 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35097]: starting. 2024-04-22T08:14:29.801 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35097]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-35060 2024-04-22T08:14:29.801 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35097]: created object 0... 2024-04-22T08:14:29.801 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35097]: created object 25... 2024-04-22T08:14:29.801 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35097]: created object 49... 2024-04-22T08:14:29.801 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35097]: finishing. 2024-04-22T08:14:29.801 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35097]: shutting down. 2024-04-22T08:14:29.843 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:14:29.843 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35099]: starting. 2024-04-22T08:14:29.843 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35099]: listing objects. 2024-04-22T08:14:29.843 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35099]: listed object 0... 2024-04-22T08:14:29.843 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35099]: listed object 25... 2024-04-22T08:14:29.843 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35099]: saw 50 objects 2024-04-22T08:14:29.843 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35099]: shutting down. 2024-04-22T08:14:30.681 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:14:30.682 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[35098]: starting. 2024-04-22T08:14:30.682 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[35098]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-35060 2024-04-22T08:14:30.682 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[35098]: shutting down. 2024-04-22T08:14:30.686 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:14:30.686 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:14:30.686 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:14:30.687 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:14:32.849 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35168]: starting. 2024-04-22T08:14:32.850 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35168]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-35167 2024-04-22T08:14:32.850 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35168]: created object 0... 2024-04-22T08:14:32.850 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35168]: created object 25... 2024-04-22T08:14:32.850 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35168]: created object 49... 2024-04-22T08:14:32.850 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35168]: finishing. 2024-04-22T08:14:32.850 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35168]: shutting down. 2024-04-22T08:14:33.741 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[35169]: starting. 2024-04-22T08:14:33.741 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[35169]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-35167 2024-04-22T08:14:33.741 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[35169]: shutting down. 2024-04-22T08:14:35.864 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:14:35.864 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35204]: starting. 2024-04-22T08:14:35.864 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35204]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-35167 2024-04-22T08:14:35.864 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35204]: created object 0... 2024-04-22T08:14:35.864 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35204]: created object 25... 2024-04-22T08:14:35.864 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35204]: created object 49... 2024-04-22T08:14:35.864 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35204]: finishing. 2024-04-22T08:14:35.865 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35204]: shutting down. 2024-04-22T08:14:35.905 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:14:35.905 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35206]: starting. 2024-04-22T08:14:35.905 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35206]: listing objects. 2024-04-22T08:14:35.905 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35206]: listed object 0... 2024-04-22T08:14:35.906 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35206]: listed object 25... 2024-04-22T08:14:35.906 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35206]: saw 50 objects 2024-04-22T08:14:35.906 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35206]: shutting down. 2024-04-22T08:14:36.548 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:14:36.548 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[35205]: starting. 2024-04-22T08:14:36.548 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[35205]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-35167 2024-04-22T08:14:36.549 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[35205]: shutting down. 2024-04-22T08:14:36.550 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:14:36.550 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:14:36.550 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:14:36.552 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:14:36.736 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-04-22T08:14:36.737 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-04-22T08:14:36.737 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-22T08:14:36.737 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-22T08:14:36.740 INFO:tasks.ceph.mon.a:Started 2024-04-22T08:14:36.740 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-22T08:14:36.740 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:14:36.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.852+0000 7fd4f6b99640 10 monclient: get_monmap_and_config 2024-04-22T08:14:36.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.852+0000 7fd4f6b99640 10 monclient: build_initial_monmap 2024-04-22T08:14:36.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.852+0000 7fd4f6b99640 10 monclient: monmap: 2024-04-22T08:14:36.852 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:14:36.852 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:36.852 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:14:36.853096+0000 2024-04-22T08:14:36.852 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:14:36.853096+0000 2024-04-22T08:14:36.852 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:14:36.852 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:36.852 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:14:36.852 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:14:36.853 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:14:36.853 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:14:36.853 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:14:36.853 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:36.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.853+0000 7fd4f6b99640 1 Processor -- start 2024-04-22T08:14:36.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.853+0000 7fd4f6b99640 1 -- start start 2024-04-22T08:14:36.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.853+0000 7fd4f6b99640 10 monclient: init 2024-04-22T08:14:36.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.853+0000 7fd4f6b99640 10 monclient: _reopen_session rank -1 2024-04-22T08:14:36.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.853+0000 7fd4f6b99640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-04-22T08:14:36.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.853+0000 7fd4f6b99640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd4f009ee10 addr v1:172.21.15.40:6789/0 2024-04-22T08:14:36.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.853+0000 7fd4f6b99640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4f00a32b0 0x7fd4f013b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:36.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.853+0000 7fd4f6b99640 10 monclient(hunting): picked mon.noname-b con 0x7fd4f00a32b0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:14:36.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.853+0000 7fd4f6b99640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd4f013b5f0 addr v1:172.21.15.189:6789/0 2024-04-22T08:14:36.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.853+0000 7fd4f6b99640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4f009f1f0 con 0x7fd4f009ee10 2024-04-22T08:14:36.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.853+0000 7fd4f6b99640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4f00a39c0 con 0x7fd4f013b5f0 2024-04-22T08:14:36.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.853+0000 7fd4f6b99640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:36.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.853+0000 7fd4f6b99640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f009fd30 con 0x7fd4f00a32b0 2024-04-22T08:14:36.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.853+0000 7fd4f6b99640 10 monclient(hunting): _renew_subs 2024-04-22T08:14:36.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.853+0000 7fd4f6b99640 10 monclient(hunting): authenticate will time out at 950.847463s 2024-04-22T08:14:36.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.853+0000 7fd4f4935640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fd4f009ee10 0x7fd4f00a1d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:35440/0 (socket says 172.21.15.40:35440) 2024-04-22T08:14:36.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.853+0000 7fd4f4935640 1 -- 172.21.15.40:0/2406967614 learned_addr learned my addr 172.21.15.40:0/2406967614 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:14:36.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.853+0000 7fd4effff640 1 --2- 172.21.15.40:0/2406967614 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4f00a32b0 0x7fd4f013b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:36.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.853+0000 7fd4effff640 10 monclient(hunting): get_auth_request con 0x7fd4f00a32b0 auth_method 0 2024-04-22T08:14:36.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.853+0000 7fd4effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:36.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.853+0000 7fd4effff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:36.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.853+0000 7fd4effff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:36.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.853+0000 7fd4effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:36.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.853+0000 7fd4effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:36.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.853+0000 7fd4effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:36.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.854+0000 7fd4effff640 10 monclient(hunting): handle_auth_done global_id 4685 payload 274 2024-04-22T08:14:36.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.854+0000 7fd4effff640 1 -- 172.21.15.40:0/2406967614 >> v1:172.21.15.189:6789/0 conn(0x7fd4f013b5f0 legacy=0x7fd4f013d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:14:36.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.854+0000 7fd4effff640 1 -- 172.21.15.40:0/2406967614 >> v1:172.21.15.40:6789/0 conn(0x7fd4f009ee10 legacy=0x7fd4f00a1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:14:36.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.854+0000 7fd4effff640 10 monclient: _finish_hunting 0 2024-04-22T08:14:36.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.854+0000 7fd4effff640 1 monclient: found mon.noname-b 2024-04-22T08:14:36.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.854+0000 7fd4effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:14:36.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.854+0000 7fd4effff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:14:36.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.854+0000 7fd4effff640 1 -- 172.21.15.40:0/2406967614 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4f00a02f0 con 0x7fd4f00a32b0 2024-04-22T08:14:36.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.857+0000 7fd4effff640 1 -- 172.21.15.40:0/2406967614 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4f00a32b0 msgr2=0x7fd4f013b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:36.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.857+0000 7fd4effff640 1 --2- 172.21.15.40:0/2406967614 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4f00a32b0 0x7fd4f013b0b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd4f0074690 tx=0x7fd4e402b6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=897f8e9104d5c17f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:36.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.857+0000 7fd4ef7fe640 1 -- 172.21.15.40:0/2406967614 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e4030a00 con 0x7fd4f00a32b0 2024-04-22T08:14:36.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.857+0000 7fd4ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:36.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.857+0000 7fd4ef7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:14:36.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.857+0000 7fd4ef7fe640 10 monclient: dump: 2024-04-22T08:14:36.863 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:36.863 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:36.863 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:36.864 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:36.864 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:36.864 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:36.864 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:36.864 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:36.864 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:36.864 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:36.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.857+0000 7fd4ef7fe640 10 monclient: _finish_auth 0 2024-04-22T08:14:36.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.857+0000 7fd4ef7fe640 10 monclient: _check_auth_tickets 2024-04-22T08:14:36.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.857+0000 7fd4ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:14:36.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.857+0000 7fd4ef7fe640 1 -- 172.21.15.40:0/2406967614 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4e402e050 con 0x7fd4f00a32b0 2024-04-22T08:14:36.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.857+0000 7fd4ef7fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:14:36.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.857+0000 7fd4ef7fe640 1 -- 172.21.15.40:0/2406967614 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e4030d80 con 0x7fd4f00a32b0 2024-04-22T08:14:36.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.857+0000 7fd4ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:36.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.857+0000 7fd4ef7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:14:36.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.857+0000 7fd4ef7fe640 10 monclient: dump: 2024-04-22T08:14:36.865 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:36.865 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:36.865 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:36.865 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:36.865 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:36.865 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:36.865 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:36.865 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:36.865 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:36.865 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:36.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.857+0000 7fd4f6b99640 5 monclient: authenticate success, global_id 4685 2024-04-22T08:14:36.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.857+0000 7fd4f6b99640 10 monclient: get_monmap_and_config success 2024-04-22T08:14:36.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.857+0000 7fd4f6b99640 4 set_mon_vals no callback set 2024-04-22T08:14:36.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.857+0000 7fd4f6b99640 10 monclient: shutdown 2024-04-22T08:14:36.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.857+0000 7fd4f6b99640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:14:36.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.857+0000 7fd4f6b99640 1 -- 172.21.15.40:0/2406967614 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4f00a32b0 msgr2=0x7fd4f013b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:36.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.857+0000 7fd4f6b99640 1 --2- 172.21.15.40:0/2406967614 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4f00a32b0 0x7fd4f013b0b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd4f0074690 tx=0x7fd4e402b6d0 comp rx=0 tx=0).stop 2024-04-22T08:14:36.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.857+0000 7fd4f6b99640 1 -- 172.21.15.40:0/2406967614 shutdown_connections 2024-04-22T08:14:36.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.857+0000 7fd4f6b99640 1 --2- 172.21.15.40:0/2406967614 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4f00a32b0 0x7fd4f013b0b0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:36.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.857+0000 7fd4f6b99640 1 -- 172.21.15.40:0/2406967614 >> 172.21.15.40:0/2406967614 conn(0x7fd4f00a4ef0 msgr2=0x7fd4f00a5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:14:36.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.857+0000 7fd4f6b99640 1 -- 172.21.15.40:0/2406967614 shutdown_connections 2024-04-22T08:14:36.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.857+0000 7fd4f6b99640 1 -- 172.21.15.40:0/2406967614 wait complete. 2024-04-22T08:14:36.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.858+0000 7fd4f6b99640 10 monclient: build_initial_monmap 2024-04-22T08:14:36.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.858+0000 7fd4f6b99640 10 monclient: monmap: 2024-04-22T08:14:36.867 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:14:36.867 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:14:36.867 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:14:36.867 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:14:36.867 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:14:36.867 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:36.867 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:14:36.867 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:14:36.867 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:14:36.867 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:36.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.858+0000 7fd4f6b99640 1 Processor -- start 2024-04-22T08:14:36.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.858+0000 7fd4f6b99640 1 -- start start 2024-04-22T08:14:36.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.858+0000 7fd4f6b99640 10 monclient: init 2024-04-22T08:14:36.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4f6b99640 10 monclient: _reopen_session rank -1 2024-04-22T08:14:36.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4f6b99640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-22T08:14:36.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4f6b99640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4f009ee10 0x7fd4f006e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:36.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4f6b99640 10 monclient(hunting): picked mon.noname-c con 0x7fd4f009ee10 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:14:36.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4f6b99640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4f00a32b0 0x7fd4f006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:36.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4f6b99640 10 monclient(hunting): picked mon.noname-a con 0x7fd4f00a32b0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:14:36.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4f6b99640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4f013b5f0 0x7fd4f006f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:36.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4f6b99640 10 monclient(hunting): picked mon.noname-b con 0x7fd4f013b5f0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:14:36.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4f6b99640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:36.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4f6b99640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f011f250 con 0x7fd4f00a32b0 2024-04-22T08:14:36.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4f6b99640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:36.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4f6b99640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f0070db0 con 0x7fd4f013b5f0 2024-04-22T08:14:36.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4f6b99640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:36.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4f6b99640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f00a2730 con 0x7fd4f009ee10 2024-04-22T08:14:36.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4f6b99640 10 monclient(hunting): _renew_subs 2024-04-22T08:14:36.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4f6b99640 10 monclient(hunting): authenticate will time out at 950.853129s 2024-04-22T08:14:36.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4f4935640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4f009ee10 0x7fd4f006e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:36.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4effff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4f00a32b0 0x7fd4f006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:36.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4f4935640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4f009ee10 0x7fd4f006e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:46854/0 (socket says 172.21.15.40:46854) 2024-04-22T08:14:36.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4effff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4f00a32b0 0x7fd4f006ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40838/0 (socket says 172.21.15.40:40838) 2024-04-22T08:14:36.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4f4935640 1 -- 172.21.15.40:0/2577502468 learned_addr learned my addr 172.21.15.40:0/2577502468 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:14:36.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4f4935640 10 monclient(hunting): get_auth_request con 0x7fd4f009ee10 auth_method 0 2024-04-22T08:14:36.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4f4935640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:36.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4f4935640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:36.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4f4935640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:36.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4effff640 10 monclient(hunting): get_auth_request con 0x7fd4f00a32b0 auth_method 0 2024-04-22T08:14:36.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:36.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4effff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:36.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4effff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:36.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4f4935640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:36.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4f4935640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:36.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4effff640 1 -- 172.21.15.40:0/2577502468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4f00a32b0 msgr2=0x7fd4f006ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-22T08:14:36.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4effff640 1 -- 172.21.15.40:0/2577502468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4f00a32b0 msgr2=0x7fd4f006ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:14:36.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4effff640 1 --2- 172.21.15.40:0/2577502468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4f00a32b0 0x7fd4f006ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:14:36.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4f4935640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:36.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4effff640 1 --2- 172.21.15.40:0/2577502468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4f00a32b0 0x7fd4f006ed70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:14:36.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4f5136640 1 --2- 172.21.15.40:0/2577502468 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4f013b5f0 0x7fd4f006f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:36.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4f5136640 10 monclient(hunting): get_auth_request con 0x7fd4f013b5f0 auth_method 0 2024-04-22T08:14:36.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4f5136640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:36.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4f5136640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:36.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4f5136640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:36.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4f4935640 10 monclient(hunting): handle_auth_done global_id 4688 payload 824 2024-04-22T08:14:36.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4f4935640 1 -- 172.21.15.40:0/2577502468 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4f013b5f0 msgr2=0x7fd4f006f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:36.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4f4935640 1 --2- 172.21.15.40:0/2577502468 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4f013b5f0 0x7fd4f006f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:36.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4f4935640 1 -- 172.21.15.40:0/2577502468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4f00a32b0 msgr2=0x7fd4f006ed70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:14:36.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4f4935640 1 --2- 172.21.15.40:0/2577502468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4f00a32b0 0x7fd4f006ed70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:36.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4f4935640 10 monclient: _finish_hunting 0 2024-04-22T08:14:36.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4f4935640 1 monclient: found mon.noname-c 2024-04-22T08:14:36.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4f4935640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:14:36.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4f4935640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:14:36.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.859+0000 7fd4f4935640 1 -- 172.21.15.40:0/2577502468 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e4009400 con 0x7fd4f009ee10 2024-04-22T08:14:36.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.862+0000 7fd4f4935640 1 -- 172.21.15.40:0/2577502468 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4f009ee10 msgr2=0x7fd4f006e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:36.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.862+0000 7fd4f4935640 1 --2- 172.21.15.40:0/2577502468 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4f009ee10 0x7fd4f006e830 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd4e000b460 tx=0x7fd4e000b920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:36.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.862+0000 7fd4ed7fa640 1 -- 172.21.15.40:0/2577502468 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e0011aa0 con 0x7fd4f009ee10 2024-04-22T08:14:36.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.862+0000 7fd4ed7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:36.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.862+0000 7fd4ed7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:14:36.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.862+0000 7fd4ed7fa640 10 monclient: dump: 2024-04-22T08:14:36.872 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:36.872 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:36.872 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:36.872 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:36.872 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:36.872 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:36.872 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:36.872 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:36.872 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:36.872 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:36.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.862+0000 7fd4ed7fa640 10 monclient: _finish_auth 0 2024-04-22T08:14:36.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.862+0000 7fd4ed7fa640 10 monclient: _check_auth_tickets 2024-04-22T08:14:36.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.862+0000 7fd4ed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:14:36.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.862+0000 7fd4f6b99640 5 monclient: authenticate success, global_id 4688 2024-04-22T08:14:36.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.862+0000 7fd4f6b99640 10 monclient: _renew_subs 2024-04-22T08:14:36.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.862+0000 7fd4ed7fa640 1 -- 172.21.15.40:0/2577502468 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4e0011c20 con 0x7fd4f009ee10 2024-04-22T08:14:36.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.862+0000 7fd4f6b99640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:14:36.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.862+0000 7fd4f6b99640 1 -- 172.21.15.40:0/2577502468 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4f00a7d30 con 0x7fd4f009ee10 2024-04-22T08:14:36.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.862+0000 7fd4ed7fa640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:14:36.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.862+0000 7fd4ed7fa640 1 -- 172.21.15.40:0/2577502468 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e00103e0 con 0x7fd4f009ee10 2024-04-22T08:14:36.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.862+0000 7fd4ed7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:36.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.862+0000 7fd4ed7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:14:36.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.862+0000 7fd4ed7fa640 10 monclient: dump: 2024-04-22T08:14:36.873 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:36.873 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:36.873 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:36.873 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:36.873 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:36.874 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:36.874 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:36.874 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:36.874 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:36.874 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:36.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.862+0000 7fd4f6b99640 10 monclient: _renew_subs 2024-04-22T08:14:36.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.862+0000 7fd4f6b99640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:14:36.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.862+0000 7fd4f6b99640 1 -- 172.21.15.40:0/2577502468 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4f0070ac0 con 0x7fd4f009ee10 2024-04-22T08:14:36.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.862+0000 7fd4eeffd640 4 set_mon_vals no callback set 2024-04-22T08:14:36.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.863+0000 7fd4ed7fa640 1 -- 172.21.15.40:0/2577502468 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fd4e0010580 con 0x7fd4f009ee10 2024-04-22T08:14:36.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.863+0000 7fd4f6b99640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:14:36.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.863+0000 7fd4f6b99640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:14:36.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.863+0000 7fd4f6b99640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:14:36.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.863+0000 7fd4f6b99640 1 -- 172.21.15.40:0/2577502468 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4f0070230 con 0x7fd4f009ee10 2024-04-22T08:14:36.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.863+0000 7fd4ed7fa640 1 --2- 172.21.15.40:0/2577502468 >> v2:172.21.15.189:6800/30168 conn(0x7fd4dc0377d0 0x7fd4dc039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:36.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.863+0000 7fd4ed7fa640 1 -- 172.21.15.40:0/2577502468 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 3506+0+0 (secure 0 0 0) 0x7fd4e004a6d0 con 0x7fd4f009ee10 2024-04-22T08:14:36.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.863+0000 7fd4effff640 1 --2- 172.21.15.40:0/2577502468 >> v2:172.21.15.189:6800/30168 conn(0x7fd4dc0377d0 0x7fd4dc039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:36.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.863+0000 7fd4effff640 10 monclient: get_auth_request con 0x7fd4dc0377d0 auth_method 0 2024-04-22T08:14:36.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.866+0000 7fd4effff640 1 --2- 172.21.15.40:0/2577502468 >> v2:172.21.15.189:6800/30168 conn(0x7fd4dc0377d0 0x7fd4dc039c90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd4f00a43e0 tx=0x7fd4e4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:36.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.868+0000 7fd4ed7fa640 1 -- 172.21.15.40:0/2577502468 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7fd4f0070230 con 0x7fd4f009ee10 2024-04-22T08:14:36.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.868+0000 7fd4ed7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:14:36.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:36.868+0000 7fd4ed7fa640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:14:37.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:37.120+0000 7fd4f6b99640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:14:37.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:37.120+0000 7fd4f6b99640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:14:37.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:37.120+0000 7fd4f6b99640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:14:37.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:37.120+0000 7fd4f6b99640 1 -- 172.21.15.40:0/2577502468 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd4f0070230 con 0x7fd4f009ee10 2024-04-22T08:14:37.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:37.120+0000 7fd4ed7fa640 1 -- 172.21.15.40:0/2577502468 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7fd4f0070230 con 0x7fd4f009ee10 2024-04-22T08:14:37.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:37.121+0000 7fd4ed7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:14:37.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:37.121+0000 7fd4ed7fa640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:14:37.121 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":22,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":22,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:14:37.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:37.122+0000 7fd4f6b99640 1 -- 172.21.15.40:0/2577502468 >> v2:172.21.15.189:6800/30168 conn(0x7fd4dc0377d0 msgr2=0x7fd4dc039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:37.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:37.122+0000 7fd4f6b99640 1 --2- 172.21.15.40:0/2577502468 >> v2:172.21.15.189:6800/30168 conn(0x7fd4dc0377d0 0x7fd4dc039c90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd4f00a43e0 tx=0x7fd4e4008030 comp rx=0 tx=0).stop 2024-04-22T08:14:37.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:37.122+0000 7fd4f6b99640 10 monclient: shutdown 2024-04-22T08:14:37.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:37.122+0000 7fd4f6b99640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:14:37.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:37.122+0000 7fd4f6b99640 1 -- 172.21.15.40:0/2577502468 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4f009ee10 msgr2=0x7fd4f006e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:37.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:37.122+0000 7fd4f6b99640 1 --2- 172.21.15.40:0/2577502468 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4f009ee10 0x7fd4f006e830 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd4e000b460 tx=0x7fd4e000b920 comp rx=0 tx=0).stop 2024-04-22T08:14:37.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:37.122+0000 7fd4f6b99640 1 -- 172.21.15.40:0/2577502468 shutdown_connections 2024-04-22T08:14:37.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:37.122+0000 7fd4f6b99640 1 --2- 172.21.15.40:0/2577502468 >> v2:172.21.15.189:6800/30168 conn(0x7fd4dc0377d0 0x7fd4dc039c90 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:37.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:37.122+0000 7fd4f6b99640 1 --2- 172.21.15.40:0/2577502468 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4f013b5f0 0x7fd4f006f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:37.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:37.122+0000 7fd4f6b99640 1 --2- 172.21.15.40:0/2577502468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4f00a32b0 0x7fd4f006ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:37.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:37.122+0000 7fd4f6b99640 1 --2- 172.21.15.40:0/2577502468 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4f009ee10 0x7fd4f006e830 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:37.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:37.122+0000 7fd4f6b99640 1 -- 172.21.15.40:0/2577502468 >> 172.21.15.40:0/2577502468 conn(0x7fd4f00a4ef0 msgr2=0x7fd4f013e2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:14:37.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:37.123+0000 7fd4f6b99640 1 -- 172.21.15.40:0/2577502468 shutdown_connections 2024-04-22T08:14:37.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:37.123+0000 7fd4f6b99640 1 -- 172.21.15.40:0/2577502468 wait complete. 2024-04-22T08:14:38.677 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35275]: starting. 2024-04-22T08:14:38.678 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35275]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-35274 2024-04-22T08:14:38.678 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35275]: created object 0... 2024-04-22T08:14:38.678 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35275]: created object 25... 2024-04-22T08:14:38.678 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35275]: created object 49... 2024-04-22T08:14:38.678 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35275]: finishing. 2024-04-22T08:14:38.678 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35275]: shutting down. 2024-04-22T08:14:39.555 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[35276]: starting. 2024-04-22T08:14:39.555 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[35276]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-35274 2024-04-22T08:14:39.555 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[35276]: shutting down. 2024-04-22T08:14:40.134 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:14:40.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.244+0000 7f062e1fd640 10 monclient: get_monmap_and_config 2024-04-22T08:14:40.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.244+0000 7f062e1fd640 10 monclient: build_initial_monmap 2024-04-22T08:14:40.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.244+0000 7f062e1fd640 10 monclient: monmap: 2024-04-22T08:14:40.244 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:14:40.244 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:40.244 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:14:40.245527+0000 2024-04-22T08:14:40.244 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:14:40.245527+0000 2024-04-22T08:14:40.245 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:14:40.245 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:40.245 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:14:40.245 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:14:40.245 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:14:40.245 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:14:40.245 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:14:40.245 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:40.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.245+0000 7f062e1fd640 1 Processor -- start 2024-04-22T08:14:40.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.245+0000 7f062e1fd640 1 -- start start 2024-04-22T08:14:40.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.245+0000 7f062e1fd640 10 monclient: init 2024-04-22T08:14:40.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.246+0000 7f062e1fd640 10 monclient: _reopen_session rank -1 2024-04-22T08:14:40.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.246+0000 7f062e1fd640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-04-22T08:14:40.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.246+0000 7f062e1fd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f062809ee10 addr v1:172.21.15.40:6789/0 2024-04-22T08:14:40.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.246+0000 7f062e1fd640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f06280a32b0 0x7f062813b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:40.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.246+0000 7f062e1fd640 10 monclient(hunting): picked mon.noname-a con 0x7f06280a32b0 addr v2:172.21.15.40:3300/0 2024-04-22T08:14:40.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.246+0000 7f062e1fd640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f062813b5f0 0x7f062813d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:40.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.246+0000 7f062e1fd640 10 monclient(hunting): picked mon.noname-c con 0x7f062813b5f0 addr v2:172.21.15.189:3300/0 2024-04-22T08:14:40.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.246+0000 7f062e1fd640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f062809f1f0 con 0x7f062809ee10 2024-04-22T08:14:40.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.246+0000 7f062e1fd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:40.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.246+0000 7f062e1fd640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06280a39c0 con 0x7f06280a32b0 2024-04-22T08:14:40.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.246+0000 7f062e1fd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:40.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.246+0000 7f062e1fd640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06280a3700 con 0x7f062813b5f0 2024-04-22T08:14:40.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.246+0000 7f062e1fd640 10 monclient(hunting): _renew_subs 2024-04-22T08:14:40.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.246+0000 7f062e1fd640 10 monclient(hunting): authenticate will time out at 954.240003s 2024-04-22T08:14:40.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.246+0000 7f0626ffd640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f06280a32b0 0x7f062813b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:40.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.246+0000 7f06277fe640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f062809ee10 0x7f06280a1d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:58952/0 (socket says 172.21.15.40:58952) 2024-04-22T08:14:40.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.246+0000 7f06277fe640 1 -- 172.21.15.40:0/1482496847 learned_addr learned my addr 172.21.15.40:0/1482496847 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:14:40.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.246+0000 7f0626ffd640 1 --2- 172.21.15.40:0/1482496847 >> v2:172.21.15.40:3300/0 conn(0x7f06280a32b0 0x7f062813b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35008/0 (socket says 172.21.15.40:35008) 2024-04-22T08:14:40.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.246+0000 7f0626ffd640 10 monclient(hunting): get_auth_request con 0x7f06280a32b0 auth_method 0 2024-04-22T08:14:40.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.246+0000 7f0626ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:40.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.246+0000 7f0626ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:40.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.246+0000 7f0626ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:40.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.246+0000 7f0626ffd640 1 -- 172.21.15.40:0/1482496847 >> v2:172.21.15.40:3300/0 conn(0x7f06280a32b0 msgr2=0x7f062813b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:14:40.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.246+0000 7f0626ffd640 1 -- 172.21.15.40:0/1482496847 >> v2:172.21.15.40:3300/0 conn(0x7f06280a32b0 msgr2=0x7f062813b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:14:40.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.246+0000 7f0626ffd640 1 --2- 172.21.15.40:0/1482496847 >> v2:172.21.15.40:3300/0 conn(0x7f06280a32b0 0x7f062813b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:14:40.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.246+0000 7f0626ffd640 1 --2- 172.21.15.40:0/1482496847 >> v2:172.21.15.40:3300/0 conn(0x7f06280a32b0 0x7f062813b0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:14:40.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.246+0000 7f0627fff640 1 --2- 172.21.15.40:0/1482496847 >> v2:172.21.15.189:3300/0 conn(0x7f062813b5f0 0x7f062813d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:40.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.246+0000 7f0627fff640 10 monclient(hunting): get_auth_request con 0x7f062813b5f0 auth_method 0 2024-04-22T08:14:40.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.246+0000 7f0627fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:40.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.246+0000 7f0627fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:40.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.246+0000 7f0627fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:40.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.246+0000 7f0627fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:40.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.246+0000 7f0627fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:40.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.246+0000 7f0627fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:40.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.246+0000 7f0627fff640 10 monclient(hunting): handle_auth_done global_id 24289 payload 274 2024-04-22T08:14:40.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.247+0000 7f0627fff640 1 -- 172.21.15.40:0/1482496847 >> v2:172.21.15.40:3300/0 conn(0x7f06280a32b0 msgr2=0x7f062813b0b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:14:40.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.247+0000 7f0627fff640 1 --2- 172.21.15.40:0/1482496847 >> v2:172.21.15.40:3300/0 conn(0x7f06280a32b0 0x7f062813b0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:40.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.247+0000 7f0627fff640 1 -- 172.21.15.40:0/1482496847 >> v1:172.21.15.40:6789/0 conn(0x7f062809ee10 legacy=0x7f06280a1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:14:40.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.247+0000 7f0627fff640 10 monclient: _finish_hunting 0 2024-04-22T08:14:40.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.247+0000 7f0627fff640 1 monclient: found mon.noname-c 2024-04-22T08:14:40.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.247+0000 7f0627fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:14:40.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.247+0000 7f0627fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.189:3300/0 2024-04-22T08:14:40.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.247+0000 7f0627fff640 1 -- 172.21.15.40:0/1482496847 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f062809fd30 con 0x7f062813b5f0 2024-04-22T08:14:40.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.250+0000 7f0627fff640 1 -- 172.21.15.40:0/1482496847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f062813b5f0 msgr2=0x7f062813d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:40.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.250+0000 7f0627fff640 1 --2- 172.21.15.40:0/1482496847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f062813b5f0 0x7f062813d9e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f062806c620 tx=0x7f062002bce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90824aec72030db server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:40.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.250+0000 7f06267fc640 1 -- 172.21.15.40:0/1482496847 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f062002cca0 con 0x7f062813b5f0 2024-04-22T08:14:40.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.250+0000 7f06267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:40.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.250+0000 7f06267fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:14:40.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.250+0000 7f06267fc640 10 monclient: dump: 2024-04-22T08:14:40.259 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:40.259 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:40.259 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:40.259 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:40.259 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:40.259 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:40.259 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:40.259 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:40.260 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:40.260 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:40.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.250+0000 7f06267fc640 10 monclient: _finish_auth 0 2024-04-22T08:14:40.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.250+0000 7f06267fc640 10 monclient: _check_auth_tickets 2024-04-22T08:14:40.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.250+0000 7f06267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:14:40.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.250+0000 7f06267fc640 1 -- 172.21.15.40:0/1482496847 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f062002ce00 con 0x7f062813b5f0 2024-04-22T08:14:40.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.250+0000 7f06267fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:14:40.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.250+0000 7f06267fc640 1 -- 172.21.15.40:0/1482496847 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0620034660 con 0x7f062813b5f0 2024-04-22T08:14:40.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.250+0000 7f062e1fd640 5 monclient: authenticate success, global_id 24289 2024-04-22T08:14:40.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.250+0000 7f062e1fd640 10 monclient: get_monmap_and_config success 2024-04-22T08:14:40.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.250+0000 7f062e1fd640 4 set_mon_vals no callback set 2024-04-22T08:14:40.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.250+0000 7f06267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:40.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.250+0000 7f06267fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:14:40.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.250+0000 7f06267fc640 10 monclient: dump: 2024-04-22T08:14:40.261 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:40.261 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:40.261 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:40.261 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:40.261 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:40.261 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:40.261 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:40.261 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:40.261 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:40.261 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:40.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.250+0000 7f062e1fd640 10 monclient: shutdown 2024-04-22T08:14:40.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.250+0000 7f062e1fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:14:40.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.250+0000 7f062e1fd640 1 -- 172.21.15.40:0/1482496847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f062813b5f0 msgr2=0x7f062813d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:40.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.250+0000 7f062e1fd640 1 --2- 172.21.15.40:0/1482496847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f062813b5f0 0x7f062813d9e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f062806c620 tx=0x7f062002bce0 comp rx=0 tx=0).stop 2024-04-22T08:14:40.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.251+0000 7f062e1fd640 1 -- 172.21.15.40:0/1482496847 shutdown_connections 2024-04-22T08:14:40.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.251+0000 7f062e1fd640 1 --2- 172.21.15.40:0/1482496847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f062813b5f0 0x7f062813d9e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:40.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.251+0000 7f062e1fd640 1 --2- 172.21.15.40:0/1482496847 >> v2:172.21.15.40:3300/0 conn(0x7f06280a32b0 0x7f062813b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:40.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.251+0000 7f062e1fd640 1 -- 172.21.15.40:0/1482496847 >> 172.21.15.40:0/1482496847 conn(0x7f06280a4ef0 msgr2=0x7f06280a5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:14:40.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.251+0000 7f062e1fd640 1 -- 172.21.15.40:0/1482496847 shutdown_connections 2024-04-22T08:14:40.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.251+0000 7f062e1fd640 1 -- 172.21.15.40:0/1482496847 wait complete. 2024-04-22T08:14:40.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.251+0000 7f062e1fd640 10 monclient: build_initial_monmap 2024-04-22T08:14:40.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.251+0000 7f062e1fd640 10 monclient: monmap: 2024-04-22T08:14:40.262 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:14:40.262 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:14:40.262 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:14:40.262 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:14:40.262 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:14:40.262 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:40.262 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:14:40.263 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:14:40.263 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:14:40.263 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:40.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.251+0000 7f062e1fd640 1 Processor -- start 2024-04-22T08:14:40.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.252+0000 7f062e1fd640 1 -- start start 2024-04-22T08:14:40.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.252+0000 7f062e1fd640 10 monclient: init 2024-04-22T08:14:40.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.252+0000 7f062e1fd640 10 monclient: _reopen_session rank -1 2024-04-22T08:14:40.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.252+0000 7f062e1fd640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-22T08:14:40.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.252+0000 7f062e1fd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f062809ee10 0x7f06281ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:40.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.252+0000 7f062e1fd640 10 monclient(hunting): picked mon.noname-b con 0x7f062809ee10 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:14:40.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.252+0000 7f062e1fd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06280a32b0 0x7f0628136e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:40.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.252+0000 7f062e1fd640 10 monclient(hunting): picked mon.noname-a con 0x7f06280a32b0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:14:40.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.252+0000 7f062e1fd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f062813b5f0 0x7f0628137370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:40.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.252+0000 7f062e1fd640 10 monclient(hunting): picked mon.noname-c con 0x7f062813b5f0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:14:40.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.252+0000 7f062e1fd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:40.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.252+0000 7f062e1fd640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0628123890 con 0x7f06280a32b0 2024-04-22T08:14:40.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.252+0000 7f062e1fd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:40.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.252+0000 7f062e1fd640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06281cae30 con 0x7f062809ee10 2024-04-22T08:14:40.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.252+0000 7f062e1fd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:40.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.252+0000 7f062e1fd640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06280a2730 con 0x7f062813b5f0 2024-04-22T08:14:40.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.252+0000 7f062e1fd640 10 monclient(hunting): _renew_subs 2024-04-22T08:14:40.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.252+0000 7f062e1fd640 10 monclient(hunting): authenticate will time out at 954.246508s 2024-04-22T08:14:40.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.252+0000 7f0626ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06280a32b0 0x7f0628136e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:40.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.252+0000 7f0626ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06280a32b0 0x7f0628136e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35014/0 (socket says 172.21.15.40:35014) 2024-04-22T08:14:40.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.252+0000 7f0626ffd640 1 -- 172.21.15.40:0/995873883 learned_addr learned my addr 172.21.15.40:0/995873883 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:14:40.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.252+0000 7f0626ffd640 10 monclient(hunting): get_auth_request con 0x7f06280a32b0 auth_method 0 2024-04-22T08:14:40.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.252+0000 7f0626ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:40.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.252+0000 7f0626ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:40.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.252+0000 7f0626ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:40.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.252+0000 7f0627fff640 1 --2- 172.21.15.40:0/995873883 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f062813b5f0 0x7f0628137370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:40.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.252+0000 7f0627fff640 10 monclient(hunting): get_auth_request con 0x7f062813b5f0 auth_method 0 2024-04-22T08:14:40.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.252+0000 7f0627fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:40.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.252+0000 7f0627fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:40.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.252+0000 7f0627fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:40.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.252+0000 7f0626ffd640 1 -- 172.21.15.40:0/995873883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06280a32b0 msgr2=0x7f0628136e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-22T08:14:40.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.252+0000 7f0626ffd640 1 -- 172.21.15.40:0/995873883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06280a32b0 msgr2=0x7f0628136e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:14:40.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.252+0000 7f0626ffd640 1 --2- 172.21.15.40:0/995873883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06280a32b0 0x7f0628136e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:14:40.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.253+0000 7f0627fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:40.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.253+0000 7f06277fe640 1 --2- 172.21.15.40:0/995873883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f062809ee10 0x7f06281ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:40.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.253+0000 7f0627fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:40.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.253+0000 7f0626ffd640 1 --2- 172.21.15.40:0/995873883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06280a32b0 0x7f0628136e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:14:40.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.253+0000 7f0627fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:40.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.253+0000 7f06277fe640 10 monclient(hunting): get_auth_request con 0x7f062809ee10 auth_method 0 2024-04-22T08:14:40.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.253+0000 7f06277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:40.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.253+0000 7f06277fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:40.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.253+0000 7f06277fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:40.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.253+0000 7f06277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:40.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.253+0000 7f06277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:40.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.253+0000 7f06277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:40.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.253+0000 7f0627fff640 10 monclient(hunting): handle_auth_done global_id 4703 payload 824 2024-04-22T08:14:40.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.253+0000 7f0627fff640 1 -- 172.21.15.40:0/995873883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f062809ee10 msgr2=0x7f06281ca8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:40.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.253+0000 7f0627fff640 1 --2- 172.21.15.40:0/995873883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f062809ee10 0x7f06281ca8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:40.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.253+0000 7f0627fff640 1 -- 172.21.15.40:0/995873883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06280a32b0 msgr2=0x7f0628136e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:14:40.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.253+0000 7f0627fff640 1 --2- 172.21.15.40:0/995873883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06280a32b0 0x7f0628136e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:40.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.253+0000 7f0627fff640 10 monclient: _finish_hunting 0 2024-04-22T08:14:40.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.253+0000 7f0627fff640 1 monclient: found mon.noname-c 2024-04-22T08:14:40.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.253+0000 7f0627fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:14:40.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.253+0000 7f0627fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:14:40.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.253+0000 7f0627fff640 1 -- 172.21.15.40:0/995873883 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0620009a70 con 0x7f062813b5f0 2024-04-22T08:14:40.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.255+0000 7f0627fff640 1 -- 172.21.15.40:0/995873883 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f062813b5f0 msgr2=0x7f0628137370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:40.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.255+0000 7f0627fff640 1 --2- 172.21.15.40:0/995873883 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f062813b5f0 0x7f0628137370 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f06280a3840 tx=0x7f0620004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:40.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.255+0000 7f060ffff640 1 -- 172.21.15.40:0/995873883 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f062003e610 con 0x7f062813b5f0 2024-04-22T08:14:40.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.255+0000 7f060ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:40.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.255+0000 7f060ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:14:40.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.255+0000 7f060ffff640 10 monclient: dump: 2024-04-22T08:14:40.267 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:40.267 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:40.267 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:40.267 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:40.267 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:40.267 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:40.268 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:40.268 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:40.268 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:40.268 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:40.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.255+0000 7f060ffff640 10 monclient: _finish_auth 0 2024-04-22T08:14:40.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.255+0000 7f060ffff640 10 monclient: _check_auth_tickets 2024-04-22T08:14:40.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.255+0000 7f060ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:14:40.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.255+0000 7f060ffff640 1 -- 172.21.15.40:0/995873883 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f062003e770 con 0x7f062813b5f0 2024-04-22T08:14:40.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.255+0000 7f060ffff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:14:40.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.255+0000 7f060ffff640 1 -- 172.21.15.40:0/995873883 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f062003eae0 con 0x7f062813b5f0 2024-04-22T08:14:40.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.255+0000 7f062e1fd640 5 monclient: authenticate success, global_id 4703 2024-04-22T08:14:40.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.255+0000 7f060ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:40.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.255+0000 7f060ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:14:40.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.255+0000 7f060ffff640 10 monclient: dump: 2024-04-22T08:14:40.269 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:40.269 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:40.269 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:40.269 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:40.269 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:40.269 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:40.269 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:40.269 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:40.269 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:40.269 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:40.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.255+0000 7f062e1fd640 10 monclient: _renew_subs 2024-04-22T08:14:40.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.255+0000 7f062e1fd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:14:40.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.255+0000 7f062e1fd640 1 -- 172.21.15.40:0/995873883 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0628138f00 con 0x7f062813b5f0 2024-04-22T08:14:40.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.255+0000 7f06257fa640 4 set_mon_vals no callback set 2024-04-22T08:14:40.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.255+0000 7f062e1fd640 10 monclient: _renew_subs 2024-04-22T08:14:40.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.255+0000 7f062e1fd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:14:40.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.255+0000 7f062e1fd640 1 -- 172.21.15.40:0/995873883 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0628122f60 con 0x7f062813b5f0 2024-04-22T08:14:40.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.256+0000 7f060ffff640 1 -- 172.21.15.40:0/995873883 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f062003b050 con 0x7f062813b5f0 2024-04-22T08:14:40.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.256+0000 7f062e1fd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:14:40.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.256+0000 7f062e1fd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:14:40.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.256+0000 7f062e1fd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:14:40.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.256+0000 7f062e1fd640 1 -- 172.21.15.40:0/995873883 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0628123360 con 0x7f062813b5f0 2024-04-22T08:14:40.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.256+0000 7f060ffff640 1 --2- 172.21.15.40:0/995873883 >> v2:172.21.15.189:6800/30168 conn(0x7f06140377d0 0x7f0614039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:40.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.256+0000 7f060ffff640 1 -- 172.21.15.40:0/995873883 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f062006c7a0 con 0x7f062813b5f0 2024-04-22T08:14:40.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.257+0000 7f06277fe640 1 --2- 172.21.15.40:0/995873883 >> v2:172.21.15.189:6800/30168 conn(0x7f06140377d0 0x7f0614039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:40.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.257+0000 7f06277fe640 10 monclient: get_auth_request con 0x7f06140377d0 auth_method 0 2024-04-22T08:14:40.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.259+0000 7f06277fe640 1 --2- 172.21.15.40:0/995873883 >> v2:172.21.15.189:6800/30168 conn(0x7f06140377d0 0x7f0614039c90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f06180092a0 tx=0x7f0618009230 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:40.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.261+0000 7f060ffff640 1 -- 172.21.15.40:0/995873883 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f062003ecc0 con 0x7f062813b5f0 2024-04-22T08:14:40.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.261+0000 7f060ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:14:40.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.261+0000 7f060ffff640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:14:40.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.512+0000 7f062e1fd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:14:40.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.512+0000 7f062e1fd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:14:40.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.512+0000 7f062e1fd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:14:40.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.512+0000 7f062e1fd640 1 -- 172.21.15.40:0/995873883 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f062809ec30 con 0x7f062813b5f0 2024-04-22T08:14:40.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.513+0000 7f060ffff640 1 -- 172.21.15.40:0/995873883 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7f0620039070 con 0x7f062813b5f0 2024-04-22T08:14:40.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.513+0000 7f060ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:14:40.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.513+0000 7f060ffff640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:14:40.513 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":22,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":25,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:14:40.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.514+0000 7f062e1fd640 1 -- 172.21.15.40:0/995873883 >> v2:172.21.15.189:6800/30168 conn(0x7f06140377d0 msgr2=0x7f0614039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:40.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.514+0000 7f062e1fd640 1 --2- 172.21.15.40:0/995873883 >> v2:172.21.15.189:6800/30168 conn(0x7f06140377d0 0x7f0614039c90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f06180092a0 tx=0x7f0618009230 comp rx=0 tx=0).stop 2024-04-22T08:14:40.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.514+0000 7f062e1fd640 10 monclient: shutdown 2024-04-22T08:14:40.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.514+0000 7f062e1fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:14:40.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.514+0000 7f062e1fd640 1 -- 172.21.15.40:0/995873883 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f062813b5f0 msgr2=0x7f0628137370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:40.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.514+0000 7f062e1fd640 1 --2- 172.21.15.40:0/995873883 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f062813b5f0 0x7f0628137370 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f06280a3840 tx=0x7f0620004a40 comp rx=0 tx=0).stop 2024-04-22T08:14:40.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.515+0000 7f062e1fd640 1 -- 172.21.15.40:0/995873883 shutdown_connections 2024-04-22T08:14:40.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.515+0000 7f062e1fd640 1 --2- 172.21.15.40:0/995873883 >> v2:172.21.15.189:6800/30168 conn(0x7f06140377d0 0x7f0614039c90 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:40.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.515+0000 7f062e1fd640 1 --2- 172.21.15.40:0/995873883 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f062813b5f0 0x7f0628137370 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:40.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.515+0000 7f062e1fd640 1 --2- 172.21.15.40:0/995873883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06280a32b0 0x7f0628136e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:40.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.515+0000 7f062e1fd640 1 --2- 172.21.15.40:0/995873883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f062809ee10 0x7f06281ca8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:40.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.515+0000 7f062e1fd640 1 -- 172.21.15.40:0/995873883 >> 172.21.15.40:0/995873883 conn(0x7f06280a4ef0 msgr2=0x7f06280a1980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:14:40.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.515+0000 7f062e1fd640 1 -- 172.21.15.40:0/995873883 shutdown_connections 2024-04-22T08:14:40.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:40.515+0000 7f062e1fd640 1 -- 172.21.15.40:0/995873883 wait complete. 2024-04-22T08:14:41.682 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:14:41.682 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35311]: starting. 2024-04-22T08:14:41.682 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35311]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-35274 2024-04-22T08:14:41.682 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35311]: created object 0... 2024-04-22T08:14:41.682 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35311]: created object 25... 2024-04-22T08:14:41.683 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35311]: created object 49... 2024-04-22T08:14:41.683 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35311]: finishing. 2024-04-22T08:14:41.683 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35311]: shutting down. 2024-04-22T08:14:41.722 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:14:41.722 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35313]: starting. 2024-04-22T08:14:41.722 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35313]: listing objects. 2024-04-22T08:14:41.723 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35313]: listed object 0... 2024-04-22T08:14:41.723 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35313]: listed object 25... 2024-04-22T08:14:41.723 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35313]: saw 50 objects 2024-04-22T08:14:41.723 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35313]: shutting down. 2024-04-22T08:14:42.572 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:14:42.572 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[35312]: starting. 2024-04-22T08:14:42.572 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[35312]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-35274 2024-04-22T08:14:42.572 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[35312]: shutting down. 2024-04-22T08:14:42.573 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:14:42.573 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:14:42.574 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:14:42.576 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:14:43.526 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:14:43.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.639+0000 7f6c959d7640 10 monclient: get_monmap_and_config 2024-04-22T08:14:43.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.639+0000 7f6c959d7640 10 monclient: build_initial_monmap 2024-04-22T08:14:43.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.639+0000 7f6c959d7640 10 monclient: monmap: 2024-04-22T08:14:43.639 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:14:43.639 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:43.639 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:14:43.639857+0000 2024-04-22T08:14:43.639 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:14:43.639857+0000 2024-04-22T08:14:43.639 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:14:43.639 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:43.639 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:14:43.639 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:14:43.639 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:14:43.639 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:14:43.640 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:14:43.640 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:43.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.639+0000 7f6c959d7640 1 Processor -- start 2024-04-22T08:14:43.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.639+0000 7f6c959d7640 1 -- start start 2024-04-22T08:14:43.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.639+0000 7f6c959d7640 10 monclient: init 2024-04-22T08:14:43.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.640+0000 7f6c959d7640 10 monclient: _reopen_session rank -1 2024-04-22T08:14:43.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.640+0000 7f6c959d7640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-04-22T08:14:43.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.640+0000 7f6c959d7640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6c9009ee10 0x7f6c900a3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:43.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.640+0000 7f6c959d7640 10 monclient(hunting): picked mon.noname-a con 0x7f6c9009ee10 addr v2:172.21.15.40:3300/0 2024-04-22T08:14:43.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.640+0000 7f6c959d7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6c900a1820 addr v1:172.21.15.189:6789/0 2024-04-22T08:14:43.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.640+0000 7f6c959d7640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6c9013b0e0 0x7f6c9013d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:43.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.640+0000 7f6c959d7640 10 monclient(hunting): picked mon.noname-b con 0x7f6c9013b0e0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:14:43.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.640+0000 7f6c959d7640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c900a4470 con 0x7f6c900a1820 2024-04-22T08:14:43.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.640+0000 7f6c959d7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:43.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.640+0000 7f6c959d7640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c9009f1f0 con 0x7f6c9009ee10 2024-04-22T08:14:43.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.640+0000 7f6c959d7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:43.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.640+0000 7f6c959d7640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9009fd30 con 0x7f6c9013b0e0 2024-04-22T08:14:43.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.640+0000 7f6c959d7640 10 monclient(hunting): _renew_subs 2024-04-22T08:14:43.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.640+0000 7f6c959d7640 10 monclient(hunting): authenticate will time out at 957.633996s 2024-04-22T08:14:43.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.640+0000 7f6c8effd640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6c9009ee10 0x7f6c900a3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:43.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.640+0000 7f6c8f7fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6c9013b0e0 0x7f6c9013d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:43.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.640+0000 7f6c8effd640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6c9009ee10 0x7f6c900a3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35030/0 (socket says 172.21.15.40:35030) 2024-04-22T08:14:43.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.640+0000 7f6c8f7fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6c9013b0e0 0x7f6c9013d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49472/0 (socket says 172.21.15.40:49472) 2024-04-22T08:14:43.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.640+0000 7f6c8effd640 1 -- 172.21.15.40:0/1890836083 learned_addr learned my addr 172.21.15.40:0/1890836083 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:14:43.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.640+0000 7f6c8effd640 10 monclient(hunting): get_auth_request con 0x7f6c9009ee10 auth_method 0 2024-04-22T08:14:43.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.640+0000 7f6c8effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:43.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.640+0000 7f6c8effd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:43.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.640+0000 7f6c8effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:43.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.640+0000 7f6c8f7fe640 10 monclient(hunting): get_auth_request con 0x7f6c9013b0e0 auth_method 0 2024-04-22T08:14:43.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.640+0000 7f6c8f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:43.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.640+0000 7f6c8f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:43.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.640+0000 7f6c8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:43.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.640+0000 7f6c8effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:43.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.640+0000 7f6c8effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:43.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.640+0000 7f6c8effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:43.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.640+0000 7f6c8f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:43.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.640+0000 7f6c8f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:43.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.640+0000 7f6c8f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:43.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.640+0000 7f6c8effd640 10 monclient(hunting): handle_auth_done global_id 34098 payload 274 2024-04-22T08:14:43.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.640+0000 7f6c8effd640 1 -- 172.21.15.40:0/1890836083 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6c9013b0e0 msgr2=0x7f6c9013d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:14:43.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.640+0000 7f6c8effd640 1 --2- 172.21.15.40:0/1890836083 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6c9013b0e0 0x7f6c9013d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:43.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.640+0000 7f6c8effd640 1 -- 172.21.15.40:0/1890836083 >> v1:172.21.15.189:6789/0 conn(0x7f6c900a1820 legacy=0x7f6c900a1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:14:43.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.640+0000 7f6c8effd640 10 monclient: _finish_hunting 0 2024-04-22T08:14:43.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.640+0000 7f6c8effd640 1 monclient: found mon.noname-a 2024-04-22T08:14:43.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.640+0000 7f6c8effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:14:43.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.640+0000 7f6c8effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:14:43.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.640+0000 7f6c8effd640 1 -- 172.21.15.40:0/1890836083 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c900a0010 con 0x7f6c9009ee10 2024-04-22T08:14:43.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.640+0000 7f6c8f7fe640 1 --2- 172.21.15.40:0/1890836083 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6c9013b0e0 0x7f6c9013d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:14:43.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.643+0000 7f6c8effd640 1 -- 172.21.15.40:0/1890836083 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c9009ee10 msgr2=0x7f6c900a3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:43.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.643+0000 7f6c8effd640 1 --2- 172.21.15.40:0/1890836083 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c9009ee10 0x7f6c900a3240 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6c84009870 tx=0x7f6c8402b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d4446a5e5e4c440 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:43.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.644+0000 7f6c8dffb640 1 -- 172.21.15.40:0/1890836083 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c84031950 con 0x7f6c9009ee10 2024-04-22T08:14:43.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.644+0000 7f6c8dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:43.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.644+0000 7f6c8dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:14:43.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.644+0000 7f6c8dffb640 10 monclient: dump: 2024-04-22T08:14:43.652 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:43.652 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:43.652 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:43.652 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:43.652 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:43.652 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:43.652 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:43.652 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:43.652 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:43.652 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:43.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.644+0000 7f6c8dffb640 10 monclient: _finish_auth 0 2024-04-22T08:14:43.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.644+0000 7f6c8dffb640 10 monclient: _check_auth_tickets 2024-04-22T08:14:43.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.644+0000 7f6c8dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:14:43.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.644+0000 7f6c8dffb640 1 -- 172.21.15.40:0/1890836083 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c84031ab0 con 0x7f6c9009ee10 2024-04-22T08:14:43.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.644+0000 7f6c959d7640 5 monclient: authenticate success, global_id 34098 2024-04-22T08:14:43.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.644+0000 7f6c959d7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:14:43.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.644+0000 7f6c8dffb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:14:43.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.644+0000 7f6c8dffb640 1 -- 172.21.15.40:0/1890836083 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c84031e40 con 0x7f6c9009ee10 2024-04-22T08:14:43.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.644+0000 7f6c959d7640 10 monclient: get_monmap_and_config success 2024-04-22T08:14:43.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.644+0000 7f6c959d7640 4 set_mon_vals no callback set 2024-04-22T08:14:43.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.644+0000 7f6c8dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:43.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.644+0000 7f6c8dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:14:43.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.644+0000 7f6c8dffb640 10 monclient: dump: 2024-04-22T08:14:43.653 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:43.653 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:43.653 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:43.653 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:43.653 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:43.654 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:43.654 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:43.654 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:43.654 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:43.654 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:43.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.644+0000 7f6c959d7640 10 monclient: shutdown 2024-04-22T08:14:43.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.644+0000 7f6c959d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:14:43.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.644+0000 7f6c959d7640 1 -- 172.21.15.40:0/1890836083 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c9009ee10 msgr2=0x7f6c900a3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:43.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.644+0000 7f6c959d7640 1 --2- 172.21.15.40:0/1890836083 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c9009ee10 0x7f6c900a3240 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6c84009870 tx=0x7f6c8402b7e0 comp rx=0 tx=0).stop 2024-04-22T08:14:43.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.644+0000 7f6c959d7640 1 -- 172.21.15.40:0/1890836083 shutdown_connections 2024-04-22T08:14:43.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.644+0000 7f6c959d7640 1 --2- 172.21.15.40:0/1890836083 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6c9013b0e0 0x7f6c9013d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:43.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.644+0000 7f6c959d7640 1 --2- 172.21.15.40:0/1890836083 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c9009ee10 0x7f6c900a3240 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:43.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.644+0000 7f6c959d7640 1 -- 172.21.15.40:0/1890836083 >> 172.21.15.40:0/1890836083 conn(0x7f6c900a4e80 msgr2=0x7f6c900a5290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:14:43.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.644+0000 7f6c959d7640 1 -- 172.21.15.40:0/1890836083 shutdown_connections 2024-04-22T08:14:43.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.644+0000 7f6c959d7640 1 -- 172.21.15.40:0/1890836083 wait complete. 2024-04-22T08:14:43.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.645+0000 7f6c959d7640 10 monclient: build_initial_monmap 2024-04-22T08:14:43.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.645+0000 7f6c959d7640 10 monclient: monmap: 2024-04-22T08:14:43.655 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:14:43.655 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:14:43.655 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:14:43.655 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:14:43.655 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:14:43.655 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:43.655 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:14:43.655 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:14:43.655 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:14:43.655 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:43.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.645+0000 7f6c959d7640 1 Processor -- start 2024-04-22T08:14:43.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.645+0000 7f6c959d7640 1 -- start start 2024-04-22T08:14:43.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.645+0000 7f6c959d7640 10 monclient: init 2024-04-22T08:14:43.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c959d7640 10 monclient: _reopen_session rank -1 2024-04-22T08:14:43.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c959d7640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:14:43.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c959d7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c9009ee10 0x7f6c9006cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:43.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c959d7640 10 monclient(hunting): picked mon.noname-a con 0x7f6c9009ee10 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:14:43.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c959d7640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c900a1820 0x7f6c9006d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:43.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c959d7640 10 monclient(hunting): picked mon.noname-b con 0x7f6c900a1820 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:14:43.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c959d7640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6c9013b0e0 0x7f6c9006da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:43.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c959d7640 10 monclient(hunting): picked mon.noname-c con 0x7f6c9013b0e0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:14:43.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c959d7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:43.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c959d7640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c90070a70 con 0x7f6c9009ee10 2024-04-22T08:14:43.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c959d7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:43.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c959d7640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c90070cc0 con 0x7f6c900a1820 2024-04-22T08:14:43.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c959d7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:43.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c959d7640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c900a4260 con 0x7f6c9013b0e0 2024-04-22T08:14:43.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c959d7640 10 monclient(hunting): _renew_subs 2024-04-22T08:14:43.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c959d7640 10 monclient(hunting): authenticate will time out at 957.640095s 2024-04-22T08:14:43.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c8effd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c9009ee10 0x7f6c9006cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:43.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c8f7fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6c9013b0e0 0x7f6c9006da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:43.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c8f7fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6c9013b0e0 0x7f6c9006da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49474/0 (socket says 172.21.15.40:49474) 2024-04-22T08:14:43.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c8f7fe640 1 -- 172.21.15.40:0/814174958 learned_addr learned my addr 172.21.15.40:0/814174958 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:14:43.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c8f7fe640 10 monclient(hunting): get_auth_request con 0x7f6c9013b0e0 auth_method 0 2024-04-22T08:14:43.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c8effd640 1 --2- 172.21.15.40:0/814174958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c9009ee10 0x7f6c9006cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35036/0 (socket says 172.21.15.40:35036) 2024-04-22T08:14:43.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c8f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:43.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c8f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:43.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:43.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c8effd640 10 monclient(hunting): get_auth_request con 0x7f6c9009ee10 auth_method 0 2024-04-22T08:14:43.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c8effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:43.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c8effd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:43.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c8effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:43.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c8e7fc640 1 --2- 172.21.15.40:0/814174958 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c900a1820 0x7f6c9006d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:43.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c8f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:43.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c8f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:43.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c8f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:43.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c8effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:43.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c8effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:43.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c8effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:43.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c8e7fc640 10 monclient(hunting): get_auth_request con 0x7f6c900a1820 auth_method 0 2024-04-22T08:14:43.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c8e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:43.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c8e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:43.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:43.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c8e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:43.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c8e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:43.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c8e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:43.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c8f7fe640 10 monclient(hunting): handle_auth_done global_id 4718 payload 824 2024-04-22T08:14:43.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c8f7fe640 1 -- 172.21.15.40:0/814174958 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c900a1820 msgr2=0x7f6c9006d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:43.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c8f7fe640 1 --2- 172.21.15.40:0/814174958 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c900a1820 0x7f6c9006d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:43.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c8f7fe640 1 -- 172.21.15.40:0/814174958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c9009ee10 msgr2=0x7f6c9006cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:43.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c8f7fe640 1 --2- 172.21.15.40:0/814174958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c9009ee10 0x7f6c9006cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:43.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c8f7fe640 10 monclient: _finish_hunting 0 2024-04-22T08:14:43.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c8f7fe640 1 monclient: found mon.noname-c 2024-04-22T08:14:43.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c8f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:14:43.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c8f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:14:43.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c8f7fe640 1 -- 172.21.15.40:0/814174958 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c88009a70 con 0x7f6c9013b0e0 2024-04-22T08:14:43.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.646+0000 7f6c8effd640 1 --2- 172.21.15.40:0/814174958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c9009ee10 0x7f6c9006cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:14:43.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.649+0000 7f6c8f7fe640 1 -- 172.21.15.40:0/814174958 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6c9013b0e0 msgr2=0x7f6c9006da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:43.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.649+0000 7f6c8f7fe640 1 --2- 172.21.15.40:0/814174958 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6c9013b0e0 0x7f6c9006da50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6c88009ee0 tx=0x7f6c8800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:43.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.649+0000 7f6c77fff640 1 -- 172.21.15.40:0/814174958 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c88009d50 con 0x7f6c9013b0e0 2024-04-22T08:14:43.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.649+0000 7f6c77fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:43.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.649+0000 7f6c77fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:14:43.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.649+0000 7f6c77fff640 10 monclient: dump: 2024-04-22T08:14:43.660 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:43.660 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:43.660 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:43.660 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:43.660 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:43.660 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:43.660 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:43.660 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:43.660 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:43.660 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:43.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.649+0000 7f6c77fff640 10 monclient: _finish_auth 0 2024-04-22T08:14:43.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.649+0000 7f6c77fff640 10 monclient: _check_auth_tickets 2024-04-22T08:14:43.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.649+0000 7f6c77fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:14:43.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.649+0000 7f6c77fff640 1 -- 172.21.15.40:0/814174958 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c8800f040 con 0x7f6c9013b0e0 2024-04-22T08:14:43.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.649+0000 7f6c77fff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:14:43.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.649+0000 7f6c77fff640 1 -- 172.21.15.40:0/814174958 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c880148a0 con 0x7f6c9013b0e0 2024-04-22T08:14:43.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.649+0000 7f6c959d7640 5 monclient: authenticate success, global_id 4718 2024-04-22T08:14:43.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.649+0000 7f6c959d7640 10 monclient: _renew_subs 2024-04-22T08:14:43.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.649+0000 7f6c959d7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:14:43.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.649+0000 7f6c959d7640 1 -- 172.21.15.40:0/814174958 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c840094d0 con 0x7f6c9013b0e0 2024-04-22T08:14:43.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.649+0000 7f6c77fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:43.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.649+0000 7f6c8d7fa640 4 set_mon_vals no callback set 2024-04-22T08:14:43.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.649+0000 7f6c77fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:14:43.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.649+0000 7f6c77fff640 10 monclient: dump: 2024-04-22T08:14:43.662 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:43.662 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:43.662 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:43.662 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:43.662 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:43.662 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:43.662 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:43.662 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:43.662 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:43.662 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:43.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.649+0000 7f6c959d7640 10 monclient: _renew_subs 2024-04-22T08:14:43.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.649+0000 7f6c959d7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:14:43.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.649+0000 7f6c959d7640 1 -- 172.21.15.40:0/814174958 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c90070420 con 0x7f6c9013b0e0 2024-04-22T08:14:43.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.650+0000 7f6c77fff640 1 -- 172.21.15.40:0/814174958 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f6c88014c90 con 0x7f6c9013b0e0 2024-04-22T08:14:43.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.650+0000 7f6c959d7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:14:43.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.650+0000 7f6c959d7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:14:43.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.650+0000 7f6c959d7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:14:43.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.650+0000 7f6c959d7640 1 -- 172.21.15.40:0/814174958 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c9004e9b0 con 0x7f6c9013b0e0 2024-04-22T08:14:43.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.650+0000 7f6c77fff640 1 --2- 172.21.15.40:0/814174958 >> v2:172.21.15.189:6800/30168 conn(0x7f6c7c0377d0 0x7f6c7c039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:43.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.650+0000 7f6c77fff640 1 -- 172.21.15.40:0/814174958 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f6c8804b5b0 con 0x7f6c9013b0e0 2024-04-22T08:14:43.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.650+0000 7f6c8effd640 1 --2- 172.21.15.40:0/814174958 >> v2:172.21.15.189:6800/30168 conn(0x7f6c7c0377d0 0x7f6c7c039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:43.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.650+0000 7f6c8effd640 10 monclient: get_auth_request con 0x7f6c7c0377d0 auth_method 0 2024-04-22T08:14:43.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.653+0000 7f6c8effd640 1 --2- 172.21.15.40:0/814174958 >> v2:172.21.15.189:6800/30168 conn(0x7f6c7c0377d0 0x7f6c7c039c90 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6c84033000 tx=0x7f6c84008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:43.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.654+0000 7f6c77fff640 1 -- 172.21.15.40:0/814174958 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f6c88024300 con 0x7f6c9013b0e0 2024-04-22T08:14:43.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.654+0000 7f6c77fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:14:43.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.654+0000 7f6c77fff640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:14:43.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.907+0000 7f6c959d7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:14:43.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.907+0000 7f6c959d7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:14:43.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.907+0000 7f6c959d7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:14:43.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.907+0000 7f6c959d7640 1 -- 172.21.15.40:0/814174958 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6c90070e00 con 0x7f6c9013b0e0 2024-04-22T08:14:43.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.908+0000 7f6c77fff640 1 -- 172.21.15.40:0/814174958 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f6c90070e00 con 0x7f6c9013b0e0 2024-04-22T08:14:43.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.908+0000 7f6c77fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:14:43.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.908+0000 7f6c77fff640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:14:43.908 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:14:43.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.910+0000 7f6c959d7640 1 -- 172.21.15.40:0/814174958 >> v2:172.21.15.189:6800/30168 conn(0x7f6c7c0377d0 msgr2=0x7f6c7c039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:43.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.910+0000 7f6c959d7640 1 --2- 172.21.15.40:0/814174958 >> v2:172.21.15.189:6800/30168 conn(0x7f6c7c0377d0 0x7f6c7c039c90 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6c84033000 tx=0x7f6c84008030 comp rx=0 tx=0).stop 2024-04-22T08:14:43.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.910+0000 7f6c959d7640 10 monclient: shutdown 2024-04-22T08:14:43.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.910+0000 7f6c959d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:14:43.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.910+0000 7f6c959d7640 1 -- 172.21.15.40:0/814174958 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6c9013b0e0 msgr2=0x7f6c9006da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:43.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.910+0000 7f6c959d7640 1 --2- 172.21.15.40:0/814174958 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6c9013b0e0 0x7f6c9006da50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6c88009ee0 tx=0x7f6c8800c490 comp rx=0 tx=0).stop 2024-04-22T08:14:43.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.910+0000 7f6c959d7640 1 -- 172.21.15.40:0/814174958 shutdown_connections 2024-04-22T08:14:43.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.910+0000 7f6c959d7640 1 --2- 172.21.15.40:0/814174958 >> v2:172.21.15.189:6800/30168 conn(0x7f6c7c0377d0 0x7f6c7c039c90 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:43.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.910+0000 7f6c959d7640 1 --2- 172.21.15.40:0/814174958 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c900a1820 0x7f6c9006d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:43.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.910+0000 7f6c959d7640 1 --2- 172.21.15.40:0/814174958 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6c9013b0e0 0x7f6c9006da50 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:43.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.910+0000 7f6c959d7640 1 --2- 172.21.15.40:0/814174958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c9009ee10 0x7f6c9006cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:43.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.910+0000 7f6c959d7640 1 -- 172.21.15.40:0/814174958 >> 172.21.15.40:0/814174958 conn(0x7f6c900a4e80 msgr2=0x7f6c900a6280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:14:43.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.910+0000 7f6c959d7640 1 -- 172.21.15.40:0/814174958 shutdown_connections 2024-04-22T08:14:43.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:43.910+0000 7f6c959d7640 1 -- 172.21.15.40:0/814174958 wait complete. 2024-04-22T08:14:43.920 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-22T08:14:43.920 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-22T08:14:43.920 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-22T08:14:44.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.020+0000 7f2a4b87d640 10 monclient: get_monmap_and_config 2024-04-22T08:14:44.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.020+0000 7f2a4b87d640 10 monclient: build_initial_monmap 2024-04-22T08:14:44.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.020+0000 7f2a4b87d640 10 monclient: monmap: 2024-04-22T08:14:44.020 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:14:44.020 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:44.020 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:14:44.021571+0000 2024-04-22T08:14:44.021 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:14:44.021571+0000 2024-04-22T08:14:44.021 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:14:44.021 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:44.021 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:14:44.021 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:14:44.021 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:14:44.021 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:14:44.021 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:14:44.021 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:44.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.021+0000 7f2a4b87d640 1 Processor -- start 2024-04-22T08:14:44.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.021+0000 7f2a4b87d640 1 -- start start 2024-04-22T08:14:44.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.021+0000 7f2a4b87d640 10 monclient: init 2024-04-22T08:14:44.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.021+0000 7f2a4b87d640 10 monclient: _reopen_session rank -1 2024-04-22T08:14:44.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.022+0000 7f2a4b87d640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-04-22T08:14:44.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.022+0000 7f2a4b87d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a440a1a80 0x7f2a440a01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:44.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.022+0000 7f2a4b87d640 10 monclient(hunting): picked mon.noname-b con 0x7f2a440a1a80 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:14:44.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.022+0000 7f2a4b87d640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2a440a1e60 0x7f2a440a2260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:44.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.022+0000 7f2a4b87d640 10 monclient(hunting): picked mon.noname-c con 0x7f2a440a1e60 addr v2:172.21.15.189:3300/0 2024-04-22T08:14:44.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.022+0000 7f2a4b87d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2a44135c20 addr v1:172.21.15.189:6789/0 2024-04-22T08:14:44.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.022+0000 7f2a4b87d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a4409d860 con 0x7f2a44135c20 2024-04-22T08:14:44.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.022+0000 7f2a4b87d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:44.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.022+0000 7f2a4b87d640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a4409f880 con 0x7f2a440a1e60 2024-04-22T08:14:44.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.022+0000 7f2a4b87d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:44.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.022+0000 7f2a4b87d640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a4409ef50 con 0x7f2a440a1a80 2024-04-22T08:14:44.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.022+0000 7f2a4b87d640 10 monclient(hunting): _renew_subs 2024-04-22T08:14:44.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.022+0000 7f2a4b87d640 10 monclient(hunting): authenticate will time out at 958.015896s 2024-04-22T08:14:44.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.022+0000 7f2a49619640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a440a1a80 0x7f2a440a01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:44.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.022+0000 7f2a49619640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a440a1a80 0x7f2a440a01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49486/0 (socket says 172.21.15.40:49486) 2024-04-22T08:14:44.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.022+0000 7f2a49619640 1 -- 172.21.15.40:0/3812824538 learned_addr learned my addr 172.21.15.40:0/3812824538 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:14:44.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.022+0000 7f2a49619640 10 monclient(hunting): get_auth_request con 0x7f2a440a1a80 auth_method 0 2024-04-22T08:14:44.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.022+0000 7f2a49619640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:44.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.022+0000 7f2a49619640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:44.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.022+0000 7f2a49619640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:44.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.022+0000 7f2a49619640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:44.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.022+0000 7f2a49619640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:44.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.022+0000 7f2a49619640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:44.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.022+0000 7f2a48e18640 1 --2- 172.21.15.40:0/3812824538 >> v2:172.21.15.189:3300/0 conn(0x7f2a440a1e60 0x7f2a440a2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:44.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.022+0000 7f2a48e18640 10 monclient(hunting): get_auth_request con 0x7f2a440a1e60 auth_method 0 2024-04-22T08:14:44.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.022+0000 7f2a48e18640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:44.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.022+0000 7f2a48e18640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:44.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.022+0000 7f2a48e18640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:44.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.022+0000 7f2a49619640 10 monclient(hunting): handle_auth_done global_id 4721 payload 274 2024-04-22T08:14:44.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.022+0000 7f2a49619640 1 -- 172.21.15.40:0/3812824538 >> v2:172.21.15.189:3300/0 conn(0x7f2a440a1e60 msgr2=0x7f2a440a2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:14:44.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.022+0000 7f2a49619640 1 --2- 172.21.15.40:0/3812824538 >> v2:172.21.15.189:3300/0 conn(0x7f2a440a1e60 0x7f2a440a2260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:44.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.022+0000 7f2a49619640 1 -- 172.21.15.40:0/3812824538 >> v1:172.21.15.189:6789/0 conn(0x7f2a44135c20 legacy=0x7f2a441360b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:14:44.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.022+0000 7f2a49619640 10 monclient: _finish_hunting 0 2024-04-22T08:14:44.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.022+0000 7f2a49619640 1 monclient: found mon.noname-b 2024-04-22T08:14:44.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.022+0000 7f2a49619640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:14:44.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.022+0000 7f2a49619640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:14:44.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.022+0000 7f2a49619640 1 -- 172.21.15.40:0/3812824538 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a44072160 con 0x7f2a440a1a80 2024-04-22T08:14:44.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.025+0000 7f2a49619640 1 -- 172.21.15.40:0/3812824538 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a440a1a80 msgr2=0x7f2a440a01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:44.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.025+0000 7f2a49619640 1 --2- 172.21.15.40:0/3812824538 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a440a1a80 0x7f2a440a01d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2a40009870 tx=0x7f2a4002b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=412f5e659500834d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:44.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.025+0000 7f2a3bfff640 1 -- 172.21.15.40:0/3812824538 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a40031950 con 0x7f2a440a1a80 2024-04-22T08:14:44.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.025+0000 7f2a3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:44.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.025+0000 7f2a3bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:14:44.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.025+0000 7f2a3bfff640 10 monclient: dump: 2024-04-22T08:14:44.033 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:44.033 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:44.033 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:44.033 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:44.034 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:44.034 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:44.034 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:44.034 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:44.034 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:44.034 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:44.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.025+0000 7f2a3bfff640 10 monclient: _finish_auth 0 2024-04-22T08:14:44.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.025+0000 7f2a3bfff640 10 monclient: _check_auth_tickets 2024-04-22T08:14:44.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.025+0000 7f2a3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:14:44.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.025+0000 7f2a3bfff640 1 -- 172.21.15.40:0/3812824538 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a40031ab0 con 0x7f2a440a1a80 2024-04-22T08:14:44.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.025+0000 7f2a3bfff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:14:44.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.025+0000 7f2a3bfff640 1 -- 172.21.15.40:0/3812824538 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a40031e40 con 0x7f2a440a1a80 2024-04-22T08:14:44.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.025+0000 7f2a4b87d640 5 monclient: authenticate success, global_id 4721 2024-04-22T08:14:44.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.025+0000 7f2a4b87d640 10 monclient: get_monmap_and_config success 2024-04-22T08:14:44.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.025+0000 7f2a4b87d640 4 set_mon_vals no callback set 2024-04-22T08:14:44.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.025+0000 7f2a3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:44.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.025+0000 7f2a3bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:14:44.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.025+0000 7f2a3bfff640 10 monclient: dump: 2024-04-22T08:14:44.035 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:44.035 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:44.035 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:44.035 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:44.035 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:44.035 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:44.035 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:44.035 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:44.035 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:44.036 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:44.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.025+0000 7f2a4b87d640 10 monclient: shutdown 2024-04-22T08:14:44.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.025+0000 7f2a4b87d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:14:44.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.025+0000 7f2a4b87d640 1 -- 172.21.15.40:0/3812824538 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a440a1a80 msgr2=0x7f2a440a01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:44.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.025+0000 7f2a4b87d640 1 --2- 172.21.15.40:0/3812824538 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a440a1a80 0x7f2a440a01d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2a40009870 tx=0x7f2a4002b7e0 comp rx=0 tx=0).stop 2024-04-22T08:14:44.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.026+0000 7f2a4b87d640 1 -- 172.21.15.40:0/3812824538 shutdown_connections 2024-04-22T08:14:44.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.026+0000 7f2a4b87d640 1 --2- 172.21.15.40:0/3812824538 >> v2:172.21.15.189:3300/0 conn(0x7f2a440a1e60 0x7f2a440a2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:44.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.026+0000 7f2a4b87d640 1 --2- 172.21.15.40:0/3812824538 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a440a1a80 0x7f2a440a01d0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:44.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.026+0000 7f2a4b87d640 1 -- 172.21.15.40:0/3812824538 >> 172.21.15.40:0/3812824538 conn(0x7f2a440a66a0 msgr2=0x7f2a440a2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:14:44.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.026+0000 7f2a4b87d640 1 -- 172.21.15.40:0/3812824538 shutdown_connections 2024-04-22T08:14:44.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.026+0000 7f2a4b87d640 1 -- 172.21.15.40:0/3812824538 wait complete. 2024-04-22T08:14:44.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.026+0000 7f2a4b87d640 10 monclient: build_initial_monmap 2024-04-22T08:14:44.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.026+0000 7f2a4b87d640 10 monclient: monmap: 2024-04-22T08:14:44.036 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:14:44.036 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:14:44.037 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:14:44.037 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:14:44.037 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:14:44.037 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:44.037 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:14:44.037 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:14:44.037 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:14:44.037 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:44.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.027+0000 7f2a4b87d640 1 Processor -- start 2024-04-22T08:14:44.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.027+0000 7f2a4b87d640 1 -- start start 2024-04-22T08:14:44.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.027+0000 7f2a4b87d640 10 monclient: init 2024-04-22T08:14:44.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.027+0000 7f2a4b87d640 10 monclient: _reopen_session rank -1 2024-04-22T08:14:44.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.027+0000 7f2a4b87d640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-22T08:14:44.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.027+0000 7f2a4b87d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a440a1a80 0x7f2a441cac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:44.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.027+0000 7f2a4b87d640 10 monclient(hunting): picked mon.noname-c con 0x7f2a440a1a80 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:14:44.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.027+0000 7f2a4b87d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a440a1e60 0x7f2a441cb170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:44.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.027+0000 7f2a4b87d640 10 monclient(hunting): picked mon.noname-b con 0x7f2a440a1e60 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:14:44.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.027+0000 7f2a4b87d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a44135c20 0x7f2a44122400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:44.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.027+0000 7f2a4b87d640 10 monclient(hunting): picked mon.noname-a con 0x7f2a44135c20 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:14:44.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.027+0000 7f2a4b87d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:44.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.027+0000 7f2a4b87d640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a44124ae0 con 0x7f2a44135c20 2024-04-22T08:14:44.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.027+0000 7f2a4b87d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:44.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.027+0000 7f2a4b87d640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a44124490 con 0x7f2a440a1e60 2024-04-22T08:14:44.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.027+0000 7f2a4b87d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:44.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.027+0000 7f2a4b87d640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a4409cc20 con 0x7f2a440a1a80 2024-04-22T08:14:44.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.027+0000 7f2a4b87d640 10 monclient(hunting): _renew_subs 2024-04-22T08:14:44.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.027+0000 7f2a4b87d640 10 monclient(hunting): authenticate will time out at 958.021434s 2024-04-22T08:14:44.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.027+0000 7f2a49619640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a440a1a80 0x7f2a441cac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:44.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.027+0000 7f2a49619640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a440a1a80 0x7f2a441cac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49490/0 (socket says 172.21.15.40:49490) 2024-04-22T08:14:44.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.027+0000 7f2a49619640 1 -- 172.21.15.40:0/3985905456 learned_addr learned my addr 172.21.15.40:0/3985905456 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:14:44.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.027+0000 7f2a49619640 10 monclient(hunting): get_auth_request con 0x7f2a440a1a80 auth_method 0 2024-04-22T08:14:44.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.027+0000 7f2a49619640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:44.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.027+0000 7f2a49619640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:44.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.027+0000 7f2a49619640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:44.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.027+0000 7f2a49619640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:44.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.027+0000 7f2a49619640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:44.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.027+0000 7f2a49619640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:44.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.027+0000 7f2a49e1a640 1 --2- 172.21.15.40:0/3985905456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a44135c20 0x7f2a44122400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:44.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.027+0000 7f2a49e1a640 10 monclient(hunting): get_auth_request con 0x7f2a44135c20 auth_method 0 2024-04-22T08:14:44.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.027+0000 7f2a49e1a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:44.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.027+0000 7f2a49e1a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:44.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.027+0000 7f2a49e1a640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:44.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.027+0000 7f2a48e18640 1 --2- 172.21.15.40:0/3985905456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a440a1e60 0x7f2a441cb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:44.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.027+0000 7f2a49e1a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:44.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.027+0000 7f2a49e1a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:44.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.027+0000 7f2a49e1a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:44.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.027+0000 7f2a48e18640 10 monclient(hunting): get_auth_request con 0x7f2a440a1e60 auth_method 0 2024-04-22T08:14:44.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.028+0000 7f2a48e18640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:44.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.028+0000 7f2a48e18640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:44.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.028+0000 7f2a48e18640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:44.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.028+0000 7f2a49619640 10 monclient(hunting): handle_auth_done global_id 4724 payload 824 2024-04-22T08:14:44.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.028+0000 7f2a49619640 1 -- 172.21.15.40:0/3985905456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a440a1e60 msgr2=0x7f2a441cb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:44.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.028+0000 7f2a49619640 1 --2- 172.21.15.40:0/3985905456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a440a1e60 0x7f2a441cb170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:44.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.028+0000 7f2a49619640 1 -- 172.21.15.40:0/3985905456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a44135c20 msgr2=0x7f2a44122400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:44.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.028+0000 7f2a49619640 1 --2- 172.21.15.40:0/3985905456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a44135c20 0x7f2a44122400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:44.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.028+0000 7f2a49619640 10 monclient: _finish_hunting 0 2024-04-22T08:14:44.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.028+0000 7f2a49619640 1 monclient: found mon.noname-c 2024-04-22T08:14:44.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.028+0000 7f2a49619640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:14:44.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.028+0000 7f2a49619640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:14:44.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.028+0000 7f2a49619640 1 -- 172.21.15.40:0/3985905456 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a400094d0 con 0x7f2a440a1a80 2024-04-22T08:14:44.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.030+0000 7f2a49619640 1 -- 172.21.15.40:0/3985905456 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a440a1a80 msgr2=0x7f2a441cac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:44.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.030+0000 7f2a49619640 1 --2- 172.21.15.40:0/3985905456 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a440a1a80 0x7f2a441cac30 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2a400090b0 tx=0x7f2a400046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:44.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.030+0000 7f2a39ffb640 1 -- 172.21.15.40:0/3985905456 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a40033d30 con 0x7f2a440a1a80 2024-04-22T08:14:44.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.030+0000 7f2a39ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:44.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.030+0000 7f2a39ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:14:44.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.030+0000 7f2a39ffb640 10 monclient: dump: 2024-04-22T08:14:44.042 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:44.042 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:44.042 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:44.042 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:44.042 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:44.042 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:44.042 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:44.042 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:44.042 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:44.043 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:44.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.030+0000 7f2a39ffb640 10 monclient: _finish_auth 0 2024-04-22T08:14:44.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.030+0000 7f2a39ffb640 10 monclient: _check_auth_tickets 2024-04-22T08:14:44.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.030+0000 7f2a39ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:14:44.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.030+0000 7f2a39ffb640 1 -- 172.21.15.40:0/3985905456 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a4003c400 con 0x7f2a440a1a80 2024-04-22T08:14:44.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.030+0000 7f2a4b87d640 5 monclient: authenticate success, global_id 4724 2024-04-22T08:14:44.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.030+0000 7f2a4b87d640 10 monclient: _renew_subs 2024-04-22T08:14:44.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.030+0000 7f2a4b87d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:14:44.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.030+0000 7f2a4b87d640 1 -- 172.21.15.40:0/3985905456 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a4409c310 con 0x7f2a440a1a80 2024-04-22T08:14:44.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.030+0000 7f2a39ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:14:44.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.030+0000 7f2a39ffb640 1 -- 172.21.15.40:0/3985905456 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a4003c7b0 con 0x7f2a440a1a80 2024-04-22T08:14:44.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.030+0000 7f2a39ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:44.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.030+0000 7f2a39ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:14:44.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.030+0000 7f2a39ffb640 10 monclient: dump: 2024-04-22T08:14:44.043 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:44.044 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:44.044 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:44.044 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:44.044 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:44.044 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:44.044 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:44.044 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:44.044 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:44.044 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:44.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.030+0000 7f2a3b7fe640 4 set_mon_vals no callback set 2024-04-22T08:14:44.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.030+0000 7f2a4b87d640 10 monclient: _renew_subs 2024-04-22T08:14:44.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.030+0000 7f2a4b87d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:14:44.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.030+0000 7f2a4b87d640 1 -- 172.21.15.40:0/3985905456 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a441262e0 con 0x7f2a440a1a80 2024-04-22T08:14:44.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.031+0000 7f2a4b87d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:14:44.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.031+0000 7f2a4b87d640 1 --2- 172.21.15.40:0/3985905456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a44123100 0x7f2a4402b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:44.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.031+0000 7f2a4b87d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:44.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.031+0000 7f2a4b87d640 1 -- 172.21.15.40:0/3985905456 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a44066770 con 0x7f2a44123100 2024-04-22T08:14:44.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.031+0000 7f2a39ffb640 1 -- 172.21.15.40:0/3985905456 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f2a4003a040 con 0x7f2a440a1a80 2024-04-22T08:14:44.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.031+0000 7f2a48e18640 1 --2- 172.21.15.40:0/3985905456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a44123100 0x7f2a4402b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:44.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.031+0000 7f2a48e18640 10 monclient: get_auth_request con 0x7f2a44123100 auth_method 0 2024-04-22T08:14:44.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.031+0000 7f2a48e18640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:44.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.031+0000 7f2a48e18640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:44.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.031+0000 7f2a48e18640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:44.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.031+0000 7f2a48e18640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:44.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.031+0000 7f2a48e18640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:44.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.031+0000 7f2a48e18640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:44.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.031+0000 7f2a48e18640 10 monclient(hunting): handle_auth_done global_id 34107 payload 824 2024-04-22T08:14:44.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.031+0000 7f2a39ffb640 1 --2- 172.21.15.40:0/3985905456 >> v2:172.21.15.189:6800/30168 conn(0x7f2a280377d0 0x7f2a28039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:44.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.031+0000 7f2a48e18640 1 -- 172.21.15.40:0/3985905456 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2a440a27a0 con 0x7f2a44123100 2024-04-22T08:14:44.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.031+0000 7f2a39ffb640 1 -- 172.21.15.40:0/3985905456 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f2a4006be70 con 0x7f2a440a1a80 2024-04-22T08:14:44.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.032+0000 7f2a49e1a640 1 --2- 172.21.15.40:0/3985905456 >> v2:172.21.15.189:6800/30168 conn(0x7f2a280377d0 0x7f2a28039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:44.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.032+0000 7f2a49e1a640 10 monclient: get_auth_request con 0x7f2a280377d0 auth_method 0 2024-04-22T08:14:44.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.034+0000 7f2a48e18640 1 -- 172.21.15.40:0/3985905456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a44123100 msgr2=0x7f2a4402b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:44.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.034+0000 7f2a48e18640 1 --2- 172.21.15.40:0/3985905456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a44123100 0x7f2a4402b200 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2a34006fd0 tx=0x7f2a3400f930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:44.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.034+0000 7f2a39ffb640 1 -- 172.21.15.40:0/3985905456 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a34004020 con 0x7f2a44123100 2024-04-22T08:14:44.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.034+0000 7f2a39ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:44.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.034+0000 7f2a39ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:14:44.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.034+0000 7f2a39ffb640 10 monclient: dump: 2024-04-22T08:14:44.049 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:44.049 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:44.049 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:44.049 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:44.049 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:44.049 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:44.049 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:44.049 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:44.050 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:44.050 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:44.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.034+0000 7f2a39ffb640 1 -- 172.21.15.40:0/3985905456 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f2a34004180 con 0x7f2a44123100 2024-04-22T08:14:44.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.034+0000 7f2a39ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:14:44.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.034+0000 7f2a39ffb640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:14:44.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.034+0000 7f2a39ffb640 1 -- 172.21.15.40:0/3985905456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a44123100 msgr2=0x7f2a4402b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:44.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.034+0000 7f2a39ffb640 1 --2- 172.21.15.40:0/3985905456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a44123100 0x7f2a4402b200 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2a34006fd0 tx=0x7f2a3400f930 comp rx=0 tx=0).stop 2024-04-22T08:14:44.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.034+0000 7f2a39ffb640 1 -- 172.21.15.40:0/3985905456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a44123100 msgr2=0x7f2a4402b200 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:14:44.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.034+0000 7f2a39ffb640 1 --2- 172.21.15.40:0/3985905456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a44123100 0x7f2a4402b200 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2a34006fd0 tx=0x7f2a3400f930 comp rx=0 tx=0).stop 2024-04-22T08:14:44.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.034+0000 7f2a49e1a640 1 --2- 172.21.15.40:0/3985905456 >> v2:172.21.15.189:6800/30168 conn(0x7f2a280377d0 0x7f2a28039c90 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c0055e0 tx=0x7f2a3c003d10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:44.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.040+0000 7f2a4b87d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:14:44.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.040+0000 7f2a4b87d640 1 --2- 172.21.15.40:0/3985905456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a441234e0 0x7f2a4402b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:44.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.040+0000 7f2a4b87d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:44.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.040+0000 7f2a4b87d640 1 -- 172.21.15.40:0/3985905456 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a4409dc10 con 0x7f2a441234e0 2024-04-22T08:14:44.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.040+0000 7f2a48e18640 1 --2- 172.21.15.40:0/3985905456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a441234e0 0x7f2a4402b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:44.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.041+0000 7f2a48e18640 10 monclient: get_auth_request con 0x7f2a441234e0 auth_method 0 2024-04-22T08:14:44.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.041+0000 7f2a48e18640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:44.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.041+0000 7f2a48e18640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:44.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.041+0000 7f2a48e18640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:44.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.041+0000 7f2a48e18640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:44.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.041+0000 7f2a48e18640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:44.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.041+0000 7f2a48e18640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:44.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.041+0000 7f2a48e18640 10 monclient(hunting): handle_auth_done global_id 34110 payload 824 2024-04-22T08:14:44.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.041+0000 7f2a48e18640 1 -- 172.21.15.40:0/3985905456 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2a441367c0 con 0x7f2a441234e0 2024-04-22T08:14:44.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.043+0000 7f2a48e18640 1 -- 172.21.15.40:0/3985905456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a441234e0 msgr2=0x7f2a4402b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:44.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.043+0000 7f2a48e18640 1 --2- 172.21.15.40:0/3985905456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a441234e0 0x7f2a4402b740 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2a34006fd0 tx=0x7f2a340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:44.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.043+0000 7f2a39ffb640 1 -- 172.21.15.40:0/3985905456 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a34024070 con 0x7f2a441234e0 2024-04-22T08:14:44.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.043+0000 7f2a39ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:44.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.043+0000 7f2a39ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:14:44.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.043+0000 7f2a39ffb640 10 monclient: dump: 2024-04-22T08:14:44.052 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:44.052 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:44.052 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:44.052 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:44.052 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:44.052 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:44.052 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:44.052 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:44.052 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:44.052 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:44.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.043+0000 7f2a39ffb640 1 -- 172.21.15.40:0/3985905456 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4463 (secure 0 0 0) 0x7f2a34003bd0 con 0x7f2a441234e0 2024-04-22T08:14:44.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.043+0000 7f2a39ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:14:44.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.043+0000 7f2a39ffb640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:14:44.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.043+0000 7f2a39ffb640 1 -- 172.21.15.40:0/3985905456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a441234e0 msgr2=0x7f2a4402b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:44.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.043+0000 7f2a39ffb640 1 --2- 172.21.15.40:0/3985905456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a441234e0 0x7f2a4402b740 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2a34006fd0 tx=0x7f2a340046c0 comp rx=0 tx=0).stop 2024-04-22T08:14:44.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.043+0000 7f2a39ffb640 1 -- 172.21.15.40:0/3985905456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a441234e0 msgr2=0x7f2a4402b740 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:14:44.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.043+0000 7f2a39ffb640 1 --2- 172.21.15.40:0/3985905456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a441234e0 0x7f2a4402b740 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2a34006fd0 tx=0x7f2a340046c0 comp rx=0 tx=0).stop 2024-04-22T08:14:44.053 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:14:44.053 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:14:44.053 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:14:44.053 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "leader", 2024-04-22T08:14:44.053 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 26, 2024-04-22T08:14:44.054 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:14:44.054 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:14:44.054 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:14:44.054 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:14:44.054 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:14:44.054 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 0, 2024-04-22T08:14:44.054 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:14:44.054 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:14:44.054 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:14:44.054 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:14:44.054 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:14:44.054 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:14:44.054 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:14:44.054 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:14:44.055 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:14:44.055 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:14:44.055 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:14:44.055 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:14:44.055 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:14:44.055 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:14:44.055 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:14:44.055 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:14:44.055 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:14:44.055 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:14:44.055 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:14:44.055 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:14:44.055 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:14:44.055 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:14:44.056 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:14:44.056 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:14:44.056 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:14:44.056 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:14:44.056 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:44.056 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:44.056 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:14:44.056 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:14:44.056 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:14:44.056 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:14:44.056 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:14:44.056 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:14:44.056 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:14:44.056 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:14:44.057 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:14:44.057 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:14:44.057 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:14:44.057 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:14:44.057 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:14:44.057 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:14:44.057 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:14:44.057 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:14:44.057 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:14:44.057 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:14:44.057 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:14:44.057 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:14:44.058 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:14:44.058 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:14:44.058 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:14:44.058 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:14:44.058 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:14:44.058 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:14:44.058 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:14:44.058 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:14:44.058 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:14:44.058 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:44.058 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:14:44.058 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:44.058 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:14:44.059 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:14:44.059 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:14:44.059 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:14:44.059 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:44.059 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:14:44.059 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:14:44.059 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:44.059 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:44.059 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:44.059 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:14:44.059 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:14:44.060 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:44.060 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:44.060 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:44.060 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:44.060 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:14:44.060 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:14:44.060 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:14:44.060 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:14:44.060 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:14:44.060 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:44.060 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:44.060 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:14:44.060 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:14:44.060 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:14:44.060 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:14:44.061 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:44.061 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:14:44.061 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:14:44.061 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:44.061 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:44.061 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:44.061 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:14:44.061 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:14:44.061 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:44.061 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:44.061 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:44.061 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:44.061 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:14:44.061 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:14:44.062 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:14:44.062 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:14:44.062 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:14:44.062 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:44.062 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:44.062 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:14:44.062 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:14:44.062 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:14:44.062 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:14:44.062 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:44.062 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:14:44.062 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:14:44.062 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:44.062 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:44.063 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:44.063 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:14:44.063 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:14:44.063 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:44.063 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:44.063 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:44.063 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:44.063 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:14:44.063 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:14:44.063 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:14:44.063 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:14:44.063 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:14:44.063 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:44.064 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:44.064 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:44.064 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:14:44.064 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:14:44.064 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:44.064 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:14:44.064 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:14:44.064 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:14:44.064 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:44.064 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:14:44.064 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:14:44.064 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:44.064 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:14:44.064 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:14:44.065 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:14:44.065 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:44.065 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:44.065 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:44.065 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:14:44.065 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:14:44.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.044+0000 7f2a4b87d640 1 -- 172.21.15.40:0/3985905456 >> v2:172.21.15.189:6800/30168 conn(0x7f2a280377d0 msgr2=0x7f2a28039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:44.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.044+0000 7f2a4b87d640 1 --2- 172.21.15.40:0/3985905456 >> v2:172.21.15.189:6800/30168 conn(0x7f2a280377d0 0x7f2a28039c90 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c0055e0 tx=0x7f2a3c003d10 comp rx=0 tx=0).stop 2024-04-22T08:14:44.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.044+0000 7f2a4b87d640 10 monclient: shutdown 2024-04-22T08:14:44.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.044+0000 7f2a4b87d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:14:44.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.044+0000 7f2a4b87d640 1 -- 172.21.15.40:0/3985905456 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a440a1a80 msgr2=0x7f2a441cac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:44.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.044+0000 7f2a4b87d640 1 --2- 172.21.15.40:0/3985905456 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a440a1a80 0x7f2a441cac30 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2a400090b0 tx=0x7f2a400046c0 comp rx=0 tx=0).stop 2024-04-22T08:14:44.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.044+0000 7f2a49e1a640 1 -- 172.21.15.40:0/3985905456 reap_dead start 2024-04-22T08:14:44.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.044+0000 7f2a4b87d640 1 -- 172.21.15.40:0/3985905456 shutdown_connections 2024-04-22T08:14:44.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.044+0000 7f2a4b87d640 1 --2- 172.21.15.40:0/3985905456 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a440a1a80 0x7f2a441cac30 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:44.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.044+0000 7f2a4b87d640 1 -- 172.21.15.40:0/3985905456 >> 172.21.15.40:0/3985905456 conn(0x7f2a440a66a0 msgr2=0x7f2a440999f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:14:44.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.044+0000 7f2a4b87d640 1 -- 172.21.15.40:0/3985905456 shutdown_connections 2024-04-22T08:14:44.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.044+0000 7f2a4b87d640 1 -- 172.21.15.40:0/3985905456 wait complete. 2024-04-22T08:14:44.067 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:14:44.067 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-22T08:14:44.068 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-22T08:14:44.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.168+0000 7fe3a8c0b640 10 monclient: get_monmap_and_config 2024-04-22T08:14:44.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.168+0000 7fe3a8c0b640 10 monclient: build_initial_monmap 2024-04-22T08:14:44.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.168+0000 7fe3a8c0b640 10 monclient: monmap: 2024-04-22T08:14:44.168 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:14:44.169 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:44.169 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:14:44.168997+0000 2024-04-22T08:14:44.169 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:14:44.168997+0000 2024-04-22T08:14:44.169 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:14:44.169 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:44.170 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:14:44.170 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:14:44.170 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:14:44.170 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:14:44.170 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:14:44.170 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:44.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.168+0000 7fe3a8c0b640 1 Processor -- start 2024-04-22T08:14:44.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.168+0000 7fe3a8c0b640 1 -- start start 2024-04-22T08:14:44.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.168+0000 7fe3a8c0b640 10 monclient: init 2024-04-22T08:14:44.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.169+0000 7fe3a8c0b640 10 monclient: _reopen_session rank -1 2024-04-22T08:14:44.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.169+0000 7fe3a8c0b640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-04-22T08:14:44.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.169+0000 7fe3a8c0b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe3a40a1c60 addr v1:172.21.15.40:6789/0 2024-04-22T08:14:44.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.169+0000 7fe3a8c0b640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe3a40a31d0 0x7fe3a413aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:44.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.169+0000 7fe3a8c0b640 10 monclient(hunting): picked mon.noname-c con 0x7fe3a40a31d0 addr v2:172.21.15.189:3300/0 2024-04-22T08:14:44.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.169+0000 7fe3a8c0b640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe3a413b410 0x7fe3a413d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:44.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.169+0000 7fe3a8c0b640 10 monclient(hunting): picked mon.noname-a con 0x7fe3a413b410 addr v2:172.21.15.40:3300/0 2024-04-22T08:14:44.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.169+0000 7fe3a8c0b640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3a40a41f0 con 0x7fe3a40a1c60 2024-04-22T08:14:44.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.169+0000 7fe3a8c0b640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:44.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.169+0000 7fe3a8c0b640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3a40a3660 con 0x7fe3a413b410 2024-04-22T08:14:44.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.169+0000 7fe3a8c0b640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:44.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.169+0000 7fe3a8c0b640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3a409fe90 con 0x7fe3a40a31d0 2024-04-22T08:14:44.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.169+0000 7fe3a8c0b640 10 monclient(hunting): _renew_subs 2024-04-22T08:14:44.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.169+0000 7fe3a8c0b640 10 monclient(hunting): authenticate will time out at 958.163192s 2024-04-22T08:14:44.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.169+0000 7fe3a2d9d640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe3a413b410 0x7fe3a413d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:44.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.169+0000 7fe3a259c640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fe3a40a1c60 0x7fe3a40a2040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:58956/0 (socket says 172.21.15.40:58956) 2024-04-22T08:14:44.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.169+0000 7fe3a259c640 1 -- 172.21.15.40:0/2534409885 learned_addr learned my addr 172.21.15.40:0/2534409885 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:14:44.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.169+0000 7fe3a2d9d640 10 monclient(hunting): get_auth_request con 0x7fe3a413b410 auth_method 0 2024-04-22T08:14:44.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.169+0000 7fe3a2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:44.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.169+0000 7fe3a2d9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:44.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.169+0000 7fe3a2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:44.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.169+0000 7fe39bfff640 1 --2- 172.21.15.40:0/2534409885 >> v2:172.21.15.189:3300/0 conn(0x7fe3a40a31d0 0x7fe3a413aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:44.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.169+0000 7fe3a2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:44.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.169+0000 7fe3a2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:44.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.169+0000 7fe3a2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:44.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.169+0000 7fe39bfff640 10 monclient(hunting): get_auth_request con 0x7fe3a40a31d0 auth_method 0 2024-04-22T08:14:44.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.169+0000 7fe39bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:44.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.169+0000 7fe39bfff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:44.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.169+0000 7fe39bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:44.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.169+0000 7fe39bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:44.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.169+0000 7fe39bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:44.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.169+0000 7fe39bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:44.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.169+0000 7fe3a2d9d640 10 monclient(hunting): handle_auth_done global_id 34113 payload 274 2024-04-22T08:14:44.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.169+0000 7fe3a2d9d640 1 -- 172.21.15.40:0/2534409885 >> v2:172.21.15.189:3300/0 conn(0x7fe3a40a31d0 msgr2=0x7fe3a413aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:14:44.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.169+0000 7fe3a2d9d640 1 --2- 172.21.15.40:0/2534409885 >> v2:172.21.15.189:3300/0 conn(0x7fe3a40a31d0 0x7fe3a413aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:44.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.169+0000 7fe3a2d9d640 1 -- 172.21.15.40:0/2534409885 >> v1:172.21.15.40:6789/0 conn(0x7fe3a40a1c60 legacy=0x7fe3a40a2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:14:44.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.169+0000 7fe3a2d9d640 10 monclient: _finish_hunting 0 2024-04-22T08:14:44.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.169+0000 7fe3a2d9d640 1 monclient: found mon.noname-a 2024-04-22T08:14:44.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.169+0000 7fe3a2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:14:44.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.169+0000 7fe3a2d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:14:44.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.169+0000 7fe3a2d9d640 1 -- 172.21.15.40:0/2534409885 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a40a2750 con 0x7fe3a413b410 2024-04-22T08:14:44.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.172+0000 7fe3a2d9d640 1 -- 172.21.15.40:0/2534409885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3a413b410 msgr2=0x7fe3a413d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:44.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.172+0000 7fe3a2d9d640 1 --2- 172.21.15.40:0/2534409885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3a413b410 0x7fe3a413d800 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe39c009d90 tx=0x7fe39c02bd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d90fe9a5e3460cb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:44.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.172+0000 7fe3a1d9b640 1 -- 172.21.15.40:0/2534409885 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe39c02ccc0 con 0x7fe3a413b410 2024-04-22T08:14:44.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.172+0000 7fe3a1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:44.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.172+0000 7fe3a1d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:14:44.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.172+0000 7fe3a1d9b640 10 monclient: dump: 2024-04-22T08:14:44.183 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:44.183 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:44.183 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:44.183 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:44.183 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:44.183 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:44.183 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:44.183 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:44.184 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:44.184 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:44.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.172+0000 7fe3a1d9b640 10 monclient: _finish_auth 0 2024-04-22T08:14:44.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.172+0000 7fe3a1d9b640 10 monclient: _check_auth_tickets 2024-04-22T08:14:44.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.172+0000 7fe3a1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:14:44.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.172+0000 7fe3a8c0b640 5 monclient: authenticate success, global_id 34113 2024-04-22T08:14:44.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.172+0000 7fe3a8c0b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:14:44.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.172+0000 7fe3a1d9b640 1 -- 172.21.15.40:0/2534409885 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe39c02ce20 con 0x7fe3a413b410 2024-04-22T08:14:44.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.172+0000 7fe3a1d9b640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:14:44.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.172+0000 7fe3a8c0b640 10 monclient: get_monmap_and_config success 2024-04-22T08:14:44.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.172+0000 7fe3a1d9b640 1 -- 172.21.15.40:0/2534409885 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe39c034660 con 0x7fe3a413b410 2024-04-22T08:14:44.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.172+0000 7fe3a1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:44.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.172+0000 7fe3a8c0b640 4 set_mon_vals no callback set 2024-04-22T08:14:44.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.172+0000 7fe3a1d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:14:44.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.172+0000 7fe3a1d9b640 10 monclient: dump: 2024-04-22T08:14:44.185 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:44.185 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:44.185 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:44.185 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:44.185 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:44.186 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:44.186 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:44.186 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:44.186 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:44.186 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:44.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.173+0000 7fe3a8c0b640 10 monclient: shutdown 2024-04-22T08:14:44.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.173+0000 7fe3a8c0b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:14:44.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.173+0000 7fe3a8c0b640 1 -- 172.21.15.40:0/2534409885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3a413b410 msgr2=0x7fe3a413d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:44.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.173+0000 7fe3a8c0b640 1 --2- 172.21.15.40:0/2534409885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3a413b410 0x7fe3a413d800 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe39c009d90 tx=0x7fe39c02bd00 comp rx=0 tx=0).stop 2024-04-22T08:14:44.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.173+0000 7fe3a8c0b640 1 -- 172.21.15.40:0/2534409885 shutdown_connections 2024-04-22T08:14:44.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.173+0000 7fe3a8c0b640 1 --2- 172.21.15.40:0/2534409885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3a413b410 0x7fe3a413d800 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:44.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.173+0000 7fe3a8c0b640 1 --2- 172.21.15.40:0/2534409885 >> v2:172.21.15.189:3300/0 conn(0x7fe3a40a31d0 0x7fe3a413aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:44.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.173+0000 7fe3a8c0b640 1 -- 172.21.15.40:0/2534409885 >> 172.21.15.40:0/2534409885 conn(0x7fe3a40a89a0 msgr2=0x7fe3a40a4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:14:44.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.173+0000 7fe3a8c0b640 1 -- 172.21.15.40:0/2534409885 shutdown_connections 2024-04-22T08:14:44.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.173+0000 7fe3a8c0b640 1 -- 172.21.15.40:0/2534409885 wait complete. 2024-04-22T08:14:44.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.174+0000 7fe3a8c0b640 10 monclient: build_initial_monmap 2024-04-22T08:14:44.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.174+0000 7fe3a8c0b640 10 monclient: monmap: 2024-04-22T08:14:44.187 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:14:44.187 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:14:44.187 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:14:44.187 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:14:44.187 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:14:44.187 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:44.187 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:14:44.187 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:14:44.187 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:14:44.187 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:44.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.174+0000 7fe3a8c0b640 1 Processor -- start 2024-04-22T08:14:44.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.174+0000 7fe3a8c0b640 1 -- start start 2024-04-22T08:14:44.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.174+0000 7fe3a8c0b640 10 monclient: init 2024-04-22T08:14:44.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe3a8c0b640 10 monclient: _reopen_session rank -1 2024-04-22T08:14:44.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe3a8c0b640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:14:44.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe3a8c0b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3a40a1c60 0x7fe3a40703d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:44.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe3a8c0b640 10 monclient(hunting): picked mon.noname-a con 0x7fe3a40a1c60 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:14:44.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe3a8c0b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3a40a31d0 0x7fe3a41349c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:44.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe3a8c0b640 10 monclient(hunting): picked mon.noname-b con 0x7fe3a40a31d0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:14:44.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe3a8c0b640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3a413b410 0x7fe3a41ca680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:44.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe3a8c0b640 10 monclient(hunting): picked mon.noname-c con 0x7fe3a413b410 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:14:44.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe3a8c0b640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:44.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe3a8c0b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a4070f90 con 0x7fe3a40a1c60 2024-04-22T08:14:44.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe3a8c0b640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:44.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe3a8c0b640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a4120db0 con 0x7fe3a40a31d0 2024-04-22T08:14:44.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe3a8c0b640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:44.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe3a8c0b640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a4121090 con 0x7fe3a413b410 2024-04-22T08:14:44.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe3a8c0b640 10 monclient(hunting): _renew_subs 2024-04-22T08:14:44.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe3a8c0b640 10 monclient(hunting): authenticate will time out at 958.168923s 2024-04-22T08:14:44.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe3a259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3a40a1c60 0x7fe3a40703d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:44.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe3a259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3a40a1c60 0x7fe3a40703d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35096/0 (socket says 172.21.15.40:35096) 2024-04-22T08:14:44.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe3a259c640 1 -- 172.21.15.40:0/3570058952 learned_addr learned my addr 172.21.15.40:0/3570058952 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:14:44.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe3a259c640 10 monclient(hunting): get_auth_request con 0x7fe3a40a1c60 auth_method 0 2024-04-22T08:14:44.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe3a2d9d640 1 --2- 172.21.15.40:0/3570058952 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3a413b410 0x7fe3a41ca680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:44.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe3a259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:44.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe3a259c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:44.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe3a259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:44.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe3a2d9d640 10 monclient(hunting): get_auth_request con 0x7fe3a413b410 auth_method 0 2024-04-22T08:14:44.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe3a2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:44.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe3a2d9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:44.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe3a2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:44.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe3a259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:44.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe3a259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:44.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe3a259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:44.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe3a2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:44.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe3a2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:44.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe39bfff640 1 --2- 172.21.15.40:0/3570058952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3a40a31d0 0x7fe3a41349c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:44.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe3a2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:44.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe39bfff640 10 monclient(hunting): get_auth_request con 0x7fe3a40a31d0 auth_method 0 2024-04-22T08:14:44.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe39bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:44.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe39bfff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:44.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe39bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:44.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe39bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:44.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe39bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:44.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe39bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:44.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe3a2d9d640 10 monclient(hunting): handle_auth_done global_id 4727 payload 824 2024-04-22T08:14:44.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe3a2d9d640 1 -- 172.21.15.40:0/3570058952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3a40a31d0 msgr2=0x7fe3a41349c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:44.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe3a2d9d640 1 --2- 172.21.15.40:0/3570058952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3a40a31d0 0x7fe3a41349c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:44.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe3a2d9d640 1 -- 172.21.15.40:0/3570058952 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3a40a1c60 msgr2=0x7fe3a40703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:44.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe3a2d9d640 1 --2- 172.21.15.40:0/3570058952 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3a40a1c60 0x7fe3a40703d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:44.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe3a2d9d640 10 monclient: _finish_hunting 0 2024-04-22T08:14:44.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe3a2d9d640 1 monclient: found mon.noname-c 2024-04-22T08:14:44.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe3a2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:14:44.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe3a2d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:14:44.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe3a2d9d640 1 -- 172.21.15.40:0/3570058952 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe39c009a70 con 0x7fe3a413b410 2024-04-22T08:14:44.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.175+0000 7fe3a259c640 1 --2- 172.21.15.40:0/3570058952 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3a40a1c60 0x7fe3a40703d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:14:44.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.177+0000 7fe3a2d9d640 1 -- 172.21.15.40:0/3570058952 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3a413b410 msgr2=0x7fe3a41ca680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:44.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.177+0000 7fe3a2d9d640 1 --2- 172.21.15.40:0/3570058952 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3a413b410 0x7fe3a41ca680 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe3a40a5980 tx=0x7fe39c004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:44.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.177+0000 7fe39affd640 1 -- 172.21.15.40:0/3570058952 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe39c040650 con 0x7fe3a413b410 2024-04-22T08:14:44.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.177+0000 7fe39affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:44.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.177+0000 7fe39affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:14:44.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.177+0000 7fe39affd640 10 monclient: dump: 2024-04-22T08:14:44.192 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:44.192 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:44.192 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:44.192 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:44.192 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:44.192 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:44.192 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:44.192 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:44.192 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:44.192 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:44.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.178+0000 7fe39affd640 10 monclient: _finish_auth 0 2024-04-22T08:14:44.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.178+0000 7fe39affd640 10 monclient: _check_auth_tickets 2024-04-22T08:14:44.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.178+0000 7fe39affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:14:44.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.178+0000 7fe39affd640 1 -- 172.21.15.40:0/3570058952 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe39c0407b0 con 0x7fe3a413b410 2024-04-22T08:14:44.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.178+0000 7fe3a8c0b640 5 monclient: authenticate success, global_id 4727 2024-04-22T08:14:44.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.178+0000 7fe3a8c0b640 10 monclient: _renew_subs 2024-04-22T08:14:44.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.178+0000 7fe3a8c0b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:14:44.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.178+0000 7fe3a8c0b640 1 -- 172.21.15.40:0/3570058952 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3a4120ad0 con 0x7fe3a413b410 2024-04-22T08:14:44.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.178+0000 7fe39affd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:14:44.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.178+0000 7fe39affd640 1 -- 172.21.15.40:0/3570058952 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe39c040b40 con 0x7fe3a413b410 2024-04-22T08:14:44.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.178+0000 7fe39affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:44.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.178+0000 7fe39affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:14:44.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.178+0000 7fe39affd640 10 monclient: dump: 2024-04-22T08:14:44.193 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:44.194 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:44.194 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:44.194 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:44.194 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:44.194 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:44.194 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:44.194 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:44.194 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:44.194 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:44.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.178+0000 7fe3a8c0b640 10 monclient: _renew_subs 2024-04-22T08:14:44.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.178+0000 7fe3a8c0b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:14:44.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.178+0000 7fe3a159a640 4 set_mon_vals no callback set 2024-04-22T08:14:44.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.178+0000 7fe3a8c0b640 1 -- 172.21.15.40:0/3570058952 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3a4120030 con 0x7fe3a413b410 2024-04-22T08:14:44.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.178+0000 7fe3a8c0b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:14:44.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.178+0000 7fe3a8c0b640 1 --2- 172.21.15.40:0/3570058952 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3a411e960 0x7fe3a41cabc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:44.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.178+0000 7fe3a8c0b640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:44.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.178+0000 7fe3a8c0b640 1 -- 172.21.15.40:0/3570058952 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a4066650 con 0x7fe3a411e960 2024-04-22T08:14:44.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.178+0000 7fe3a2d9d640 1 --2- 172.21.15.40:0/3570058952 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3a413b410 0x7fe3a41ca680 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe3a40a5980 tx=0x7fe39c004ab0 comp rx=0 tx=0).queue_received will delay after 1.71377e+09 on 0x7fe39c06ce50 osd_map(70..70 src has 1..70) v4 2024-04-22T08:14:44.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.178+0000 7fe39affd640 1 -- 172.21.15.40:0/3570058952 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fe39c040d20 con 0x7fe3a413b410 2024-04-22T08:14:44.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.179+0000 7fe3a259c640 1 --2- 172.21.15.40:0/3570058952 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3a411e960 0x7fe3a41cabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:44.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.179+0000 7fe3a259c640 10 monclient: get_auth_request con 0x7fe3a411e960 auth_method 0 2024-04-22T08:14:44.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.179+0000 7fe3a259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:44.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.179+0000 7fe3a259c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:44.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.179+0000 7fe3a259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:44.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.179+0000 7fe3a259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:44.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.179+0000 7fe3a259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:44.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.179+0000 7fe3a259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:44.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.179+0000 7fe39affd640 1 --2- 172.21.15.40:0/3570058952 >> v2:172.21.15.189:6800/30168 conn(0x7fe384037800 0x7fe384039cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:44.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.179+0000 7fe3a259c640 10 monclient(hunting): handle_auth_done global_id 4730 payload 824 2024-04-22T08:14:44.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.179+0000 7fe3a259c640 1 -- 172.21.15.40:0/3570058952 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe3a40a0450 con 0x7fe3a411e960 2024-04-22T08:14:44.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.179+0000 7fe39bfff640 1 --2- 172.21.15.40:0/3570058952 >> v2:172.21.15.189:6800/30168 conn(0x7fe384037800 0x7fe384039cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:44.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.179+0000 7fe39bfff640 10 monclient: get_auth_request con 0x7fe384037800 auth_method 0 2024-04-22T08:14:44.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.181+0000 7fe3a259c640 1 -- 172.21.15.40:0/3570058952 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3a411e960 msgr2=0x7fe3a41cabc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:44.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.181+0000 7fe3a259c640 1 --2- 172.21.15.40:0/3570058952 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3a411e960 0x7fe3a41cabc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe3940072e0 tx=0x7fe3940077a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:44.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.182+0000 7fe39bfff640 1 --2- 172.21.15.40:0/3570058952 >> v2:172.21.15.189:6800/30168 conn(0x7fe384037800 0x7fe384039cc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe3a40a8120 tx=0x7fe38c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:44.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.185+0000 7fe39affd640 1 -- 172.21.15.40:0/3570058952 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe39400fd40 con 0x7fe3a411e960 2024-04-22T08:14:44.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.185+0000 7fe39affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:44.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.185+0000 7fe39affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:14:44.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.185+0000 7fe39affd640 10 monclient: dump: 2024-04-22T08:14:44.198 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:44.198 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:44.198 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:44.198 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:44.198 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:44.198 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:44.198 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:44.198 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:44.199 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:44.199 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:44.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.185+0000 7fe39affd640 1 -- 172.21.15.40:0/3570058952 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7fe39400fea0 con 0x7fe3a411e960 2024-04-22T08:14:44.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.185+0000 7fe39affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:14:44.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.185+0000 7fe39affd640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:14:44.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.185+0000 7fe39affd640 1 -- 172.21.15.40:0/3570058952 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3a411e960 msgr2=0x7fe3a41cabc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:44.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.185+0000 7fe39affd640 1 --2- 172.21.15.40:0/3570058952 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3a411e960 0x7fe3a41cabc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe3940072e0 tx=0x7fe3940077a0 comp rx=0 tx=0).stop 2024-04-22T08:14:44.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.185+0000 7fe39affd640 1 -- 172.21.15.40:0/3570058952 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3a411e960 msgr2=0x7fe3a41cabc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:14:44.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.185+0000 7fe39affd640 1 --2- 172.21.15.40:0/3570058952 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3a411e960 0x7fe3a41cabc0 secure :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe3940072e0 tx=0x7fe3940077a0 comp rx=0 tx=0).stop 2024-04-22T08:14:44.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.198+0000 7fe3a8c0b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:14:44.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.198+0000 7fe3a8c0b640 1 --2- 172.21.15.40:0/3570058952 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3a41cb100 0x7fe3a406c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:44.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.198+0000 7fe3a8c0b640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:44.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.198+0000 7fe3a8c0b640 1 -- 172.21.15.40:0/3570058952 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a4120c60 con 0x7fe3a41cb100 2024-04-22T08:14:44.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.198+0000 7fe3a259c640 1 --2- 172.21.15.40:0/3570058952 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3a41cb100 0x7fe3a406c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:44.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.198+0000 7fe3a259c640 10 monclient: get_auth_request con 0x7fe3a41cb100 auth_method 0 2024-04-22T08:14:44.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.198+0000 7fe3a259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:44.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.198+0000 7fe3a259c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:44.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.198+0000 7fe3a259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:44.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.199+0000 7fe3a259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:44.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.199+0000 7fe3a259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:44.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.199+0000 7fe3a259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:44.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.199+0000 7fe3a259c640 10 monclient(hunting): handle_auth_done global_id 4733 payload 824 2024-04-22T08:14:44.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.199+0000 7fe3a259c640 1 -- 172.21.15.40:0/3570058952 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe3a41cb510 con 0x7fe3a41cb100 2024-04-22T08:14:44.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.201+0000 7fe3a259c640 1 -- 172.21.15.40:0/3570058952 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3a41cb100 msgr2=0x7fe3a406c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:44.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.201+0000 7fe3a259c640 1 --2- 172.21.15.40:0/3570058952 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3a41cb100 0x7fe3a406c570 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe3940072e0 tx=0x7fe394004940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:44.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.201+0000 7fe39affd640 1 -- 172.21.15.40:0/3570058952 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe394024070 con 0x7fe3a41cb100 2024-04-22T08:14:44.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.201+0000 7fe39affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:44.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.201+0000 7fe39affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:14:44.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.201+0000 7fe39affd640 10 monclient: dump: 2024-04-22T08:14:44.203 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:44.203 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:44.203 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:44.203 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:44.203 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:44.203 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:44.203 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:44.203 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:44.203 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:44.203 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:44.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.202+0000 7fe39affd640 1 -- 172.21.15.40:0/3570058952 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4631 (secure 0 0 0) 0x7fe39401d4b0 con 0x7fe3a41cb100 2024-04-22T08:14:44.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.202+0000 7fe39affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:14:44.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.202+0000 7fe39affd640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:14:44.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.202+0000 7fe39affd640 1 -- 172.21.15.40:0/3570058952 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3a41cb100 msgr2=0x7fe3a406c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:44.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.202+0000 7fe39affd640 1 --2- 172.21.15.40:0/3570058952 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3a41cb100 0x7fe3a406c570 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe3940072e0 tx=0x7fe394004940 comp rx=0 tx=0).stop 2024-04-22T08:14:44.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.202+0000 7fe39affd640 1 -- 172.21.15.40:0/3570058952 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3a41cb100 msgr2=0x7fe3a406c570 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:14:44.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.202+0000 7fe39affd640 1 --2- 172.21.15.40:0/3570058952 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3a41cb100 0x7fe3a406c570 secure :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe3940072e0 tx=0x7fe394004940 comp rx=0 tx=0).stop 2024-04-22T08:14:44.204 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:14:44.204 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:14:44.204 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:14:44.204 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:14:44.204 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 26, 2024-04-22T08:14:44.204 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:14:44.204 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:14:44.205 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:14:44.205 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:14:44.205 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:14:44.205 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 1, 2024-04-22T08:14:44.205 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:14:44.205 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:14:44.205 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:14:44.205 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:14:44.205 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:14:44.205 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:14:44.205 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:14:44.205 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:14:44.205 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:14:44.205 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:14:44.206 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:14:44.206 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:14:44.206 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:14:44.206 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:14:44.206 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:14:44.206 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:14:44.206 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:14:44.206 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:14:44.206 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:14:44.206 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:14:44.206 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:14:44.206 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:14:44.206 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:14:44.206 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:14:44.207 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:14:44.207 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:14:44.207 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:44.207 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:44.207 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:14:44.207 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:14:44.207 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:14:44.207 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:14:44.207 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:14:44.207 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:14:44.207 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:14:44.207 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:14:44.207 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:14:44.207 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:14:44.208 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:14:44.208 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:14:44.208 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:14:44.208 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:14:44.208 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:14:44.208 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:14:44.208 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:14:44.208 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:14:44.208 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:14:44.208 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:14:44.208 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:14:44.208 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:14:44.208 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:14:44.208 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:14:44.209 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:14:44.209 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:14:44.209 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:14:44.209 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:14:44.209 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:14:44.209 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:44.209 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:14:44.209 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:44.209 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:14:44.209 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:14:44.209 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:14:44.209 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:14:44.209 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:44.209 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:14:44.210 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:14:44.210 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:44.210 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:44.210 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:44.210 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:14:44.210 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:14:44.210 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:44.210 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:44.210 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:44.210 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:44.210 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:14:44.210 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:14:44.210 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:14:44.210 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:14:44.210 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:14:44.211 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:44.211 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:44.211 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:14:44.211 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:14:44.211 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:14:44.211 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:14:44.211 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:44.211 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:14:44.211 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:14:44.211 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:44.211 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:44.211 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:44.211 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:14:44.211 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:14:44.212 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:44.212 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:44.212 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:44.212 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:44.212 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:14:44.212 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:14:44.212 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:14:44.212 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:14:44.212 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:14:44.212 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:44.212 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:44.212 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:14:44.212 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:14:44.212 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:14:44.213 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:14:44.213 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:44.213 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:14:44.213 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:14:44.213 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:44.213 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:44.213 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:44.213 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:14:44.213 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:14:44.213 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:44.213 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:44.214 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:44.214 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:44.214 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:14:44.214 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:14:44.214 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:14:44.214 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:14:44.214 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:14:44.214 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:44.214 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:44.215 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:44.215 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:14:44.215 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:14:44.215 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:44.215 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:14:44.215 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:14:44.215 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:14:44.215 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:44.215 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:14:44.215 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:14:44.215 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:44.216 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:14:44.216 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:14:44.216 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 5 2024-04-22T08:14:44.216 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:44.216 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:14:44.216 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:14:44.216 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:44.216 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:14:44.216 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:14:44.216 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 3 2024-04-22T08:14:44.217 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:44.217 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:44.217 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:44.217 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:14:44.217 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:14:44.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.202+0000 7fe3a8c0b640 1 -- 172.21.15.40:0/3570058952 >> v2:172.21.15.189:6800/30168 conn(0x7fe384037800 msgr2=0x7fe384039cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:44.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.202+0000 7fe3a8c0b640 1 --2- 172.21.15.40:0/3570058952 >> v2:172.21.15.189:6800/30168 conn(0x7fe384037800 0x7fe384039cc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe3a40a8120 tx=0x7fe38c009190 comp rx=0 tx=0).stop 2024-04-22T08:14:44.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.202+0000 7fe3a8c0b640 10 monclient: shutdown 2024-04-22T08:14:44.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.202+0000 7fe3a8c0b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:14:44.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.202+0000 7fe3a8c0b640 1 -- 172.21.15.40:0/3570058952 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3a413b410 msgr2=0x7fe3a41ca680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:44.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.202+0000 7fe3a8c0b640 1 --2- 172.21.15.40:0/3570058952 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3a413b410 0x7fe3a41ca680 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe3a40a5980 tx=0x7fe39c004ab0 comp rx=0 tx=0).stop 2024-04-22T08:14:44.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.202+0000 7fe3a2d9d640 1 -- 172.21.15.40:0/3570058952 reap_dead start 2024-04-22T08:14:44.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.202+0000 7fe3a2d9d640 1 -- 172.21.15.40:0/3570058952 reap_dead start 2024-04-22T08:14:44.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.202+0000 7fe39affd640 1 -- 172.21.15.40:0/3570058952 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 3898+0+0 (unknown 0 0 0) 0x7fe39c06ce50 con 0x7fe3a413b410 2024-04-22T08:14:44.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.202+0000 7fe3a8c0b640 1 -- 172.21.15.40:0/3570058952 shutdown_connections 2024-04-22T08:14:44.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.202+0000 7fe3a8c0b640 1 -- 172.21.15.40:0/3570058952 >> 172.21.15.40:0/3570058952 conn(0x7fe3a40a89a0 msgr2=0x7fe3a409de00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:14:44.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.202+0000 7fe3a8c0b640 1 -- 172.21.15.40:0/3570058952 shutdown_connections 2024-04-22T08:14:44.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.203+0000 7fe3a8c0b640 1 -- 172.21.15.40:0/3570058952 wait complete. 2024-04-22T08:14:44.220 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:14:44.220 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-22T08:14:44.220 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-22T08:14:44.301 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35382]: starting. 2024-04-22T08:14:44.302 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35382]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-35381 2024-04-22T08:14:44.302 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35382]: created object 0... 2024-04-22T08:14:44.302 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35382]: created object 25... 2024-04-22T08:14:44.302 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35382]: created object 49... 2024-04-22T08:14:44.302 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35382]: finishing. 2024-04-22T08:14:44.302 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35382]: shutting down. 2024-04-22T08:14:44.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.326+0000 7f7fc73a9640 10 monclient: get_monmap_and_config 2024-04-22T08:14:44.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.326+0000 7f7fc73a9640 10 monclient: build_initial_monmap 2024-04-22T08:14:44.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.326+0000 7f7fc73a9640 10 monclient: monmap: 2024-04-22T08:14:44.326 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:14:44.326 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:44.326 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:14:44.327293+0000 2024-04-22T08:14:44.326 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:14:44.327293+0000 2024-04-22T08:14:44.326 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:14:44.326 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:44.326 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:14:44.326 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:14:44.327 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:14:44.327 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:14:44.327 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:14:44.327 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:44.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.327+0000 7f7fc73a9640 1 Processor -- start 2024-04-22T08:14:44.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.327+0000 7f7fc73a9640 1 -- start start 2024-04-22T08:14:44.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.327+0000 7f7fc73a9640 10 monclient: init 2024-04-22T08:14:44.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.327+0000 7f7fc73a9640 10 monclient: _reopen_session rank -1 2024-04-22T08:14:44.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.327+0000 7f7fc73a9640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-04-22T08:14:44.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.327+0000 7f7fc73a9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7fc00a1c60 addr v1:172.21.15.189:6789/0 2024-04-22T08:14:44.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.327+0000 7f7fc73a9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7fc00a31d0 addr v1:172.21.15.40:6789/0 2024-04-22T08:14:44.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.327+0000 7f7fc73a9640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7fc013c5f0 0x7f7fc013e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:44.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.327+0000 7f7fc73a9640 10 monclient(hunting): picked mon.noname-b con 0x7f7fc013c5f0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:14:44.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.327+0000 7f7fc73a9640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc00a41f0 con 0x7f7fc00a31d0 2024-04-22T08:14:44.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.327+0000 7f7fc73a9640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc009fe90 con 0x7f7fc00a1c60 2024-04-22T08:14:44.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.327+0000 7f7fc73a9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:44.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.327+0000 7f7fc73a9640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc00a3660 con 0x7f7fc013c5f0 2024-04-22T08:14:44.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.327+0000 7f7fc73a9640 10 monclient(hunting): _renew_subs 2024-04-22T08:14:44.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.327+0000 7f7fc73a9640 10 monclient(hunting): authenticate will time out at 958.321638s 2024-04-22T08:14:44.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.327+0000 7f7fc5946640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7fc013c5f0 0x7f7fc013e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:44.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.327+0000 7f7fc4944640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f7fc00a31d0 0x7f7fc013aed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:58960/0 (socket says 172.21.15.40:58960) 2024-04-22T08:14:44.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.327+0000 7f7fc4944640 1 -- 172.21.15.40:0/48249015 learned_addr learned my addr 172.21.15.40:0/48249015 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:14:44.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.328+0000 7f7fc5946640 10 monclient(hunting): get_auth_request con 0x7f7fc013c5f0 auth_method 0 2024-04-22T08:14:44.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.328+0000 7f7fc5946640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:44.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.328+0000 7f7fc5946640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:44.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.328+0000 7f7fc5946640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:44.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.328+0000 7f7fc5946640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:44.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.328+0000 7f7fc5946640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:44.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.328+0000 7f7fc5946640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:44.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.328+0000 7f7fc5946640 10 monclient(hunting): handle_auth_done global_id 4736 payload 274 2024-04-22T08:14:44.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.328+0000 7f7fc5946640 1 -- 172.21.15.40:0/48249015 >> v1:172.21.15.189:6789/0 conn(0x7f7fc00a1c60 legacy=0x7f7fc00a2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:14:44.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.328+0000 7f7fc5946640 1 -- 172.21.15.40:0/48249015 >> v1:172.21.15.40:6789/0 conn(0x7f7fc00a31d0 legacy=0x7f7fc013aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:14:44.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.328+0000 7f7fc5946640 10 monclient: _finish_hunting 0 2024-04-22T08:14:44.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.328+0000 7f7fc5946640 1 monclient: found mon.noname-b 2024-04-22T08:14:44.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.328+0000 7f7fc5946640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:14:44.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.328+0000 7f7fc5946640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:14:44.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.328+0000 7f7fc5946640 1 -- 172.21.15.40:0/48249015 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc00a2750 con 0x7f7fc013c5f0 2024-04-22T08:14:44.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.331+0000 7f7fc5946640 1 -- 172.21.15.40:0/48249015 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7fc013c5f0 msgr2=0x7f7fc013e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:44.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.331+0000 7f7fc5946640 1 --2- 172.21.15.40:0/48249015 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7fc013c5f0 0x7f7fc013e9e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7fc00a37a0 tx=0x7f7fb802bce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5e2ece7d2d64ce3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:44.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.331+0000 7f7fb7fff640 1 -- 172.21.15.40:0/48249015 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb802cd80 con 0x7f7fc013c5f0 2024-04-22T08:14:44.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.331+0000 7f7fb7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:44.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.331+0000 7f7fb7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:14:44.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.331+0000 7f7fb7fff640 10 monclient: dump: 2024-04-22T08:14:44.335 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:44.335 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:44.335 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:44.335 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:44.335 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:44.335 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:44.335 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:44.336 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:44.336 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:44.336 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:44.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.331+0000 7f7fb7fff640 10 monclient: _finish_auth 0 2024-04-22T08:14:44.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.331+0000 7f7fb7fff640 10 monclient: _check_auth_tickets 2024-04-22T08:14:44.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.331+0000 7f7fb7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:14:44.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.331+0000 7f7fb7fff640 1 -- 172.21.15.40:0/48249015 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fb8034440 con 0x7f7fc013c5f0 2024-04-22T08:14:44.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.331+0000 7f7fc73a9640 5 monclient: authenticate success, global_id 4736 2024-04-22T08:14:44.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.331+0000 7f7fc73a9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:14:44.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.331+0000 7f7fb7fff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:14:44.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.331+0000 7f7fb7fff640 1 -- 172.21.15.40:0/48249015 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb80347d0 con 0x7f7fc013c5f0 2024-04-22T08:14:44.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.331+0000 7f7fc73a9640 10 monclient: get_monmap_and_config success 2024-04-22T08:14:44.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.331+0000 7f7fc73a9640 4 set_mon_vals no callback set 2024-04-22T08:14:44.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.331+0000 7f7fb7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:44.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.331+0000 7f7fb7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:14:44.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.331+0000 7f7fb7fff640 10 monclient: dump: 2024-04-22T08:14:44.337 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:44.337 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:44.337 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:44.337 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:44.337 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:44.337 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:44.337 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:44.337 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:44.337 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:44.337 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:44.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.331+0000 7f7fc73a9640 10 monclient: shutdown 2024-04-22T08:14:44.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.331+0000 7f7fc73a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:14:44.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.331+0000 7f7fc73a9640 1 -- 172.21.15.40:0/48249015 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7fc013c5f0 msgr2=0x7f7fc013e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:44.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.331+0000 7f7fc73a9640 1 --2- 172.21.15.40:0/48249015 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7fc013c5f0 0x7f7fc013e9e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7fc00a37a0 tx=0x7f7fb802bce0 comp rx=0 tx=0).stop 2024-04-22T08:14:44.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.331+0000 7f7fc73a9640 1 -- 172.21.15.40:0/48249015 shutdown_connections 2024-04-22T08:14:44.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.331+0000 7f7fc73a9640 1 --2- 172.21.15.40:0/48249015 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7fc013c5f0 0x7f7fc013e9e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:44.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.331+0000 7f7fc73a9640 1 -- 172.21.15.40:0/48249015 >> 172.21.15.40:0/48249015 conn(0x7f7fc00a89a0 msgr2=0x7f7fc00a4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:14:44.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.332+0000 7f7fc73a9640 1 -- 172.21.15.40:0/48249015 shutdown_connections 2024-04-22T08:14:44.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.332+0000 7f7fc73a9640 1 -- 172.21.15.40:0/48249015 wait complete. 2024-04-22T08:14:44.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.332+0000 7f7fc73a9640 10 monclient: build_initial_monmap 2024-04-22T08:14:44.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.332+0000 7f7fc73a9640 10 monclient: monmap: 2024-04-22T08:14:44.338 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:14:44.338 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:14:44.338 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:14:44.338 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:14:44.338 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:14:44.338 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:44.339 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:14:44.339 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:14:44.339 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:14:44.339 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:44.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.332+0000 7f7fc73a9640 1 Processor -- start 2024-04-22T08:14:44.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.332+0000 7f7fc73a9640 1 -- start start 2024-04-22T08:14:44.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.332+0000 7f7fc73a9640 10 monclient: init 2024-04-22T08:14:44.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc73a9640 10 monclient: _reopen_session rank -1 2024-04-22T08:14:44.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc73a9640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-22T08:14:44.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc73a9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7fc00a1c60 0x7f7fc013a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:44.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc73a9640 10 monclient(hunting): picked mon.noname-b con 0x7f7fc00a1c60 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:14:44.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc73a9640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7fc00a31d0 0x7f7fc0135a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:44.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc73a9640 10 monclient(hunting): picked mon.noname-c con 0x7f7fc00a31d0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:14:44.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc73a9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7fc013c5f0 0x7f7fc011e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:44.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc73a9640 10 monclient(hunting): picked mon.noname-a con 0x7f7fc013c5f0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:14:44.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc73a9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:44.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc73a9640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc00a4790 con 0x7f7fc013c5f0 2024-04-22T08:14:44.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc73a9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:44.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc73a9640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc013bf80 con 0x7f7fc00a1c60 2024-04-22T08:14:44.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc73a9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:44.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc73a9640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc01213b0 con 0x7f7fc00a31d0 2024-04-22T08:14:44.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc73a9640 10 monclient(hunting): _renew_subs 2024-04-22T08:14:44.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc73a9640 10 monclient(hunting): authenticate will time out at 958.327121s 2024-04-22T08:14:44.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc5946640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7fc013c5f0 0x7f7fc011e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:44.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc4944640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7fc00a31d0 0x7f7fc0135a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:44.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc5946640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7fc013c5f0 0x7f7fc011e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35098/0 (socket says 172.21.15.40:35098) 2024-04-22T08:14:44.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc5946640 1 -- 172.21.15.40:0/3702734453 learned_addr learned my addr 172.21.15.40:0/3702734453 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:14:44.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc5946640 10 monclient(hunting): get_auth_request con 0x7f7fc013c5f0 auth_method 0 2024-04-22T08:14:44.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc5946640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:44.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc5946640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:44.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc5946640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:44.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc4944640 10 monclient(hunting): get_auth_request con 0x7f7fc00a31d0 auth_method 0 2024-04-22T08:14:44.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc4944640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:44.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc4944640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:44.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc4944640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:44.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc5946640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:44.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc5946640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:44.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc5145640 1 --2- 172.21.15.40:0/3702734453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7fc00a1c60 0x7f7fc013a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:44.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc5946640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:44.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc5145640 10 monclient(hunting): get_auth_request con 0x7f7fc00a1c60 auth_method 0 2024-04-22T08:14:44.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc5145640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:44.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc5145640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:44.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc5145640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:44.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc4944640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:44.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc4944640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:44.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc4944640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:44.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc5145640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:44.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc5145640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:44.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc5145640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:44.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc4944640 10 monclient(hunting): handle_auth_done global_id 4739 payload 824 2024-04-22T08:14:44.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc4944640 1 -- 172.21.15.40:0/3702734453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7fc00a1c60 msgr2=0x7f7fc013a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:44.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc4944640 1 --2- 172.21.15.40:0/3702734453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7fc00a1c60 0x7f7fc013a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:44.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc4944640 1 -- 172.21.15.40:0/3702734453 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7fc013c5f0 msgr2=0x7f7fc011e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:44.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc4944640 1 --2- 172.21.15.40:0/3702734453 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7fc013c5f0 0x7f7fc011e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:44.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc4944640 10 monclient: _finish_hunting 0 2024-04-22T08:14:44.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc4944640 1 monclient: found mon.noname-c 2024-04-22T08:14:44.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc4944640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:14:44.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc4944640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:14:44.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc4944640 1 -- 172.21.15.40:0/3702734453 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb8009a70 con 0x7f7fc00a31d0 2024-04-22T08:14:44.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.333+0000 7f7fc5946640 1 --2- 172.21.15.40:0/3702734453 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7fc013c5f0 0x7f7fc011e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:14:44.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.336+0000 7f7fc4944640 1 -- 172.21.15.40:0/3702734453 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7fc00a31d0 msgr2=0x7f7fc0135a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:44.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.336+0000 7f7fc4944640 1 --2- 172.21.15.40:0/3702734453 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7fc00a31d0 0x7f7fc0135a10 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7fb000c3e0 tx=0x7f7fb000c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:44.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.336+0000 7f7fb5ffb640 1 -- 172.21.15.40:0/3702734453 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb00070d0 con 0x7f7fc00a31d0 2024-04-22T08:14:44.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.336+0000 7f7fb5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:44.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.336+0000 7f7fb5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:14:44.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.336+0000 7f7fb5ffb640 10 monclient: dump: 2024-04-22T08:14:44.347 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:44.347 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:44.348 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:44.348 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:44.348 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:44.348 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:44.348 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:44.348 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:44.348 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:44.348 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:44.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.336+0000 7f7fb5ffb640 10 monclient: _finish_auth 0 2024-04-22T08:14:44.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.336+0000 7f7fb5ffb640 10 monclient: _check_auth_tickets 2024-04-22T08:14:44.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.336+0000 7f7fb5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:14:44.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.336+0000 7f7fb5ffb640 1 -- 172.21.15.40:0/3702734453 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fb0007230 con 0x7f7fc00a31d0 2024-04-22T08:14:44.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.336+0000 7f7fc73a9640 5 monclient: authenticate success, global_id 4739 2024-04-22T08:14:44.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.336+0000 7f7fc73a9640 10 monclient: _renew_subs 2024-04-22T08:14:44.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.336+0000 7f7fc73a9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:14:44.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.336+0000 7f7fc73a9640 1 -- 172.21.15.40:0/3702734453 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fc0120b10 con 0x7f7fc00a31d0 2024-04-22T08:14:44.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.336+0000 7f7fb5ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:14:44.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.336+0000 7f7fb5ffb640 1 -- 172.21.15.40:0/3702734453 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb00075e0 con 0x7f7fc00a31d0 2024-04-22T08:14:44.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.336+0000 7f7fb5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:44.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.336+0000 7f7fb5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:14:44.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.336+0000 7f7fb5ffb640 10 monclient: dump: 2024-04-22T08:14:44.349 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:44.349 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:44.349 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:44.349 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:44.349 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:44.349 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:44.349 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:44.349 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:44.350 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:44.350 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:44.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.336+0000 7f7fb77fe640 4 set_mon_vals no callback set 2024-04-22T08:14:44.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.336+0000 7f7fc73a9640 10 monclient: _renew_subs 2024-04-22T08:14:44.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.336+0000 7f7fc73a9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:14:44.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.336+0000 7f7fc73a9640 1 -- 172.21.15.40:0/3702734453 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fc013b430 con 0x7f7fc00a31d0 2024-04-22T08:14:44.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.337+0000 7f7fb5ffb640 1 -- 172.21.15.40:0/3702734453 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f7fb001b3b0 con 0x7f7fc00a31d0 2024-04-22T08:14:44.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.337+0000 7f7fb5ffb640 1 --2- 172.21.15.40:0/3702734453 >> v2:172.21.15.189:6800/30168 conn(0x7f7fa40377d0 0x7f7fa4039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:44.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.337+0000 7f7fc73a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:14:44.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.337+0000 7f7fc73a9640 1 --2- 172.21.15.40:0/3702734453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7fc01c16e0 0x7f7fc011eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:44.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.337+0000 7f7fc73a9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:44.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.337+0000 7f7fc73a9640 1 -- 172.21.15.40:0/3702734453 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc0066650 con 0x7f7fc01c16e0 2024-04-22T08:14:44.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.337+0000 7f7fb5ffb640 1 -- 172.21.15.40:0/3702734453 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f7fb0049a20 con 0x7f7fc00a31d0 2024-04-22T08:14:44.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.337+0000 7f7fc5946640 1 --2- 172.21.15.40:0/3702734453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7fc01c16e0 0x7f7fc011eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:44.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.337+0000 7f7fc5145640 1 --2- 172.21.15.40:0/3702734453 >> v2:172.21.15.189:6800/30168 conn(0x7f7fa40377d0 0x7f7fa4039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:44.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.337+0000 7f7fc5946640 10 monclient: get_auth_request con 0x7f7fc01c16e0 auth_method 0 2024-04-22T08:14:44.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.337+0000 7f7fc5946640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:44.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.337+0000 7f7fc5946640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:44.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.337+0000 7f7fc5946640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:44.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.337+0000 7f7fc5145640 10 monclient: get_auth_request con 0x7f7fa40377d0 auth_method 0 2024-04-22T08:14:44.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.337+0000 7f7fc5946640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:44.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.337+0000 7f7fc5946640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:44.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.337+0000 7f7fc5946640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:44.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.337+0000 7f7fc5946640 10 monclient(hunting): handle_auth_done global_id 24325 payload 824 2024-04-22T08:14:44.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.338+0000 7f7fc5946640 1 -- 172.21.15.40:0/3702734453 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7fc009fac0 con 0x7f7fc01c16e0 2024-04-22T08:14:44.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.340+0000 7f7fc5145640 1 --2- 172.21.15.40:0/3702734453 >> v2:172.21.15.189:6800/30168 conn(0x7f7fa40377d0 0x7f7fa4039c90 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc004f60 tx=0x7f7fbc007e30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:44.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.340+0000 7f7fc5946640 1 -- 172.21.15.40:0/3702734453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7fc01c16e0 msgr2=0x7f7fc011eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:44.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.340+0000 7f7fc5946640 1 --2- 172.21.15.40:0/3702734453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7fc01c16e0 0x7f7fc011eea0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7fb8037040 tx=0x7f7fb8004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:44.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.340+0000 7f7fb5ffb640 1 -- 172.21.15.40:0/3702734453 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb803f800 con 0x7f7fc01c16e0 2024-04-22T08:14:44.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.340+0000 7f7fb5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:44.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.340+0000 7f7fb5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:14:44.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.340+0000 7f7fb5ffb640 10 monclient: dump: 2024-04-22T08:14:44.352 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:44.352 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:44.352 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:44.352 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:44.352 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:44.352 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:44.352 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:44.352 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:44.353 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:44.353 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:44.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.340+0000 7f7fb5ffb640 1 -- 172.21.15.40:0/3702734453 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f7fb8043070 con 0x7f7fc01c16e0 2024-04-22T08:14:44.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.340+0000 7f7fb5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:14:44.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.340+0000 7f7fb5ffb640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:14:44.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.340+0000 7f7fb5ffb640 1 -- 172.21.15.40:0/3702734453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7fc01c16e0 msgr2=0x7f7fc011eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:44.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.340+0000 7f7fb5ffb640 1 --2- 172.21.15.40:0/3702734453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7fc01c16e0 0x7f7fc011eea0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7fb8037040 tx=0x7f7fb8004a40 comp rx=0 tx=0).stop 2024-04-22T08:14:44.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.340+0000 7f7fb5ffb640 1 -- 172.21.15.40:0/3702734453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7fc01c16e0 msgr2=0x7f7fc011eea0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:14:44.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.340+0000 7f7fb5ffb640 1 --2- 172.21.15.40:0/3702734453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7fc01c16e0 0x7f7fc011eea0 secure :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7fb8037040 tx=0x7f7fb8004a40 comp rx=0 tx=0).stop 2024-04-22T08:14:44.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.347+0000 7f7fc73a9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:14:44.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.347+0000 7f7fc73a9640 1 --2- 172.21.15.40:0/3702734453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7fc011f3e0 0x7f7fc0136c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:44.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.347+0000 7f7fc73a9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:44.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.347+0000 7f7fc73a9640 1 -- 172.21.15.40:0/3702734453 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc0121030 con 0x7f7fc011f3e0 2024-04-22T08:14:44.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.347+0000 7f7fc5946640 1 --2- 172.21.15.40:0/3702734453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7fc011f3e0 0x7f7fc0136c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:44.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.347+0000 7f7fc5946640 10 monclient: get_auth_request con 0x7f7fc011f3e0 auth_method 0 2024-04-22T08:14:44.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.347+0000 7f7fc5946640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:44.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.347+0000 7f7fc5946640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:44.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.347+0000 7f7fc5946640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:44.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.347+0000 7f7fc5946640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:44.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.347+0000 7f7fc5946640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:44.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.347+0000 7f7fc5946640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:44.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.347+0000 7f7fc5946640 10 monclient(hunting): handle_auth_done global_id 24328 payload 824 2024-04-22T08:14:44.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.347+0000 7f7fc5946640 1 -- 172.21.15.40:0/3702734453 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7fc011f7f0 con 0x7f7fc011f3e0 2024-04-22T08:14:44.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.349+0000 7f7fc5946640 1 -- 172.21.15.40:0/3702734453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7fc011f3e0 msgr2=0x7f7fc0136c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:44.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.350+0000 7f7fc5946640 1 --2- 172.21.15.40:0/3702734453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7fc011f3e0 0x7f7fc0136c50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7fb8046000 tx=0x7f7fb8031000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:44.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.350+0000 7f7fb5ffb640 1 -- 172.21.15.40:0/3702734453 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb8004040 con 0x7f7fc011f3e0 2024-04-22T08:14:44.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.350+0000 7f7fb5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:44.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.350+0000 7f7fb5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:14:44.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.350+0000 7f7fb5ffb640 10 monclient: dump: 2024-04-22T08:14:44.355 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:44.356 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:44.356 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:44.356 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:44.356 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:44.356 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:44.356 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:44.356 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:44.356 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:44.356 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:44.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.350+0000 7f7fb5ffb640 1 -- 172.21.15.40:0/3702734453 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4801 (secure 0 0 0) 0x7f7fb80041a0 con 0x7f7fc011f3e0 2024-04-22T08:14:44.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.350+0000 7f7fb5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:14:44.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.350+0000 7f7fb5ffb640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:14:44.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.350+0000 7f7fb5ffb640 1 -- 172.21.15.40:0/3702734453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7fc011f3e0 msgr2=0x7f7fc0136c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:44.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.350+0000 7f7fb5ffb640 1 --2- 172.21.15.40:0/3702734453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7fc011f3e0 0x7f7fc0136c50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7fb8046000 tx=0x7f7fb8031000 comp rx=0 tx=0).stop 2024-04-22T08:14:44.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.350+0000 7f7fb5ffb640 1 -- 172.21.15.40:0/3702734453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7fc011f3e0 msgr2=0x7f7fc0136c50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:14:44.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.350+0000 7f7fb5ffb640 1 --2- 172.21.15.40:0/3702734453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7fc011f3e0 0x7f7fc0136c50 secure :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7fb8046000 tx=0x7f7fb8031000 comp rx=0 tx=0).stop 2024-04-22T08:14:44.357 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:14:44.357 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:14:44.357 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:14:44.357 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:14:44.357 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 26, 2024-04-22T08:14:44.357 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:14:44.357 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:14:44.357 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:14:44.357 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:14:44.357 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:14:44.358 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 1, 2024-04-22T08:14:44.358 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:14:44.358 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:14:44.358 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:14:44.358 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:14:44.358 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:14:44.358 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:14:44.358 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:14:44.358 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:14:44.358 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:14:44.358 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:14:44.358 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:14:44.358 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:14:44.358 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:14:44.359 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:14:44.359 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:14:44.359 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:14:44.359 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:14:44.359 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:14:44.359 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:14:44.359 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:14:44.359 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:14:44.359 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:14:44.359 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:14:44.359 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:14:44.359 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:14:44.359 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:14:44.359 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:44.360 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:44.360 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:14:44.360 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:14:44.360 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:14:44.360 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:14:44.360 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:14:44.360 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:14:44.360 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:14:44.360 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:14:44.360 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:14:44.360 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:14:44.360 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:14:44.360 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:14:44.360 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:14:44.360 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:14:44.361 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:14:44.361 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:14:44.361 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:14:44.361 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:14:44.361 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:14:44.361 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:14:44.361 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:14:44.361 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:14:44.361 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:14:44.361 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:14:44.361 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:14:44.361 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:14:44.361 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:14:44.361 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:14:44.362 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:14:44.362 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:44.362 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:14:44.362 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:44.362 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:14:44.362 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:14:44.362 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:14:44.362 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:14:44.362 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:44.362 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:14:44.362 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:14:44.362 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:44.362 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:44.362 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:44.363 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:14:44.363 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:14:44.363 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:44.363 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:44.363 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:44.363 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:44.363 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:14:44.363 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:14:44.363 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:14:44.363 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:14:44.363 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:14:44.363 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:44.363 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:44.363 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:14:44.363 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:14:44.364 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:14:44.364 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:14:44.364 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:44.364 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:14:44.364 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:14:44.364 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:44.364 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:44.364 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:44.364 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:14:44.364 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:14:44.364 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:44.364 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:44.364 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:44.364 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:44.365 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:14:44.365 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:14:44.365 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:14:44.365 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:14:44.365 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:14:44.365 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:44.365 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:44.365 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:14:44.365 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:14:44.365 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:14:44.365 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:14:44.365 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:44.365 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:14:44.365 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:14:44.365 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:44.366 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:44.366 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:44.366 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:14:44.366 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:14:44.366 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:44.366 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:44.366 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:44.366 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:44.366 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:14:44.366 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:14:44.366 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:14:44.366 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:14:44.366 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:14:44.366 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:44.367 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:44.367 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:44.367 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:14:44.367 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:14:44.367 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:44.367 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:14:44.367 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:14:44.367 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:14:44.367 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:44.367 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:14:44.367 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:14:44.367 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:44.367 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:14:44.367 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:14:44.368 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:14:44.368 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:44.368 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:14:44.368 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:14:44.368 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:44.368 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:14:44.368 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:14:44.368 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 5 2024-04-22T08:14:44.368 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:44.368 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:14:44.368 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": [ 2024-04-22T08:14:44.368 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:44.368 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:14:44.368 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:14:44.368 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:14:44.369 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:44.369 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:44.369 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:44.369 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:14:44.369 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:14:44.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.350+0000 7f7fc73a9640 1 -- 172.21.15.40:0/3702734453 >> v2:172.21.15.189:6800/30168 conn(0x7f7fa40377d0 msgr2=0x7f7fa4039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:44.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.350+0000 7f7fc73a9640 1 --2- 172.21.15.40:0/3702734453 >> v2:172.21.15.189:6800/30168 conn(0x7f7fa40377d0 0x7f7fa4039c90 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc004f60 tx=0x7f7fbc007e30 comp rx=0 tx=0).stop 2024-04-22T08:14:44.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.350+0000 7f7fc73a9640 10 monclient: shutdown 2024-04-22T08:14:44.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.350+0000 7f7fc73a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:14:44.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.350+0000 7f7fc73a9640 1 -- 172.21.15.40:0/3702734453 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7fc00a31d0 msgr2=0x7f7fc0135a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:44.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.350+0000 7f7fc5946640 1 -- 172.21.15.40:0/3702734453 reap_dead start 2024-04-22T08:14:44.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.350+0000 7f7fc73a9640 1 --2- 172.21.15.40:0/3702734453 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7fc00a31d0 0x7f7fc0135a10 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7fb000c3e0 tx=0x7f7fb000c8a0 comp rx=0 tx=0).stop 2024-04-22T08:14:44.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.350+0000 7f7fc73a9640 1 -- 172.21.15.40:0/3702734453 shutdown_connections 2024-04-22T08:14:44.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.350+0000 7f7fc73a9640 1 --2- 172.21.15.40:0/3702734453 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7fc00a31d0 0x7f7fc0135a10 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:44.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.350+0000 7f7fc73a9640 1 -- 172.21.15.40:0/3702734453 >> 172.21.15.40:0/3702734453 conn(0x7f7fc00a89a0 msgr2=0x7f7fc006e580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:14:44.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.351+0000 7f7fc73a9640 1 -- 172.21.15.40:0/3702734453 shutdown_connections 2024-04-22T08:14:44.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.351+0000 7f7fc73a9640 1 -- 172.21.15.40:0/3702734453 wait complete. 2024-04-22T08:14:44.373 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:14:44.373 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-22T08:14:44.373 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-22T08:14:44.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.473+0000 7f6fd6ee0640 10 monclient: get_monmap_and_config 2024-04-22T08:14:44.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.473+0000 7f6fd6ee0640 10 monclient: build_initial_monmap 2024-04-22T08:14:44.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.473+0000 7f6fd6ee0640 10 monclient: monmap: 2024-04-22T08:14:44.473 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:14:44.473 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:44.473 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:14:44.473848+0000 2024-04-22T08:14:44.473 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:14:44.473848+0000 2024-04-22T08:14:44.473 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:14:44.473 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:44.473 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:14:44.473 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:14:44.474 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:14:44.474 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:14:44.474 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:14:44.474 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:44.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.473+0000 7f6fd6ee0640 1 Processor -- start 2024-04-22T08:14:44.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.473+0000 7f6fd6ee0640 1 -- start start 2024-04-22T08:14:44.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.473+0000 7f6fd6ee0640 10 monclient: init 2024-04-22T08:14:44.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.474+0000 7f6fd6ee0640 10 monclient: _reopen_session rank -1 2024-04-22T08:14:44.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.474+0000 7f6fd6ee0640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-04-22T08:14:44.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.474+0000 7f6fd6ee0640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6fd009ee10 0x7f6fd00a3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:44.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.474+0000 7f6fd6ee0640 10 monclient(hunting): picked mon.noname-c con 0x7f6fd009ee10 addr v2:172.21.15.189:3300/0 2024-04-22T08:14:44.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.474+0000 7f6fd6ee0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6fd00a1820 addr v1:172.21.15.189:6789/0 2024-04-22T08:14:44.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.474+0000 7f6fd6ee0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6fd013b0e0 addr v1:172.21.15.40:6789/0 2024-04-22T08:14:44.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.474+0000 7f6fd6ee0640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd00a4470 con 0x7f6fd013b0e0 2024-04-22T08:14:44.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.474+0000 7f6fd6ee0640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd009f1f0 con 0x7f6fd00a1820 2024-04-22T08:14:44.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.474+0000 7f6fd6ee0640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:44.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.474+0000 7f6fd6ee0640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fd009fd30 con 0x7f6fd009ee10 2024-04-22T08:14:44.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.474+0000 7f6fd6ee0640 10 monclient(hunting): _renew_subs 2024-04-22T08:14:44.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.474+0000 7f6fd6ee0640 10 monclient(hunting): authenticate will time out at 958.468100s 2024-04-22T08:14:44.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.474+0000 7f6fd547d640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f6fd013b0e0 0x7f6fd013d4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:58976/0 (socket says 172.21.15.40:58976) 2024-04-22T08:14:44.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.474+0000 7f6fd547d640 1 -- 172.21.15.40:0/2420967932 learned_addr learned my addr 172.21.15.40:0/2420967932 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:14:44.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.474+0000 7f6fd4c7c640 1 --2- 172.21.15.40:0/2420967932 >> v2:172.21.15.189:3300/0 conn(0x7f6fd009ee10 0x7f6fd00a3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:44.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.474+0000 7f6fd4c7c640 10 monclient(hunting): get_auth_request con 0x7f6fd009ee10 auth_method 0 2024-04-22T08:14:44.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.474+0000 7f6fd4c7c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:44.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.474+0000 7f6fd4c7c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:44.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.474+0000 7f6fd4c7c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:44.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.474+0000 7f6fd4c7c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:44.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.474+0000 7f6fd4c7c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:44.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.474+0000 7f6fd4c7c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:44.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.476+0000 7f6fd547d640 1 -- 172.21.15.40:0/2420967932 >> v1:172.21.15.40:6789/0 conn(0x7f6fd013b0e0 legacy=0x7f6fd013d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:44.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.476+0000 7f6fcf7fe640 1 -- 172.21.15.40:0/2420967932 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3211315238 0 0) 0x7f6fd00a4470 con 0x7f6fd013b0e0 2024-04-22T08:14:44.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.476+0000 7f6fcf7fe640 10 monclient(hunting): _negotiate 2024-04-22T08:14:44.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.476+0000 7f6fcf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:44.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.476+0000 7f6fcf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:44.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.476+0000 7f6fcf7fe640 10 monclient(hunting): my global_id is 34122 2024-04-22T08:14:44.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.476+0000 7f6fcf7fe640 1 -- 172.21.15.40:0/2420967932 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fbc003520 con 0x7f6fd013b0e0 2024-04-22T08:14:44.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.476+0000 7f6fcffff640 1 -- 172.21.15.40:0/2420967932 >> v1:172.21.15.189:6789/0 conn(0x7f6fd00a1820 legacy=0x7f6fd00a1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:44.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.477+0000 7f6fd4c7c640 10 monclient(hunting): handle_auth_done global_id 24331 payload 274 2024-04-22T08:14:44.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.477+0000 7f6fd4c7c640 1 -- 172.21.15.40:0/2420967932 >> v1:172.21.15.189:6789/0 conn(0x7f6fd00a1820 legacy=0x7f6fd00a1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:44.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.477+0000 7f6fcf7fe640 1 -- 172.21.15.40:0/2420967932 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1279744992 0 0) 0x7f6fbc003520 con 0x7f6fd013b0e0 2024-04-22T08:14:44.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.477+0000 7f6fd4c7c640 1 -- 172.21.15.40:0/2420967932 >> v1:172.21.15.40:6789/0 conn(0x7f6fd013b0e0 legacy=0x7f6fd013d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:44.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.477+0000 7f6fd4c7c640 10 monclient: _finish_hunting 0 2024-04-22T08:14:44.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.477+0000 7f6fd4c7c640 1 monclient: found mon.noname-c 2024-04-22T08:14:44.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.477+0000 7f6fd4c7c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:14:44.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.477+0000 7f6fd4c7c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.189:3300/0 2024-04-22T08:14:44.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.477+0000 7f6fd4c7c640 1 -- 172.21.15.40:0/2420967932 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd00a0010 con 0x7f6fd009ee10 2024-04-22T08:14:44.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.477+0000 7f6fcf7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-04-22T08:14:44.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.480+0000 7f6fd4c7c640 1 -- 172.21.15.40:0/2420967932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6fd009ee10 msgr2=0x7f6fd00a3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:44.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.480+0000 7f6fd4c7c640 1 --2- 172.21.15.40:0/2420967932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6fd009ee10 0x7f6fd00a3240 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8009e90 tx=0x7f6fc802bde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d3e09c732f049e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:44.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.480+0000 7f6fcf7fe640 1 -- 172.21.15.40:0/2420967932 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc00019e0 con 0x7f6fd009ee10 2024-04-22T08:14:44.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.480+0000 7f6fcf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:44.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.480+0000 7f6fcf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:14:44.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.480+0000 7f6fcf7fe640 10 monclient: dump: 2024-04-22T08:14:44.488 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:44.488 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:44.489 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:44.489 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:44.489 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:44.489 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:44.489 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:44.489 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:44.489 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:44.489 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:44.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.480+0000 7f6fcf7fe640 10 monclient: _finish_auth 0 2024-04-22T08:14:44.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.480+0000 7f6fcf7fe640 10 monclient: _check_auth_tickets 2024-04-22T08:14:44.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.480+0000 7f6fcf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:14:44.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.480+0000 7f6fcf7fe640 1 -- 172.21.15.40:0/2420967932 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fc802cb20 con 0x7f6fd009ee10 2024-04-22T08:14:44.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.480+0000 7f6fcf7fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:14:44.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.480+0000 7f6fcf7fe640 1 -- 172.21.15.40:0/2420967932 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc8034440 con 0x7f6fd009ee10 2024-04-22T08:14:44.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.480+0000 7f6fcf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:44.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.480+0000 7f6fcf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:14:44.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.480+0000 7f6fcf7fe640 10 monclient: dump: 2024-04-22T08:14:44.490 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:44.490 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:44.490 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:44.490 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:44.490 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:44.490 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:44.490 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:44.490 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:44.490 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:44.490 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:44.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.480+0000 7f6fd6ee0640 5 monclient: authenticate success, global_id 24331 2024-04-22T08:14:44.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.480+0000 7f6fd6ee0640 10 monclient: get_monmap_and_config success 2024-04-22T08:14:44.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.480+0000 7f6fd6ee0640 4 set_mon_vals no callback set 2024-04-22T08:14:44.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.481+0000 7f6fd6ee0640 10 monclient: shutdown 2024-04-22T08:14:44.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.481+0000 7f6fd6ee0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:14:44.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.481+0000 7f6fd6ee0640 1 -- 172.21.15.40:0/2420967932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6fd009ee10 msgr2=0x7f6fd00a3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:44.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.481+0000 7f6fd6ee0640 1 --2- 172.21.15.40:0/2420967932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6fd009ee10 0x7f6fd00a3240 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8009e90 tx=0x7f6fc802bde0 comp rx=0 tx=0).stop 2024-04-22T08:14:44.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.481+0000 7f6fd6ee0640 1 -- 172.21.15.40:0/2420967932 shutdown_connections 2024-04-22T08:14:44.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.481+0000 7f6fd6ee0640 1 --2- 172.21.15.40:0/2420967932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6fd009ee10 0x7f6fd00a3240 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:44.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.481+0000 7f6fd6ee0640 1 -- 172.21.15.40:0/2420967932 >> 172.21.15.40:0/2420967932 conn(0x7f6fd00a4e80 msgr2=0x7f6fd00a5290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:14:44.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.481+0000 7f6fd6ee0640 1 -- 172.21.15.40:0/2420967932 shutdown_connections 2024-04-22T08:14:44.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.481+0000 7f6fd6ee0640 1 -- 172.21.15.40:0/2420967932 wait complete. 2024-04-22T08:14:44.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.481+0000 7f6fd6ee0640 10 monclient: build_initial_monmap 2024-04-22T08:14:44.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.481+0000 7f6fd6ee0640 10 monclient: monmap: 2024-04-22T08:14:44.491 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:14:44.491 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:14:44.491 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:14:44.492 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:14:44.492 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:14:44.492 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:44.492 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:14:44.492 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:14:44.492 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:14:44.492 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:44.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.482+0000 7f6fd6ee0640 1 Processor -- start 2024-04-22T08:14:44.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.482+0000 7f6fd6ee0640 1 -- start start 2024-04-22T08:14:44.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.482+0000 7f6fd6ee0640 10 monclient: init 2024-04-22T08:14:44.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.482+0000 7f6fd6ee0640 10 monclient: _reopen_session rank -1 2024-04-22T08:14:44.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.482+0000 7f6fd6ee0640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-22T08:14:44.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.482+0000 7f6fd6ee0640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6fd009ee10 0x7f6fd01cadf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:44.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.482+0000 7f6fd6ee0640 10 monclient(hunting): picked mon.noname-c con 0x7f6fd009ee10 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:14:44.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.482+0000 7f6fd6ee0640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fd00a1820 0x7f6fd01cb330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:44.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.482+0000 7f6fd6ee0640 10 monclient(hunting): picked mon.noname-a con 0x7f6fd00a1820 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:14:44.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.482+0000 7f6fd6ee0640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6fd013b0e0 0x7f6fd006f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:44.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.482+0000 7f6fd6ee0640 10 monclient(hunting): picked mon.noname-b con 0x7f6fd013b0e0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:14:44.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.482+0000 7f6fd6ee0640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:44.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.482+0000 7f6fd6ee0640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd0071190 con 0x7f6fd00a1820 2024-04-22T08:14:44.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.482+0000 7f6fd6ee0640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:44.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.482+0000 7f6fd6ee0640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd00a4260 con 0x7f6fd013b0e0 2024-04-22T08:14:44.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.482+0000 7f6fd6ee0640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:44.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.482+0000 7f6fd6ee0640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd0130410 con 0x7f6fd009ee10 2024-04-22T08:14:44.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.482+0000 7f6fd6ee0640 10 monclient(hunting): _renew_subs 2024-04-22T08:14:44.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.482+0000 7f6fd6ee0640 10 monclient(hunting): authenticate will time out at 958.476492s 2024-04-22T08:14:44.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.482+0000 7f6fd4c7c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6fd009ee10 0x7f6fd01cadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:44.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.482+0000 7f6fcffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fd00a1820 0x7f6fd01cb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:44.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.482+0000 7f6fd4c7c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6fd009ee10 0x7f6fd01cadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49544/0 (socket says 172.21.15.40:49544) 2024-04-22T08:14:44.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.482+0000 7f6fd4c7c640 1 -- 172.21.15.40:0/4184637026 learned_addr learned my addr 172.21.15.40:0/4184637026 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:14:44.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.482+0000 7f6fd4c7c640 10 monclient(hunting): get_auth_request con 0x7f6fd009ee10 auth_method 0 2024-04-22T08:14:44.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.482+0000 7f6fd4c7c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:44.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.482+0000 7f6fd4c7c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:44.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.482+0000 7f6fd4c7c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:44.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.482+0000 7f6fcffff640 10 monclient(hunting): get_auth_request con 0x7f6fd00a1820 auth_method 0 2024-04-22T08:14:44.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.482+0000 7f6fcffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:44.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.482+0000 7f6fcffff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:44.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.482+0000 7f6fcffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:44.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.482+0000 7f6fd4c7c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:44.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.482+0000 7f6fd547d640 1 --2- 172.21.15.40:0/4184637026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6fd013b0e0 0x7f6fd006f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:44.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.482+0000 7f6fd4c7c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:44.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.482+0000 7f6fd4c7c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:44.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.482+0000 7f6fcffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:44.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.482+0000 7f6fcffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:44.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.482+0000 7f6fcffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:44.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.482+0000 7f6fd547d640 10 monclient(hunting): get_auth_request con 0x7f6fd013b0e0 auth_method 0 2024-04-22T08:14:44.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.482+0000 7f6fd547d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:44.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.482+0000 7f6fd547d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:44.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.482+0000 7f6fd547d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:44.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.483+0000 7f6fd547d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:44.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.483+0000 7f6fd547d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:44.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.483+0000 7f6fd547d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:44.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.483+0000 7f6fd4c7c640 10 monclient(hunting): handle_auth_done global_id 4742 payload 824 2024-04-22T08:14:44.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.483+0000 7f6fd4c7c640 1 -- 172.21.15.40:0/4184637026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6fd013b0e0 msgr2=0x7f6fd006f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:44.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.483+0000 7f6fd4c7c640 1 --2- 172.21.15.40:0/4184637026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6fd013b0e0 0x7f6fd006f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:44.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.483+0000 7f6fd4c7c640 1 -- 172.21.15.40:0/4184637026 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fd00a1820 msgr2=0x7f6fd01cb330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:44.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.483+0000 7f6fd4c7c640 1 --2- 172.21.15.40:0/4184637026 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fd00a1820 0x7f6fd01cb330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:44.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.483+0000 7f6fd4c7c640 10 monclient: _finish_hunting 0 2024-04-22T08:14:44.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.483+0000 7f6fd4c7c640 1 monclient: found mon.noname-c 2024-04-22T08:14:44.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.483+0000 7f6fd4c7c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:14:44.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.483+0000 7f6fd4c7c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:14:44.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.483+0000 7f6fd4c7c640 1 -- 172.21.15.40:0/4184637026 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc8009a70 con 0x7f6fd009ee10 2024-04-22T08:14:44.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.483+0000 7f6fcffff640 1 --2- 172.21.15.40:0/4184637026 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fd00a1820 0x7f6fd01cb330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:14:44.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.485+0000 7f6fd4c7c640 1 -- 172.21.15.40:0/4184637026 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6fd009ee10 msgr2=0x7f6fd01cadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:44.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.485+0000 7f6fd4c7c640 1 --2- 172.21.15.40:0/4184637026 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6fd009ee10 0x7f6fd01cadf0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8009e90 tx=0x7f6fc8004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:44.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.485+0000 7f6fcd7fa640 1 -- 172.21.15.40:0/4184637026 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc803f630 con 0x7f6fd009ee10 2024-04-22T08:14:44.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.485+0000 7f6fcd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:44.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.485+0000 7f6fcd7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:14:44.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.485+0000 7f6fcd7fa640 10 monclient: dump: 2024-04-22T08:14:44.497 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:44.497 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:44.497 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:44.497 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:44.497 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:44.497 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:44.497 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:44.497 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:44.497 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:44.497 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:44.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.485+0000 7f6fcd7fa640 10 monclient: _finish_auth 0 2024-04-22T08:14:44.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.485+0000 7f6fcd7fa640 10 monclient: _check_auth_tickets 2024-04-22T08:14:44.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.485+0000 7f6fcd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:14:44.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.485+0000 7f6fd6ee0640 5 monclient: authenticate success, global_id 4742 2024-04-22T08:14:44.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.485+0000 7f6fcd7fa640 1 -- 172.21.15.40:0/4184637026 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fc803f790 con 0x7f6fd009ee10 2024-04-22T08:14:44.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.485+0000 7f6fcd7fa640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:14:44.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.485+0000 7f6fd6ee0640 10 monclient: _renew_subs 2024-04-22T08:14:44.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.485+0000 7f6fcd7fa640 1 -- 172.21.15.40:0/4184637026 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc803fb20 con 0x7f6fd009ee10 2024-04-22T08:14:44.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.485+0000 7f6fd6ee0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:14:44.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.485+0000 7f6fd6ee0640 1 -- 172.21.15.40:0/4184637026 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fd00713d0 con 0x7f6fd009ee10 2024-04-22T08:14:44.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.485+0000 7f6fceffd640 4 set_mon_vals no callback set 2024-04-22T08:14:44.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.485+0000 7f6fcd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:44.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.485+0000 7f6fcd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:14:44.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.485+0000 7f6fcd7fa640 10 monclient: dump: 2024-04-22T08:14:44.498 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:44.498 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:44.499 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:44.499 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:44.499 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:44.499 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:44.499 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:44.499 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:44.499 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:44.499 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:44.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.485+0000 7f6fd6ee0640 10 monclient: _renew_subs 2024-04-22T08:14:44.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.485+0000 7f6fd6ee0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:14:44.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.485+0000 7f6fd6ee0640 1 -- 172.21.15.40:0/4184637026 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fd0070c80 con 0x7f6fd009ee10 2024-04-22T08:14:44.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.486+0000 7f6fcd7fa640 1 -- 172.21.15.40:0/4184637026 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f6fc803fc80 con 0x7f6fd009ee10 2024-04-22T08:14:44.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.486+0000 7f6fd6ee0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:14:44.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.486+0000 7f6fd6ee0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:14:44.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.486+0000 7f6fd6ee0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:14:44.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.486+0000 7f6fd6ee0640 1 -- 172.21.15.40:0/4184637026 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fd0065bd0 con 0x7f6fd009ee10 2024-04-22T08:14:44.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.486+0000 7f6fcd7fa640 1 --2- 172.21.15.40:0/4184637026 >> v2:172.21.15.189:6800/30168 conn(0x7f6fb00377d0 0x7f6fb0039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:44.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.486+0000 7f6fcd7fa640 1 -- 172.21.15.40:0/4184637026 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f6fc806d270 con 0x7f6fd009ee10 2024-04-22T08:14:44.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.487+0000 7f6fcffff640 1 --2- 172.21.15.40:0/4184637026 >> v2:172.21.15.189:6800/30168 conn(0x7f6fb00377d0 0x7f6fb0039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:44.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.487+0000 7f6fcffff640 10 monclient: get_auth_request con 0x7f6fb00377d0 auth_method 0 2024-04-22T08:14:44.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.489+0000 7f6fcffff640 1 --2- 172.21.15.40:0/4184637026 >> v2:172.21.15.189:6800/30168 conn(0x7f6fb00377d0 0x7f6fb0039c90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6fd00717e0 tx=0x7f6fb8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:44.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.491+0000 7f6fcd7fa640 1 -- 172.21.15.40:0/4184637026 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f6fc80473c0 con 0x7f6fd009ee10 2024-04-22T08:14:44.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.491+0000 7f6fcd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:14:44.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.491+0000 7f6fcd7fa640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:14:44.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.742+0000 7f6fd6ee0640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-22T08:14:44.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.742+0000 7f6fd6ee0640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-22T08:14:44.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.742+0000 7f6fd6ee0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:14:44.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.742+0000 7f6fd6ee0640 1 -- 172.21.15.40:0/4184637026 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f6fd004e160 con 0x7f6fd009ee10 2024-04-22T08:14:44.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.744+0000 7f6fcd7fa640 1 -- 172.21.15.40:0/4184637026 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f6fc804d080 con 0x7f6fd009ee10 2024-04-22T08:14:44.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.744+0000 7f6fcd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-22T08:14:44.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.744+0000 7f6fcd7fa640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:14:44.744 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:14:44.744+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:14:44.745 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:14:44.744+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=8,config=2,health=11,kv=9,logm=70} crc {auth=2585056491,config=1618278320,health=1211899592,kv=3730709594,logm=182054406}) 2024-04-22T08:14:44.745 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:14:44.744+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=9,config=2,health=10,kv=9,logm=70} crc {auth=3831754339,config=1618278320,health=103516700,kv=2184066631,logm=855090864}) 2024-04-22T08:14:44.745 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:14:44.744+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:14:44.745 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:14:44.744+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=8,config=2,health=11,kv=9,logm=70} crc {auth=2585056491,config=1618278320,health=1211899592,kv=3730709594,logm=182054406}) 2024-04-22T08:14:44.745 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:14:44.744+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=9,config=2,health=12,kv=8,logm=69} crc {auth=3600112391,config=1618278320,health=4205301480,kv=3809403368,logm=170525780}) 2024-04-22T08:14:44.745 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:14:44.745+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:14:44.745 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:14:44.745+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1814278670}) 2024-04-22T08:14:44.745 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:14:44.745+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2987309924}) 2024-04-22T08:14:44.745 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:14:44.745+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:14:44.745 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:14:44.745+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1814278670}) 2024-04-22T08:14:44.746 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:14:44.745+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2055032450}) 2024-04-22T08:14:44.746 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:14:44.746+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:14:44.746 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:14:44.746+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1244546591}) 2024-04-22T08:14:44.747 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:14:44.746+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=353771738}) 2024-04-22T08:14:44.747 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:14:44.746+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:14:44.747 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:14:44.746+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1244546591}) 2024-04-22T08:14:44.747 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:14:44.746+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=928343000}) 2024-04-22T08:14:44.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.746+0000 7f6fd6ee0640 1 -- 172.21.15.40:0/4184637026 >> v2:172.21.15.189:6800/30168 conn(0x7f6fb00377d0 msgr2=0x7f6fb0039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:44.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.746+0000 7f6fd6ee0640 1 --2- 172.21.15.40:0/4184637026 >> v2:172.21.15.189:6800/30168 conn(0x7f6fb00377d0 0x7f6fb0039c90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6fd00717e0 tx=0x7f6fb8009040 comp rx=0 tx=0).stop 2024-04-22T08:14:44.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.746+0000 7f6fd6ee0640 10 monclient: shutdown 2024-04-22T08:14:44.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.746+0000 7f6fd6ee0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:14:44.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.746+0000 7f6fd6ee0640 1 -- 172.21.15.40:0/4184637026 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6fd009ee10 msgr2=0x7f6fd01cadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:44.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.746+0000 7f6fd6ee0640 1 --2- 172.21.15.40:0/4184637026 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6fd009ee10 0x7f6fd01cadf0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8009e90 tx=0x7f6fc8004a40 comp rx=0 tx=0).stop 2024-04-22T08:14:44.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.746+0000 7f6fd6ee0640 1 -- 172.21.15.40:0/4184637026 shutdown_connections 2024-04-22T08:14:44.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.746+0000 7f6fd6ee0640 1 --2- 172.21.15.40:0/4184637026 >> v2:172.21.15.189:6800/30168 conn(0x7f6fb00377d0 0x7f6fb0039c90 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:44.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.746+0000 7f6fd6ee0640 1 --2- 172.21.15.40:0/4184637026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6fd013b0e0 0x7f6fd006f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:44.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.746+0000 7f6fd6ee0640 1 --2- 172.21.15.40:0/4184637026 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fd00a1820 0x7f6fd01cb330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:44.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.746+0000 7f6fd6ee0640 1 --2- 172.21.15.40:0/4184637026 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6fd009ee10 0x7f6fd01cadf0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:44.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.746+0000 7f6fd6ee0640 1 -- 172.21.15.40:0/4184637026 >> 172.21.15.40:0/4184637026 conn(0x7f6fd00a4e80 msgr2=0x7f6fd00a6180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:14:44.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.746+0000 7f6fd6ee0640 1 -- 172.21.15.40:0/4184637026 shutdown_connections 2024-04-22T08:14:44.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:44.746+0000 7f6fd6ee0640 1 -- 172.21.15.40:0/4184637026 wait complete. 2024-04-22T08:14:44.751 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:14:44.747+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:14:44.752 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:14:44.747+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2486061713}) 2024-04-22T08:14:44.752 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:14:44.747+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3362525381}) 2024-04-22T08:14:44.752 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:14:44.747+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:14:44.752 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:14:44.747+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2486061713}) 2024-04-22T08:14:44.752 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:14:44.747+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2318093733}) 2024-04-22T08:14:44.752 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:14:44.747+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:14:44.752 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:14:44.747+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1868562959}) 2024-04-22T08:14:44.752 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:14:44.747+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2369494983}) 2024-04-22T08:14:44.752 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:14:44.747+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:14:44.752 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:14:44.747+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1868562959}) 2024-04-22T08:14:44.752 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:14:44.747+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1397379373}) 2024-04-22T08:14:44.752 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:14:44.748+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:14:44.752 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:14:44.748+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=62,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=8,monmap=3,osd_metadata=5} crc {logm=3385122432,mdsmap=1531151339,mgr=310612464,mgr_command_descs=470045235,mgr_metadata=4222303120,mgrstat=540170348,mon_config_key=4255067121,monmap=3469086636,osd_metadata=3591713508}) 2024-04-22T08:14:44.753 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:14:44.748+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=59,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=8,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=3} crc {logm=3161726889,mdsmap=1531151339,mgr=1219260174,mgr_command_descs=470045235,mgr_metadata=4222303120,mgrstat=2047834581,mon_config_key=3293354343,monmap=3469086636,osd_metadata=4151711904,osd_pg_creating=3689313388,osdmap=3932689550}) 2024-04-22T08:14:44.753 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:14:44.748+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:14:44.753 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:14:44.748+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=62,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=8,monmap=3,osd_metadata=5} crc {logm=3385122432,mdsmap=1531151339,mgr=310612464,mgr_command_descs=470045235,mgr_metadata=4222303120,mgrstat=540170348,mon_config_key=4255067121,monmap=3469086636,osd_metadata=3591713508}) 2024-04-22T08:14:44.753 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:14:44.748+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=62,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=8,monmap=3,osd_metadata=6,osd_pg_creating=1} crc {logm=849354142,mdsmap=1531151339,mgr=36904295,mgr_command_descs=470045235,mgr_metadata=4222303120,mgrstat=620460427,mon_config_key=3037593030,monmap=3469086636,osd_metadata=4151711904,osd_pg_creating=3689313388}) 2024-04-22T08:14:44.753 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:14:44.749+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:14:44.753 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:14:44.749+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=1,osd_pg_creating=1,osdmap=98} crc {osd_metadata=4277766693,osd_pg_creating=3689313388,osdmap=2464700246}) 2024-04-22T08:14:44.753 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:14:44.749+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_metadata=1,osd_pg_creating=1,osdmap=98} crc {osd_metadata=4277766693,osd_pg_creating=3689313388,osdmap=2278238775}) 2024-04-22T08:14:44.753 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:14:44.749+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:14:44.753 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:14:44.749+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=1,osd_pg_creating=1,osdmap=98} crc {osd_metadata=4277766693,osd_pg_creating=3689313388,osdmap=2464700246}) 2024-04-22T08:14:44.753 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:14:44.749+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_metadata=1,osd_pg_creating=1,osdmap=98} crc {osd_metadata=4277766693,osd_pg_creating=3689313388,osdmap=1914032038}) 2024-04-22T08:14:44.753 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:14:44.751+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:14:44.753 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:14:44.751+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=19} crc {osdmap=4163738363}) 2024-04-22T08:14:44.753 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:14:44.751+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=18} crc {osdmap=1265699825}) 2024-04-22T08:14:44.757 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-22T08:14:45.196 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[35383]: starting. 2024-04-22T08:14:45.196 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[35383]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-35381 2024-04-22T08:14:45.196 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[35383]: shutting down. 2024-04-22T08:14:45.758 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-22T08:14:45.759 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:14:45.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.869+0000 7f9a3f27d640 10 monclient: get_monmap_and_config 2024-04-22T08:14:45.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.869+0000 7f9a3f27d640 10 monclient: build_initial_monmap 2024-04-22T08:14:45.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.869+0000 7f9a3f27d640 10 monclient: monmap: 2024-04-22T08:14:45.869 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:14:45.869 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:45.869 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:14:45.870721+0000 2024-04-22T08:14:45.870 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:14:45.870721+0000 2024-04-22T08:14:45.870 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:14:45.870 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:45.870 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:14:45.870 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:14:45.870 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:14:45.870 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:14:45.870 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:14:45.870 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:45.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.870+0000 7f9a3f27d640 1 Processor -- start 2024-04-22T08:14:45.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.870+0000 7f9a3f27d640 1 -- start start 2024-04-22T08:14:45.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.870+0000 7f9a3f27d640 10 monclient: init 2024-04-22T08:14:45.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.871+0000 7f9a3f27d640 10 monclient: _reopen_session rank -1 2024-04-22T08:14:45.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.871+0000 7f9a3f27d640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-04-22T08:14:45.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.871+0000 7f9a3f27d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9a380a20d0 0x7f9a380a24b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:45.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.871+0000 7f9a3f27d640 10 monclient(hunting): picked mon.noname-b con 0x7f9a380a20d0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:14:45.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.871+0000 7f9a3f27d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9a380a2d10 addr v1:172.21.15.189:6789/0 2024-04-22T08:14:45.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.871+0000 7f9a3f27d640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9a380a38a0 0x7f9a3813db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:45.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.871+0000 7f9a3f27d640 10 monclient(hunting): picked mon.noname-c con 0x7f9a380a38a0 addr v2:172.21.15.189:3300/0 2024-04-22T08:14:45.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.871+0000 7f9a3f27d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a380a45f0 con 0x7f9a380a2d10 2024-04-22T08:14:45.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.871+0000 7f9a3f27d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:45.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.871+0000 7f9a3f27d640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a380a0360 con 0x7f9a380a38a0 2024-04-22T08:14:45.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.871+0000 7f9a3f27d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:45.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.871+0000 7f9a3f27d640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a380a0640 con 0x7f9a380a20d0 2024-04-22T08:14:45.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.871+0000 7f9a3f27d640 10 monclient(hunting): _renew_subs 2024-04-22T08:14:45.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.871+0000 7f9a3f27d640 10 monclient(hunting): authenticate will time out at 959.865039s 2024-04-22T08:14:45.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.871+0000 7f9a3d019640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9a380a20d0 0x7f9a380a24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:45.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.871+0000 7f9a3d019640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9a380a20d0 0x7f9a380a24b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49556/0 (socket says 172.21.15.40:49556) 2024-04-22T08:14:45.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.871+0000 7f9a3d019640 1 -- 172.21.15.40:0/3228402463 learned_addr learned my addr 172.21.15.40:0/3228402463 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:14:45.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.871+0000 7f9a3d019640 10 monclient(hunting): get_auth_request con 0x7f9a380a20d0 auth_method 0 2024-04-22T08:14:45.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.871+0000 7f9a3d019640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:45.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.871+0000 7f9a3d019640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:45.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.871+0000 7f9a3d019640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:45.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.871+0000 7f9a3d019640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:45.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.871+0000 7f9a3d019640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:45.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.871+0000 7f9a3d019640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:45.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.871+0000 7f9a3d019640 10 monclient(hunting): handle_auth_done global_id 4760 payload 274 2024-04-22T08:14:45.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.871+0000 7f9a3d81a640 1 --2- 172.21.15.40:0/3228402463 >> v2:172.21.15.189:3300/0 conn(0x7f9a380a38a0 0x7f9a3813db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:45.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.871+0000 7f9a3d019640 1 -- 172.21.15.40:0/3228402463 >> v2:172.21.15.189:3300/0 conn(0x7f9a380a38a0 msgr2=0x7f9a3813db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:14:45.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.871+0000 7f9a3d019640 1 --2- 172.21.15.40:0/3228402463 >> v2:172.21.15.189:3300/0 conn(0x7f9a380a38a0 0x7f9a3813db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:45.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.871+0000 7f9a3d019640 1 -- 172.21.15.40:0/3228402463 >> v1:172.21.15.189:6789/0 conn(0x7f9a380a2d10 legacy=0x7f9a380a3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:14:45.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.871+0000 7f9a3d019640 10 monclient: _finish_hunting 0 2024-04-22T08:14:45.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.871+0000 7f9a3d019640 1 monclient: found mon.noname-b 2024-04-22T08:14:45.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.871+0000 7f9a3d019640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:14:45.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.871+0000 7f9a3d019640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:14:45.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.871+0000 7f9a3d019640 1 -- 172.21.15.40:0/3228402463 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a380a0920 con 0x7f9a380a20d0 2024-04-22T08:14:45.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.871+0000 7f9a3d81a640 10 monclient: get_auth_request con 0x7f9a380a38a0 auth_method 0 2024-04-22T08:14:45.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.871+0000 7f9a3d81a640 1 --2- 172.21.15.40:0/3228402463 >> v2:172.21.15.189:3300/0 conn(0x7f9a380a38a0 0x7f9a3813db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T08:14:45.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.874+0000 7f9a3d019640 1 -- 172.21.15.40:0/3228402463 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9a380a20d0 msgr2=0x7f9a380a24b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:45.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.874+0000 7f9a3d019640 1 --2- 172.21.15.40:0/3228402463 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9a380a20d0 0x7f9a380a24b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9a30009870 tx=0x7f9a3002b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb7bbb8fe2c2ab4f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:45.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.874+0000 7f9a2ffff640 1 -- 172.21.15.40:0/3228402463 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a30031a30 con 0x7f9a380a20d0 2024-04-22T08:14:45.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.874+0000 7f9a2ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:45.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.874+0000 7f9a2ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:14:45.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.874+0000 7f9a2ffff640 10 monclient: dump: 2024-04-22T08:14:45.881 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:45.881 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:45.882 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:45.882 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:45.882 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:45.882 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:45.882 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:45.882 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:45.882 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:45.882 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:45.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.874+0000 7f9a2ffff640 10 monclient: _finish_auth 0 2024-04-22T08:14:45.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.874+0000 7f9a2ffff640 10 monclient: _check_auth_tickets 2024-04-22T08:14:45.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.874+0000 7f9a2ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:14:45.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.874+0000 7f9a2ffff640 1 -- 172.21.15.40:0/3228402463 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a30031b90 con 0x7f9a380a20d0 2024-04-22T08:14:45.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.874+0000 7f9a3f27d640 5 monclient: authenticate success, global_id 4760 2024-04-22T08:14:45.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.874+0000 7f9a3f27d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:14:45.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.874+0000 7f9a2ffff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:14:45.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.874+0000 7f9a3f27d640 10 monclient: get_monmap_and_config success 2024-04-22T08:14:45.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.874+0000 7f9a3f27d640 4 set_mon_vals no callback set 2024-04-22T08:14:45.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.874+0000 7f9a2ffff640 1 -- 172.21.15.40:0/3228402463 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a30033430 con 0x7f9a380a20d0 2024-04-22T08:14:45.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.874+0000 7f9a2ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:45.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.874+0000 7f9a2ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:14:45.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.874+0000 7f9a2ffff640 10 monclient: dump: 2024-04-22T08:14:45.883 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:45.883 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:45.883 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:45.883 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:45.883 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:45.883 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:45.883 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:45.884 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:45.884 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:45.884 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:45.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.875+0000 7f9a3f27d640 10 monclient: shutdown 2024-04-22T08:14:45.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.875+0000 7f9a3f27d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:14:45.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.875+0000 7f9a3f27d640 1 -- 172.21.15.40:0/3228402463 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9a380a20d0 msgr2=0x7f9a380a24b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:45.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.875+0000 7f9a3f27d640 1 --2- 172.21.15.40:0/3228402463 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9a380a20d0 0x7f9a380a24b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9a30009870 tx=0x7f9a3002b7e0 comp rx=0 tx=0).stop 2024-04-22T08:14:45.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.875+0000 7f9a3f27d640 1 -- 172.21.15.40:0/3228402463 shutdown_connections 2024-04-22T08:14:45.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.875+0000 7f9a3f27d640 1 --2- 172.21.15.40:0/3228402463 >> v2:172.21.15.189:3300/0 conn(0x7f9a380a38a0 0x7f9a3813db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:45.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.875+0000 7f9a3f27d640 1 --2- 172.21.15.40:0/3228402463 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9a380a20d0 0x7f9a380a24b0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:45.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.875+0000 7f9a3f27d640 1 -- 172.21.15.40:0/3228402463 >> 172.21.15.40:0/3228402463 conn(0x7f9a380a8e60 msgr2=0x7f9a380a4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:14:45.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.875+0000 7f9a3f27d640 1 -- 172.21.15.40:0/3228402463 shutdown_connections 2024-04-22T08:14:45.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.875+0000 7f9a3f27d640 1 -- 172.21.15.40:0/3228402463 wait complete. 2024-04-22T08:14:45.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.876+0000 7f9a3f27d640 10 monclient: build_initial_monmap 2024-04-22T08:14:45.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.876+0000 7f9a3f27d640 10 monclient: monmap: 2024-04-22T08:14:45.885 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:14:45.885 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:14:45.885 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:14:45.885 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:14:45.885 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:14:45.885 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:45.885 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:14:45.885 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:14:45.885 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:14:45.885 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:45.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.876+0000 7f9a3f27d640 1 Processor -- start 2024-04-22T08:14:45.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.876+0000 7f9a3f27d640 1 -- start start 2024-04-22T08:14:45.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.876+0000 7f9a3f27d640 10 monclient: init 2024-04-22T08:14:45.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.876+0000 7f9a3f27d640 10 monclient: _reopen_session rank -1 2024-04-22T08:14:45.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.876+0000 7f9a3f27d640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:14:45.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.876+0000 7f9a3f27d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a380a20d0 0x7f9a381c4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:45.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.876+0000 7f9a3f27d640 10 monclient(hunting): picked mon.noname-a con 0x7f9a380a20d0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:14:45.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.876+0000 7f9a3f27d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a380a2d10 0x7f9a381ca880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:45.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.876+0000 7f9a3f27d640 10 monclient(hunting): picked mon.noname-b con 0x7f9a380a2d10 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:14:45.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.876+0000 7f9a3f27d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9a380a38a0 0x7f9a381cadc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:45.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.876+0000 7f9a3f27d640 10 monclient(hunting): picked mon.noname-c con 0x7f9a380a38a0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:14:45.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.876+0000 7f9a3f27d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:45.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.876+0000 7f9a3f27d640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a381cb300 con 0x7f9a380a20d0 2024-04-22T08:14:45.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.876+0000 7f9a3f27d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:45.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.876+0000 7f9a3f27d640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a381a9d70 con 0x7f9a380a2d10 2024-04-22T08:14:45.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.876+0000 7f9a3f27d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:45.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.876+0000 7f9a3f27d640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a38070e40 con 0x7f9a380a38a0 2024-04-22T08:14:45.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.876+0000 7f9a3f27d640 10 monclient(hunting): _renew_subs 2024-04-22T08:14:45.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.876+0000 7f9a3f27d640 10 monclient(hunting): authenticate will time out at 959.870758s 2024-04-22T08:14:45.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.877+0000 7f9a3d81a640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9a380a38a0 0x7f9a381cadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:45.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.877+0000 7f9a3d81a640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9a380a38a0 0x7f9a381cadc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49560/0 (socket says 172.21.15.40:49560) 2024-04-22T08:14:45.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.877+0000 7f9a3d81a640 1 -- 172.21.15.40:0/2021032484 learned_addr learned my addr 172.21.15.40:0/2021032484 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:14:45.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.877+0000 7f9a3d81a640 10 monclient(hunting): get_auth_request con 0x7f9a380a38a0 auth_method 0 2024-04-22T08:14:45.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.877+0000 7f9a3d019640 1 --2- 172.21.15.40:0/2021032484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a380a20d0 0x7f9a381c4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:45.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.877+0000 7f9a3d81a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:45.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.877+0000 7f9a3d81a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:45.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.877+0000 7f9a3d81a640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:45.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.877+0000 7f9a3d019640 10 monclient(hunting): get_auth_request con 0x7f9a380a20d0 auth_method 0 2024-04-22T08:14:45.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.877+0000 7f9a3d019640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:45.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.877+0000 7f9a3d019640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:45.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.877+0000 7f9a3d019640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:45.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.877+0000 7f9a3d81a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:45.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.877+0000 7f9a3d81a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:45.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.877+0000 7f9a3d81a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:45.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.877+0000 7f9a3d019640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:45.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.877+0000 7f9a3d019640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:45.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.877+0000 7f9a3d019640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:45.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.877+0000 7f9a3c818640 1 --2- 172.21.15.40:0/2021032484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a380a2d10 0x7f9a381ca880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:45.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.877+0000 7f9a3c818640 10 monclient(hunting): get_auth_request con 0x7f9a380a2d10 auth_method 0 2024-04-22T08:14:45.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.877+0000 7f9a3c818640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:45.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.877+0000 7f9a3c818640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:45.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.877+0000 7f9a3c818640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:45.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.877+0000 7f9a3d81a640 10 monclient(hunting): handle_auth_done global_id 4763 payload 824 2024-04-22T08:14:45.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.877+0000 7f9a3d81a640 1 -- 172.21.15.40:0/2021032484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a380a2d10 msgr2=0x7f9a381ca880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:45.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.877+0000 7f9a3d81a640 1 --2- 172.21.15.40:0/2021032484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a380a2d10 0x7f9a381ca880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:45.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.877+0000 7f9a3d81a640 1 -- 172.21.15.40:0/2021032484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a380a20d0 msgr2=0x7f9a381c4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:45.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.877+0000 7f9a3d81a640 1 --2- 172.21.15.40:0/2021032484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a380a20d0 0x7f9a381c4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:45.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.877+0000 7f9a3d81a640 10 monclient: _finish_hunting 0 2024-04-22T08:14:45.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.877+0000 7f9a3d81a640 1 monclient: found mon.noname-c 2024-04-22T08:14:45.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.877+0000 7f9a3d81a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:14:45.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.877+0000 7f9a3d81a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:14:45.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.877+0000 7f9a3d81a640 1 -- 172.21.15.40:0/2021032484 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a300094d0 con 0x7f9a380a38a0 2024-04-22T08:14:45.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.877+0000 7f9a3c818640 1 --2- 172.21.15.40:0/2021032484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a380a2d10 0x7f9a381ca880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T08:14:45.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.879+0000 7f9a3d81a640 1 -- 172.21.15.40:0/2021032484 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9a380a38a0 msgr2=0x7f9a381cadc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:45.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.879+0000 7f9a3d81a640 1 --2- 172.21.15.40:0/2021032484 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9a380a38a0 0x7f9a381cadc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9a3400cef0 tx=0x7f9a3400a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:45.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.879+0000 7f9a2dffb640 1 -- 172.21.15.40:0/2021032484 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a3400da50 con 0x7f9a380a38a0 2024-04-22T08:14:45.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.879+0000 7f9a2dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:45.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.879+0000 7f9a2dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:14:45.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.879+0000 7f9a2dffb640 10 monclient: dump: 2024-04-22T08:14:45.889 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:45.889 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:45.890 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:45.890 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:45.890 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:45.890 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:45.890 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:45.890 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:45.890 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:45.890 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:45.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.879+0000 7f9a2dffb640 10 monclient: _finish_auth 0 2024-04-22T08:14:45.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.879+0000 7f9a2dffb640 10 monclient: _check_auth_tickets 2024-04-22T08:14:45.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.879+0000 7f9a2dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:14:45.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.879+0000 7f9a2dffb640 1 -- 172.21.15.40:0/2021032484 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a3400dbd0 con 0x7f9a380a38a0 2024-04-22T08:14:45.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.879+0000 7f9a3f27d640 5 monclient: authenticate success, global_id 4763 2024-04-22T08:14:45.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.879+0000 7f9a3f27d640 10 monclient: _renew_subs 2024-04-22T08:14:45.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.879+0000 7f9a3f27d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:14:45.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.879+0000 7f9a3f27d640 1 -- 172.21.15.40:0/2021032484 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a381ab7d0 con 0x7f9a380a38a0 2024-04-22T08:14:45.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.879+0000 7f9a2dffb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:14:45.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.879+0000 7f9a2dffb640 1 -- 172.21.15.40:0/2021032484 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a34015440 con 0x7f9a380a38a0 2024-04-22T08:14:45.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.879+0000 7f9a2dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:45.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.879+0000 7f9a2dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:14:45.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.879+0000 7f9a2dffb640 10 monclient: dump: 2024-04-22T08:14:45.891 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:45.891 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:45.891 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:45.891 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:45.891 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:45.891 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:45.891 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:45.892 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:45.892 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:45.892 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:45.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.879+0000 7f9a2f7fe640 4 set_mon_vals no callback set 2024-04-22T08:14:45.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.880+0000 7f9a3f27d640 10 monclient: _renew_subs 2024-04-22T08:14:45.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.880+0000 7f9a3f27d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:14:45.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.880+0000 7f9a3f27d640 1 -- 172.21.15.40:0/2021032484 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a381abd90 con 0x7f9a380a38a0 2024-04-22T08:14:45.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.880+0000 7f9a2dffb640 1 -- 172.21.15.40:0/2021032484 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f9a3401d360 con 0x7f9a380a38a0 2024-04-22T08:14:45.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.881+0000 7f9a2dffb640 1 --2- 172.21.15.40:0/2021032484 >> v2:172.21.15.189:6800/30168 conn(0x7f9a1c037850 0x7f9a1c039d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:45.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.881+0000 7f9a3f27d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:14:45.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.881+0000 7f9a2dffb640 1 -- 172.21.15.40:0/2021032484 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f9a3404aae0 con 0x7f9a380a38a0 2024-04-22T08:14:45.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.881+0000 7f9a3f27d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:14:45.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.881+0000 7f9a3f27d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:14:45.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.881+0000 7f9a3f27d640 1 -- 172.21.15.40:0/2021032484 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a3804e9b0 con 0x7f9a380a38a0 2024-04-22T08:14:45.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.881+0000 7f9a3d019640 1 --2- 172.21.15.40:0/2021032484 >> v2:172.21.15.189:6800/30168 conn(0x7f9a1c037850 0x7f9a1c039d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:45.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.881+0000 7f9a3d019640 10 monclient: get_auth_request con 0x7f9a1c037850 auth_method 0 2024-04-22T08:14:45.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.884+0000 7f9a3d019640 1 --2- 172.21.15.40:0/2021032484 >> v2:172.21.15.189:6800/30168 conn(0x7f9a1c037850 0x7f9a1c039d10 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9a30035000 tx=0x7f9a30008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:45.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.886+0000 7f9a2dffb640 1 -- 172.21.15.40:0/2021032484 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f9a3401a020 con 0x7f9a380a38a0 2024-04-22T08:14:45.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.886+0000 7f9a2dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:14:45.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:45.886+0000 7f9a2dffb640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:14:46.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.143+0000 7f9a3f27d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:14:46.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.143+0000 7f9a3f27d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:14:46.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.143+0000 7f9a3f27d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:14:46.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.143+0000 7f9a3f27d640 1 -- 172.21.15.40:0/2021032484 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9a3806ca50 con 0x7f9a380a38a0 2024-04-22T08:14:46.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.143+0000 7f9a2dffb640 1 -- 172.21.15.40:0/2021032484 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f9a34013340 con 0x7f9a380a38a0 2024-04-22T08:14:46.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.143+0000 7f9a2dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:14:46.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.143+0000 7f9a2dffb640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:14:46.143 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:14:46.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.145+0000 7f9a3f27d640 1 -- 172.21.15.40:0/2021032484 >> v2:172.21.15.189:6800/30168 conn(0x7f9a1c037850 msgr2=0x7f9a1c039d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:46.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.145+0000 7f9a3f27d640 1 --2- 172.21.15.40:0/2021032484 >> v2:172.21.15.189:6800/30168 conn(0x7f9a1c037850 0x7f9a1c039d10 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9a30035000 tx=0x7f9a30008030 comp rx=0 tx=0).stop 2024-04-22T08:14:46.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.145+0000 7f9a3f27d640 10 monclient: shutdown 2024-04-22T08:14:46.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.145+0000 7f9a3f27d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:14:46.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.145+0000 7f9a3f27d640 1 -- 172.21.15.40:0/2021032484 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9a380a38a0 msgr2=0x7f9a381cadc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:46.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.145+0000 7f9a3f27d640 1 --2- 172.21.15.40:0/2021032484 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9a380a38a0 0x7f9a381cadc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9a3400cef0 tx=0x7f9a3400a710 comp rx=0 tx=0).stop 2024-04-22T08:14:46.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.145+0000 7f9a3f27d640 1 -- 172.21.15.40:0/2021032484 shutdown_connections 2024-04-22T08:14:46.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.145+0000 7f9a3f27d640 1 --2- 172.21.15.40:0/2021032484 >> v2:172.21.15.189:6800/30168 conn(0x7f9a1c037850 0x7f9a1c039d10 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:46.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.145+0000 7f9a3f27d640 1 --2- 172.21.15.40:0/2021032484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a380a2d10 0x7f9a381ca880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:46.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.145+0000 7f9a3f27d640 1 --2- 172.21.15.40:0/2021032484 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9a380a38a0 0x7f9a381cadc0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:46.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.145+0000 7f9a3f27d640 1 --2- 172.21.15.40:0/2021032484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a380a20d0 0x7f9a381c4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:46.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.145+0000 7f9a3f27d640 1 -- 172.21.15.40:0/2021032484 >> 172.21.15.40:0/2021032484 conn(0x7f9a380a8e60 msgr2=0x7f9a3806ef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:14:46.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.145+0000 7f9a3f27d640 1 -- 172.21.15.40:0/2021032484 shutdown_connections 2024-04-22T08:14:46.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.145+0000 7f9a3f27d640 1 -- 172.21.15.40:0/2021032484 wait complete. 2024-04-22T08:14:46.155 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-22T08:14:46.155 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-22T08:14:46.155 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-22T08:14:46.156 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-22T08:14:46.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.264+0000 7fb6bc2e9640 10 monclient: get_monmap_and_config 2024-04-22T08:14:46.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.264+0000 7fb6bc2e9640 10 monclient: build_initial_monmap 2024-04-22T08:14:46.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.264+0000 7fb6bc2e9640 10 monclient: monmap: 2024-04-22T08:14:46.264 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:14:46.264 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:46.264 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:14:46.265090+0000 2024-04-22T08:14:46.264 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:14:46.265090+0000 2024-04-22T08:14:46.264 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:14:46.264 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:46.264 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:14:46.265 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:14:46.265 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:14:46.265 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:14:46.265 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:14:46.265 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:46.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.264+0000 7fb6bc2e9640 1 Processor -- start 2024-04-22T08:14:46.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.264+0000 7fb6bc2e9640 1 -- start start 2024-04-22T08:14:46.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.264+0000 7fb6bc2e9640 10 monclient: init 2024-04-22T08:14:46.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.265+0000 7fb6bc2e9640 10 monclient: _reopen_session rank -1 2024-04-22T08:14:46.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.265+0000 7fb6bc2e9640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-04-22T08:14:46.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.265+0000 7fb6bc2e9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb6b412fad0 addr v1:172.21.15.40:6789/0 2024-04-22T08:14:46.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.265+0000 7fb6bc2e9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb6b4135c30 addr v1:172.21.15.189:6789/0 2024-04-22T08:14:46.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.265+0000 7fb6bc2e9640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb6b4139740 0x7fb6b413bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:46.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.265+0000 7fb6bc2e9640 10 monclient(hunting): picked mon.noname-a con 0x7fb6b4139740 addr v2:172.21.15.40:3300/0 2024-04-22T08:14:46.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.265+0000 7fb6bc2e9640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6b41305c0 con 0x7fb6b412fad0 2024-04-22T08:14:46.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.265+0000 7fb6bc2e9640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6b409ae50 con 0x7fb6b4135c30 2024-04-22T08:14:46.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.265+0000 7fb6bc2e9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:46.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.265+0000 7fb6bc2e9640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6b409efc0 con 0x7fb6b4139740 2024-04-22T08:14:46.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.265+0000 7fb6bc2e9640 10 monclient(hunting): _renew_subs 2024-04-22T08:14:46.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.265+0000 7fb6bc2e9640 10 monclient(hunting): authenticate will time out at 960.259295s 2024-04-22T08:14:46.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.265+0000 7fb6ba085640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fb6b412fad0 0x7fb6b412feb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:58984/0 (socket says 172.21.15.40:58984) 2024-04-22T08:14:46.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.265+0000 7fb6ba085640 1 -- 172.21.15.40:0/3105216380 learned_addr learned my addr 172.21.15.40:0/3105216380 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:14:46.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.265+0000 7fb6ba886640 1 --2- 172.21.15.40:0/3105216380 >> v2:172.21.15.40:3300/0 conn(0x7fb6b4139740 0x7fb6b413bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:46.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.265+0000 7fb6ba886640 10 monclient(hunting): get_auth_request con 0x7fb6b4139740 auth_method 0 2024-04-22T08:14:46.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.265+0000 7fb6ba886640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:46.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.265+0000 7fb6ba886640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:46.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.265+0000 7fb6ba886640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:46.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.265+0000 7fb6ba886640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:46.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.265+0000 7fb6ba886640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:46.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.265+0000 7fb6ba886640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:46.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.265+0000 7fb6ba886640 10 monclient(hunting): handle_auth_done global_id 34149 payload 274 2024-04-22T08:14:46.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.265+0000 7fb6ba886640 1 -- 172.21.15.40:0/3105216380 >> v1:172.21.15.189:6789/0 conn(0x7fb6b4135c30 legacy=0x7fb6b4138020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:14:46.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.265+0000 7fb6ba886640 1 -- 172.21.15.40:0/3105216380 >> v1:172.21.15.40:6789/0 conn(0x7fb6b412fad0 legacy=0x7fb6b412feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:14:46.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.265+0000 7fb6ba886640 10 monclient: _finish_hunting 0 2024-04-22T08:14:46.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.265+0000 7fb6ba886640 1 monclient: found mon.noname-a 2024-04-22T08:14:46.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.265+0000 7fb6ba886640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:14:46.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.265+0000 7fb6ba886640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:14:46.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.265+0000 7fb6ba886640 1 -- 172.21.15.40:0/3105216380 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6b409d070 con 0x7fb6b4139740 2024-04-22T08:14:46.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.268+0000 7fb6ba886640 1 -- 172.21.15.40:0/3105216380 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6b4139740 msgr2=0x7fb6b413bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:46.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.268+0000 7fb6ba886640 1 --2- 172.21.15.40:0/3105216380 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6b4139740 0x7fb6b413bb30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb6a4009d90 tx=0x7fb6a402bd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d2b3dcb0f20657e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:46.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.269+0000 7fb6b9083640 1 -- 172.21.15.40:0/3105216380 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a402cda0 con 0x7fb6b4139740 2024-04-22T08:14:46.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.269+0000 7fb6b9083640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:46.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.269+0000 7fb6b9083640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:14:46.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.269+0000 7fb6b9083640 10 monclient: dump: 2024-04-22T08:14:46.275 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:46.275 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:46.275 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:46.275 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:46.275 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:46.276 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:46.276 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:46.276 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:46.276 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:46.276 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:46.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.269+0000 7fb6b9083640 10 monclient: _finish_auth 0 2024-04-22T08:14:46.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.269+0000 7fb6b9083640 10 monclient: _check_auth_tickets 2024-04-22T08:14:46.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.269+0000 7fb6b9083640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:14:46.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.269+0000 7fb6bc2e9640 5 monclient: authenticate success, global_id 34149 2024-04-22T08:14:46.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.269+0000 7fb6bc2e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:14:46.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.269+0000 7fb6b9083640 1 -- 172.21.15.40:0/3105216380 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6a4034440 con 0x7fb6b4139740 2024-04-22T08:14:46.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.269+0000 7fb6b9083640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:14:46.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.269+0000 7fb6bc2e9640 10 monclient: get_monmap_and_config success 2024-04-22T08:14:46.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.269+0000 7fb6bc2e9640 4 set_mon_vals no callback set 2024-04-22T08:14:46.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.269+0000 7fb6b9083640 1 -- 172.21.15.40:0/3105216380 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a40347d0 con 0x7fb6b4139740 2024-04-22T08:14:46.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.269+0000 7fb6b9083640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:46.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.269+0000 7fb6b9083640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:14:46.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.269+0000 7fb6b9083640 10 monclient: dump: 2024-04-22T08:14:46.277 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:46.277 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:46.277 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:46.277 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:46.277 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:46.277 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:46.277 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:46.277 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:46.277 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:46.277 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:46.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.269+0000 7fb6bc2e9640 10 monclient: shutdown 2024-04-22T08:14:46.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.269+0000 7fb6bc2e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:14:46.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.269+0000 7fb6bc2e9640 1 -- 172.21.15.40:0/3105216380 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6b4139740 msgr2=0x7fb6b413bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:46.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.269+0000 7fb6bc2e9640 1 --2- 172.21.15.40:0/3105216380 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6b4139740 0x7fb6b413bb30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb6a4009d90 tx=0x7fb6a402bd00 comp rx=0 tx=0).stop 2024-04-22T08:14:46.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.269+0000 7fb6bc2e9640 1 -- 172.21.15.40:0/3105216380 shutdown_connections 2024-04-22T08:14:46.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.269+0000 7fb6bc2e9640 1 --2- 172.21.15.40:0/3105216380 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6b4139740 0x7fb6b413bb30 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:46.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.269+0000 7fb6bc2e9640 1 -- 172.21.15.40:0/3105216380 >> 172.21.15.40:0/3105216380 conn(0x7fb6b40a0ae0 msgr2=0x7fb6b40a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:14:46.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.269+0000 7fb6bc2e9640 1 -- 172.21.15.40:0/3105216380 shutdown_connections 2024-04-22T08:14:46.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.269+0000 7fb6bc2e9640 1 -- 172.21.15.40:0/3105216380 wait complete. 2024-04-22T08:14:46.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.270+0000 7fb6bc2e9640 10 monclient: build_initial_monmap 2024-04-22T08:14:46.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.270+0000 7fb6bc2e9640 10 monclient: monmap: 2024-04-22T08:14:46.278 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:14:46.278 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:14:46.278 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:14:46.278 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:14:46.279 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:14:46.279 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:46.279 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:14:46.279 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:14:46.279 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:14:46.279 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:46.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.270+0000 7fb6bc2e9640 1 Processor -- start 2024-04-22T08:14:46.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.270+0000 7fb6bc2e9640 1 -- start start 2024-04-22T08:14:46.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.270+0000 7fb6bc2e9640 10 monclient: init 2024-04-22T08:14:46.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6bc2e9640 10 monclient: _reopen_session rank -1 2024-04-22T08:14:46.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6bc2e9640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-22T08:14:46.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6bc2e9640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6b412fad0 0x7fb6b4132db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:46.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6bc2e9640 10 monclient(hunting): picked mon.noname-c con 0x7fb6b412fad0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:14:46.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6bc2e9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6b4135c30 0x7fb6b412da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:46.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6bc2e9640 10 monclient(hunting): picked mon.noname-a con 0x7fb6b4135c30 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:14:46.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6bc2e9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6b4139740 0x7fb6b412dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:46.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6bc2e9640 10 monclient(hunting): picked mon.noname-b con 0x7fb6b4139740 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:14:46.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6bc2e9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:46.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6bc2e9640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b412f740 con 0x7fb6b4135c30 2024-04-22T08:14:46.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6bc2e9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:46.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6bc2e9640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b413c600 con 0x7fb6b4139740 2024-04-22T08:14:46.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6bc2e9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:46.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6bc2e9640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b4071870 con 0x7fb6b412fad0 2024-04-22T08:14:46.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6bc2e9640 10 monclient(hunting): _renew_subs 2024-04-22T08:14:46.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6bc2e9640 10 monclient(hunting): authenticate will time out at 960.265011s 2024-04-22T08:14:46.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6ba085640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6b412fad0 0x7fb6b4132db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:46.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6b9884640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6b4135c30 0x7fb6b412da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:46.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6ba085640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6b412fad0 0x7fb6b4132db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49570/0 (socket says 172.21.15.40:49570) 2024-04-22T08:14:46.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6ba085640 1 -- 172.21.15.40:0/2112127888 learned_addr learned my addr 172.21.15.40:0/2112127888 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:14:46.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6ba085640 10 monclient(hunting): get_auth_request con 0x7fb6b412fad0 auth_method 0 2024-04-22T08:14:46.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6ba085640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:46.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6ba085640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:46.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6ba085640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:46.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6b9884640 10 monclient(hunting): get_auth_request con 0x7fb6b4135c30 auth_method 0 2024-04-22T08:14:46.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6b9884640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:46.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6b9884640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:46.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6b9884640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:46.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6ba085640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:46.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6ba085640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:46.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6ba085640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:46.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6ba886640 1 --2- 172.21.15.40:0/2112127888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6b4139740 0x7fb6b412dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:46.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6b9884640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:46.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6b9884640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:46.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6b9884640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:46.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6ba886640 10 monclient(hunting): get_auth_request con 0x7fb6b4139740 auth_method 0 2024-04-22T08:14:46.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6ba886640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:46.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6ba886640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:46.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6ba886640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:46.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6ba886640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:46.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6ba886640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:46.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6ba886640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:46.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6ba085640 10 monclient(hunting): handle_auth_done global_id 4766 payload 824 2024-04-22T08:14:46.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6ba085640 1 -- 172.21.15.40:0/2112127888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6b4139740 msgr2=0x7fb6b412dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:46.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6ba085640 1 --2- 172.21.15.40:0/2112127888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6b4139740 0x7fb6b412dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:46.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6ba085640 1 -- 172.21.15.40:0/2112127888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6b4135c30 msgr2=0x7fb6b412da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:46.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6ba085640 1 --2- 172.21.15.40:0/2112127888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6b4135c30 0x7fb6b412da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:46.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6ba085640 10 monclient: _finish_hunting 0 2024-04-22T08:14:46.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6ba085640 1 monclient: found mon.noname-c 2024-04-22T08:14:46.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6ba085640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:14:46.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6ba085640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:14:46.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6ba085640 1 -- 172.21.15.40:0/2112127888 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a4009a70 con 0x7fb6b412fad0 2024-04-22T08:14:46.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.271+0000 7fb6b9884640 1 --2- 172.21.15.40:0/2112127888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6b4135c30 0x7fb6b412da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:14:46.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.273+0000 7fb6ba085640 1 -- 172.21.15.40:0/2112127888 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6b412fad0 msgr2=0x7fb6b4132db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:46.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.273+0000 7fb6ba085640 1 --2- 172.21.15.40:0/2112127888 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6b412fad0 0x7fb6b4132db0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac00c460 tx=0x7fb6ac00c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:46.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6ba085640 1 -- 172.21.15.40:0/2112127888 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6b412fad0 msgr2=0x7fb6b4132db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-22T08:14:46.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6ba085640 1 -- 172.21.15.40:0/2112127888 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6b412fad0 msgr2=0x7fb6b4132db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:14:46.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6ba085640 1 --2- 172.21.15.40:0/2112127888 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6b412fad0 0x7fb6b4132db0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac00c460 tx=0x7fb6ac00c920 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:14:46.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6ba085640 1 --2- 172.21.15.40:0/2112127888 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6b412fad0 0x7fb6b4132db0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac00c460 tx=0x7fb6ac00c920 comp rx=0 tx=0).stop 2024-04-22T08:14:46.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6b2ffd640 10 monclient: ms_handle_reset current mon [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:14:46.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6b2ffd640 10 monclient: _reopen_session rank -1 2024-04-22T08:14:46.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6b2ffd640 1 -- 172.21.15.40:0/2112127888 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6b412fad0 msgr2=0x7fb6b4132db0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:14:46.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6b2ffd640 1 --2- 172.21.15.40:0/2112127888 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6b412fad0 0x7fb6b4132db0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:46.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6b2ffd640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-22T08:14:46.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6b2ffd640 1 --2- 172.21.15.40:0/2112127888 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6a00034a0 0x7fb6a00058f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:46.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6b2ffd640 10 monclient(hunting): picked mon.noname-c con 0x7fb6a00034a0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:14:46.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6b2ffd640 1 --2- 172.21.15.40:0/2112127888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6b4135c30 0x7fb6a0008420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:46.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6b2ffd640 10 monclient(hunting): picked mon.noname-a con 0x7fb6b4135c30 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:14:46.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6b2ffd640 1 --2- 172.21.15.40:0/2112127888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6b4139740 0x7fb6a000aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:46.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6b2ffd640 10 monclient(hunting): picked mon.noname-b con 0x7fb6b4139740 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:14:46.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6b2ffd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:46.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6b2ffd640 1 -- 172.21.15.40:0/2112127888 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a000b950 con 0x7fb6b4135c30 2024-04-22T08:14:46.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6b2ffd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:46.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6b2ffd640 1 -- 172.21.15.40:0/2112127888 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a000bac0 con 0x7fb6b4139740 2024-04-22T08:14:46.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6b2ffd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:46.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6b2ffd640 1 -- 172.21.15.40:0/2112127888 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a000bc80 con 0x7fb6a00034a0 2024-04-22T08:14:46.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6b2ffd640 10 monclient(hunting): _renew_subs 2024-04-22T08:14:46.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6ba085640 1 --2- 172.21.15.40:0/2112127888 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6a00034a0 0x7fb6a00058f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:46.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6ba085640 10 monclient(hunting): get_auth_request con 0x7fb6a00034a0 auth_method 0 2024-04-22T08:14:46.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6b9884640 1 --2- 172.21.15.40:0/2112127888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6b4135c30 0x7fb6a0008420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:46.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6ba085640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:46.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6ba085640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:46.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6ba085640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:14:46.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6b9884640 10 monclient(hunting): get_auth_request con 0x7fb6b4135c30 auth_method 0 2024-04-22T08:14:46.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6b9884640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:46.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6b9884640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:46.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6b9884640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:14:46.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6ba085640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:46.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6ba085640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:46.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6ba085640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-22T08:14:46.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6b9884640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:46.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6b9884640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:46.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6b9884640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-22T08:14:46.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6ba085640 10 monclient(hunting): handle_auth_done global_id 4766 payload 293 2024-04-22T08:14:46.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6ba085640 1 -- 172.21.15.40:0/2112127888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6b4139740 msgr2=0x7fb6a000aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:46.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6ba085640 1 --2- 172.21.15.40:0/2112127888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6b4139740 0x7fb6a000aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:46.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6ba085640 1 -- 172.21.15.40:0/2112127888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6b4135c30 msgr2=0x7fb6a0008420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:46.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6ba085640 1 --2- 172.21.15.40:0/2112127888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6b4135c30 0x7fb6a0008420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:46.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6ba085640 10 monclient: _finish_hunting 0 2024-04-22T08:14:46.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6ba085640 1 monclient: found mon.noname-c 2024-04-22T08:14:46.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6ba085640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:14:46.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6ba085640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:14:46.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6ba085640 1 -- 172.21.15.40:0/2112127888 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a00102b0 con 0x7fb6a00034a0 2024-04-22T08:14:46.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.274+0000 7fb6b9884640 1 --2- 172.21.15.40:0/2112127888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6b4135c30 0x7fb6a0008420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:14:46.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.276+0000 7fb6ba085640 1 -- 172.21.15.40:0/2112127888 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6a00034a0 msgr2=0x7fb6a00058f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:46.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.276+0000 7fb6ba085640 1 --2- 172.21.15.40:0/2112127888 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6a00034a0 0x7fb6a00058f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac015000 tx=0x7fb6ac004a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:46.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.276+0000 7fb6b2ffd640 1 -- 172.21.15.40:0/2112127888 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6ac007210 con 0x7fb6a00034a0 2024-04-22T08:14:46.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.276+0000 7fb6b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:46.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.276+0000 7fb6b2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:14:46.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.276+0000 7fb6b2ffd640 10 monclient: dump: 2024-04-22T08:14:46.292 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:46.292 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:46.292 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:46.292 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:46.292 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:46.292 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:46.292 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:46.292 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:46.292 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:46.292 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:46.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.276+0000 7fb6b2ffd640 10 monclient: _finish_auth 0 2024-04-22T08:14:46.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.276+0000 7fb6b2ffd640 10 monclient: _check_auth_tickets 2024-04-22T08:14:46.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.276+0000 7fb6b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:14:46.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.276+0000 7fb6b2ffd640 1 -- 172.21.15.40:0/2112127888 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6ac007370 con 0x7fb6a00034a0 2024-04-22T08:14:46.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.276+0000 7fb6bc2e9640 5 monclient: authenticate success, global_id 4766 2024-04-22T08:14:46.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.276+0000 7fb6bc2e9640 10 monclient: _renew_subs 2024-04-22T08:14:46.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.276+0000 7fb6bc2e9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:14:46.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.276+0000 7fb6bc2e9640 1 -- 172.21.15.40:0/2112127888 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6b413de60 con 0x7fb6a00034a0 2024-04-22T08:14:46.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.276+0000 7fb6b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:14:46.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.276+0000 7fb6b2ffd640 1 -- 172.21.15.40:0/2112127888 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6ac007720 con 0x7fb6a00034a0 2024-04-22T08:14:46.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.276+0000 7fb6b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:46.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.276+0000 7fb6b2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:14:46.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.276+0000 7fb6b2ffd640 10 monclient: dump: 2024-04-22T08:14:46.293 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:46.293 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:46.293 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:46.293 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:46.294 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:46.294 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:46.294 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:46.294 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:46.294 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:46.294 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:46.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.276+0000 7fb6bc2e9640 10 monclient: _renew_subs 2024-04-22T08:14:46.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.276+0000 7fb6bc2e9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:14:46.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.276+0000 7fb6bc2e9640 1 -- 172.21.15.40:0/2112127888 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6b413b830 con 0x7fb6a00034a0 2024-04-22T08:14:46.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.276+0000 7fb6b8882640 4 set_mon_vals no callback set 2024-04-22T08:14:46.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.277+0000 7fb6b2ffd640 1 -- 172.21.15.40:0/2112127888 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fb6ac01c470 con 0x7fb6a00034a0 2024-04-22T08:14:46.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.277+0000 7fb6bc2e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:14:46.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.277+0000 7fb6bc2e9640 1 --2- 172.21.15.40:0/2112127888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6b4132db0 0x7fb6b412da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:46.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.277+0000 7fb6bc2e9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:46.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.277+0000 7fb6bc2e9640 1 -- 172.21.15.40:0/2112127888 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b40667a0 con 0x7fb6b4132db0 2024-04-22T08:14:46.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.277+0000 7fb6b2ffd640 1 --2- 172.21.15.40:0/2112127888 >> v2:172.21.15.189:6800/30168 conn(0x7fb6b412fad0 0x7fb6a0046110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:46.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.277+0000 7fb6b9884640 1 --2- 172.21.15.40:0/2112127888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6b4132db0 0x7fb6b412da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:46.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.277+0000 7fb6b9884640 10 monclient: get_auth_request con 0x7fb6b4132db0 auth_method 0 2024-04-22T08:14:46.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.277+0000 7fb6b9884640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:46.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.277+0000 7fb6b9884640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:46.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.277+0000 7fb6b9884640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:46.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.277+0000 7fb6b2ffd640 1 -- 172.21.15.40:0/2112127888 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 3898+0+0 (secure 0 0 0) 0x7fb6ac04bfa0 con 0x7fb6a00034a0 2024-04-22T08:14:46.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.278+0000 7fb6b9884640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:46.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.278+0000 7fb6b9884640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:46.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.278+0000 7fb6b9884640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:46.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.278+0000 7fb6b9884640 10 monclient(hunting): handle_auth_done global_id 34155 payload 824 2024-04-22T08:14:46.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.278+0000 7fb6b9884640 1 -- 172.21.15.40:0/2112127888 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb6b409c9b0 con 0x7fb6b4132db0 2024-04-22T08:14:46.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.278+0000 7fb6ba886640 1 --2- 172.21.15.40:0/2112127888 >> v2:172.21.15.189:6800/30168 conn(0x7fb6b412fad0 0x7fb6a0046110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:46.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.278+0000 7fb6ba886640 10 monclient: get_auth_request con 0x7fb6b412fad0 auth_method 0 2024-04-22T08:14:46.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.280+0000 7fb6b9884640 1 -- 172.21.15.40:0/2112127888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6b4132db0 msgr2=0x7fb6b412da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:46.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.280+0000 7fb6b9884640 1 --2- 172.21.15.40:0/2112127888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6b4132db0 0x7fb6b412da90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb6b413dc30 tx=0x7fb6a8010610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:46.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.280+0000 7fb6b2ffd640 1 -- 172.21.15.40:0/2112127888 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a8010e40 con 0x7fb6b4132db0 2024-04-22T08:14:46.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.280+0000 7fb6b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:46.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.280+0000 7fb6b2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:14:46.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.280+0000 7fb6b2ffd640 10 monclient: dump: 2024-04-22T08:14:46.296 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:46.296 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:46.296 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:46.296 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:46.296 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:46.296 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:46.297 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:46.297 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:46.297 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:46.297 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:46.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.280+0000 7fb6ba886640 1 --2- 172.21.15.40:0/2112127888 >> v2:172.21.15.189:6800/30168 conn(0x7fb6b412fad0 0x7fb6a0046110 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb6a4036000 tx=0x7fb6a402e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:46.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.280+0000 7fb6b2ffd640 1 -- 172.21.15.40:0/2112127888 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7fb6a8015860 con 0x7fb6b4132db0 2024-04-22T08:14:46.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.280+0000 7fb6b2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:14:46.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.280+0000 7fb6b2ffd640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:14:46.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.280+0000 7fb6b2ffd640 1 -- 172.21.15.40:0/2112127888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6b4132db0 msgr2=0x7fb6b412da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:46.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.280+0000 7fb6b2ffd640 1 --2- 172.21.15.40:0/2112127888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6b4132db0 0x7fb6b412da90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb6b413dc30 tx=0x7fb6a8010610 comp rx=0 tx=0).stop 2024-04-22T08:14:46.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.280+0000 7fb6b2ffd640 1 -- 172.21.15.40:0/2112127888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6b4132db0 msgr2=0x7fb6b412da90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:14:46.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.280+0000 7fb6b2ffd640 1 --2- 172.21.15.40:0/2112127888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6b4132db0 0x7fb6b412da90 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb6b413dc30 tx=0x7fb6a8010610 comp rx=0 tx=0).stop 2024-04-22T08:14:46.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.287+0000 7fb6bc2e9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:14:46.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.287+0000 7fb6bc2e9640 1 --2- 172.21.15.40:0/2112127888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6b412dfd0 0x7fb6b412e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:46.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.287+0000 7fb6bc2e9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:46.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.287+0000 7fb6bc2e9640 1 -- 172.21.15.40:0/2112127888 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b4133190 con 0x7fb6b412dfd0 2024-04-22T08:14:46.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.287+0000 7fb6b9884640 1 --2- 172.21.15.40:0/2112127888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6b412dfd0 0x7fb6b412e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:46.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.287+0000 7fb6b9884640 10 monclient: get_auth_request con 0x7fb6b412dfd0 auth_method 0 2024-04-22T08:14:46.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.287+0000 7fb6b9884640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:46.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.287+0000 7fb6b9884640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:46.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.287+0000 7fb6b9884640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:46.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.287+0000 7fb6b9884640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:46.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.287+0000 7fb6b9884640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:46.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.287+0000 7fb6b9884640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:46.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.287+0000 7fb6b9884640 10 monclient(hunting): handle_auth_done global_id 34158 payload 824 2024-04-22T08:14:46.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.287+0000 7fb6b9884640 1 -- 172.21.15.40:0/2112127888 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb6b412e920 con 0x7fb6b412dfd0 2024-04-22T08:14:46.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.289+0000 7fb6b9884640 1 -- 172.21.15.40:0/2112127888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6b412dfd0 msgr2=0x7fb6b412e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:46.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.289+0000 7fb6b9884640 1 --2- 172.21.15.40:0/2112127888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6b412dfd0 0x7fb6b412e3e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb6b413dc30 tx=0x7fb6a80036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:46.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.289+0000 7fb6b2ffd640 1 -- 172.21.15.40:0/2112127888 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a8029600 con 0x7fb6b412dfd0 2024-04-22T08:14:46.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.289+0000 7fb6b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:46.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.289+0000 7fb6b2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:14:46.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.289+0000 7fb6b2ffd640 10 monclient: dump: 2024-04-22T08:14:46.299 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:46.299 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:46.299 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:46.299 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:46.299 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:46.299 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:46.299 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:46.299 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:46.299 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:46.300 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:46.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.290+0000 7fb6b2ffd640 1 -- 172.21.15.40:0/2112127888 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4463 (secure 0 0 0) 0x7fb6a80283a0 con 0x7fb6b412dfd0 2024-04-22T08:14:46.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.290+0000 7fb6b2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:14:46.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.290+0000 7fb6b2ffd640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:14:46.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.290+0000 7fb6b2ffd640 1 -- 172.21.15.40:0/2112127888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6b412dfd0 msgr2=0x7fb6b412e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:46.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.290+0000 7fb6b2ffd640 1 --2- 172.21.15.40:0/2112127888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6b412dfd0 0x7fb6b412e3e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb6b413dc30 tx=0x7fb6a80036f0 comp rx=0 tx=0).stop 2024-04-22T08:14:46.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.290+0000 7fb6b2ffd640 1 -- 172.21.15.40:0/2112127888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6b412dfd0 msgr2=0x7fb6b412e3e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:14:46.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.290+0000 7fb6b2ffd640 1 --2- 172.21.15.40:0/2112127888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6b412dfd0 0x7fb6b412e3e0 secure :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb6b413dc30 tx=0x7fb6a80036f0 comp rx=0 tx=0).stop 2024-04-22T08:14:46.300 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:14:46.300 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:14:46.300 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:14:46.301 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "leader", 2024-04-22T08:14:46.301 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 26, 2024-04-22T08:14:46.301 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:14:46.301 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:14:46.301 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:14:46.301 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:14:46.301 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:14:46.301 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 3, 2024-04-22T08:14:46.301 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:14:46.301 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:14:46.301 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:14:46.301 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:14:46.301 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:14:46.301 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:14:46.302 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:14:46.302 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:14:46.302 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:14:46.302 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:14:46.302 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:14:46.302 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:14:46.302 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:14:46.302 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:14:46.302 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:14:46.302 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:14:46.302 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:14:46.302 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:14:46.302 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:14:46.302 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:14:46.303 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:14:46.303 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:14:46.303 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:14:46.303 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:14:46.303 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:14:46.303 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:14:46.303 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:46.303 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:46.303 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:14:46.303 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:14:46.303 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:14:46.303 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:14:46.303 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:14:46.303 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:14:46.304 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:14:46.304 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:14:46.304 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:14:46.304 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:14:46.304 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:14:46.304 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:14:46.304 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:14:46.304 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:14:46.304 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:14:46.304 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:14:46.304 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:14:46.304 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:14:46.304 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:14:46.304 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:14:46.305 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:14:46.305 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:14:46.305 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:14:46.305 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:14:46.305 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:14:46.305 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:14:46.305 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:14:46.305 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:14:46.305 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:14:46.305 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:46.305 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:14:46.305 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:46.305 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:14:46.305 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:14:46.306 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:14:46.306 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:14:46.306 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:46.306 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:14:46.306 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:14:46.306 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:46.306 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:46.306 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:46.306 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:14:46.306 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:14:46.306 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:46.306 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:46.306 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:46.307 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:46.307 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:14:46.307 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:14:46.307 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:14:46.307 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:14:46.307 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:14:46.307 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:46.307 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:46.307 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:14:46.307 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:14:46.307 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:14:46.307 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:14:46.307 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:46.307 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:14:46.308 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:14:46.308 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:46.308 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:46.308 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:46.308 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:14:46.308 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:14:46.308 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:46.308 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:46.308 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:46.308 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:46.308 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:14:46.308 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:14:46.308 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:14:46.308 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:14:46.309 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:14:46.309 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:46.309 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:46.309 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:14:46.309 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:14:46.309 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:14:46.309 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:14:46.309 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:46.309 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:14:46.309 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:14:46.309 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:46.309 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:46.309 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:46.309 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:14:46.309 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:14:46.310 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:46.310 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:46.310 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:46.310 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:46.310 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:14:46.310 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:14:46.310 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:14:46.310 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:14:46.310 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:14:46.310 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:46.311 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:46.311 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:46.311 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:14:46.311 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:14:46.311 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:46.311 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:14:46.311 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:14:46.311 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:14:46.311 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:46.311 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:14:46.311 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:14:46.311 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:46.311 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:14:46.311 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:14:46.311 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 2 2024-04-22T08:14:46.312 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:46.312 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:46.312 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:46.312 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:14:46.312 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:14:46.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.290+0000 7fb6bc2e9640 1 -- 172.21.15.40:0/2112127888 >> v2:172.21.15.189:6800/30168 conn(0x7fb6b412fad0 msgr2=0x7fb6a0046110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:46.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.290+0000 7fb6bc2e9640 1 --2- 172.21.15.40:0/2112127888 >> v2:172.21.15.189:6800/30168 conn(0x7fb6b412fad0 0x7fb6a0046110 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb6a4036000 tx=0x7fb6a402e000 comp rx=0 tx=0).stop 2024-04-22T08:14:46.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.290+0000 7fb6bc2e9640 10 monclient: shutdown 2024-04-22T08:14:46.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.290+0000 7fb6bc2e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:14:46.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.290+0000 7fb6bc2e9640 1 -- 172.21.15.40:0/2112127888 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6a00034a0 msgr2=0x7fb6a00058f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:46.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.290+0000 7fb6bc2e9640 1 --2- 172.21.15.40:0/2112127888 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6a00034a0 0x7fb6a00058f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac015000 tx=0x7fb6ac004a10 comp rx=0 tx=0).stop 2024-04-22T08:14:46.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.290+0000 7fb6ba886640 1 -- 172.21.15.40:0/2112127888 reap_dead start 2024-04-22T08:14:46.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.290+0000 7fb6bc2e9640 1 -- 172.21.15.40:0/2112127888 shutdown_connections 2024-04-22T08:14:46.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.290+0000 7fb6bc2e9640 1 --2- 172.21.15.40:0/2112127888 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6a00034a0 0x7fb6a00058f0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:46.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.290+0000 7fb6bc2e9640 1 -- 172.21.15.40:0/2112127888 >> 172.21.15.40:0/2112127888 conn(0x7fb6b40a0ae0 msgr2=0x7fb6b4135010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:14:46.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.290+0000 7fb6bc2e9640 1 -- 172.21.15.40:0/2112127888 shutdown_connections 2024-04-22T08:14:46.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.290+0000 7fb6bc2e9640 1 -- 172.21.15.40:0/2112127888 wait complete. 2024-04-22T08:14:46.315 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:14:46.315 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-22T08:14:46.315 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-22T08:14:46.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.415+0000 7f696f0ce640 10 monclient: get_monmap_and_config 2024-04-22T08:14:46.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.415+0000 7f696f0ce640 10 monclient: build_initial_monmap 2024-04-22T08:14:46.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.415+0000 7f696f0ce640 10 monclient: monmap: 2024-04-22T08:14:46.427 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:14:46.427 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:46.427 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:14:46.416175+0000 2024-04-22T08:14:46.427 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:14:46.416175+0000 2024-04-22T08:14:46.427 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:14:46.427 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:46.427 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:14:46.427 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:14:46.427 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:14:46.427 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:14:46.427 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:14:46.427 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:46.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.416+0000 7f696f0ce640 1 Processor -- start 2024-04-22T08:14:46.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.416+0000 7f696f0ce640 1 -- start start 2024-04-22T08:14:46.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.416+0000 7f696f0ce640 10 monclient: init 2024-04-22T08:14:46.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.416+0000 7f696f0ce640 10 monclient: _reopen_session rank -1 2024-04-22T08:14:46.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.416+0000 7f696f0ce640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-04-22T08:14:46.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.416+0000 7f696f0ce640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69680a1ba0 0x7f69680a1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:46.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.416+0000 7f696f0ce640 10 monclient(hunting): picked mon.noname-b con 0x7f69680a1ba0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:14:46.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.416+0000 7f696f0ce640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6968136230 addr v1:172.21.15.189:6789/0 2024-04-22T08:14:46.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.416+0000 7f696f0ce640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6968136670 addr v1:172.21.15.40:6789/0 2024-04-22T08:14:46.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.416+0000 7f696f0ce640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69680a3ba0 con 0x7f6968136670 2024-04-22T08:14:46.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.416+0000 7f696f0ce640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f696809fe80 con 0x7f6968136230 2024-04-22T08:14:46.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.416+0000 7f696f0ce640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:46.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.416+0000 7f696f0ce640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69680722a0 con 0x7f69680a1ba0 2024-04-22T08:14:46.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.416+0000 7f696f0ce640 10 monclient(hunting): _renew_subs 2024-04-22T08:14:46.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.416+0000 7f696f0ce640 10 monclient(hunting): authenticate will time out at 960.410524s 2024-04-22T08:14:46.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.416+0000 7f696ce6a640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69680a1ba0 0x7f69680a1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:46.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.416+0000 7f696d66b640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f6968136670 0x7f69680a3490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:59000/0 (socket says 172.21.15.40:59000) 2024-04-22T08:14:46.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.416+0000 7f696d66b640 1 -- 172.21.15.40:0/3521943871 learned_addr learned my addr 172.21.15.40:0/3521943871 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:14:46.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.416+0000 7f696ce6a640 10 monclient(hunting): get_auth_request con 0x7f69680a1ba0 auth_method 0 2024-04-22T08:14:46.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.416+0000 7f696ce6a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:46.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.416+0000 7f696ce6a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:46.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.416+0000 7f696ce6a640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:46.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.416+0000 7f696ce6a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:46.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.416+0000 7f696ce6a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:46.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.417+0000 7f696ce6a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:46.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.417+0000 7f696ce6a640 10 monclient(hunting): handle_auth_done global_id 4769 payload 274 2024-04-22T08:14:46.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.417+0000 7f696ce6a640 1 -- 172.21.15.40:0/3521943871 >> v1:172.21.15.189:6789/0 conn(0x7f6968136230 legacy=0x7f69680a2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:14:46.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.417+0000 7f696ce6a640 1 -- 172.21.15.40:0/3521943871 >> v1:172.21.15.40:6789/0 conn(0x7f6968136670 legacy=0x7f69680a3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:14:46.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.417+0000 7f696ce6a640 10 monclient: _finish_hunting 0 2024-04-22T08:14:46.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.417+0000 7f696ce6a640 1 monclient: found mon.noname-b 2024-04-22T08:14:46.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.417+0000 7f696ce6a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:14:46.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.417+0000 7f696ce6a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:14:46.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.417+0000 7f696ce6a640 1 -- 172.21.15.40:0/3521943871 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69680a0160 con 0x7f69680a1ba0 2024-04-22T08:14:46.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.420+0000 7f696ce6a640 1 -- 172.21.15.40:0/3521943871 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69680a1ba0 msgr2=0x7f69680a1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:46.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.420+0000 7f696ce6a640 1 --2- 172.21.15.40:0/3521943871 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69680a1ba0 0x7f69680a1f80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f695c0097d0 tx=0x7f695c02b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60453939404c6078 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:46.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.420+0000 7f69677fe640 1 -- 172.21.15.40:0/3521943871 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f695c0319e0 con 0x7f69680a1ba0 2024-04-22T08:14:46.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.420+0000 7f69677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:46.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.420+0000 7f69677fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:14:46.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.420+0000 7f69677fe640 10 monclient: dump: 2024-04-22T08:14:46.431 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:46.431 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:46.431 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:46.431 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:46.431 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:46.431 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:46.431 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:46.431 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:46.431 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:46.431 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:46.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.420+0000 7f69677fe640 10 monclient: _finish_auth 0 2024-04-22T08:14:46.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.420+0000 7f69677fe640 10 monclient: _check_auth_tickets 2024-04-22T08:14:46.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.420+0000 7f69677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:14:46.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.420+0000 7f696f0ce640 5 monclient: authenticate success, global_id 4769 2024-04-22T08:14:46.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.420+0000 7f696f0ce640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:14:46.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.420+0000 7f69677fe640 1 -- 172.21.15.40:0/3521943871 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f695c031b40 con 0x7f69680a1ba0 2024-04-22T08:14:46.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.420+0000 7f69677fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:14:46.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.420+0000 7f696f0ce640 10 monclient: get_monmap_and_config success 2024-04-22T08:14:46.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.420+0000 7f696f0ce640 4 set_mon_vals no callback set 2024-04-22T08:14:46.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.420+0000 7f69677fe640 1 -- 172.21.15.40:0/3521943871 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f695c034480 con 0x7f69680a1ba0 2024-04-22T08:14:46.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.420+0000 7f69677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:46.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.420+0000 7f69677fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:14:46.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.420+0000 7f69677fe640 10 monclient: dump: 2024-04-22T08:14:46.432 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:46.433 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:46.433 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:46.433 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:46.433 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:46.433 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:46.433 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:46.433 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:46.433 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:46.433 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:46.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.420+0000 7f696f0ce640 10 monclient: shutdown 2024-04-22T08:14:46.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.420+0000 7f696f0ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:14:46.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.420+0000 7f696f0ce640 1 -- 172.21.15.40:0/3521943871 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69680a1ba0 msgr2=0x7f69680a1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:46.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.420+0000 7f696f0ce640 1 --2- 172.21.15.40:0/3521943871 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69680a1ba0 0x7f69680a1f80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f695c0097d0 tx=0x7f695c02b790 comp rx=0 tx=0).stop 2024-04-22T08:14:46.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.420+0000 7f696f0ce640 1 -- 172.21.15.40:0/3521943871 shutdown_connections 2024-04-22T08:14:46.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.420+0000 7f696f0ce640 1 --2- 172.21.15.40:0/3521943871 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69680a1ba0 0x7f69680a1f80 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:46.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.420+0000 7f696f0ce640 1 -- 172.21.15.40:0/3521943871 >> 172.21.15.40:0/3521943871 conn(0x7f69680a4e80 msgr2=0x7f69680a5290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:14:46.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.420+0000 7f696f0ce640 1 -- 172.21.15.40:0/3521943871 shutdown_connections 2024-04-22T08:14:46.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.420+0000 7f696f0ce640 1 -- 172.21.15.40:0/3521943871 wait complete. 2024-04-22T08:14:46.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.421+0000 7f696f0ce640 10 monclient: build_initial_monmap 2024-04-22T08:14:46.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.421+0000 7f696f0ce640 10 monclient: monmap: 2024-04-22T08:14:46.434 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:14:46.434 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:14:46.434 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:14:46.434 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:14:46.435 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:14:46.435 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:46.435 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:14:46.435 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:14:46.435 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:14:46.435 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:46.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.421+0000 7f696f0ce640 1 Processor -- start 2024-04-22T08:14:46.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.421+0000 7f696f0ce640 1 -- start start 2024-04-22T08:14:46.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.421+0000 7f696f0ce640 10 monclient: init 2024-04-22T08:14:46.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f696f0ce640 10 monclient: _reopen_session rank -1 2024-04-22T08:14:46.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f696f0ce640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-22T08:14:46.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f696f0ce640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69680a1ba0 0x7f69681313e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:46.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f696f0ce640 10 monclient(hunting): picked mon.noname-a con 0x7f69680a1ba0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:14:46.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f696f0ce640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6968136230 0x7f6968131920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:46.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f696f0ce640 10 monclient(hunting): picked mon.noname-c con 0x7f6968136230 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:14:46.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f696f0ce640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6968136670 0x7f6968131e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:46.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f696f0ce640 10 monclient(hunting): picked mon.noname-b con 0x7f6968136670 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:14:46.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f696f0ce640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:46.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f696f0ce640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6968070500 con 0x7f69680a1ba0 2024-04-22T08:14:46.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f696f0ce640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:46.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f696f0ce640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69681339e0 con 0x7f6968136670 2024-04-22T08:14:46.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f696f0ce640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:46.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f696f0ce640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6968133490 con 0x7f6968136230 2024-04-22T08:14:46.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f696f0ce640 10 monclient(hunting): _renew_subs 2024-04-22T08:14:46.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f696f0ce640 10 monclient(hunting): authenticate will time out at 960.416161s 2024-04-22T08:14:46.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f696ce6a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69680a1ba0 0x7f69681313e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:46.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f6967fff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6968136230 0x7f6968131920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:46.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f696ce6a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69680a1ba0 0x7f69681313e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35152/0 (socket says 172.21.15.40:35152) 2024-04-22T08:14:46.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f6967fff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6968136230 0x7f6968131920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49582/0 (socket says 172.21.15.40:49582) 2024-04-22T08:14:46.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f696ce6a640 1 -- 172.21.15.40:0/1860033543 learned_addr learned my addr 172.21.15.40:0/1860033543 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:14:46.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f696ce6a640 10 monclient(hunting): get_auth_request con 0x7f69680a1ba0 auth_method 0 2024-04-22T08:14:46.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f696ce6a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:46.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f696ce6a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:46.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f696ce6a640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:46.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f6967fff640 10 monclient(hunting): get_auth_request con 0x7f6968136230 auth_method 0 2024-04-22T08:14:46.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f6967fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:46.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f6967fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:46.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f6967fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:46.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f696d66b640 1 --2- 172.21.15.40:0/1860033543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6968136670 0x7f6968131e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:46.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f6967fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:46.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f6967fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:46.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f6967fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:46.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f696d66b640 10 monclient(hunting): get_auth_request con 0x7f6968136670 auth_method 0 2024-04-22T08:14:46.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f696d66b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:46.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f696d66b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:46.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f696d66b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:46.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f696ce6a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:46.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f696ce6a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:46.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f696ce6a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:46.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f696d66b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:46.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f696d66b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:46.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f696d66b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:46.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f6967fff640 10 monclient(hunting): handle_auth_done global_id 4772 payload 824 2024-04-22T08:14:46.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f6967fff640 1 -- 172.21.15.40:0/1860033543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6968136670 msgr2=0x7f6968131e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:46.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f6967fff640 1 --2- 172.21.15.40:0/1860033543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6968136670 0x7f6968131e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:46.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f6967fff640 1 -- 172.21.15.40:0/1860033543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69680a1ba0 msgr2=0x7f69681313e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:46.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f6967fff640 1 --2- 172.21.15.40:0/1860033543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69680a1ba0 0x7f69681313e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:46.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f6967fff640 10 monclient: _finish_hunting 0 2024-04-22T08:14:46.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f6967fff640 1 monclient: found mon.noname-c 2024-04-22T08:14:46.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f6967fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:14:46.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f6967fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:14:46.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f6967fff640 1 -- 172.21.15.40:0/1860033543 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f695c009400 con 0x7f6968136230 2024-04-22T08:14:46.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.422+0000 7f696ce6a640 1 --2- 172.21.15.40:0/1860033543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69680a1ba0 0x7f69681313e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:14:46.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.425+0000 7f6967fff640 1 -- 172.21.15.40:0/1860033543 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6968136230 msgr2=0x7f6968131920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:46.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.425+0000 7f6967fff640 1 --2- 172.21.15.40:0/1860033543 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6968136230 0x7f6968131920 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f695400c440 tx=0x7f695400c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:46.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.425+0000 7f69657fa640 1 -- 172.21.15.40:0/1860033543 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69540119a0 con 0x7f6968136230 2024-04-22T08:14:46.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.425+0000 7f69657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:46.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.425+0000 7f69657fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:14:46.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.425+0000 7f69657fa640 10 monclient: dump: 2024-04-22T08:14:46.440 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:46.440 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:46.440 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:46.440 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:46.440 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:46.440 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:46.440 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:46.440 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:46.440 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:46.440 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:46.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.425+0000 7f69657fa640 10 monclient: _finish_auth 0 2024-04-22T08:14:46.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.425+0000 7f69657fa640 10 monclient: _check_auth_tickets 2024-04-22T08:14:46.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.425+0000 7f69657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:14:46.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.425+0000 7f696f0ce640 5 monclient: authenticate success, global_id 4772 2024-04-22T08:14:46.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.425+0000 7f69657fa640 1 -- 172.21.15.40:0/1860033543 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6954011b00 con 0x7f6968136230 2024-04-22T08:14:46.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.425+0000 7f69657fa640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:14:46.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.425+0000 7f69657fa640 1 -- 172.21.15.40:0/1860033543 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6954012420 con 0x7f6968136230 2024-04-22T08:14:46.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.425+0000 7f696f0ce640 10 monclient: _renew_subs 2024-04-22T08:14:46.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.425+0000 7f696f0ce640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:14:46.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.425+0000 7f696f0ce640 1 -- 172.21.15.40:0/1860033543 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69680a2770 con 0x7f6968136230 2024-04-22T08:14:46.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.425+0000 7f69657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:46.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.425+0000 7f6966ffd640 4 set_mon_vals no callback set 2024-04-22T08:14:46.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.425+0000 7f69657fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:14:46.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.425+0000 7f69657fa640 10 monclient: dump: 2024-04-22T08:14:46.441 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:46.442 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:46.442 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:46.442 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:46.442 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:46.442 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:46.442 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:46.442 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:46.442 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:46.442 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:46.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.425+0000 7f696f0ce640 10 monclient: _renew_subs 2024-04-22T08:14:46.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.425+0000 7f696f0ce640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:14:46.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.425+0000 7f696f0ce640 1 -- 172.21.15.40:0/1860033543 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6968070b30 con 0x7f6968136230 2024-04-22T08:14:46.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.426+0000 7f696f0ce640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:14:46.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.426+0000 7f69657fa640 1 -- 172.21.15.40:0/1860033543 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f6954012580 con 0x7f6968136230 2024-04-22T08:14:46.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.426+0000 7f696f0ce640 1 --2- 172.21.15.40:0/1860033543 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69681346a0 0x7f69681cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:46.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.426+0000 7f696f0ce640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:46.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.426+0000 7f696f0ce640 1 -- 172.21.15.40:0/1860033543 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6968066650 con 0x7f69681346a0 2024-04-22T08:14:46.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.426+0000 7f696ce6a640 1 --2- 172.21.15.40:0/1860033543 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69681346a0 0x7f69681cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:46.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.426+0000 7f696ce6a640 10 monclient: get_auth_request con 0x7f69681346a0 auth_method 0 2024-04-22T08:14:46.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.426+0000 7f696ce6a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:46.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.426+0000 7f696ce6a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:46.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.426+0000 7f696ce6a640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:46.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.426+0000 7f69657fa640 1 --2- 172.21.15.40:0/1860033543 >> v2:172.21.15.189:6800/30168 conn(0x7f694c0377d0 0x7f694c039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:46.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.426+0000 7f696ce6a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:46.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.426+0000 7f696ce6a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:46.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.426+0000 7f696ce6a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:46.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.426+0000 7f69657fa640 1 -- 172.21.15.40:0/1860033543 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f695404a290 con 0x7f6968136230 2024-04-22T08:14:46.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.426+0000 7f696ce6a640 10 monclient(hunting): handle_auth_done global_id 4775 payload 824 2024-04-22T08:14:46.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.426+0000 7f696ce6a640 1 -- 172.21.15.40:0/1860033543 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f69680a24c0 con 0x7f69681346a0 2024-04-22T08:14:46.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.426+0000 7f696d66b640 1 --2- 172.21.15.40:0/1860033543 >> v2:172.21.15.189:6800/30168 conn(0x7f694c0377d0 0x7f694c039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:46.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.426+0000 7f696d66b640 10 monclient: get_auth_request con 0x7f694c0377d0 auth_method 0 2024-04-22T08:14:46.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.428+0000 7f696ce6a640 1 -- 172.21.15.40:0/1860033543 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69681346a0 msgr2=0x7f69681cae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:46.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.428+0000 7f696ce6a640 1 --2- 172.21.15.40:0/1860033543 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69681346a0 0x7f69681cae10 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f695c009720 tx=0x7f695c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:46.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.429+0000 7f69657fa640 1 -- 172.21.15.40:0/1860033543 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f695c034cd0 con 0x7f69681346a0 2024-04-22T08:14:46.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.429+0000 7f69657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:46.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.429+0000 7f69657fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:14:46.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.429+0000 7f69657fa640 10 monclient: dump: 2024-04-22T08:14:46.446 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:46.446 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:46.446 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:46.446 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:46.446 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:46.447 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:46.447 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:46.447 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:46.447 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:46.447 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:46.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.429+0000 7f69657fa640 1 -- 172.21.15.40:0/1860033543 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f695c034e30 con 0x7f69681346a0 2024-04-22T08:14:46.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.429+0000 7f69657fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:14:46.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.429+0000 7f69657fa640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:14:46.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.429+0000 7f69657fa640 1 -- 172.21.15.40:0/1860033543 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69681346a0 msgr2=0x7f69681cae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:46.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.429+0000 7f69657fa640 1 --2- 172.21.15.40:0/1860033543 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69681346a0 0x7f69681cae10 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f695c009720 tx=0x7f695c0046c0 comp rx=0 tx=0).stop 2024-04-22T08:14:46.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.429+0000 7f696d66b640 1 -- 172.21.15.40:0/1860033543 >> v2:172.21.15.189:6800/30168 conn(0x7f694c0377d0 msgr2=0x7f694c039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-22T08:14:46.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.429+0000 7f69657fa640 1 -- 172.21.15.40:0/1860033543 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69681346a0 msgr2=0x7f69681cae10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:14:46.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.429+0000 7f696d66b640 1 -- 172.21.15.40:0/1860033543 >> v2:172.21.15.189:6800/30168 conn(0x7f694c0377d0 msgr2=0x7f694c039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:14:46.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.429+0000 7f696d66b640 1 --2- 172.21.15.40:0/1860033543 >> v2:172.21.15.189:6800/30168 conn(0x7f694c0377d0 0x7f694c039c90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f69600055a0 tx=0x7f6960005530 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:14:46.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.429+0000 7f69657fa640 1 --2- 172.21.15.40:0/1860033543 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69681346a0 0x7f69681cae10 secure :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f695c009720 tx=0x7f695c0046c0 comp rx=0 tx=0).stop 2024-04-22T08:14:46.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.429+0000 7f696d66b640 1 --2- 172.21.15.40:0/1860033543 >> v2:172.21.15.189:6800/30168 conn(0x7f694c0377d0 0x7f694c039c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:14:46.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.435+0000 7f696f0ce640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:14:46.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.435+0000 7f696f0ce640 1 --2- 172.21.15.40:0/1860033543 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69681cb350 0x7f69681cb760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:46.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.435+0000 7f696f0ce640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:46.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.435+0000 7f696f0ce640 1 -- 172.21.15.40:0/1860033543 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69681cbca0 con 0x7f69681cb350 2024-04-22T08:14:46.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.435+0000 7f696ce6a640 1 --2- 172.21.15.40:0/1860033543 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69681cb350 0x7f69681cb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:46.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.435+0000 7f696ce6a640 10 monclient: get_auth_request con 0x7f69681cb350 auth_method 0 2024-04-22T08:14:46.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.435+0000 7f696ce6a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:46.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.435+0000 7f696ce6a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:46.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.435+0000 7f696ce6a640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:46.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.435+0000 7f696ce6a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:46.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.435+0000 7f696ce6a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:46.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.435+0000 7f696ce6a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:46.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.435+0000 7f696ce6a640 10 monclient(hunting): handle_auth_done global_id 4778 payload 824 2024-04-22T08:14:46.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.435+0000 7f696ce6a640 1 -- 172.21.15.40:0/1860033543 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6968075260 con 0x7f69681cb350 2024-04-22T08:14:46.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.438+0000 7f696ce6a640 1 -- 172.21.15.40:0/1860033543 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69681cb350 msgr2=0x7f69681cb760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:46.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.438+0000 7f696ce6a640 1 --2- 172.21.15.40:0/1860033543 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69681cb350 0x7f69681cb760 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f695c046000 tx=0x7f695c031000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:46.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.438+0000 7f69657fa640 1 -- 172.21.15.40:0/1860033543 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f695c051b50 con 0x7f69681cb350 2024-04-22T08:14:46.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.438+0000 7f69657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:46.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.438+0000 7f69657fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:14:46.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.438+0000 7f69657fa640 10 monclient: dump: 2024-04-22T08:14:46.449 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:46.449 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:46.449 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:46.449 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:46.450 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:46.450 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:46.450 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:46.450 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:46.450 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:46.450 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:46.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.438+0000 7f69657fa640 1 -- 172.21.15.40:0/1860033543 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4631 (secure 0 0 0) 0x7f695c051cb0 con 0x7f69681cb350 2024-04-22T08:14:46.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.438+0000 7f69657fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:14:46.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.438+0000 7f69657fa640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:14:46.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.438+0000 7f69657fa640 1 -- 172.21.15.40:0/1860033543 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69681cb350 msgr2=0x7f69681cb760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:46.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.438+0000 7f69657fa640 1 --2- 172.21.15.40:0/1860033543 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69681cb350 0x7f69681cb760 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f695c046000 tx=0x7f695c031000 comp rx=0 tx=0).stop 2024-04-22T08:14:46.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.438+0000 7f69657fa640 1 -- 172.21.15.40:0/1860033543 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69681cb350 msgr2=0x7f69681cb760 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:14:46.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.438+0000 7f69657fa640 1 --2- 172.21.15.40:0/1860033543 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69681cb350 0x7f69681cb760 secure :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f695c046000 tx=0x7f695c031000 comp rx=0 tx=0).stop 2024-04-22T08:14:46.451 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:14:46.451 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:14:46.451 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:14:46.451 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:14:46.451 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 26, 2024-04-22T08:14:46.451 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:14:46.451 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:14:46.451 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:14:46.451 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:14:46.451 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:14:46.451 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 3, 2024-04-22T08:14:46.451 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:14:46.452 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:14:46.452 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:14:46.452 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:14:46.452 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:14:46.452 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:14:46.452 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:14:46.452 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:14:46.452 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:14:46.452 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:14:46.452 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:14:46.452 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:14:46.452 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:14:46.452 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:14:46.453 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:14:46.453 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:14:46.453 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:14:46.453 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:14:46.453 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:14:46.453 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:14:46.453 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:14:46.453 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:14:46.453 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:14:46.453 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:14:46.453 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:14:46.453 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:14:46.453 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:46.453 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:46.453 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:14:46.453 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:14:46.454 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:14:46.454 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:14:46.454 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:14:46.454 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:14:46.454 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:14:46.454 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:14:46.454 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:14:46.454 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:14:46.454 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:14:46.454 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:14:46.454 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:14:46.454 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:14:46.454 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:14:46.454 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:14:46.455 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:14:46.455 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:14:46.455 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:14:46.455 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:14:46.455 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:14:46.455 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:14:46.455 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:14:46.455 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:14:46.455 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:14:46.455 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:14:46.455 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:14:46.455 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:14:46.455 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:14:46.455 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:46.456 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:14:46.456 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:46.456 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:14:46.456 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:14:46.456 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:14:46.456 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:14:46.456 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:46.456 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:14:46.456 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:14:46.456 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:46.456 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:46.456 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:46.456 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:14:46.456 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:14:46.456 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:46.457 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:46.457 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:46.457 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:46.457 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:14:46.457 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:14:46.457 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:14:46.457 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:14:46.457 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:14:46.457 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:46.457 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:46.457 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:14:46.457 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:14:46.458 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:14:46.458 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:14:46.458 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:46.458 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:14:46.458 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:14:46.458 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:46.458 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:46.458 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:46.458 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:14:46.458 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:14:46.458 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:46.458 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:46.458 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:46.458 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:46.459 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:14:46.459 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:14:46.459 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:14:46.459 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:14:46.459 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:14:46.459 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:46.459 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:46.459 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:14:46.459 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:14:46.459 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:14:46.459 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:14:46.459 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:46.459 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:14:46.459 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:14:46.459 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:46.460 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:46.460 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:46.460 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:14:46.460 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:14:46.460 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:46.460 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:46.460 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:46.460 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:46.460 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:14:46.460 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:14:46.460 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:14:46.460 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:14:46.460 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:14:46.460 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:46.460 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:46.460 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:46.461 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:14:46.461 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:14:46.461 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:46.461 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:14:46.461 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:14:46.461 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:14:46.461 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:46.461 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:14:46.461 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:14:46.461 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:46.461 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:14:46.461 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:14:46.461 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 5 2024-04-22T08:14:46.461 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:46.461 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:14:46.461 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:14:46.462 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:46.462 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:14:46.462 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:14:46.462 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 3 2024-04-22T08:14:46.462 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:46.462 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:46.462 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:46.462 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:14:46.462 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:14:46.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.438+0000 7f696f0ce640 1 -- 172.21.15.40:0/1860033543 >> v2:172.21.15.189:6800/30168 conn(0x7f694c0377d0 msgr2=0x7f694c039c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:14:46.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.438+0000 7f696f0ce640 1 --2- 172.21.15.40:0/1860033543 >> v2:172.21.15.189:6800/30168 conn(0x7f694c0377d0 0x7f694c039c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:46.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.438+0000 7f696f0ce640 10 monclient: shutdown 2024-04-22T08:14:46.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.438+0000 7f696f0ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:14:46.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.438+0000 7f696f0ce640 1 -- 172.21.15.40:0/1860033543 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6968136230 msgr2=0x7f6968131920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:46.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.438+0000 7f696d66b640 1 -- 172.21.15.40:0/1860033543 reap_dead start 2024-04-22T08:14:46.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.438+0000 7f696f0ce640 1 --2- 172.21.15.40:0/1860033543 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6968136230 0x7f6968131920 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f695400c440 tx=0x7f695400c900 comp rx=0 tx=0).stop 2024-04-22T08:14:46.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.439+0000 7f696f0ce640 1 -- 172.21.15.40:0/1860033543 shutdown_connections 2024-04-22T08:14:46.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.439+0000 7f696f0ce640 1 --2- 172.21.15.40:0/1860033543 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6968136230 0x7f6968131920 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:46.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.439+0000 7f696f0ce640 1 -- 172.21.15.40:0/1860033543 >> 172.21.15.40:0/1860033543 conn(0x7f69680a4e80 msgr2=0x7f69680a08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:14:46.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.439+0000 7f696f0ce640 1 -- 172.21.15.40:0/1860033543 shutdown_connections 2024-04-22T08:14:46.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.439+0000 7f696f0ce640 1 -- 172.21.15.40:0/1860033543 wait complete. 2024-04-22T08:14:46.465 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:14:46.465 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-22T08:14:46.465 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-22T08:14:46.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.566+0000 7fad8a2c7640 10 monclient: get_monmap_and_config 2024-04-22T08:14:46.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.566+0000 7fad8a2c7640 10 monclient: build_initial_monmap 2024-04-22T08:14:46.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.566+0000 7fad8a2c7640 10 monclient: monmap: 2024-04-22T08:14:46.566 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:14:46.566 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:46.566 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:14:46.566810+0000 2024-04-22T08:14:46.567 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:14:46.566810+0000 2024-04-22T08:14:46.567 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:14:46.567 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:46.567 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:14:46.567 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:14:46.567 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:14:46.567 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:14:46.567 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:14:46.567 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:46.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.566+0000 7fad8a2c7640 1 Processor -- start 2024-04-22T08:14:46.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.566+0000 7fad8a2c7640 1 -- start start 2024-04-22T08:14:46.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.566+0000 7fad8a2c7640 10 monclient: init 2024-04-22T08:14:46.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad8a2c7640 10 monclient: _reopen_session rank -1 2024-04-22T08:14:46.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad8a2c7640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-04-22T08:14:46.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad8a2c7640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fad840a1ba0 0x7fad840a1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:46.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad8a2c7640 10 monclient(hunting): picked mon.noname-c con 0x7fad840a1ba0 addr v2:172.21.15.189:3300/0 2024-04-22T08:14:46.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad8a2c7640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fad84136230 0x7fad840a2d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:46.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad8a2c7640 10 monclient(hunting): picked mon.noname-a con 0x7fad84136230 addr v2:172.21.15.40:3300/0 2024-04-22T08:14:46.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad8a2c7640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fad84136670 0x7fad840a32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:46.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad8a2c7640 10 monclient(hunting): picked mon.noname-b con 0x7fad84136670 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:14:46.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad8a2c7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:46.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad8a2c7640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad840722a0 con 0x7fad84136230 2024-04-22T08:14:46.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad8a2c7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:46.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad8a2c7640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad840a4290 con 0x7fad840a1ba0 2024-04-22T08:14:46.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad8a2c7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:46.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad8a2c7640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad8409fe80 con 0x7fad84136670 2024-04-22T08:14:46.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad8a2c7640 10 monclient(hunting): _renew_subs 2024-04-22T08:14:46.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad8a2c7640 10 monclient(hunting): authenticate will time out at 960.560992s 2024-04-22T08:14:46.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad88864640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fad84136670 0x7fad840a32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:46.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad837fe640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fad84136230 0x7fad840a2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:46.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad88864640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fad84136670 0x7fad840a32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49592/0 (socket says 172.21.15.40:49592) 2024-04-22T08:14:46.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad837fe640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fad84136230 0x7fad840a2d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35156/0 (socket says 172.21.15.40:35156) 2024-04-22T08:14:46.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad88864640 1 -- 172.21.15.40:0/1184584436 learned_addr learned my addr 172.21.15.40:0/1184584436 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:14:46.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad88864640 10 monclient(hunting): get_auth_request con 0x7fad84136670 auth_method 0 2024-04-22T08:14:46.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad88864640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:46.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad88864640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:46.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad88864640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:46.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad837fe640 10 monclient(hunting): get_auth_request con 0x7fad84136230 auth_method 0 2024-04-22T08:14:46.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:46.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad837fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:46.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad837fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:46.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad88864640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:46.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad88864640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:46.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad88864640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:46.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:46.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:46.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:46.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad83fff640 1 --2- 172.21.15.40:0/1184584436 >> v2:172.21.15.189:3300/0 conn(0x7fad840a1ba0 0x7fad840a1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:46.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad83fff640 10 monclient(hunting): get_auth_request con 0x7fad840a1ba0 auth_method 0 2024-04-22T08:14:46.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:46.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad83fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:46.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad83fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:46.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad88864640 10 monclient(hunting): handle_auth_done global_id 4781 payload 274 2024-04-22T08:14:46.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad88864640 1 -- 172.21.15.40:0/1184584436 >> v2:172.21.15.189:3300/0 conn(0x7fad840a1ba0 msgr2=0x7fad840a1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:14:46.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad88864640 1 --2- 172.21.15.40:0/1184584436 >> v2:172.21.15.189:3300/0 conn(0x7fad840a1ba0 0x7fad840a1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:46.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad88864640 1 -- 172.21.15.40:0/1184584436 >> v2:172.21.15.40:3300/0 conn(0x7fad84136230 msgr2=0x7fad840a2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:14:46.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad88864640 1 --2- 172.21.15.40:0/1184584436 >> v2:172.21.15.40:3300/0 conn(0x7fad84136230 0x7fad840a2d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:46.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad88864640 10 monclient: _finish_hunting 0 2024-04-22T08:14:46.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad88864640 1 monclient: found mon.noname-b 2024-04-22T08:14:46.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad88864640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:14:46.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad88864640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:14:46.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad88864640 1 -- 172.21.15.40:0/1184584436 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad840a0160 con 0x7fad84136670 2024-04-22T08:14:46.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.567+0000 7fad837fe640 1 --2- 172.21.15.40:0/1184584436 >> v2:172.21.15.40:3300/0 conn(0x7fad84136230 0x7fad840a2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:14:46.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.570+0000 7fad88864640 1 -- 172.21.15.40:0/1184584436 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fad84136670 msgr2=0x7fad840a32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:46.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.570+0000 7fad88864640 1 --2- 172.21.15.40:0/1184584436 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fad84136670 0x7fad840a32c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fad840a5c50 tx=0x7fad7c02be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=184975ad1c73e592 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:46.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.570+0000 7fad82ffd640 1 -- 172.21.15.40:0/1184584436 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad7c02cbf0 con 0x7fad84136670 2024-04-22T08:14:46.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.570+0000 7fad82ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:46.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.570+0000 7fad82ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:14:46.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.570+0000 7fad82ffd640 10 monclient: dump: 2024-04-22T08:14:46.579 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:46.580 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:46.580 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:46.580 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:46.580 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:46.580 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:46.580 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:46.580 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:46.580 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:46.580 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:46.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.570+0000 7fad82ffd640 10 monclient: _finish_auth 0 2024-04-22T08:14:46.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.570+0000 7fad82ffd640 10 monclient: _check_auth_tickets 2024-04-22T08:14:46.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.570+0000 7fad82ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:14:46.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.570+0000 7fad8a2c7640 5 monclient: authenticate success, global_id 4781 2024-04-22T08:14:46.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.570+0000 7fad8a2c7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:14:46.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.570+0000 7fad82ffd640 1 -- 172.21.15.40:0/1184584436 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad7c02cd50 con 0x7fad84136670 2024-04-22T08:14:46.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.570+0000 7fad82ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:14:46.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.570+0000 7fad8a2c7640 10 monclient: get_monmap_and_config success 2024-04-22T08:14:46.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.570+0000 7fad8a2c7640 4 set_mon_vals no callback set 2024-04-22T08:14:46.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.570+0000 7fad82ffd640 1 -- 172.21.15.40:0/1184584436 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad7c034610 con 0x7fad84136670 2024-04-22T08:14:46.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.570+0000 7fad82ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:46.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.570+0000 7fad82ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:14:46.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.570+0000 7fad82ffd640 10 monclient: dump: 2024-04-22T08:14:46.581 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:46.581 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:46.581 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:46.582 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:46.582 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:46.582 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:46.582 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:46.582 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:46.582 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:46.582 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:46.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.571+0000 7fad8a2c7640 10 monclient: shutdown 2024-04-22T08:14:46.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.571+0000 7fad8a2c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:14:46.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.571+0000 7fad8a2c7640 1 -- 172.21.15.40:0/1184584436 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fad84136670 msgr2=0x7fad840a32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:46.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.571+0000 7fad8a2c7640 1 --2- 172.21.15.40:0/1184584436 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fad84136670 0x7fad840a32c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fad840a5c50 tx=0x7fad7c02be40 comp rx=0 tx=0).stop 2024-04-22T08:14:46.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.571+0000 7fad8a2c7640 1 -- 172.21.15.40:0/1184584436 shutdown_connections 2024-04-22T08:14:46.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.571+0000 7fad8a2c7640 1 --2- 172.21.15.40:0/1184584436 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fad84136670 0x7fad840a32c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:46.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.571+0000 7fad8a2c7640 1 --2- 172.21.15.40:0/1184584436 >> v2:172.21.15.40:3300/0 conn(0x7fad84136230 0x7fad840a2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:46.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.571+0000 7fad8a2c7640 1 --2- 172.21.15.40:0/1184584436 >> v2:172.21.15.189:3300/0 conn(0x7fad840a1ba0 0x7fad840a1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:46.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.571+0000 7fad8a2c7640 1 -- 172.21.15.40:0/1184584436 >> 172.21.15.40:0/1184584436 conn(0x7fad840a4e80 msgr2=0x7fad840a5290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:14:46.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.571+0000 7fad8a2c7640 1 -- 172.21.15.40:0/1184584436 shutdown_connections 2024-04-22T08:14:46.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.571+0000 7fad8a2c7640 1 -- 172.21.15.40:0/1184584436 wait complete. 2024-04-22T08:14:46.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.571+0000 7fad8a2c7640 10 monclient: build_initial_monmap 2024-04-22T08:14:46.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.571+0000 7fad8a2c7640 10 monclient: monmap: 2024-04-22T08:14:46.583 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:14:46.583 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:14:46.583 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:14:46.583 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:14:46.584 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:14:46.584 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:46.584 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:14:46.584 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:14:46.584 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:14:46.584 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:46.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.572+0000 7fad8a2c7640 1 Processor -- start 2024-04-22T08:14:46.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.572+0000 7fad8a2c7640 1 -- start start 2024-04-22T08:14:46.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.572+0000 7fad8a2c7640 10 monclient: init 2024-04-22T08:14:46.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.572+0000 7fad8a2c7640 10 monclient: _reopen_session rank -1 2024-04-22T08:14:46.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.572+0000 7fad8a2c7640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:14:46.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.572+0000 7fad8a2c7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad840a1ba0 0x7fad8411eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:46.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.572+0000 7fad8a2c7640 10 monclient(hunting): picked mon.noname-a con 0x7fad840a1ba0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:14:46.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.572+0000 7fad8a2c7640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad84136230 0x7fad84133bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:46.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.572+0000 7fad8a2c7640 10 monclient(hunting): picked mon.noname-b con 0x7fad84136230 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:14:46.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.572+0000 7fad8a2c7640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fad84136670 0x7fad84134130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:46.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.572+0000 7fad8a2c7640 10 monclient(hunting): picked mon.noname-c con 0x7fad84136670 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:14:46.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.572+0000 7fad8a2c7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:46.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.572+0000 7fad8a2c7640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad84135060 con 0x7fad840a1ba0 2024-04-22T08:14:46.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.572+0000 7fad8a2c7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:46.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.572+0000 7fad8a2c7640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad8411fdb0 con 0x7fad84136230 2024-04-22T08:14:46.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.572+0000 7fad8a2c7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:46.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.572+0000 7fad8a2c7640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad84120d10 con 0x7fad84136670 2024-04-22T08:14:46.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.572+0000 7fad8a2c7640 10 monclient(hunting): _renew_subs 2024-04-22T08:14:46.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.572+0000 7fad8a2c7640 10 monclient(hunting): authenticate will time out at 960.566689s 2024-04-22T08:14:46.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.573+0000 7fad88864640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fad84136670 0x7fad84134130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:46.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.573+0000 7fad83fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad840a1ba0 0x7fad8411eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:46.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.573+0000 7fad88864640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fad84136670 0x7fad84134130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49608/0 (socket says 172.21.15.40:49608) 2024-04-22T08:14:46.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.573+0000 7fad83fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad840a1ba0 0x7fad8411eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35172/0 (socket says 172.21.15.40:35172) 2024-04-22T08:14:46.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.573+0000 7fad88864640 1 -- 172.21.15.40:0/2355621812 learned_addr learned my addr 172.21.15.40:0/2355621812 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:14:46.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.573+0000 7fad88864640 10 monclient(hunting): get_auth_request con 0x7fad84136670 auth_method 0 2024-04-22T08:14:46.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.573+0000 7fad88864640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:46.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.573+0000 7fad88864640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:46.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.573+0000 7fad88864640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:46.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.573+0000 7fad83fff640 10 monclient(hunting): get_auth_request con 0x7fad840a1ba0 auth_method 0 2024-04-22T08:14:46.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.573+0000 7fad83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:46.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.573+0000 7fad83fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:46.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.573+0000 7fad83fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:46.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.573+0000 7fad88864640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:46.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.573+0000 7fad88864640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:46.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.573+0000 7fad837fe640 1 --2- 172.21.15.40:0/2355621812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad84136230 0x7fad84133bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:46.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.573+0000 7fad88864640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:46.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.573+0000 7fad83fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:46.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.573+0000 7fad83fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:46.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.573+0000 7fad83fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:46.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.573+0000 7fad837fe640 10 monclient(hunting): get_auth_request con 0x7fad84136230 auth_method 0 2024-04-22T08:14:46.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.573+0000 7fad837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:46.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.573+0000 7fad837fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:46.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.573+0000 7fad837fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:46.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.573+0000 7fad837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:46.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.573+0000 7fad837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:46.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.573+0000 7fad837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:46.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.573+0000 7fad83fff640 10 monclient(hunting): handle_auth_done global_id 34167 payload 824 2024-04-22T08:14:46.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.573+0000 7fad83fff640 1 -- 172.21.15.40:0/2355621812 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fad84136670 msgr2=0x7fad84134130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:46.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.573+0000 7fad83fff640 1 --2- 172.21.15.40:0/2355621812 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fad84136670 0x7fad84134130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:46.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.573+0000 7fad83fff640 1 -- 172.21.15.40:0/2355621812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad84136230 msgr2=0x7fad84133bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:46.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.573+0000 7fad83fff640 1 --2- 172.21.15.40:0/2355621812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad84136230 0x7fad84133bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:46.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.573+0000 7fad83fff640 10 monclient: _finish_hunting 0 2024-04-22T08:14:46.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.573+0000 7fad83fff640 1 monclient: found mon.noname-a 2024-04-22T08:14:46.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.573+0000 7fad83fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:14:46.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.573+0000 7fad83fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:14:46.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.573+0000 7fad83fff640 1 -- 172.21.15.40:0/2355621812 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad78009400 con 0x7fad840a1ba0 2024-04-22T08:14:46.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.573+0000 7fad88864640 1 --2- 172.21.15.40:0/2355621812 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fad84136670 0x7fad84134130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:14:46.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.575+0000 7fad83fff640 1 -- 172.21.15.40:0/2355621812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad840a1ba0 msgr2=0x7fad8411eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:46.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.575+0000 7fad83fff640 1 --2- 172.21.15.40:0/2355621812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad840a1ba0 0x7fad8411eeb0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fad7400d5f0 tx=0x7fad7400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:46.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.575+0000 7fad80ff9640 1 -- 172.21.15.40:0/2355621812 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad74012800 con 0x7fad840a1ba0 2024-04-22T08:14:46.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.575+0000 7fad80ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:46.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.575+0000 7fad80ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:14:46.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.575+0000 7fad80ff9640 10 monclient: dump: 2024-04-22T08:14:46.594 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:46.594 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:46.594 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:46.594 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:46.595 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:46.595 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:46.595 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:46.595 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:46.595 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:46.595 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:46.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.575+0000 7fad80ff9640 10 monclient: _finish_auth 0 2024-04-22T08:14:46.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.575+0000 7fad80ff9640 10 monclient: _check_auth_tickets 2024-04-22T08:14:46.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.575+0000 7fad80ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:14:46.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.575+0000 7fad8a2c7640 5 monclient: authenticate success, global_id 34167 2024-04-22T08:14:46.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.575+0000 7fad80ff9640 1 -- 172.21.15.40:0/2355621812 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad74012960 con 0x7fad840a1ba0 2024-04-22T08:14:46.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.575+0000 7fad80ff9640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:14:46.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.575+0000 7fad80ff9640 1 -- 172.21.15.40:0/2355621812 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad74012d10 con 0x7fad840a1ba0 2024-04-22T08:14:46.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.575+0000 7fad8a2c7640 10 monclient: _renew_subs 2024-04-22T08:14:46.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.575+0000 7fad8a2c7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:14:46.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.575+0000 7fad8a2c7640 1 -- 172.21.15.40:0/2355621812 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad7c009b30 con 0x7fad840a1ba0 2024-04-22T08:14:46.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.575+0000 7fad827fc640 4 set_mon_vals no callback set 2024-04-22T08:14:46.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.575+0000 7fad80ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:46.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.575+0000 7fad80ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:14:46.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.575+0000 7fad80ff9640 10 monclient: dump: 2024-04-22T08:14:46.596 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:46.596 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:46.596 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:46.596 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:46.596 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:46.597 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:46.597 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:46.597 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:46.597 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:46.597 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:46.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.575+0000 7fad8a2c7640 10 monclient: _renew_subs 2024-04-22T08:14:46.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.575+0000 7fad8a2c7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:14:46.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.575+0000 7fad8a2c7640 1 -- 172.21.15.40:0/2355621812 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad841212d0 con 0x7fad840a1ba0 2024-04-22T08:14:46.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.576+0000 7fad8a2c7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:14:46.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.576+0000 7fad8a2c7640 1 --2- 172.21.15.40:0/2355621812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad841ca3c0 0x7fad841ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:46.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.576+0000 7fad8a2c7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:46.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.576+0000 7fad8a2c7640 1 -- 172.21.15.40:0/2355621812 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad84066650 con 0x7fad841ca3c0 2024-04-22T08:14:46.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.576+0000 7fad80ff9640 1 -- 172.21.15.40:0/2355621812 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fad7401e460 con 0x7fad840a1ba0 2024-04-22T08:14:46.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.577+0000 7fad837fe640 1 --2- 172.21.15.40:0/2355621812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad841ca3c0 0x7fad841ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:46.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.577+0000 7fad80ff9640 1 --2- 172.21.15.40:0/2355621812 >> v2:172.21.15.189:6800/30168 conn(0x7fad700377d0 0x7fad70039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:46.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.577+0000 7fad837fe640 10 monclient: get_auth_request con 0x7fad841ca3c0 auth_method 0 2024-04-22T08:14:46.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.577+0000 7fad837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:46.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.577+0000 7fad837fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:46.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.577+0000 7fad837fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:46.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.577+0000 7fad80ff9640 1 -- 172.21.15.40:0/2355621812 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 3898+0+0 (secure 0 0 0) 0x7fad7404a8b0 con 0x7fad840a1ba0 2024-04-22T08:14:46.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.577+0000 7fad837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:46.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.577+0000 7fad837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:46.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.577+0000 7fad837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:46.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.577+0000 7fad88864640 1 --2- 172.21.15.40:0/2355621812 >> v2:172.21.15.189:6800/30168 conn(0x7fad700377d0 0x7fad70039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:46.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.577+0000 7fad88864640 10 monclient: get_auth_request con 0x7fad700377d0 auth_method 0 2024-04-22T08:14:46.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.577+0000 7fad837fe640 10 monclient(hunting): handle_auth_done global_id 24367 payload 824 2024-04-22T08:14:46.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.577+0000 7fad837fe640 1 -- 172.21.15.40:0/2355621812 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fad840a24c0 con 0x7fad841ca3c0 2024-04-22T08:14:46.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.579+0000 7fad837fe640 1 -- 172.21.15.40:0/2355621812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad841ca3c0 msgr2=0x7fad841ca7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:46.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.579+0000 7fad837fe640 1 --2- 172.21.15.40:0/2355621812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad841ca3c0 0x7fad841ca7a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fad84071d90 tx=0x7fad7800f610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:46.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.579+0000 7fad88864640 1 --2- 172.21.15.40:0/2355621812 >> v2:172.21.15.189:6800/30168 conn(0x7fad700377d0 0x7fad70039c90 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fad8411fb40 tx=0x7fad7c030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:46.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.580+0000 7fad80ff9640 1 -- 172.21.15.40:0/2355621812 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad78011e80 con 0x7fad841ca3c0 2024-04-22T08:14:46.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.580+0000 7fad80ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:46.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.580+0000 7fad80ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:14:46.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.580+0000 7fad80ff9640 10 monclient: dump: 2024-04-22T08:14:46.599 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:46.599 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:46.600 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:46.600 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:46.600 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:46.600 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:46.600 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:46.600 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:46.600 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:46.600 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:46.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.580+0000 7fad80ff9640 1 -- 172.21.15.40:0/2355621812 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7fad78004240 con 0x7fad841ca3c0 2024-04-22T08:14:46.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.580+0000 7fad80ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:14:46.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.580+0000 7fad80ff9640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:14:46.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.580+0000 7fad80ff9640 1 -- 172.21.15.40:0/2355621812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad841ca3c0 msgr2=0x7fad841ca7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:46.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.580+0000 7fad80ff9640 1 --2- 172.21.15.40:0/2355621812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad841ca3c0 0x7fad841ca7a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fad84071d90 tx=0x7fad7800f610 comp rx=0 tx=0).stop 2024-04-22T08:14:46.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.580+0000 7fad80ff9640 1 -- 172.21.15.40:0/2355621812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad841ca3c0 msgr2=0x7fad841ca7a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:14:46.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.580+0000 7fad80ff9640 1 --2- 172.21.15.40:0/2355621812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad841ca3c0 0x7fad841ca7a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:46.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.586+0000 7fad8a2c7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:14:46.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.586+0000 7fad8a2c7640 1 --2- 172.21.15.40:0/2355621812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad841cace0 0x7fad8406cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:46.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.586+0000 7fad8a2c7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:46.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.586+0000 7fad8a2c7640 1 -- 172.21.15.40:0/2355621812 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad8404e9b0 con 0x7fad841cace0 2024-04-22T08:14:46.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.586+0000 7fad837fe640 1 --2- 172.21.15.40:0/2355621812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad841cace0 0x7fad8406cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:46.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.586+0000 7fad837fe640 10 monclient: get_auth_request con 0x7fad841cace0 auth_method 0 2024-04-22T08:14:46.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.587+0000 7fad837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:46.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.587+0000 7fad837fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:46.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.587+0000 7fad837fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:46.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.587+0000 7fad837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:46.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.587+0000 7fad837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:46.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.587+0000 7fad837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:46.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.587+0000 7fad837fe640 10 monclient(hunting): handle_auth_done global_id 24370 payload 824 2024-04-22T08:14:46.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.587+0000 7fad837fe640 1 -- 172.21.15.40:0/2355621812 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fad841cb0f0 con 0x7fad841cace0 2024-04-22T08:14:46.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.589+0000 7fad837fe640 1 -- 172.21.15.40:0/2355621812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad841cace0 msgr2=0x7fad8406cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:46.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.589+0000 7fad837fe640 1 --2- 172.21.15.40:0/2355621812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad841cace0 0x7fad8406cd70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fad7801b000 tx=0x7fad780046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:46.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.589+0000 7fad80ff9640 1 -- 172.21.15.40:0/2355621812 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad7801ea70 con 0x7fad841cace0 2024-04-22T08:14:46.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.589+0000 7fad80ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:46.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.589+0000 7fad80ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:14:46.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.589+0000 7fad80ff9640 10 monclient: dump: 2024-04-22T08:14:46.603 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:46.603 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:46.603 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:46.603 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:46.603 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:46.603 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:46.603 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:46.603 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:46.603 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:46.603 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:46.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.590+0000 7fad80ff9640 1 -- 172.21.15.40:0/2355621812 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4801 (secure 0 0 0) 0x7fad780283a0 con 0x7fad841cace0 2024-04-22T08:14:46.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.590+0000 7fad80ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:14:46.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.590+0000 7fad80ff9640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:14:46.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.590+0000 7fad80ff9640 1 -- 172.21.15.40:0/2355621812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad841cace0 msgr2=0x7fad8406cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:46.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.590+0000 7fad80ff9640 1 --2- 172.21.15.40:0/2355621812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad841cace0 0x7fad8406cd70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fad7801b000 tx=0x7fad780046c0 comp rx=0 tx=0).stop 2024-04-22T08:14:46.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.590+0000 7fad80ff9640 1 -- 172.21.15.40:0/2355621812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad841cace0 msgr2=0x7fad8406cd70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:14:46.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.590+0000 7fad80ff9640 1 --2- 172.21.15.40:0/2355621812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad841cace0 0x7fad8406cd70 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:46.604 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:14:46.604 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:14:46.604 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:14:46.605 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:14:46.605 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 26, 2024-04-22T08:14:46.605 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:14:46.605 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:14:46.605 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:14:46.605 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:14:46.605 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:14:46.605 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 3, 2024-04-22T08:14:46.605 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:14:46.605 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:14:46.606 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:14:46.606 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:14:46.606 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:14:46.606 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:14:46.606 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:14:46.606 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:14:46.606 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:14:46.606 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:14:46.606 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:14:46.606 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:14:46.606 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:14:46.606 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:14:46.607 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:14:46.607 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:14:46.607 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:14:46.607 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:14:46.607 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:14:46.607 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:14:46.607 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:14:46.607 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:14:46.607 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:14:46.607 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:14:46.607 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:14:46.607 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:14:46.607 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:46.608 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:46.608 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:14:46.608 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:14:46.608 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:14:46.608 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:14:46.608 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:14:46.608 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:14:46.608 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:14:46.608 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:14:46.608 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:14:46.608 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:14:46.608 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:14:46.609 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:14:46.609 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:14:46.609 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:14:46.609 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:14:46.609 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:14:46.609 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:14:46.609 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:14:46.609 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:14:46.609 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:14:46.609 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:14:46.609 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:14:46.609 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:14:46.609 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:14:46.610 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:14:46.610 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:14:46.610 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:14:46.610 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:14:46.610 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:14:46.610 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:46.610 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:14:46.610 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:46.610 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:14:46.610 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:14:46.610 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:14:46.611 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:14:46.611 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:46.611 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:14:46.611 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:14:46.611 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:46.611 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:46.611 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:46.611 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:14:46.611 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:14:46.611 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:46.611 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:46.611 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:46.612 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:46.612 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:14:46.612 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:14:46.612 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:14:46.612 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:14:46.612 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:14:46.612 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:46.612 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:46.612 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:14:46.612 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:14:46.612 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:14:46.612 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:14:46.613 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:46.613 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:14:46.613 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:14:46.613 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:46.613 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:46.613 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:46.613 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:14:46.613 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:14:46.613 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:46.613 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:46.613 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:46.613 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:46.613 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:14:46.614 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:14:46.614 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:14:46.614 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:14:46.614 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:14:46.614 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:46.614 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:46.614 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:14:46.614 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:14:46.614 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:14:46.614 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:14:46.614 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:46.614 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:14:46.615 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:14:46.615 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:46.615 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:46.615 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:46.615 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:14:46.615 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:14:46.615 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:14:46.615 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:46.615 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:46.615 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:46.615 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:14:46.616 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:14:46.616 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:14:46.616 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:14:46.616 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:14:46.616 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:46.616 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:46.616 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:46.616 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:14:46.616 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:14:46.616 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:46.617 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:14:46.617 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:14:46.617 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:14:46.617 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:46.617 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:14:46.617 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:14:46.617 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:46.617 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:14:46.617 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:14:46.617 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:14:46.617 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:46.618 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:14:46.618 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:14:46.618 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:46.618 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:14:46.618 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:14:46.618 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 6 2024-04-22T08:14:46.618 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:46.618 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:14:46.618 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": [ 2024-04-22T08:14:46.618 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:14:46.619 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:14:46.619 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:14:46.619 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:14:46.619 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:14:46.619 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:14:46.619 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:14:46.619 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:14:46.619 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:14:46.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.590+0000 7fad8a2c7640 1 -- 172.21.15.40:0/2355621812 >> v2:172.21.15.189:6800/30168 conn(0x7fad700377d0 msgr2=0x7fad70039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:46.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.590+0000 7fad8a2c7640 1 --2- 172.21.15.40:0/2355621812 >> v2:172.21.15.189:6800/30168 conn(0x7fad700377d0 0x7fad70039c90 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fad8411fb40 tx=0x7fad7c030000 comp rx=0 tx=0).stop 2024-04-22T08:14:46.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.590+0000 7fad8a2c7640 10 monclient: shutdown 2024-04-22T08:14:46.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.590+0000 7fad8a2c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:14:46.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.590+0000 7fad8a2c7640 1 -- 172.21.15.40:0/2355621812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad840a1ba0 msgr2=0x7fad8411eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:46.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.590+0000 7fad8a2c7640 1 --2- 172.21.15.40:0/2355621812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad840a1ba0 0x7fad8411eeb0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fad7400d5f0 tx=0x7fad7400dab0 comp rx=0 tx=0).stop 2024-04-22T08:14:46.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.590+0000 7fad88864640 1 -- 172.21.15.40:0/2355621812 reap_dead start 2024-04-22T08:14:46.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.590+0000 7fad88864640 1 -- 172.21.15.40:0/2355621812 reap_dead start 2024-04-22T08:14:46.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.590+0000 7fad8a2c7640 1 -- 172.21.15.40:0/2355621812 shutdown_connections 2024-04-22T08:14:46.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.590+0000 7fad8a2c7640 1 -- 172.21.15.40:0/2355621812 >> 172.21.15.40:0/2355621812 conn(0x7fad840a4e80 msgr2=0x7fad84135300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:14:46.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.590+0000 7fad8a2c7640 1 -- 172.21.15.40:0/2355621812 shutdown_connections 2024-04-22T08:14:46.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.590+0000 7fad8a2c7640 1 -- 172.21.15.40:0/2355621812 wait complete. 2024-04-22T08:14:46.622 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:14:46.622 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-04-22T08:14:46.622 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-04-22T08:14:46.622 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-04-22T08:14:46.622 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-04-22T08:14:46.623 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-22T08:14:46.623 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T08:14:46.635 INFO:tasks.ceph.mon.c:Stopped 2024-04-22T08:14:46.635 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-22T08:14:46.635 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:14:46.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.774+0000 7f6d503fd640 10 monclient: get_monmap_and_config 2024-04-22T08:14:46.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.774+0000 7f6d503fd640 10 monclient: build_initial_monmap 2024-04-22T08:14:46.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.774+0000 7f6d503fd640 10 monclient: monmap: 2024-04-22T08:14:46.774 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:14:46.774 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:46.774 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:14:46.774936+0000 2024-04-22T08:14:46.774 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:14:46.774936+0000 2024-04-22T08:14:46.774 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:14:46.774 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:46.774 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:14:46.774 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:14:46.775 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:14:46.775 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:14:46.775 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:14:46.775 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:46.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.774+0000 7f6d503fd640 1 Processor -- start 2024-04-22T08:14:46.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.774+0000 7f6d503fd640 1 -- start start 2024-04-22T08:14:46.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.774+0000 7f6d503fd640 10 monclient: init 2024-04-22T08:14:46.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.775+0000 7f6d503fd640 10 monclient: _reopen_session rank -1 2024-04-22T08:14:46.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.775+0000 7f6d503fd640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-04-22T08:14:46.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.775+0000 7f6d503fd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d4809b760 0x7f6d4809ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:46.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.775+0000 7f6d503fd640 10 monclient(hunting): picked mon.noname-b con 0x7f6d4809b760 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:14:46.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.775+0000 7f6d503fd640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6d4809ef90 0x7f6d4809f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:46.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.775+0000 7f6d503fd640 10 monclient(hunting): picked mon.noname-c con 0x7f6d4809ef90 addr v2:172.21.15.189:3300/0 2024-04-22T08:14:46.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.775+0000 7f6d503fd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6d4809f8f0 addr v1:172.21.15.40:6789/0 2024-04-22T08:14:46.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.775+0000 7f6d503fd640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d4809cf70 con 0x7f6d4809f8f0 2024-04-22T08:14:46.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.775+0000 7f6d503fd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:46.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.775+0000 7f6d503fd640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d4809c6d0 con 0x7f6d4809ef90 2024-04-22T08:14:46.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.775+0000 7f6d503fd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:46.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.775+0000 7f6d503fd640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4809c9b0 con 0x7f6d4809b760 2024-04-22T08:14:46.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.775+0000 7f6d503fd640 10 monclient(hunting): _renew_subs 2024-04-22T08:14:46.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.775+0000 7f6d4e199640 1 -- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d4809b760 msgr2=0x7f6d4809ea50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:14:46.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.775+0000 7f6d503fd640 10 monclient(hunting): authenticate will time out at 960.769241s 2024-04-22T08:14:46.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.775+0000 7f6d4e199640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d4809b760 0x7f6d4809ea50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:14:46.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.775+0000 7f6d4e99a640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f6d4809f8f0 0x7f6d48139960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:59008/0 (socket says 172.21.15.40:59008) 2024-04-22T08:14:46.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.775+0000 7f6d4e99a640 1 -- 172.21.15.40:0/2514025756 learned_addr learned my addr 172.21.15.40:0/2514025756 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:14:46.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.775+0000 7f6d4d998640 1 --2- 172.21.15.40:0/2514025756 >> v2:172.21.15.189:3300/0 conn(0x7f6d4809ef90 0x7f6d4809f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:46.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.775+0000 7f6d4d998640 10 monclient(hunting): get_auth_request con 0x7f6d4809ef90 auth_method 0 2024-04-22T08:14:46.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.775+0000 7f6d4d998640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:46.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.775+0000 7f6d4d998640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:46.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.775+0000 7f6d4d998640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:46.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.775+0000 7f6d4d998640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:46.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.775+0000 7f6d4d998640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:46.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.775+0000 7f6d4d998640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:46.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.776+0000 7f6d4d998640 10 monclient(hunting): handle_auth_done global_id 24373 payload 274 2024-04-22T08:14:46.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.776+0000 7f6d4d998640 1 -- 172.21.15.40:0/2514025756 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d4809b760 msgr2=0x7f6d4809ea50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:14:46.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.776+0000 7f6d4d998640 1 --2- 172.21.15.40:0/2514025756 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d4809b760 0x7f6d4809ea50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:46.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.776+0000 7f6d4d998640 1 -- 172.21.15.40:0/2514025756 >> v1:172.21.15.40:6789/0 conn(0x7f6d4809f8f0 legacy=0x7f6d48139960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:14:46.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.776+0000 7f6d4d998640 10 monclient: _finish_hunting 0 2024-04-22T08:14:46.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.776+0000 7f6d4d998640 1 monclient: found mon.noname-c 2024-04-22T08:14:46.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.776+0000 7f6d4d998640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:14:46.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.776+0000 7f6d4d998640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.189:3300/0 2024-04-22T08:14:46.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.776+0000 7f6d4d998640 1 -- 172.21.15.40:0/2514025756 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d4809cc90 con 0x7f6d4809ef90 2024-04-22T08:14:46.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.779+0000 7f6d4d998640 1 -- 172.21.15.40:0/2514025756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d4809ef90 msgr2=0x7f6d4809f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:46.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.779+0000 7f6d4d998640 1 --2- 172.21.15.40:0/2514025756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d4809ef90 0x7f6d4809f3b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6d380097a0 tx=0x7f6d3802b730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3548fee7418adf60 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:46.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.779+0000 7f6d4d197640 1 -- 172.21.15.40:0/2514025756 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d38030a00 con 0x7f6d4809ef90 2024-04-22T08:14:46.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.779+0000 7f6d4d197640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:46.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.779+0000 7f6d4d197640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:14:46.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.779+0000 7f6d4d197640 10 monclient: dump: 2024-04-22T08:14:46.787 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:46.787 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:46.787 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:46.787 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:46.787 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:46.787 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:46.788 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:46.788 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:46.788 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:46.788 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:46.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.779+0000 7f6d4d197640 10 monclient: _finish_auth 0 2024-04-22T08:14:46.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.779+0000 7f6d4d197640 10 monclient: _check_auth_tickets 2024-04-22T08:14:46.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.779+0000 7f6d4d197640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:14:46.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.779+0000 7f6d4d197640 1 -- 172.21.15.40:0/2514025756 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d3802e050 con 0x7f6d4809ef90 2024-04-22T08:14:46.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.779+0000 7f6d4d197640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:14:46.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.779+0000 7f6d4d197640 1 -- 172.21.15.40:0/2514025756 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d38030d80 con 0x7f6d4809ef90 2024-04-22T08:14:46.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.779+0000 7f6d4d197640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:46.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.779+0000 7f6d4d197640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:14:46.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.779+0000 7f6d4d197640 10 monclient: dump: 2024-04-22T08:14:46.789 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:46.789 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:46.789 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:46.789 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:46.789 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:46.789 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:46.789 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:46.789 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:46.789 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:46.790 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:46.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.779+0000 7f6d503fd640 5 monclient: authenticate success, global_id 24373 2024-04-22T08:14:46.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.779+0000 7f6d503fd640 10 monclient: get_monmap_and_config success 2024-04-22T08:14:46.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.779+0000 7f6d503fd640 4 set_mon_vals no callback set 2024-04-22T08:14:46.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.779+0000 7f6d503fd640 10 monclient: shutdown 2024-04-22T08:14:46.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.779+0000 7f6d503fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:14:46.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.779+0000 7f6d503fd640 1 -- 172.21.15.40:0/2514025756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d4809ef90 msgr2=0x7f6d4809f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:46.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.779+0000 7f6d503fd640 1 --2- 172.21.15.40:0/2514025756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d4809ef90 0x7f6d4809f3b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6d380097a0 tx=0x7f6d3802b730 comp rx=0 tx=0).stop 2024-04-22T08:14:46.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.780+0000 7f6d503fd640 1 -- 172.21.15.40:0/2514025756 shutdown_connections 2024-04-22T08:14:46.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.780+0000 7f6d503fd640 1 --2- 172.21.15.40:0/2514025756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d4809ef90 0x7f6d4809f3b0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:46.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.780+0000 7f6d503fd640 1 --2- 172.21.15.40:0/2514025756 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d4809b760 0x7f6d4809ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:46.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.780+0000 7f6d503fd640 1 -- 172.21.15.40:0/2514025756 >> 172.21.15.40:0/2514025756 conn(0x7f6d480a0b50 msgr2=0x7f6d480a0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:14:46.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.780+0000 7f6d503fd640 1 -- 172.21.15.40:0/2514025756 shutdown_connections 2024-04-22T08:14:46.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.780+0000 7f6d503fd640 1 -- 172.21.15.40:0/2514025756 wait complete. 2024-04-22T08:14:46.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.780+0000 7f6d503fd640 10 monclient: build_initial_monmap 2024-04-22T08:14:46.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.780+0000 7f6d503fd640 10 monclient: monmap: 2024-04-22T08:14:46.791 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:14:46.791 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:14:46.791 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:14:46.791 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:14:46.791 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:14:46.791 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:46.792 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:14:46.792 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:14:46.792 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:14:46.792 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:46.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.780+0000 7f6d503fd640 1 Processor -- start 2024-04-22T08:14:46.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.781+0000 7f6d503fd640 1 -- start start 2024-04-22T08:14:46.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.781+0000 7f6d503fd640 10 monclient: init 2024-04-22T08:14:46.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.781+0000 7f6d503fd640 10 monclient: _reopen_session rank -1 2024-04-22T08:14:46.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.781+0000 7f6d503fd640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-22T08:14:46.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.781+0000 7f6d503fd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d4809ef90 0x7f6d481c02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:46.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.781+0000 7f6d503fd640 10 monclient(hunting): picked mon.noname-a con 0x7f6d4809ef90 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:14:46.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.781+0000 7f6d503fd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d4809f8f0 0x7f6d481c2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:46.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.781+0000 7f6d503fd640 10 monclient(hunting): picked mon.noname-c con 0x7f6d4809f8f0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:14:46.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.781+0000 7f6d503fd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d481c3470 0x7f6d481c6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:46.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.781+0000 7f6d503fd640 10 monclient(hunting): picked mon.noname-b con 0x7f6d481c3470 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:14:46.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.781+0000 7f6d503fd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:46.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.781+0000 7f6d503fd640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d481a7650 con 0x7f6d4809ef90 2024-04-22T08:14:46.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.781+0000 7f6d503fd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:46.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.781+0000 7f6d503fd640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4809bbc0 con 0x7f6d481c3470 2024-04-22T08:14:46.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.781+0000 7f6d503fd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:46.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.781+0000 7f6d503fd640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d481c38f0 con 0x7f6d4809f8f0 2024-04-22T08:14:46.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.781+0000 7f6d503fd640 10 monclient(hunting): _renew_subs 2024-04-22T08:14:46.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.781+0000 7f6d4d998640 1 -- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d4809f8f0 msgr2=0x7f6d481c2f30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:14:46.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.781+0000 7f6d503fd640 10 monclient(hunting): authenticate will time out at 960.775302s 2024-04-22T08:14:46.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.781+0000 7f6d4d998640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d4809f8f0 0x7f6d481c2f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:14:46.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.781+0000 7f6d4e199640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d4809ef90 0x7f6d481c02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:46.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.781+0000 7f6d4e199640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d4809ef90 0x7f6d481c02b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35206/0 (socket says 172.21.15.40:35206) 2024-04-22T08:14:46.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.781+0000 7f6d4e199640 1 -- 172.21.15.40:0/3599378576 learned_addr learned my addr 172.21.15.40:0/3599378576 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:14:46.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.781+0000 7f6d4e199640 10 monclient(hunting): get_auth_request con 0x7f6d4809ef90 auth_method 0 2024-04-22T08:14:46.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.781+0000 7f6d4e199640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:46.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.781+0000 7f6d4e199640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:46.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.781+0000 7f6d4e199640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:46.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.781+0000 7f6d4e199640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:46.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.781+0000 7f6d4e199640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:46.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.781+0000 7f6d4e199640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:46.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.781+0000 7f6d4e199640 10 monclient(hunting): handle_auth_done global_id 34170 payload 824 2024-04-22T08:14:46.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.781+0000 7f6d4e199640 1 -- 172.21.15.40:0/3599378576 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d4809f8f0 msgr2=0x7f6d481c2f30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:14:46.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.781+0000 7f6d4e199640 1 --2- 172.21.15.40:0/3599378576 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d4809f8f0 0x7f6d481c2f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:46.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.781+0000 7f6d4e199640 1 -- 172.21.15.40:0/3599378576 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d481c3470 msgr2=0x7f6d481c6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:46.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.781+0000 7f6d4e199640 1 --2- 172.21.15.40:0/3599378576 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d481c3470 0x7f6d481c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:46.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.781+0000 7f6d4e199640 10 monclient: _finish_hunting 0 2024-04-22T08:14:46.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.781+0000 7f6d4e199640 1 monclient: found mon.noname-a 2024-04-22T08:14:46.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.781+0000 7f6d4e199640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:14:46.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.781+0000 7f6d4e199640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:14:46.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.781+0000 7f6d4e199640 1 -- 172.21.15.40:0/3599378576 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d38009400 con 0x7f6d4809ef90 2024-04-22T08:14:46.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.784+0000 7f6d4e199640 1 -- 172.21.15.40:0/3599378576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d4809ef90 msgr2=0x7f6d481c02b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:46.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.784+0000 7f6d4e199640 1 --2- 172.21.15.40:0/3599378576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d4809ef90 0x7f6d481c02b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6d4000b550 tx=0x7f6d4000ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:46.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.784+0000 7f6d3effd640 1 -- 172.21.15.40:0/3599378576 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d40003dd0 con 0x7f6d4809ef90 2024-04-22T08:14:46.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.784+0000 7f6d3effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:46.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.784+0000 7f6d3effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:14:46.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.784+0000 7f6d3effd640 10 monclient: dump: 2024-04-22T08:14:46.796 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:46.796 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:46.796 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:46.797 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:46.797 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:46.797 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:46.797 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:46.797 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:46.797 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:46.797 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:46.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.784+0000 7f6d3effd640 10 monclient: _finish_auth 0 2024-04-22T08:14:46.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.784+0000 7f6d3effd640 10 monclient: _check_auth_tickets 2024-04-22T08:14:46.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.784+0000 7f6d3effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:14:46.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.784+0000 7f6d503fd640 5 monclient: authenticate success, global_id 34170 2024-04-22T08:14:46.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.784+0000 7f6d3effd640 1 -- 172.21.15.40:0/3599378576 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d40005070 con 0x7f6d4809ef90 2024-04-22T08:14:46.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.784+0000 7f6d503fd640 10 monclient: _renew_subs 2024-04-22T08:14:46.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.784+0000 7f6d503fd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:14:46.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.784+0000 7f6d503fd640 1 -- 172.21.15.40:0/3599378576 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d481a5130 con 0x7f6d4809ef90 2024-04-22T08:14:46.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.784+0000 7f6d3effd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:14:46.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.784+0000 7f6d3effd640 1 -- 172.21.15.40:0/3599378576 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d40005420 con 0x7f6d4809ef90 2024-04-22T08:14:46.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.784+0000 7f6d3effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:46.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.784+0000 7f6d3effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:14:46.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.784+0000 7f6d3effd640 10 monclient: dump: 2024-04-22T08:14:46.798 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:46.798 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:46.798 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:46.799 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:46.799 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:46.799 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:46.799 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:46.799 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:46.799 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:46.799 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:46.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.784+0000 7f6d4c996640 4 set_mon_vals no callback set 2024-04-22T08:14:46.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.784+0000 7f6d503fd640 10 monclient: _renew_subs 2024-04-22T08:14:46.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.784+0000 7f6d503fd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:14:46.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.784+0000 7f6d503fd640 1 -- 172.21.15.40:0/3599378576 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d48071070 con 0x7f6d4809ef90 2024-04-22T08:14:46.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.785+0000 7f6d3effd640 1 -- 172.21.15.40:0/3599378576 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f6d40005580 con 0x7f6d4809ef90 2024-04-22T08:14:46.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.785+0000 7f6d503fd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:14:46.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.785+0000 7f6d503fd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:14:46.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.785+0000 7f6d503fd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:14:46.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.785+0000 7f6d503fd640 1 -- 172.21.15.40:0/3599378576 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d4804e9b0 con 0x7f6d4809ef90 2024-04-22T08:14:46.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.785+0000 7f6d3effd640 1 --2- 172.21.15.40:0/3599378576 >> v2:172.21.15.189:6800/30168 conn(0x7f6d2c0377b0 0x7f6d2c039c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:46.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.785+0000 7f6d3effd640 1 -- 172.21.15.40:0/3599378576 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f6d4004a000 con 0x7f6d4809ef90 2024-04-22T08:14:46.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.785+0000 7f6d4d998640 1 --2- 172.21.15.40:0/3599378576 >> v2:172.21.15.189:6800/30168 conn(0x7f6d2c0377b0 0x7f6d2c039c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:46.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.785+0000 7f6d4d998640 10 monclient: get_auth_request con 0x7f6d2c0377b0 auth_method 0 2024-04-22T08:14:46.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.788+0000 7f6d4d998640 1 --2- 172.21.15.40:0/3599378576 >> v2:172.21.15.189:6800/30168 conn(0x7f6d2c0377b0 0x7f6d2c039c70 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6d38032000 tx=0x7f6d38008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:46.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.790+0000 7f6d3effd640 1 -- 172.21.15.40:0/3599378576 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f6d40005830 con 0x7f6d4809ef90 2024-04-22T08:14:46.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.790+0000 7f6d3effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:14:46.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:46.790+0000 7f6d3effd640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:14:47.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:47.043+0000 7f6d503fd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:14:47.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:47.043+0000 7f6d503fd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:14:47.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:47.043+0000 7f6d503fd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:14:47.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:47.043+0000 7f6d503fd640 1 -- 172.21.15.40:0/3599378576 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6d4806a290 con 0x7f6d4809ef90 2024-04-22T08:14:47.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:47.044+0000 7f6d3effd640 1 -- 172.21.15.40:0/3599378576 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f6d4003c620 con 0x7f6d4809ef90 2024-04-22T08:14:47.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:47.044+0000 7f6d3effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:14:47.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:47.044+0000 7f6d3effd640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:14:47.044 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:14:47.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:47.046+0000 7f6d503fd640 1 -- 172.21.15.40:0/3599378576 >> v2:172.21.15.189:6800/30168 conn(0x7f6d2c0377b0 msgr2=0x7f6d2c039c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:47.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:47.046+0000 7f6d503fd640 1 --2- 172.21.15.40:0/3599378576 >> v2:172.21.15.189:6800/30168 conn(0x7f6d2c0377b0 0x7f6d2c039c70 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6d38032000 tx=0x7f6d38008030 comp rx=0 tx=0).stop 2024-04-22T08:14:47.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:47.046+0000 7f6d503fd640 10 monclient: shutdown 2024-04-22T08:14:47.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:47.046+0000 7f6d503fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:14:47.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:47.046+0000 7f6d503fd640 1 -- 172.21.15.40:0/3599378576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d4809ef90 msgr2=0x7f6d481c02b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:47.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:47.046+0000 7f6d503fd640 1 --2- 172.21.15.40:0/3599378576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d4809ef90 0x7f6d481c02b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6d4000b550 tx=0x7f6d4000ba10 comp rx=0 tx=0).stop 2024-04-22T08:14:47.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:47.046+0000 7f6d503fd640 1 -- 172.21.15.40:0/3599378576 shutdown_connections 2024-04-22T08:14:47.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:47.046+0000 7f6d503fd640 1 --2- 172.21.15.40:0/3599378576 >> v2:172.21.15.189:6800/30168 conn(0x7f6d2c0377b0 0x7f6d2c039c70 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:47.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:47.046+0000 7f6d503fd640 1 --2- 172.21.15.40:0/3599378576 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d481c3470 0x7f6d481c6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:47.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:47.046+0000 7f6d503fd640 1 --2- 172.21.15.40:0/3599378576 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d4809f8f0 0x7f6d481c2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:47.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:47.046+0000 7f6d503fd640 1 --2- 172.21.15.40:0/3599378576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d4809ef90 0x7f6d481c02b0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:47.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:47.046+0000 7f6d503fd640 1 -- 172.21.15.40:0/3599378576 >> 172.21.15.40:0/3599378576 conn(0x7f6d480a0b50 msgr2=0x7f6d48071280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:14:47.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:47.046+0000 7f6d503fd640 1 -- 172.21.15.40:0/3599378576 shutdown_connections 2024-04-22T08:14:47.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:47.046+0000 7f6d503fd640 1 -- 172.21.15.40:0/3599378576 wait complete. 2024-04-22T08:14:50.056 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:14:50.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.167+0000 7f9381407640 10 monclient: get_monmap_and_config 2024-04-22T08:14:50.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.167+0000 7f9381407640 10 monclient: build_initial_monmap 2024-04-22T08:14:50.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.167+0000 7f9381407640 10 monclient: monmap: 2024-04-22T08:14:50.168 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:14:50.168 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:50.168 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:14:50.169279+0000 2024-04-22T08:14:50.168 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:14:50.169279+0000 2024-04-22T08:14:50.168 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:14:50.168 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:50.168 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:14:50.169 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:14:50.169 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:14:50.169 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:14:50.169 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:14:50.169 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:50.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.168+0000 7f9381407640 1 Processor -- start 2024-04-22T08:14:50.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.168+0000 7f9381407640 1 -- start start 2024-04-22T08:14:50.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.168+0000 7f9381407640 10 monclient: init 2024-04-22T08:14:50.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.168+0000 7f9381407640 10 monclient: _reopen_session rank -1 2024-04-22T08:14:50.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.168+0000 7f9381407640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-04-22T08:14:50.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.168+0000 7f9381407640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f937c0a20d0 0x7f937c0a24b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:50.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.168+0000 7f9381407640 10 monclient(hunting): picked mon.noname-c con 0x7f937c0a20d0 addr v2:172.21.15.189:3300/0 2024-04-22T08:14:50.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.168+0000 7f9381407640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f937c0a2d10 0x7f937c0a31c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:50.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.168+0000 7f9381407640 10 monclient(hunting): picked mon.noname-a con 0x7f937c0a2d10 addr v2:172.21.15.40:3300/0 2024-04-22T08:14:50.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.168+0000 7f9381407640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f937c0a3700 addr v1:172.21.15.189:6789/0 2024-04-22T08:14:50.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.168+0000 7f9381407640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f937c0a3b70 con 0x7f937c0a3700 2024-04-22T08:14:50.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.168+0000 7f9381407640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:50.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.168+0000 7f9381407640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f937c0a45f0 con 0x7f937c0a2d10 2024-04-22T08:14:50.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.168+0000 7f9381407640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:50.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.168+0000 7f9381407640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f937c0a0360 con 0x7f937c0a20d0 2024-04-22T08:14:50.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.168+0000 7f9381407640 10 monclient(hunting): _renew_subs 2024-04-22T08:14:50.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.168+0000 7f9381407640 10 monclient(hunting): authenticate will time out at 964.163589s 2024-04-22T08:14:50.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.168+0000 7f9373fff640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f937c0a2d10 0x7f937c0a31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:50.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.169+0000 7f9373fff640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f937c0a2d10 0x7f937c0a31c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33922/0 (socket says 172.21.15.40:33922) 2024-04-22T08:14:50.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.169+0000 7f9373fff640 1 -- 172.21.15.40:0/54925511 learned_addr learned my addr 172.21.15.40:0/54925511 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:14:50.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.169+0000 7f9373fff640 10 monclient(hunting): get_auth_request con 0x7f937c0a2d10 auth_method 0 2024-04-22T08:14:50.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.169+0000 7f9373fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:50.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.169+0000 7f9373fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:50.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.169+0000 7f9373fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:50.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.169+0000 7f9373fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:50.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.169+0000 7f9373fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:50.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.169+0000 7f9373fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:50.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.169+0000 7f937affd640 1 --2- 172.21.15.40:0/54925511 >> v2:172.21.15.189:3300/0 conn(0x7f937c0a20d0 0x7f937c0a24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:50.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.169+0000 7f937affd640 10 monclient(hunting): get_auth_request con 0x7f937c0a20d0 auth_method 0 2024-04-22T08:14:50.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.169+0000 7f937affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:50.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.169+0000 7f937affd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:50.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.169+0000 7f937affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:50.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.169+0000 7f9373fff640 10 monclient(hunting): handle_auth_done global_id 34173 payload 274 2024-04-22T08:14:50.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.169+0000 7f9373fff640 1 -- 172.21.15.40:0/54925511 >> v2:172.21.15.189:3300/0 conn(0x7f937c0a20d0 msgr2=0x7f937c0a24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:14:50.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.169+0000 7f9373fff640 1 --2- 172.21.15.40:0/54925511 >> v2:172.21.15.189:3300/0 conn(0x7f937c0a20d0 0x7f937c0a24b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:50.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.169+0000 7f9373fff640 1 -- 172.21.15.40:0/54925511 >> v1:172.21.15.189:6789/0 conn(0x7f937c0a3700 legacy=0x7f937c13cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:14:50.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.169+0000 7f9373fff640 10 monclient: _finish_hunting 0 2024-04-22T08:14:50.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.169+0000 7f9373fff640 1 monclient: found mon.noname-a 2024-04-22T08:14:50.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.169+0000 7f9373fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:14:50.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.169+0000 7f9373fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:14:50.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.169+0000 7f9373fff640 1 -- 172.21.15.40:0/54925511 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f937c0a0640 con 0x7f937c0a2d10 2024-04-22T08:14:50.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.172+0000 7f9373fff640 1 -- 172.21.15.40:0/54925511 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f937c0a2d10 msgr2=0x7f937c0a31c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:50.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.172+0000 7f9373fff640 1 --2- 172.21.15.40:0/54925511 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f937c0a2d10 0x7f937c0a31c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f937c0a45c0 tx=0x7f936402b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fced860d28457dfb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:50.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.172+0000 7f937a7fc640 1 -- 172.21.15.40:0/54925511 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9364031960 con 0x7f937c0a2d10 2024-04-22T08:14:50.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.172+0000 7f937a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:50.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.172+0000 7f937a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:14:50.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.172+0000 7f937a7fc640 10 monclient: dump: 2024-04-22T08:14:50.180 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:50.180 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:50.180 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:50.180 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:50.180 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:50.180 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:50.180 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:50.181 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:50.181 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:50.181 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:50.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.172+0000 7f937a7fc640 10 monclient: _finish_auth 0 2024-04-22T08:14:50.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.172+0000 7f937a7fc640 10 monclient: _check_auth_tickets 2024-04-22T08:14:50.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.172+0000 7f937a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:14:50.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.172+0000 7f937a7fc640 1 -- 172.21.15.40:0/54925511 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9364031ac0 con 0x7f937c0a2d10 2024-04-22T08:14:50.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.172+0000 7f937a7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:14:50.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.172+0000 7f937a7fc640 1 -- 172.21.15.40:0/54925511 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9364031e50 con 0x7f937c0a2d10 2024-04-22T08:14:50.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.172+0000 7f9381407640 5 monclient: authenticate success, global_id 34173 2024-04-22T08:14:50.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.172+0000 7f9381407640 10 monclient: get_monmap_and_config success 2024-04-22T08:14:50.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.172+0000 7f937a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:50.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.172+0000 7f9381407640 4 set_mon_vals no callback set 2024-04-22T08:14:50.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.172+0000 7f937a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:14:50.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.172+0000 7f937a7fc640 10 monclient: dump: 2024-04-22T08:14:50.182 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:50.182 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:50.182 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:50.182 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:50.182 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:50.182 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:50.182 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:50.182 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:50.182 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:50.183 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:50.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.173+0000 7f9381407640 10 monclient: shutdown 2024-04-22T08:14:50.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.173+0000 7f9381407640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:14:50.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.173+0000 7f9381407640 1 -- 172.21.15.40:0/54925511 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f937c0a2d10 msgr2=0x7f937c0a31c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:50.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.173+0000 7f9381407640 1 --2- 172.21.15.40:0/54925511 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f937c0a2d10 0x7f937c0a31c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f937c0a45c0 tx=0x7f936402b7f0 comp rx=0 tx=0).stop 2024-04-22T08:14:50.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.173+0000 7f9381407640 1 -- 172.21.15.40:0/54925511 shutdown_connections 2024-04-22T08:14:50.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.173+0000 7f9381407640 1 --2- 172.21.15.40:0/54925511 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f937c0a2d10 0x7f937c0a31c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:50.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.173+0000 7f9381407640 1 --2- 172.21.15.40:0/54925511 >> v2:172.21.15.189:3300/0 conn(0x7f937c0a20d0 0x7f937c0a24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:50.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.173+0000 7f9381407640 1 -- 172.21.15.40:0/54925511 >> 172.21.15.40:0/54925511 conn(0x7f937c0a8e60 msgr2=0x7f937c0a4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:14:50.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.173+0000 7f9381407640 1 -- 172.21.15.40:0/54925511 shutdown_connections 2024-04-22T08:14:50.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.173+0000 7f9381407640 1 -- 172.21.15.40:0/54925511 wait complete. 2024-04-22T08:14:50.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.173+0000 7f9381407640 10 monclient: build_initial_monmap 2024-04-22T08:14:50.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.173+0000 7f9381407640 10 monclient: monmap: 2024-04-22T08:14:50.183 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:14:50.184 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:14:50.184 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:14:50.184 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:14:50.184 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:14:50.184 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:50.184 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:14:50.184 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:14:50.184 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:14:50.184 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:50.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.174+0000 7f9381407640 1 Processor -- start 2024-04-22T08:14:50.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.174+0000 7f9381407640 1 -- start start 2024-04-22T08:14:50.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.174+0000 7f9381407640 10 monclient: init 2024-04-22T08:14:50.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.174+0000 7f9381407640 10 monclient: _reopen_session rank -1 2024-04-22T08:14:50.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.174+0000 7f9381407640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-22T08:14:50.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.174+0000 7f9381407640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f937c0a20d0 0x7f937c1c46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:50.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.174+0000 7f9381407640 10 monclient(hunting): picked mon.noname-b con 0x7f937c0a20d0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:14:50.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.174+0000 7f9381407640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f937c0a2d10 0x7f937c1c7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:50.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.174+0000 7f9381407640 10 monclient(hunting): picked mon.noname-a con 0x7f937c0a2d10 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:14:50.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.174+0000 7f9381407640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f937c0a3700 0x7f937c1c7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:50.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.174+0000 7f9381407640 10 monclient(hunting): picked mon.noname-c con 0x7f937c0a3700 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:14:50.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.174+0000 7f9381407640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:50.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.174+0000 7f9381407640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f937c071750 con 0x7f937c0a2d10 2024-04-22T08:14:50.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.174+0000 7f9381407640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:50.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.174+0000 7f9381407640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f937c1a95e0 con 0x7f937c0a20d0 2024-04-22T08:14:50.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.174+0000 7f9381407640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:50.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.174+0000 7f9381407640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f937c1a9cf0 con 0x7f937c0a3700 2024-04-22T08:14:50.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.174+0000 7f9381407640 10 monclient(hunting): _renew_subs 2024-04-22T08:14:50.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.174+0000 7f9381407640 10 monclient(hunting): authenticate will time out at 964.169665s 2024-04-22T08:14:50.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.174+0000 7f937b7fe640 1 -- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f937c0a3700 msgr2=0x7f937c1c7890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:14:50.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.174+0000 7f937b7fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f937c0a3700 0x7f937c1c7890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:14:50.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.174+0000 7f9373fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f937c0a2d10 0x7f937c1c7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:50.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.174+0000 7f9373fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f937c0a2d10 0x7f937c1c7350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33932/0 (socket says 172.21.15.40:33932) 2024-04-22T08:14:50.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.175+0000 7f9373fff640 1 -- 172.21.15.40:0/4217693485 learned_addr learned my addr 172.21.15.40:0/4217693485 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:14:50.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.175+0000 7f9373fff640 10 monclient(hunting): get_auth_request con 0x7f937c0a2d10 auth_method 0 2024-04-22T08:14:50.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.175+0000 7f9373fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:50.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.175+0000 7f9373fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:50.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.175+0000 7f9373fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:50.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.175+0000 7f9373fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:14:50.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.175+0000 7f9373fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:14:50.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.175+0000 7f9373fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:14:50.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.175+0000 7f937affd640 1 --2- 172.21.15.40:0/4217693485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f937c0a20d0 0x7f937c1c46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:50.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.175+0000 7f937affd640 10 monclient(hunting): get_auth_request con 0x7f937c0a20d0 auth_method 0 2024-04-22T08:14:50.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.175+0000 7f937affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:50.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.175+0000 7f937affd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:50.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.175+0000 7f937affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:50.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.175+0000 7f9373fff640 10 monclient(hunting): handle_auth_done global_id 34176 payload 824 2024-04-22T08:14:50.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.175+0000 7f9373fff640 1 -- 172.21.15.40:0/4217693485 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f937c0a3700 msgr2=0x7f937c1c7890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:14:50.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.175+0000 7f9373fff640 1 --2- 172.21.15.40:0/4217693485 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f937c0a3700 0x7f937c1c7890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:50.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.175+0000 7f9373fff640 1 -- 172.21.15.40:0/4217693485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f937c0a20d0 msgr2=0x7f937c1c46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:50.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.175+0000 7f9373fff640 1 --2- 172.21.15.40:0/4217693485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f937c0a20d0 0x7f937c1c46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:50.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.175+0000 7f9373fff640 10 monclient: _finish_hunting 0 2024-04-22T08:14:50.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.175+0000 7f9373fff640 1 monclient: found mon.noname-a 2024-04-22T08:14:50.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.175+0000 7f9373fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:14:50.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.175+0000 7f9373fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:14:50.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.175+0000 7f9373fff640 1 -- 172.21.15.40:0/4217693485 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93640094d0 con 0x7f937c0a2d10 2024-04-22T08:14:50.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.177+0000 7f9373fff640 1 -- 172.21.15.40:0/4217693485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f937c0a2d10 msgr2=0x7f937c1c7350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:50.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.177+0000 7f9373fff640 1 --2- 172.21.15.40:0/4217693485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f937c0a2d10 0x7f937c1c7350 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f93640090b0 tx=0x7f93640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:50.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.177+0000 7f93737fe640 1 -- 172.21.15.40:0/4217693485 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9364031e30 con 0x7f937c0a2d10 2024-04-22T08:14:50.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.177+0000 7f93737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:50.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.177+0000 7f93737fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:14:50.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.177+0000 7f93737fe640 10 monclient: dump: 2024-04-22T08:14:50.188 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:50.188 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:50.188 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:50.188 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:50.188 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:50.188 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:50.189 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:50.189 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:50.189 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:50.189 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:50.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.177+0000 7f93737fe640 10 monclient: _finish_auth 0 2024-04-22T08:14:50.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.177+0000 7f93737fe640 10 monclient: _check_auth_tickets 2024-04-22T08:14:50.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.177+0000 7f93737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:14:50.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.177+0000 7f93737fe640 1 -- 172.21.15.40:0/4217693485 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f936403c400 con 0x7f937c0a2d10 2024-04-22T08:14:50.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.177+0000 7f93737fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:14:50.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.177+0000 7f93737fe640 1 -- 172.21.15.40:0/4217693485 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f936403c790 con 0x7f937c0a2d10 2024-04-22T08:14:50.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.177+0000 7f9381407640 5 monclient: authenticate success, global_id 34176 2024-04-22T08:14:50.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.177+0000 7f9381407640 10 monclient: _renew_subs 2024-04-22T08:14:50.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.177+0000 7f9381407640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:14:50.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.177+0000 7f9379ffb640 4 set_mon_vals no callback set 2024-04-22T08:14:50.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.177+0000 7f9381407640 1 -- 172.21.15.40:0/4217693485 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f937c070d40 con 0x7f937c0a2d10 2024-04-22T08:14:50.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.177+0000 7f93737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:14:50.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.177+0000 7f93737fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:14:50.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.177+0000 7f93737fe640 10 monclient: dump: 2024-04-22T08:14:50.190 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:14:50.190 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:50.190 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:50.190 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:14:50.190 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:14:50.190 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:50.190 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:14:50.190 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:14:50.190 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:14:50.190 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:50.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.177+0000 7f9381407640 10 monclient: _renew_subs 2024-04-22T08:14:50.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.177+0000 7f9381407640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:14:50.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.177+0000 7f9381407640 1 -- 172.21.15.40:0/4217693485 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f937c1aba70 con 0x7f937c0a2d10 2024-04-22T08:14:50.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.178+0000 7f93737fe640 1 -- 172.21.15.40:0/4217693485 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f936403a040 con 0x7f937c0a2d10 2024-04-22T08:14:50.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.178+0000 7f9381407640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:14:50.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.178+0000 7f9381407640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:14:50.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.178+0000 7f9381407640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:14:50.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.178+0000 7f9381407640 1 -- 172.21.15.40:0/4217693485 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f937c04e9b0 con 0x7f937c0a2d10 2024-04-22T08:14:50.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.178+0000 7f93737fe640 1 --2- 172.21.15.40:0/4217693485 >> v2:172.21.15.189:6800/30168 conn(0x7f935c0377d0 0x7f935c039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:50.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.178+0000 7f93737fe640 1 -- 172.21.15.40:0/4217693485 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f936406bd40 con 0x7f937c0a2d10 2024-04-22T08:14:50.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.179+0000 7f937affd640 1 --2- 172.21.15.40:0/4217693485 >> v2:172.21.15.189:6800/30168 conn(0x7f935c0377d0 0x7f935c039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:50.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.179+0000 7f937affd640 10 monclient: get_auth_request con 0x7f935c0377d0 auth_method 0 2024-04-22T08:14:50.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.181+0000 7f937affd640 1 --2- 172.21.15.40:0/4217693485 >> v2:172.21.15.189:6800/30168 conn(0x7f935c0377d0 0x7f935c039c90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f936c006fd0 tx=0x7f936c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:14:50.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.183+0000 7f93737fe640 1 -- 172.21.15.40:0/4217693485 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f936403a700 con 0x7f937c0a2d10 2024-04-22T08:14:50.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.183+0000 7f93737fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:14:50.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.183+0000 7f93737fe640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:14:50.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.434+0000 7f9381407640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:14:50.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.434+0000 7f9381407640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:14:50.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.434+0000 7f9381407640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:14:50.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.434+0000 7f9381407640 1 -- 172.21.15.40:0/4217693485 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f937c0a2760 con 0x7f937c0a2d10 2024-04-22T08:14:50.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.435+0000 7f93737fe640 1 -- 172.21.15.40:0/4217693485 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f9364038070 con 0x7f937c0a2d10 2024-04-22T08:14:50.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.435+0000 7f93737fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:14:50.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.435+0000 7f93737fe640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:14:50.436 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:14:50.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.436+0000 7f9381407640 1 -- 172.21.15.40:0/4217693485 >> v2:172.21.15.189:6800/30168 conn(0x7f935c0377d0 msgr2=0x7f935c039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:50.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.436+0000 7f9381407640 1 --2- 172.21.15.40:0/4217693485 >> v2:172.21.15.189:6800/30168 conn(0x7f935c0377d0 0x7f935c039c90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f936c006fd0 tx=0x7f936c008040 comp rx=0 tx=0).stop 2024-04-22T08:14:50.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.436+0000 7f9381407640 10 monclient: shutdown 2024-04-22T08:14:50.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.436+0000 7f9381407640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:14:50.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.436+0000 7f9381407640 1 -- 172.21.15.40:0/4217693485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f937c0a2d10 msgr2=0x7f937c1c7350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:50.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.436+0000 7f9381407640 1 --2- 172.21.15.40:0/4217693485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f937c0a2d10 0x7f937c1c7350 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f93640090b0 tx=0x7f93640046c0 comp rx=0 tx=0).stop 2024-04-22T08:14:50.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.436+0000 7f9381407640 1 -- 172.21.15.40:0/4217693485 shutdown_connections 2024-04-22T08:14:50.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.436+0000 7f9381407640 1 --2- 172.21.15.40:0/4217693485 >> v2:172.21.15.189:6800/30168 conn(0x7f935c0377d0 0x7f935c039c90 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:50.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.436+0000 7f9381407640 1 --2- 172.21.15.40:0/4217693485 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f937c0a3700 0x7f937c1c7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:50.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.436+0000 7f9381407640 1 --2- 172.21.15.40:0/4217693485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f937c0a2d10 0x7f937c1c7350 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:50.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.436+0000 7f9381407640 1 --2- 172.21.15.40:0/4217693485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f937c0a20d0 0x7f937c1c46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:50.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.436+0000 7f9381407640 1 -- 172.21.15.40:0/4217693485 >> 172.21.15.40:0/4217693485 conn(0x7f937c0a8e60 msgr2=0x7f937c06ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:14:50.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.437+0000 7f9381407640 1 -- 172.21.15.40:0/4217693485 shutdown_connections 2024-04-22T08:14:50.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:50.437+0000 7f9381407640 1 -- 172.21.15.40:0/4217693485 wait complete. 2024-04-22T08:14:53.450 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:14:53.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:53.559+0000 7fdbbaace640 10 monclient: get_monmap_and_config 2024-04-22T08:14:53.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:53.559+0000 7fdbbaace640 10 monclient: build_initial_monmap 2024-04-22T08:14:53.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:53.560+0000 7fdbbaace640 10 monclient: monmap: 2024-04-22T08:14:53.560 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:14:53.561 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:14:53.561 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:14:53.561785+0000 2024-04-22T08:14:53.561 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:14:53.561785+0000 2024-04-22T08:14:53.561 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:14:53.561 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:14:53.561 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:14:53.561 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:14:53.561 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:14:53.561 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:14:53.561 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:14:53.561 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:14:53.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:53.560+0000 7fdbbaace640 1 Processor -- start 2024-04-22T08:14:53.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:53.560+0000 7fdbbaace640 1 -- start start 2024-04-22T08:14:53.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:53.560+0000 7fdbbaace640 10 monclient: init 2024-04-22T08:14:53.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:53.561+0000 7fdbbaace640 10 monclient: _reopen_session rank -1 2024-04-22T08:14:53.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:53.561+0000 7fdbbaace640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-04-22T08:14:53.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:53.561+0000 7fdbbaace640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbb4135e30 0x7fdbb4136210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:53.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:53.561+0000 7fdbbaace640 10 monclient(hunting): picked mon.noname-b con 0x7fdbb4135e30 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:14:53.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:53.561+0000 7fdbbaace640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdbb4071c90 addr v1:172.21.15.40:6789/0 2024-04-22T08:14:53.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:53.561+0000 7fdbbaace640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdbb4136750 addr v1:172.21.15.189:6789/0 2024-04-22T08:14:53.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:53.561+0000 7fdbbaace640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbb40a27c0 con 0x7fdbb4071c90 2024-04-22T08:14:53.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:53.561+0000 7fdbbaace640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbb409e310 con 0x7fdbb4136750 2024-04-22T08:14:53.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:53.561+0000 7fdbbaace640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:53.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:53.561+0000 7fdbb886a640 1 -- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbb4135e30 msgr2=0x7fdbb4136210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:14:53.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:53.561+0000 7fdbbaace640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb409de80 con 0x7fdbb4135e30 2024-04-22T08:14:53.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:53.561+0000 7fdbbaace640 10 monclient(hunting): _renew_subs 2024-04-22T08:14:53.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:53.561+0000 7fdbbaace640 10 monclient(hunting): authenticate will time out at 967.556176s 2024-04-22T08:14:53.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:53.561+0000 7fdbb886a640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbb4135e30 0x7fdbb4136210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:14:53.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:53.561+0000 7fdbb3fff640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fdbb4071c90 0x7fdbb40a19a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:50604/0 (socket says 172.21.15.40:50604) 2024-04-22T08:14:53.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:53.561+0000 7fdbb3fff640 1 -- 172.21.15.40:0/3904135143 learned_addr learned my addr 172.21.15.40:0/3904135143 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:14:53.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:53.563+0000 7fdbb3fff640 1 -- 172.21.15.40:0/3904135143 >> v1:172.21.15.40:6789/0 conn(0x7fdbb4071c90 legacy=0x7fdbb40a19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:53.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:53.564+0000 7fdbb906b640 1 -- 172.21.15.40:0/3904135143 >> v1:172.21.15.189:6789/0 conn(0x7fdbb4136750 legacy=0x7fdbb40a20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:53.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:53.762+0000 7fdbb886a640 1 -- 172.21.15.40:0/3904135143 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbb4135e30 msgr2=0x7fdbb4136210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:14:53.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:53.762+0000 7fdbb886a640 1 --2- 172.21.15.40:0/3904135143 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbb4135e30 0x7fdbb4136210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:14:54.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:54.163+0000 7fdbb886a640 1 -- 172.21.15.40:0/3904135143 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbb4135e30 msgr2=0x7fdbb4136210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:14:54.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:54.163+0000 7fdbb886a640 1 --2- 172.21.15.40:0/3904135143 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbb4135e30 0x7fdbb4136210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:14:54.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:54.964+0000 7fdbb886a640 1 -- 172.21.15.40:0/3904135143 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbb4135e30 msgr2=0x7fdbb4136210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:14:54.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:54.964+0000 7fdbb886a640 1 --2- 172.21.15.40:0/3904135143 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbb4135e30 0x7fdbb4136210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T08:14:56.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.561+0000 7fdbb27fc640 10 monclient(hunting): tick 2024-04-22T08:14:56.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.561+0000 7fdbb27fc640 10 monclient(hunting): _check_auth_tickets 2024-04-22T08:14:56.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.561+0000 7fdbb27fc640 1 monclient(hunting): continuing hunt 2024-04-22T08:14:56.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.561+0000 7fdbb27fc640 10 monclient(hunting): _reopen_session rank -1 2024-04-22T08:14:56.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.561+0000 7fdbb27fc640 1 -- 172.21.15.40:0/3904135143 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbb4135e30 msgr2=0x7fdbb4136210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:14:56.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.561+0000 7fdbb27fc640 1 --2- 172.21.15.40:0/3904135143 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbb4135e30 0x7fdbb4136210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:56.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.561+0000 7fdbb27fc640 1 -- 172.21.15.40:0/3904135143 >> v1:172.21.15.189:6789/0 conn(0x7fdbb4136750 legacy=0x7fdbb40a20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:56.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.561+0000 7fdbb27fc640 1 -- 172.21.15.40:0/3904135143 >> v1:172.21.15.40:6789/0 conn(0x7fdbb4071c90 legacy=0x7fdbb40a19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:14:56.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.561+0000 7fdbb27fc640 10 monclient: _add_conns ranks=[3,4] 2024-04-22T08:14:56.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.561+0000 7fdbb27fc640 1 --2- 172.21.15.40:0/3904135143 >> v2:172.21.15.40:3300/0 conn(0x7fdba0003810 0x7fdba0005c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:56.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.561+0000 7fdbb27fc640 10 monclient(hunting): picked mon.noname-a con 0x7fdba0003810 addr v2:172.21.15.40:3300/0 2024-04-22T08:14:56.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.561+0000 7fdbb27fc640 1 --2- 172.21.15.40:0/3904135143 >> v2:172.21.15.189:3300/0 conn(0x7fdba0006380 0x7fdba0008840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:56.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.561+0000 7fdbb27fc640 10 monclient(hunting): picked mon.noname-c con 0x7fdba0006380 addr v2:172.21.15.189:3300/0 2024-04-22T08:14:56.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.561+0000 7fdbb27fc640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:56.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.561+0000 7fdbb27fc640 1 -- 172.21.15.40:0/3904135143 --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbb409de80 con 0x7fdba0003810 2024-04-22T08:14:56.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.561+0000 7fdbb27fc640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:56.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.561+0000 7fdbb27fc640 1 -- 172.21.15.40:0/3904135143 --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdba0008f00 con 0x7fdba0006380 2024-04-22T08:14:56.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.561+0000 7fdbb27fc640 10 monclient(hunting): _renew_subs 2024-04-22T08:14:56.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.561+0000 7fdbb886a640 1 --2- 172.21.15.40:0/3904135143 >> v2:172.21.15.40:3300/0 conn(0x7fdba0003810 0x7fdba0005c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:56.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.561+0000 7fdbb886a640 10 monclient(hunting): get_auth_request con 0x7fdba0003810 auth_method 0 2024-04-22T08:14:56.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.561+0000 7fdbb886a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:56.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.561+0000 7fdbb886a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:56.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.561+0000 7fdbb886a640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:56.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.561+0000 7fdbb3fff640 1 --2- 172.21.15.40:0/3904135143 >> v2:172.21.15.189:3300/0 conn(0x7fdba0006380 0x7fdba0008840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:56.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.561+0000 7fdbb886a640 1 -- 172.21.15.40:0/3904135143 >> v2:172.21.15.40:3300/0 conn(0x7fdba0003810 msgr2=0x7fdba0005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:14:56.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.561+0000 7fdbb886a640 1 -- 172.21.15.40:0/3904135143 >> v2:172.21.15.40:3300/0 conn(0x7fdba0003810 msgr2=0x7fdba0005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:14:56.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.561+0000 7fdbb886a640 1 --2- 172.21.15.40:0/3904135143 >> v2:172.21.15.40:3300/0 conn(0x7fdba0003810 0x7fdba0005c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:14:56.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.561+0000 7fdbb3fff640 10 monclient(hunting): get_auth_request con 0x7fdba0006380 auth_method 0 2024-04-22T08:14:56.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.561+0000 7fdbb3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:56.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.561+0000 7fdbb3fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:56.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.561+0000 7fdbb3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:56.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.561+0000 7fdbb886a640 1 --2- 172.21.15.40:0/3904135143 >> v2:172.21.15.40:3300/0 conn(0x7fdba0003810 0x7fdba0005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:14:56.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.561+0000 7fdbb3fff640 1 -- 172.21.15.40:0/3904135143 >> v2:172.21.15.189:3300/0 conn(0x7fdba0006380 msgr2=0x7fdba0008840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:14:56.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.561+0000 7fdbb3fff640 1 -- 172.21.15.40:0/3904135143 >> v2:172.21.15.189:3300/0 conn(0x7fdba0006380 msgr2=0x7fdba0008840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:14:56.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.561+0000 7fdbb3fff640 1 --2- 172.21.15.40:0/3904135143 >> v2:172.21.15.189:3300/0 conn(0x7fdba0006380 0x7fdba0008840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:14:56.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.561+0000 7fdbb3fff640 1 --2- 172.21.15.40:0/3904135143 >> v2:172.21.15.189:3300/0 conn(0x7fdba0006380 0x7fdba0008840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:14:56.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.762+0000 7fdbb886a640 1 --2- 172.21.15.40:0/3904135143 >> v2:172.21.15.40:3300/0 conn(0x7fdba0003810 0x7fdba0005c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:56.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.762+0000 7fdbb886a640 10 monclient(hunting): get_auth_request con 0x7fdba0003810 auth_method 0 2024-04-22T08:14:56.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.762+0000 7fdbb886a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:56.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.762+0000 7fdbb886a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:56.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.762+0000 7fdbb886a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:14:56.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.762+0000 7fdbb886a640 1 -- 172.21.15.40:0/3904135143 >> v2:172.21.15.40:3300/0 conn(0x7fdba0003810 msgr2=0x7fdba0005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:14:56.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.762+0000 7fdbb886a640 1 -- 172.21.15.40:0/3904135143 >> v2:172.21.15.40:3300/0 conn(0x7fdba0003810 msgr2=0x7fdba0005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:14:56.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.762+0000 7fdbb886a640 1 --2- 172.21.15.40:0/3904135143 >> v2:172.21.15.40:3300/0 conn(0x7fdba0003810 0x7fdba0005c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:14:56.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.762+0000 7fdbb886a640 1 --2- 172.21.15.40:0/3904135143 >> v2:172.21.15.40:3300/0 conn(0x7fdba0003810 0x7fdba0005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:14:56.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.762+0000 7fdbb3fff640 1 --2- 172.21.15.40:0/3904135143 >> v2:172.21.15.189:3300/0 conn(0x7fdba0006380 0x7fdba0008840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:56.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.762+0000 7fdbb3fff640 10 monclient(hunting): get_auth_request con 0x7fdba0006380 auth_method 0 2024-04-22T08:14:56.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.762+0000 7fdbb3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:56.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.762+0000 7fdbb3fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:56.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.762+0000 7fdbb3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:14:56.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.762+0000 7fdbb3fff640 1 -- 172.21.15.40:0/3904135143 >> v2:172.21.15.189:3300/0 conn(0x7fdba0006380 msgr2=0x7fdba0008840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:14:56.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.762+0000 7fdbb3fff640 1 -- 172.21.15.40:0/3904135143 >> v2:172.21.15.189:3300/0 conn(0x7fdba0006380 msgr2=0x7fdba0008840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:14:56.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.762+0000 7fdbb3fff640 1 --2- 172.21.15.40:0/3904135143 >> v2:172.21.15.189:3300/0 conn(0x7fdba0006380 0x7fdba0008840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:14:56.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:56.762+0000 7fdbb3fff640 1 --2- 172.21.15.40:0/3904135143 >> v2:172.21.15.189:3300/0 conn(0x7fdba0006380 0x7fdba0008840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:14:57.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:57.163+0000 7fdbb886a640 1 --2- 172.21.15.40:0/3904135143 >> v2:172.21.15.40:3300/0 conn(0x7fdba0003810 0x7fdba0005c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:57.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:57.163+0000 7fdbb886a640 10 monclient(hunting): get_auth_request con 0x7fdba0003810 auth_method 0 2024-04-22T08:14:57.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:57.163+0000 7fdbb886a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:57.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:57.163+0000 7fdbb886a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:57.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:57.163+0000 7fdbb886a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:14:57.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:57.163+0000 7fdbb886a640 1 -- 172.21.15.40:0/3904135143 >> v2:172.21.15.40:3300/0 conn(0x7fdba0003810 msgr2=0x7fdba0005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:14:57.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:57.163+0000 7fdbb886a640 1 -- 172.21.15.40:0/3904135143 >> v2:172.21.15.40:3300/0 conn(0x7fdba0003810 msgr2=0x7fdba0005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:14:57.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:57.163+0000 7fdbb886a640 1 --2- 172.21.15.40:0/3904135143 >> v2:172.21.15.40:3300/0 conn(0x7fdba0003810 0x7fdba0005c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:14:57.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:57.163+0000 7fdbb886a640 1 --2- 172.21.15.40:0/3904135143 >> v2:172.21.15.40:3300/0 conn(0x7fdba0003810 0x7fdba0005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:14:57.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:57.163+0000 7fdbb3fff640 1 --2- 172.21.15.40:0/3904135143 >> v2:172.21.15.189:3300/0 conn(0x7fdba0006380 0x7fdba0008840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:57.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:57.163+0000 7fdbb3fff640 10 monclient(hunting): get_auth_request con 0x7fdba0006380 auth_method 0 2024-04-22T08:14:57.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:57.163+0000 7fdbb3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:57.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:57.163+0000 7fdbb3fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:57.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:57.163+0000 7fdbb3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:14:57.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:57.163+0000 7fdbb3fff640 1 -- 172.21.15.40:0/3904135143 >> v2:172.21.15.189:3300/0 conn(0x7fdba0006380 msgr2=0x7fdba0008840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:14:57.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:57.163+0000 7fdbb3fff640 1 -- 172.21.15.40:0/3904135143 >> v2:172.21.15.189:3300/0 conn(0x7fdba0006380 msgr2=0x7fdba0008840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:14:57.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:57.163+0000 7fdbb3fff640 1 --2- 172.21.15.40:0/3904135143 >> v2:172.21.15.189:3300/0 conn(0x7fdba0006380 0x7fdba0008840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:14:57.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:57.163+0000 7fdbb3fff640 1 --2- 172.21.15.40:0/3904135143 >> v2:172.21.15.189:3300/0 conn(0x7fdba0006380 0x7fdba0008840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:14:57.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:57.965+0000 7fdbb886a640 1 --2- 172.21.15.40:0/3904135143 >> v2:172.21.15.40:3300/0 conn(0x7fdba0003810 0x7fdba0005c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:57.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:57.965+0000 7fdbb886a640 10 monclient(hunting): get_auth_request con 0x7fdba0003810 auth_method 0 2024-04-22T08:14:57.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:57.965+0000 7fdbb886a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:57.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:57.965+0000 7fdbb886a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:57.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:57.965+0000 7fdbb886a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:14:57.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:57.965+0000 7fdbb886a640 1 -- 172.21.15.40:0/3904135143 >> v2:172.21.15.40:3300/0 conn(0x7fdba0003810 msgr2=0x7fdba0005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:14:57.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:57.965+0000 7fdbb886a640 1 -- 172.21.15.40:0/3904135143 >> v2:172.21.15.40:3300/0 conn(0x7fdba0003810 msgr2=0x7fdba0005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:14:57.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:57.965+0000 7fdbb886a640 1 --2- 172.21.15.40:0/3904135143 >> v2:172.21.15.40:3300/0 conn(0x7fdba0003810 0x7fdba0005c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:14:57.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:57.965+0000 7fdbb886a640 1 --2- 172.21.15.40:0/3904135143 >> v2:172.21.15.40:3300/0 conn(0x7fdba0003810 0x7fdba0005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T08:14:57.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:57.965+0000 7fdbb3fff640 1 --2- 172.21.15.40:0/3904135143 >> v2:172.21.15.189:3300/0 conn(0x7fdba0006380 0x7fdba0008840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:57.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:57.965+0000 7fdbb3fff640 10 monclient(hunting): get_auth_request con 0x7fdba0006380 auth_method 0 2024-04-22T08:14:57.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:57.965+0000 7fdbb3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:57.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:57.965+0000 7fdbb3fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:57.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:57.965+0000 7fdbb3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:14:57.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:57.965+0000 7fdbb3fff640 1 -- 172.21.15.40:0/3904135143 >> v2:172.21.15.189:3300/0 conn(0x7fdba0006380 msgr2=0x7fdba0008840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:14:57.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:57.965+0000 7fdbb3fff640 1 -- 172.21.15.40:0/3904135143 >> v2:172.21.15.189:3300/0 conn(0x7fdba0006380 msgr2=0x7fdba0008840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:14:57.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:57.965+0000 7fdbb3fff640 1 --2- 172.21.15.40:0/3904135143 >> v2:172.21.15.189:3300/0 conn(0x7fdba0006380 0x7fdba0008840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:14:57.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:57.965+0000 7fdbb3fff640 1 --2- 172.21.15.40:0/3904135143 >> v2:172.21.15.189:3300/0 conn(0x7fdba0006380 0x7fdba0008840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T08:14:59.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:59.561+0000 7fdbb27fc640 10 monclient(hunting): tick 2024-04-22T08:14:59.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:59.561+0000 7fdbb27fc640 10 monclient(hunting): _check_auth_tickets 2024-04-22T08:14:59.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:59.561+0000 7fdbb27fc640 1 monclient(hunting): continuing hunt 2024-04-22T08:14:59.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:59.561+0000 7fdbb27fc640 10 monclient(hunting): _reopen_session rank -1 2024-04-22T08:14:59.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:59.561+0000 7fdbb27fc640 1 -- 172.21.15.40:0/3904135143 >> v2:172.21.15.189:3300/0 conn(0x7fdba0006380 msgr2=0x7fdba0008840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:14:59.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:59.561+0000 7fdbb27fc640 1 --2- 172.21.15.40:0/3904135143 >> v2:172.21.15.189:3300/0 conn(0x7fdba0006380 0x7fdba0008840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:59.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:59.561+0000 7fdbb27fc640 1 -- 172.21.15.40:0/3904135143 >> v2:172.21.15.40:3300/0 conn(0x7fdba0003810 msgr2=0x7fdba0005c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:14:59.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:59.561+0000 7fdbb27fc640 1 --2- 172.21.15.40:0/3904135143 >> v2:172.21.15.40:3300/0 conn(0x7fdba0003810 0x7fdba0005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:14:59.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:59.561+0000 7fdbb27fc640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-04-22T08:14:59.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:59.561+0000 7fdbb27fc640 1 --2- 172.21.15.40:0/3904135143 >> v2:172.21.15.40:3300/0 conn(0x7fdba0009310 0x7fdba000b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:59.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:59.561+0000 7fdbb906b640 1 -- 172.21.15.40:0/3904135143 reap_dead start 2024-04-22T08:14:59.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:59.561+0000 7fdbb27fc640 10 monclient(hunting): picked mon.noname-a con 0x7fdba0009310 addr v2:172.21.15.40:3300/0 2024-04-22T08:14:59.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:59.561+0000 7fdbb27fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdba0003c80 addr v1:172.21.15.189:6789/0 2024-04-22T08:14:59.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:59.561+0000 7fdbb27fc640 1 --2- 172.21.15.40:0/3904135143 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdba0007f50 0x7fdba00083a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:14:59.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:59.561+0000 7fdbb27fc640 10 monclient(hunting): picked mon.noname-b con 0x7fdba0007f50 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:14:59.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:59.561+0000 7fdbb27fc640 1 -- 172.21.15.40:0/3904135143 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbb409f130 con 0x7fdba0003c80 2024-04-22T08:14:59.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:59.561+0000 7fdbb27fc640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:59.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:59.561+0000 7fdbb27fc640 1 -- 172.21.15.40:0/3904135143 --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbb409de80 con 0x7fdba0009310 2024-04-22T08:14:59.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:59.561+0000 7fdbb27fc640 10 monclient(hunting): start opening mon connection 2024-04-22T08:14:59.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:59.561+0000 7fdbb27fc640 1 -- 172.21.15.40:0/3904135143 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdba0008f00 con 0x7fdba0007f50 2024-04-22T08:14:59.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:59.561+0000 7fdbb27fc640 10 monclient(hunting): _renew_subs 2024-04-22T08:14:59.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:59.561+0000 7fdbb906b640 1 -- 172.21.15.40:0/3904135143 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdba0007f50 msgr2=0x7fdba00083a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:14:59.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:59.561+0000 7fdbb906b640 1 --2- 172.21.15.40:0/3904135143 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdba0007f50 0x7fdba00083a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:14:59.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:59.561+0000 7fdbb886a640 1 --2- 172.21.15.40:0/3904135143 >> v2:172.21.15.40:3300/0 conn(0x7fdba0009310 0x7fdba000b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:59.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:59.561+0000 7fdbb886a640 10 monclient(hunting): get_auth_request con 0x7fdba0009310 auth_method 0 2024-04-22T08:14:59.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:59.561+0000 7fdbb886a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:59.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:59.561+0000 7fdbb886a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:59.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:59.561+0000 7fdbb886a640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:14:59.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:59.562+0000 7fdbb886a640 1 -- 172.21.15.40:0/3904135143 >> v2:172.21.15.40:3300/0 conn(0x7fdba0009310 msgr2=0x7fdba000b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:14:59.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:59.562+0000 7fdbb886a640 1 -- 172.21.15.40:0/3904135143 >> v2:172.21.15.40:3300/0 conn(0x7fdba0009310 msgr2=0x7fdba000b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:14:59.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:59.562+0000 7fdbb886a640 1 --2- 172.21.15.40:0/3904135143 >> v2:172.21.15.40:3300/0 conn(0x7fdba0009310 0x7fdba000b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:14:59.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:59.562+0000 7fdbb886a640 1 --2- 172.21.15.40:0/3904135143 >> v2:172.21.15.40:3300/0 conn(0x7fdba0009310 0x7fdba000b720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:14:59.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:59.564+0000 7fdbb3fff640 1 -- 172.21.15.40:0/3904135143 >> v1:172.21.15.189:6789/0 conn(0x7fdba0003c80 legacy=0x7fdba0006830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:14:59.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:59.762+0000 7fdbb906b640 1 -- 172.21.15.40:0/3904135143 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdba0007f50 msgr2=0x7fdba00083a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:14:59.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:59.762+0000 7fdbb906b640 1 --2- 172.21.15.40:0/3904135143 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdba0007f50 0x7fdba00083a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:14:59.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:59.763+0000 7fdbb886a640 1 --2- 172.21.15.40:0/3904135143 >> v2:172.21.15.40:3300/0 conn(0x7fdba0009310 0x7fdba000b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:14:59.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:59.763+0000 7fdbb886a640 10 monclient(hunting): get_auth_request con 0x7fdba0009310 auth_method 0 2024-04-22T08:14:59.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:59.763+0000 7fdbb886a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:14:59.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:59.763+0000 7fdbb886a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:14:59.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:59.763+0000 7fdbb886a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:14:59.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:59.763+0000 7fdbb886a640 1 -- 172.21.15.40:0/3904135143 >> v2:172.21.15.40:3300/0 conn(0x7fdba0009310 msgr2=0x7fdba000b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:14:59.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:59.763+0000 7fdbb886a640 1 -- 172.21.15.40:0/3904135143 >> v2:172.21.15.40:3300/0 conn(0x7fdba0009310 msgr2=0x7fdba000b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:14:59.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:59.763+0000 7fdbb886a640 1 --2- 172.21.15.40:0/3904135143 >> v2:172.21.15.40:3300/0 conn(0x7fdba0009310 0x7fdba000b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:14:59.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:14:59.763+0000 7fdbb886a640 1 --2- 172.21.15.40:0/3904135143 >> v2:172.21.15.40:3300/0 conn(0x7fdba0009310 0x7fdba000b720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:15:00.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:00.163+0000 7fdbb906b640 1 -- 172.21.15.40:0/3904135143 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdba0007f50 msgr2=0x7fdba00083a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:15:00.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:00.163+0000 7fdbb906b640 1 --2- 172.21.15.40:0/3904135143 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdba0007f50 0x7fdba00083a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:15:00.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:00.164+0000 7fdbb886a640 1 --2- 172.21.15.40:0/3904135143 >> v2:172.21.15.40:3300/0 conn(0x7fdba0009310 0x7fdba000b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:00.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:00.164+0000 7fdbb886a640 10 monclient(hunting): get_auth_request con 0x7fdba0009310 auth_method 0 2024-04-22T08:15:00.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:00.164+0000 7fdbb886a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:00.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:00.164+0000 7fdbb886a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:00.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:00.164+0000 7fdbb886a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:15:00.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:00.164+0000 7fdbb886a640 1 -- 172.21.15.40:0/3904135143 >> v2:172.21.15.40:3300/0 conn(0x7fdba0009310 msgr2=0x7fdba000b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:15:00.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:00.164+0000 7fdbb886a640 1 -- 172.21.15.40:0/3904135143 >> v2:172.21.15.40:3300/0 conn(0x7fdba0009310 msgr2=0x7fdba000b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:15:00.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:00.164+0000 7fdbb886a640 1 --2- 172.21.15.40:0/3904135143 >> v2:172.21.15.40:3300/0 conn(0x7fdba0009310 0x7fdba000b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:15:00.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:00.164+0000 7fdbb886a640 1 --2- 172.21.15.40:0/3904135143 >> v2:172.21.15.40:3300/0 conn(0x7fdba0009310 0x7fdba000b720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:15:00.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:00.964+0000 7fdbb906b640 1 -- 172.21.15.40:0/3904135143 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdba0007f50 msgr2=0x7fdba00083a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:15:00.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:00.964+0000 7fdbb906b640 1 --2- 172.21.15.40:0/3904135143 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdba0007f50 0x7fdba00083a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T08:15:00.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:00.965+0000 7fdbb886a640 1 --2- 172.21.15.40:0/3904135143 >> v2:172.21.15.40:3300/0 conn(0x7fdba0009310 0x7fdba000b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:00.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:00.965+0000 7fdbb886a640 10 monclient(hunting): get_auth_request con 0x7fdba0009310 auth_method 0 2024-04-22T08:15:00.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:00.965+0000 7fdbb886a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:00.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:00.965+0000 7fdbb886a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:00.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:00.965+0000 7fdbb886a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:15:00.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:00.965+0000 7fdbb886a640 1 -- 172.21.15.40:0/3904135143 >> v2:172.21.15.40:3300/0 conn(0x7fdba0009310 msgr2=0x7fdba000b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:15:00.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:00.965+0000 7fdbb886a640 1 -- 172.21.15.40:0/3904135143 >> v2:172.21.15.40:3300/0 conn(0x7fdba0009310 msgr2=0x7fdba000b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:15:00.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:00.965+0000 7fdbb886a640 1 --2- 172.21.15.40:0/3904135143 >> v2:172.21.15.40:3300/0 conn(0x7fdba0009310 0x7fdba000b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:15:00.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:00.965+0000 7fdbb886a640 1 --2- 172.21.15.40:0/3904135143 >> v2:172.21.15.40:3300/0 conn(0x7fdba0009310 0x7fdba000b720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T08:15:01.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.219+0000 7fdbb37fe640 1 -- 172.21.15.40:0/3904135143 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 270642005 0 0) 0x7fdbb409f130 con 0x7fdba0003c80 2024-04-22T08:15:01.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.219+0000 7fdbb37fe640 10 monclient(hunting): _negotiate 2024-04-22T08:15:01.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.219+0000 7fdbb37fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:01.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.219+0000 7fdbb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:01.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.219+0000 7fdbb37fe640 10 monclient(hunting): my global_id is 24382 2024-04-22T08:15:01.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.219+0000 7fdbb37fe640 1 -- 172.21.15.40:0/3904135143 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb9c003520 con 0x7fdba0003c80 2024-04-22T08:15:01.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.229+0000 7fdbb37fe640 1 -- 172.21.15.40:0/3904135143 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4167157443 0 0) 0x7fdb9c003520 con 0x7fdba0003c80 2024-04-22T08:15:01.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.229+0000 7fdbb37fe640 1 -- 172.21.15.40:0/3904135143 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdba0007f50 msgr2=0x7fdba00083a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:15:01.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.229+0000 7fdbb37fe640 1 --2- 172.21.15.40:0/3904135143 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdba0007f50 0x7fdba00083a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:01.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.229+0000 7fdbb37fe640 1 -- 172.21.15.40:0/3904135143 >> v2:172.21.15.40:3300/0 conn(0x7fdba0009310 msgr2=0x7fdba000b720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:15:01.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.229+0000 7fdbb37fe640 1 --2- 172.21.15.40:0/3904135143 >> v2:172.21.15.40:3300/0 conn(0x7fdba0009310 0x7fdba000b720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:01.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.229+0000 7fdbb37fe640 10 monclient: _finish_hunting 0 2024-04-22T08:15:01.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.229+0000 7fdbb37fe640 1 monclient: found mon.noname-c-legacy 2024-04-22T08:15:01.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.229+0000 7fdbb37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:15:01.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.229+0000 7fdbb37fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.189:6789/0 2024-04-22T08:15:01.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.229+0000 7fdbb37fe640 1 -- 172.21.15.40:0/3904135143 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba0008b10 con 0x7fdba0003c80 2024-04-22T08:15:01.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.229+0000 7fdbb37fe640 10 monclient: _finish_auth 0 2024-04-22T08:15:01.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.229+0000 7fdbb37fe640 10 monclient: _check_auth_tickets 2024-04-22T08:15:01.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.229+0000 7fdbb37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:15:01.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.229+0000 7fdbb37fe640 1 -- 172.21.15.40:0/3904135143 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1087006497 0 0) 0x7fdba8007fe0 con 0x7fdba0003c80 2024-04-22T08:15:01.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.229+0000 7fdbbaace640 5 monclient: authenticate success, global_id 24382 2024-04-22T08:15:01.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.229+0000 7fdbbaace640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:15:01.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.229+0000 7fdbb37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:01.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.229+0000 7fdbb37fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-04-22T08:15:01.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.229+0000 7fdbb37fe640 10 monclient: dump: 2024-04-22T08:15:01.233 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:01.233 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:01.233 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:01.233 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:01.234 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:01.234 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:01.234 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:01.234 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:01.234 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:01.234 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:01.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.229+0000 7fdbb37fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] but i'm connected to v1:172.21.15.189:6789/0, reconnecting 2024-04-22T08:15:01.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.229+0000 7fdbb37fe640 10 monclient: _reopen_session rank -1 2024-04-22T08:15:01.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.229+0000 7fdbb37fe640 1 -- 172.21.15.40:0/3904135143 >> v1:172.21.15.189:6789/0 conn(0x7fdba0003c80 legacy=0x7fdba0006830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:01.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.229+0000 7fdbb37fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:15:01.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.229+0000 7fdbb37fe640 1 --2- 172.21.15.40:0/3904135143 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb9c003970 0x7fdb9c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:01.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.229+0000 7fdbb37fe640 10 monclient(hunting): picked mon.a con 0x7fdb9c003970 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:15:01.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.229+0000 7fdbb37fe640 1 --2- 172.21.15.40:0/3904135143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb9c0047c0 0x7fdb9c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:01.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.229+0000 7fdbb37fe640 10 monclient(hunting): picked mon.b con 0x7fdb9c0047c0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:15:01.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.229+0000 7fdbb886a640 1 --2- 172.21.15.40:0/3904135143 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb9c003970 0x7fdb9c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:01.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.229+0000 7fdbb37fe640 1 --2- 172.21.15.40:0/3904135143 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdba0007f50 0x7fdb9c005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:01.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.230+0000 7fdbb906b640 1 -- 172.21.15.40:0/3904135143 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdba0007f50 msgr2=0x7fdb9c005250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:15:01.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.230+0000 7fdbb906b640 1 --2- 172.21.15.40:0/3904135143 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdba0007f50 0x7fdb9c005250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:15:01.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.230+0000 7fdbb37fe640 10 monclient(hunting): picked mon.c con 0x7fdba0007f50 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:15:01.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.230+0000 7fdbb37fe640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:01.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.230+0000 7fdbb37fe640 1 -- 172.21.15.40:0/3904135143 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb409de80 con 0x7fdb9c003970 2024-04-22T08:15:01.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.230+0000 7fdbb37fe640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:01.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.230+0000 7fdbb37fe640 1 -- 172.21.15.40:0/3904135143 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba0008f00 con 0x7fdb9c0047c0 2024-04-22T08:15:01.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.230+0000 7fdbb37fe640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:01.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.230+0000 7fdbb37fe640 1 -- 172.21.15.40:0/3904135143 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9c00bbc0 con 0x7fdba0007f50 2024-04-22T08:15:01.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.230+0000 7fdbb37fe640 10 monclient(hunting): _renew_subs 2024-04-22T08:15:01.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.230+0000 7fdbb37fe640 10 monclient(hunting): _finish_auth 0 2024-04-22T08:15:01.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.230+0000 7fdbb886a640 10 monclient(hunting): get_auth_request con 0x7fdb9c003970 auth_method 0 2024-04-22T08:15:01.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.230+0000 7fdbb886a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:01.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.230+0000 7fdbb886a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:01.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.230+0000 7fdbb886a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:15:01.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.230+0000 7fdbbaace640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-04-22T08:15:01.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.230+0000 7fdbb3fff640 1 --2- 172.21.15.40:0/3904135143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb9c0047c0 0x7fdb9c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:01.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.230+0000 7fdbb886a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:01.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.230+0000 7fdbb886a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:01.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.230+0000 7fdbb886a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-22T08:15:01.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.230+0000 7fdbb3fff640 10 monclient(hunting): get_auth_request con 0x7fdb9c0047c0 auth_method 0 2024-04-22T08:15:01.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.230+0000 7fdbb3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:01.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.230+0000 7fdbb3fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:01.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.230+0000 7fdbb3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:15:01.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.230+0000 7fdbb3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:01.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.230+0000 7fdbb3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:01.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.230+0000 7fdbb3fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-22T08:15:01.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.230+0000 7fdbb886a640 10 monclient(hunting): handle_auth_done global_id 24382 payload 293 2024-04-22T08:15:01.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.230+0000 7fdbb886a640 1 -- 172.21.15.40:0/3904135143 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdba0007f50 msgr2=0x7fdb9c005250 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:15:01.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.230+0000 7fdbb886a640 1 --2- 172.21.15.40:0/3904135143 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdba0007f50 0x7fdb9c005250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:01.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.230+0000 7fdbb886a640 1 -- 172.21.15.40:0/3904135143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb9c0047c0 msgr2=0x7fdb9c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:15:01.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.230+0000 7fdbb886a640 1 --2- 172.21.15.40:0/3904135143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb9c0047c0 0x7fdb9c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:01.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.230+0000 7fdbb886a640 10 monclient: _finish_hunting 0 2024-04-22T08:15:01.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.230+0000 7fdbb886a640 1 monclient: found mon.a 2024-04-22T08:15:01.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.230+0000 7fdbb886a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:15:01.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.230+0000 7fdbb886a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:01.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.230+0000 7fdbb886a640 1 -- 172.21.15.40:0/3904135143 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb9c003520 con 0x7fdb9c003970 2024-04-22T08:15:01.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.230+0000 7fdbb886a640 10 monclient: _finish_auth 0 2024-04-22T08:15:01.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.230+0000 7fdbb886a640 10 monclient: _check_auth_tickets 2024-04-22T08:15:01.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.230+0000 7fdbb886a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:15:01.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.234+0000 7fdbb886a640 1 -- 172.21.15.40:0/3904135143 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb9c003970 msgr2=0x7fdb9c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:01.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.234+0000 7fdbb886a640 1 --2- 172.21.15.40:0/3904135143 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb9c003970 0x7fdb9c003d90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdba00062c0 tx=0x7fdb9c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bcc4990d2b43e25 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:01.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.234+0000 7fdbb37fe640 1 -- 172.21.15.40:0/3904135143 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbac02e970 con 0x7fdb9c003970 2024-04-22T08:15:01.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.234+0000 7fdbb37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:01.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.234+0000 7fdbb37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:15:01.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.234+0000 7fdbb37fe640 10 monclient: dump: 2024-04-22T08:15:01.244 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:01.244 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:01.245 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:01.245 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:01.245 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:01.245 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:01.245 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:01.245 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:01.245 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:01.245 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:01.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.234+0000 7fdbb37fe640 1 -- 172.21.15.40:0/3904135143 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbac016040 con 0x7fdb9c003970 2024-04-22T08:15:01.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.234+0000 7fdbbaace640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:15:01.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.234+0000 7fdbb37fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:15:01.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.234+0000 7fdbb37fe640 1 -- 172.21.15.40:0/3904135143 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbac02ed10 con 0x7fdb9c003970 2024-04-22T08:15:01.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.234+0000 7fdbbaace640 10 monclient: get_monmap_and_config success 2024-04-22T08:15:01.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.234+0000 7fdbbaace640 4 set_mon_vals no callback set 2024-04-22T08:15:01.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.234+0000 7fdbb37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:01.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.234+0000 7fdbb37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:15:01.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.234+0000 7fdbb37fe640 10 monclient: dump: 2024-04-22T08:15:01.246 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:01.246 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:01.246 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:01.246 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:01.246 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:01.246 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:01.246 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:01.247 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:01.247 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:01.247 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:01.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.235+0000 7fdbbaace640 10 monclient: shutdown 2024-04-22T08:15:01.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.235+0000 7fdbbaace640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:15:01.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.235+0000 7fdbbaace640 1 -- 172.21.15.40:0/3904135143 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb9c003970 msgr2=0x7fdb9c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:01.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.235+0000 7fdbbaace640 1 --2- 172.21.15.40:0/3904135143 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb9c003970 0x7fdb9c003d90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdba00062c0 tx=0x7fdb9c0057f0 comp rx=0 tx=0).stop 2024-04-22T08:15:01.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.235+0000 7fdbb906b640 1 -- 172.21.15.40:0/3904135143 reap_dead start 2024-04-22T08:15:01.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.235+0000 7fdbbaace640 1 -- 172.21.15.40:0/3904135143 shutdown_connections 2024-04-22T08:15:01.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.235+0000 7fdbbaace640 1 -- 172.21.15.40:0/3904135143 >> 172.21.15.40:0/3904135143 conn(0x7fdbb40a65c0 msgr2=0x7fdbb40a2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:15:01.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.235+0000 7fdbbaace640 1 -- 172.21.15.40:0/3904135143 shutdown_connections 2024-04-22T08:15:01.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.235+0000 7fdbbaace640 1 -- 172.21.15.40:0/3904135143 wait complete. 2024-04-22T08:15:01.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.236+0000 7fdbbaace640 10 monclient: build_initial_monmap 2024-04-22T08:15:01.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.236+0000 7fdbbaace640 10 monclient: monmap: 2024-04-22T08:15:01.248 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:15:01.248 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:15:01.248 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:15:01.248 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:15:01.248 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:15:01.248 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:01.248 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:15:01.248 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:15:01.248 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:15:01.248 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:01.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.236+0000 7fdbbaace640 1 Processor -- start 2024-04-22T08:15:01.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.236+0000 7fdbbaace640 1 -- start start 2024-04-22T08:15:01.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.236+0000 7fdbbaace640 10 monclient: init 2024-04-22T08:15:01.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.237+0000 7fdbbaace640 10 monclient: _reopen_session rank -1 2024-04-22T08:15:01.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.237+0000 7fdbbaace640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:15:01.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.237+0000 7fdbbaace640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb9c003970 0x7fdbb41ca4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:01.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.237+0000 7fdbbaace640 10 monclient(hunting): picked mon.noname-a con 0x7fdb9c003970 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:15:01.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.237+0000 7fdbbaace640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbb4133d90 0x7fdbb41341f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:01.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.237+0000 7fdbbaace640 10 monclient(hunting): picked mon.noname-b con 0x7fdbb4133d90 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:15:01.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.237+0000 7fdbbaace640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbb406ea30 0x7fdbb406eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:01.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.237+0000 7fdbbaace640 10 monclient(hunting): picked mon.noname-c con 0x7fdbb406ea30 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:15:01.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.237+0000 7fdbbaace640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:01.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.237+0000 7fdbbaace640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb41217e0 con 0x7fdb9c003970 2024-04-22T08:15:01.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.237+0000 7fdbbaace640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:01.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.237+0000 7fdbbaace640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb40720e0 con 0x7fdbb4133d90 2024-04-22T08:15:01.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.237+0000 7fdbbaace640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:01.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.237+0000 7fdbbaace640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb4134ff0 con 0x7fdbb406ea30 2024-04-22T08:15:01.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.237+0000 7fdbbaace640 10 monclient(hunting): _renew_subs 2024-04-22T08:15:01.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.237+0000 7fdbbaace640 10 monclient(hunting): authenticate will time out at 975.232153s 2024-04-22T08:15:01.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.237+0000 7fdbb906b640 1 -- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbb406ea30 msgr2=0x7fdbb406eee0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:15:01.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.237+0000 7fdbb906b640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbb406ea30 0x7fdbb406eee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:15:01.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.237+0000 7fdbb886a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb9c003970 0x7fdbb41ca4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:01.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.237+0000 7fdbb886a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb9c003970 0x7fdbb41ca4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52310/0 (socket says 172.21.15.40:52310) 2024-04-22T08:15:01.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.237+0000 7fdbb886a640 1 -- 172.21.15.40:0/2760348187 learned_addr learned my addr 172.21.15.40:0/2760348187 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:15:01.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.237+0000 7fdbb886a640 10 monclient(hunting): get_auth_request con 0x7fdb9c003970 auth_method 0 2024-04-22T08:15:01.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.237+0000 7fdbb886a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:01.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.237+0000 7fdbb886a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:01.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.237+0000 7fdbb886a640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:01.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.237+0000 7fdbb3fff640 1 --2- 172.21.15.40:0/2760348187 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbb4133d90 0x7fdbb41341f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:01.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.237+0000 7fdbb3fff640 10 monclient(hunting): get_auth_request con 0x7fdbb4133d90 auth_method 0 2024-04-22T08:15:01.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.237+0000 7fdbb3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:01.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.237+0000 7fdbb3fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:01.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.237+0000 7fdbb3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:01.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.237+0000 7fdbb886a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:01.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.237+0000 7fdbb886a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:01.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.237+0000 7fdbb886a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:01.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.237+0000 7fdbb3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:01.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.237+0000 7fdbb3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:01.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.237+0000 7fdbb3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:01.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.237+0000 7fdbb886a640 10 monclient(hunting): handle_auth_done global_id 34179 payload 824 2024-04-22T08:15:01.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.237+0000 7fdbb886a640 1 -- 172.21.15.40:0/2760348187 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbb406ea30 msgr2=0x7fdbb406eee0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:15:01.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.237+0000 7fdbb886a640 1 --2- 172.21.15.40:0/2760348187 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbb406ea30 0x7fdbb406eee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:01.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.237+0000 7fdbb886a640 1 -- 172.21.15.40:0/2760348187 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbb4133d90 msgr2=0x7fdbb41341f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:01.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.237+0000 7fdbb886a640 1 --2- 172.21.15.40:0/2760348187 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbb4133d90 0x7fdbb41341f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:01.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.237+0000 7fdbb886a640 10 monclient: _finish_hunting 0 2024-04-22T08:15:01.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.237+0000 7fdbb886a640 1 monclient: found mon.noname-a 2024-04-22T08:15:01.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.237+0000 7fdbb886a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:15:01.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.237+0000 7fdbb886a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:15:01.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.237+0000 7fdbb886a640 1 -- 172.21.15.40:0/2760348187 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbb4070550 con 0x7fdb9c003970 2024-04-22T08:15:01.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.240+0000 7fdbb886a640 1 -- 172.21.15.40:0/2760348187 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb9c003970 msgr2=0x7fdbb41ca4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:01.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.240+0000 7fdbb886a640 1 --2- 172.21.15.40:0/2760348187 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb9c003970 0x7fdbb41ca4f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdbac003d00 tx=0x7fdbac0060c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:01.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.240+0000 7fdbb17fa640 1 -- 172.21.15.40:0/2760348187 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbac036a60 con 0x7fdb9c003970 2024-04-22T08:15:01.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.240+0000 7fdbb17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:01.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.240+0000 7fdbb17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:15:01.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.240+0000 7fdbb17fa640 10 monclient: dump: 2024-04-22T08:15:01.253 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:01.253 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:01.253 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:01.253 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:01.253 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:01.253 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:01.253 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:01.253 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:01.253 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:01.253 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:01.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.240+0000 7fdbb17fa640 10 monclient: _finish_auth 0 2024-04-22T08:15:01.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.240+0000 7fdbb17fa640 10 monclient: _check_auth_tickets 2024-04-22T08:15:01.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.240+0000 7fdbb17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:15:01.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.240+0000 7fdbb17fa640 1 -- 172.21.15.40:0/2760348187 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbac036bc0 con 0x7fdb9c003970 2024-04-22T08:15:01.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.240+0000 7fdbb17fa640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:15:01.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.240+0000 7fdbb17fa640 1 -- 172.21.15.40:0/2760348187 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbac040410 con 0x7fdb9c003970 2024-04-22T08:15:01.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.240+0000 7fdbb17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:01.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.240+0000 7fdbb17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:15:01.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.240+0000 7fdbb17fa640 10 monclient: dump: 2024-04-22T08:15:01.254 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:01.254 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:01.254 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:01.254 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:01.254 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:01.254 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:01.254 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:01.255 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:01.255 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:01.255 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:01.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.240+0000 7fdbbaace640 5 monclient: authenticate success, global_id 34179 2024-04-22T08:15:01.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.240+0000 7fdbbaace640 10 monclient: _renew_subs 2024-04-22T08:15:01.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.240+0000 7fdbbaace640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:01.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.240+0000 7fdbbaace640 1 -- 172.21.15.40:0/2760348187 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbb4120eb0 con 0x7fdb9c003970 2024-04-22T08:15:01.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.240+0000 7fdbb2ffd640 4 set_mon_vals no callback set 2024-04-22T08:15:01.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.240+0000 7fdbbaace640 10 monclient: _renew_subs 2024-04-22T08:15:01.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.240+0000 7fdbbaace640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:01.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.240+0000 7fdbbaace640 1 -- 172.21.15.40:0/2760348187 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbb40712f0 con 0x7fdb9c003970 2024-04-22T08:15:01.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.241+0000 7fdbb17fa640 1 -- 172.21.15.40:0/2760348187 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fdbac03d050 con 0x7fdb9c003970 2024-04-22T08:15:01.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.241+0000 7fdbbaace640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:01.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.241+0000 7fdbb17fa640 1 --2- 172.21.15.40:0/2760348187 >> v2:172.21.15.189:6800/30168 conn(0x7fdba00392e0 0x7fdba003b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:01.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.241+0000 7fdbbaace640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:01.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.241+0000 7fdbbaace640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:01.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.241+0000 7fdbbaace640 1 -- 172.21.15.40:0/2760348187 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbb404e9b0 con 0x7fdb9c003970 2024-04-22T08:15:01.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.241+0000 7fdbb17fa640 1 -- 172.21.15.40:0/2760348187 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 3898+0+0 (secure 0 0 0) 0x7fdbac06dc30 con 0x7fdb9c003970 2024-04-22T08:15:01.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.242+0000 7fdbb3fff640 1 --2- 172.21.15.40:0/2760348187 >> v2:172.21.15.189:6800/30168 conn(0x7fdba00392e0 0x7fdba003b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:01.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.242+0000 7fdbb3fff640 10 monclient: get_auth_request con 0x7fdba00392e0 auth_method 0 2024-04-22T08:15:01.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.244+0000 7fdbb3fff640 1 --2- 172.21.15.40:0/2760348187 >> v2:172.21.15.189:6800/30168 conn(0x7fdba00392e0 0x7fdba003b7a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fdbb40a5160 tx=0x7fdba8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:01.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.247+0000 7fdbb17fa640 1 -- 172.21.15.40:0/2760348187 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7fdbac03b070 con 0x7fdb9c003970 2024-04-22T08:15:01.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.247+0000 7fdbb17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:01.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.247+0000 7fdbb17fa640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:15:01.332 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:15:01.332 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35418]: starting. 2024-04-22T08:15:01.332 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35418]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-35381 2024-04-22T08:15:01.332 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35418]: created object 0... 2024-04-22T08:15:01.332 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35418]: created object 25... 2024-04-22T08:15:01.332 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35418]: created object 49... 2024-04-22T08:15:01.332 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35418]: finishing. 2024-04-22T08:15:01.332 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35418]: shutting down. 2024-04-22T08:15:01.371 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:15:01.371 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35420]: starting. 2024-04-22T08:15:01.371 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35420]: listing objects. 2024-04-22T08:15:01.371 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35420]: listed object 0... 2024-04-22T08:15:01.371 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35420]: listed object 25... 2024-04-22T08:15:01.371 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35420]: saw 50 objects 2024-04-22T08:15:01.372 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35420]: shutting down. 2024-04-22T08:15:01.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.507+0000 7fdbbaace640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:15:01.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.507+0000 7fdbbaace640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:15:01.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.507+0000 7fdbbaace640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:01.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.507+0000 7fdbbaace640 1 -- 172.21.15.40:0/2760348187 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fdbb41209f0 con 0x7fdb9c003970 2024-04-22T08:15:01.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.507+0000 7fdbb17fa640 1 -- 172.21.15.40:0/2760348187 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1537 (secure 0 0 0) 0x7fdbac058080 con 0x7fdb9c003970 2024-04-22T08:15:01.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.507+0000 7fdbb17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:15:01.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.507+0000 7fdbb17fa640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:15:01.551 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":30,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:15:01.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.509+0000 7fdbbaace640 1 -- 172.21.15.40:0/2760348187 >> v2:172.21.15.189:6800/30168 conn(0x7fdba00392e0 msgr2=0x7fdba003b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:01.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.509+0000 7fdbbaace640 1 --2- 172.21.15.40:0/2760348187 >> v2:172.21.15.189:6800/30168 conn(0x7fdba00392e0 0x7fdba003b7a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fdbb40a5160 tx=0x7fdba8009040 comp rx=0 tx=0).stop 2024-04-22T08:15:01.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.509+0000 7fdbbaace640 10 monclient: shutdown 2024-04-22T08:15:01.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.509+0000 7fdbbaace640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:15:01.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.509+0000 7fdbbaace640 1 -- 172.21.15.40:0/2760348187 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb9c003970 msgr2=0x7fdbb41ca4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:01.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.509+0000 7fdbbaace640 1 --2- 172.21.15.40:0/2760348187 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb9c003970 0x7fdbb41ca4f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdbac003d00 tx=0x7fdbac0060c0 comp rx=0 tx=0).stop 2024-04-22T08:15:01.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.509+0000 7fdbbaace640 1 -- 172.21.15.40:0/2760348187 shutdown_connections 2024-04-22T08:15:01.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.509+0000 7fdbbaace640 1 --2- 172.21.15.40:0/2760348187 >> v2:172.21.15.189:6800/30168 conn(0x7fdba00392e0 0x7fdba003b7a0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:01.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.509+0000 7fdbbaace640 1 --2- 172.21.15.40:0/2760348187 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbb4133d90 0x7fdbb41341f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:01.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.509+0000 7fdbbaace640 1 --2- 172.21.15.40:0/2760348187 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbb406ea30 0x7fdbb406eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:01.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.509+0000 7fdbbaace640 1 --2- 172.21.15.40:0/2760348187 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb9c003970 0x7fdbb41ca4f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:01.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.509+0000 7fdbbaace640 1 -- 172.21.15.40:0/2760348187 >> 172.21.15.40:0/2760348187 conn(0x7fdbb40a65c0 msgr2=0x7fdbb41bddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:15:01.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.509+0000 7fdbbaace640 1 -- 172.21.15.40:0/2760348187 shutdown_connections 2024-04-22T08:15:01.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.509+0000 7fdbbaace640 1 -- 172.21.15.40:0/2760348187 wait complete. 2024-04-22T08:15:01.554 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-22T08:15:01.554 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-04-22T08:15:01.554 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-22T08:15:01.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.654+0000 7fe9c8046640 10 monclient: get_monmap_and_config 2024-04-22T08:15:01.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.654+0000 7fe9c8046640 10 monclient: build_initial_monmap 2024-04-22T08:15:01.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.654+0000 7fe9c8046640 10 monclient: monmap: 2024-04-22T08:15:01.655 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:15:01.655 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:01.655 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:15:01.655942+0000 2024-04-22T08:15:01.655 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:15:01.655942+0000 2024-04-22T08:15:01.655 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:15:01.655 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:01.655 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:15:01.655 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:15:01.655 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:15:01.655 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:15:01.655 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:15:01.655 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:01.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.654+0000 7fe9c8046640 1 Processor -- start 2024-04-22T08:15:01.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.654+0000 7fe9c8046640 1 -- start start 2024-04-22T08:15:01.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.654+0000 7fe9c8046640 10 monclient: init 2024-04-22T08:15:01.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.655+0000 7fe9c8046640 10 monclient: _reopen_session rank -1 2024-04-22T08:15:01.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.655+0000 7fe9c8046640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-04-22T08:15:01.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.655+0000 7fe9c8046640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe9c00a1c60 addr v1:172.21.15.189:6789/0 2024-04-22T08:15:01.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.655+0000 7fe9c8046640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe9c00a31d0 0x7fe9c013aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:01.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.655+0000 7fe9c8046640 10 monclient(hunting): picked mon.noname-b con 0x7fe9c00a31d0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:15:01.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.655+0000 7fe9c8046640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe9c013b410 0x7fe9c013d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:01.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.655+0000 7fe9c8046640 10 monclient(hunting): picked mon.noname-a con 0x7fe9c013b410 addr v2:172.21.15.40:3300/0 2024-04-22T08:15:01.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.655+0000 7fe9c8046640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c00a41f0 con 0x7fe9c00a1c60 2024-04-22T08:15:01.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.655+0000 7fe9c8046640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:01.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.655+0000 7fe9c8046640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9c00a3630 con 0x7fe9c013b410 2024-04-22T08:15:01.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.655+0000 7fe9c8046640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:01.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.655+0000 7fe9c8046640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c009fe90 con 0x7fe9c00a31d0 2024-04-22T08:15:01.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.655+0000 7fe9c8046640 10 monclient(hunting): _renew_subs 2024-04-22T08:15:01.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.655+0000 7fe9c8046640 10 monclient(hunting): authenticate will time out at 975.650183s 2024-04-22T08:15:01.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.655+0000 7fe9c55e1640 1 -- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe9c00a31d0 msgr2=0x7fe9c013aed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:15:01.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.655+0000 7fe9c55e1640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe9c00a31d0 0x7fe9c013aed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:15:01.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.655+0000 7fe9c65e3640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe9c013b410 0x7fe9c013d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:01.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.655+0000 7fe9c65e3640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe9c013b410 0x7fe9c013d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52320/0 (socket says 172.21.15.40:52320) 2024-04-22T08:15:01.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.655+0000 7fe9c65e3640 1 -- 172.21.15.40:0/4139073738 learned_addr learned my addr 172.21.15.40:0/4139073738 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:15:01.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.655+0000 7fe9c65e3640 10 monclient(hunting): get_auth_request con 0x7fe9c013b410 auth_method 0 2024-04-22T08:15:01.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.655+0000 7fe9c65e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:01.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.655+0000 7fe9c65e3640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:01.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.655+0000 7fe9c65e3640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:01.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.655+0000 7fe9c65e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:01.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.655+0000 7fe9c65e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:01.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.655+0000 7fe9c65e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:01.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.655+0000 7fe9c65e3640 10 monclient(hunting): handle_auth_done global_id 34182 payload 274 2024-04-22T08:15:01.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.655+0000 7fe9c65e3640 1 -- 172.21.15.40:0/4139073738 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe9c00a31d0 msgr2=0x7fe9c013aed0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:15:01.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.655+0000 7fe9c65e3640 1 --2- 172.21.15.40:0/4139073738 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe9c00a31d0 0x7fe9c013aed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:01.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.655+0000 7fe9c65e3640 1 -- 172.21.15.40:0/4139073738 >> v1:172.21.15.189:6789/0 conn(0x7fe9c00a1c60 legacy=0x7fe9c00a2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:15:01.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.655+0000 7fe9c65e3640 10 monclient: _finish_hunting 0 2024-04-22T08:15:01.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.655+0000 7fe9c65e3640 1 monclient: found mon.noname-a 2024-04-22T08:15:01.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.655+0000 7fe9c65e3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:15:01.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.655+0000 7fe9c65e3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:15:01.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.655+0000 7fe9c65e3640 1 -- 172.21.15.40:0/4139073738 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c00a2750 con 0x7fe9c013b410 2024-04-22T08:15:01.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.658+0000 7fe9c65e3640 1 -- 172.21.15.40:0/4139073738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9c013b410 msgr2=0x7fe9c013d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:01.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.658+0000 7fe9c65e3640 1 --2- 172.21.15.40:0/4139073738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9c013b410 0x7fe9c013d800 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe9b8009e50 tx=0x7fe9b802bde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ed19221865cb9a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:01.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.660+0000 7fe9c4de0640 1 -- 172.21.15.40:0/4139073738 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b802cb90 con 0x7fe9c013b410 2024-04-22T08:15:01.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.661+0000 7fe9c4de0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:01.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.661+0000 7fe9c4de0640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:15:01.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.661+0000 7fe9c4de0640 10 monclient: dump: 2024-04-22T08:15:01.664 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:01.664 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:01.664 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:01.664 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:01.664 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:01.664 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:01.664 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:01.664 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:01.664 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:01.665 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:01.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.661+0000 7fe9c4de0640 10 monclient: _finish_auth 0 2024-04-22T08:15:01.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.661+0000 7fe9c4de0640 10 monclient: _check_auth_tickets 2024-04-22T08:15:01.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.661+0000 7fe9c4de0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:15:01.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.661+0000 7fe9c4de0640 1 -- 172.21.15.40:0/4139073738 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9b802ccf0 con 0x7fe9c013b410 2024-04-22T08:15:01.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.661+0000 7fe9c8046640 5 monclient: authenticate success, global_id 34182 2024-04-22T08:15:01.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.661+0000 7fe9c8046640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:15:01.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.661+0000 7fe9c4de0640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:15:01.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.661+0000 7fe9c8046640 10 monclient: get_monmap_and_config success 2024-04-22T08:15:01.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.661+0000 7fe9c8046640 4 set_mon_vals no callback set 2024-04-22T08:15:01.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.661+0000 7fe9c4de0640 1 -- 172.21.15.40:0/4139073738 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b8034610 con 0x7fe9c013b410 2024-04-22T08:15:01.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.661+0000 7fe9c4de0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:01.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.661+0000 7fe9c4de0640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:15:01.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.661+0000 7fe9c4de0640 10 monclient: dump: 2024-04-22T08:15:01.665 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:01.666 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:01.666 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:01.666 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:01.666 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:01.666 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:01.666 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:01.666 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:01.666 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:01.666 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:01.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.661+0000 7fe9c8046640 10 monclient: shutdown 2024-04-22T08:15:01.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.661+0000 7fe9c8046640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:15:01.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.661+0000 7fe9c8046640 1 -- 172.21.15.40:0/4139073738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9c013b410 msgr2=0x7fe9c013d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:01.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.661+0000 7fe9c8046640 1 --2- 172.21.15.40:0/4139073738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9c013b410 0x7fe9c013d800 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe9b8009e50 tx=0x7fe9b802bde0 comp rx=0 tx=0).stop 2024-04-22T08:15:01.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.661+0000 7fe9c8046640 1 -- 172.21.15.40:0/4139073738 shutdown_connections 2024-04-22T08:15:01.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.661+0000 7fe9c8046640 1 --2- 172.21.15.40:0/4139073738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9c013b410 0x7fe9c013d800 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:01.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.661+0000 7fe9c8046640 1 --2- 172.21.15.40:0/4139073738 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe9c00a31d0 0x7fe9c013aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:01.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.661+0000 7fe9c8046640 1 -- 172.21.15.40:0/4139073738 >> 172.21.15.40:0/4139073738 conn(0x7fe9c00a89a0 msgr2=0x7fe9c00a4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:15:01.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.661+0000 7fe9c8046640 1 -- 172.21.15.40:0/4139073738 shutdown_connections 2024-04-22T08:15:01.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.661+0000 7fe9c8046640 1 -- 172.21.15.40:0/4139073738 wait complete. 2024-04-22T08:15:01.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.662+0000 7fe9c8046640 10 monclient: build_initial_monmap 2024-04-22T08:15:01.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.662+0000 7fe9c8046640 10 monclient: monmap: 2024-04-22T08:15:01.674 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:15:01.674 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:15:01.674 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:15:01.674 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:15:01.674 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:15:01.674 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:01.674 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:15:01.674 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:15:01.674 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:15:01.674 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:01.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.662+0000 7fe9c8046640 1 Processor -- start 2024-04-22T08:15:01.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.662+0000 7fe9c8046640 1 -- start start 2024-04-22T08:15:01.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.662+0000 7fe9c8046640 10 monclient: init 2024-04-22T08:15:01.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.663+0000 7fe9c8046640 10 monclient: _reopen_session rank -1 2024-04-22T08:15:01.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.663+0000 7fe9c8046640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-22T08:15:01.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.663+0000 7fe9c8046640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe9c00a1c60 0x7fe9c01ca680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:01.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.663+0000 7fe9c8046640 10 monclient(hunting): picked mon.noname-c con 0x7fe9c00a1c60 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:15:01.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.663+0000 7fe9c8046640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9c00a31d0 0x7fe9c0139c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:01.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.663+0000 7fe9c8046640 10 monclient(hunting): picked mon.noname-b con 0x7fe9c00a31d0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:15:01.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.663+0000 7fe9c8046640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9c013b410 0x7fe9c013a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:01.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.663+0000 7fe9c8046640 10 monclient(hunting): picked mon.noname-a con 0x7fe9c013b410 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:15:01.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.663+0000 7fe9c8046640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:01.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.663+0000 7fe9c8046640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c0126150 con 0x7fe9c013b410 2024-04-22T08:15:01.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.663+0000 7fe9c8046640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:01.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.663+0000 7fe9c8046640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c01cabc0 con 0x7fe9c00a31d0 2024-04-22T08:15:01.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.663+0000 7fe9c8046640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:01.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.663+0000 7fe9c8046640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c013ae20 con 0x7fe9c00a1c60 2024-04-22T08:15:01.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.663+0000 7fe9c8046640 10 monclient(hunting): _renew_subs 2024-04-22T08:15:01.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.663+0000 7fe9c8046640 10 monclient(hunting): authenticate will time out at 975.657893s 2024-04-22T08:15:01.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.663+0000 7fe9c5de2640 1 -- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe9c00a1c60 msgr2=0x7fe9c01ca680 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:15:01.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.663+0000 7fe9c5de2640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe9c00a1c60 0x7fe9c01ca680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:15:01.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.663+0000 7fe9c65e3640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9c013b410 0x7fe9c013a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:01.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.663+0000 7fe9c65e3640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9c013b410 0x7fe9c013a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52334/0 (socket says 172.21.15.40:52334) 2024-04-22T08:15:01.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.663+0000 7fe9c65e3640 1 -- 172.21.15.40:0/2102560643 learned_addr learned my addr 172.21.15.40:0/2102560643 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:15:01.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.663+0000 7fe9c65e3640 10 monclient(hunting): get_auth_request con 0x7fe9c013b410 auth_method 0 2024-04-22T08:15:01.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.663+0000 7fe9c65e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:01.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.663+0000 7fe9c65e3640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:01.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.663+0000 7fe9c65e3640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:01.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.663+0000 7fe9c65e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:01.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.663+0000 7fe9c65e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:01.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.663+0000 7fe9c65e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:01.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.663+0000 7fe9c65e3640 10 monclient(hunting): handle_auth_done global_id 34185 payload 824 2024-04-22T08:15:01.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.663+0000 7fe9c55e1640 1 --2- 172.21.15.40:0/2102560643 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9c00a31d0 0x7fe9c0139c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:01.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.663+0000 7fe9c65e3640 1 -- 172.21.15.40:0/2102560643 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe9c00a1c60 msgr2=0x7fe9c01ca680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:15:01.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.663+0000 7fe9c65e3640 1 --2- 172.21.15.40:0/2102560643 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe9c00a1c60 0x7fe9c01ca680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:01.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.663+0000 7fe9c65e3640 1 -- 172.21.15.40:0/2102560643 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9c00a31d0 msgr2=0x7fe9c0139c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:01.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.663+0000 7fe9c65e3640 1 --2- 172.21.15.40:0/2102560643 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9c00a31d0 0x7fe9c0139c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:01.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.663+0000 7fe9c65e3640 10 monclient: _finish_hunting 0 2024-04-22T08:15:01.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.663+0000 7fe9c65e3640 1 monclient: found mon.noname-a 2024-04-22T08:15:01.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.663+0000 7fe9c65e3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:15:01.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.663+0000 7fe9c65e3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:15:01.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.663+0000 7fe9c65e3640 1 -- 172.21.15.40:0/2102560643 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b8009b30 con 0x7fe9c013b410 2024-04-22T08:15:01.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.663+0000 7fe9c55e1640 10 monclient: get_auth_request con 0x7fe9c00a31d0 auth_method 0 2024-04-22T08:15:01.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.663+0000 7fe9c55e1640 1 --2- 172.21.15.40:0/2102560643 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9c00a31d0 0x7fe9c0139c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T08:15:01.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.665+0000 7fe9c65e3640 1 -- 172.21.15.40:0/2102560643 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9c013b410 msgr2=0x7fe9c013a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:01.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.665+0000 7fe9c65e3640 1 --2- 172.21.15.40:0/2102560643 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9c013b410 0x7fe9c013a1a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe9b8036040 tx=0x7fe9b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:01.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.665+0000 7fe9b67fc640 1 -- 172.21.15.40:0/2102560643 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b803e660 con 0x7fe9c013b410 2024-04-22T08:15:01.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.665+0000 7fe9b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:01.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.665+0000 7fe9b67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:15:01.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.665+0000 7fe9b67fc640 10 monclient: dump: 2024-04-22T08:15:01.678 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:01.678 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:01.678 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:01.678 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:01.678 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:01.678 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:01.678 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:01.678 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:01.678 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:01.678 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:01.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.665+0000 7fe9b67fc640 10 monclient: _finish_auth 0 2024-04-22T08:15:01.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.665+0000 7fe9b67fc640 10 monclient: _check_auth_tickets 2024-04-22T08:15:01.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.665+0000 7fe9b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:15:01.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.665+0000 7fe9c8046640 5 monclient: authenticate success, global_id 34185 2024-04-22T08:15:01.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.665+0000 7fe9b67fc640 1 -- 172.21.15.40:0/2102560643 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9b803e7e0 con 0x7fe9c013b410 2024-04-22T08:15:01.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.665+0000 7fe9c8046640 10 monclient: _renew_subs 2024-04-22T08:15:01.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.665+0000 7fe9c8046640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:01.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.665+0000 7fe9c8046640 1 -- 172.21.15.40:0/2102560643 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9c0126610 con 0x7fe9c013b410 2024-04-22T08:15:01.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.666+0000 7fe9b67fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:15:01.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.666+0000 7fe9b67fc640 1 -- 172.21.15.40:0/2102560643 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b803eb90 con 0x7fe9c013b410 2024-04-22T08:15:01.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.666+0000 7fe9b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:01.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.666+0000 7fe9b67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:15:01.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.666+0000 7fe9b67fc640 10 monclient: dump: 2024-04-22T08:15:01.679 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:01.679 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:01.679 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:01.679 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:01.680 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:01.680 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:01.680 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:01.680 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:01.680 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:01.680 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:01.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.666+0000 7fe9b7fff640 4 set_mon_vals no callback set 2024-04-22T08:15:01.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.666+0000 7fe9c8046640 10 monclient: _renew_subs 2024-04-22T08:15:01.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.666+0000 7fe9c8046640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:01.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.666+0000 7fe9c8046640 1 -- 172.21.15.40:0/2102560643 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9c01cb520 con 0x7fe9c013b410 2024-04-22T08:15:01.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.666+0000 7fe9c8046640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:01.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.666+0000 7fe9b67fc640 1 -- 172.21.15.40:0/2102560643 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fe9b8048410 con 0x7fe9c013b410 2024-04-22T08:15:01.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.666+0000 7fe9c8046640 1 --2- 172.21.15.40:0/2102560643 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9c01226f0 0x7fe9c0122ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:01.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.666+0000 7fe9c8046640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:01.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.666+0000 7fe9c8046640 1 -- 172.21.15.40:0/2102560643 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c0066650 con 0x7fe9c01226f0 2024-04-22T08:15:01.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.666+0000 7fe9c5de2640 1 --2- 172.21.15.40:0/2102560643 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9c01226f0 0x7fe9c0122ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:01.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.666+0000 7fe9c5de2640 10 monclient: get_auth_request con 0x7fe9c01226f0 auth_method 0 2024-04-22T08:15:01.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.666+0000 7fe9c5de2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:01.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.666+0000 7fe9c5de2640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:01.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.666+0000 7fe9c5de2640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:01.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.666+0000 7fe9b67fc640 1 --2- 172.21.15.40:0/2102560643 >> v2:172.21.15.189:6800/30168 conn(0x7fe9a4037780 0x7fe9a4039c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:01.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.667+0000 7fe9c5de2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:01.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.667+0000 7fe9c5de2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:01.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.667+0000 7fe9b67fc640 1 -- 172.21.15.40:0/2102560643 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 3898+0+0 (secure 0 0 0) 0x7fe9b806d4f0 con 0x7fe9c013b410 2024-04-22T08:15:01.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.667+0000 7fe9c5de2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:01.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.667+0000 7fe9c5de2640 10 monclient(hunting): handle_auth_done global_id 34188 payload 824 2024-04-22T08:15:01.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.667+0000 7fe9c5de2640 1 -- 172.21.15.40:0/2102560643 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe9c00a0490 con 0x7fe9c01226f0 2024-04-22T08:15:01.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.667+0000 7fe9c55e1640 1 --2- 172.21.15.40:0/2102560643 >> v2:172.21.15.189:6800/30168 conn(0x7fe9a4037780 0x7fe9a4039c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:01.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.667+0000 7fe9c55e1640 10 monclient: get_auth_request con 0x7fe9a4037780 auth_method 0 2024-04-22T08:15:01.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.669+0000 7fe9c5de2640 1 -- 172.21.15.40:0/2102560643 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9c01226f0 msgr2=0x7fe9c0122ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:01.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.669+0000 7fe9c5de2640 1 --2- 172.21.15.40:0/2102560643 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9c01226f0 0x7fe9c0122ad0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc00c310 tx=0x7fe9bc00c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:01.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.669+0000 7fe9b67fc640 1 -- 172.21.15.40:0/2102560643 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc004970 con 0x7fe9c01226f0 2024-04-22T08:15:01.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.669+0000 7fe9b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:01.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.669+0000 7fe9b67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:15:01.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.669+0000 7fe9b67fc640 10 monclient: dump: 2024-04-22T08:15:01.682 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:01.682 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:01.682 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:01.682 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:01.682 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:01.682 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:01.682 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:01.682 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:01.682 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:01.683 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:01.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.669+0000 7fe9b67fc640 1 -- 172.21.15.40:0/2102560643 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7fe9bc0075d0 con 0x7fe9c01226f0 2024-04-22T08:15:01.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.669+0000 7fe9b67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:01.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.669+0000 7fe9b67fc640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:15:01.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.669+0000 7fe9b67fc640 1 -- 172.21.15.40:0/2102560643 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9c01226f0 msgr2=0x7fe9c0122ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:01.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.669+0000 7fe9b67fc640 1 --2- 172.21.15.40:0/2102560643 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9c01226f0 0x7fe9c0122ad0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc00c310 tx=0x7fe9bc00c7d0 comp rx=0 tx=0).stop 2024-04-22T08:15:01.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.669+0000 7fe9b67fc640 1 -- 172.21.15.40:0/2102560643 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9c01226f0 msgr2=0x7fe9c0122ad0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:15:01.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.669+0000 7fe9b67fc640 1 --2- 172.21.15.40:0/2102560643 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9c01226f0 0x7fe9c0122ad0 secure :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc00c310 tx=0x7fe9bc00c7d0 comp rx=0 tx=0).stop 2024-04-22T08:15:01.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.669+0000 7fe9c55e1640 1 --2- 172.21.15.40:0/2102560643 >> v2:172.21.15.189:6800/30168 conn(0x7fe9a4037780 0x7fe9a4039c40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0006fd0 tx=0x7fe9b0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:01.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.676+0000 7fe9c8046640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:15:01.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.676+0000 7fe9c8046640 1 --2- 172.21.15.40:0/2102560643 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9c0123010 0x7fe9c011e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:01.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.676+0000 7fe9c8046640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:01.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.676+0000 7fe9c8046640 1 -- 172.21.15.40:0/2102560643 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c0126470 con 0x7fe9c0123010 2024-04-22T08:15:01.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.676+0000 7fe9c5de2640 1 --2- 172.21.15.40:0/2102560643 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9c0123010 0x7fe9c011e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:01.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.676+0000 7fe9c5de2640 10 monclient: get_auth_request con 0x7fe9c0123010 auth_method 0 2024-04-22T08:15:01.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.676+0000 7fe9c5de2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:01.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.676+0000 7fe9c5de2640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:01.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.676+0000 7fe9c5de2640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:01.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.676+0000 7fe9c5de2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:01.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.676+0000 7fe9c5de2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:01.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.676+0000 7fe9c5de2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:01.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.676+0000 7fe9c5de2640 10 monclient(hunting): handle_auth_done global_id 34191 payload 824 2024-04-22T08:15:01.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.676+0000 7fe9c5de2640 1 -- 172.21.15.40:0/2102560643 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe9c004e9b0 con 0x7fe9c0123010 2024-04-22T08:15:01.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.678+0000 7fe9c5de2640 1 -- 172.21.15.40:0/2102560643 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9c0123010 msgr2=0x7fe9c011e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:01.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.678+0000 7fe9c5de2640 1 --2- 172.21.15.40:0/2102560643 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9c0123010 0x7fe9c011e960 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc00c810 tx=0x7fe9bc0010a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:01.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.678+0000 7fe9b67fc640 1 -- 172.21.15.40:0/2102560643 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc01c670 con 0x7fe9c0123010 2024-04-22T08:15:01.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.678+0000 7fe9b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:01.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.678+0000 7fe9b67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:15:01.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.678+0000 7fe9b67fc640 10 monclient: dump: 2024-04-22T08:15:01.686 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:01.687 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:01.687 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:01.687 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:01.687 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:01.687 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:01.687 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:01.687 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:01.687 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:01.687 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:01.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.678+0000 7fe9b67fc640 1 -- 172.21.15.40:0/2102560643 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4622 (secure 0 0 0) 0x7fe9bc023070 con 0x7fe9c0123010 2024-04-22T08:15:01.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.679+0000 7fe9b67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:15:01.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.679+0000 7fe9b67fc640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:15:01.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.679+0000 7fe9b67fc640 1 -- 172.21.15.40:0/2102560643 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9c0123010 msgr2=0x7fe9c011e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:01.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.679+0000 7fe9b67fc640 1 --2- 172.21.15.40:0/2102560643 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9c0123010 0x7fe9c011e960 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc00c810 tx=0x7fe9bc0010a0 comp rx=0 tx=0).stop 2024-04-22T08:15:01.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.679+0000 7fe9b67fc640 1 -- 172.21.15.40:0/2102560643 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9c0123010 msgr2=0x7fe9c011e960 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:15:01.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.679+0000 7fe9b67fc640 1 --2- 172.21.15.40:0/2102560643 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9c0123010 0x7fe9c011e960 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc00c810 tx=0x7fe9bc0010a0 comp rx=0 tx=0).stop 2024-04-22T08:15:01.688 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:15:01.688 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:15:01.688 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:15:01.688 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "leader", 2024-04-22T08:15:01.688 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 30, 2024-04-22T08:15:01.688 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:15:01.688 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:15:01.688 INFO:teuthology.orchestra.run.smithi040.stdout: 1 2024-04-22T08:15:01.688 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:01.689 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 0, 2024-04-22T08:15:01.689 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:15:01.689 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:15:01.689 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:15:01.689 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:15:01.689 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:15:01.689 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:15:01.689 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:15:01.689 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:15:01.689 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:15:01.689 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:15:01.689 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:15:01.689 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:15:01.689 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:15:01.689 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:01.690 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:15:01.690 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:15:01.690 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:15:01.690 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:15:01.690 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:15:01.690 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:15:01.690 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:15:01.690 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:15:01.690 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:15:01.690 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:15:01.690 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:15:01.690 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:15:01.690 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:01.691 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:01.691 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:15:01.691 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:15:01.691 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:15:01.691 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:15:01.691 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:15:01.691 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:15:01.691 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:15:01.691 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:15:01.691 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:15:01.691 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:15:01.691 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:15:01.691 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:15:01.691 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:15:01.692 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:15:01.692 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:15:01.692 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:15:01.692 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:15:01.692 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:15:01.692 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:15:01.692 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:15:01.692 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:15:01.692 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:15:01.692 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:15:01.692 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:15:01.692 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:15:01.692 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:15:01.692 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:15:01.692 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:01.693 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:15:01.693 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:01.693 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:15:01.693 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:01.693 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:15:01.693 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:15:01.693 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:15:01.693 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:15:01.693 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:01.693 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:15:01.693 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:15:01.693 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:01.693 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:01.693 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:01.694 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:15:01.694 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:15:01.694 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:01.694 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:01.694 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:01.694 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:01.694 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:15:01.694 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:15:01.694 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:15:01.694 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:15:01.694 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:15:01.694 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:01.694 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:01.694 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:15:01.695 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:15:01.695 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:15:01.695 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:15:01.695 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:01.695 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:15:01.695 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:15:01.695 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:01.695 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:01.695 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:01.695 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:15:01.695 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:15:01.695 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:01.695 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:01.695 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:01.695 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:01.696 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:15:01.696 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:15:01.696 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:15:01.696 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:15:01.696 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:15:01.696 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:01.696 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:01.696 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:15:01.696 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:15:01.696 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:15:01.696 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:15:01.696 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:01.696 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:15:01.696 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:15:01.697 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:01.697 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:01.697 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:01.697 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:15:01.697 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:15:01.697 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:01.697 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:01.697 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:01.697 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:01.697 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:15:01.697 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:15:01.697 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:15:01.697 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:15:01.697 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:15:01.697 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:01.698 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:01.698 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:01.698 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:15:01.698 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:15:01.698 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:01.698 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:15:01.698 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:15:01.698 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:15:01.698 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:01.698 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:01.698 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:15:01.698 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:01.698 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:15:01.698 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:15:01.698 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 4 2024-04-22T08:15:01.699 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:01.699 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:01.699 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:15:01.699 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:01.699 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:15:01.699 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:15:01.699 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 2 2024-04-22T08:15:01.699 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:01.699 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:01.699 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:01.699 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:15:01.699 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:15:01.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.679+0000 7fe9c8046640 1 -- 172.21.15.40:0/2102560643 >> v2:172.21.15.189:6800/30168 conn(0x7fe9a4037780 msgr2=0x7fe9a4039c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:01.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.679+0000 7fe9c8046640 1 --2- 172.21.15.40:0/2102560643 >> v2:172.21.15.189:6800/30168 conn(0x7fe9a4037780 0x7fe9a4039c40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0006fd0 tx=0x7fe9b0008040 comp rx=0 tx=0).stop 2024-04-22T08:15:01.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.679+0000 7fe9c8046640 10 monclient: shutdown 2024-04-22T08:15:01.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.679+0000 7fe9c8046640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:15:01.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.679+0000 7fe9c8046640 1 -- 172.21.15.40:0/2102560643 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9c013b410 msgr2=0x7fe9c013a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:01.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.679+0000 7fe9c8046640 1 --2- 172.21.15.40:0/2102560643 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9c013b410 0x7fe9c013a1a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe9b8036040 tx=0x7fe9b8004a40 comp rx=0 tx=0).stop 2024-04-22T08:15:01.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.679+0000 7fe9c65e3640 1 -- 172.21.15.40:0/2102560643 reap_dead start 2024-04-22T08:15:01.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.679+0000 7fe9c65e3640 1 -- 172.21.15.40:0/2102560643 reap_dead start 2024-04-22T08:15:01.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.679+0000 7fe9c8046640 1 -- 172.21.15.40:0/2102560643 shutdown_connections 2024-04-22T08:15:01.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.679+0000 7fe9c8046640 1 -- 172.21.15.40:0/2102560643 >> 172.21.15.40:0/2102560643 conn(0x7fe9c00a89a0 msgr2=0x7fe9c00a6110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:15:01.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.679+0000 7fe9c8046640 1 -- 172.21.15.40:0/2102560643 shutdown_connections 2024-04-22T08:15:01.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.679+0000 7fe9c8046640 1 -- 172.21.15.40:0/2102560643 wait complete. 2024-04-22T08:15:01.702 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-22T08:15:01.702 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-04-22T08:15:01.702 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-22T08:15:01.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.807+0000 7feb5f45f640 10 monclient: get_monmap_and_config 2024-04-22T08:15:01.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.807+0000 7feb5f45f640 10 monclient: build_initial_monmap 2024-04-22T08:15:01.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.807+0000 7feb5f45f640 10 monclient: monmap: 2024-04-22T08:15:01.809 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:15:01.809 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:01.809 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:15:01.809574+0000 2024-04-22T08:15:01.809 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:15:01.809574+0000 2024-04-22T08:15:01.809 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:15:01.809 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:01.809 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:15:01.809 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:15:01.809 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:15:01.809 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:15:01.809 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:15:01.809 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:01.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.808+0000 7feb5f45f640 1 Processor -- start 2024-04-22T08:15:01.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.808+0000 7feb5f45f640 1 -- start start 2024-04-22T08:15:01.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.808+0000 7feb5f45f640 10 monclient: init 2024-04-22T08:15:01.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.808+0000 7feb5f45f640 10 monclient: _reopen_session rank -1 2024-04-22T08:15:01.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.808+0000 7feb5f45f640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-04-22T08:15:01.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.808+0000 7feb5f45f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feb5812fad0 addr v1:172.21.15.40:6789/0 2024-04-22T08:15:01.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.809+0000 7feb5f45f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7feb58135c30 addr v1:172.21.15.189:6789/0 2024-04-22T08:15:01.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.809+0000 7feb5f45f640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7feb58139740 0x7feb5813bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:01.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.809+0000 7feb5f45f640 10 monclient(hunting): picked mon.noname-a con 0x7feb58139740 addr v2:172.21.15.40:3300/0 2024-04-22T08:15:01.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.809+0000 7feb5f45f640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb581305c0 con 0x7feb5812fad0 2024-04-22T08:15:01.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.809+0000 7feb5f45f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb5809ae50 con 0x7feb58135c30 2024-04-22T08:15:01.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.809+0000 7feb5f45f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:01.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.809+0000 7feb5f45f640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb5809efc0 con 0x7feb58139740 2024-04-22T08:15:01.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.809+0000 7feb5f45f640 10 monclient(hunting): _renew_subs 2024-04-22T08:15:01.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.809+0000 7feb5f45f640 10 monclient(hunting): authenticate will time out at 975.803837s 2024-04-22T08:15:01.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.809+0000 7feb5d9fc640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7feb58139740 0x7feb5813bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:01.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.809+0000 7feb5d1fb640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7feb5812fad0 0x7feb5812feb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:50588/0 (socket says 172.21.15.40:50588) 2024-04-22T08:15:01.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.809+0000 7feb5d1fb640 1 -- 172.21.15.40:0/1251894937 learned_addr learned my addr 172.21.15.40:0/1251894937 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:15:01.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.809+0000 7feb5d9fc640 10 monclient(hunting): get_auth_request con 0x7feb58139740 auth_method 0 2024-04-22T08:15:01.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.809+0000 7feb5d9fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:01.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.809+0000 7feb5d9fc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:01.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.809+0000 7feb5d9fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:01.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.809+0000 7feb5d9fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:01.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.809+0000 7feb5d9fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:01.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.809+0000 7feb5d9fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:01.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.809+0000 7feb5d9fc640 10 monclient(hunting): handle_auth_done global_id 34194 payload 274 2024-04-22T08:15:01.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.809+0000 7feb5d9fc640 1 -- 172.21.15.40:0/1251894937 >> v1:172.21.15.189:6789/0 conn(0x7feb58135c30 legacy=0x7feb58138020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:15:01.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.809+0000 7feb5d9fc640 1 -- 172.21.15.40:0/1251894937 >> v1:172.21.15.40:6789/0 conn(0x7feb5812fad0 legacy=0x7feb5812feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:15:01.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.809+0000 7feb5d9fc640 10 monclient: _finish_hunting 0 2024-04-22T08:15:01.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.809+0000 7feb5d9fc640 1 monclient: found mon.noname-a 2024-04-22T08:15:01.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.809+0000 7feb5d9fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:15:01.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.809+0000 7feb5d9fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:15:01.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.809+0000 7feb5d9fc640 1 -- 172.21.15.40:0/1251894937 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb5809d070 con 0x7feb58139740 2024-04-22T08:15:01.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.812+0000 7feb5d9fc640 1 -- 172.21.15.40:0/1251894937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb58139740 msgr2=0x7feb5813bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:01.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.812+0000 7feb5d9fc640 1 --2- 172.21.15.40:0/1251894937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb58139740 0x7feb5813bb30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7feb54009d90 tx=0x7feb5402bd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=769ab3ee72ccd058 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:01.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.812+0000 7feb4ffff640 1 -- 172.21.15.40:0/1251894937 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb5402cda0 con 0x7feb58139740 2024-04-22T08:15:01.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.812+0000 7feb4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:01.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.812+0000 7feb4ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:15:01.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.812+0000 7feb4ffff640 10 monclient: dump: 2024-04-22T08:15:01.817 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:01.817 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:01.817 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:01.817 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:01.817 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:01.817 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:01.817 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:01.818 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:01.818 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:01.818 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:01.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.812+0000 7feb4ffff640 10 monclient: _finish_auth 0 2024-04-22T08:15:01.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.812+0000 7feb4ffff640 10 monclient: _check_auth_tickets 2024-04-22T08:15:01.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.812+0000 7feb4ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:15:01.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.812+0000 7feb4ffff640 1 -- 172.21.15.40:0/1251894937 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb54034440 con 0x7feb58139740 2024-04-22T08:15:01.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.812+0000 7feb4ffff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:15:01.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.812+0000 7feb4ffff640 1 -- 172.21.15.40:0/1251894937 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb540347d0 con 0x7feb58139740 2024-04-22T08:15:01.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.812+0000 7feb4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:01.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.812+0000 7feb4ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:15:01.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.812+0000 7feb4ffff640 10 monclient: dump: 2024-04-22T08:15:01.818 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:01.818 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:01.819 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:01.819 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:01.819 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:01.819 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:01.819 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:01.819 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:01.819 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:01.819 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:01.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.812+0000 7feb5f45f640 5 monclient: authenticate success, global_id 34194 2024-04-22T08:15:01.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.812+0000 7feb5f45f640 10 monclient: get_monmap_and_config success 2024-04-22T08:15:01.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.812+0000 7feb5f45f640 4 set_mon_vals no callback set 2024-04-22T08:15:01.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.813+0000 7feb5f45f640 10 monclient: shutdown 2024-04-22T08:15:01.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.813+0000 7feb5f45f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:15:01.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.813+0000 7feb5f45f640 1 -- 172.21.15.40:0/1251894937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb58139740 msgr2=0x7feb5813bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:01.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.813+0000 7feb5f45f640 1 --2- 172.21.15.40:0/1251894937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb58139740 0x7feb5813bb30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7feb54009d90 tx=0x7feb5402bd00 comp rx=0 tx=0).stop 2024-04-22T08:15:01.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.813+0000 7feb5f45f640 1 -- 172.21.15.40:0/1251894937 shutdown_connections 2024-04-22T08:15:01.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.813+0000 7feb5f45f640 1 --2- 172.21.15.40:0/1251894937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb58139740 0x7feb5813bb30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:01.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.813+0000 7feb5f45f640 1 -- 172.21.15.40:0/1251894937 >> 172.21.15.40:0/1251894937 conn(0x7feb580a0ae0 msgr2=0x7feb580a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:15:01.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.813+0000 7feb5f45f640 1 -- 172.21.15.40:0/1251894937 shutdown_connections 2024-04-22T08:15:01.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.813+0000 7feb5f45f640 1 -- 172.21.15.40:0/1251894937 wait complete. 2024-04-22T08:15:01.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.813+0000 7feb5f45f640 10 monclient: build_initial_monmap 2024-04-22T08:15:01.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.813+0000 7feb5f45f640 10 monclient: monmap: 2024-04-22T08:15:01.820 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:15:01.820 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:15:01.820 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:15:01.820 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:15:01.820 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:15:01.820 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:01.820 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:15:01.821 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:15:01.821 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:15:01.821 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:01.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.814+0000 7feb5f45f640 1 Processor -- start 2024-04-22T08:15:01.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.814+0000 7feb5f45f640 1 -- start start 2024-04-22T08:15:01.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.814+0000 7feb5f45f640 10 monclient: init 2024-04-22T08:15:01.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.814+0000 7feb5f45f640 10 monclient: _reopen_session rank -1 2024-04-22T08:15:01.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.814+0000 7feb5f45f640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-22T08:15:01.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.814+0000 7feb5f45f640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feb5812fad0 0x7feb5812ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:01.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.814+0000 7feb5f45f640 10 monclient(hunting): picked mon.noname-c con 0x7feb5812fad0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:15:01.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.814+0000 7feb5f45f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb58135c30 0x7feb5812f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:01.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.814+0000 7feb5f45f640 10 monclient(hunting): picked mon.noname-a con 0x7feb58135c30 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:15:01.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.814+0000 7feb5f45f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb58139740 0x7feb58133a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:01.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.814+0000 7feb5f45f640 10 monclient(hunting): picked mon.noname-b con 0x7feb58139740 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:15:01.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.814+0000 7feb5f45f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:01.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.814+0000 7feb5f45f640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb58071870 con 0x7feb58135c30 2024-04-22T08:15:01.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.814+0000 7feb5f45f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:01.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.814+0000 7feb5f45f640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb5806fae0 con 0x7feb58139740 2024-04-22T08:15:01.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.814+0000 7feb5f45f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:01.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.814+0000 7feb5f45f640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb5806fdc0 con 0x7feb5812fad0 2024-04-22T08:15:01.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.814+0000 7feb5f45f640 10 monclient(hunting): _renew_subs 2024-04-22T08:15:01.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.814+0000 7feb5f45f640 10 monclient(hunting): authenticate will time out at 975.809556s 2024-04-22T08:15:01.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.814+0000 7feb5d1fb640 1 -- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feb5812fad0 msgr2=0x7feb5812ed70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:15:01.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.814+0000 7feb5d1fb640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feb5812fad0 0x7feb5812ed70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:15:01.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.814+0000 7feb5c9fa640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb58135c30 0x7feb5812f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:01.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.814+0000 7feb5c9fa640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb58135c30 0x7feb5812f2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52364/0 (socket says 172.21.15.40:52364) 2024-04-22T08:15:01.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.814+0000 7feb5c9fa640 1 -- 172.21.15.40:0/3109962663 learned_addr learned my addr 172.21.15.40:0/3109962663 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:15:01.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.814+0000 7feb5c9fa640 10 monclient(hunting): get_auth_request con 0x7feb58135c30 auth_method 0 2024-04-22T08:15:01.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.814+0000 7feb5c9fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:01.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.814+0000 7feb5c9fa640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:01.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.814+0000 7feb5c9fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:01.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.815+0000 7feb5c9fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:01.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.815+0000 7feb5c9fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:01.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.815+0000 7feb5c9fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:01.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.815+0000 7feb5d9fc640 1 --2- 172.21.15.40:0/3109962663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb58139740 0x7feb58133a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:01.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.815+0000 7feb5c9fa640 10 monclient(hunting): handle_auth_done global_id 34197 payload 824 2024-04-22T08:15:01.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.815+0000 7feb5c9fa640 1 -- 172.21.15.40:0/3109962663 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feb5812fad0 msgr2=0x7feb5812ed70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:15:01.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.815+0000 7feb5c9fa640 1 --2- 172.21.15.40:0/3109962663 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feb5812fad0 0x7feb5812ed70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:01.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.815+0000 7feb5c9fa640 1 -- 172.21.15.40:0/3109962663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb58139740 msgr2=0x7feb58133a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:01.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.815+0000 7feb5c9fa640 1 --2- 172.21.15.40:0/3109962663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb58139740 0x7feb58133a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:01.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.815+0000 7feb5c9fa640 10 monclient: _finish_hunting 0 2024-04-22T08:15:01.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.815+0000 7feb5c9fa640 1 monclient: found mon.noname-a 2024-04-22T08:15:01.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.815+0000 7feb5c9fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:15:01.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.815+0000 7feb5c9fa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:15:01.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.815+0000 7feb5c9fa640 1 -- 172.21.15.40:0/3109962663 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb54009a70 con 0x7feb58135c30 2024-04-22T08:15:01.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.815+0000 7feb5d9fc640 10 monclient: get_auth_request con 0x7feb58139740 auth_method 0 2024-04-22T08:15:01.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.815+0000 7feb5d9fc640 1 --2- 172.21.15.40:0/3109962663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb58139740 0x7feb58133a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T08:15:01.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.817+0000 7feb5c9fa640 1 -- 172.21.15.40:0/3109962663 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb58135c30 msgr2=0x7feb5812f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:01.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.817+0000 7feb5c9fa640 1 --2- 172.21.15.40:0/3109962663 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb58135c30 0x7feb5812f2b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7feb4800c430 tx=0x7feb4800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:01.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.817+0000 7feb5c9fa640 1 --2- 172.21.15.40:0/3109962663 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb58135c30 0x7feb5812f2b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7feb4800c430 tx=0x7feb4800c8f0 comp rx=0 tx=0).queue_received will delay after 1.71377e+09 on 0x7feb480119a0 mon_map magic: 0 v1 2024-04-22T08:15:01.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.817+0000 7feb4dffb640 1 -- 172.21.15.40:0/3109962663 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb480119a0 con 0x7feb58135c30 2024-04-22T08:15:01.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.817+0000 7feb4dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:01.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.817+0000 7feb4dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:15:01.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.817+0000 7feb4dffb640 10 monclient: dump: 2024-04-22T08:15:01.827 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:01.828 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:01.828 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:01.828 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:01.828 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:01.828 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:01.828 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:01.828 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:01.828 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:01.828 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:01.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.817+0000 7feb4dffb640 10 monclient: _finish_auth 0 2024-04-22T08:15:01.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.817+0000 7feb4dffb640 10 monclient: _check_auth_tickets 2024-04-22T08:15:01.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.817+0000 7feb4dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:15:01.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.817+0000 7feb4dffb640 1 -- 172.21.15.40:0/3109962663 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb48011b00 con 0x7feb58135c30 2024-04-22T08:15:01.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.817+0000 7feb5f45f640 5 monclient: authenticate success, global_id 34197 2024-04-22T08:15:01.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.817+0000 7feb5f45f640 10 monclient: _renew_subs 2024-04-22T08:15:01.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.817+0000 7feb5f45f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:01.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.817+0000 7feb5f45f640 1 -- 172.21.15.40:0/3109962663 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb581355f0 con 0x7feb58135c30 2024-04-22T08:15:01.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.817+0000 7feb4dffb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:15:01.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.817+0000 7feb4f7fe640 4 set_mon_vals no callback set 2024-04-22T08:15:01.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.817+0000 7feb5f45f640 10 monclient: _renew_subs 2024-04-22T08:15:01.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.817+0000 7feb5f45f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:01.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.817+0000 7feb5f45f640 1 -- 172.21.15.40:0/3109962663 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb5806f520 con 0x7feb58135c30 2024-04-22T08:15:01.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.818+0000 7feb4dffb640 1 -- 172.21.15.40:0/3109962663 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb48012420 con 0x7feb58135c30 2024-04-22T08:15:01.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.818+0000 7feb4dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:01.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.818+0000 7feb4dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:15:01.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.818+0000 7feb4dffb640 10 monclient: dump: 2024-04-22T08:15:01.829 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:01.829 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:01.830 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:01.830 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:01.830 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:01.830 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:01.830 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:01.830 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:01.830 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:01.830 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:01.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.818+0000 7feb4dffb640 1 -- 172.21.15.40:0/3109962663 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7feb48012600 con 0x7feb58135c30 2024-04-22T08:15:01.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.818+0000 7feb4dffb640 1 --2- 172.21.15.40:0/3109962663 >> v2:172.21.15.189:6800/30168 conn(0x7feb3c037800 0x7feb3c039cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:01.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.818+0000 7feb5f45f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:01.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.818+0000 7feb5f45f640 1 --2- 172.21.15.40:0/3109962663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb58133f50 0x7feb58134330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:01.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.818+0000 7feb5f45f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:01.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.818+0000 7feb5f45f640 1 -- 172.21.15.40:0/3109962663 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb580667a0 con 0x7feb58133f50 2024-04-22T08:15:01.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.819+0000 7feb5d9fc640 1 --2- 172.21.15.40:0/3109962663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb58133f50 0x7feb58134330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:01.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.819+0000 7feb5d9fc640 10 monclient: get_auth_request con 0x7feb58133f50 auth_method 0 2024-04-22T08:15:01.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.819+0000 7feb5d9fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:01.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.819+0000 7feb5d9fc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:01.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.819+0000 7feb5d9fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:01.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.819+0000 7feb5d1fb640 1 --2- 172.21.15.40:0/3109962663 >> v2:172.21.15.189:6800/30168 conn(0x7feb3c037800 0x7feb3c039cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:01.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.819+0000 7feb5d1fb640 10 monclient: get_auth_request con 0x7feb3c037800 auth_method 0 2024-04-22T08:15:01.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.819+0000 7feb5d9fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:01.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.819+0000 7feb5d9fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:01.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.819+0000 7feb5d9fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:01.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.819+0000 7feb5d9fc640 10 monclient(hunting): handle_auth_done global_id 24388 payload 824 2024-04-22T08:15:01.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.819+0000 7feb5d9fc640 1 -- 172.21.15.40:0/3109962663 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feb5809c9b0 con 0x7feb58133f50 2024-04-22T08:15:01.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.821+0000 7feb5d1fb640 1 --2- 172.21.15.40:0/3109962663 >> v2:172.21.15.189:6800/30168 conn(0x7feb3c037800 0x7feb3c039cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7feb50007fd0 tx=0x7feb50007b20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:01.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.821+0000 7feb5d9fc640 1 -- 172.21.15.40:0/3109962663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb58133f50 msgr2=0x7feb58134330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:01.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.821+0000 7feb5d9fc640 1 --2- 172.21.15.40:0/3109962663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb58133f50 0x7feb58134330 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7feb5806bb30 tx=0x7feb54004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:01.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.821+0000 7feb4dffb640 1 -- 172.21.15.40:0/3109962663 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb5403f800 con 0x7feb58133f50 2024-04-22T08:15:01.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.821+0000 7feb4dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:01.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.821+0000 7feb4dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:15:01.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.821+0000 7feb4dffb640 10 monclient: dump: 2024-04-22T08:15:01.832 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:01.832 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:01.832 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:01.832 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:01.832 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:01.832 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:01.832 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:01.832 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:01.832 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:01.832 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:01.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.822+0000 7feb4dffb640 1 -- 172.21.15.40:0/3109962663 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7feb5403f960 con 0x7feb58133f50 2024-04-22T08:15:01.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.822+0000 7feb4dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:01.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.822+0000 7feb4dffb640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:15:01.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.822+0000 7feb4dffb640 1 -- 172.21.15.40:0/3109962663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb58133f50 msgr2=0x7feb58134330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:01.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.822+0000 7feb4dffb640 1 --2- 172.21.15.40:0/3109962663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb58133f50 0x7feb58134330 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7feb5806bb30 tx=0x7feb54004a40 comp rx=0 tx=0).stop 2024-04-22T08:15:01.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.822+0000 7feb4dffb640 1 -- 172.21.15.40:0/3109962663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb58133f50 msgr2=0x7feb58134330 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:15:01.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.822+0000 7feb4dffb640 1 --2- 172.21.15.40:0/3109962663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb58133f50 0x7feb58134330 secure :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7feb5806bb30 tx=0x7feb54004a40 comp rx=0 tx=0).stop 2024-04-22T08:15:01.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.828+0000 7feb5f45f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:15:01.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.828+0000 7feb5f45f640 1 --2- 172.21.15.40:0/3109962663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb581317a0 0x7feb58131bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:01.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.828+0000 7feb5f45f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:01.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.828+0000 7feb5f45f640 1 -- 172.21.15.40:0/3109962663 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb58134870 con 0x7feb581317a0 2024-04-22T08:15:01.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.828+0000 7feb5d9fc640 1 --2- 172.21.15.40:0/3109962663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb581317a0 0x7feb58131bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:01.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.828+0000 7feb5d9fc640 10 monclient: get_auth_request con 0x7feb581317a0 auth_method 0 2024-04-22T08:15:01.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.828+0000 7feb5d9fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:01.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.828+0000 7feb5d9fc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:01.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.828+0000 7feb5d9fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:01.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.828+0000 7feb5d9fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:01.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.828+0000 7feb5d9fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:01.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.828+0000 7feb5d9fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:01.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.828+0000 7feb5d9fc640 10 monclient(hunting): handle_auth_done global_id 24391 payload 824 2024-04-22T08:15:01.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.829+0000 7feb5d9fc640 1 -- 172.21.15.40:0/3109962663 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7feb5804e9b0 con 0x7feb581317a0 2024-04-22T08:15:01.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.831+0000 7feb5d9fc640 1 -- 172.21.15.40:0/3109962663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb581317a0 msgr2=0x7feb58131bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:01.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.831+0000 7feb5d9fc640 1 --2- 172.21.15.40:0/3109962663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb581317a0 0x7feb58131bb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7feb5806ff80 tx=0x7feb54031000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:01.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.831+0000 7feb4dffb640 1 -- 172.21.15.40:0/3109962663 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb54004240 con 0x7feb581317a0 2024-04-22T08:15:01.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.831+0000 7feb4dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:01.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.831+0000 7feb4dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:15:01.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.831+0000 7feb4dffb640 10 monclient: dump: 2024-04-22T08:15:01.836 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:01.836 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:01.836 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:01.836 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:01.836 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:01.836 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:01.836 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:01.836 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:01.836 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:01.837 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:01.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.832+0000 7feb4dffb640 1 -- 172.21.15.40:0/3109962663 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4790 (secure 0 0 0) 0x7feb540043a0 con 0x7feb581317a0 2024-04-22T08:15:01.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.832+0000 7feb4dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:15:01.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.832+0000 7feb4dffb640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:15:01.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.832+0000 7feb4dffb640 1 -- 172.21.15.40:0/3109962663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb581317a0 msgr2=0x7feb58131bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:01.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.832+0000 7feb4dffb640 1 --2- 172.21.15.40:0/3109962663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb581317a0 0x7feb58131bb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7feb5806ff80 tx=0x7feb54031000 comp rx=0 tx=0).stop 2024-04-22T08:15:01.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.832+0000 7feb4dffb640 1 -- 172.21.15.40:0/3109962663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb581317a0 msgr2=0x7feb58131bb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:15:01.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.832+0000 7feb4dffb640 1 --2- 172.21.15.40:0/3109962663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb581317a0 0x7feb58131bb0 secure :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7feb5806ff80 tx=0x7feb54031000 comp rx=0 tx=0).stop 2024-04-22T08:15:01.837 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:15:01.837 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:15:01.837 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:15:01.838 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:15:01.838 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 30, 2024-04-22T08:15:01.838 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:15:01.838 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:15:01.838 INFO:teuthology.orchestra.run.smithi040.stdout: 1 2024-04-22T08:15:01.838 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:01.838 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 0, 2024-04-22T08:15:01.838 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:15:01.838 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:15:01.838 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:15:01.838 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:15:01.838 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:15:01.838 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:15:01.839 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:15:01.839 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:15:01.839 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:15:01.839 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:15:01.839 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:15:01.839 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:15:01.839 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:15:01.839 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:01.839 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:15:01.839 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:15:01.839 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:15:01.839 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:15:01.839 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:15:01.839 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:15:01.840 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:15:01.840 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:15:01.840 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:15:01.840 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:15:01.840 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:15:01.840 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:15:01.840 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:01.840 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:01.840 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:15:01.840 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:15:01.840 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:15:01.840 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:15:01.840 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:15:01.840 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:15:01.840 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:15:01.841 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:15:01.841 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:15:01.841 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:15:01.841 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:15:01.841 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:15:01.841 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:15:01.841 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:15:01.841 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:15:01.841 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:15:01.841 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:15:01.841 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:15:01.841 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:15:01.841 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:15:01.841 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:15:01.842 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:15:01.842 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:15:01.842 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:15:01.842 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:15:01.842 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:15:01.842 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:15:01.842 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:01.842 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:15:01.842 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:01.842 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:15:01.842 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:01.842 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:15:01.842 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:15:01.842 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:15:01.843 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:15:01.843 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:01.843 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:15:01.843 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:15:01.843 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:01.843 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:01.843 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:01.843 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:15:01.843 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:15:01.843 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:01.843 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:01.843 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:01.843 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:01.843 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:15:01.843 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:15:01.844 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:15:01.844 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:15:01.844 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:15:01.844 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:01.844 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:01.844 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:15:01.844 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:15:01.844 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:15:01.844 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:15:01.844 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:01.844 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:15:01.844 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:15:01.844 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:01.844 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:01.844 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:01.845 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:15:01.845 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:15:01.845 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:01.845 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:01.845 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:01.845 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:01.845 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:15:01.845 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:15:01.845 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:15:01.845 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:15:01.845 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:15:01.845 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:01.845 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:01.845 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:15:01.845 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:15:01.845 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:15:01.846 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:15:01.846 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:01.846 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:15:01.846 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:15:01.846 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:01.846 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:01.846 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:01.846 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:15:01.846 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:15:01.846 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:01.846 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:01.846 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:01.846 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:01.846 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:15:01.846 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:15:01.847 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:15:01.847 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:15:01.847 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:15:01.847 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:01.847 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:01.847 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:01.847 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:15:01.847 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:15:01.847 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:01.847 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:15:01.847 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:15:01.847 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:15:01.847 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:01.847 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:01.847 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:15:01.848 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:01.848 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:15:01.848 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:15:01.848 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 2 2024-04-22T08:15:01.848 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:01.848 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:01.848 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:15:01.848 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:01.848 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:15:01.848 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:15:01.848 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 6 2024-04-22T08:15:01.848 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:01.848 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:01.848 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": [ 2024-04-22T08:15:01.849 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:01.849 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:15:01.849 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:15:01.849 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:15:01.849 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:01.849 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:01.849 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:01.849 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:15:01.849 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:15:01.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.832+0000 7feb5f45f640 1 -- 172.21.15.40:0/3109962663 >> v2:172.21.15.189:6800/30168 conn(0x7feb3c037800 msgr2=0x7feb3c039cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:01.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.832+0000 7feb5f45f640 1 --2- 172.21.15.40:0/3109962663 >> v2:172.21.15.189:6800/30168 conn(0x7feb3c037800 0x7feb3c039cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7feb50007fd0 tx=0x7feb50007b20 comp rx=0 tx=0).stop 2024-04-22T08:15:01.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.832+0000 7feb5f45f640 10 monclient: shutdown 2024-04-22T08:15:01.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.832+0000 7feb5f45f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:15:01.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.832+0000 7feb5f45f640 1 -- 172.21.15.40:0/3109962663 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb58135c30 msgr2=0x7feb5812f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:01.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.832+0000 7feb5f45f640 1 --2- 172.21.15.40:0/3109962663 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb58135c30 0x7feb5812f2b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7feb4800c430 tx=0x7feb4800c8f0 comp rx=0 tx=0).stop 2024-04-22T08:15:01.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.832+0000 7feb5d9fc640 1 -- 172.21.15.40:0/3109962663 reap_dead start 2024-04-22T08:15:01.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.832+0000 7feb5d9fc640 1 -- 172.21.15.40:0/3109962663 reap_dead start 2024-04-22T08:15:01.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.832+0000 7feb4dffb640 1 -- 172.21.15.40:0/3109962663 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 3898+0+0 (unknown 0 0 0) 0x7feb4804a490 con 0x7feb58135c30 2024-04-22T08:15:01.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.832+0000 7feb5f45f640 1 -- 172.21.15.40:0/3109962663 shutdown_connections 2024-04-22T08:15:01.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.832+0000 7feb5f45f640 1 -- 172.21.15.40:0/3109962663 >> 172.21.15.40:0/3109962663 conn(0x7feb580a0ae0 msgr2=0x7feb5809f270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:15:01.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.832+0000 7feb5f45f640 1 -- 172.21.15.40:0/3109962663 shutdown_connections 2024-04-22T08:15:01.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:01.832+0000 7feb5f45f640 1 -- 172.21.15.40:0/3109962663 wait complete. 2024-04-22T08:15:01.852 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-22T08:15:01.852 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-04-22T08:15:01.853 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:15:01.853 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[35419]: starting. 2024-04-22T08:15:01.853 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[35419]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-35381 2024-04-22T08:15:01.853 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[35419]: shutting down. 2024-04-22T08:15:01.853 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:15:01.853 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:15:01.853 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:15:01.853 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:15:04.180 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35487]: starting. 2024-04-22T08:15:04.180 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35487]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-35486 2024-04-22T08:15:04.181 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35487]: created object 0... 2024-04-22T08:15:04.181 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35487]: created object 25... 2024-04-22T08:15:04.181 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35487]: created object 49... 2024-04-22T08:15:04.181 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35487]: finishing. 2024-04-22T08:15:04.181 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35487]: shutting down. 2024-04-22T08:15:04.856 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[35488]: starting. 2024-04-22T08:15:04.856 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[35488]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-35486 2024-04-22T08:15:04.856 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[35488]: shutting down. 2024-04-22T08:15:06.983 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:15:06.984 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35521]: starting. 2024-04-22T08:15:06.984 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35521]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-35486 2024-04-22T08:15:06.984 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35521]: created object 0... 2024-04-22T08:15:06.984 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35521]: created object 25... 2024-04-22T08:15:06.985 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35521]: created object 49... 2024-04-22T08:15:06.985 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35521]: finishing. 2024-04-22T08:15:06.986 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35521]: shutting down. 2024-04-22T08:15:07.226 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:15:07.226 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35523]: starting. 2024-04-22T08:15:07.226 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35523]: listing objects. 2024-04-22T08:15:07.226 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35523]: listed object 0... 2024-04-22T08:15:07.226 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35523]: listed object 25... 2024-04-22T08:15:07.227 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35523]: saw 50 objects 2024-04-22T08:15:07.227 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35523]: shutting down. 2024-04-22T08:15:07.873 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:15:07.873 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[35522]: starting. 2024-04-22T08:15:07.873 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[35522]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-35486 2024-04-22T08:15:07.873 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[35522]: shutting down. 2024-04-22T08:15:07.874 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:15:07.875 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:15:07.875 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:15:07.877 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:15:10.026 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35590]: starting. 2024-04-22T08:15:10.026 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35590]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-35589 2024-04-22T08:15:10.026 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35590]: created object 0... 2024-04-22T08:15:10.026 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35590]: created object 25... 2024-04-22T08:15:10.026 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35590]: created object 49... 2024-04-22T08:15:10.026 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35590]: finishing. 2024-04-22T08:15:10.026 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35590]: shutting down. 2024-04-22T08:15:10.887 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[35591]: starting. 2024-04-22T08:15:10.887 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[35591]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-35589 2024-04-22T08:15:10.887 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[35591]: shutting down. 2024-04-22T08:15:12.962 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:15:12.962 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35624]: starting. 2024-04-22T08:15:12.962 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35624]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-35589 2024-04-22T08:15:12.962 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35624]: created object 0... 2024-04-22T08:15:12.963 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35624]: created object 25... 2024-04-22T08:15:12.963 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35624]: created object 49... 2024-04-22T08:15:12.963 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35624]: finishing. 2024-04-22T08:15:12.963 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35624]: shutting down. 2024-04-22T08:15:13.002 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:15:13.002 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35626]: starting. 2024-04-22T08:15:13.002 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35626]: listing objects. 2024-04-22T08:15:13.002 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35626]: listed object 0... 2024-04-22T08:15:13.002 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35626]: listed object 25... 2024-04-22T08:15:13.002 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35626]: saw 50 objects 2024-04-22T08:15:13.002 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35626]: shutting down. 2024-04-22T08:15:13.852 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:15:13.852 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[35625]: starting. 2024-04-22T08:15:13.852 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[35625]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-35589 2024-04-22T08:15:13.852 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[35625]: shutting down. 2024-04-22T08:15:13.854 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:15:13.854 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:15:13.854 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:15:13.856 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:15:15.968 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35693]: starting. 2024-04-22T08:15:15.968 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35693]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-35692 2024-04-22T08:15:15.968 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35693]: created object 0... 2024-04-22T08:15:15.968 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35693]: created object 25... 2024-04-22T08:15:15.968 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35693]: created object 49... 2024-04-22T08:15:15.968 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35693]: finishing. 2024-04-22T08:15:15.968 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35693]: shutting down. 2024-04-22T08:15:16.873 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[35694]: starting. 2024-04-22T08:15:16.873 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[35694]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-35692 2024-04-22T08:15:16.873 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[35694]: shutting down. 2024-04-22T08:15:18.994 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:15:18.994 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35727]: starting. 2024-04-22T08:15:18.994 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35727]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-35692 2024-04-22T08:15:18.994 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35727]: created object 0... 2024-04-22T08:15:18.994 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35727]: created object 25... 2024-04-22T08:15:18.994 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35727]: created object 49... 2024-04-22T08:15:18.995 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35727]: finishing. 2024-04-22T08:15:18.995 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35727]: shutting down. 2024-04-22T08:15:19.039 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:15:19.039 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35729]: starting. 2024-04-22T08:15:19.039 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35729]: listing objects. 2024-04-22T08:15:19.039 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35729]: listed object 0... 2024-04-22T08:15:19.039 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35729]: listed object 25... 2024-04-22T08:15:19.039 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35729]: saw 50 objects 2024-04-22T08:15:19.039 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35729]: shutting down. 2024-04-22T08:15:19.882 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:15:19.882 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[35728]: starting. 2024-04-22T08:15:19.882 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[35728]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-35692 2024-04-22T08:15:19.882 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[35728]: shutting down. 2024-04-22T08:15:19.883 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:15:19.883 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:15:19.883 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:15:19.886 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:15:21.853 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-04-22T08:15:21.853 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-04-22T08:15:21.853 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-22T08:15:21.853 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-22T08:15:21.856 INFO:tasks.ceph.mon.c:Started 2024-04-22T08:15:21.856 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-22T08:15:21.857 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:15:21.965 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35796]: starting. 2024-04-22T08:15:21.965 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35796]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-35795 2024-04-22T08:15:21.965 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35796]: created object 0... 2024-04-22T08:15:21.965 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35796]: created object 25... 2024-04-22T08:15:21.965 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35796]: created object 49... 2024-04-22T08:15:21.966 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35796]: finishing. 2024-04-22T08:15:21.966 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35796]: shutting down. 2024-04-22T08:15:21.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.973+0000 7f8e164ce640 10 monclient: get_monmap_and_config 2024-04-22T08:15:21.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.973+0000 7f8e164ce640 10 monclient: build_initial_monmap 2024-04-22T08:15:21.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.974+0000 7f8e164ce640 10 monclient: monmap: 2024-04-22T08:15:21.973 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:15:21.973 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:21.973 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:15:21.974778+0000 2024-04-22T08:15:21.973 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:15:21.974778+0000 2024-04-22T08:15:21.973 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:15:21.974 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:21.974 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:15:21.974 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:15:21.974 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:15:21.974 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:15:21.974 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:15:21.974 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:21.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.974+0000 7f8e164ce640 1 Processor -- start 2024-04-22T08:15:21.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.974+0000 7f8e164ce640 1 -- start start 2024-04-22T08:15:21.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.974+0000 7f8e164ce640 10 monclient: init 2024-04-22T08:15:21.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.975+0000 7f8e164ce640 10 monclient: _reopen_session rank -1 2024-04-22T08:15:21.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.975+0000 7f8e164ce640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-04-22T08:15:21.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.975+0000 7f8e164ce640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8e1009ee10 addr v1:172.21.15.189:6789/0 2024-04-22T08:15:21.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.975+0000 7f8e164ce640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8e100a32b0 0x7f8e1013b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:21.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.975+0000 7f8e164ce640 10 monclient(hunting): picked mon.noname-a con 0x7f8e100a32b0 addr v2:172.21.15.40:3300/0 2024-04-22T08:15:21.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.975+0000 7f8e164ce640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8e1013b5f0 0x7f8e1013d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:21.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.975+0000 7f8e164ce640 10 monclient(hunting): picked mon.noname-b con 0x7f8e1013b5f0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:15:21.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.975+0000 7f8e164ce640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e1009f1f0 con 0x7f8e1009ee10 2024-04-22T08:15:21.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.975+0000 7f8e164ce640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:21.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.975+0000 7f8e164ce640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e100a39c0 con 0x7f8e100a32b0 2024-04-22T08:15:21.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.975+0000 7f8e164ce640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:21.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.975+0000 7f8e164ce640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e100a3700 con 0x7f8e1013b5f0 2024-04-22T08:15:21.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.975+0000 7f8e164ce640 10 monclient(hunting): _renew_subs 2024-04-22T08:15:21.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.975+0000 7f8e164ce640 10 monclient(hunting): authenticate will time out at 995.969199s 2024-04-22T08:15:21.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.975+0000 7f8e0f7fe640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8e100a32b0 0x7f8e1013b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:21.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.975+0000 7f8e14a6b640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8e1013b5f0 0x7f8e1013d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:21.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.975+0000 7f8e0f7fe640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8e100a32b0 0x7f8e1013b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60224/0 (socket says 172.21.15.40:60224) 2024-04-22T08:15:21.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.975+0000 7f8e0f7fe640 1 -- 172.21.15.40:0/3426681890 learned_addr learned my addr 172.21.15.40:0/3426681890 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:15:21.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.975+0000 7f8e0f7fe640 10 monclient(hunting): get_auth_request con 0x7f8e100a32b0 auth_method 0 2024-04-22T08:15:21.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.975+0000 7f8e0f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:21.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.975+0000 7f8e0f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:21.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.975+0000 7f8e0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:21.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.975+0000 7f8e14a6b640 10 monclient(hunting): get_auth_request con 0x7f8e1013b5f0 auth_method 0 2024-04-22T08:15:21.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.975+0000 7f8e14a6b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:21.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.975+0000 7f8e14a6b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:21.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.975+0000 7f8e14a6b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:21.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.975+0000 7f8e0f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:21.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.975+0000 7f8e0f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:21.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.975+0000 7f8e0f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:21.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.975+0000 7f8e14a6b640 1 -- 172.21.15.40:0/3426681890 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8e1013b5f0 msgr2=0x7f8e1013d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-22T08:15:21.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.975+0000 7f8e14a6b640 1 -- 172.21.15.40:0/3426681890 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8e1013b5f0 msgr2=0x7f8e1013d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:15:21.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.975+0000 7f8e14a6b640 1 --2- 172.21.15.40:0/3426681890 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8e1013b5f0 0x7f8e1013d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:15:21.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.975+0000 7f8e14a6b640 1 --2- 172.21.15.40:0/3426681890 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8e1013b5f0 0x7f8e1013d9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:15:21.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.975+0000 7f8e0f7fe640 10 monclient(hunting): handle_auth_done global_id 34287 payload 274 2024-04-22T08:15:21.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.975+0000 7f8e0f7fe640 1 -- 172.21.15.40:0/3426681890 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8e1013b5f0 msgr2=0x7f8e1013d9e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:15:21.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.975+0000 7f8e0f7fe640 1 --2- 172.21.15.40:0/3426681890 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8e1013b5f0 0x7f8e1013d9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:21.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.975+0000 7f8e0f7fe640 1 -- 172.21.15.40:0/3426681890 >> v1:172.21.15.189:6789/0 conn(0x7f8e1009ee10 legacy=0x7f8e100a1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:15:21.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.975+0000 7f8e0f7fe640 10 monclient: _finish_hunting 0 2024-04-22T08:15:21.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.975+0000 7f8e0f7fe640 1 monclient: found mon.noname-a 2024-04-22T08:15:21.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.975+0000 7f8e0f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:15:21.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.975+0000 7f8e0f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:15:21.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.975+0000 7f8e0f7fe640 1 -- 172.21.15.40:0/3426681890 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e1009fd30 con 0x7f8e100a32b0 2024-04-22T08:15:21.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.978+0000 7f8e0f7fe640 1 -- 172.21.15.40:0/3426681890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e100a32b0 msgr2=0x7f8e1013b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:21.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.978+0000 7f8e0f7fe640 1 --2- 172.21.15.40:0/3426681890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e100a32b0 0x7f8e1013b0b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8e1006c620 tx=0x7f8e0002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b39eabca9dc90d78 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:21.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.978+0000 7f8e0effd640 1 -- 172.21.15.40:0/3426681890 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e0002f050 con 0x7f8e100a32b0 2024-04-22T08:15:21.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.978+0000 7f8e0effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:21.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.978+0000 7f8e0effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:15:21.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.978+0000 7f8e0effd640 10 monclient: dump: 2024-04-22T08:15:21.987 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:21.988 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:21.988 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:21.988 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:21.988 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:21.988 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:21.988 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:21.988 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:21.988 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:21.988 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:21.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.978+0000 7f8e0effd640 10 monclient: _finish_auth 0 2024-04-22T08:15:21.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.978+0000 7f8e0effd640 10 monclient: _check_auth_tickets 2024-04-22T08:15:21.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.978+0000 7f8e0effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:15:21.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.978+0000 7f8e164ce640 5 monclient: authenticate success, global_id 34287 2024-04-22T08:15:21.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.978+0000 7f8e164ce640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:15:21.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.978+0000 7f8e0effd640 1 -- 172.21.15.40:0/3426681890 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e00031910 con 0x7f8e100a32b0 2024-04-22T08:15:21.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.978+0000 7f8e0effd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:15:21.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.978+0000 7f8e164ce640 10 monclient: get_monmap_and_config success 2024-04-22T08:15:21.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.978+0000 7f8e0effd640 1 -- 172.21.15.40:0/3426681890 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e00031ca0 con 0x7f8e100a32b0 2024-04-22T08:15:21.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.978+0000 7f8e0effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:21.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.978+0000 7f8e164ce640 4 set_mon_vals no callback set 2024-04-22T08:15:21.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.978+0000 7f8e0effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:15:21.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.978+0000 7f8e0effd640 10 monclient: dump: 2024-04-22T08:15:21.989 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:21.989 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:21.989 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:21.989 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:21.989 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:21.990 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:21.990 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:21.990 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:21.991 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:21.991 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:21.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.979+0000 7f8e164ce640 10 monclient: shutdown 2024-04-22T08:15:21.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.979+0000 7f8e164ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:15:21.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.979+0000 7f8e164ce640 1 -- 172.21.15.40:0/3426681890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e100a32b0 msgr2=0x7f8e1013b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:21.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.979+0000 7f8e164ce640 1 --2- 172.21.15.40:0/3426681890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e100a32b0 0x7f8e1013b0b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8e1006c620 tx=0x7f8e0002b7a0 comp rx=0 tx=0).stop 2024-04-22T08:15:21.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.979+0000 7f8e164ce640 1 -- 172.21.15.40:0/3426681890 shutdown_connections 2024-04-22T08:15:21.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.979+0000 7f8e164ce640 1 --2- 172.21.15.40:0/3426681890 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8e1013b5f0 0x7f8e1013d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:21.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.979+0000 7f8e164ce640 1 --2- 172.21.15.40:0/3426681890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e100a32b0 0x7f8e1013b0b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:21.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.979+0000 7f8e164ce640 1 -- 172.21.15.40:0/3426681890 >> 172.21.15.40:0/3426681890 conn(0x7f8e100a4ef0 msgr2=0x7f8e100a5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:15:21.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.979+0000 7f8e164ce640 1 -- 172.21.15.40:0/3426681890 shutdown_connections 2024-04-22T08:15:21.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.979+0000 7f8e164ce640 1 -- 172.21.15.40:0/3426681890 wait complete. 2024-04-22T08:15:21.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.980+0000 7f8e164ce640 10 monclient: build_initial_monmap 2024-04-22T08:15:21.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.980+0000 7f8e164ce640 10 monclient: monmap: 2024-04-22T08:15:21.992 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:15:21.993 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:15:21.993 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:15:21.993 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:15:21.993 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:15:21.993 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:21.993 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:15:21.993 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:15:21.993 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:15:21.993 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:21.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.980+0000 7f8e164ce640 1 Processor -- start 2024-04-22T08:15:21.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.980+0000 7f8e164ce640 1 -- start start 2024-04-22T08:15:21.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.980+0000 7f8e164ce640 10 monclient: init 2024-04-22T08:15:21.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.980+0000 7f8e164ce640 10 monclient: _reopen_session rank -1 2024-04-22T08:15:21.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.980+0000 7f8e164ce640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-22T08:15:21.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.980+0000 7f8e164ce640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e1009ee10 0x7f8e101cad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:21.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.980+0000 7f8e164ce640 10 monclient(hunting): picked mon.noname-a con 0x7f8e1009ee10 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:15:21.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.980+0000 7f8e164ce640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8e100a32b0 0x7f8e101cb280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:21.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.980+0000 7f8e164ce640 10 monclient(hunting): picked mon.noname-c con 0x7f8e100a32b0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:15:21.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e164ce640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e1013b5f0 0x7f8e101cb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:21.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e164ce640 10 monclient(hunting): picked mon.noname-b con 0x7f8e1013b5f0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:15:21.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e164ce640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:21.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e164ce640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e10136c40 con 0x7f8e1009ee10 2024-04-22T08:15:21.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e164ce640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:21.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e164ce640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e100a2730 con 0x7f8e1013b5f0 2024-04-22T08:15:21.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e164ce640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:21.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e164ce640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1011f6b0 con 0x7f8e100a32b0 2024-04-22T08:15:21.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e164ce640 10 monclient(hunting): _renew_subs 2024-04-22T08:15:21.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e164ce640 10 monclient(hunting): authenticate will time out at 995.974812s 2024-04-22T08:15:21.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e0ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e1009ee10 0x7f8e101cad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:21.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e0f7fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8e100a32b0 0x7f8e101cb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:21.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e0ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e1009ee10 0x7f8e101cad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60228/0 (socket says 172.21.15.40:60228) 2024-04-22T08:15:21.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e0f7fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8e100a32b0 0x7f8e101cb280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:41780/0 (socket says 172.21.15.40:41780) 2024-04-22T08:15:21.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e0ffff640 1 -- 172.21.15.40:0/470847775 learned_addr learned my addr 172.21.15.40:0/470847775 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:15:21.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e0ffff640 10 monclient(hunting): get_auth_request con 0x7f8e1009ee10 auth_method 0 2024-04-22T08:15:21.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:21.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e0ffff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:21.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:21.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e0f7fe640 10 monclient(hunting): get_auth_request con 0x7f8e100a32b0 auth_method 0 2024-04-22T08:15:21.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e0f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:21.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e0f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:21.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:21.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e0ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:21.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e0ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:21.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e14a6b640 1 --2- 172.21.15.40:0/470847775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e1013b5f0 0x7f8e101cb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:21.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e0ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:21.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e14a6b640 10 monclient(hunting): get_auth_request con 0x7f8e1013b5f0 auth_method 0 2024-04-22T08:15:21.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e14a6b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:21.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e14a6b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:21.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e14a6b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:21.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e0f7fe640 1 -- 172.21.15.40:0/470847775 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8e100a32b0 msgr2=0x7f8e101cb280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-22T08:15:21.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e0f7fe640 1 -- 172.21.15.40:0/470847775 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8e100a32b0 msgr2=0x7f8e101cb280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:15:21.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e0f7fe640 1 --2- 172.21.15.40:0/470847775 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8e100a32b0 0x7f8e101cb280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:15:21.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e0f7fe640 1 --2- 172.21.15.40:0/470847775 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8e100a32b0 0x7f8e101cb280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:15:21.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e14a6b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:21.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e14a6b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:21.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e14a6b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:21.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e0ffff640 10 monclient(hunting): handle_auth_done global_id 34290 payload 824 2024-04-22T08:15:21.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e0ffff640 1 -- 172.21.15.40:0/470847775 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8e100a32b0 msgr2=0x7f8e101cb280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:15:21.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e0ffff640 1 --2- 172.21.15.40:0/470847775 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8e100a32b0 0x7f8e101cb280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:21.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e0ffff640 1 -- 172.21.15.40:0/470847775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e1013b5f0 msgr2=0x7f8e101cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:21.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e0ffff640 1 --2- 172.21.15.40:0/470847775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e1013b5f0 0x7f8e101cb7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:21.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e0ffff640 10 monclient: _finish_hunting 0 2024-04-22T08:15:21.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e0ffff640 1 monclient: found mon.noname-a 2024-04-22T08:15:21.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e0ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:15:21.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e0ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:15:21.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.981+0000 7f8e0ffff640 1 -- 172.21.15.40:0/470847775 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e000094d0 con 0x7f8e1009ee10 2024-04-22T08:15:21.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.983+0000 7f8e0ffff640 1 -- 172.21.15.40:0/470847775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e1009ee10 msgr2=0x7f8e101cad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:21.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.983+0000 7f8e0ffff640 1 --2- 172.21.15.40:0/470847775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e1009ee10 0x7f8e101cad40 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8e0400b3f0 tx=0x7f8e0400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:21.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.983+0000 7f8e0cff9640 1 -- 172.21.15.40:0/470847775 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e04011a40 con 0x7f8e1009ee10 2024-04-22T08:15:21.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.983+0000 7f8e0cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:21.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.983+0000 7f8e0cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:15:21.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.983+0000 7f8e0cff9640 10 monclient: dump: 2024-04-22T08:15:21.999 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:21.999 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:21.999 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:21.999 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:21.999 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:21.999 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:22.000 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:22.000 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:22.000 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:22.000 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:22.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.983+0000 7f8e0cff9640 10 monclient: _finish_auth 0 2024-04-22T08:15:22.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.983+0000 7f8e0cff9640 10 monclient: _check_auth_tickets 2024-04-22T08:15:22.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.983+0000 7f8e0cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:15:22.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.983+0000 7f8e164ce640 5 monclient: authenticate success, global_id 34290 2024-04-22T08:15:22.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.983+0000 7f8e0cff9640 1 -- 172.21.15.40:0/470847775 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e04011ba0 con 0x7f8e1009ee10 2024-04-22T08:15:22.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.983+0000 7f8e0cff9640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:15:22.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.983+0000 7f8e164ce640 10 monclient: _renew_subs 2024-04-22T08:15:22.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.983+0000 7f8e0cff9640 1 -- 172.21.15.40:0/470847775 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e04010380 con 0x7f8e1009ee10 2024-04-22T08:15:22.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.983+0000 7f8e164ce640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:22.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.983+0000 7f8e164ce640 1 -- 172.21.15.40:0/470847775 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e1011f990 con 0x7f8e1009ee10 2024-04-22T08:15:22.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.983+0000 7f8e0cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:22.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.983+0000 7f8e0dffb640 4 set_mon_vals no callback set 2024-04-22T08:15:22.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.983+0000 7f8e0cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:15:22.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.983+0000 7f8e0cff9640 10 monclient: dump: 2024-04-22T08:15:22.001 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:22.001 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:22.001 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:22.001 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:22.002 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:22.002 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:22.002 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:22.002 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:22.002 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:22.002 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:22.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.983+0000 7f8e164ce640 10 monclient: _renew_subs 2024-04-22T08:15:22.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.984+0000 7f8e164ce640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:22.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.984+0000 7f8e164ce640 1 -- 172.21.15.40:0/470847775 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e10071020 con 0x7f8e1009ee10 2024-04-22T08:15:22.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.984+0000 7f8e164ce640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:22.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.984+0000 7f8e164ce640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:22.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.984+0000 7f8e0cff9640 1 -- 172.21.15.40:0/470847775 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f8e04010560 con 0x7f8e1009ee10 2024-04-22T08:15:22.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.984+0000 7f8e164ce640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:22.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.984+0000 7f8e164ce640 1 -- 172.21.15.40:0/470847775 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e1004e9b0 con 0x7f8e1009ee10 2024-04-22T08:15:22.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.985+0000 7f8e0cff9640 1 --2- 172.21.15.40:0/470847775 >> v2:172.21.15.189:6800/30168 conn(0x7f8dfc0377d0 0x7f8dfc039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:22.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.985+0000 7f8e0cff9640 1 -- 172.21.15.40:0/470847775 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f8e0404a250 con 0x7f8e1009ee10 2024-04-22T08:15:22.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.985+0000 7f8e0f7fe640 1 --2- 172.21.15.40:0/470847775 >> v2:172.21.15.189:6800/30168 conn(0x7f8dfc0377d0 0x7f8dfc039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:22.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.985+0000 7f8e0f7fe640 10 monclient: get_auth_request con 0x7f8dfc0377d0 auth_method 0 2024-04-22T08:15:22.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.988+0000 7f8e0f7fe640 1 --2- 172.21.15.40:0/470847775 >> v2:172.21.15.189:6800/30168 conn(0x7f8dfc0377d0 0x7f8dfc039c90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8e00035000 tx=0x7f8e00008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:22.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.990+0000 7f8e0cff9640 1 -- 172.21.15.40:0/470847775 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f8e040208a0 con 0x7f8e1009ee10 2024-04-22T08:15:22.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.990+0000 7f8e0cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:22.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:21.990+0000 7f8e0cff9640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:15:22.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:22.238+0000 7f8e164ce640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:15:22.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:22.238+0000 7f8e164ce640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:15:22.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:22.238+0000 7f8e164ce640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:22.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:22.238+0000 7f8e164ce640 1 -- 172.21.15.40:0/470847775 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8e1011feb0 con 0x7f8e1009ee10 2024-04-22T08:15:22.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:22.238+0000 7f8e0cff9640 1 -- 172.21.15.40:0/470847775 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7f8e04015070 con 0x7f8e1009ee10 2024-04-22T08:15:22.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:22.238+0000 7f8e0cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:15:22.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:22.238+0000 7f8e0cff9640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:15:22.241 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":30,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:15:22.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:22.240+0000 7f8e164ce640 1 -- 172.21.15.40:0/470847775 >> v2:172.21.15.189:6800/30168 conn(0x7f8dfc0377d0 msgr2=0x7f8dfc039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:22.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:22.240+0000 7f8e164ce640 1 --2- 172.21.15.40:0/470847775 >> v2:172.21.15.189:6800/30168 conn(0x7f8dfc0377d0 0x7f8dfc039c90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8e00035000 tx=0x7f8e00008030 comp rx=0 tx=0).stop 2024-04-22T08:15:22.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:22.240+0000 7f8e164ce640 10 monclient: shutdown 2024-04-22T08:15:22.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:22.240+0000 7f8e164ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:15:22.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:22.240+0000 7f8e164ce640 1 -- 172.21.15.40:0/470847775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e1009ee10 msgr2=0x7f8e101cad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:22.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:22.240+0000 7f8e164ce640 1 --2- 172.21.15.40:0/470847775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e1009ee10 0x7f8e101cad40 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8e0400b3f0 tx=0x7f8e0400b8b0 comp rx=0 tx=0).stop 2024-04-22T08:15:22.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:22.240+0000 7f8e164ce640 1 -- 172.21.15.40:0/470847775 shutdown_connections 2024-04-22T08:15:22.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:22.240+0000 7f8e164ce640 1 --2- 172.21.15.40:0/470847775 >> v2:172.21.15.189:6800/30168 conn(0x7f8dfc0377d0 0x7f8dfc039c90 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:22.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:22.240+0000 7f8e164ce640 1 --2- 172.21.15.40:0/470847775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e1013b5f0 0x7f8e101cb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:22.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:22.240+0000 7f8e164ce640 1 --2- 172.21.15.40:0/470847775 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8e100a32b0 0x7f8e101cb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:22.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:22.240+0000 7f8e164ce640 1 --2- 172.21.15.40:0/470847775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e1009ee10 0x7f8e101cad40 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:22.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:22.240+0000 7f8e164ce640 1 -- 172.21.15.40:0/470847775 >> 172.21.15.40:0/470847775 conn(0x7f8e100a4ef0 msgr2=0x7f8e100a1980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:15:22.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:22.240+0000 7f8e164ce640 1 -- 172.21.15.40:0/470847775 shutdown_connections 2024-04-22T08:15:22.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:22.240+0000 7f8e164ce640 1 -- 172.21.15.40:0/470847775 wait complete. 2024-04-22T08:15:22.852 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[35797]: starting. 2024-04-22T08:15:22.852 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[35797]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-35795 2024-04-22T08:15:22.852 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[35797]: shutting down. 2024-04-22T08:15:24.964 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:15:24.965 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35830]: starting. 2024-04-22T08:15:24.965 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35830]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-35795 2024-04-22T08:15:24.965 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35830]: created object 0... 2024-04-22T08:15:24.965 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35830]: created object 25... 2024-04-22T08:15:24.965 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35830]: created object 49... 2024-04-22T08:15:24.965 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35830]: finishing. 2024-04-22T08:15:24.965 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35830]: shutting down. 2024-04-22T08:15:25.011 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:15:25.011 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35832]: starting. 2024-04-22T08:15:25.011 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35832]: listing objects. 2024-04-22T08:15:25.011 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35832]: listed object 0... 2024-04-22T08:15:25.011 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35832]: listed object 25... 2024-04-22T08:15:25.011 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35832]: saw 50 objects 2024-04-22T08:15:25.011 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35832]: shutting down. 2024-04-22T08:15:25.251 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:15:25.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.362+0000 7fd4a4d8a640 10 monclient: get_monmap_and_config 2024-04-22T08:15:25.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.362+0000 7fd4a4d8a640 10 monclient: build_initial_monmap 2024-04-22T08:15:25.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.362+0000 7fd4a4d8a640 10 monclient: monmap: 2024-04-22T08:15:25.362 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:15:25.362 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:25.362 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:15:25.363291+0000 2024-04-22T08:15:25.362 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:15:25.363291+0000 2024-04-22T08:15:25.362 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:15:25.362 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:25.362 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:15:25.363 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:15:25.363 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:15:25.363 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:15:25.363 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:15:25.363 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:25.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.363+0000 7fd4a4d8a640 1 Processor -- start 2024-04-22T08:15:25.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.363+0000 7fd4a4d8a640 1 -- start start 2024-04-22T08:15:25.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.363+0000 7fd4a4d8a640 10 monclient: init 2024-04-22T08:15:25.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.363+0000 7fd4a4d8a640 10 monclient: _reopen_session rank -1 2024-04-22T08:15:25.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.363+0000 7fd4a4d8a640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-04-22T08:15:25.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.363+0000 7fd4a4d8a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd4a009f0b0 addr v1:172.21.15.40:6789/0 2024-04-22T08:15:25.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.363+0000 7fd4a4d8a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd4a009e2b0 addr v1:172.21.15.189:6789/0 2024-04-22T08:15:25.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.363+0000 7fd4a4d8a640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd4a013a560 0x7fd4a013c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:25.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.363+0000 7fd4a4d8a640 10 monclient(hunting): picked mon.noname-c con 0x7fd4a013a560 addr v2:172.21.15.189:3300/0 2024-04-22T08:15:25.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.363+0000 7fd4a4d8a640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4a0071670 con 0x7fd4a009f0b0 2024-04-22T08:15:25.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.363+0000 7fd4a4d8a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4a009fba0 con 0x7fd4a009e2b0 2024-04-22T08:15:25.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.363+0000 7fd4a4d8a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:25.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.363+0000 7fd4a4d8a640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4a009d9e0 con 0x7fd4a013a560 2024-04-22T08:15:25.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.363+0000 7fd4a4d8a640 10 monclient(hunting): _renew_subs 2024-04-22T08:15:25.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.363+0000 7fd4a4d8a640 10 monclient(hunting): authenticate will time out at 999.357640s 2024-04-22T08:15:25.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.363+0000 7fd49e59c640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fd4a009f0b0 0x7fd4a009f490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53220/0 (socket says 172.21.15.40:53220) 2024-04-22T08:15:25.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.364+0000 7fd49e59c640 1 -- 172.21.15.40:0/3661643926 learned_addr learned my addr 172.21.15.40:0/3661643926 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:15:25.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.364+0000 7fd49ed9d640 1 --2- 172.21.15.40:0/3661643926 >> v2:172.21.15.189:3300/0 conn(0x7fd4a013a560 0x7fd4a013c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:25.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.364+0000 7fd49ed9d640 10 monclient(hunting): get_auth_request con 0x7fd4a013a560 auth_method 0 2024-04-22T08:15:25.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.364+0000 7fd49ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:25.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.364+0000 7fd49ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:25.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.364+0000 7fd49ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:25.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.364+0000 7fd49ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:25.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.364+0000 7fd49ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:25.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.364+0000 7fd49ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:25.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.366+0000 7fd49e59c640 1 -- 172.21.15.40:0/3661643926 >> v1:172.21.15.40:6789/0 conn(0x7fd4a009f0b0 legacy=0x7fd4a009f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:25.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.366+0000 7fd49dd9b640 1 -- 172.21.15.40:0/3661643926 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2139917555 0 0) 0x7fd4a0071670 con 0x7fd4a009f0b0 2024-04-22T08:15:25.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.366+0000 7fd49dd9b640 10 monclient(hunting): _negotiate 2024-04-22T08:15:25.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.366+0000 7fd49dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:25.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.366+0000 7fd49dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:25.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.366+0000 7fd49dd9b640 10 monclient(hunting): my global_id is 34305 2024-04-22T08:15:25.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.366+0000 7fd49dd9b640 1 -- 172.21.15.40:0/3661643926 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd48c003520 con 0x7fd4a009f0b0 2024-04-22T08:15:25.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.366+0000 7fd497fff640 1 -- 172.21.15.40:0/3661643926 >> v1:172.21.15.189:6789/0 conn(0x7fd4a009e2b0 legacy=0x7fd4a0138e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:25.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.366+0000 7fd49ed9d640 10 monclient(hunting): handle_auth_done global_id 24508 payload 274 2024-04-22T08:15:25.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.366+0000 7fd49ed9d640 1 -- 172.21.15.40:0/3661643926 >> v1:172.21.15.189:6789/0 conn(0x7fd4a009e2b0 legacy=0x7fd4a0138e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:25.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.366+0000 7fd49ed9d640 1 -- 172.21.15.40:0/3661643926 >> v1:172.21.15.40:6789/0 conn(0x7fd4a009f0b0 legacy=0x7fd4a009f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:25.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.366+0000 7fd49ed9d640 10 monclient: _finish_hunting 0 2024-04-22T08:15:25.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.366+0000 7fd49ed9d640 1 monclient: found mon.noname-c 2024-04-22T08:15:25.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.366+0000 7fd49ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:15:25.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.366+0000 7fd49ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.189:3300/0 2024-04-22T08:15:25.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.366+0000 7fd49ed9d640 1 -- 172.21.15.40:0/3661643926 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4a009d7a0 con 0x7fd4a013a560 2024-04-22T08:15:25.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.369+0000 7fd49ed9d640 1 -- 172.21.15.40:0/3661643926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4a013a560 msgr2=0x7fd4a013c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:25.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.369+0000 7fd49ed9d640 1 --2- 172.21.15.40:0/3661643926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4a013a560 0x7fd4a013c950 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd498009e90 tx=0x7fd49802bf20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf1a6777e097b59d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:25.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.370+0000 7fd49dd9b640 1 -- 172.21.15.40:0/3661643926 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd498034440 con 0x7fd4a013a560 2024-04-22T08:15:25.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.370+0000 7fd49dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:25.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.370+0000 7fd49dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:15:25.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.370+0000 7fd49dd9b640 10 monclient: dump: 2024-04-22T08:15:25.371 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:25.371 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:25.371 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:25.371 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:25.371 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:25.371 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:25.371 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:25.371 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:25.371 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:25.371 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:25.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.370+0000 7fd49dd9b640 10 monclient: _finish_auth 0 2024-04-22T08:15:25.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.370+0000 7fd49dd9b640 10 monclient: _check_auth_tickets 2024-04-22T08:15:25.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.370+0000 7fd49dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:15:25.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.370+0000 7fd4a4d8a640 5 monclient: authenticate success, global_id 24508 2024-04-22T08:15:25.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.370+0000 7fd4a4d8a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:15:25.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.370+0000 7fd49dd9b640 1 -- 172.21.15.40:0/3661643926 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4980345a0 con 0x7fd4a013a560 2024-04-22T08:15:25.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.370+0000 7fd49dd9b640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:15:25.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.370+0000 7fd49dd9b640 1 -- 172.21.15.40:0/3661643926 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd498034910 con 0x7fd4a013a560 2024-04-22T08:15:25.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.370+0000 7fd4a4d8a640 10 monclient: get_monmap_and_config success 2024-04-22T08:15:25.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.370+0000 7fd4a4d8a640 4 set_mon_vals no callback set 2024-04-22T08:15:25.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.370+0000 7fd49dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:25.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.370+0000 7fd49dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:15:25.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.370+0000 7fd49dd9b640 10 monclient: dump: 2024-04-22T08:15:25.373 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:25.373 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:25.373 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:25.373 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:25.373 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:25.373 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:25.373 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:25.373 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:25.373 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:25.373 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:25.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.370+0000 7fd4a4d8a640 10 monclient: shutdown 2024-04-22T08:15:25.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.370+0000 7fd4a4d8a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:15:25.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.370+0000 7fd4a4d8a640 1 -- 172.21.15.40:0/3661643926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4a013a560 msgr2=0x7fd4a013c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:25.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.370+0000 7fd4a4d8a640 1 --2- 172.21.15.40:0/3661643926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4a013a560 0x7fd4a013c950 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd498009e90 tx=0x7fd49802bf20 comp rx=0 tx=0).stop 2024-04-22T08:15:25.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.370+0000 7fd4a4d8a640 1 -- 172.21.15.40:0/3661643926 shutdown_connections 2024-04-22T08:15:25.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.370+0000 7fd4a4d8a640 1 --2- 172.21.15.40:0/3661643926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4a013a560 0x7fd4a013c950 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:25.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.370+0000 7fd4a4d8a640 1 -- 172.21.15.40:0/3661643926 >> 172.21.15.40:0/3661643926 conn(0x7fd4a00a0b50 msgr2=0x7fd4a00a0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:15:25.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.370+0000 7fd4a4d8a640 1 -- 172.21.15.40:0/3661643926 shutdown_connections 2024-04-22T08:15:25.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.370+0000 7fd4a4d8a640 1 -- 172.21.15.40:0/3661643926 wait complete. 2024-04-22T08:15:25.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.371+0000 7fd4a4d8a640 10 monclient: build_initial_monmap 2024-04-22T08:15:25.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.371+0000 7fd4a4d8a640 10 monclient: monmap: 2024-04-22T08:15:25.378 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:15:25.379 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:15:25.379 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:15:25.379 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:15:25.379 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:15:25.379 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:25.379 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:15:25.379 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:15:25.379 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:15:25.379 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:25.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.371+0000 7fd4a4d8a640 1 Processor -- start 2024-04-22T08:15:25.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd4a4d8a640 1 -- start start 2024-04-22T08:15:25.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd4a4d8a640 10 monclient: init 2024-04-22T08:15:25.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd4a4d8a640 10 monclient: _reopen_session rank -1 2024-04-22T08:15:25.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd4a4d8a640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-22T08:15:25.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd4a4d8a640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4a009e2b0 0x7fd4a01386c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:25.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd4a4d8a640 10 monclient(hunting): picked mon.noname-c con 0x7fd4a009e2b0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:15:25.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd4a4d8a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4a009f0b0 0x7fd4a0138c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:25.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd4a4d8a640 10 monclient(hunting): picked mon.noname-a con 0x7fd4a009f0b0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:15:25.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd4a4d8a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4a013a560 0x7fd4a009b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:25.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd4a4d8a640 10 monclient(hunting): picked mon.noname-b con 0x7fd4a013a560 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:15:25.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd4a4d8a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:25.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd4a4d8a640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a0125ee0 con 0x7fd4a009f0b0 2024-04-22T08:15:25.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd4a4d8a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:25.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd4a4d8a640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a0139c10 con 0x7fd4a013a560 2024-04-22T08:15:25.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd4a4d8a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:25.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd4a4d8a640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a0125920 con 0x7fd4a009e2b0 2024-04-22T08:15:25.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd4a4d8a640 10 monclient(hunting): _renew_subs 2024-04-22T08:15:25.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd4a4d8a640 10 monclient(hunting): authenticate will time out at 999.366240s 2024-04-22T08:15:25.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd497fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4a009f0b0 0x7fd4a0138c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:25.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd49e59c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4a009e2b0 0x7fd4a01386c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:25.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd497fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4a009f0b0 0x7fd4a0138c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60272/0 (socket says 172.21.15.40:60272) 2024-04-22T08:15:25.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd497fff640 1 -- 172.21.15.40:0/523487106 learned_addr learned my addr 172.21.15.40:0/523487106 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:15:25.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd497fff640 10 monclient(hunting): get_auth_request con 0x7fd4a009f0b0 auth_method 0 2024-04-22T08:15:25.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd497fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:25.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd497fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:25.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd497fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:25.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd49e59c640 10 monclient(hunting): get_auth_request con 0x7fd4a009e2b0 auth_method 0 2024-04-22T08:15:25.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd49e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:25.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd49e59c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:25.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd49e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:25.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd497fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:25.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd497fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:25.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd497fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:25.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd49e59c640 1 -- 172.21.15.40:0/523487106 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4a009e2b0 msgr2=0x7fd4a01386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-22T08:15:25.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd49ed9d640 1 --2- 172.21.15.40:0/523487106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4a013a560 0x7fd4a009b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:25.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd49e59c640 1 -- 172.21.15.40:0/523487106 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4a009e2b0 msgr2=0x7fd4a01386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:15:25.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd49e59c640 1 --2- 172.21.15.40:0/523487106 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4a009e2b0 0x7fd4a01386c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:15:25.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd49ed9d640 10 monclient(hunting): get_auth_request con 0x7fd4a013a560 auth_method 0 2024-04-22T08:15:25.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd49ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:25.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd49ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:25.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd49ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:25.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd49e59c640 1 --2- 172.21.15.40:0/523487106 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4a009e2b0 0x7fd4a01386c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:15:25.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd49ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:25.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd49ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:25.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd49ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:25.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd497fff640 10 monclient(hunting): handle_auth_done global_id 34308 payload 824 2024-04-22T08:15:25.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd497fff640 1 -- 172.21.15.40:0/523487106 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4a009e2b0 msgr2=0x7fd4a01386c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:15:25.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd497fff640 1 --2- 172.21.15.40:0/523487106 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4a009e2b0 0x7fd4a01386c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:25.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd497fff640 1 -- 172.21.15.40:0/523487106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4a013a560 msgr2=0x7fd4a009b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:25.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd497fff640 1 --2- 172.21.15.40:0/523487106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4a013a560 0x7fd4a009b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:25.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd497fff640 10 monclient: _finish_hunting 0 2024-04-22T08:15:25.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd497fff640 1 monclient: found mon.noname-a 2024-04-22T08:15:25.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd497fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:15:25.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd497fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:15:25.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.372+0000 7fd497fff640 1 -- 172.21.15.40:0/523487106 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd498009a70 con 0x7fd4a009f0b0 2024-04-22T08:15:25.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.375+0000 7fd497fff640 1 -- 172.21.15.40:0/523487106 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4a009f0b0 msgr2=0x7fd4a0138c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:25.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.375+0000 7fd497fff640 1 --2- 172.21.15.40:0/523487106 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4a009f0b0 0x7fd4a0138c00 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd48800c480 tx=0x7fd48800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:25.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.375+0000 7fd496ffd640 1 -- 172.21.15.40:0/523487106 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd488007690 con 0x7fd4a009f0b0 2024-04-22T08:15:25.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.375+0000 7fd496ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:25.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.375+0000 7fd496ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:15:25.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.375+0000 7fd497fff640 1 --2- 172.21.15.40:0/523487106 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4a009f0b0 0x7fd4a0138c00 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd48800c480 tx=0x7fd48800c940 comp rx=0 tx=0).queue_received will delay after 1.71377e+09 on 0x7fd488005b70 mon_map magic: 0 v1 2024-04-22T08:15:25.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.375+0000 7fd496ffd640 10 monclient: dump: 2024-04-22T08:15:25.384 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:25.384 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:25.384 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:25.384 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:25.384 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:25.384 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:25.384 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:25.384 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:25.384 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:25.384 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:25.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.375+0000 7fd496ffd640 10 monclient: _finish_auth 0 2024-04-22T08:15:25.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.375+0000 7fd496ffd640 10 monclient: _check_auth_tickets 2024-04-22T08:15:25.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.375+0000 7fd496ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:15:25.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.375+0000 7fd4a4d8a640 5 monclient: authenticate success, global_id 34308 2024-04-22T08:15:25.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.375+0000 7fd496ffd640 1 -- 172.21.15.40:0/523487106 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4880057c0 con 0x7fd4a009f0b0 2024-04-22T08:15:25.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.375+0000 7fd496ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:15:25.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.375+0000 7fd4a4d8a640 10 monclient: _renew_subs 2024-04-22T08:15:25.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.375+0000 7fd4a4d8a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:25.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.375+0000 7fd4a4d8a640 1 -- 172.21.15.40:0/523487106 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4a0125c00 con 0x7fd4a009f0b0 2024-04-22T08:15:25.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.375+0000 7fd49d59a640 4 set_mon_vals no callback set 2024-04-22T08:15:25.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.375+0000 7fd4a4d8a640 10 monclient: _renew_subs 2024-04-22T08:15:25.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.375+0000 7fd4a4d8a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:25.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.375+0000 7fd4a4d8a640 1 -- 172.21.15.40:0/523487106 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4a0124a30 con 0x7fd4a009f0b0 2024-04-22T08:15:25.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.376+0000 7fd496ffd640 1 -- 172.21.15.40:0/523487106 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd488005b70 con 0x7fd4a009f0b0 2024-04-22T08:15:25.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.376+0000 7fd496ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:25.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.376+0000 7fd496ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:15:25.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.376+0000 7fd496ffd640 10 monclient: dump: 2024-04-22T08:15:25.386 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:25.386 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:25.386 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:25.386 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:25.386 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:25.386 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:25.386 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:25.386 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:25.386 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:25.387 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:25.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.376+0000 7fd496ffd640 1 -- 172.21.15.40:0/523487106 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fd488005d50 con 0x7fd4a009f0b0 2024-04-22T08:15:25.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.376+0000 7fd4a4d8a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:25.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.376+0000 7fd4a4d8a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:25.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.376+0000 7fd4a4d8a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:25.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.376+0000 7fd4a4d8a640 1 -- 172.21.15.40:0/523487106 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4a004e9b0 con 0x7fd4a009f0b0 2024-04-22T08:15:25.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.376+0000 7fd496ffd640 1 --2- 172.21.15.40:0/523487106 >> v2:172.21.15.189:6800/30168 conn(0x7fd4800377d0 0x7fd480039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:25.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.376+0000 7fd49e59c640 1 --2- 172.21.15.40:0/523487106 >> v2:172.21.15.189:6800/30168 conn(0x7fd4800377d0 0x7fd480039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:25.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.376+0000 7fd49e59c640 10 monclient: get_auth_request con 0x7fd4800377d0 auth_method 0 2024-04-22T08:15:25.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.379+0000 7fd49e59c640 1 --2- 172.21.15.40:0/523487106 >> v2:172.21.15.189:6800/30168 conn(0x7fd4800377d0 0x7fd480039c90 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd490001550 tx=0x7fd490009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:25.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:25.381+0000 7fd496ffd640 1 -- 172.21.15.40:0/523487106 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 3898+0+0 (secure 0 0 0) 0x7fd488049ab0 con 0x7fd4a009f0b0 2024-04-22T08:15:25.858 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:15:25.858 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[35831]: starting. 2024-04-22T08:15:25.858 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[35831]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-35795 2024-04-22T08:15:25.859 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[35831]: shutting down. 2024-04-22T08:15:25.860 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:15:25.860 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:15:25.860 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:15:25.863 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:15:26.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:26.203+0000 7fd496ffd640 1 -- 172.21.15.40:0/523487106 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7fd48800e280 con 0x7fd4a009f0b0 2024-04-22T08:15:26.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:26.203+0000 7fd496ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:26.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:26.203+0000 7fd496ffd640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:15:26.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:26.454+0000 7fd4a4d8a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:15:26.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:26.454+0000 7fd4a4d8a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:15:26.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:26.454+0000 7fd4a4d8a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:26.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:26.454+0000 7fd4a4d8a640 1 -- 172.21.15.40:0/523487106 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd4a006a290 con 0x7fd4a009f0b0 2024-04-22T08:15:26.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:26.455+0000 7fd496ffd640 1 -- 172.21.15.40:0/523487106 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7fd488017030 con 0x7fd4a009f0b0 2024-04-22T08:15:26.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:26.455+0000 7fd496ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:15:26.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:26.455+0000 7fd496ffd640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:15:26.455 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":30,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":25,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:15:26.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:26.456+0000 7fd4a4d8a640 1 -- 172.21.15.40:0/523487106 >> v2:172.21.15.189:6800/30168 conn(0x7fd4800377d0 msgr2=0x7fd480039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:26.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:26.456+0000 7fd4a4d8a640 1 --2- 172.21.15.40:0/523487106 >> v2:172.21.15.189:6800/30168 conn(0x7fd4800377d0 0x7fd480039c90 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd490001550 tx=0x7fd490009040 comp rx=0 tx=0).stop 2024-04-22T08:15:26.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:26.457+0000 7fd4a4d8a640 10 monclient: shutdown 2024-04-22T08:15:26.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:26.457+0000 7fd4a4d8a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:15:26.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:26.457+0000 7fd4a4d8a640 1 -- 172.21.15.40:0/523487106 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4a009f0b0 msgr2=0x7fd4a0138c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:26.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:26.457+0000 7fd4a4d8a640 1 --2- 172.21.15.40:0/523487106 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4a009f0b0 0x7fd4a0138c00 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd48800c480 tx=0x7fd48800c940 comp rx=0 tx=0).stop 2024-04-22T08:15:26.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:26.457+0000 7fd4a4d8a640 1 -- 172.21.15.40:0/523487106 shutdown_connections 2024-04-22T08:15:26.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:26.457+0000 7fd4a4d8a640 1 --2- 172.21.15.40:0/523487106 >> v2:172.21.15.189:6800/30168 conn(0x7fd4800377d0 0x7fd480039c90 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:26.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:26.457+0000 7fd4a4d8a640 1 --2- 172.21.15.40:0/523487106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4a013a560 0x7fd4a009b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:26.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:26.457+0000 7fd4a4d8a640 1 --2- 172.21.15.40:0/523487106 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4a009f0b0 0x7fd4a0138c00 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:26.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:26.457+0000 7fd4a4d8a640 1 --2- 172.21.15.40:0/523487106 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4a009e2b0 0x7fd4a01386c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:26.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:26.457+0000 7fd4a4d8a640 1 -- 172.21.15.40:0/523487106 >> 172.21.15.40:0/523487106 conn(0x7fd4a00a0b50 msgr2=0x7fd4a00a3570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:15:26.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:26.457+0000 7fd4a4d8a640 1 -- 172.21.15.40:0/523487106 shutdown_connections 2024-04-22T08:15:26.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:26.457+0000 7fd4a4d8a640 1 -- 172.21.15.40:0/523487106 wait complete. 2024-04-22T08:15:29.469 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:15:29.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.579+0000 7fd3dece9640 10 monclient: get_monmap_and_config 2024-04-22T08:15:29.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.579+0000 7fd3dece9640 10 monclient: build_initial_monmap 2024-04-22T08:15:29.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.579+0000 7fd3dece9640 10 monclient: monmap: 2024-04-22T08:15:29.579 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:15:29.579 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:29.579 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:15:29.580625+0000 2024-04-22T08:15:29.579 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:15:29.580625+0000 2024-04-22T08:15:29.580 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:15:29.580 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:29.580 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:15:29.580 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:15:29.580 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:15:29.580 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:15:29.580 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:15:29.580 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:29.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.580+0000 7fd3dece9640 1 Processor -- start 2024-04-22T08:15:29.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.580+0000 7fd3dece9640 1 -- start start 2024-04-22T08:15:29.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.580+0000 7fd3dece9640 10 monclient: init 2024-04-22T08:15:29.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.581+0000 7fd3dece9640 10 monclient: _reopen_session rank -1 2024-04-22T08:15:29.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.581+0000 7fd3dece9640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-04-22T08:15:29.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.581+0000 7fd3dece9640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd3d8135e30 0x7fd3d8136210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:29.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.581+0000 7fd3dece9640 10 monclient(hunting): picked mon.noname-c con 0x7fd3d8135e30 addr v2:172.21.15.189:3300/0 2024-04-22T08:15:29.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.581+0000 7fd3dece9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd3d8071c90 addr v1:172.21.15.189:6789/0 2024-04-22T08:15:29.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.581+0000 7fd3dece9640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3d8136750 0x7fd3d80a20b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:29.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.581+0000 7fd3dece9640 10 monclient(hunting): picked mon.noname-b con 0x7fd3d8136750 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:15:29.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.581+0000 7fd3dece9640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3d809e310 con 0x7fd3d8071c90 2024-04-22T08:15:29.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.581+0000 7fd3dece9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:29.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.581+0000 7fd3dece9640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3d809de80 con 0x7fd3d8135e30 2024-04-22T08:15:29.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.581+0000 7fd3dece9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:29.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.581+0000 7fd3dece9640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d809f130 con 0x7fd3d8136750 2024-04-22T08:15:29.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.581+0000 7fd3dece9640 10 monclient(hunting): _renew_subs 2024-04-22T08:15:29.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.581+0000 7fd3dece9640 10 monclient(hunting): authenticate will time out at 1003.574996s 2024-04-22T08:15:29.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.581+0000 7fd3dd286640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3d8136750 0x7fd3d80a20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:29.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.581+0000 7fd3dd286640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3d8136750 0x7fd3d80a20b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:60680/0 (socket says 172.21.15.40:60680) 2024-04-22T08:15:29.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.581+0000 7fd3dd286640 1 -- 172.21.15.40:0/3133758165 learned_addr learned my addr 172.21.15.40:0/3133758165 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:15:29.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.581+0000 7fd3dd286640 10 monclient(hunting): get_auth_request con 0x7fd3d8136750 auth_method 0 2024-04-22T08:15:29.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.581+0000 7fd3dd286640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:29.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.581+0000 7fd3dd286640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:29.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.581+0000 7fd3dd286640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:29.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.581+0000 7fd3dca85640 1 --2- 172.21.15.40:0/3133758165 >> v2:172.21.15.189:3300/0 conn(0x7fd3d8135e30 0x7fd3d8136210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:29.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.581+0000 7fd3dca85640 10 monclient(hunting): get_auth_request con 0x7fd3d8135e30 auth_method 0 2024-04-22T08:15:29.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.581+0000 7fd3dca85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:29.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.581+0000 7fd3dca85640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:29.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.581+0000 7fd3dca85640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:29.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.581+0000 7fd3dd286640 1 -- 172.21.15.40:0/3133758165 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3d8136750 msgr2=0x7fd3d80a20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-22T08:15:29.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.581+0000 7fd3dd286640 1 -- 172.21.15.40:0/3133758165 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3d8136750 msgr2=0x7fd3d80a20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:15:29.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.581+0000 7fd3dd286640 1 --2- 172.21.15.40:0/3133758165 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3d8136750 0x7fd3d80a20b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:15:29.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.581+0000 7fd3dd286640 1 --2- 172.21.15.40:0/3133758165 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3d8136750 0x7fd3d80a20b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:15:29.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.581+0000 7fd3dca85640 1 -- 172.21.15.40:0/3133758165 >> v2:172.21.15.189:3300/0 conn(0x7fd3d8135e30 msgr2=0x7fd3d8136210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:15:29.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.581+0000 7fd3dca85640 1 -- 172.21.15.40:0/3133758165 >> v2:172.21.15.189:3300/0 conn(0x7fd3d8135e30 msgr2=0x7fd3d8136210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:15:29.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.581+0000 7fd3dca85640 1 --2- 172.21.15.40:0/3133758165 >> v2:172.21.15.189:3300/0 conn(0x7fd3d8135e30 0x7fd3d8136210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:15:29.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.581+0000 7fd3dca85640 1 --2- 172.21.15.40:0/3133758165 >> v2:172.21.15.189:3300/0 conn(0x7fd3d8135e30 0x7fd3d8136210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:15:29.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.583+0000 7fd3d7fff640 1 -- 172.21.15.40:0/3133758165 >> v1:172.21.15.189:6789/0 conn(0x7fd3d8071c90 legacy=0x7fd3d80a19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:29.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.782+0000 7fd3dd286640 1 --2- 172.21.15.40:0/3133758165 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3d8136750 0x7fd3d80a20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:29.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.782+0000 7fd3dd286640 10 monclient(hunting): get_auth_request con 0x7fd3d8136750 auth_method 0 2024-04-22T08:15:29.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.782+0000 7fd3dd286640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:29.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.782+0000 7fd3dd286640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:29.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.782+0000 7fd3dd286640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:15:29.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.782+0000 7fd3dd286640 1 -- 172.21.15.40:0/3133758165 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3d8136750 msgr2=0x7fd3d80a20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:15:29.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.782+0000 7fd3dd286640 1 -- 172.21.15.40:0/3133758165 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3d8136750 msgr2=0x7fd3d80a20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:15:29.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.782+0000 7fd3dd286640 1 --2- 172.21.15.40:0/3133758165 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3d8136750 0x7fd3d80a20b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:15:29.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.782+0000 7fd3dd286640 1 --2- 172.21.15.40:0/3133758165 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3d8136750 0x7fd3d80a20b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:15:29.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.782+0000 7fd3dca85640 1 --2- 172.21.15.40:0/3133758165 >> v2:172.21.15.189:3300/0 conn(0x7fd3d8135e30 0x7fd3d8136210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:29.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.782+0000 7fd3dca85640 10 monclient(hunting): get_auth_request con 0x7fd3d8135e30 auth_method 0 2024-04-22T08:15:29.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.782+0000 7fd3dca85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:29.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.782+0000 7fd3dca85640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:29.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.782+0000 7fd3dca85640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:15:29.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.782+0000 7fd3dca85640 1 -- 172.21.15.40:0/3133758165 >> v2:172.21.15.189:3300/0 conn(0x7fd3d8135e30 msgr2=0x7fd3d8136210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-22T08:15:29.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.782+0000 7fd3dca85640 1 -- 172.21.15.40:0/3133758165 >> v2:172.21.15.189:3300/0 conn(0x7fd3d8135e30 msgr2=0x7fd3d8136210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:15:29.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.782+0000 7fd3dca85640 1 --2- 172.21.15.40:0/3133758165 >> v2:172.21.15.189:3300/0 conn(0x7fd3d8135e30 0x7fd3d8136210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:15:29.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:29.782+0000 7fd3dca85640 1 --2- 172.21.15.40:0/3133758165 >> v2:172.21.15.189:3300/0 conn(0x7fd3d8135e30 0x7fd3d8136210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:15:30.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:30.183+0000 7fd3dd286640 1 --2- 172.21.15.40:0/3133758165 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3d8136750 0x7fd3d80a20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:30.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:30.183+0000 7fd3dd286640 10 monclient(hunting): get_auth_request con 0x7fd3d8136750 auth_method 0 2024-04-22T08:15:30.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:30.183+0000 7fd3dd286640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:30.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:30.183+0000 7fd3dd286640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:30.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:30.183+0000 7fd3dd286640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:15:30.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:30.183+0000 7fd3dd286640 1 -- 172.21.15.40:0/3133758165 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3d8136750 msgr2=0x7fd3d80a20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:15:30.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:30.183+0000 7fd3dd286640 1 -- 172.21.15.40:0/3133758165 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3d8136750 msgr2=0x7fd3d80a20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:15:30.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:30.183+0000 7fd3dd286640 1 --2- 172.21.15.40:0/3133758165 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3d8136750 0x7fd3d80a20b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:15:30.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:30.183+0000 7fd3dd286640 1 --2- 172.21.15.40:0/3133758165 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3d8136750 0x7fd3d80a20b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:15:30.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:30.183+0000 7fd3dca85640 1 --2- 172.21.15.40:0/3133758165 >> v2:172.21.15.189:3300/0 conn(0x7fd3d8135e30 0x7fd3d8136210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:30.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:30.183+0000 7fd3dca85640 10 monclient(hunting): get_auth_request con 0x7fd3d8135e30 auth_method 0 2024-04-22T08:15:30.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:30.183+0000 7fd3dca85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:30.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:30.183+0000 7fd3dca85640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:30.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:30.183+0000 7fd3dca85640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:15:30.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:30.183+0000 7fd3dca85640 1 -- 172.21.15.40:0/3133758165 >> v2:172.21.15.189:3300/0 conn(0x7fd3d8135e30 msgr2=0x7fd3d8136210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-22T08:15:30.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:30.183+0000 7fd3dca85640 1 -- 172.21.15.40:0/3133758165 >> v2:172.21.15.189:3300/0 conn(0x7fd3d8135e30 msgr2=0x7fd3d8136210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:15:30.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:30.183+0000 7fd3dca85640 1 --2- 172.21.15.40:0/3133758165 >> v2:172.21.15.189:3300/0 conn(0x7fd3d8135e30 0x7fd3d8136210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:15:30.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:30.183+0000 7fd3dca85640 1 --2- 172.21.15.40:0/3133758165 >> v2:172.21.15.189:3300/0 conn(0x7fd3d8135e30 0x7fd3d8136210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:15:30.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:30.984+0000 7fd3dd286640 1 --2- 172.21.15.40:0/3133758165 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3d8136750 0x7fd3d80a20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:30.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:30.984+0000 7fd3dd286640 10 monclient(hunting): get_auth_request con 0x7fd3d8136750 auth_method 0 2024-04-22T08:15:30.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:30.984+0000 7fd3dd286640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:30.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:30.984+0000 7fd3dd286640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:30.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:30.984+0000 7fd3dd286640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:15:30.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:30.984+0000 7fd3dd286640 1 -- 172.21.15.40:0/3133758165 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3d8136750 msgr2=0x7fd3d80a20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:15:30.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:30.984+0000 7fd3dd286640 1 -- 172.21.15.40:0/3133758165 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3d8136750 msgr2=0x7fd3d80a20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:15:30.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:30.984+0000 7fd3dd286640 1 --2- 172.21.15.40:0/3133758165 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3d8136750 0x7fd3d80a20b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:15:30.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:30.984+0000 7fd3dd286640 1 --2- 172.21.15.40:0/3133758165 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3d8136750 0x7fd3d80a20b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T08:15:30.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:30.984+0000 7fd3dca85640 1 --2- 172.21.15.40:0/3133758165 >> v2:172.21.15.189:3300/0 conn(0x7fd3d8135e30 0x7fd3d8136210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:30.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:30.984+0000 7fd3dca85640 10 monclient(hunting): get_auth_request con 0x7fd3d8135e30 auth_method 0 2024-04-22T08:15:30.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:30.984+0000 7fd3dca85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:30.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:30.984+0000 7fd3dca85640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:30.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:30.984+0000 7fd3dca85640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:15:30.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:30.984+0000 7fd3dca85640 1 -- 172.21.15.40:0/3133758165 >> v2:172.21.15.189:3300/0 conn(0x7fd3d8135e30 msgr2=0x7fd3d8136210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-22T08:15:30.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:30.984+0000 7fd3dca85640 1 -- 172.21.15.40:0/3133758165 >> v2:172.21.15.189:3300/0 conn(0x7fd3d8135e30 msgr2=0x7fd3d8136210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:15:30.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:30.984+0000 7fd3dca85640 1 --2- 172.21.15.40:0/3133758165 >> v2:172.21.15.189:3300/0 conn(0x7fd3d8135e30 0x7fd3d8136210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:15:30.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:30.984+0000 7fd3dca85640 1 --2- 172.21.15.40:0/3133758165 >> v2:172.21.15.189:3300/0 conn(0x7fd3d8135e30 0x7fd3d8136210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T08:15:32.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.216+0000 7fd3d77fe640 1 -- 172.21.15.40:0/3133758165 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 245288487 0 0) 0x7fd3d809e310 con 0x7fd3d8071c90 2024-04-22T08:15:32.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.216+0000 7fd3d77fe640 10 monclient(hunting): _negotiate 2024-04-22T08:15:32.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.216+0000 7fd3d77fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:32.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.216+0000 7fd3d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:32.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.216+0000 7fd3d77fe640 10 monclient(hunting): my global_id is 24526 2024-04-22T08:15:32.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.216+0000 7fd3d77fe640 1 -- 172.21.15.40:0/3133758165 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3c8003520 con 0x7fd3d8071c90 2024-04-22T08:15:32.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.216+0000 7fd3d77fe640 1 -- 172.21.15.40:0/3133758165 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 588124334 0 0) 0x7fd3c8003520 con 0x7fd3d8071c90 2024-04-22T08:15:32.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.216+0000 7fd3d77fe640 1 -- 172.21.15.40:0/3133758165 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3d8136750 msgr2=0x7fd3d80a20b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:15:32.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.216+0000 7fd3d77fe640 1 --2- 172.21.15.40:0/3133758165 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3d8136750 0x7fd3d80a20b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:32.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.216+0000 7fd3d77fe640 1 -- 172.21.15.40:0/3133758165 >> v2:172.21.15.189:3300/0 conn(0x7fd3d8135e30 msgr2=0x7fd3d8136210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:15:32.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.216+0000 7fd3d77fe640 1 --2- 172.21.15.40:0/3133758165 >> v2:172.21.15.189:3300/0 conn(0x7fd3d8135e30 0x7fd3d8136210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:32.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.216+0000 7fd3d77fe640 10 monclient: _finish_hunting 0 2024-04-22T08:15:32.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.216+0000 7fd3d77fe640 1 monclient: found mon.noname-c-legacy 2024-04-22T08:15:32.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.216+0000 7fd3d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:15:32.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.216+0000 7fd3d77fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.189:6789/0 2024-04-22T08:15:32.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.216+0000 7fd3d77fe640 1 -- 172.21.15.40:0/3133758165 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3d809f410 con 0x7fd3d8071c90 2024-04-22T08:15:32.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.216+0000 7fd3d77fe640 10 monclient: _finish_auth 0 2024-04-22T08:15:32.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.216+0000 7fd3d77fe640 10 monclient: _check_auth_tickets 2024-04-22T08:15:32.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.216+0000 7fd3d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:15:32.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.216+0000 7fd3d77fe640 1 -- 172.21.15.40:0/3133758165 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1087006497 0 0) 0x7fd3c4001970 con 0x7fd3d8071c90 2024-04-22T08:15:32.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.216+0000 7fd3d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:32.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.216+0000 7fd3d77fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-04-22T08:15:32.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.216+0000 7fd3d77fe640 10 monclient: dump: 2024-04-22T08:15:32.220 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:32.220 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:32.220 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:32.220 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:32.220 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:32.220 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:32.220 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:32.221 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:32.221 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:32.221 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:32.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.216+0000 7fd3d77fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] but i'm connected to v1:172.21.15.189:6789/0, reconnecting 2024-04-22T08:15:32.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.216+0000 7fd3d77fe640 10 monclient: _reopen_session rank -1 2024-04-22T08:15:32.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.216+0000 7fd3d77fe640 1 -- 172.21.15.40:0/3133758165 >> v1:172.21.15.189:6789/0 conn(0x7fd3d8071c90 legacy=0x7fd3d80a19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:32.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.216+0000 7fd3d77fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:15:32.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.216+0000 7fd3d77fe640 1 --2- 172.21.15.40:0/3133758165 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3c80039f0 0x7fd3c8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:32.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.216+0000 7fd3d77fe640 10 monclient(hunting): picked mon.a con 0x7fd3c80039f0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:15:32.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.216+0000 7fd3d77fe640 1 --2- 172.21.15.40:0/3133758165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3c80047c0 0x7fd3c8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:32.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.216+0000 7fd3d77fe640 10 monclient(hunting): picked mon.b con 0x7fd3c80047c0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:15:32.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.216+0000 7fd3d77fe640 1 --2- 172.21.15.40:0/3133758165 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3d8136750 0x7fd3c800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:32.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.216+0000 7fd3d77fe640 10 monclient(hunting): picked mon.c con 0x7fd3d8136750 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:15:32.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.216+0000 7fd3d77fe640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:32.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.216+0000 7fd3d77fe640 1 -- 172.21.15.40:0/3133758165 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d809de80 con 0x7fd3c80039f0 2024-04-22T08:15:32.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.216+0000 7fd3d77fe640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:32.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.216+0000 7fd3d77fe640 1 -- 172.21.15.40:0/3133758165 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d809f130 con 0x7fd3c80047c0 2024-04-22T08:15:32.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.216+0000 7fd3d77fe640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:32.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.216+0000 7fd3d77fe640 1 -- 172.21.15.40:0/3133758165 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c8005540 con 0x7fd3d8136750 2024-04-22T08:15:32.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.216+0000 7fd3d77fe640 10 monclient(hunting): _renew_subs 2024-04-22T08:15:32.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.216+0000 7fd3d77fe640 10 monclient(hunting): _finish_auth 0 2024-04-22T08:15:32.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.217+0000 7fd3d7fff640 1 --2- 172.21.15.40:0/3133758165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3c80047c0 0x7fd3c8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:32.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.217+0000 7fd3dd286640 1 --2- 172.21.15.40:0/3133758165 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3d8136750 0x7fd3c800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:32.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.217+0000 7fd3d7fff640 10 monclient(hunting): get_auth_request con 0x7fd3c80047c0 auth_method 0 2024-04-22T08:15:32.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.217+0000 7fd3d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:32.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.217+0000 7fd3d7fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:32.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.217+0000 7fd3d7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:15:32.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.217+0000 7fd3dd286640 10 monclient(hunting): get_auth_request con 0x7fd3d8136750 auth_method 0 2024-04-22T08:15:32.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.217+0000 7fd3dd286640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:32.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.217+0000 7fd3dd286640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:32.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.217+0000 7fd3dd286640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:15:32.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.217+0000 7fd3d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:32.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.217+0000 7fd3d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:32.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.217+0000 7fd3d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-22T08:15:32.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.217+0000 7fd3dd286640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:32.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.217+0000 7fd3dd286640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:32.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.217+0000 7fd3dd286640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-22T08:15:32.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.217+0000 7fd3d7fff640 10 monclient(hunting): handle_auth_done global_id 24526 payload 293 2024-04-22T08:15:32.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.217+0000 7fd3d7fff640 1 -- 172.21.15.40:0/3133758165 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3d8136750 msgr2=0x7fd3c800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:15:32.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.217+0000 7fd3d7fff640 1 --2- 172.21.15.40:0/3133758165 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3d8136750 0x7fd3c800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:32.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.217+0000 7fd3d7fff640 1 -- 172.21.15.40:0/3133758165 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3c80039f0 msgr2=0x7fd3c8003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:15:32.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.217+0000 7fd3d7fff640 1 --2- 172.21.15.40:0/3133758165 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3c80039f0 0x7fd3c8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:32.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.217+0000 7fd3d7fff640 10 monclient: _finish_hunting 0 2024-04-22T08:15:32.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.217+0000 7fd3d7fff640 1 monclient: found mon.b 2024-04-22T08:15:32.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.217+0000 7fd3d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:15:32.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.217+0000 7fd3d7fff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.189:3300/0 2024-04-22T08:15:32.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.217+0000 7fd3d7fff640 1 -- 172.21.15.40:0/3133758165 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3c8003520 con 0x7fd3c80047c0 2024-04-22T08:15:32.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.217+0000 7fd3d7fff640 10 monclient: _finish_auth 0 2024-04-22T08:15:32.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.217+0000 7fd3d7fff640 10 monclient: _check_auth_tickets 2024-04-22T08:15:32.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.217+0000 7fd3d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:15:32.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.217+0000 7fd3dece9640 5 monclient: authenticate success, global_id 24526 2024-04-22T08:15:32.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.217+0000 7fd3dece9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:15:32.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.217+0000 7fd3dd286640 1 --2- 172.21.15.40:0/3133758165 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3d8136750 0x7fd3c800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:15:32.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.221+0000 7fd3d7fff640 1 -- 172.21.15.40:0/3133758165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3c80047c0 msgr2=0x7fd3c8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:32.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.221+0000 7fd3d7fff640 1 --2- 172.21.15.40:0/3133758165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3c80047c0 0x7fd3c8004c40 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd3c4001b50 tx=0x7fd3c80057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ea87b6d93de5f7e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:32.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.221+0000 7fd3d77fe640 1 -- 172.21.15.40:0/3133758165 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c402ca90 con 0x7fd3c80047c0 2024-04-22T08:15:32.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.221+0000 7fd3d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:32.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.221+0000 7fd3d77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:15:32.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.221+0000 7fd3d77fe640 10 monclient: dump: 2024-04-22T08:15:32.231 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:32.231 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:32.231 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:32.232 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:32.232 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:32.232 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:32.232 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:32.232 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:32.232 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:32.232 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:32.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.221+0000 7fd3dece9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:15:32.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.221+0000 7fd3d77fe640 1 -- 172.21.15.40:0/3133758165 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3c402cbf0 con 0x7fd3c80047c0 2024-04-22T08:15:32.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.221+0000 7fd3d77fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:15:32.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.221+0000 7fd3d77fe640 1 -- 172.21.15.40:0/3133758165 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c4007070 con 0x7fd3c80047c0 2024-04-22T08:15:32.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.221+0000 7fd3dece9640 10 monclient: get_monmap_and_config success 2024-04-22T08:15:32.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.221+0000 7fd3dece9640 4 set_mon_vals no callback set 2024-04-22T08:15:32.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.221+0000 7fd3d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:32.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.221+0000 7fd3d77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:15:32.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.221+0000 7fd3d77fe640 10 monclient: dump: 2024-04-22T08:15:32.233 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:32.233 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:32.233 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:32.233 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:32.233 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:32.233 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:32.233 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:32.233 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:32.233 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:32.233 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:32.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.222+0000 7fd3dece9640 10 monclient: shutdown 2024-04-22T08:15:32.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.222+0000 7fd3dece9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:15:32.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.222+0000 7fd3dece9640 1 -- 172.21.15.40:0/3133758165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3c80047c0 msgr2=0x7fd3c8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:32.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.222+0000 7fd3dece9640 1 --2- 172.21.15.40:0/3133758165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3c80047c0 0x7fd3c8004c40 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd3c4001b50 tx=0x7fd3c80057f0 comp rx=0 tx=0).stop 2024-04-22T08:15:32.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.222+0000 7fd3dd286640 1 -- 172.21.15.40:0/3133758165 reap_dead start 2024-04-22T08:15:32.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.222+0000 7fd3dece9640 1 -- 172.21.15.40:0/3133758165 shutdown_connections 2024-04-22T08:15:32.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.222+0000 7fd3dece9640 1 -- 172.21.15.40:0/3133758165 >> 172.21.15.40:0/3133758165 conn(0x7fd3d80a65c0 msgr2=0x7fd3d80a2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:15:32.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.222+0000 7fd3dece9640 1 -- 172.21.15.40:0/3133758165 shutdown_connections 2024-04-22T08:15:32.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.222+0000 7fd3dece9640 1 -- 172.21.15.40:0/3133758165 wait complete. 2024-04-22T08:15:32.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.223+0000 7fd3dece9640 10 monclient: build_initial_monmap 2024-04-22T08:15:32.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.223+0000 7fd3dece9640 10 monclient: monmap: 2024-04-22T08:15:32.234 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:15:32.234 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:15:32.234 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:15:32.234 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:15:32.234 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:15:32.235 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:32.235 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:15:32.235 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:15:32.235 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:15:32.235 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:32.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.224+0000 7fd3dece9640 1 Processor -- start 2024-04-22T08:15:32.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.224+0000 7fd3dece9640 1 -- start start 2024-04-22T08:15:32.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.224+0000 7fd3dece9640 10 monclient: init 2024-04-22T08:15:32.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.224+0000 7fd3dece9640 10 monclient: _reopen_session rank -1 2024-04-22T08:15:32.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.224+0000 7fd3dece9640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:15:32.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.224+0000 7fd3dece9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3c80047c0 0x7fd3d81c7130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:32.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.224+0000 7fd3dece9640 10 monclient(hunting): picked mon.noname-a con 0x7fd3c80047c0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:15:32.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.224+0000 7fd3dece9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3d8134130 0x7fd3d8134570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:32.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.224+0000 7fd3dece9640 10 monclient(hunting): picked mon.noname-b con 0x7fd3d8134130 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:15:32.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.224+0000 7fd3dece9640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3d81ca690 0x7fd3d81cab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:32.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.224+0000 7fd3dece9640 10 monclient(hunting): picked mon.noname-c con 0x7fd3d81ca690 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:15:32.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.224+0000 7fd3dece9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:32.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.224+0000 7fd3dece9640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d8121830 con 0x7fd3c80047c0 2024-04-22T08:15:32.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.224+0000 7fd3dece9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:32.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.224+0000 7fd3dece9640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d8072150 con 0x7fd3d8134130 2024-04-22T08:15:32.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.224+0000 7fd3dece9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:32.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.224+0000 7fd3dece9640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d81c7b90 con 0x7fd3d81ca690 2024-04-22T08:15:32.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.224+0000 7fd3dece9640 10 monclient(hunting): _renew_subs 2024-04-22T08:15:32.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.224+0000 7fd3dece9640 10 monclient(hunting): authenticate will time out at 1006.218741s 2024-04-22T08:15:32.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.225+0000 7fd3dca85640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3c80047c0 0x7fd3d81c7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:32.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.225+0000 7fd3dd286640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3d81ca690 0x7fd3d81cab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:32.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.225+0000 7fd3dca85640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3c80047c0 0x7fd3d81c7130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58796/0 (socket says 172.21.15.40:58796) 2024-04-22T08:15:32.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.225+0000 7fd3dca85640 1 -- 172.21.15.40:0/3033340345 learned_addr learned my addr 172.21.15.40:0/3033340345 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:15:32.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.225+0000 7fd3dca85640 10 monclient(hunting): get_auth_request con 0x7fd3c80047c0 auth_method 0 2024-04-22T08:15:32.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.225+0000 7fd3dca85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:32.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.225+0000 7fd3dca85640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:32.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.225+0000 7fd3dca85640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:32.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.225+0000 7fd3dd286640 10 monclient(hunting): get_auth_request con 0x7fd3d81ca690 auth_method 0 2024-04-22T08:15:32.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.225+0000 7fd3dd286640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:32.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.225+0000 7fd3dd286640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:32.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.225+0000 7fd3dd286640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:32.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.225+0000 7fd3d7fff640 1 --2- 172.21.15.40:0/3033340345 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3d8134130 0x7fd3d8134570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:32.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.225+0000 7fd3dca85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:32.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.225+0000 7fd3dca85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:32.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.225+0000 7fd3dca85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:32.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.225+0000 7fd3d7fff640 10 monclient(hunting): get_auth_request con 0x7fd3d8134130 auth_method 0 2024-04-22T08:15:32.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.225+0000 7fd3d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:32.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.225+0000 7fd3d7fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:32.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.225+0000 7fd3d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:32.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.225+0000 7fd3dd286640 1 -- 172.21.15.40:0/3033340345 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3d81ca690 msgr2=0x7fd3d81cab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-22T08:15:32.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.225+0000 7fd3dd286640 1 -- 172.21.15.40:0/3033340345 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3d81ca690 msgr2=0x7fd3d81cab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:15:32.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.225+0000 7fd3dd286640 1 --2- 172.21.15.40:0/3033340345 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3d81ca690 0x7fd3d81cab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:15:32.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.225+0000 7fd3dd286640 1 --2- 172.21.15.40:0/3033340345 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3d81ca690 0x7fd3d81cab10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:15:32.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.225+0000 7fd3d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:32.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.225+0000 7fd3d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:32.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.225+0000 7fd3d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:32.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.225+0000 7fd3dca85640 10 monclient(hunting): handle_auth_done global_id 34320 payload 824 2024-04-22T08:15:32.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.225+0000 7fd3dca85640 1 -- 172.21.15.40:0/3033340345 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3d81ca690 msgr2=0x7fd3d81cab10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:15:32.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.225+0000 7fd3dca85640 1 --2- 172.21.15.40:0/3033340345 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3d81ca690 0x7fd3d81cab10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:32.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.225+0000 7fd3dca85640 1 -- 172.21.15.40:0/3033340345 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3d8134130 msgr2=0x7fd3d8134570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:32.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.225+0000 7fd3dca85640 1 --2- 172.21.15.40:0/3033340345 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3d8134130 0x7fd3d8134570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:32.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.225+0000 7fd3dca85640 10 monclient: _finish_hunting 0 2024-04-22T08:15:32.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.225+0000 7fd3dca85640 1 monclient: found mon.noname-a 2024-04-22T08:15:32.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.225+0000 7fd3dca85640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:15:32.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.225+0000 7fd3dca85640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:15:32.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.225+0000 7fd3dca85640 1 -- 172.21.15.40:0/3033340345 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3d8134ef0 con 0x7fd3c80047c0 2024-04-22T08:15:32.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.227+0000 7fd3dca85640 1 -- 172.21.15.40:0/3033340345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3c80047c0 msgr2=0x7fd3d81c7130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:32.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.227+0000 7fd3dca85640 1 --2- 172.21.15.40:0/3033340345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3c80047c0 0x7fd3d81c7130 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc013950 tx=0x7fd3cc013e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:32.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.228+0000 7fd3d57fa640 1 -- 172.21.15.40:0/3033340345 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3cc011c60 con 0x7fd3c80047c0 2024-04-22T08:15:32.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.228+0000 7fd3d57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:32.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.228+0000 7fd3d57fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:15:32.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.228+0000 7fd3d57fa640 10 monclient: dump: 2024-04-22T08:15:32.240 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:32.240 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:32.240 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:32.240 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:32.240 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:32.240 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:32.240 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:32.241 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:32.241 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:32.241 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:32.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.228+0000 7fd3d57fa640 10 monclient: _finish_auth 0 2024-04-22T08:15:32.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.228+0000 7fd3d57fa640 10 monclient: _check_auth_tickets 2024-04-22T08:15:32.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.228+0000 7fd3d57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:15:32.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.228+0000 7fd3d57fa640 1 -- 172.21.15.40:0/3033340345 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3cc011dc0 con 0x7fd3c80047c0 2024-04-22T08:15:32.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.228+0000 7fd3dece9640 5 monclient: authenticate success, global_id 34320 2024-04-22T08:15:32.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.228+0000 7fd3dece9640 10 monclient: _renew_subs 2024-04-22T08:15:32.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.228+0000 7fd3dece9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:32.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.228+0000 7fd3dece9640 1 -- 172.21.15.40:0/3033340345 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3d8121270 con 0x7fd3c80047c0 2024-04-22T08:15:32.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.228+0000 7fd3d57fa640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:15:32.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.228+0000 7fd3d57fa640 1 -- 172.21.15.40:0/3033340345 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3cc0166a0 con 0x7fd3c80047c0 2024-04-22T08:15:32.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.228+0000 7fd3d57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:32.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.228+0000 7fd3d57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:15:32.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.228+0000 7fd3d57fa640 10 monclient: dump: 2024-04-22T08:15:32.242 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:32.242 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:32.242 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:32.242 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:32.242 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:32.242 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:32.242 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:32.242 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:32.242 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:32.243 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:32.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.228+0000 7fd3d6ffd640 4 set_mon_vals no callback set 2024-04-22T08:15:32.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.228+0000 7fd3dece9640 10 monclient: _renew_subs 2024-04-22T08:15:32.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.228+0000 7fd3dece9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:32.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.228+0000 7fd3dece9640 1 -- 172.21.15.40:0/3033340345 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3d80717e0 con 0x7fd3c80047c0 2024-04-22T08:15:32.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.229+0000 7fd3d57fa640 1 -- 172.21.15.40:0/3033340345 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fd3cc016800 con 0x7fd3c80047c0 2024-04-22T08:15:32.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.229+0000 7fd3d57fa640 1 --2- 172.21.15.40:0/3033340345 >> v2:172.21.15.189:6800/30168 conn(0x7fd3bc037830 0x7fd3bc039cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:32.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.229+0000 7fd3d57fa640 1 -- 172.21.15.40:0/3033340345 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 3898+0+0 (secure 0 0 0) 0x7fd3cc04ab50 con 0x7fd3c80047c0 2024-04-22T08:15:32.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.229+0000 7fd3dece9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:32.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.229+0000 7fd3dece9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:32.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.229+0000 7fd3dece9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:32.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.229+0000 7fd3dece9640 1 -- 172.21.15.40:0/3033340345 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3d804e680 con 0x7fd3c80047c0 2024-04-22T08:15:32.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.229+0000 7fd3d7fff640 1 --2- 172.21.15.40:0/3033340345 >> v2:172.21.15.189:6800/30168 conn(0x7fd3bc037830 0x7fd3bc039cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:32.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.229+0000 7fd3d7fff640 10 monclient: get_auth_request con 0x7fd3bc037830 auth_method 0 2024-04-22T08:15:32.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.232+0000 7fd3d7fff640 1 --2- 172.21.15.40:0/3033340345 >> v2:172.21.15.189:6800/30168 conn(0x7fd3bc037830 0x7fd3bc039cf0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd3d80a2a60 tx=0x7fd3c4009030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:32.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.234+0000 7fd3d57fa640 1 -- 172.21.15.40:0/3033340345 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7fd3cc026020 con 0x7fd3c80047c0 2024-04-22T08:15:32.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.234+0000 7fd3d57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:32.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.234+0000 7fd3d57fa640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:15:32.383 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35899]: starting. 2024-04-22T08:15:32.532 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35899]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-35898 2024-04-22T08:15:32.532 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35899]: created object 0... 2024-04-22T08:15:32.532 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35899]: created object 25... 2024-04-22T08:15:32.533 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35899]: created object 49... 2024-04-22T08:15:32.533 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35899]: finishing. 2024-04-22T08:15:32.533 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[35899]: shutting down. 2024-04-22T08:15:32.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.504+0000 7fd3dece9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:15:32.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.504+0000 7fd3dece9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:15:32.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.504+0000 7fd3dece9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:32.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.504+0000 7fd3dece9640 1 -- 172.21.15.40:0/3033340345 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd3d804e9b0 con 0x7fd3c80047c0 2024-04-22T08:15:32.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.505+0000 7fd3d57fa640 1 -- 172.21.15.40:0/3033340345 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1537 (secure 0 0 0) 0x7fd3cc0209a0 con 0x7fd3c80047c0 2024-04-22T08:15:32.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.505+0000 7fd3d57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:15:32.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.505+0000 7fd3d57fa640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:15:32.535 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":34,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:15:32.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.507+0000 7fd3dece9640 1 -- 172.21.15.40:0/3033340345 >> v2:172.21.15.189:6800/30168 conn(0x7fd3bc037830 msgr2=0x7fd3bc039cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:32.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.507+0000 7fd3dece9640 1 --2- 172.21.15.40:0/3033340345 >> v2:172.21.15.189:6800/30168 conn(0x7fd3bc037830 0x7fd3bc039cf0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd3d80a2a60 tx=0x7fd3c4009030 comp rx=0 tx=0).stop 2024-04-22T08:15:32.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.507+0000 7fd3dece9640 10 monclient: shutdown 2024-04-22T08:15:32.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.507+0000 7fd3dece9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:15:32.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.507+0000 7fd3dece9640 1 -- 172.21.15.40:0/3033340345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3c80047c0 msgr2=0x7fd3d81c7130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:32.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.507+0000 7fd3dece9640 1 --2- 172.21.15.40:0/3033340345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3c80047c0 0x7fd3d81c7130 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc013950 tx=0x7fd3cc013e10 comp rx=0 tx=0).stop 2024-04-22T08:15:32.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.507+0000 7fd3dece9640 1 -- 172.21.15.40:0/3033340345 shutdown_connections 2024-04-22T08:15:32.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.507+0000 7fd3dece9640 1 --2- 172.21.15.40:0/3033340345 >> v2:172.21.15.189:6800/30168 conn(0x7fd3bc037830 0x7fd3bc039cf0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:32.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.507+0000 7fd3dece9640 1 --2- 172.21.15.40:0/3033340345 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3d8134130 0x7fd3d8134570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:32.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.507+0000 7fd3dece9640 1 --2- 172.21.15.40:0/3033340345 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3d81ca690 0x7fd3d81cab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:32.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.507+0000 7fd3dece9640 1 --2- 172.21.15.40:0/3033340345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3c80047c0 0x7fd3d81c7130 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:32.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.507+0000 7fd3dece9640 1 -- 172.21.15.40:0/3033340345 >> 172.21.15.40:0/3033340345 conn(0x7fd3d80a65c0 msgr2=0x7fd3d80a3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:15:32.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.507+0000 7fd3dece9640 1 -- 172.21.15.40:0/3033340345 shutdown_connections 2024-04-22T08:15:32.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:32.507+0000 7fd3dece9640 1 -- 172.21.15.40:0/3033340345 wait complete. 2024-04-22T08:15:33.737 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[35900]: starting. 2024-04-22T08:15:33.738 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[35900]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-35898 2024-04-22T08:15:33.738 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[35900]: shutting down. 2024-04-22T08:15:35.539 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:15:35.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.650+0000 7f822342f640 10 monclient: get_monmap_and_config 2024-04-22T08:15:35.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.650+0000 7f822342f640 10 monclient: build_initial_monmap 2024-04-22T08:15:35.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.650+0000 7f822342f640 10 monclient: monmap: 2024-04-22T08:15:35.651 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:15:35.651 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:35.651 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:15:35.651394+0000 2024-04-22T08:15:35.651 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:15:35.651394+0000 2024-04-22T08:15:35.651 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:15:35.651 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:35.651 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:15:35.651 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:15:35.651 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:15:35.651 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:15:35.652 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:15:35.652 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:35.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.651+0000 7f822342f640 1 Processor -- start 2024-04-22T08:15:35.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.651+0000 7f822342f640 1 -- start start 2024-04-22T08:15:35.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.651+0000 7f822342f640 10 monclient: init 2024-04-22T08:15:35.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.651+0000 7f822342f640 10 monclient: _reopen_session rank -1 2024-04-22T08:15:35.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.651+0000 7f822342f640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-04-22T08:15:35.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.651+0000 7f822342f640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f821c132b00 0x7f821c09af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:35.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.651+0000 7f822342f640 10 monclient(hunting): picked mon.noname-b con 0x7f821c132b00 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:15:35.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.651+0000 7f822342f640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f821c132ee0 0x7f821c0a0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:35.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.651+0000 7f822342f640 10 monclient(hunting): picked mon.noname-c con 0x7f821c132ee0 addr v2:172.21.15.189:3300/0 2024-04-22T08:15:35.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.651+0000 7f822342f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f821c133320 addr v1:172.21.15.189:6789/0 2024-04-22T08:15:35.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.651+0000 7f822342f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f821c09b4b0 con 0x7f821c133320 2024-04-22T08:15:35.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.651+0000 7f822342f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:35.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.651+0000 7f822342f640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f821c06c840 con 0x7f821c132ee0 2024-04-22T08:15:35.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.651+0000 7f822342f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:35.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.651+0000 7f822342f640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f821c09a220 con 0x7f821c132b00 2024-04-22T08:15:35.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.651+0000 7f822342f640 10 monclient(hunting): _renew_subs 2024-04-22T08:15:35.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.651+0000 7f822342f640 10 monclient(hunting): authenticate will time out at 1009.645696s 2024-04-22T08:15:35.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.652+0000 7f82211cb640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f821c132b00 0x7f821c09af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:35.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.652+0000 7f82211cb640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f821c132b00 0x7f821c09af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:60760/0 (socket says 172.21.15.40:60760) 2024-04-22T08:15:35.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.652+0000 7f82211cb640 1 -- 172.21.15.40:0/513271624 learned_addr learned my addr 172.21.15.40:0/513271624 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:15:35.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.652+0000 7f82211cb640 10 monclient(hunting): get_auth_request con 0x7f821c132b00 auth_method 0 2024-04-22T08:15:35.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.652+0000 7f82211cb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:35.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.652+0000 7f82211cb640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:35.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.652+0000 7f82211cb640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:35.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.652+0000 7f82211cb640 1 -- 172.21.15.40:0/513271624 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f821c132b00 msgr2=0x7f821c09af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:15:35.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.652+0000 7f82211cb640 1 -- 172.21.15.40:0/513271624 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f821c132b00 msgr2=0x7f821c09af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:15:35.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.652+0000 7f82211cb640 1 --2- 172.21.15.40:0/513271624 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f821c132b00 0x7f821c09af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:15:35.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.652+0000 7f82211cb640 1 --2- 172.21.15.40:0/513271624 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f821c132b00 0x7f821c09af70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:15:35.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.652+0000 7f82209ca640 1 --2- 172.21.15.40:0/513271624 >> v2:172.21.15.189:3300/0 conn(0x7f821c132ee0 0x7f821c0a0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:35.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.652+0000 7f82209ca640 10 monclient(hunting): get_auth_request con 0x7f821c132ee0 auth_method 0 2024-04-22T08:15:35.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.652+0000 7f82209ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:35.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.652+0000 7f82209ca640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:35.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.652+0000 7f82209ca640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:35.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.652+0000 7f82209ca640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:35.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.652+0000 7f82209ca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:35.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.652+0000 7f82209ca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:35.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.652+0000 7f82209ca640 10 monclient(hunting): handle_auth_done global_id 24550 payload 274 2024-04-22T08:15:35.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.652+0000 7f82209ca640 1 -- 172.21.15.40:0/513271624 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f821c132b00 msgr2=0x7f821c09af70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:15:35.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.652+0000 7f82209ca640 1 --2- 172.21.15.40:0/513271624 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f821c132b00 0x7f821c09af70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:35.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.652+0000 7f82209ca640 1 -- 172.21.15.40:0/513271624 >> v1:172.21.15.189:6789/0 conn(0x7f821c133320 legacy=0x7f821c0a0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:15:35.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.652+0000 7f82209ca640 10 monclient: _finish_hunting 0 2024-04-22T08:15:35.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.652+0000 7f82209ca640 1 monclient: found mon.noname-c 2024-04-22T08:15:35.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.652+0000 7f82209ca640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:15:35.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.652+0000 7f82209ca640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.189:3300/0 2024-04-22T08:15:35.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.652+0000 7f82209ca640 1 -- 172.21.15.40:0/513271624 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f821c09a500 con 0x7f821c132ee0 2024-04-22T08:15:35.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.655+0000 7f82209ca640 1 -- 172.21.15.40:0/513271624 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f821c132ee0 msgr2=0x7f821c0a0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:35.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.655+0000 7f82209ca640 1 --2- 172.21.15.40:0/513271624 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f821c132ee0 0x7f821c0a0a00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f82100097a0 tx=0x7f821002b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=edd5ff774199837e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:35.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.656+0000 7f820ffff640 1 -- 172.21.15.40:0/513271624 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8210030920 con 0x7f821c132ee0 2024-04-22T08:15:35.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.656+0000 7f820ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:35.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.656+0000 7f820ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:15:35.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.656+0000 7f820ffff640 10 monclient: dump: 2024-04-22T08:15:35.664 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:35.664 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:35.664 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:35.664 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:35.664 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:35.664 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:35.664 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:35.664 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:35.664 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:35.664 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:35.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.656+0000 7f820ffff640 10 monclient: _finish_auth 0 2024-04-22T08:15:35.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.656+0000 7f820ffff640 10 monclient: _check_auth_tickets 2024-04-22T08:15:35.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.656+0000 7f820ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:15:35.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.656+0000 7f820ffff640 1 -- 172.21.15.40:0/513271624 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f821002e050 con 0x7f821c132ee0 2024-04-22T08:15:35.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.656+0000 7f820ffff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:15:35.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.656+0000 7f820ffff640 1 -- 172.21.15.40:0/513271624 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8210030ca0 con 0x7f821c132ee0 2024-04-22T08:15:35.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.656+0000 7f820ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:35.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.656+0000 7f820ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:15:35.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.656+0000 7f820ffff640 10 monclient: dump: 2024-04-22T08:15:35.665 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:35.665 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:35.665 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:35.665 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:35.666 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:35.666 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:35.666 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:35.666 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:35.666 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:35.666 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:35.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.656+0000 7f822342f640 5 monclient: authenticate success, global_id 24550 2024-04-22T08:15:35.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.656+0000 7f822342f640 10 monclient: get_monmap_and_config success 2024-04-22T08:15:35.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.656+0000 7f822342f640 4 set_mon_vals no callback set 2024-04-22T08:15:35.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.656+0000 7f822342f640 10 monclient: shutdown 2024-04-22T08:15:35.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.656+0000 7f822342f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:15:35.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.656+0000 7f822342f640 1 -- 172.21.15.40:0/513271624 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f821c132ee0 msgr2=0x7f821c0a0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:35.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.656+0000 7f822342f640 1 --2- 172.21.15.40:0/513271624 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f821c132ee0 0x7f821c0a0a00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f82100097a0 tx=0x7f821002b710 comp rx=0 tx=0).stop 2024-04-22T08:15:35.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.656+0000 7f822342f640 1 -- 172.21.15.40:0/513271624 shutdown_connections 2024-04-22T08:15:35.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.656+0000 7f822342f640 1 --2- 172.21.15.40:0/513271624 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f821c132ee0 0x7f821c0a0a00 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:35.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.656+0000 7f822342f640 1 --2- 172.21.15.40:0/513271624 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f821c132b00 0x7f821c09af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:35.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.656+0000 7f822342f640 1 -- 172.21.15.40:0/513271624 >> 172.21.15.40:0/513271624 conn(0x7f821c0a5960 msgr2=0x7f821c0a1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:15:35.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.657+0000 7f822342f640 1 -- 172.21.15.40:0/513271624 shutdown_connections 2024-04-22T08:15:35.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.657+0000 7f822342f640 1 -- 172.21.15.40:0/513271624 wait complete. 2024-04-22T08:15:35.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.657+0000 7f822342f640 10 monclient: build_initial_monmap 2024-04-22T08:15:35.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.657+0000 7f822342f640 10 monclient: monmap: 2024-04-22T08:15:35.668 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:15:35.668 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:15:35.668 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:15:35.668 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:15:35.668 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:15:35.668 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:35.668 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:15:35.668 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:15:35.668 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:15:35.668 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:35.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.657+0000 7f822342f640 1 Processor -- start 2024-04-22T08:15:35.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.657+0000 7f822342f640 1 -- start start 2024-04-22T08:15:35.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.657+0000 7f822342f640 10 monclient: init 2024-04-22T08:15:35.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f822342f640 10 monclient: _reopen_session rank -1 2024-04-22T08:15:35.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f822342f640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-22T08:15:35.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f822342f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f821c132b00 0x7f821c1caae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:35.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f822342f640 10 monclient(hunting): picked mon.noname-b con 0x7f821c132b00 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:15:35.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f822342f640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f821c132ee0 0x7f821c1cb020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:35.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f822342f640 10 monclient(hunting): picked mon.noname-c con 0x7f821c132ee0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:15:35.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f822342f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f821c133320 0x7f821c121fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:35.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f822342f640 10 monclient(hunting): picked mon.noname-a con 0x7f821c133320 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:15:35.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f822342f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:35.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f822342f640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f821c09ee30 con 0x7f821c133320 2024-04-22T08:15:35.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f822342f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:35.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f822342f640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f821c1268a0 con 0x7f821c132b00 2024-04-22T08:15:35.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f822342f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:35.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f822342f640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f821c125be0 con 0x7f821c132ee0 2024-04-22T08:15:35.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f822342f640 10 monclient(hunting): _renew_subs 2024-04-22T08:15:35.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f822342f640 10 monclient(hunting): authenticate will time out at 1009.652200s 2024-04-22T08:15:35.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f82209ca640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f821c132ee0 0x7f821c1cb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:35.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f82209ca640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f821c132ee0 0x7f821c1cb020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:60776/0 (socket says 172.21.15.40:60776) 2024-04-22T08:15:35.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f82209ca640 1 -- 172.21.15.40:0/1507096144 learned_addr learned my addr 172.21.15.40:0/1507096144 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:15:35.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f82209ca640 10 monclient(hunting): get_auth_request con 0x7f821c132ee0 auth_method 0 2024-04-22T08:15:35.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f82209ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:35.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f82209ca640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:35.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f82219cc640 1 --2- 172.21.15.40:0/1507096144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f821c133320 0x7f821c121fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:35.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f82209ca640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:35.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f82219cc640 10 monclient(hunting): get_auth_request con 0x7f821c133320 auth_method 0 2024-04-22T08:15:35.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f82219cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:35.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f82219cc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:35.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f82219cc640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:35.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f82209ca640 1 -- 172.21.15.40:0/1507096144 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f821c132ee0 msgr2=0x7f821c1cb020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-22T08:15:35.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f82209ca640 1 -- 172.21.15.40:0/1507096144 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f821c132ee0 msgr2=0x7f821c1cb020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:15:35.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f82209ca640 1 --2- 172.21.15.40:0/1507096144 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f821c132ee0 0x7f821c1cb020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:15:35.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f82219cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:35.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f82219cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:35.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f82219cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:35.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f82209ca640 1 --2- 172.21.15.40:0/1507096144 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f821c132ee0 0x7f821c1cb020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:15:35.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f82211cb640 1 --2- 172.21.15.40:0/1507096144 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f821c132b00 0x7f821c1caae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:35.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f82211cb640 10 monclient(hunting): get_auth_request con 0x7f821c132b00 auth_method 0 2024-04-22T08:15:35.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f82211cb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:35.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f82211cb640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:35.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f82211cb640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:35.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f82211cb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:35.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f82211cb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:35.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f82211cb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:35.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f82219cc640 10 monclient(hunting): handle_auth_done global_id 34335 payload 824 2024-04-22T08:15:35.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f82219cc640 1 -- 172.21.15.40:0/1507096144 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f821c132ee0 msgr2=0x7f821c1cb020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:15:35.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f82219cc640 1 --2- 172.21.15.40:0/1507096144 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f821c132ee0 0x7f821c1cb020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:35.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f82219cc640 1 -- 172.21.15.40:0/1507096144 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f821c132b00 msgr2=0x7f821c1caae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:35.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f82219cc640 1 --2- 172.21.15.40:0/1507096144 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f821c132b00 0x7f821c1caae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:35.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f82219cc640 10 monclient: _finish_hunting 0 2024-04-22T08:15:35.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.658+0000 7f82219cc640 1 monclient: found mon.noname-a 2024-04-22T08:15:35.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.659+0000 7f82219cc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:15:35.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.659+0000 7f82219cc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:15:35.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.659+0000 7f82219cc640 1 -- 172.21.15.40:0/1507096144 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8210009400 con 0x7f821c133320 2024-04-22T08:15:35.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.661+0000 7f82219cc640 1 -- 172.21.15.40:0/1507096144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f821c133320 msgr2=0x7f821c121fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:35.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.661+0000 7f82219cc640 1 --2- 172.21.15.40:0/1507096144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f821c133320 0x7f821c121fd0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f821400c7b0 tx=0x7f821400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:35.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.661+0000 7f820dffb640 1 -- 172.21.15.40:0/1507096144 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f821400ac60 con 0x7f821c133320 2024-04-22T08:15:35.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.661+0000 7f820dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:35.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.661+0000 7f820dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:15:35.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.661+0000 7f820dffb640 10 monclient: dump: 2024-04-22T08:15:35.676 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:35.676 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:35.676 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:35.676 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:35.676 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:35.676 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:35.676 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:35.676 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:35.676 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:35.676 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:35.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.661+0000 7f820dffb640 10 monclient: _finish_auth 0 2024-04-22T08:15:35.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.661+0000 7f820dffb640 10 monclient: _check_auth_tickets 2024-04-22T08:15:35.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.661+0000 7f820dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:15:35.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.661+0000 7f822342f640 5 monclient: authenticate success, global_id 34335 2024-04-22T08:15:35.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.661+0000 7f820dffb640 1 -- 172.21.15.40:0/1507096144 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f821400adc0 con 0x7f821c133320 2024-04-22T08:15:35.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.661+0000 7f820dffb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:15:35.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.661+0000 7f822342f640 10 monclient: _renew_subs 2024-04-22T08:15:35.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.661+0000 7f820dffb640 1 -- 172.21.15.40:0/1507096144 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8214012660 con 0x7f821c133320 2024-04-22T08:15:35.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.661+0000 7f822342f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:35.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.661+0000 7f822342f640 1 -- 172.21.15.40:0/1507096144 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f821c124730 con 0x7f821c133320 2024-04-22T08:15:35.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.661+0000 7f820dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:35.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.661+0000 7f820dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:15:35.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.661+0000 7f820dffb640 10 monclient: dump: 2024-04-22T08:15:35.678 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:35.678 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:35.678 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:35.678 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:35.678 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:35.679 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:35.679 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:35.679 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:35.679 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:35.679 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:35.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.661+0000 7f820f7fe640 4 set_mon_vals no callback set 2024-04-22T08:15:35.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.661+0000 7f822342f640 10 monclient: _renew_subs 2024-04-22T08:15:35.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.661+0000 7f822342f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:35.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.661+0000 7f822342f640 1 -- 172.21.15.40:0/1507096144 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f821c1244b0 con 0x7f821c133320 2024-04-22T08:15:35.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.662+0000 7f820dffb640 1 -- 172.21.15.40:0/1507096144 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f8214012840 con 0x7f821c133320 2024-04-22T08:15:35.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.662+0000 7f820dffb640 1 --2- 172.21.15.40:0/1507096144 >> v2:172.21.15.189:6800/30168 conn(0x7f8200037820 0x7f8200039ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:35.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.662+0000 7f820dffb640 1 -- 172.21.15.40:0/1507096144 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f821404a440 con 0x7f821c133320 2024-04-22T08:15:35.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.662+0000 7f822342f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:35.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.662+0000 7f822342f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:35.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.662+0000 7f822342f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:35.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.662+0000 7f822342f640 1 -- 172.21.15.40:0/1507096144 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f821c0a1600 con 0x7f821c133320 2024-04-22T08:15:35.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.662+0000 7f82211cb640 1 --2- 172.21.15.40:0/1507096144 >> v2:172.21.15.189:6800/30168 conn(0x7f8200037820 0x7f8200039ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:35.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.662+0000 7f82211cb640 10 monclient: get_auth_request con 0x7f8200037820 auth_method 0 2024-04-22T08:15:35.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.665+0000 7f82211cb640 1 --2- 172.21.15.40:0/1507096144 >> v2:172.21.15.189:6800/30168 conn(0x7f8200037820 0x7f8200039ce0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8218009100 tx=0x7f8218008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:35.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.667+0000 7f820dffb640 1 -- 172.21.15.40:0/1507096144 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f8214016020 con 0x7f821c133320 2024-04-22T08:15:35.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.667+0000 7f820dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:35.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.667+0000 7f820dffb640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:15:35.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.926+0000 7f822342f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:15:35.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.926+0000 7f822342f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:15:35.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.926+0000 7f822342f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:35.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.926+0000 7f822342f640 1 -- 172.21.15.40:0/1507096144 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f821c09b8c0 con 0x7f821c133320 2024-04-22T08:15:35.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.927+0000 7f820dffb640 1 -- 172.21.15.40:0/1507096144 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f82140106f0 con 0x7f821c133320 2024-04-22T08:15:35.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.927+0000 7f820dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:15:35.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.927+0000 7f820dffb640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:15:35.927 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":36,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:15:35.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.929+0000 7f822342f640 1 -- 172.21.15.40:0/1507096144 >> v2:172.21.15.189:6800/30168 conn(0x7f8200037820 msgr2=0x7f8200039ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:35.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.929+0000 7f822342f640 1 --2- 172.21.15.40:0/1507096144 >> v2:172.21.15.189:6800/30168 conn(0x7f8200037820 0x7f8200039ce0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8218009100 tx=0x7f8218008040 comp rx=0 tx=0).stop 2024-04-22T08:15:35.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.929+0000 7f822342f640 10 monclient: shutdown 2024-04-22T08:15:35.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.929+0000 7f822342f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:15:35.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.929+0000 7f822342f640 1 -- 172.21.15.40:0/1507096144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f821c133320 msgr2=0x7f821c121fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:35.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.929+0000 7f822342f640 1 --2- 172.21.15.40:0/1507096144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f821c133320 0x7f821c121fd0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f821400c7b0 tx=0x7f821400cc70 comp rx=0 tx=0).stop 2024-04-22T08:15:35.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.929+0000 7f822342f640 1 -- 172.21.15.40:0/1507096144 shutdown_connections 2024-04-22T08:15:35.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.929+0000 7f822342f640 1 --2- 172.21.15.40:0/1507096144 >> v2:172.21.15.189:6800/30168 conn(0x7f8200037820 0x7f8200039ce0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:35.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.929+0000 7f822342f640 1 --2- 172.21.15.40:0/1507096144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f821c133320 0x7f821c121fd0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:35.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.929+0000 7f822342f640 1 --2- 172.21.15.40:0/1507096144 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f821c132ee0 0x7f821c1cb020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:35.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.929+0000 7f822342f640 1 --2- 172.21.15.40:0/1507096144 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f821c132b00 0x7f821c1caae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:35.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.929+0000 7f822342f640 1 -- 172.21.15.40:0/1507096144 >> 172.21.15.40:0/1507096144 conn(0x7f821c0a5960 msgr2=0x7f821c09c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:15:35.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.929+0000 7f822342f640 1 -- 172.21.15.40:0/1507096144 shutdown_connections 2024-04-22T08:15:35.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:35.929+0000 7f822342f640 1 -- 172.21.15.40:0/1507096144 wait complete. 2024-04-22T08:15:35.939 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-22T08:15:35.939 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-22T08:15:35.939 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-22T08:15:36.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.039+0000 7fe0e9bfd640 10 monclient: get_monmap_and_config 2024-04-22T08:15:36.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.039+0000 7fe0e9bfd640 10 monclient: build_initial_monmap 2024-04-22T08:15:36.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.039+0000 7fe0e9bfd640 10 monclient: monmap: 2024-04-22T08:15:36.039 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:15:36.039 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:36.039 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:15:36.040120+0000 2024-04-22T08:15:36.039 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:15:36.040120+0000 2024-04-22T08:15:36.039 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:15:36.039 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:36.039 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:15:36.039 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:15:36.039 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:15:36.039 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:15:36.040 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:15:36.040 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:36.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.040+0000 7fe0e9bfd640 1 Processor -- start 2024-04-22T08:15:36.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.040+0000 7fe0e9bfd640 1 -- start start 2024-04-22T08:15:36.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.040+0000 7fe0e9bfd640 10 monclient: init 2024-04-22T08:15:36.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.040+0000 7fe0e9bfd640 10 monclient: _reopen_session rank -1 2024-04-22T08:15:36.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.040+0000 7fe0e9bfd640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-04-22T08:15:36.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.040+0000 7fe0e9bfd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe0e40a5a10 addr v1:172.21.15.40:6789/0 2024-04-22T08:15:36.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.040+0000 7fe0e9bfd640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe0e40a32b0 0x7fe0e413af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:36.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.040+0000 7fe0e9bfd640 10 monclient(hunting): picked mon.noname-c con 0x7fe0e40a32b0 addr v2:172.21.15.189:3300/0 2024-04-22T08:15:36.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.040+0000 7fe0e9bfd640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe0e413b480 0x7fe0e413d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:36.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.040+0000 7fe0e9bfd640 10 monclient(hunting): picked mon.noname-a con 0x7fe0e413b480 addr v2:172.21.15.40:3300/0 2024-04-22T08:15:36.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.040+0000 7fe0e9bfd640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0e409fe80 con 0x7fe0e40a5a10 2024-04-22T08:15:36.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.040+0000 7fe0e9bfd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:36.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.040+0000 7fe0e9bfd640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0e40a25b0 con 0x7fe0e413b480 2024-04-22T08:15:36.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.040+0000 7fe0e9bfd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:36.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.040+0000 7fe0e9bfd640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0e409fba0 con 0x7fe0e40a32b0 2024-04-22T08:15:36.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.040+0000 7fe0e9bfd640 10 monclient(hunting): _renew_subs 2024-04-22T08:15:36.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.040+0000 7fe0e9bfd640 10 monclient(hunting): authenticate will time out at 1010.034361s 2024-04-22T08:15:36.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.040+0000 7fe0e3fff640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe0e413b480 0x7fe0e413d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:36.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.040+0000 7fe0e3fff640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe0e413b480 0x7fe0e413d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58812/0 (socket says 172.21.15.40:58812) 2024-04-22T08:15:36.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.040+0000 7fe0e37fe640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fe0e40a5a10 0x7fe0e40a1970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:40024/0 (socket says 172.21.15.40:40024) 2024-04-22T08:15:36.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.040+0000 7fe0e3fff640 1 -- 172.21.15.40:0/1823461223 learned_addr learned my addr 172.21.15.40:0/1823461223 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:15:36.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.040+0000 7fe0e3fff640 10 monclient(hunting): get_auth_request con 0x7fe0e413b480 auth_method 0 2024-04-22T08:15:36.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.040+0000 7fe0e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:36.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.040+0000 7fe0e3fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:36.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.040+0000 7fe0e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:36.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.040+0000 7fe0e2ffd640 1 --2- 172.21.15.40:0/1823461223 >> v2:172.21.15.189:3300/0 conn(0x7fe0e40a32b0 0x7fe0e413af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:36.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.040+0000 7fe0e2ffd640 10 monclient(hunting): get_auth_request con 0x7fe0e40a32b0 auth_method 0 2024-04-22T08:15:36.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.040+0000 7fe0e2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:36.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.040+0000 7fe0e2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:36.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.040+0000 7fe0e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:36.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.040+0000 7fe0e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:36.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.040+0000 7fe0e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:36.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.040+0000 7fe0e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:36.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.040+0000 7fe0e2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:36.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.040+0000 7fe0e2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:36.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.040+0000 7fe0e2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:36.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.040+0000 7fe0e3fff640 10 monclient(hunting): handle_auth_done global_id 34338 payload 274 2024-04-22T08:15:36.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.041+0000 7fe0e3fff640 1 -- 172.21.15.40:0/1823461223 >> v2:172.21.15.189:3300/0 conn(0x7fe0e40a32b0 msgr2=0x7fe0e413af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:15:36.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.041+0000 7fe0e3fff640 1 --2- 172.21.15.40:0/1823461223 >> v2:172.21.15.189:3300/0 conn(0x7fe0e40a32b0 0x7fe0e413af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:36.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.041+0000 7fe0e3fff640 1 -- 172.21.15.40:0/1823461223 >> v1:172.21.15.40:6789/0 conn(0x7fe0e40a5a10 legacy=0x7fe0e40a1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:15:36.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.041+0000 7fe0e3fff640 10 monclient: _finish_hunting 0 2024-04-22T08:15:36.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.041+0000 7fe0e3fff640 1 monclient: found mon.noname-a 2024-04-22T08:15:36.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.041+0000 7fe0e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:15:36.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.041+0000 7fe0e3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:15:36.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.041+0000 7fe0e3fff640 1 -- 172.21.15.40:0/1823461223 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0e40a0160 con 0x7fe0e413b480 2024-04-22T08:15:36.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.043+0000 7fe0e3fff640 1 -- 172.21.15.40:0/1823461223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0e413b480 msgr2=0x7fe0e413d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:36.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.043+0000 7fe0e3fff640 1 --2- 172.21.15.40:0/1823461223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0e413b480 0x7fe0e413d870 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe0e406c060 tx=0x7fe0dc02bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfa73c1f7a1e5162 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:36.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.044+0000 7fe0e27fc640 1 -- 172.21.15.40:0/1823461223 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0dc02cb70 con 0x7fe0e413b480 2024-04-22T08:15:36.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.044+0000 7fe0e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:36.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.044+0000 7fe0e27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:15:36.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.044+0000 7fe0e27fc640 10 monclient: dump: 2024-04-22T08:15:36.054 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:36.054 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:36.054 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:36.054 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:36.054 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:36.054 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:36.054 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:36.054 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:36.054 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:36.054 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:36.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.044+0000 7fe0e27fc640 10 monclient: _finish_auth 0 2024-04-22T08:15:36.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.044+0000 7fe0e27fc640 10 monclient: _check_auth_tickets 2024-04-22T08:15:36.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.044+0000 7fe0e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:15:36.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.044+0000 7fe0e27fc640 1 -- 172.21.15.40:0/1823461223 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0dc02ccd0 con 0x7fe0e413b480 2024-04-22T08:15:36.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.044+0000 7fe0e9bfd640 5 monclient: authenticate success, global_id 34338 2024-04-22T08:15:36.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.044+0000 7fe0e9bfd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:15:36.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.044+0000 7fe0e27fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:15:36.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.044+0000 7fe0e9bfd640 10 monclient: get_monmap_and_config success 2024-04-22T08:15:36.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.044+0000 7fe0e27fc640 1 -- 172.21.15.40:0/1823461223 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0dc034610 con 0x7fe0e413b480 2024-04-22T08:15:36.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.044+0000 7fe0e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:36.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.044+0000 7fe0e9bfd640 4 set_mon_vals no callback set 2024-04-22T08:15:36.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.044+0000 7fe0e27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:15:36.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.044+0000 7fe0e27fc640 10 monclient: dump: 2024-04-22T08:15:36.056 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:36.056 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:36.056 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:36.056 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:36.056 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:36.056 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:36.057 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:36.057 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:36.057 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:36.057 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:36.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.044+0000 7fe0e9bfd640 10 monclient: shutdown 2024-04-22T08:15:36.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.044+0000 7fe0e9bfd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:15:36.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.044+0000 7fe0e9bfd640 1 -- 172.21.15.40:0/1823461223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0e413b480 msgr2=0x7fe0e413d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:36.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.044+0000 7fe0e9bfd640 1 --2- 172.21.15.40:0/1823461223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0e413b480 0x7fe0e413d870 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe0e406c060 tx=0x7fe0dc02bdc0 comp rx=0 tx=0).stop 2024-04-22T08:15:36.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.044+0000 7fe0e9bfd640 1 -- 172.21.15.40:0/1823461223 shutdown_connections 2024-04-22T08:15:36.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.044+0000 7fe0e9bfd640 1 --2- 172.21.15.40:0/1823461223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0e413b480 0x7fe0e413d870 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:36.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.044+0000 7fe0e9bfd640 1 --2- 172.21.15.40:0/1823461223 >> v2:172.21.15.189:3300/0 conn(0x7fe0e40a32b0 0x7fe0e413af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:36.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.044+0000 7fe0e9bfd640 1 -- 172.21.15.40:0/1823461223 >> 172.21.15.40:0/1823461223 conn(0x7fe0e40a4ef0 msgr2=0x7fe0e40a5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:15:36.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.044+0000 7fe0e9bfd640 1 -- 172.21.15.40:0/1823461223 shutdown_connections 2024-04-22T08:15:36.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.044+0000 7fe0e9bfd640 1 -- 172.21.15.40:0/1823461223 wait complete. 2024-04-22T08:15:36.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.045+0000 7fe0e9bfd640 10 monclient: build_initial_monmap 2024-04-22T08:15:36.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.045+0000 7fe0e9bfd640 10 monclient: monmap: 2024-04-22T08:15:36.058 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:15:36.058 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:15:36.058 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:15:36.059 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:15:36.059 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:15:36.059 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:36.059 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:15:36.059 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:15:36.059 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:15:36.059 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:36.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.045+0000 7fe0e9bfd640 1 Processor -- start 2024-04-22T08:15:36.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.045+0000 7fe0e9bfd640 1 -- start start 2024-04-22T08:15:36.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.045+0000 7fe0e9bfd640 10 monclient: init 2024-04-22T08:15:36.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e9bfd640 10 monclient: _reopen_session rank -1 2024-04-22T08:15:36.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e9bfd640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-22T08:15:36.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e9bfd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0e40a32b0 0x7fe0e4133930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:36.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e9bfd640 10 monclient(hunting): picked mon.noname-b con 0x7fe0e40a32b0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:15:36.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e9bfd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe0e40a5a10 0x7fe0e4133e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:36.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e9bfd640 10 monclient(hunting): picked mon.noname-c con 0x7fe0e40a5a10 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:15:36.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e9bfd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0e413b480 0x7fe0e41343b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:36.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e9bfd640 10 monclient(hunting): picked mon.noname-a con 0x7fe0e413b480 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:15:36.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e9bfd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:36.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e9bfd640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e4137200 con 0x7fe0e413b480 2024-04-22T08:15:36.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e9bfd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:36.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e9bfd640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e4137590 con 0x7fe0e40a32b0 2024-04-22T08:15:36.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e9bfd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:36.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e9bfd640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e40a27c0 con 0x7fe0e40a5a10 2024-04-22T08:15:36.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e9bfd640 10 monclient(hunting): _renew_subs 2024-04-22T08:15:36.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e9bfd640 10 monclient(hunting): authenticate will time out at 1010.040104s 2024-04-22T08:15:36.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e2ffd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe0e40a5a10 0x7fe0e4133e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:36.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e2ffd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe0e40a5a10 0x7fe0e4133e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:60790/0 (socket says 172.21.15.40:60790) 2024-04-22T08:15:36.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e3fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0e413b480 0x7fe0e41343b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:36.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e2ffd640 1 -- 172.21.15.40:0/1392102772 learned_addr learned my addr 172.21.15.40:0/1392102772 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:15:36.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e2ffd640 10 monclient(hunting): get_auth_request con 0x7fe0e40a5a10 auth_method 0 2024-04-22T08:15:36.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:36.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:36.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:36.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e3fff640 10 monclient(hunting): get_auth_request con 0x7fe0e413b480 auth_method 0 2024-04-22T08:15:36.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:36.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e3fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:36.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:36.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e2ffd640 1 -- 172.21.15.40:0/1392102772 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe0e40a5a10 msgr2=0x7fe0e4133e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-22T08:15:36.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e2ffd640 1 -- 172.21.15.40:0/1392102772 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe0e40a5a10 msgr2=0x7fe0e4133e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:15:36.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e2ffd640 1 --2- 172.21.15.40:0/1392102772 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe0e40a5a10 0x7fe0e4133e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:15:36.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:36.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:36.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:36.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e2ffd640 1 --2- 172.21.15.40:0/1392102772 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe0e40a5a10 0x7fe0e4133e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:15:36.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e37fe640 1 --2- 172.21.15.40:0/1392102772 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0e40a32b0 0x7fe0e4133930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:36.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e37fe640 10 monclient(hunting): get_auth_request con 0x7fe0e40a32b0 auth_method 0 2024-04-22T08:15:36.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:36.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e37fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:36.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:36.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e3fff640 10 monclient(hunting): handle_auth_done global_id 34341 payload 824 2024-04-22T08:15:36.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e3fff640 1 -- 172.21.15.40:0/1392102772 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe0e40a5a10 msgr2=0x7fe0e4133e70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:15:36.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e3fff640 1 --2- 172.21.15.40:0/1392102772 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe0e40a5a10 0x7fe0e4133e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:36.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e3fff640 1 -- 172.21.15.40:0/1392102772 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0e40a32b0 msgr2=0x7fe0e4133930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:36.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e3fff640 1 --2- 172.21.15.40:0/1392102772 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0e40a32b0 0x7fe0e4133930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:36.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e3fff640 10 monclient: _finish_hunting 0 2024-04-22T08:15:36.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e3fff640 1 monclient: found mon.noname-a 2024-04-22T08:15:36.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:15:36.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:15:36.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.046+0000 7fe0e3fff640 1 -- 172.21.15.40:0/1392102772 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0dc009b30 con 0x7fe0e413b480 2024-04-22T08:15:36.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.049+0000 7fe0e3fff640 1 -- 172.21.15.40:0/1392102772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0e413b480 msgr2=0x7fe0e41343b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:36.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.049+0000 7fe0e3fff640 1 --2- 172.21.15.40:0/1392102772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0e413b480 0x7fe0e41343b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc032000 tx=0x7fe0dc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:36.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.049+0000 7fe0cbfff640 1 -- 172.21.15.40:0/1392102772 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0dc03e450 con 0x7fe0e413b480 2024-04-22T08:15:36.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.049+0000 7fe0cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:36.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.049+0000 7fe0cbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:15:36.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.049+0000 7fe0cbfff640 10 monclient: dump: 2024-04-22T08:15:36.065 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:36.066 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:36.066 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:36.066 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:36.066 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:36.066 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:36.066 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:36.066 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:36.066 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:36.066 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:36.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.049+0000 7fe0cbfff640 10 monclient: _finish_auth 0 2024-04-22T08:15:36.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.049+0000 7fe0cbfff640 10 monclient: _check_auth_tickets 2024-04-22T08:15:36.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.049+0000 7fe0cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:15:36.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.049+0000 7fe0cbfff640 1 -- 172.21.15.40:0/1392102772 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0dc03e5b0 con 0x7fe0e413b480 2024-04-22T08:15:36.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.049+0000 7fe0e9bfd640 5 monclient: authenticate success, global_id 34341 2024-04-22T08:15:36.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.049+0000 7fe0e9bfd640 10 monclient: _renew_subs 2024-04-22T08:15:36.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.049+0000 7fe0e9bfd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:36.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.049+0000 7fe0e9bfd640 1 -- 172.21.15.40:0/1392102772 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0e413aab0 con 0x7fe0e413b480 2024-04-22T08:15:36.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.049+0000 7fe0cbfff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:15:36.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.049+0000 7fe0cbfff640 1 -- 172.21.15.40:0/1392102772 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0dc03e940 con 0x7fe0e413b480 2024-04-22T08:15:36.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.049+0000 7fe0cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:36.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.049+0000 7fe0cbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:15:36.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.049+0000 7fe0cbfff640 10 monclient: dump: 2024-04-22T08:15:36.068 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:36.068 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:36.068 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:36.068 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:36.068 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:36.068 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:36.068 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:36.068 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:36.068 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:36.068 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:36.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.049+0000 7fe0e9bfd640 10 monclient: _renew_subs 2024-04-22T08:15:36.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.049+0000 7fe0e9bfd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:36.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.049+0000 7fe0e1ffb640 4 set_mon_vals no callback set 2024-04-22T08:15:36.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.049+0000 7fe0e9bfd640 1 -- 172.21.15.40:0/1392102772 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0e41368a0 con 0x7fe0e413b480 2024-04-22T08:15:36.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.049+0000 7fe0e9bfd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:36.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.050+0000 7fe0cbfff640 1 -- 172.21.15.40:0/1392102772 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fe0dc03eaa0 con 0x7fe0e413b480 2024-04-22T08:15:36.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.049+0000 7fe0e9bfd640 1 --2- 172.21.15.40:0/1392102772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0e4134db0 0x7fe0e413c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:36.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.050+0000 7fe0e9bfd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:36.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.050+0000 7fe0e9bfd640 1 -- 172.21.15.40:0/1392102772 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e40667a0 con 0x7fe0e4134db0 2024-04-22T08:15:36.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.050+0000 7fe0e37fe640 1 --2- 172.21.15.40:0/1392102772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0e4134db0 0x7fe0e413c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:36.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.050+0000 7fe0e37fe640 10 monclient: get_auth_request con 0x7fe0e4134db0 auth_method 0 2024-04-22T08:15:36.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.050+0000 7fe0e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:36.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.050+0000 7fe0e37fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:36.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.050+0000 7fe0e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:36.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.050+0000 7fe0e37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:36.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.050+0000 7fe0e37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:36.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.050+0000 7fe0e37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:36.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.050+0000 7fe0cbfff640 1 --2- 172.21.15.40:0/1392102772 >> v2:172.21.15.189:6800/30168 conn(0x7fe0d00377d0 0x7fe0d0039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:36.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.050+0000 7fe0cbfff640 1 -- 172.21.15.40:0/1392102772 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 3898+0+0 (secure 0 0 0) 0x7fe0dc02ce40 con 0x7fe0e413b480 2024-04-22T08:15:36.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.050+0000 7fe0e37fe640 10 monclient(hunting): handle_auth_done global_id 34344 payload 824 2024-04-22T08:15:36.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.050+0000 7fe0e37fe640 1 -- 172.21.15.40:0/1392102772 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe0e40a2080 con 0x7fe0e4134db0 2024-04-22T08:15:36.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.050+0000 7fe0e2ffd640 1 --2- 172.21.15.40:0/1392102772 >> v2:172.21.15.189:6800/30168 conn(0x7fe0d00377d0 0x7fe0d0039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:36.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.050+0000 7fe0e2ffd640 10 monclient: get_auth_request con 0x7fe0d00377d0 auth_method 0 2024-04-22T08:15:36.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.052+0000 7fe0e37fe640 1 -- 172.21.15.40:0/1392102772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0e4134db0 msgr2=0x7fe0e413c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:36.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.052+0000 7fe0e37fe640 1 --2- 172.21.15.40:0/1392102772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0e4134db0 0x7fe0e413c070 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe0d400e450 tx=0x7fe0d400e910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:36.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.052+0000 7fe0cbfff640 1 -- 172.21.15.40:0/1392102772 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d4004e20 con 0x7fe0e4134db0 2024-04-22T08:15:36.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.052+0000 7fe0cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:36.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.052+0000 7fe0cbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:15:36.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.052+0000 7fe0cbfff640 10 monclient: dump: 2024-04-22T08:15:36.070 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:36.070 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:36.070 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:36.070 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:36.071 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:36.071 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:36.071 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:36.071 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:36.071 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:36.071 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:36.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.053+0000 7fe0cbfff640 1 -- 172.21.15.40:0/1392102772 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7fe0d40060d0 con 0x7fe0e4134db0 2024-04-22T08:15:36.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.053+0000 7fe0cbfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:36.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.053+0000 7fe0cbfff640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:15:36.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.053+0000 7fe0cbfff640 1 -- 172.21.15.40:0/1392102772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0e4134db0 msgr2=0x7fe0e413c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:36.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.053+0000 7fe0cbfff640 1 --2- 172.21.15.40:0/1392102772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0e4134db0 0x7fe0e413c070 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe0d400e450 tx=0x7fe0d400e910 comp rx=0 tx=0).stop 2024-04-22T08:15:36.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.053+0000 7fe0e2ffd640 1 --2- 172.21.15.40:0/1392102772 >> v2:172.21.15.189:6800/30168 conn(0x7fe0d00377d0 0x7fe0d0039c90 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe0e40a3dd0 tx=0x7fe0d8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:36.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.053+0000 7fe0cbfff640 1 -- 172.21.15.40:0/1392102772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0e4134db0 msgr2=0x7fe0e413c070 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:15:36.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.053+0000 7fe0cbfff640 1 --2- 172.21.15.40:0/1392102772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0e4134db0 0x7fe0e413c070 secure :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe0d400e450 tx=0x7fe0d400e910 comp rx=0 tx=0).stop 2024-04-22T08:15:36.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.059+0000 7fe0e9bfd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:15:36.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.059+0000 7fe0e9bfd640 1 --2- 172.21.15.40:0/1392102772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0e413c5b0 0x7fe0e413c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:36.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.059+0000 7fe0e9bfd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:36.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.059+0000 7fe0e9bfd640 1 -- 172.21.15.40:0/1392102772 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e40a3700 con 0x7fe0e413c5b0 2024-04-22T08:15:36.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.059+0000 7fe0e37fe640 1 --2- 172.21.15.40:0/1392102772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0e413c5b0 0x7fe0e413c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:36.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.059+0000 7fe0e37fe640 10 monclient: get_auth_request con 0x7fe0e413c5b0 auth_method 0 2024-04-22T08:15:36.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.059+0000 7fe0e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:36.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.059+0000 7fe0e37fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:36.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.059+0000 7fe0e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:36.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.060+0000 7fe0e37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:36.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.060+0000 7fe0e37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:36.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.060+0000 7fe0e37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:36.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.060+0000 7fe0e37fe640 10 monclient(hunting): handle_auth_done global_id 34347 payload 824 2024-04-22T08:15:36.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.060+0000 7fe0e37fe640 1 -- 172.21.15.40:0/1392102772 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe0e413ddb0 con 0x7fe0e413c5b0 2024-04-22T08:15:36.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.062+0000 7fe0e37fe640 1 -- 172.21.15.40:0/1392102772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0e413c5b0 msgr2=0x7fe0e413c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:36.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.062+0000 7fe0e37fe640 1 --2- 172.21.15.40:0/1392102772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0e413c5b0 0x7fe0e413c9c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe0d400e450 tx=0x7fe0d4004930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:36.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.062+0000 7fe0cbfff640 1 -- 172.21.15.40:0/1392102772 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d4022070 con 0x7fe0e413c5b0 2024-04-22T08:15:36.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.062+0000 7fe0cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:36.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.062+0000 7fe0cbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:15:36.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.062+0000 7fe0cbfff640 10 monclient: dump: 2024-04-22T08:15:36.074 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:36.074 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:36.075 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:36.075 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:36.075 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:36.075 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:36.075 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:36.075 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:36.075 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:36.075 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:36.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.063+0000 7fe0cbfff640 1 -- 172.21.15.40:0/1392102772 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4633 (secure 0 0 0) 0x7fe0d4013420 con 0x7fe0e413c5b0 2024-04-22T08:15:36.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.063+0000 7fe0cbfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:15:36.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.063+0000 7fe0cbfff640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:15:36.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.063+0000 7fe0cbfff640 1 -- 172.21.15.40:0/1392102772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0e413c5b0 msgr2=0x7fe0e413c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:36.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.063+0000 7fe0cbfff640 1 --2- 172.21.15.40:0/1392102772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0e413c5b0 0x7fe0e413c9c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe0d400e450 tx=0x7fe0d4004930 comp rx=0 tx=0).stop 2024-04-22T08:15:36.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.063+0000 7fe0cbfff640 1 -- 172.21.15.40:0/1392102772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0e413c5b0 msgr2=0x7fe0e413c9c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:15:36.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.063+0000 7fe0cbfff640 1 --2- 172.21.15.40:0/1392102772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0e413c5b0 0x7fe0e413c9c0 secure :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe0d400e450 tx=0x7fe0d4004930 comp rx=0 tx=0).stop 2024-04-22T08:15:36.076 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:15:36.076 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:15:36.076 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:15:36.076 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "leader", 2024-04-22T08:15:36.076 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 36, 2024-04-22T08:15:36.076 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:15:36.076 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:15:36.076 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:15:36.076 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:15:36.076 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:36.077 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 2, 2024-04-22T08:15:36.077 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:15:36.077 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:15:36.077 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:15:36.077 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:15:36.077 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:15:36.077 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:15:36.077 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:15:36.077 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:15:36.077 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:15:36.077 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:15:36.077 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:15:36.077 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:15:36.077 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:15:36.078 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:36.078 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:15:36.078 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:15:36.078 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:15:36.078 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:15:36.078 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:15:36.078 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:15:36.078 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:15:36.078 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:15:36.078 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:15:36.078 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:15:36.078 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:15:36.078 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:15:36.078 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:36.079 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:36.079 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:15:36.079 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:15:36.079 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:15:36.079 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:15:36.079 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:15:36.079 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:15:36.079 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:15:36.079 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:15:36.079 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:15:36.079 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:15:36.079 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:15:36.079 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:15:36.080 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:15:36.080 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:15:36.080 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:15:36.080 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:15:36.080 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:15:36.080 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:15:36.080 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:15:36.080 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:15:36.080 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:15:36.080 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:15:36.080 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:15:36.080 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:15:36.080 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:15:36.080 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:15:36.080 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:15:36.081 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:36.081 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:15:36.081 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:36.081 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:15:36.081 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:36.081 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:15:36.081 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:15:36.081 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:15:36.081 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:15:36.081 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:36.081 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:15:36.081 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:15:36.081 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:36.082 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:36.082 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:36.082 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:15:36.082 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:15:36.082 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:36.082 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:36.082 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:36.082 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:36.082 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:15:36.082 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:15:36.082 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:15:36.082 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:15:36.082 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:15:36.082 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:36.083 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:36.083 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:15:36.083 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:15:36.083 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:15:36.083 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:15:36.083 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:36.083 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:15:36.083 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:15:36.083 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:36.083 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:36.083 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:36.083 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:15:36.083 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:15:36.084 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:36.084 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:36.084 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:36.084 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:36.084 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:15:36.084 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:15:36.084 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:15:36.084 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:15:36.084 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:15:36.084 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:36.085 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:36.085 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:15:36.085 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:15:36.085 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:15:36.085 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:15:36.085 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:36.085 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:15:36.085 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:15:36.085 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:36.085 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:36.085 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:36.085 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:15:36.085 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:15:36.086 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:36.086 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:36.086 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:36.086 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:36.086 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:15:36.086 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:15:36.086 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:15:36.086 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:15:36.086 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:15:36.086 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:36.086 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:36.087 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:36.087 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:15:36.087 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:15:36.087 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:36.087 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:15:36.087 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:15:36.087 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:15:36.087 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:36.087 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:36.087 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:15:36.088 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:36.088 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:15:36.088 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:15:36.088 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 4 2024-04-22T08:15:36.088 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:36.088 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:36.088 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:15:36.088 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:36.088 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:15:36.089 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:15:36.089 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 3 2024-04-22T08:15:36.089 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:36.089 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:36.089 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:36.089 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:15:36.089 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:15:36.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.064+0000 7fe0c9ffb640 1 -- 172.21.15.40:0/1392102772 >> v2:172.21.15.189:6800/30168 conn(0x7fe0d00377d0 msgr2=0x7fe0d0039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:36.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.064+0000 7fe0c9ffb640 1 --2- 172.21.15.40:0/1392102772 >> v2:172.21.15.189:6800/30168 conn(0x7fe0d00377d0 0x7fe0d0039c90 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe0e40a3dd0 tx=0x7fe0d8008040 comp rx=0 tx=0).stop 2024-04-22T08:15:36.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.064+0000 7fe0c9ffb640 10 monclient: shutdown 2024-04-22T08:15:36.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.064+0000 7fe0c9ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:15:36.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.064+0000 7fe0c9ffb640 1 -- 172.21.15.40:0/1392102772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0e413b480 msgr2=0x7fe0e41343b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:36.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.064+0000 7fe0c9ffb640 1 --2- 172.21.15.40:0/1392102772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0e413b480 0x7fe0e41343b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc032000 tx=0x7fe0dc004a40 comp rx=0 tx=0).stop 2024-04-22T08:15:36.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.064+0000 7fe0e3fff640 1 -- 172.21.15.40:0/1392102772 reap_dead start 2024-04-22T08:15:36.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.064+0000 7fe0e3fff640 1 -- 172.21.15.40:0/1392102772 reap_dead start 2024-04-22T08:15:36.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.064+0000 7fe0c9ffb640 1 -- 172.21.15.40:0/1392102772 shutdown_connections 2024-04-22T08:15:36.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.064+0000 7fe0c9ffb640 1 -- 172.21.15.40:0/1392102772 >> 172.21.15.40:0/1392102772 conn(0x7fe0e40a4ef0 msgr2=0x7fe0e4132380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:15:36.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.065+0000 7fe0c9ffb640 1 -- 172.21.15.40:0/1392102772 shutdown_connections 2024-04-22T08:15:36.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.066+0000 7fe0c9ffb640 1 -- 172.21.15.40:0/1392102772 wait complete. 2024-04-22T08:15:36.094 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:15:36.094 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-22T08:15:36.094 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-22T08:15:36.176 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:15:36.176 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35933]: starting. 2024-04-22T08:15:36.176 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35933]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-35898 2024-04-22T08:15:36.176 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35933]: created object 0... 2024-04-22T08:15:36.176 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35933]: created object 25... 2024-04-22T08:15:36.176 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35933]: created object 49... 2024-04-22T08:15:36.177 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35933]: finishing. 2024-04-22T08:15:36.177 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[35933]: shutting down. 2024-04-22T08:15:36.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.201+0000 7f12a4f8d640 10 monclient: get_monmap_and_config 2024-04-22T08:15:36.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.201+0000 7f12a4f8d640 10 monclient: build_initial_monmap 2024-04-22T08:15:36.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.201+0000 7f12a4f8d640 10 monclient: monmap: 2024-04-22T08:15:36.201 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:15:36.201 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:36.201 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:15:36.202593+0000 2024-04-22T08:15:36.201 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:15:36.202593+0000 2024-04-22T08:15:36.202 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:15:36.202 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:36.202 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:15:36.202 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:15:36.202 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:15:36.202 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:15:36.202 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:15:36.202 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:36.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.202+0000 7f12a4f8d640 1 Processor -- start 2024-04-22T08:15:36.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.202+0000 7f12a4f8d640 1 -- start start 2024-04-22T08:15:36.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.202+0000 7f12a4f8d640 10 monclient: init 2024-04-22T08:15:36.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.202+0000 7f12a4f8d640 10 monclient: _reopen_session rank -1 2024-04-22T08:15:36.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.202+0000 7f12a4f8d640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-04-22T08:15:36.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.203+0000 7f12a4f8d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f12a00a5a10 addr v1:172.21.15.40:6789/0 2024-04-22T08:15:36.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.203+0000 7f12a4f8d640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f12a00a32b0 0x7f12a013af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:36.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.203+0000 7f12a4f8d640 10 monclient(hunting): picked mon.noname-c con 0x7f12a00a32b0 addr v2:172.21.15.189:3300/0 2024-04-22T08:15:36.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.203+0000 7f12a4f8d640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f12a013b480 0x7f12a013d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:36.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.203+0000 7f12a4f8d640 10 monclient(hunting): picked mon.noname-a con 0x7f12a013b480 addr v2:172.21.15.40:3300/0 2024-04-22T08:15:36.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.203+0000 7f12a4f8d640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12a009fe80 con 0x7f12a00a5a10 2024-04-22T08:15:36.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.203+0000 7f12a4f8d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:36.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.203+0000 7f12a4f8d640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12a00a25b0 con 0x7f12a013b480 2024-04-22T08:15:36.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.203+0000 7f12a4f8d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:36.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.203+0000 7f12a4f8d640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12a009fba0 con 0x7f12a00a32b0 2024-04-22T08:15:36.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.203+0000 7f12a4f8d640 10 monclient(hunting): _renew_subs 2024-04-22T08:15:36.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.203+0000 7f12a4f8d640 10 monclient(hunting): authenticate will time out at 1010.196832s 2024-04-22T08:15:36.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.203+0000 7f129ed9d640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f12a013b480 0x7f12a013d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:36.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.203+0000 7f129e59c640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f12a00a5a10 0x7f12a00a1970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:40036/0 (socket says 172.21.15.40:40036) 2024-04-22T08:15:36.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.203+0000 7f129e59c640 1 -- 172.21.15.40:0/1864835997 learned_addr learned my addr 172.21.15.40:0/1864835997 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:15:36.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.203+0000 7f129ed9d640 10 monclient(hunting): get_auth_request con 0x7f12a013b480 auth_method 0 2024-04-22T08:15:36.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.203+0000 7f129ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:36.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.203+0000 7f129ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:36.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.203+0000 7f129ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:36.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.203+0000 7f129ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:36.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.203+0000 7f129ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:36.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.203+0000 7f129ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:36.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.203+0000 7f129dd9b640 1 --2- 172.21.15.40:0/1864835997 >> v2:172.21.15.189:3300/0 conn(0x7f12a00a32b0 0x7f12a013af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:36.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.203+0000 7f129dd9b640 10 monclient(hunting): get_auth_request con 0x7f12a00a32b0 auth_method 0 2024-04-22T08:15:36.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.203+0000 7f129dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:36.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.203+0000 7f129dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:36.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.203+0000 7f129dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:36.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.203+0000 7f129ed9d640 10 monclient(hunting): handle_auth_done global_id 34350 payload 274 2024-04-22T08:15:36.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.203+0000 7f129ed9d640 1 -- 172.21.15.40:0/1864835997 >> v2:172.21.15.189:3300/0 conn(0x7f12a00a32b0 msgr2=0x7f12a013af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:15:36.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.203+0000 7f129ed9d640 1 --2- 172.21.15.40:0/1864835997 >> v2:172.21.15.189:3300/0 conn(0x7f12a00a32b0 0x7f12a013af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:36.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.203+0000 7f129ed9d640 1 -- 172.21.15.40:0/1864835997 >> v1:172.21.15.40:6789/0 conn(0x7f12a00a5a10 legacy=0x7f12a00a1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:15:36.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.203+0000 7f129ed9d640 10 monclient: _finish_hunting 0 2024-04-22T08:15:36.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.203+0000 7f129ed9d640 1 monclient: found mon.noname-a 2024-04-22T08:15:36.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.203+0000 7f129ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:15:36.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.203+0000 7f129ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:15:36.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.203+0000 7f129ed9d640 1 -- 172.21.15.40:0/1864835997 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a00a0160 con 0x7f12a013b480 2024-04-22T08:15:36.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.206+0000 7f129ed9d640 1 -- 172.21.15.40:0/1864835997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12a013b480 msgr2=0x7f12a013d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:36.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.206+0000 7f129ed9d640 1 --2- 172.21.15.40:0/1864835997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12a013b480 0x7f12a013d870 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f12a006c060 tx=0x7f129002bd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c576998f44dfeb5f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:36.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.206+0000 7f129d59a640 1 -- 172.21.15.40:0/1864835997 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f129002cce0 con 0x7f12a013b480 2024-04-22T08:15:36.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.206+0000 7f129d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:36.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.206+0000 7f129d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:15:36.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.206+0000 7f129d59a640 10 monclient: dump: 2024-04-22T08:15:36.214 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:36.214 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:36.214 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:36.214 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:36.214 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:36.215 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:36.215 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:36.215 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:36.215 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:36.215 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:36.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.206+0000 7f129d59a640 10 monclient: _finish_auth 0 2024-04-22T08:15:36.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.206+0000 7f129d59a640 10 monclient: _check_auth_tickets 2024-04-22T08:15:36.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.206+0000 7f129d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:15:36.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.206+0000 7f129d59a640 1 -- 172.21.15.40:0/1864835997 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f129002ce40 con 0x7f12a013b480 2024-04-22T08:15:36.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.206+0000 7f129d59a640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:15:36.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.206+0000 7f129d59a640 1 -- 172.21.15.40:0/1864835997 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1290034660 con 0x7f12a013b480 2024-04-22T08:15:36.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.206+0000 7f12a4f8d640 5 monclient: authenticate success, global_id 34350 2024-04-22T08:15:36.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.206+0000 7f12a4f8d640 10 monclient: get_monmap_and_config success 2024-04-22T08:15:36.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.206+0000 7f12a4f8d640 4 set_mon_vals no callback set 2024-04-22T08:15:36.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.206+0000 7f129d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:36.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.206+0000 7f129d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:15:36.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.206+0000 7f129d59a640 10 monclient: dump: 2024-04-22T08:15:36.216 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:36.216 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:36.216 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:36.216 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:36.216 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:36.216 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:36.217 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:36.217 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:36.217 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:36.217 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:36.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.207+0000 7f12a4f8d640 10 monclient: shutdown 2024-04-22T08:15:36.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.207+0000 7f12a4f8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:15:36.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.207+0000 7f12a4f8d640 1 -- 172.21.15.40:0/1864835997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12a013b480 msgr2=0x7f12a013d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:36.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.207+0000 7f12a4f8d640 1 --2- 172.21.15.40:0/1864835997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12a013b480 0x7f12a013d870 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f12a006c060 tx=0x7f129002bd20 comp rx=0 tx=0).stop 2024-04-22T08:15:36.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.207+0000 7f12a4f8d640 1 -- 172.21.15.40:0/1864835997 shutdown_connections 2024-04-22T08:15:36.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.207+0000 7f12a4f8d640 1 --2- 172.21.15.40:0/1864835997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12a013b480 0x7f12a013d870 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:36.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.207+0000 7f12a4f8d640 1 --2- 172.21.15.40:0/1864835997 >> v2:172.21.15.189:3300/0 conn(0x7f12a00a32b0 0x7f12a013af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:36.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.207+0000 7f12a4f8d640 1 -- 172.21.15.40:0/1864835997 >> 172.21.15.40:0/1864835997 conn(0x7f12a00a4ef0 msgr2=0x7f12a00a5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:15:36.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.207+0000 7f12a4f8d640 1 -- 172.21.15.40:0/1864835997 shutdown_connections 2024-04-22T08:15:36.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.207+0000 7f12a4f8d640 1 -- 172.21.15.40:0/1864835997 wait complete. 2024-04-22T08:15:36.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.207+0000 7f12a4f8d640 10 monclient: build_initial_monmap 2024-04-22T08:15:36.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.207+0000 7f12a4f8d640 10 monclient: monmap: 2024-04-22T08:15:36.218 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:15:36.218 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:15:36.218 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:15:36.218 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:15:36.218 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:15:36.218 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:36.218 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:15:36.218 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:15:36.218 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:15:36.218 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:36.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.208+0000 7f12a4f8d640 1 Processor -- start 2024-04-22T08:15:36.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.208+0000 7f12a4f8d640 1 -- start start 2024-04-22T08:15:36.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.208+0000 7f12a4f8d640 10 monclient: init 2024-04-22T08:15:36.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.208+0000 7f12a4f8d640 10 monclient: _reopen_session rank -1 2024-04-22T08:15:36.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.208+0000 7f12a4f8d640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-22T08:15:36.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.208+0000 7f12a4f8d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12a00a32b0 0x7f12a0133930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:36.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.208+0000 7f12a4f8d640 10 monclient(hunting): picked mon.noname-b con 0x7f12a00a32b0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:15:36.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.208+0000 7f12a4f8d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12a00a5a10 0x7f12a0133e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:36.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.208+0000 7f12a4f8d640 10 monclient(hunting): picked mon.noname-c con 0x7f12a00a5a10 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:15:36.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.208+0000 7f12a4f8d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12a013b480 0x7f12a01343b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:36.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.208+0000 7f12a4f8d640 10 monclient(hunting): picked mon.noname-a con 0x7f12a013b480 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:15:36.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.208+0000 7f12a4f8d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:36.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.208+0000 7f12a4f8d640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a0137200 con 0x7f12a013b480 2024-04-22T08:15:36.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.208+0000 7f12a4f8d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:36.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.208+0000 7f12a4f8d640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a0137590 con 0x7f12a00a32b0 2024-04-22T08:15:36.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.208+0000 7f12a4f8d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:36.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.208+0000 7f12a4f8d640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12a00a27c0 con 0x7f12a00a5a10 2024-04-22T08:15:36.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.208+0000 7f12a4f8d640 10 monclient(hunting): _renew_subs 2024-04-22T08:15:36.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.208+0000 7f12a4f8d640 10 monclient(hunting): authenticate will time out at 1010.202479s 2024-04-22T08:15:36.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.208+0000 7f129dd9b640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12a00a5a10 0x7f12a0133e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:36.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.208+0000 7f129ed9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12a013b480 0x7f12a01343b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:36.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.208+0000 7f129dd9b640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12a00a5a10 0x7f12a0133e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:60796/0 (socket says 172.21.15.40:60796) 2024-04-22T08:15:36.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.208+0000 7f129dd9b640 1 -- 172.21.15.40:0/1665602295 learned_addr learned my addr 172.21.15.40:0/1665602295 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:15:36.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.208+0000 7f129dd9b640 10 monclient(hunting): get_auth_request con 0x7f12a00a5a10 auth_method 0 2024-04-22T08:15:36.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.208+0000 7f129dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:36.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.208+0000 7f129dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:36.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.208+0000 7f129dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:36.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.208+0000 7f129e59c640 1 --2- 172.21.15.40:0/1665602295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12a00a32b0 0x7f12a0133930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:36.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.208+0000 7f129ed9d640 10 monclient(hunting): get_auth_request con 0x7f12a013b480 auth_method 0 2024-04-22T08:15:36.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.208+0000 7f129ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:36.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.208+0000 7f129ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:36.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.208+0000 7f129ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:36.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.208+0000 7f129e59c640 10 monclient(hunting): get_auth_request con 0x7f12a00a32b0 auth_method 0 2024-04-22T08:15:36.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.208+0000 7f129e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:36.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.208+0000 7f129e59c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:36.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.208+0000 7f129e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:36.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.208+0000 7f129dd9b640 1 -- 172.21.15.40:0/1665602295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12a00a5a10 msgr2=0x7f12a0133e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-22T08:15:36.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.208+0000 7f129dd9b640 1 -- 172.21.15.40:0/1665602295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12a00a5a10 msgr2=0x7f12a0133e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:15:36.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.208+0000 7f129dd9b640 1 --2- 172.21.15.40:0/1665602295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12a00a5a10 0x7f12a0133e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:15:36.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.208+0000 7f129ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:36.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.208+0000 7f129ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:36.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.208+0000 7f129ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:36.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.208+0000 7f129e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:36.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.208+0000 7f129e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:36.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.208+0000 7f129e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:36.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.208+0000 7f129dd9b640 1 --2- 172.21.15.40:0/1665602295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12a00a5a10 0x7f12a0133e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:15:36.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.209+0000 7f129ed9d640 10 monclient(hunting): handle_auth_done global_id 34353 payload 824 2024-04-22T08:15:36.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.209+0000 7f129ed9d640 1 -- 172.21.15.40:0/1665602295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12a00a5a10 msgr2=0x7f12a0133e70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:15:36.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.209+0000 7f129ed9d640 1 --2- 172.21.15.40:0/1665602295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12a00a5a10 0x7f12a0133e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:36.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.209+0000 7f129ed9d640 1 -- 172.21.15.40:0/1665602295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12a00a32b0 msgr2=0x7f12a0133930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:36.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.209+0000 7f129ed9d640 1 --2- 172.21.15.40:0/1665602295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12a00a32b0 0x7f12a0133930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:36.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.209+0000 7f129ed9d640 10 monclient: _finish_hunting 0 2024-04-22T08:15:36.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.209+0000 7f129ed9d640 1 monclient: found mon.noname-a 2024-04-22T08:15:36.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.209+0000 7f129ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:15:36.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.209+0000 7f129ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:15:36.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.209+0000 7f129ed9d640 1 -- 172.21.15.40:0/1665602295 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1290009a70 con 0x7f12a013b480 2024-04-22T08:15:36.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.211+0000 7f129ed9d640 1 -- 172.21.15.40:0/1665602295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12a013b480 msgr2=0x7f12a01343b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:36.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.211+0000 7f129ed9d640 1 --2- 172.21.15.40:0/1665602295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12a013b480 0x7f12a01343b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1290036000 tx=0x7f1290004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:36.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.211+0000 7f1296ffd640 1 -- 172.21.15.40:0/1665602295 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f129003e610 con 0x7f12a013b480 2024-04-22T08:15:36.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.211+0000 7f1296ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:36.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.211+0000 7f1296ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:15:36.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.211+0000 7f1296ffd640 10 monclient: dump: 2024-04-22T08:15:36.223 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:36.223 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:36.223 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:36.223 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:36.223 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:36.224 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:36.224 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:36.224 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:36.224 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:36.224 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:36.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.211+0000 7f1296ffd640 10 monclient: _finish_auth 0 2024-04-22T08:15:36.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.211+0000 7f1296ffd640 10 monclient: _check_auth_tickets 2024-04-22T08:15:36.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.211+0000 7f1296ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:15:36.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.211+0000 7f12a4f8d640 5 monclient: authenticate success, global_id 34353 2024-04-22T08:15:36.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.211+0000 7f1296ffd640 1 -- 172.21.15.40:0/1665602295 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f129003e770 con 0x7f12a013b480 2024-04-22T08:15:36.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.211+0000 7f1296ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:15:36.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.211+0000 7f1296ffd640 1 -- 172.21.15.40:0/1665602295 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f129003eae0 con 0x7f12a013b480 2024-04-22T08:15:36.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.211+0000 7f12a4f8d640 10 monclient: _renew_subs 2024-04-22T08:15:36.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.211+0000 7f12a4f8d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:36.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.211+0000 7f12a4f8d640 1 -- 172.21.15.40:0/1665602295 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12a013aab0 con 0x7f12a013b480 2024-04-22T08:15:36.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.211+0000 7f129cd99640 4 set_mon_vals no callback set 2024-04-22T08:15:36.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.211+0000 7f1296ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:36.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.211+0000 7f1296ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:15:36.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.211+0000 7f1296ffd640 10 monclient: dump: 2024-04-22T08:15:36.225 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:36.225 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:36.225 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:36.225 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:36.225 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:36.225 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:36.225 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:36.225 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:36.225 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:36.225 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:36.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.211+0000 7f12a4f8d640 10 monclient: _renew_subs 2024-04-22T08:15:36.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.211+0000 7f12a4f8d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:36.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.211+0000 7f12a4f8d640 1 -- 172.21.15.40:0/1665602295 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12a01368a0 con 0x7f12a013b480 2024-04-22T08:15:36.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.212+0000 7f1296ffd640 1 -- 172.21.15.40:0/1665602295 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f129003b050 con 0x7f12a013b480 2024-04-22T08:15:36.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.212+0000 7f12a4f8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:36.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.212+0000 7f12a4f8d640 1 --2- 172.21.15.40:0/1665602295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12a0134db0 0x7f12a013c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:36.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.212+0000 7f12a4f8d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:36.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.212+0000 7f12a4f8d640 1 -- 172.21.15.40:0/1665602295 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12a0136a30 con 0x7f12a0134db0 2024-04-22T08:15:36.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.212+0000 7f129e59c640 1 --2- 172.21.15.40:0/1665602295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12a0134db0 0x7f12a013c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:36.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.212+0000 7f129e59c640 10 monclient: get_auth_request con 0x7f12a0134db0 auth_method 0 2024-04-22T08:15:36.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.212+0000 7f129e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:36.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.212+0000 7f129e59c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:36.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.212+0000 7f129e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:36.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.212+0000 7f1296ffd640 1 --2- 172.21.15.40:0/1665602295 >> v2:172.21.15.189:6800/30168 conn(0x7f12800377d0 0x7f1280039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:36.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.212+0000 7f129e59c640 1 -- 172.21.15.40:0/1665602295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12a0134db0 msgr2=0x7f12a013c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-22T08:15:36.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.212+0000 7f129e59c640 1 -- 172.21.15.40:0/1665602295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12a0134db0 msgr2=0x7f12a013c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:15:36.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.212+0000 7f129e59c640 1 --2- 172.21.15.40:0/1665602295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12a0134db0 0x7f12a013c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:15:36.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.212+0000 7f1296ffd640 1 -- 172.21.15.40:0/1665602295 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f129006c970 con 0x7f12a013b480 2024-04-22T08:15:36.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.212+0000 7f129e59c640 1 --2- 172.21.15.40:0/1665602295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12a0134db0 0x7f12a013c070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:15:36.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.212+0000 7f129dd9b640 1 --2- 172.21.15.40:0/1665602295 >> v2:172.21.15.189:6800/30168 conn(0x7f12800377d0 0x7f1280039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:36.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.212+0000 7f129dd9b640 10 monclient: get_auth_request con 0x7f12800377d0 auth_method 0 2024-04-22T08:15:36.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.215+0000 7f129dd9b640 1 --2- 172.21.15.40:0/1665602295 >> v2:172.21.15.189:6800/30168 conn(0x7f12800377d0 0x7f1280039c90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f12a00a3dd0 tx=0x7f1288008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:36.228 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:15:36.228 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35935]: starting. 2024-04-22T08:15:36.228 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35935]: listing objects. 2024-04-22T08:15:36.228 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35935]: listed object 0... 2024-04-22T08:15:36.228 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35935]: listed object 25... 2024-04-22T08:15:36.228 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35935]: saw 50 objects 2024-04-22T08:15:36.228 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[35935]: shutting down. 2024-04-22T08:15:36.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.413+0000 7f129e59c640 1 --2- 172.21.15.40:0/1665602295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12a0134db0 0x7f12a013c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:36.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.413+0000 7f129e59c640 10 monclient: get_auth_request con 0x7f12a0134db0 auth_method 0 2024-04-22T08:15:36.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.413+0000 7f129e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:36.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.413+0000 7f129e59c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:36.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.413+0000 7f129e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:15:36.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.413+0000 7f129e59c640 1 -- 172.21.15.40:0/1665602295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12a0134db0 msgr2=0x7f12a013c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-22T08:15:36.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.413+0000 7f129e59c640 1 -- 172.21.15.40:0/1665602295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12a0134db0 msgr2=0x7f12a013c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:15:36.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.413+0000 7f129e59c640 1 --2- 172.21.15.40:0/1665602295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12a0134db0 0x7f12a013c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:15:36.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.413+0000 7f129e59c640 1 --2- 172.21.15.40:0/1665602295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12a0134db0 0x7f12a013c070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:15:36.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.814+0000 7f129e59c640 1 --2- 172.21.15.40:0/1665602295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12a0134db0 0x7f12a013c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:36.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.814+0000 7f129e59c640 10 monclient: get_auth_request con 0x7f12a0134db0 auth_method 0 2024-04-22T08:15:36.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.814+0000 7f129e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:36.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.814+0000 7f129e59c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:36.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.814+0000 7f129e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:15:36.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.814+0000 7f129e59c640 1 -- 172.21.15.40:0/1665602295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12a0134db0 msgr2=0x7f12a013c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-22T08:15:36.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.814+0000 7f129e59c640 1 -- 172.21.15.40:0/1665602295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12a0134db0 msgr2=0x7f12a013c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:15:36.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.814+0000 7f129e59c640 1 --2- 172.21.15.40:0/1665602295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12a0134db0 0x7f12a013c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:15:36.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:36.814+0000 7f129e59c640 1 --2- 172.21.15.40:0/1665602295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12a0134db0 0x7f12a013c070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:15:36.850 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:15:36.851 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[35934]: starting. 2024-04-22T08:15:36.851 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[35934]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-35898 2024-04-22T08:15:36.851 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[35934]: shutting down. 2024-04-22T08:15:36.852 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:15:36.852 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:15:36.852 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:15:36.855 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:15:37.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.615+0000 7f129e59c640 1 --2- 172.21.15.40:0/1665602295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12a0134db0 0x7f12a013c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:37.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.615+0000 7f129e59c640 10 monclient: get_auth_request con 0x7f12a0134db0 auth_method 0 2024-04-22T08:15:37.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.615+0000 7f129e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:37.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.615+0000 7f129e59c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:37.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.615+0000 7f129e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:15:37.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.615+0000 7f129e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:37.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.615+0000 7f129e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:37.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.615+0000 7f129e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:37.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.615+0000 7f129e59c640 10 monclient(hunting): handle_auth_done global_id 44099 payload 824 2024-04-22T08:15:37.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.615+0000 7f129e59c640 1 -- 172.21.15.40:0/1665602295 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f12a00a2080 con 0x7f12a0134db0 2024-04-22T08:15:37.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.618+0000 7f129e59c640 1 -- 172.21.15.40:0/1665602295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12a0134db0 msgr2=0x7f12a013c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:37.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.618+0000 7f129e59c640 1 --2- 172.21.15.40:0/1665602295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12a0134db0 0x7f12a013c070 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f12980149c0 tx=0x7f1298014e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:37.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.618+0000 7f1296ffd640 1 -- 172.21.15.40:0/1665602295 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1298012d80 con 0x7f12a0134db0 2024-04-22T08:15:37.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.618+0000 7f1296ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:37.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.618+0000 7f1296ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:15:37.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.618+0000 7f1296ffd640 10 monclient: dump: 2024-04-22T08:15:37.618 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:37.618 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:37.618 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:37.618 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:37.618 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:37.618 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:37.618 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:37.618 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:37.618 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:37.618 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:37.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.618+0000 7f1296ffd640 1 -- 172.21.15.40:0/1665602295 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f1298017460 con 0x7f12a0134db0 2024-04-22T08:15:37.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.618+0000 7f1296ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:37.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.618+0000 7f1296ffd640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:15:37.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.618+0000 7f1296ffd640 1 -- 172.21.15.40:0/1665602295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12a0134db0 msgr2=0x7f12a013c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:37.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.618+0000 7f1296ffd640 1 --2- 172.21.15.40:0/1665602295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12a0134db0 0x7f12a013c070 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f12980149c0 tx=0x7f1298014e80 comp rx=0 tx=0).stop 2024-04-22T08:15:37.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.618+0000 7f1296ffd640 1 -- 172.21.15.40:0/1665602295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12a0134db0 msgr2=0x7f12a013c070 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:15:37.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.618+0000 7f1296ffd640 1 --2- 172.21.15.40:0/1665602295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12a0134db0 0x7f12a013c070 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:37.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.625+0000 7f12a4f8d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:15:37.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.625+0000 7f12a4f8d640 1 --2- 172.21.15.40:0/1665602295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12a013c5b0 0x7f12a013c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:37.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.625+0000 7f12a4f8d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:37.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.625+0000 7f12a4f8d640 1 -- 172.21.15.40:0/1665602295 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12a00667a0 con 0x7f12a013c5b0 2024-04-22T08:15:37.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.625+0000 7f129e59c640 1 --2- 172.21.15.40:0/1665602295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12a013c5b0 0x7f12a013c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:37.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.625+0000 7f129e59c640 10 monclient: get_auth_request con 0x7f12a013c5b0 auth_method 0 2024-04-22T08:15:37.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.625+0000 7f129e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:37.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.625+0000 7f129e59c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:37.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.625+0000 7f129e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:37.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.625+0000 7f129e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:37.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.625+0000 7f129e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:37.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.625+0000 7f129e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:37.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.626+0000 7f129e59c640 10 monclient(hunting): handle_auth_done global_id 44102 payload 824 2024-04-22T08:15:37.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.626+0000 7f129e59c640 1 -- 172.21.15.40:0/1665602295 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f12a00a3700 con 0x7f12a013c5b0 2024-04-22T08:15:37.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.628+0000 7f129e59c640 1 -- 172.21.15.40:0/1665602295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12a013c5b0 msgr2=0x7f12a013c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:37.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.628+0000 7f129e59c640 1 --2- 172.21.15.40:0/1665602295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12a013c5b0 0x7f12a013c9c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f129801e000 tx=0x7f1298007f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:37.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.628+0000 7f1296ffd640 1 -- 172.21.15.40:0/1665602295 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1298027070 con 0x7f12a013c5b0 2024-04-22T08:15:37.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.628+0000 7f1296ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:37.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.628+0000 7f1296ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:15:37.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.628+0000 7f1296ffd640 10 monclient: dump: 2024-04-22T08:15:37.629 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:37.629 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:37.629 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:37.629 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:37.629 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:37.629 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:37.630 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:37.630 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:37.630 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:37.630 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:37.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.629+0000 7f1296ffd640 1 -- 172.21.15.40:0/1665602295 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4461 (secure 0 0 0) 0x7f12980163c0 con 0x7f12a013c5b0 2024-04-22T08:15:37.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.629+0000 7f1296ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:15:37.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.629+0000 7f1296ffd640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:15:37.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.629+0000 7f1296ffd640 1 -- 172.21.15.40:0/1665602295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12a013c5b0 msgr2=0x7f12a013c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:37.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.629+0000 7f1296ffd640 1 --2- 172.21.15.40:0/1665602295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12a013c5b0 0x7f12a013c9c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f129801e000 tx=0x7f1298007f20 comp rx=0 tx=0).stop 2024-04-22T08:15:37.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.629+0000 7f1296ffd640 1 -- 172.21.15.40:0/1665602295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12a013c5b0 msgr2=0x7f12a013c9c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:15:37.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.629+0000 7f1296ffd640 1 --2- 172.21.15.40:0/1665602295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12a013c5b0 0x7f12a013c9c0 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f129801e000 tx=0x7f1298007f20 comp rx=0 tx=0).stop 2024-04-22T08:15:37.631 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:15:37.632 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:15:37.632 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:15:37.632 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:15:37.632 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 36, 2024-04-22T08:15:37.632 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:15:37.632 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:15:37.632 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:15:37.632 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:15:37.632 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:37.633 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 4, 2024-04-22T08:15:37.633 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:15:37.633 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:15:37.633 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:15:37.633 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:15:37.633 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:15:37.633 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:15:37.633 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:15:37.633 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:15:37.634 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:15:37.634 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:15:37.634 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:15:37.634 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:15:37.634 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:15:37.634 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:37.634 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:15:37.634 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:15:37.634 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:15:37.634 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:15:37.634 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:15:37.634 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:15:37.634 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:15:37.635 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:15:37.635 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:15:37.635 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:15:37.635 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:15:37.635 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:15:37.635 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:37.635 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:37.635 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:15:37.635 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:15:37.635 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:15:37.635 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:15:37.635 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:15:37.635 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:15:37.635 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:15:37.636 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:15:37.636 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:15:37.636 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:15:37.636 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:15:37.636 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:15:37.636 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:15:37.636 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:15:37.636 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:15:37.636 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:15:37.636 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:15:37.636 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:15:37.636 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:15:37.636 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:15:37.636 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:15:37.637 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:15:37.637 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:15:37.637 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:15:37.637 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:15:37.637 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:15:37.637 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:15:37.637 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:37.637 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:15:37.637 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:37.637 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:15:37.637 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:37.637 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:15:37.637 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:15:37.637 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:15:37.638 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:15:37.638 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:37.638 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:15:37.638 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:15:37.638 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:37.638 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:37.638 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:37.638 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:15:37.638 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:15:37.638 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:37.638 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:37.638 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:37.638 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:37.638 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:15:37.638 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:15:37.639 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:15:37.639 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:15:37.639 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:15:37.639 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:37.639 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:37.639 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:15:37.639 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:15:37.639 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:15:37.639 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:15:37.639 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:37.640 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:15:37.640 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:15:37.640 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:37.640 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:37.640 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:37.640 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:15:37.640 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:15:37.640 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:37.641 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:37.641 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:37.641 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:37.641 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:15:37.641 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:15:37.641 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:15:37.641 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:15:37.641 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:15:37.641 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:37.641 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:37.641 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:15:37.641 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:15:37.641 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:15:37.642 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:15:37.642 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:37.642 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:15:37.642 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:15:37.642 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:37.642 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:37.642 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:37.642 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:15:37.642 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:15:37.642 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:37.642 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:37.642 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:37.642 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:37.642 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:15:37.643 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:15:37.643 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:15:37.643 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:15:37.643 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:15:37.643 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:37.643 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:37.643 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:37.643 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:15:37.643 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:15:37.643 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:37.643 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:15:37.643 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:15:37.644 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:15:37.644 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:37.644 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:37.644 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:15:37.644 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:37.644 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:15:37.644 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:15:37.644 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:15:37.644 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:37.644 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:37.644 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:37.644 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:15:37.644 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:15:37.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.629+0000 7f12a4f8d640 1 -- 172.21.15.40:0/1665602295 >> v2:172.21.15.189:6800/30168 conn(0x7f12800377d0 msgr2=0x7f1280039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:37.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.629+0000 7f12a4f8d640 1 --2- 172.21.15.40:0/1665602295 >> v2:172.21.15.189:6800/30168 conn(0x7f12800377d0 0x7f1280039c90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f12a00a3dd0 tx=0x7f1288008040 comp rx=0 tx=0).stop 2024-04-22T08:15:37.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.629+0000 7f12a4f8d640 10 monclient: shutdown 2024-04-22T08:15:37.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.629+0000 7f12a4f8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:15:37.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.629+0000 7f12a4f8d640 1 -- 172.21.15.40:0/1665602295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12a013b480 msgr2=0x7f12a01343b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:37.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.629+0000 7f12a4f8d640 1 --2- 172.21.15.40:0/1665602295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12a013b480 0x7f12a01343b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1290036000 tx=0x7f1290004a40 comp rx=0 tx=0).stop 2024-04-22T08:15:37.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.629+0000 7f129ed9d640 1 -- 172.21.15.40:0/1665602295 reap_dead start 2024-04-22T08:15:37.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.629+0000 7f12a4f8d640 1 -- 172.21.15.40:0/1665602295 shutdown_connections 2024-04-22T08:15:37.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.629+0000 7f12a4f8d640 1 --2- 172.21.15.40:0/1665602295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12a013b480 0x7f12a01343b0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:37.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.629+0000 7f12a4f8d640 1 -- 172.21.15.40:0/1665602295 >> 172.21.15.40:0/1665602295 conn(0x7f12a00a4ef0 msgr2=0x7f12a0132380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:15:37.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.629+0000 7f12a4f8d640 1 -- 172.21.15.40:0/1665602295 shutdown_connections 2024-04-22T08:15:37.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.629+0000 7f12a4f8d640 1 -- 172.21.15.40:0/1665602295 wait complete. 2024-04-22T08:15:37.647 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:15:37.647 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-22T08:15:37.647 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-22T08:15:37.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.756+0000 7f26268a1640 10 monclient: get_monmap_and_config 2024-04-22T08:15:37.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.756+0000 7f26268a1640 10 monclient: build_initial_monmap 2024-04-22T08:15:37.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.756+0000 7f26268a1640 10 monclient: monmap: 2024-04-22T08:15:37.756 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:15:37.756 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:37.756 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:15:37.757156+0000 2024-04-22T08:15:37.756 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:15:37.757156+0000 2024-04-22T08:15:37.756 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:15:37.756 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:37.756 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:15:37.756 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:15:37.757 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:15:37.757 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:15:37.757 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:15:37.757 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:37.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.757+0000 7f26268a1640 1 Processor -- start 2024-04-22T08:15:37.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.757+0000 7f26268a1640 1 -- start start 2024-04-22T08:15:37.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.757+0000 7f26268a1640 10 monclient: init 2024-04-22T08:15:37.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.757+0000 7f26268a1640 10 monclient: _reopen_session rank -1 2024-04-22T08:15:37.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.757+0000 7f26268a1640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-04-22T08:15:37.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.757+0000 7f26268a1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f262009f270 addr v1:172.21.15.40:6789/0 2024-04-22T08:15:37.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.757+0000 7f26268a1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f262009f650 addr v1:172.21.15.189:6789/0 2024-04-22T08:15:37.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.757+0000 7f26268a1640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f26201387f0 0x7f262013abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:37.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.757+0000 7f26268a1640 10 monclient(hunting): picked mon.noname-a con 0x7f26201387f0 addr v2:172.21.15.40:3300/0 2024-04-22T08:15:37.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.757+0000 7f26268a1640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f262009e510 con 0x7f262009f270 2024-04-22T08:15:37.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.757+0000 7f26268a1640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f262009ba30 con 0x7f262009f650 2024-04-22T08:15:37.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.757+0000 7f26268a1640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:37.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.757+0000 7f26268a1640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f262009b760 con 0x7f26201387f0 2024-04-22T08:15:37.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.757+0000 7f26268a1640 10 monclient(hunting): _renew_subs 2024-04-22T08:15:37.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.757+0000 7f26268a1640 10 monclient(hunting): authenticate will time out at 1011.751557s 2024-04-22T08:15:37.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.757+0000 7f2624e3e640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f26201387f0 0x7f262013abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:37.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.757+0000 7f2624e3e640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f26201387f0 0x7f262013abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58870/0 (socket says 172.21.15.40:58870) 2024-04-22T08:15:37.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.757+0000 7f2624e3e640 1 -- 172.21.15.40:0/1061823730 learned_addr learned my addr 172.21.15.40:0/1061823730 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:15:37.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.757+0000 7f2624e3e640 10 monclient(hunting): get_auth_request con 0x7f26201387f0 auth_method 0 2024-04-22T08:15:37.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.757+0000 7f2624e3e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:37.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.757+0000 7f2624e3e640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:37.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.757+0000 7f2624e3e640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:37.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.758+0000 7f2624e3e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:37.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.758+0000 7f2624e3e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:37.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.758+0000 7f2624e3e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:37.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.758+0000 7f2624e3e640 10 monclient(hunting): handle_auth_done global_id 34371 payload 274 2024-04-22T08:15:37.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.758+0000 7f2624e3e640 1 -- 172.21.15.40:0/1061823730 >> v1:172.21.15.189:6789/0 conn(0x7f262009f650 legacy=0x7f262009fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:15:37.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.758+0000 7f2624e3e640 1 -- 172.21.15.40:0/1061823730 >> v1:172.21.15.40:6789/0 conn(0x7f262009f270 legacy=0x7f262009d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:15:37.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.758+0000 7f2624e3e640 10 monclient: _finish_hunting 0 2024-04-22T08:15:37.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.758+0000 7f2624e3e640 1 monclient: found mon.noname-a 2024-04-22T08:15:37.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.758+0000 7f2624e3e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:15:37.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.758+0000 7f2624e3e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:15:37.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.758+0000 7f2624e3e640 1 -- 172.21.15.40:0/1061823730 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f262009c790 con 0x7f26201387f0 2024-04-22T08:15:37.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.761+0000 7f2624e3e640 1 -- 172.21.15.40:0/1061823730 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26201387f0 msgr2=0x7f262013abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:37.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.761+0000 7f2624e3e640 1 --2- 172.21.15.40:0/1061823730 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26201387f0 0x7f262013abe0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f262006c6d0 tx=0x7f261802df90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cdd9a3dddfd013e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:37.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.761+0000 7f261effd640 1 -- 172.21.15.40:0/1061823730 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2618038600 con 0x7f26201387f0 2024-04-22T08:15:37.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.761+0000 7f261effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:37.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.761+0000 7f261effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:15:37.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.761+0000 7f261effd640 10 monclient: dump: 2024-04-22T08:15:37.769 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:37.769 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:37.769 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:37.769 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:37.769 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:37.769 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:37.769 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:37.769 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:37.770 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:37.770 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:37.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.761+0000 7f261effd640 10 monclient: _finish_auth 0 2024-04-22T08:15:37.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.761+0000 7f261effd640 10 monclient: _check_auth_tickets 2024-04-22T08:15:37.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.761+0000 7f261effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:15:37.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.761+0000 7f26268a1640 5 monclient: authenticate success, global_id 34371 2024-04-22T08:15:37.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.761+0000 7f261effd640 1 -- 172.21.15.40:0/1061823730 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2618032070 con 0x7f26201387f0 2024-04-22T08:15:37.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.761+0000 7f26268a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:15:37.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.761+0000 7f261effd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:15:37.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.761+0000 7f26268a1640 10 monclient: get_monmap_and_config success 2024-04-22T08:15:37.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.761+0000 7f26268a1640 4 set_mon_vals no callback set 2024-04-22T08:15:37.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.761+0000 7f261effd640 1 -- 172.21.15.40:0/1061823730 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2618038980 con 0x7f26201387f0 2024-04-22T08:15:37.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.761+0000 7f261effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:37.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.761+0000 7f261effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:15:37.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.761+0000 7f261effd640 10 monclient: dump: 2024-04-22T08:15:37.771 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:37.771 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:37.771 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:37.771 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:37.771 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:37.771 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:37.771 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:37.771 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:37.771 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:37.771 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:37.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.761+0000 7f26268a1640 10 monclient: shutdown 2024-04-22T08:15:37.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.761+0000 7f26268a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:15:37.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.761+0000 7f26268a1640 1 -- 172.21.15.40:0/1061823730 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26201387f0 msgr2=0x7f262013abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:37.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.761+0000 7f26268a1640 1 --2- 172.21.15.40:0/1061823730 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26201387f0 0x7f262013abe0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f262006c6d0 tx=0x7f261802df90 comp rx=0 tx=0).stop 2024-04-22T08:15:37.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.761+0000 7f26268a1640 1 -- 172.21.15.40:0/1061823730 shutdown_connections 2024-04-22T08:15:37.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.761+0000 7f26268a1640 1 --2- 172.21.15.40:0/1061823730 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26201387f0 0x7f262013abe0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:37.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.761+0000 7f26268a1640 1 -- 172.21.15.40:0/1061823730 >> 172.21.15.40:0/1061823730 conn(0x7f26200a0ae0 msgr2=0x7f26200a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:15:37.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.761+0000 7f26268a1640 1 -- 172.21.15.40:0/1061823730 shutdown_connections 2024-04-22T08:15:37.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.761+0000 7f26268a1640 1 -- 172.21.15.40:0/1061823730 wait complete. 2024-04-22T08:15:37.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.762+0000 7f26268a1640 10 monclient: build_initial_monmap 2024-04-22T08:15:37.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.762+0000 7f26268a1640 10 monclient: monmap: 2024-04-22T08:15:37.772 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:15:37.772 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:15:37.772 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:15:37.772 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:15:37.773 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:15:37.773 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:37.773 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:15:37.773 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:15:37.773 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:15:37.773 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:37.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.762+0000 7f26268a1640 1 Processor -- start 2024-04-22T08:15:37.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.762+0000 7f26268a1640 1 -- start start 2024-04-22T08:15:37.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.762+0000 7f26268a1640 10 monclient: init 2024-04-22T08:15:37.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f26268a1640 10 monclient: _reopen_session rank -1 2024-04-22T08:15:37.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f26268a1640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-22T08:15:37.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f26268a1640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f262009f270 0x7f26201c6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:37.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f26268a1640 10 monclient(hunting): picked mon.noname-c con 0x7f262009f270 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:15:37.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f26268a1640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f262009f650 0x7f26201c6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:37.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f26268a1640 10 monclient(hunting): picked mon.noname-b con 0x7f262009f650 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:15:37.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f26268a1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26201387f0 0x7f26201c73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:37.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f26268a1640 10 monclient(hunting): picked mon.noname-a con 0x7f26201387f0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:15:37.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f26268a1640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:37.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f26268a1640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f262011b5c0 con 0x7f26201387f0 2024-04-22T08:15:37.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f26268a1640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:37.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f26268a1640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f262006ff20 con 0x7f262009f650 2024-04-22T08:15:37.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f26268a1640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:37.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f26268a1640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26200709e0 con 0x7f262009f270 2024-04-22T08:15:37.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f26268a1640 10 monclient(hunting): _renew_subs 2024-04-22T08:15:37.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f26268a1640 10 monclient(hunting): authenticate will time out at 1011.757175s 2024-04-22T08:15:37.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f2624e3e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26201387f0 0x7f26201c73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:37.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f261ffff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f262009f270 0x7f26201c6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:37.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f2624e3e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26201387f0 0x7f26201c73f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58886/0 (socket says 172.21.15.40:58886) 2024-04-22T08:15:37.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f2624e3e640 1 -- 172.21.15.40:0/235732126 learned_addr learned my addr 172.21.15.40:0/235732126 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:15:37.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f2624e3e640 10 monclient(hunting): get_auth_request con 0x7f26201387f0 auth_method 0 2024-04-22T08:15:37.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f2624e3e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:37.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f2624e3e640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:37.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f2624e3e640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:37.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f261ffff640 10 monclient(hunting): get_auth_request con 0x7f262009f270 auth_method 0 2024-04-22T08:15:37.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f261ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:37.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f261ffff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:37.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f261ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:37.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f2624e3e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:37.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f2624e3e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:37.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f2624e3e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:37.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f261f7fe640 1 --2- 172.21.15.40:0/235732126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f262009f650 0x7f26201c6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:37.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f261ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:37.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f261ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:37.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f261ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:37.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f261f7fe640 10 monclient(hunting): get_auth_request con 0x7f262009f650 auth_method 0 2024-04-22T08:15:37.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f261f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:37.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f261f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:37.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f261f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:37.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f261f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:37.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f261f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:37.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f261f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:37.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f2624e3e640 10 monclient(hunting): handle_auth_done global_id 34374 payload 824 2024-04-22T08:15:37.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f2624e3e640 1 -- 172.21.15.40:0/235732126 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f262009f270 msgr2=0x7f26201c6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:37.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f2624e3e640 1 --2- 172.21.15.40:0/235732126 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f262009f270 0x7f26201c6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:37.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f2624e3e640 1 -- 172.21.15.40:0/235732126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f262009f650 msgr2=0x7f26201c6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:37.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f2624e3e640 1 --2- 172.21.15.40:0/235732126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f262009f650 0x7f26201c6eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:37.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f2624e3e640 10 monclient: _finish_hunting 0 2024-04-22T08:15:37.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f2624e3e640 1 monclient: found mon.noname-a 2024-04-22T08:15:37.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f2624e3e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:15:37.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f2624e3e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:15:37.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f2624e3e640 1 -- 172.21.15.40:0/235732126 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f261800bd40 con 0x7f26201387f0 2024-04-22T08:15:37.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.763+0000 7f261ffff640 1 --2- 172.21.15.40:0/235732126 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f262009f270 0x7f26201c6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:15:37.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.766+0000 7f2624e3e640 1 -- 172.21.15.40:0/235732126 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26201387f0 msgr2=0x7f26201c73f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:37.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.766+0000 7f2624e3e640 1 --2- 172.21.15.40:0/235732126 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26201387f0 0x7f26201c73f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f261802e5a0 tx=0x7f2618004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:37.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.766+0000 7f261cff9640 1 -- 172.21.15.40:0/235732126 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2618043850 con 0x7f26201387f0 2024-04-22T08:15:37.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.766+0000 7f261cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:37.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.766+0000 7f261cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:15:37.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.766+0000 7f261cff9640 10 monclient: dump: 2024-04-22T08:15:37.778 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:37.778 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:37.778 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:37.778 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:37.778 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:37.778 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:37.778 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:37.778 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:37.778 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:37.778 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:37.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.766+0000 7f261cff9640 10 monclient: _finish_auth 0 2024-04-22T08:15:37.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.766+0000 7f261cff9640 10 monclient: _check_auth_tickets 2024-04-22T08:15:37.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.766+0000 7f261cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:15:37.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.766+0000 7f261cff9640 1 -- 172.21.15.40:0/235732126 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26180439b0 con 0x7f26201387f0 2024-04-22T08:15:37.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.766+0000 7f261cff9640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:15:37.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.766+0000 7f261cff9640 1 -- 172.21.15.40:0/235732126 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2618043d60 con 0x7f26201387f0 2024-04-22T08:15:37.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.766+0000 7f26268a1640 5 monclient: authenticate success, global_id 34374 2024-04-22T08:15:37.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.766+0000 7f26268a1640 10 monclient: _renew_subs 2024-04-22T08:15:37.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.766+0000 7f26268a1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:37.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.766+0000 7f26268a1640 1 -- 172.21.15.40:0/235732126 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f262009bc10 con 0x7f26201387f0 2024-04-22T08:15:37.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.766+0000 7f261e7fc640 4 set_mon_vals no callback set 2024-04-22T08:15:37.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.766+0000 7f261cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:37.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.766+0000 7f261cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:15:37.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.766+0000 7f261cff9640 10 monclient: dump: 2024-04-22T08:15:37.779 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:37.780 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:37.780 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:37.780 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:37.780 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:37.780 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:37.780 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:37.780 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:37.780 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:37.780 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:37.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.766+0000 7f26268a1640 10 monclient: _renew_subs 2024-04-22T08:15:37.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.766+0000 7f26268a1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:37.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.766+0000 7f26268a1640 1 -- 172.21.15.40:0/235732126 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f262011baf0 con 0x7f26201387f0 2024-04-22T08:15:37.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.767+0000 7f26268a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:37.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.767+0000 7f26268a1640 1 --2- 172.21.15.40:0/235732126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2620134cc0 0x7f26201350a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:37.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.767+0000 7f26268a1640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:37.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.767+0000 7f261cff9640 1 -- 172.21.15.40:0/235732126 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f2618041410 con 0x7f26201387f0 2024-04-22T08:15:37.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.767+0000 7f26268a1640 1 -- 172.21.15.40:0/235732126 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26200667a0 con 0x7f2620134cc0 2024-04-22T08:15:37.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.767+0000 7f261ffff640 1 --2- 172.21.15.40:0/235732126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2620134cc0 0x7f26201350a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:37.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.767+0000 7f261cff9640 1 --2- 172.21.15.40:0/235732126 >> v2:172.21.15.189:6800/30168 conn(0x7f260c0377d0 0x7f260c039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:37.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.767+0000 7f261ffff640 10 monclient: get_auth_request con 0x7f2620134cc0 auth_method 0 2024-04-22T08:15:37.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.767+0000 7f261ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:37.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.767+0000 7f261ffff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:37.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.767+0000 7f261ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:37.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.767+0000 7f261cff9640 1 -- 172.21.15.40:0/235732126 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f261806f100 con 0x7f26201387f0 2024-04-22T08:15:37.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.767+0000 7f261ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:37.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.767+0000 7f261ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:37.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.767+0000 7f261ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:37.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.767+0000 7f261f7fe640 1 --2- 172.21.15.40:0/235732126 >> v2:172.21.15.189:6800/30168 conn(0x7f260c0377d0 0x7f260c039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:37.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.767+0000 7f261f7fe640 10 monclient: get_auth_request con 0x7f260c0377d0 auth_method 0 2024-04-22T08:15:37.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.767+0000 7f261ffff640 10 monclient(hunting): handle_auth_done global_id 24577 payload 824 2024-04-22T08:15:37.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.767+0000 7f261ffff640 1 -- 172.21.15.40:0/235732126 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f262009e9e0 con 0x7f2620134cc0 2024-04-22T08:15:37.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.770+0000 7f261ffff640 1 -- 172.21.15.40:0/235732126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2620134cc0 msgr2=0x7f26201350a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:37.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.770+0000 7f261ffff640 1 --2- 172.21.15.40:0/235732126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2620134cc0 0x7f26201350a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f261000fb10 tx=0x7f26100044c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:37.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.770+0000 7f261f7fe640 1 --2- 172.21.15.40:0/235732126 >> v2:172.21.15.189:6800/30168 conn(0x7f260c0377d0 0x7f260c039c90 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f262009efc0 tx=0x7f2614007e30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:37.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.770+0000 7f261cff9640 1 -- 172.21.15.40:0/235732126 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f261000dd90 con 0x7f2620134cc0 2024-04-22T08:15:37.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.770+0000 7f261cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:37.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.770+0000 7f261cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:15:37.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.770+0000 7f261cff9640 10 monclient: dump: 2024-04-22T08:15:37.785 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:37.785 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:37.785 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:37.785 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:37.785 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:37.785 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:37.785 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:37.785 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:37.786 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:37.786 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:37.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.770+0000 7f261cff9640 1 -- 172.21.15.40:0/235732126 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f2610014470 con 0x7f2620134cc0 2024-04-22T08:15:37.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.770+0000 7f261cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:37.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.770+0000 7f261cff9640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:15:37.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.770+0000 7f261cff9640 1 -- 172.21.15.40:0/235732126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2620134cc0 msgr2=0x7f26201350a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:37.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.770+0000 7f261cff9640 1 --2- 172.21.15.40:0/235732126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2620134cc0 0x7f26201350a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f261000fb10 tx=0x7f26100044c0 comp rx=0 tx=0).stop 2024-04-22T08:15:37.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.770+0000 7f261cff9640 1 -- 172.21.15.40:0/235732126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2620134cc0 msgr2=0x7f26201350a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:15:37.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.770+0000 7f261cff9640 1 --2- 172.21.15.40:0/235732126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2620134cc0 0x7f26201350a0 secure :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f261000fb10 tx=0x7f26100044c0 comp rx=0 tx=0).stop 2024-04-22T08:15:37.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.777+0000 7f26268a1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:15:37.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.777+0000 7f26268a1640 1 --2- 172.21.15.40:0/235732126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26201355e0 0x7f26201359f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:37.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.777+0000 7f26268a1640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:37.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.777+0000 7f26268a1640 1 -- 172.21.15.40:0/235732126 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f262011b2e0 con 0x7f26201355e0 2024-04-22T08:15:37.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.777+0000 7f261ffff640 1 --2- 172.21.15.40:0/235732126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26201355e0 0x7f26201359f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:37.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.777+0000 7f261ffff640 10 monclient: get_auth_request con 0x7f26201355e0 auth_method 0 2024-04-22T08:15:37.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.777+0000 7f261ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:37.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.777+0000 7f261ffff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:37.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.777+0000 7f261ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:37.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.777+0000 7f261ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:37.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.777+0000 7f261ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:37.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.777+0000 7f261ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:37.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.777+0000 7f261ffff640 10 monclient(hunting): handle_auth_done global_id 24580 payload 824 2024-04-22T08:15:37.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.777+0000 7f261ffff640 1 -- 172.21.15.40:0/235732126 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f262004e9b0 con 0x7f26201355e0 2024-04-22T08:15:37.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.780+0000 7f261ffff640 1 -- 172.21.15.40:0/235732126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26201355e0 msgr2=0x7f26201359f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:37.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.780+0000 7f261ffff640 1 --2- 172.21.15.40:0/235732126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26201355e0 0x7f26201359f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f261000fb10 tx=0x7f2610003760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:37.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.780+0000 7f261cff9640 1 -- 172.21.15.40:0/235732126 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2610024070 con 0x7f26201355e0 2024-04-22T08:15:37.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.780+0000 7f261cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:37.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.780+0000 7f261cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:15:37.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.780+0000 7f261cff9640 10 monclient: dump: 2024-04-22T08:15:37.788 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:37.788 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:37.788 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:37.788 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:37.788 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:37.788 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:37.788 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:37.788 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:37.788 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:37.788 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:37.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.780+0000 7f261cff9640 1 -- 172.21.15.40:0/235732126 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4801 (secure 0 0 0) 0x7f261001d420 con 0x7f26201355e0 2024-04-22T08:15:37.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.780+0000 7f261cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:15:37.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.780+0000 7f261cff9640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:15:37.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.780+0000 7f261cff9640 1 -- 172.21.15.40:0/235732126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26201355e0 msgr2=0x7f26201359f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:37.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.780+0000 7f261cff9640 1 --2- 172.21.15.40:0/235732126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26201355e0 0x7f26201359f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f261000fb10 tx=0x7f2610003760 comp rx=0 tx=0).stop 2024-04-22T08:15:37.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.780+0000 7f261cff9640 1 -- 172.21.15.40:0/235732126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26201355e0 msgr2=0x7f26201359f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:15:37.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.780+0000 7f261cff9640 1 --2- 172.21.15.40:0/235732126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26201355e0 0x7f26201359f0 secure :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f261000fb10 tx=0x7f2610003760 comp rx=0 tx=0).stop 2024-04-22T08:15:37.789 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:15:37.789 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:15:37.789 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:15:37.790 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:15:37.790 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 36, 2024-04-22T08:15:37.790 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:15:37.790 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:15:37.790 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:15:37.790 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:15:37.790 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:37.790 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 4, 2024-04-22T08:15:37.790 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:15:37.790 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:15:37.790 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:15:37.790 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:15:37.790 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:15:37.791 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:15:37.791 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:15:37.791 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:15:37.791 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:15:37.791 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:15:37.791 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:15:37.791 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:15:37.791 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:15:37.791 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:37.791 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:15:37.791 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:15:37.791 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:15:37.792 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:15:37.792 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:15:37.792 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:15:37.792 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:15:37.792 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:15:37.792 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:15:37.792 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:15:37.792 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:15:37.792 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:15:37.792 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:37.792 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:37.792 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:15:37.792 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:15:37.792 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:15:37.793 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:15:37.793 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:15:37.793 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:15:37.793 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:15:37.793 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:15:37.793 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:15:37.793 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:15:37.793 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:15:37.793 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:15:37.793 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:15:37.793 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:15:37.793 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:15:37.793 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:15:37.793 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:15:37.793 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:15:37.794 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:15:37.794 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:15:37.794 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:15:37.794 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:15:37.794 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:15:37.794 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:15:37.794 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:15:37.794 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:15:37.794 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:15:37.794 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:37.794 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:15:37.794 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:37.794 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:15:37.794 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:37.795 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:15:37.795 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:15:37.795 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:15:37.795 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:15:37.795 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:37.795 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:15:37.795 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:15:37.795 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:37.795 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:37.795 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:37.795 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:15:37.795 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:15:37.795 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:37.795 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:37.796 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:37.796 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:37.796 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:15:37.796 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:15:37.796 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:15:37.796 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:15:37.796 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:15:37.796 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:37.796 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:37.796 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:15:37.796 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:15:37.796 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:15:37.796 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:15:37.796 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:37.797 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:15:37.797 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:15:37.797 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:37.797 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:37.797 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:37.797 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:15:37.797 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:15:37.797 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:37.797 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:37.797 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:37.797 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:37.797 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:15:37.797 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:15:37.798 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:15:37.798 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:15:37.798 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:15:37.798 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:37.798 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:37.798 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:15:37.798 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:15:37.798 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:15:37.798 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:15:37.798 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:37.798 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:15:37.798 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:15:37.798 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:37.798 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:37.799 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:37.799 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:15:37.799 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:15:37.799 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:37.799 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:37.799 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:37.799 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:37.799 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:15:37.799 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:15:37.799 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:15:37.799 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:15:37.799 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:15:37.799 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:37.799 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:37.800 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:37.800 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:15:37.800 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:15:37.800 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:37.800 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:15:37.800 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:15:37.800 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:15:37.800 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:37.800 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:37.800 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:15:37.800 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:37.800 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:15:37.800 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:15:37.800 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 2 2024-04-22T08:15:37.800 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:37.801 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:37.801 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:15:37.801 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:37.801 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:15:37.801 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:15:37.801 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 6 2024-04-22T08:15:37.801 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:37.801 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:37.801 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": [ 2024-04-22T08:15:37.801 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:37.801 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:15:37.801 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:15:37.801 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:15:37.801 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:37.802 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:37.802 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:37.802 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:15:37.802 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:15:37.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.781+0000 7f26268a1640 1 -- 172.21.15.40:0/235732126 >> v2:172.21.15.189:6800/30168 conn(0x7f260c0377d0 msgr2=0x7f260c039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:37.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.781+0000 7f26268a1640 1 --2- 172.21.15.40:0/235732126 >> v2:172.21.15.189:6800/30168 conn(0x7f260c0377d0 0x7f260c039c90 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f262009efc0 tx=0x7f2614007e30 comp rx=0 tx=0).stop 2024-04-22T08:15:37.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.781+0000 7f26268a1640 10 monclient: shutdown 2024-04-22T08:15:37.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.781+0000 7f26268a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:15:37.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.781+0000 7f26268a1640 1 -- 172.21.15.40:0/235732126 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26201387f0 msgr2=0x7f26201c73f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:37.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.781+0000 7f26268a1640 1 --2- 172.21.15.40:0/235732126 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26201387f0 0x7f26201c73f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f261802e5a0 tx=0x7f2618004a40 comp rx=0 tx=0).stop 2024-04-22T08:15:37.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.781+0000 7f2624e3e640 1 -- 172.21.15.40:0/235732126 reap_dead start 2024-04-22T08:15:37.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.781+0000 7f2624e3e640 1 -- 172.21.15.40:0/235732126 reap_dead start 2024-04-22T08:15:37.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.781+0000 7f26268a1640 1 -- 172.21.15.40:0/235732126 shutdown_connections 2024-04-22T08:15:37.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.781+0000 7f26268a1640 1 -- 172.21.15.40:0/235732126 >> 172.21.15.40:0/235732126 conn(0x7f26200a0ae0 msgr2=0x7f26200710c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:15:37.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.781+0000 7f26268a1640 1 -- 172.21.15.40:0/235732126 shutdown_connections 2024-04-22T08:15:37.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.781+0000 7f26268a1640 1 -- 172.21.15.40:0/235732126 wait complete. 2024-04-22T08:15:37.804 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:15:37.804 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-22T08:15:37.804 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-22T08:15:37.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.912+0000 7fc18da7d640 10 monclient: get_monmap_and_config 2024-04-22T08:15:37.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.912+0000 7fc18da7d640 10 monclient: build_initial_monmap 2024-04-22T08:15:37.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.912+0000 7fc18da7d640 10 monclient: monmap: 2024-04-22T08:15:37.912 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:15:37.912 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:37.912 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:15:37.913025+0000 2024-04-22T08:15:37.912 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:15:37.913025+0000 2024-04-22T08:15:37.912 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:15:37.912 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:37.912 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:15:37.912 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:15:37.912 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:15:37.912 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:15:37.913 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:15:37.913 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:37.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.912+0000 7fc18da7d640 1 Processor -- start 2024-04-22T08:15:37.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.912+0000 7fc18da7d640 1 -- start start 2024-04-22T08:15:37.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.912+0000 7fc18da7d640 10 monclient: init 2024-04-22T08:15:37.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.913+0000 7fc18da7d640 10 monclient: _reopen_session rank -1 2024-04-22T08:15:37.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.913+0000 7fc18da7d640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-04-22T08:15:37.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.913+0000 7fc18da7d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc18809f2e0 0x7fc18809f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:37.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.913+0000 7fc18da7d640 10 monclient(hunting): picked mon.noname-b con 0x7fc18809f2e0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:15:37.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.913+0000 7fc18da7d640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc18809b760 0x7fc18809fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:37.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.913+0000 7fc18da7d640 10 monclient(hunting): picked mon.noname-c con 0x7fc18809b760 addr v2:172.21.15.189:3300/0 2024-04-22T08:15:37.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.913+0000 7fc18da7d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc188136840 addr v1:172.21.15.40:6789/0 2024-04-22T08:15:37.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.913+0000 7fc18da7d640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc18809cf70 con 0x7fc188136840 2024-04-22T08:15:37.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.913+0000 7fc18da7d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:37.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.913+0000 7fc18da7d640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1880a0140 con 0x7fc18809b760 2024-04-22T08:15:37.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.913+0000 7fc18da7d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:37.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.913+0000 7fc18da7d640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc18809c6d0 con 0x7fc18809f2e0 2024-04-22T08:15:37.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.913+0000 7fc18da7d640 10 monclient(hunting): _renew_subs 2024-04-22T08:15:37.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.913+0000 7fc18da7d640 10 monclient(hunting): authenticate will time out at 1011.907238s 2024-04-22T08:15:37.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.913+0000 7fc1877fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc18809f2e0 0x7fc18809f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:37.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.913+0000 7fc187fff640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fc188136840 0x7fc188138c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:40044/0 (socket says 172.21.15.40:40044) 2024-04-22T08:15:37.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.913+0000 7fc187fff640 1 -- 172.21.15.40:0/2275447388 learned_addr learned my addr 172.21.15.40:0/2275447388 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:15:37.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.913+0000 7fc1877fe640 10 monclient(hunting): get_auth_request con 0x7fc18809f2e0 auth_method 0 2024-04-22T08:15:37.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.913+0000 7fc1877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:37.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.913+0000 7fc1877fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:37.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.913+0000 7fc1877fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:37.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.913+0000 7fc1877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:37.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.913+0000 7fc1877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:37.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.913+0000 7fc1877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:37.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.913+0000 7fc17effd640 1 --2- 172.21.15.40:0/2275447388 >> v2:172.21.15.189:3300/0 conn(0x7fc18809b760 0x7fc18809fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:37.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.913+0000 7fc17effd640 10 monclient(hunting): get_auth_request con 0x7fc18809b760 auth_method 0 2024-04-22T08:15:37.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.913+0000 7fc17effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:37.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.913+0000 7fc17effd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:37.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.913+0000 7fc17effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:37.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.913+0000 7fc1877fe640 10 monclient(hunting): handle_auth_done global_id 44108 payload 274 2024-04-22T08:15:37.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.913+0000 7fc1877fe640 1 -- 172.21.15.40:0/2275447388 >> v2:172.21.15.189:3300/0 conn(0x7fc18809b760 msgr2=0x7fc18809fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:15:37.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.913+0000 7fc1877fe640 1 --2- 172.21.15.40:0/2275447388 >> v2:172.21.15.189:3300/0 conn(0x7fc18809b760 0x7fc18809fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:37.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.913+0000 7fc1877fe640 1 -- 172.21.15.40:0/2275447388 >> v1:172.21.15.40:6789/0 conn(0x7fc188136840 legacy=0x7fc188138c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:15:37.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.913+0000 7fc1877fe640 10 monclient: _finish_hunting 0 2024-04-22T08:15:37.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.913+0000 7fc1877fe640 1 monclient: found mon.noname-b 2024-04-22T08:15:37.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.913+0000 7fc1877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:15:37.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.913+0000 7fc1877fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:15:37.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.913+0000 7fc1877fe640 1 -- 172.21.15.40:0/2275447388 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc18809c9b0 con 0x7fc18809f2e0 2024-04-22T08:15:37.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.916+0000 7fc1877fe640 1 -- 172.21.15.40:0/2275447388 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc18809f2e0 msgr2=0x7fc18809f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:37.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.916+0000 7fc1877fe640 1 --2- 172.21.15.40:0/2275447388 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc18809f2e0 0x7fc18809f6c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc1780097a0 tx=0x7fc17802c820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcfb3470ac76d14c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:37.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.916+0000 7fc186ffd640 1 -- 172.21.15.40:0/2275447388 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc178032990 con 0x7fc18809f2e0 2024-04-22T08:15:37.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.916+0000 7fc186ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:37.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.916+0000 7fc186ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:15:37.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.916+0000 7fc186ffd640 10 monclient: dump: 2024-04-22T08:15:37.924 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:37.924 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:37.924 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:37.924 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:37.924 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:37.924 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:37.924 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:37.924 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:37.924 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:37.925 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:37.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.916+0000 7fc186ffd640 10 monclient: _finish_auth 0 2024-04-22T08:15:37.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.916+0000 7fc186ffd640 10 monclient: _check_auth_tickets 2024-04-22T08:15:37.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.916+0000 7fc186ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:15:37.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.916+0000 7fc186ffd640 1 -- 172.21.15.40:0/2275447388 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc178032af0 con 0x7fc18809f2e0 2024-04-22T08:15:37.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.916+0000 7fc18da7d640 5 monclient: authenticate success, global_id 44108 2024-04-22T08:15:37.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.916+0000 7fc18da7d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:15:37.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.916+0000 7fc186ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:15:37.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.916+0000 7fc18da7d640 10 monclient: get_monmap_and_config success 2024-04-22T08:15:37.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.916+0000 7fc186ffd640 1 -- 172.21.15.40:0/2275447388 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc178032e80 con 0x7fc18809f2e0 2024-04-22T08:15:37.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.916+0000 7fc186ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:37.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.916+0000 7fc18da7d640 4 set_mon_vals no callback set 2024-04-22T08:15:37.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.916+0000 7fc186ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:15:37.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.916+0000 7fc186ffd640 10 monclient: dump: 2024-04-22T08:15:37.925 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:37.926 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:37.926 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:37.926 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:37.926 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:37.926 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:37.926 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:37.926 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:37.926 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:37.926 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:37.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.917+0000 7fc18da7d640 10 monclient: shutdown 2024-04-22T08:15:37.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.917+0000 7fc18da7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:15:37.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.917+0000 7fc18da7d640 1 -- 172.21.15.40:0/2275447388 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc18809f2e0 msgr2=0x7fc18809f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:37.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.917+0000 7fc18da7d640 1 --2- 172.21.15.40:0/2275447388 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc18809f2e0 0x7fc18809f6c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc1780097a0 tx=0x7fc17802c820 comp rx=0 tx=0).stop 2024-04-22T08:15:37.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.917+0000 7fc18da7d640 1 -- 172.21.15.40:0/2275447388 shutdown_connections 2024-04-22T08:15:37.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.917+0000 7fc18da7d640 1 --2- 172.21.15.40:0/2275447388 >> v2:172.21.15.189:3300/0 conn(0x7fc18809b760 0x7fc18809fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:37.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.917+0000 7fc18da7d640 1 --2- 172.21.15.40:0/2275447388 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc18809f2e0 0x7fc18809f6c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:37.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.917+0000 7fc18da7d640 1 -- 172.21.15.40:0/2275447388 >> 172.21.15.40:0/2275447388 conn(0x7fc1880a0b50 msgr2=0x7fc1880a0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:15:37.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.917+0000 7fc18da7d640 1 -- 172.21.15.40:0/2275447388 shutdown_connections 2024-04-22T08:15:37.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.917+0000 7fc18da7d640 1 -- 172.21.15.40:0/2275447388 wait complete. 2024-04-22T08:15:37.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.918+0000 7fc18da7d640 10 monclient: build_initial_monmap 2024-04-22T08:15:37.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.918+0000 7fc18da7d640 10 monclient: monmap: 2024-04-22T08:15:37.927 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:15:37.927 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:15:37.927 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:15:37.927 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:15:37.927 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:15:37.927 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:37.927 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:15:37.928 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:15:37.928 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:15:37.928 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:37.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.918+0000 7fc18da7d640 1 Processor -- start 2024-04-22T08:15:37.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.918+0000 7fc18da7d640 1 -- start start 2024-04-22T08:15:37.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.918+0000 7fc18da7d640 10 monclient: init 2024-04-22T08:15:37.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.918+0000 7fc18da7d640 10 monclient: _reopen_session rank -1 2024-04-22T08:15:37.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.918+0000 7fc18da7d640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:15:37.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.918+0000 7fc18da7d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc18809f2e0 0x7fc1881c5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:37.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.918+0000 7fc18da7d640 10 monclient(hunting): picked mon.noname-a con 0x7fc18809f2e0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:15:37.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.918+0000 7fc18da7d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc188136840 0x7fc1881c5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:37.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.918+0000 7fc18da7d640 10 monclient(hunting): picked mon.noname-b con 0x7fc188136840 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:15:37.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.918+0000 7fc18da7d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc188130d80 0x7fc188131210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:37.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.918+0000 7fc18da7d640 10 monclient(hunting): picked mon.noname-c con 0x7fc188130d80 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:15:37.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.918+0000 7fc18da7d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:37.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.918+0000 7fc18da7d640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc18809bb50 con 0x7fc18809f2e0 2024-04-22T08:15:37.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.918+0000 7fc18da7d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:37.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.918+0000 7fc18da7d640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc188070fc0 con 0x7fc188136840 2024-04-22T08:15:37.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.918+0000 7fc18da7d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:37.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.918+0000 7fc18da7d640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1880712a0 con 0x7fc188130d80 2024-04-22T08:15:37.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.918+0000 7fc18da7d640 10 monclient(hunting): _renew_subs 2024-04-22T08:15:37.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.918+0000 7fc18da7d640 10 monclient(hunting): authenticate will time out at 1011.912754s 2024-04-22T08:15:37.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.919+0000 7fc1877fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc18809f2e0 0x7fc1881c5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:37.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.919+0000 7fc187fff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc188130d80 0x7fc188131210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:37.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.919+0000 7fc187fff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc188130d80 0x7fc188131210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:60852/0 (socket says 172.21.15.40:60852) 2024-04-22T08:15:37.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.919+0000 7fc1877fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc18809f2e0 0x7fc1881c5690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58898/0 (socket says 172.21.15.40:58898) 2024-04-22T08:15:37.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.919+0000 7fc187fff640 1 -- 172.21.15.40:0/2558798848 learned_addr learned my addr 172.21.15.40:0/2558798848 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:15:37.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.919+0000 7fc187fff640 10 monclient(hunting): get_auth_request con 0x7fc188130d80 auth_method 0 2024-04-22T08:15:37.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.919+0000 7fc187fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:37.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.919+0000 7fc187fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:37.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.919+0000 7fc187fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:37.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.919+0000 7fc1877fe640 10 monclient(hunting): get_auth_request con 0x7fc18809f2e0 auth_method 0 2024-04-22T08:15:37.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.919+0000 7fc1877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:37.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.919+0000 7fc1877fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:37.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.919+0000 7fc1877fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:37.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.919+0000 7fc187fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:37.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.919+0000 7fc187fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:37.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.919+0000 7fc187fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:37.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.919+0000 7fc1877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:37.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.919+0000 7fc1877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:37.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.919+0000 7fc17effd640 1 --2- 172.21.15.40:0/2558798848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc188136840 0x7fc1881c5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:37.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.919+0000 7fc1877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:37.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.919+0000 7fc17effd640 10 monclient(hunting): get_auth_request con 0x7fc188136840 auth_method 0 2024-04-22T08:15:37.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.919+0000 7fc17effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:37.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.919+0000 7fc17effd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:37.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.919+0000 7fc17effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:37.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.919+0000 7fc17effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:37.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.919+0000 7fc17effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:37.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.919+0000 7fc17effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:37.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.919+0000 7fc187fff640 10 monclient(hunting): handle_auth_done global_id 44111 payload 824 2024-04-22T08:15:37.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.919+0000 7fc187fff640 1 -- 172.21.15.40:0/2558798848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc188136840 msgr2=0x7fc1881c5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:37.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.919+0000 7fc187fff640 1 --2- 172.21.15.40:0/2558798848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc188136840 0x7fc1881c5bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:37.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.919+0000 7fc187fff640 1 -- 172.21.15.40:0/2558798848 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc18809f2e0 msgr2=0x7fc1881c5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:37.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.919+0000 7fc187fff640 1 --2- 172.21.15.40:0/2558798848 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc18809f2e0 0x7fc1881c5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:37.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.919+0000 7fc187fff640 10 monclient: _finish_hunting 0 2024-04-22T08:15:37.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.919+0000 7fc187fff640 1 monclient: found mon.noname-c 2024-04-22T08:15:37.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.919+0000 7fc187fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:15:37.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.919+0000 7fc187fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:15:37.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.919+0000 7fc187fff640 1 -- 172.21.15.40:0/2558798848 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc178009400 con 0x7fc188130d80 2024-04-22T08:15:37.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.919+0000 7fc1877fe640 1 --2- 172.21.15.40:0/2558798848 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc18809f2e0 0x7fc1881c5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:15:37.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.921+0000 7fc187fff640 1 -- 172.21.15.40:0/2558798848 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc188130d80 msgr2=0x7fc188131210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:37.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.921+0000 7fc187fff640 1 --2- 172.21.15.40:0/2558798848 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc188130d80 0x7fc188131210 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc18000cac0 tx=0x7fc18000cf80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:37.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.921+0000 7fc184ff9640 1 -- 172.21.15.40:0/2558798848 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc18000ae40 con 0x7fc188130d80 2024-04-22T08:15:37.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.921+0000 7fc184ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:37.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.921+0000 7fc184ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:15:37.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.921+0000 7fc184ff9640 10 monclient: dump: 2024-04-22T08:15:37.933 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:37.933 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:37.933 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:37.933 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:37.933 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:37.933 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:37.933 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:37.933 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:37.933 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:37.933 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:37.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.921+0000 7fc184ff9640 10 monclient: _finish_auth 0 2024-04-22T08:15:37.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.921+0000 7fc184ff9640 10 monclient: _check_auth_tickets 2024-04-22T08:15:37.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.921+0000 7fc184ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:15:37.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.921+0000 7fc184ff9640 1 -- 172.21.15.40:0/2558798848 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc180012400 con 0x7fc188130d80 2024-04-22T08:15:37.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.921+0000 7fc184ff9640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:15:37.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.921+0000 7fc184ff9640 1 -- 172.21.15.40:0/2558798848 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1800127b0 con 0x7fc188130d80 2024-04-22T08:15:37.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.921+0000 7fc18da7d640 5 monclient: authenticate success, global_id 44111 2024-04-22T08:15:37.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.921+0000 7fc184ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:37.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.921+0000 7fc184ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:15:37.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.921+0000 7fc184ff9640 10 monclient: dump: 2024-04-22T08:15:37.934 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:37.934 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:37.934 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:37.934 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:37.934 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:37.934 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:37.934 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:37.934 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:37.935 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:37.935 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:37.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.921+0000 7fc1867fc640 4 set_mon_vals no callback set 2024-04-22T08:15:37.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.921+0000 7fc18da7d640 10 monclient: _renew_subs 2024-04-22T08:15:37.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.921+0000 7fc18da7d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:15:37.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.921+0000 7fc18da7d640 1 -- 172.21.15.40:0/2558798848 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc18809cd40 con 0x7fc188130d80 2024-04-22T08:15:37.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.922+0000 7fc18da7d640 10 monclient: _renew_subs 2024-04-22T08:15:37.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.922+0000 7fc18da7d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:15:37.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.922+0000 7fc18da7d640 1 -- 172.21.15.40:0/2558798848 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1881c8920 con 0x7fc188130d80 2024-04-22T08:15:37.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.922+0000 7fc184ff9640 1 -- 172.21.15.40:0/2558798848 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fc180012950 con 0x7fc188130d80 2024-04-22T08:15:37.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.922+0000 7fc18da7d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:37.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.922+0000 7fc18da7d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:37.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.922+0000 7fc18da7d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:15:37.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.922+0000 7fc18da7d640 1 -- 172.21.15.40:0/2558798848 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc18804e9b0 con 0x7fc188130d80 2024-04-22T08:15:37.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.923+0000 7fc184ff9640 1 --2- 172.21.15.40:0/2558798848 >> v2:172.21.15.189:6800/30168 conn(0x7fc168037820 0x7fc168039ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:37.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.923+0000 7fc184ff9640 1 -- 172.21.15.40:0/2558798848 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 3898+0+0 (secure 0 0 0) 0x7fc18004a940 con 0x7fc188130d80 2024-04-22T08:15:37.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.923+0000 7fc1877fe640 1 --2- 172.21.15.40:0/2558798848 >> v2:172.21.15.189:6800/30168 conn(0x7fc168037820 0x7fc168039ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:37.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.923+0000 7fc1877fe640 10 monclient: get_auth_request con 0x7fc168037820 auth_method 0 2024-04-22T08:15:37.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.926+0000 7fc1877fe640 1 --2- 172.21.15.40:0/2558798848 >> v2:172.21.15.189:6800/30168 conn(0x7fc168037820 0x7fc168039ce0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc178037000 tx=0x7fc17803e040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:37.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.927+0000 7fc184ff9640 1 -- 172.21.15.40:0/2558798848 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7fc180012c00 con 0x7fc188130d80 2024-04-22T08:15:37.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.927+0000 7fc184ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:37.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:37.927+0000 7fc184ff9640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:15:38.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:38.184+0000 7fc18da7d640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-22T08:15:38.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:38.185+0000 7fc18da7d640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-22T08:15:38.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:38.185+0000 7fc18da7d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:15:38.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:38.185+0000 7fc18da7d640 1 -- 172.21.15.40:0/2558798848 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fc1880a2db0 con 0x7fc188130d80 2024-04-22T08:15:38.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:38.186+0000 7fc184ff9640 1 -- 172.21.15.40:0/2558798848 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fc18003d080 con 0x7fc188130d80 2024-04-22T08:15:38.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:38.186+0000 7fc184ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-22T08:15:38.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:38.186+0000 7fc184ff9640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:15:38.186 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.186+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:15:38.186 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.186+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=11,config=2,health=12,kv=9,logm=66} crc {auth=2309834352,config=1618278320,health=788884224,kv=1350343662,logm=1229271870}) 2024-04-22T08:15:38.187 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.186+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=10,config=2,health=12,kv=7,logm=69} crc {auth=2395462518,config=1618278320,health=788884224,kv=2693807855,logm=2136984815}) 2024-04-22T08:15:38.187 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.186+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:15:38.187 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.186+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=11,config=2,health=12,kv=9,logm=66} crc {auth=2309834352,config=1618278320,health=788884224,kv=1350343662,logm=1229271870}) 2024-04-22T08:15:38.187 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.186+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=10,config=1,health=11,kv=9,logm=69} crc {auth=308302130,config=2457401934,health=3357621518,kv=1350343662,logm=2167693425}) 2024-04-22T08:15:38.187 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.187+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:15:38.188 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.187+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3146427729}) 2024-04-22T08:15:38.188 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.187+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2633313992}) 2024-04-22T08:15:38.188 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.187+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:15:38.188 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.187+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3146427729}) 2024-04-22T08:15:38.188 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.188+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1785736501}) 2024-04-22T08:15:38.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:38.188+0000 7fc18da7d640 1 -- 172.21.15.40:0/2558798848 >> v2:172.21.15.189:6800/30168 conn(0x7fc168037820 msgr2=0x7fc168039ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:38.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:38.188+0000 7fc18da7d640 1 --2- 172.21.15.40:0/2558798848 >> v2:172.21.15.189:6800/30168 conn(0x7fc168037820 0x7fc168039ce0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc178037000 tx=0x7fc17803e040 comp rx=0 tx=0).stop 2024-04-22T08:15:38.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:38.188+0000 7fc18da7d640 10 monclient: shutdown 2024-04-22T08:15:38.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:38.188+0000 7fc18da7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:15:38.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:38.188+0000 7fc18da7d640 1 -- 172.21.15.40:0/2558798848 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc188130d80 msgr2=0x7fc188131210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:38.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:38.188+0000 7fc18da7d640 1 --2- 172.21.15.40:0/2558798848 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc188130d80 0x7fc188131210 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc18000cac0 tx=0x7fc18000cf80 comp rx=0 tx=0).stop 2024-04-22T08:15:38.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:38.188+0000 7fc18da7d640 1 -- 172.21.15.40:0/2558798848 shutdown_connections 2024-04-22T08:15:38.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:38.188+0000 7fc18da7d640 1 --2- 172.21.15.40:0/2558798848 >> v2:172.21.15.189:6800/30168 conn(0x7fc168037820 0x7fc168039ce0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:38.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:38.188+0000 7fc18da7d640 1 --2- 172.21.15.40:0/2558798848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc188136840 0x7fc1881c5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:38.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:38.188+0000 7fc18da7d640 1 --2- 172.21.15.40:0/2558798848 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc188130d80 0x7fc188131210 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:38.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:38.188+0000 7fc18da7d640 1 --2- 172.21.15.40:0/2558798848 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc18809f2e0 0x7fc1881c5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:38.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:38.189+0000 7fc18da7d640 1 -- 172.21.15.40:0/2558798848 >> 172.21.15.40:0/2558798848 conn(0x7fc1880a0b50 msgr2=0x7fc188135c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:15:38.191 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.189+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:15:38.191 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.189+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3356274001}) 2024-04-22T08:15:38.191 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.189+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=830234827}) 2024-04-22T08:15:38.191 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.189+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:15:38.191 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.189+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3356274001}) 2024-04-22T08:15:38.191 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.189+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3310592886}) 2024-04-22T08:15:38.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:38.189+0000 7fc18da7d640 1 -- 172.21.15.40:0/2558798848 shutdown_connections 2024-04-22T08:15:38.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:38.189+0000 7fc18da7d640 1 -- 172.21.15.40:0/2558798848 wait complete. 2024-04-22T08:15:38.193 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.189+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:15:38.194 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.190+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3847894333}) 2024-04-22T08:15:38.194 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.190+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2044694382}) 2024-04-22T08:15:38.194 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.190+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:15:38.194 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.190+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3847894333}) 2024-04-22T08:15:38.194 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.190+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4169054155}) 2024-04-22T08:15:38.194 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.191+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:15:38.194 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.191+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1431151372}) 2024-04-22T08:15:38.194 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.191+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1163864271}) 2024-04-22T08:15:38.194 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.191+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:15:38.194 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.191+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1431151372}) 2024-04-22T08:15:38.194 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.191+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=654048278}) 2024-04-22T08:15:38.194 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.191+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:15:38.194 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.191+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1486495856}) 2024-04-22T08:15:38.195 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.191+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1622917938}) 2024-04-22T08:15:38.195 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.191+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:15:38.195 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.191+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1486495856}) 2024-04-22T08:15:38.195 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.191+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1031840091}) 2024-04-22T08:15:38.195 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.192+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:15:38.195 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.192+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2374643780}) 2024-04-22T08:15:38.195 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.192+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=668958957}) 2024-04-22T08:15:38.195 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.192+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:15:38.195 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.192+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2374643780}) 2024-04-22T08:15:38.195 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.192+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3343443115}) 2024-04-22T08:15:38.195 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.193+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:15:38.195 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.193+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=41,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=8,monmap=2,osd_metadata=6,osd_pg_creating=1,osdmap=18} crc {logm=125369322,mdsmap=1531151339,mgr=878880162,mgr_command_descs=470045235,mgr_metadata=4222303120,mgrstat=2306595999,mon_config_key=4255067121,monmap=3140514009,osd_metadata=4151711904,osd_pg_creating=3102974103,osdmap=4264408449}) 2024-04-22T08:15:38.195 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.193+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=45,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=8,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=13} crc {logm=3308199556,mdsmap=1531151339,mgr=878880162,mgr_command_descs=470045235,mgr_metadata=4222303120,mgrstat=341000996,mon_config_key=4255067121,monmap=3469086636,osd_metadata=4151711904,osd_pg_creating=3102974103,osdmap=1868396987}) 2024-04-22T08:15:38.195 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.193+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:15:38.196 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.193+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=41,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=8,monmap=2,osd_metadata=6,osd_pg_creating=1,osdmap=18} crc {logm=125369322,mdsmap=1531151339,mgr=878880162,mgr_command_descs=470045235,mgr_metadata=4222303120,mgrstat=2306595999,mon_config_key=4255067121,monmap=3140514009,osd_metadata=4151711904,osd_pg_creating=3102974103,osdmap=4264408449}) 2024-04-22T08:15:38.196 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.194+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=42,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=11,mon_config_key=8,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=17} crc {logm=2167143488,mdsmap=1531151339,mgr=878880162,mgr_metadata=4222303120,mgrstat=1533236661,mon_config_key=3293354343,monmap=3469086636,osd_metadata=4151711904,osd_pg_creating=3102974103,osdmap=2330239301}) 2024-04-22T08:15:38.196 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.195+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:15:38.196 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.195+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2007148924}) 2024-04-22T08:15:38.196 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.195+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1302763656}) 2024-04-22T08:15:38.196 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.195+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:15:38.196 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.195+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2007148924}) 2024-04-22T08:15:38.197 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.195+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4070108437}) 2024-04-22T08:15:38.197 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.198+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:15:38.197 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.198+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=56} crc {osdmap=2363904953}) 2024-04-22T08:15:38.198 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.198+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=49} crc {osdmap=1753296140}) 2024-04-22T08:15:38.198 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.198+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:15:38.198 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.198+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=56} crc {osdmap=2363904953}) 2024-04-22T08:15:38.198 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:15:38.198+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=55} crc {osdmap=3913412460}) 2024-04-22T08:15:38.199 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-22T08:15:38.984 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36002]: starting. 2024-04-22T08:15:38.984 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36002]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-36001 2024-04-22T08:15:38.984 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36002]: created object 0... 2024-04-22T08:15:38.984 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36002]: created object 25... 2024-04-22T08:15:38.984 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36002]: created object 49... 2024-04-22T08:15:38.984 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36002]: finishing. 2024-04-22T08:15:38.984 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36002]: shutting down. 2024-04-22T08:15:39.200 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-22T08:15:39.200 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:15:39.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.303+0000 7fbdd15e4640 10 monclient: get_monmap_and_config 2024-04-22T08:15:39.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.303+0000 7fbdd15e4640 10 monclient: build_initial_monmap 2024-04-22T08:15:39.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.303+0000 7fbdd15e4640 10 monclient: monmap: 2024-04-22T08:15:39.303 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:15:39.303 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:39.303 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:15:39.304213+0000 2024-04-22T08:15:39.303 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:15:39.304213+0000 2024-04-22T08:15:39.303 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:15:39.303 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:39.303 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:15:39.303 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:15:39.303 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:15:39.303 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:15:39.304 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:15:39.304 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:39.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.304+0000 7fbdd15e4640 1 Processor -- start 2024-04-22T08:15:39.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.304+0000 7fbdd15e4640 1 -- start start 2024-04-22T08:15:39.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.304+0000 7fbdd15e4640 10 monclient: init 2024-04-22T08:15:39.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.304+0000 7fbdd15e4640 10 monclient: _reopen_session rank -1 2024-04-22T08:15:39.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.304+0000 7fbdd15e4640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-04-22T08:15:39.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.304+0000 7fbdd15e4640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbdcc09f2e0 0x7fbdcc09f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:39.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.304+0000 7fbdd15e4640 10 monclient(hunting): picked mon.noname-c con 0x7fbdcc09f2e0 addr v2:172.21.15.189:3300/0 2024-04-22T08:15:39.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.304+0000 7fbdd15e4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbdcc09b760 addr v1:172.21.15.189:6789/0 2024-04-22T08:15:39.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.304+0000 7fbdd15e4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbdcc09fc00 addr v1:172.21.15.40:6789/0 2024-04-22T08:15:39.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.304+0000 7fbdd15e4640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdcc09cf70 con 0x7fbdcc09fc00 2024-04-22T08:15:39.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.304+0000 7fbdd15e4640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdcc09c6d0 con 0x7fbdcc09b760 2024-04-22T08:15:39.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.304+0000 7fbdd15e4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:39.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.304+0000 7fbdd15e4640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdcc09c9b0 con 0x7fbdcc09f2e0 2024-04-22T08:15:39.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.304+0000 7fbdd15e4640 10 monclient(hunting): _renew_subs 2024-04-22T08:15:39.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.304+0000 7fbdd15e4640 10 monclient(hunting): authenticate will time out at 1013.298456s 2024-04-22T08:15:39.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.304+0000 7fbdcb7fe640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fbdcc09fc00 0x7fbdcc139f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41014/0 (socket says 172.21.15.40:41014) 2024-04-22T08:15:39.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.304+0000 7fbdcb7fe640 1 -- 172.21.15.40:0/339231271 learned_addr learned my addr 172.21.15.40:0/339231271 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:15:39.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.304+0000 7fbdcaffd640 1 --2- 172.21.15.40:0/339231271 >> v2:172.21.15.189:3300/0 conn(0x7fbdcc09f2e0 0x7fbdcc09f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:39.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.304+0000 7fbdcaffd640 10 monclient(hunting): get_auth_request con 0x7fbdcc09f2e0 auth_method 0 2024-04-22T08:15:39.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.304+0000 7fbdcaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:39.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.304+0000 7fbdcaffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:39.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.304+0000 7fbdcaffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:39.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.305+0000 7fbdcaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:39.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.305+0000 7fbdcaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:39.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.305+0000 7fbdcaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:39.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.307+0000 7fbdcb7fe640 1 -- 172.21.15.40:0/339231271 >> v1:172.21.15.40:6789/0 conn(0x7fbdcc09fc00 legacy=0x7fbdcc139f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:39.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.307+0000 7fbdc9ffb640 1 -- 172.21.15.40:0/339231271 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3462176977 0 0) 0x7fbdcc09cf70 con 0x7fbdcc09fc00 2024-04-22T08:15:39.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.307+0000 7fbdc9ffb640 10 monclient(hunting): _negotiate 2024-04-22T08:15:39.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.307+0000 7fbdc9ffb640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:39.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.307+0000 7fbdc9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:39.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.307+0000 7fbdc9ffb640 10 monclient(hunting): my global_id is 34380 2024-04-22T08:15:39.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.307+0000 7fbdc9ffb640 1 -- 172.21.15.40:0/339231271 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdb4003520 con 0x7fbdcc09fc00 2024-04-22T08:15:39.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.307+0000 7fbdca7fc640 1 -- 172.21.15.40:0/339231271 >> v1:172.21.15.189:6789/0 conn(0x7fbdcc09b760 legacy=0x7fbdcc136840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:39.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.307+0000 7fbdcaffd640 10 monclient(hunting): handle_auth_done global_id 24589 payload 274 2024-04-22T08:15:39.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.307+0000 7fbdcaffd640 1 -- 172.21.15.40:0/339231271 >> v1:172.21.15.189:6789/0 conn(0x7fbdcc09b760 legacy=0x7fbdcc136840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:39.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.307+0000 7fbdcaffd640 1 -- 172.21.15.40:0/339231271 >> v1:172.21.15.40:6789/0 conn(0x7fbdcc09fc00 legacy=0x7fbdcc139f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:39.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.307+0000 7fbdcaffd640 10 monclient: _finish_hunting 0 2024-04-22T08:15:39.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.307+0000 7fbdcaffd640 1 monclient: found mon.noname-c 2024-04-22T08:15:39.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.307+0000 7fbdcaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:15:39.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.307+0000 7fbdcaffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.189:3300/0 2024-04-22T08:15:39.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.307+0000 7fbdcaffd640 1 -- 172.21.15.40:0/339231271 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdcc09cc90 con 0x7fbdcc09f2e0 2024-04-22T08:15:39.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.310+0000 7fbdcaffd640 1 -- 172.21.15.40:0/339231271 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbdcc09f2e0 msgr2=0x7fbdcc09f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:39.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.310+0000 7fbdcaffd640 1 --2- 172.21.15.40:0/339231271 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbdcc09f2e0 0x7fbdcc09f6c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fbdbc009e90 tx=0x7fbdbc02d030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57eed9e6248d50ed server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:39.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.310+0000 7fbdc9ffb640 1 -- 172.21.15.40:0/339231271 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdbc035440 con 0x7fbdcc09f2e0 2024-04-22T08:15:39.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.310+0000 7fbdc9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:39.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.310+0000 7fbdc9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:15:39.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.310+0000 7fbdc9ffb640 10 monclient: dump: 2024-04-22T08:15:39.311 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:39.311 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:39.311 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:39.311 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:39.311 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:39.311 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:39.311 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:39.311 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:39.311 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:39.312 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:39.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.310+0000 7fbdc9ffb640 10 monclient: _finish_auth 0 2024-04-22T08:15:39.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.310+0000 7fbdc9ffb640 10 monclient: _check_auth_tickets 2024-04-22T08:15:39.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.310+0000 7fbdc9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:15:39.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.310+0000 7fbdd15e4640 5 monclient: authenticate success, global_id 24589 2024-04-22T08:15:39.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.310+0000 7fbdd15e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:15:39.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.310+0000 7fbdc9ffb640 1 -- 172.21.15.40:0/339231271 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdbc0355a0 con 0x7fbdcc09f2e0 2024-04-22T08:15:39.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.310+0000 7fbdc9ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:15:39.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.310+0000 7fbdd15e4640 10 monclient: get_monmap_and_config success 2024-04-22T08:15:39.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.310+0000 7fbdc9ffb640 1 -- 172.21.15.40:0/339231271 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdbc035910 con 0x7fbdcc09f2e0 2024-04-22T08:15:39.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.310+0000 7fbdd15e4640 4 set_mon_vals no callback set 2024-04-22T08:15:39.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.310+0000 7fbdc9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:39.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.310+0000 7fbdc9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:15:39.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.310+0000 7fbdc9ffb640 10 monclient: dump: 2024-04-22T08:15:39.312 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:39.313 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:39.313 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:39.313 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:39.313 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:39.313 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:39.313 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:39.313 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:39.313 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:39.313 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:39.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.311+0000 7fbdd15e4640 10 monclient: shutdown 2024-04-22T08:15:39.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.311+0000 7fbdd15e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:15:39.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.311+0000 7fbdd15e4640 1 -- 172.21.15.40:0/339231271 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbdcc09f2e0 msgr2=0x7fbdcc09f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:39.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.311+0000 7fbdd15e4640 1 --2- 172.21.15.40:0/339231271 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbdcc09f2e0 0x7fbdcc09f6c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fbdbc009e90 tx=0x7fbdbc02d030 comp rx=0 tx=0).stop 2024-04-22T08:15:39.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.311+0000 7fbdd15e4640 1 -- 172.21.15.40:0/339231271 shutdown_connections 2024-04-22T08:15:39.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.311+0000 7fbdd15e4640 1 --2- 172.21.15.40:0/339231271 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbdcc09f2e0 0x7fbdcc09f6c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:39.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.311+0000 7fbdd15e4640 1 -- 172.21.15.40:0/339231271 >> 172.21.15.40:0/339231271 conn(0x7fbdcc0a0b50 msgr2=0x7fbdcc0a0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:15:39.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.311+0000 7fbdd15e4640 1 -- 172.21.15.40:0/339231271 shutdown_connections 2024-04-22T08:15:39.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.311+0000 7fbdd15e4640 1 -- 172.21.15.40:0/339231271 wait complete. 2024-04-22T08:15:39.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.311+0000 7fbdd15e4640 10 monclient: build_initial_monmap 2024-04-22T08:15:39.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.311+0000 7fbdd15e4640 10 monclient: monmap: 2024-04-22T08:15:39.319 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:15:39.319 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:15:39.319 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:15:39.319 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:15:39.319 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:15:39.319 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:39.319 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:15:39.320 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:15:39.320 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:15:39.320 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:39.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.312+0000 7fbdd15e4640 1 Processor -- start 2024-04-22T08:15:39.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.312+0000 7fbdd15e4640 1 -- start start 2024-04-22T08:15:39.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.312+0000 7fbdd15e4640 10 monclient: init 2024-04-22T08:15:39.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.312+0000 7fbdd15e4640 10 monclient: _reopen_session rank -1 2024-04-22T08:15:39.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.312+0000 7fbdd15e4640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-22T08:15:39.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.312+0000 7fbdd15e4640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbdcc09f2e0 0x7fbdcc1c2350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:39.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.312+0000 7fbdd15e4640 10 monclient(hunting): picked mon.noname-c con 0x7fbdcc09f2e0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:15:39.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.312+0000 7fbdd15e4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbdcc09fc00 0x7fbdcc1c84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:39.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.312+0000 7fbdd15e4640 10 monclient(hunting): picked mon.noname-a con 0x7fbdcc09fc00 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:15:39.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.312+0000 7fbdd15e4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbdcc1c8a00 0x7fbdcc1c4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:39.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.312+0000 7fbdd15e4640 10 monclient(hunting): picked mon.noname-b con 0x7fbdcc1c8a00 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:15:39.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.312+0000 7fbdd15e4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:39.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.312+0000 7fbdd15e4640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdcc1a7a70 con 0x7fbdcc09fc00 2024-04-22T08:15:39.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.312+0000 7fbdd15e4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:39.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.312+0000 7fbdd15e4640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdcc0712d0 con 0x7fbdcc1c8a00 2024-04-22T08:15:39.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.312+0000 7fbdd15e4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:39.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.312+0000 7fbdd15e4640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdcc1a9410 con 0x7fbdcc09f2e0 2024-04-22T08:15:39.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.312+0000 7fbdd15e4640 10 monclient(hunting): _renew_subs 2024-04-22T08:15:39.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.312+0000 7fbdd15e4640 10 monclient(hunting): authenticate will time out at 1013.306711s 2024-04-22T08:15:39.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.313+0000 7fbdca7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbdcc09fc00 0x7fbdcc1c84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:39.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.313+0000 7fbdcaffd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbdcc09f2e0 0x7fbdcc1c2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:39.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.313+0000 7fbdca7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbdcc09fc00 0x7fbdcc1c84c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33536/0 (socket says 172.21.15.40:33536) 2024-04-22T08:15:39.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.313+0000 7fbdca7fc640 1 -- 172.21.15.40:0/2146874720 learned_addr learned my addr 172.21.15.40:0/2146874720 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:15:39.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.313+0000 7fbdca7fc640 10 monclient(hunting): get_auth_request con 0x7fbdcc09fc00 auth_method 0 2024-04-22T08:15:39.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.313+0000 7fbdcaffd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbdcc09f2e0 0x7fbdcc1c2350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:44726/0 (socket says 172.21.15.40:44726) 2024-04-22T08:15:39.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.313+0000 7fbdca7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:39.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.313+0000 7fbdca7fc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:39.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.313+0000 7fbdca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:39.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.313+0000 7fbdcaffd640 10 monclient(hunting): get_auth_request con 0x7fbdcc09f2e0 auth_method 0 2024-04-22T08:15:39.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.313+0000 7fbdcaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:39.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.313+0000 7fbdcaffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:39.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.313+0000 7fbdcaffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:39.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.313+0000 7fbdca7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:39.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.313+0000 7fbdca7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:39.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.313+0000 7fbdca7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:39.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.313+0000 7fbdcaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:39.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.313+0000 7fbdcaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:39.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.313+0000 7fbdcaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:39.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.313+0000 7fbdcb7fe640 1 --2- 172.21.15.40:0/2146874720 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbdcc1c8a00 0x7fbdcc1c4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:39.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.313+0000 7fbdcb7fe640 10 monclient(hunting): get_auth_request con 0x7fbdcc1c8a00 auth_method 0 2024-04-22T08:15:39.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.313+0000 7fbdcb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:39.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.313+0000 7fbdcb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:39.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.313+0000 7fbdcb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:39.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.313+0000 7fbdca7fc640 10 monclient(hunting): handle_auth_done global_id 34383 payload 824 2024-04-22T08:15:39.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.313+0000 7fbdca7fc640 1 -- 172.21.15.40:0/2146874720 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbdcc09f2e0 msgr2=0x7fbdcc1c2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:39.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.313+0000 7fbdca7fc640 1 --2- 172.21.15.40:0/2146874720 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbdcc09f2e0 0x7fbdcc1c2350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:39.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.313+0000 7fbdca7fc640 1 -- 172.21.15.40:0/2146874720 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbdcc1c8a00 msgr2=0x7fbdcc1c4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:39.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.313+0000 7fbdca7fc640 1 --2- 172.21.15.40:0/2146874720 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbdcc1c8a00 0x7fbdcc1c4f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:39.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.313+0000 7fbdca7fc640 10 monclient: _finish_hunting 0 2024-04-22T08:15:39.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.313+0000 7fbdca7fc640 1 monclient: found mon.noname-a 2024-04-22T08:15:39.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.313+0000 7fbdca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:15:39.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.313+0000 7fbdca7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:15:39.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.313+0000 7fbdca7fc640 1 -- 172.21.15.40:0/2146874720 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdbc009a70 con 0x7fbdcc09fc00 2024-04-22T08:15:39.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.313+0000 7fbdcaffd640 1 --2- 172.21.15.40:0/2146874720 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbdcc09f2e0 0x7fbdcc1c2350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:15:39.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.315+0000 7fbdca7fc640 1 -- 172.21.15.40:0/2146874720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbdcc09fc00 msgr2=0x7fbdcc1c84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:39.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.315+0000 7fbdca7fc640 1 --2- 172.21.15.40:0/2146874720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbdcc09fc00 0x7fbdcc1c84c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fbdc000c450 tx=0x7fbdc000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:39.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.315+0000 7fbdca7fc640 1 --2- 172.21.15.40:0/2146874720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbdcc09fc00 0x7fbdcc1c84c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fbdc000c450 tx=0x7fbdc000c910 comp rx=0 tx=0).queue_received will delay after 1.71377e+09 on 0x7fbdc0005b70 mon_map magic: 0 v1 2024-04-22T08:15:39.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.315+0000 7fbdb37fe640 1 -- 172.21.15.40:0/2146874720 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc0007690 con 0x7fbdcc09fc00 2024-04-22T08:15:39.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.315+0000 7fbdb37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:39.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.315+0000 7fbdb37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:15:39.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.315+0000 7fbdb37fe640 10 monclient: dump: 2024-04-22T08:15:39.324 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:39.324 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:39.324 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:39.324 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:39.324 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:39.324 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:39.324 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:39.325 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:39.325 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:39.325 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:39.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.315+0000 7fbdb37fe640 10 monclient: _finish_auth 0 2024-04-22T08:15:39.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.315+0000 7fbdb37fe640 10 monclient: _check_auth_tickets 2024-04-22T08:15:39.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.315+0000 7fbdb37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:15:39.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.315+0000 7fbdd15e4640 5 monclient: authenticate success, global_id 34383 2024-04-22T08:15:39.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.315+0000 7fbdb37fe640 1 -- 172.21.15.40:0/2146874720 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdc00057c0 con 0x7fbdcc09fc00 2024-04-22T08:15:39.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.315+0000 7fbdb37fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:15:39.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.315+0000 7fbdd15e4640 10 monclient: _renew_subs 2024-04-22T08:15:39.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.315+0000 7fbdd15e4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:39.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.315+0000 7fbdd15e4640 1 -- 172.21.15.40:0/2146874720 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdcc1a96f0 con 0x7fbdcc09fc00 2024-04-22T08:15:39.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.315+0000 7fbdc97fa640 4 set_mon_vals no callback set 2024-04-22T08:15:39.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.315+0000 7fbdd15e4640 10 monclient: _renew_subs 2024-04-22T08:15:39.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.315+0000 7fbdd15e4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:39.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.315+0000 7fbdd15e4640 1 -- 172.21.15.40:0/2146874720 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdcc1a8970 con 0x7fbdcc09fc00 2024-04-22T08:15:39.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.316+0000 7fbdb37fe640 1 -- 172.21.15.40:0/2146874720 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc0005b70 con 0x7fbdcc09fc00 2024-04-22T08:15:39.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.316+0000 7fbdb37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:39.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.316+0000 7fbdb37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:15:39.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.316+0000 7fbdb37fe640 10 monclient: dump: 2024-04-22T08:15:39.326 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:39.326 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:39.326 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:39.326 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:39.326 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:39.326 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:39.326 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:39.326 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:39.326 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:39.327 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:39.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.316+0000 7fbdb37fe640 1 -- 172.21.15.40:0/2146874720 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fbdc0005d10 con 0x7fbdcc09fc00 2024-04-22T08:15:39.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.316+0000 7fbdd15e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:39.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.316+0000 7fbdd15e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:39.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.316+0000 7fbdd15e4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:39.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.316+0000 7fbdd15e4640 1 -- 172.21.15.40:0/2146874720 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdcc1a9ae0 con 0x7fbdcc09fc00 2024-04-22T08:15:39.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.316+0000 7fbdb37fe640 1 --2- 172.21.15.40:0/2146874720 >> v2:172.21.15.189:6800/30168 conn(0x7fbdb8037820 0x7fbdb8039ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:39.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.317+0000 7fbdcaffd640 1 --2- 172.21.15.40:0/2146874720 >> v2:172.21.15.189:6800/30168 conn(0x7fbdb8037820 0x7fbdb8039ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:39.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.317+0000 7fbdcaffd640 10 monclient: get_auth_request con 0x7fbdb8037820 auth_method 0 2024-04-22T08:15:39.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.319+0000 7fbdcaffd640 1 --2- 172.21.15.40:0/2146874720 >> v2:172.21.15.189:6800/30168 conn(0x7fbdb8037820 0x7fbdb8039ce0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc09d390 tx=0x7fbdbc031000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:39.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:39.321+0000 7fbdb37fe640 1 -- 172.21.15.40:0/2146874720 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 3898+0+0 (secure 0 0 0) 0x7fbdc00499a0 con 0x7fbdcc09fc00 2024-04-22T08:15:39.857 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[36003]: starting. 2024-04-22T08:15:39.857 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[36003]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-36001 2024-04-22T08:15:39.857 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[36003]: shutting down. 2024-04-22T08:15:40.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.144+0000 7fbdb37fe640 1 -- 172.21.15.40:0/2146874720 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7fbdc001bb10 con 0x7fbdcc09fc00 2024-04-22T08:15:40.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.144+0000 7fbdb37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:40.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.144+0000 7fbdb37fe640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:15:40.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.397+0000 7fbdd15e4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:15:40.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.397+0000 7fbdd15e4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:15:40.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.397+0000 7fbdd15e4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:40.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.397+0000 7fbdd15e4640 1 -- 172.21.15.40:0/2146874720 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fbdcc1a98e0 con 0x7fbdcc09fc00 2024-04-22T08:15:40.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.398+0000 7fbdb37fe640 1 -- 172.21.15.40:0/2146874720 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7fbdc0017070 con 0x7fbdcc09fc00 2024-04-22T08:15:40.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.398+0000 7fbdb37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:15:40.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.398+0000 7fbdb37fe640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:15:40.398 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":36,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:15:40.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.400+0000 7fbdd15e4640 1 -- 172.21.15.40:0/2146874720 >> v2:172.21.15.189:6800/30168 conn(0x7fbdb8037820 msgr2=0x7fbdb8039ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:40.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.400+0000 7fbdd15e4640 1 --2- 172.21.15.40:0/2146874720 >> v2:172.21.15.189:6800/30168 conn(0x7fbdb8037820 0x7fbdb8039ce0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc09d390 tx=0x7fbdbc031000 comp rx=0 tx=0).stop 2024-04-22T08:15:40.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.400+0000 7fbdd15e4640 10 monclient: shutdown 2024-04-22T08:15:40.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.400+0000 7fbdd15e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:15:40.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.400+0000 7fbdd15e4640 1 -- 172.21.15.40:0/2146874720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbdcc09fc00 msgr2=0x7fbdcc1c84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:40.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.400+0000 7fbdd15e4640 1 --2- 172.21.15.40:0/2146874720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbdcc09fc00 0x7fbdcc1c84c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fbdc000c450 tx=0x7fbdc000c910 comp rx=0 tx=0).stop 2024-04-22T08:15:40.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.400+0000 7fbdd15e4640 1 -- 172.21.15.40:0/2146874720 shutdown_connections 2024-04-22T08:15:40.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.400+0000 7fbdd15e4640 1 --2- 172.21.15.40:0/2146874720 >> v2:172.21.15.189:6800/30168 conn(0x7fbdb8037820 0x7fbdb8039ce0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:40.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.400+0000 7fbdd15e4640 1 --2- 172.21.15.40:0/2146874720 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbdcc1c8a00 0x7fbdcc1c4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:40.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.400+0000 7fbdd15e4640 1 --2- 172.21.15.40:0/2146874720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbdcc09fc00 0x7fbdcc1c84c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:40.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.400+0000 7fbdd15e4640 1 --2- 172.21.15.40:0/2146874720 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbdcc09f2e0 0x7fbdcc1c2350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:40.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.400+0000 7fbdd15e4640 1 -- 172.21.15.40:0/2146874720 >> 172.21.15.40:0/2146874720 conn(0x7fbdcc0a0b50 msgr2=0x7fbdcc06f370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:15:40.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.400+0000 7fbdd15e4640 1 -- 172.21.15.40:0/2146874720 shutdown_connections 2024-04-22T08:15:40.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.400+0000 7fbdd15e4640 1 -- 172.21.15.40:0/2146874720 wait complete. 2024-04-22T08:15:40.410 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-22T08:15:40.410 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-22T08:15:40.410 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-22T08:15:40.411 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-22T08:15:40.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.511+0000 7f96b700b640 10 monclient: get_monmap_and_config 2024-04-22T08:15:40.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.511+0000 7f96b700b640 10 monclient: build_initial_monmap 2024-04-22T08:15:40.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.511+0000 7f96b700b640 10 monclient: monmap: 2024-04-22T08:15:40.511 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:15:40.511 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:40.511 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:15:40.511925+0000 2024-04-22T08:15:40.511 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:15:40.511925+0000 2024-04-22T08:15:40.511 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:15:40.511 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:40.511 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:15:40.511 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:15:40.512 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:15:40.512 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:15:40.512 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:15:40.512 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:40.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.511+0000 7f96b700b640 1 Processor -- start 2024-04-22T08:15:40.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.511+0000 7f96b700b640 1 -- start start 2024-04-22T08:15:40.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.511+0000 7f96b700b640 10 monclient: init 2024-04-22T08:15:40.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.512+0000 7f96b700b640 10 monclient: _reopen_session rank -1 2024-04-22T08:15:40.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.512+0000 7f96b700b640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-04-22T08:15:40.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.512+0000 7f96b700b640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f96b00a1ba0 0x7f96b00a1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:40.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.512+0000 7f96b700b640 10 monclient(hunting): picked mon.noname-c con 0x7f96b00a1ba0 addr v2:172.21.15.189:3300/0 2024-04-22T08:15:40.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.512+0000 7f96b700b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f96b0136230 addr v1:172.21.15.40:6789/0 2024-04-22T08:15:40.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.512+0000 7f96b700b640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f96b0136670 0x7f96b00a3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:40.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.512+0000 7f96b700b640 10 monclient(hunting): picked mon.noname-b con 0x7f96b0136670 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:15:40.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.512+0000 7f96b700b640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96b009fe80 con 0x7f96b0136230 2024-04-22T08:15:40.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.512+0000 7f96b700b640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:40.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.512+0000 7f96b700b640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96b00722a0 con 0x7f96b00a1ba0 2024-04-22T08:15:40.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.512+0000 7f96b700b640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:40.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.512+0000 7f96b700b640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96b00a4290 con 0x7f96b0136670 2024-04-22T08:15:40.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.512+0000 7f96b700b640 10 monclient(hunting): _renew_subs 2024-04-22T08:15:40.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.512+0000 7f96b700b640 10 monclient(hunting): authenticate will time out at 1014.506077s 2024-04-22T08:15:40.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.512+0000 7f96b55a8640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f96b0136670 0x7f96b00a3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:40.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.512+0000 7f96affff640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f96b0136230 0x7f96b00a2d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41016/0 (socket says 172.21.15.40:41016) 2024-04-22T08:15:40.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.512+0000 7f96affff640 1 -- 172.21.15.40:0/261791003 learned_addr learned my addr 172.21.15.40:0/261791003 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:15:40.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.512+0000 7f96b55a8640 10 monclient(hunting): get_auth_request con 0x7f96b0136670 auth_method 0 2024-04-22T08:15:40.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.512+0000 7f96b55a8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:40.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.512+0000 7f96b55a8640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:40.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.512+0000 7f96b55a8640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:40.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.512+0000 7f96b55a8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:40.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.512+0000 7f96b55a8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:40.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.512+0000 7f96b55a8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:40.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.512+0000 7f96b4da7640 1 --2- 172.21.15.40:0/261791003 >> v2:172.21.15.189:3300/0 conn(0x7f96b00a1ba0 0x7f96b00a1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:40.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.512+0000 7f96b4da7640 10 monclient(hunting): get_auth_request con 0x7f96b00a1ba0 auth_method 0 2024-04-22T08:15:40.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.512+0000 7f96b4da7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:40.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.512+0000 7f96b4da7640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:40.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.512+0000 7f96b4da7640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:40.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.512+0000 7f96b55a8640 10 monclient(hunting): handle_auth_done global_id 44132 payload 274 2024-04-22T08:15:40.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.512+0000 7f96b55a8640 1 -- 172.21.15.40:0/261791003 >> v2:172.21.15.189:3300/0 conn(0x7f96b00a1ba0 msgr2=0x7f96b00a1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:15:40.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.512+0000 7f96b55a8640 1 --2- 172.21.15.40:0/261791003 >> v2:172.21.15.189:3300/0 conn(0x7f96b00a1ba0 0x7f96b00a1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:40.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.512+0000 7f96b55a8640 1 -- 172.21.15.40:0/261791003 >> v1:172.21.15.40:6789/0 conn(0x7f96b0136230 legacy=0x7f96b00a2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:15:40.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.512+0000 7f96b55a8640 10 monclient: _finish_hunting 0 2024-04-22T08:15:40.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.512+0000 7f96b55a8640 1 monclient: found mon.noname-b 2024-04-22T08:15:40.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.512+0000 7f96b55a8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:15:40.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.512+0000 7f96b55a8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:15:40.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.512+0000 7f96b55a8640 1 -- 172.21.15.40:0/261791003 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b00a0160 con 0x7f96b0136670 2024-04-22T08:15:40.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.512+0000 7f96b4da7640 1 --2- 172.21.15.40:0/261791003 >> v2:172.21.15.189:3300/0 conn(0x7f96b00a1ba0 0x7f96b00a1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T08:15:40.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.515+0000 7f96b55a8640 1 -- 172.21.15.40:0/261791003 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f96b0136670 msgr2=0x7f96b00a3490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:40.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.515+0000 7f96b55a8640 1 --2- 172.21.15.40:0/261791003 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f96b0136670 0x7f96b00a3490 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f96a8009d90 tx=0x7f96a802bd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbc5839f19761c87 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:40.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.516+0000 7f96af7fe640 1 -- 172.21.15.40:0/261791003 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a802cd40 con 0x7f96b0136670 2024-04-22T08:15:40.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.516+0000 7f96af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:40.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.516+0000 7f96af7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:15:40.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.516+0000 7f96af7fe640 10 monclient: dump: 2024-04-22T08:15:40.525 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:40.525 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:40.525 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:40.525 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:40.525 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:40.525 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:40.525 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:40.525 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:40.526 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:40.526 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:40.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.516+0000 7f96af7fe640 10 monclient: _finish_auth 0 2024-04-22T08:15:40.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.516+0000 7f96af7fe640 10 monclient: _check_auth_tickets 2024-04-22T08:15:40.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.516+0000 7f96af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:15:40.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.516+0000 7f96af7fe640 1 -- 172.21.15.40:0/261791003 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96a8034440 con 0x7f96b0136670 2024-04-22T08:15:40.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.516+0000 7f96b700b640 5 monclient: authenticate success, global_id 44132 2024-04-22T08:15:40.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.516+0000 7f96b700b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:15:40.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.516+0000 7f96af7fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:15:40.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.516+0000 7f96b700b640 10 monclient: get_monmap_and_config success 2024-04-22T08:15:40.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.516+0000 7f96af7fe640 1 -- 172.21.15.40:0/261791003 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a802cea0 con 0x7f96b0136670 2024-04-22T08:15:40.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.516+0000 7f96af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:40.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.516+0000 7f96b700b640 4 set_mon_vals no callback set 2024-04-22T08:15:40.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.516+0000 7f96af7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:15:40.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.516+0000 7f96af7fe640 10 monclient: dump: 2024-04-22T08:15:40.527 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:40.527 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:40.527 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:40.527 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:40.527 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:40.527 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:40.527 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:40.527 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:40.527 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:40.527 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:40.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.516+0000 7f96b700b640 10 monclient: shutdown 2024-04-22T08:15:40.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.516+0000 7f96b700b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:15:40.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.516+0000 7f96b700b640 1 -- 172.21.15.40:0/261791003 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f96b0136670 msgr2=0x7f96b00a3490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:40.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.516+0000 7f96b700b640 1 --2- 172.21.15.40:0/261791003 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f96b0136670 0x7f96b00a3490 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f96a8009d90 tx=0x7f96a802bd80 comp rx=0 tx=0).stop 2024-04-22T08:15:40.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.516+0000 7f96b700b640 1 -- 172.21.15.40:0/261791003 shutdown_connections 2024-04-22T08:15:40.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.516+0000 7f96b700b640 1 --2- 172.21.15.40:0/261791003 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f96b0136670 0x7f96b00a3490 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:40.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.516+0000 7f96b700b640 1 --2- 172.21.15.40:0/261791003 >> v2:172.21.15.189:3300/0 conn(0x7f96b00a1ba0 0x7f96b00a1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:40.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.516+0000 7f96b700b640 1 -- 172.21.15.40:0/261791003 >> 172.21.15.40:0/261791003 conn(0x7f96b00a4e80 msgr2=0x7f96b00a5290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:15:40.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.516+0000 7f96b700b640 1 -- 172.21.15.40:0/261791003 shutdown_connections 2024-04-22T08:15:40.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.516+0000 7f96b700b640 1 -- 172.21.15.40:0/261791003 wait complete. 2024-04-22T08:15:40.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.517+0000 7f96b700b640 10 monclient: build_initial_monmap 2024-04-22T08:15:40.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.517+0000 7f96b700b640 10 monclient: monmap: 2024-04-22T08:15:40.528 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:15:40.528 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:15:40.528 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:15:40.528 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:15:40.528 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:15:40.528 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:40.529 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:15:40.529 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:15:40.529 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:15:40.529 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:40.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.517+0000 7f96b700b640 1 Processor -- start 2024-04-22T08:15:40.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.517+0000 7f96b700b640 1 -- start start 2024-04-22T08:15:40.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.517+0000 7f96b700b640 10 monclient: init 2024-04-22T08:15:40.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96b700b640 10 monclient: _reopen_session rank -1 2024-04-22T08:15:40.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96b700b640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-22T08:15:40.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96b700b640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f96b00a1ba0 0x7f96b011e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:40.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96b700b640 10 monclient(hunting): picked mon.noname-c con 0x7f96b00a1ba0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:15:40.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96b700b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96b0136230 0x7f96b011ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:40.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96b700b640 10 monclient(hunting): picked mon.noname-a con 0x7f96b0136230 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:15:40.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96b700b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96b0136670 0x7f96b011f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:40.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96b700b640 10 monclient(hunting): picked mon.noname-b con 0x7f96b0136670 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:15:40.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96b700b640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:40.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96b700b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b01206d0 con 0x7f96b0136230 2024-04-22T08:15:40.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96b700b640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:40.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96b700b640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b0122280 con 0x7f96b0136670 2024-04-22T08:15:40.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96b700b640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:40.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96b700b640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96b00a2710 con 0x7f96b00a1ba0 2024-04-22T08:15:40.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96b700b640 10 monclient(hunting): _renew_subs 2024-04-22T08:15:40.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96b700b640 10 monclient(hunting): authenticate will time out at 1014.511896s 2024-04-22T08:15:40.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96b4da7640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f96b00a1ba0 0x7f96b011e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:40.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96affff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96b0136230 0x7f96b011ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:40.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96b4da7640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f96b00a1ba0 0x7f96b011e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:44746/0 (socket says 172.21.15.40:44746) 2024-04-22T08:15:40.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96b4da7640 1 -- 172.21.15.40:0/2988329375 learned_addr learned my addr 172.21.15.40:0/2988329375 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:15:40.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96b4da7640 10 monclient(hunting): get_auth_request con 0x7f96b00a1ba0 auth_method 0 2024-04-22T08:15:40.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96b4da7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:40.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96b4da7640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:40.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96b4da7640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:40.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96affff640 10 monclient(hunting): get_auth_request con 0x7f96b0136230 auth_method 0 2024-04-22T08:15:40.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:40.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96affff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:40.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96affff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:40.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96b4da7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:40.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96b4da7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:40.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96b4da7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:40.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:40.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:40.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:40.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96b55a8640 1 --2- 172.21.15.40:0/2988329375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96b0136670 0x7f96b011f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:40.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96b55a8640 10 monclient(hunting): get_auth_request con 0x7f96b0136670 auth_method 0 2024-04-22T08:15:40.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96b55a8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:40.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96b55a8640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:40.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96b55a8640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:40.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96b4da7640 10 monclient(hunting): handle_auth_done global_id 44135 payload 824 2024-04-22T08:15:40.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96b4da7640 1 -- 172.21.15.40:0/2988329375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96b0136670 msgr2=0x7f96b011f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:40.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96b4da7640 1 --2- 172.21.15.40:0/2988329375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96b0136670 0x7f96b011f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:40.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96b4da7640 1 -- 172.21.15.40:0/2988329375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96b0136230 msgr2=0x7f96b011ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:40.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96b4da7640 1 --2- 172.21.15.40:0/2988329375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96b0136230 0x7f96b011ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:40.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96b4da7640 10 monclient: _finish_hunting 0 2024-04-22T08:15:40.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96b4da7640 1 monclient: found mon.noname-c 2024-04-22T08:15:40.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96b4da7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:15:40.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96b4da7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:15:40.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96b4da7640 1 -- 172.21.15.40:0/2988329375 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96a8009a70 con 0x7f96b00a1ba0 2024-04-22T08:15:40.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96b55a8640 1 --2- 172.21.15.40:0/2988329375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96b0136670 0x7f96b011f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T08:15:40.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.518+0000 7f96affff640 1 --2- 172.21.15.40:0/2988329375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96b0136230 0x7f96b011ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:15:40.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.520+0000 7f96b4da7640 1 -- 172.21.15.40:0/2988329375 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f96b00a1ba0 msgr2=0x7f96b011e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:40.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.520+0000 7f96b4da7640 1 --2- 172.21.15.40:0/2988329375 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f96b00a1ba0 0x7f96b011e9d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f96a400e760 tx=0x7f96a400ec20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:40.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.520+0000 7f96ad7fa640 1 -- 172.21.15.40:0/2988329375 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a4004230 con 0x7f96b00a1ba0 2024-04-22T08:15:40.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.520+0000 7f96ad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:40.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.520+0000 7f96ad7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:15:40.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.520+0000 7f96ad7fa640 10 monclient: dump: 2024-04-22T08:15:40.533 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:40.533 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:40.533 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:40.534 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:40.534 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:40.534 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:40.534 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:40.534 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:40.534 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:40.534 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:40.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.520+0000 7f96ad7fa640 10 monclient: _finish_auth 0 2024-04-22T08:15:40.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.520+0000 7f96ad7fa640 10 monclient: _check_auth_tickets 2024-04-22T08:15:40.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.520+0000 7f96ad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:15:40.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.521+0000 7f96b700b640 5 monclient: authenticate success, global_id 44135 2024-04-22T08:15:40.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.521+0000 7f96ad7fa640 1 -- 172.21.15.40:0/2988329375 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96a4004390 con 0x7f96b00a1ba0 2024-04-22T08:15:40.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.521+0000 7f96ad7fa640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:15:40.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.521+0000 7f96ad7fa640 1 -- 172.21.15.40:0/2988329375 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a400cb00 con 0x7f96b00a1ba0 2024-04-22T08:15:40.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.521+0000 7f96ad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:40.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.521+0000 7f96ad7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:15:40.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.521+0000 7f96ad7fa640 10 monclient: dump: 2024-04-22T08:15:40.535 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:40.535 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:40.535 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:40.535 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:40.535 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:40.535 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:40.535 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:40.535 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:40.535 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:40.535 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:40.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.521+0000 7f96b700b640 10 monclient: _renew_subs 2024-04-22T08:15:40.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.521+0000 7f96aeffd640 4 set_mon_vals no callback set 2024-04-22T08:15:40.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.521+0000 7f96b700b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:15:40.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.521+0000 7f96b700b640 1 -- 172.21.15.40:0/2988329375 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96b0135110 con 0x7f96b00a1ba0 2024-04-22T08:15:40.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.521+0000 7f96b700b640 10 monclient: _renew_subs 2024-04-22T08:15:40.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.521+0000 7f96b700b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:15:40.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.521+0000 7f96b700b640 1 -- 172.21.15.40:0/2988329375 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96b01215c0 con 0x7f96b00a1ba0 2024-04-22T08:15:40.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.521+0000 7f96b700b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:40.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.521+0000 7f96b700b640 1 --2- 172.21.15.40:0/2988329375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96b002c380 0x7f96b0060ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:40.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.521+0000 7f96b700b640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:40.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.521+0000 7f96b700b640 1 -- 172.21.15.40:0/2988329375 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b0066650 con 0x7f96b002c380 2024-04-22T08:15:40.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.521+0000 7f96affff640 1 --2- 172.21.15.40:0/2988329375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96b002c380 0x7f96b0060ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:40.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.521+0000 7f96affff640 10 monclient: get_auth_request con 0x7f96b002c380 auth_method 0 2024-04-22T08:15:40.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.521+0000 7f96affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:40.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.521+0000 7f96affff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:40.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.521+0000 7f96affff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:40.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.521+0000 7f96ad7fa640 1 -- 172.21.15.40:0/2988329375 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f96a400cca0 con 0x7f96b00a1ba0 2024-04-22T08:15:40.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.521+0000 7f96affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:40.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.521+0000 7f96affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:40.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.522+0000 7f96affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:40.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.522+0000 7f96affff640 10 monclient(hunting): handle_auth_done global_id 34404 payload 824 2024-04-22T08:15:40.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.522+0000 7f96affff640 1 -- 172.21.15.40:0/2988329375 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f96b00a2500 con 0x7f96b002c380 2024-04-22T08:15:40.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.522+0000 7f96ad7fa640 1 --2- 172.21.15.40:0/2988329375 >> v2:172.21.15.189:6800/30168 conn(0x7f9694037820 0x7f9694039ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:40.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.522+0000 7f96ad7fa640 1 -- 172.21.15.40:0/2988329375 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f96a4049ef0 con 0x7f96b00a1ba0 2024-04-22T08:15:40.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.522+0000 7f96b55a8640 1 --2- 172.21.15.40:0/2988329375 >> v2:172.21.15.189:6800/30168 conn(0x7f9694037820 0x7f9694039ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:40.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.522+0000 7f96b55a8640 10 monclient: get_auth_request con 0x7f9694037820 auth_method 0 2024-04-22T08:15:40.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.524+0000 7f96affff640 1 -- 172.21.15.40:0/2988329375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96b002c380 msgr2=0x7f96b0060ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:40.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.524+0000 7f96affff640 1 --2- 172.21.15.40:0/2988329375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96b002c380 0x7f96b0060ff0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f969c007fd0 tx=0x7f969c007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:40.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.524+0000 7f96ad7fa640 1 -- 172.21.15.40:0/2988329375 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969c00fc40 con 0x7f96b002c380 2024-04-22T08:15:40.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.524+0000 7f96ad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:40.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.524+0000 7f96ad7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:15:40.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.524+0000 7f96ad7fa640 10 monclient: dump: 2024-04-22T08:15:40.538 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:40.538 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:40.538 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:40.538 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:40.538 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:40.538 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:40.538 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:40.538 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:40.539 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:40.539 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:40.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.524+0000 7f96ad7fa640 1 -- 172.21.15.40:0/2988329375 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f969c00fda0 con 0x7f96b002c380 2024-04-22T08:15:40.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.524+0000 7f96ad7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:40.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.524+0000 7f96ad7fa640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:15:40.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.524+0000 7f96ad7fa640 1 -- 172.21.15.40:0/2988329375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96b002c380 msgr2=0x7f96b0060ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:40.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.524+0000 7f96ad7fa640 1 --2- 172.21.15.40:0/2988329375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96b002c380 0x7f96b0060ff0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f969c007fd0 tx=0x7f969c007700 comp rx=0 tx=0).stop 2024-04-22T08:15:40.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.524+0000 7f96ad7fa640 1 -- 172.21.15.40:0/2988329375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96b002c380 msgr2=0x7f96b0060ff0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:15:40.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.524+0000 7f96ad7fa640 1 --2- 172.21.15.40:0/2988329375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96b002c380 0x7f96b0060ff0 secure :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f969c007fd0 tx=0x7f969c007700 comp rx=0 tx=0).stop 2024-04-22T08:15:40.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.524+0000 7f96b55a8640 1 --2- 172.21.15.40:0/2988329375 >> v2:172.21.15.189:6800/30168 conn(0x7f9694037820 0x7f9694039ce0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f96b00a4500 tx=0x7f96a8030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:40.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.531+0000 7f96b700b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:15:40.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.531+0000 7f96b700b640 1 --2- 172.21.15.40:0/2988329375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96b0061530 0x7f96b01d4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:40.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.531+0000 7f96b700b640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:40.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.531+0000 7f96b700b640 1 -- 172.21.15.40:0/2988329375 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b002c760 con 0x7f96b0061530 2024-04-22T08:15:40.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.531+0000 7f96affff640 1 --2- 172.21.15.40:0/2988329375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96b0061530 0x7f96b01d4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:40.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.531+0000 7f96affff640 10 monclient: get_auth_request con 0x7f96b0061530 auth_method 0 2024-04-22T08:15:40.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.531+0000 7f96affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:40.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.531+0000 7f96affff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:40.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.531+0000 7f96affff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:40.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.531+0000 7f96affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:40.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.531+0000 7f96affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:40.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.531+0000 7f96affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:40.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.531+0000 7f96affff640 10 monclient(hunting): handle_auth_done global_id 34407 payload 824 2024-04-22T08:15:40.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.531+0000 7f96affff640 1 -- 172.21.15.40:0/2988329375 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f96b0121a30 con 0x7f96b0061530 2024-04-22T08:15:40.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.533+0000 7f96affff640 1 -- 172.21.15.40:0/2988329375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96b0061530 msgr2=0x7f96b01d4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:40.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.533+0000 7f96affff640 1 --2- 172.21.15.40:0/2988329375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96b0061530 0x7f96b01d4af0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f969c007fd0 tx=0x7f969c004940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:40.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.534+0000 7f96ad7fa640 1 -- 172.21.15.40:0/2988329375 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969c024070 con 0x7f96b0061530 2024-04-22T08:15:40.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.534+0000 7f96ad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:40.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.534+0000 7f96ad7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:15:40.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.534+0000 7f96ad7fa640 10 monclient: dump: 2024-04-22T08:15:40.543 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:40.543 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:40.543 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:40.543 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:40.543 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:40.543 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:40.543 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:40.543 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:40.543 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:40.543 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:40.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.534+0000 7f96ad7fa640 1 -- 172.21.15.40:0/2988329375 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4633 (secure 0 0 0) 0x7f969c0058b0 con 0x7f96b0061530 2024-04-22T08:15:40.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.534+0000 7f96ad7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:15:40.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.534+0000 7f96ad7fa640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:15:40.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.534+0000 7f96ad7fa640 1 -- 172.21.15.40:0/2988329375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96b0061530 msgr2=0x7f96b01d4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:40.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.534+0000 7f96ad7fa640 1 --2- 172.21.15.40:0/2988329375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96b0061530 0x7f96b01d4af0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f969c007fd0 tx=0x7f969c004940 comp rx=0 tx=0).stop 2024-04-22T08:15:40.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.534+0000 7f96ad7fa640 1 -- 172.21.15.40:0/2988329375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96b0061530 msgr2=0x7f96b01d4af0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:15:40.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.534+0000 7f96ad7fa640 1 --2- 172.21.15.40:0/2988329375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96b0061530 0x7f96b01d4af0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:40.544 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:15:40.544 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:15:40.544 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:15:40.544 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "leader", 2024-04-22T08:15:40.545 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 36, 2024-04-22T08:15:40.545 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:15:40.545 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:15:40.545 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:15:40.545 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:15:40.545 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:40.545 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 7, 2024-04-22T08:15:40.545 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:15:40.545 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:15:40.545 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:15:40.545 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:15:40.545 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:15:40.545 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:15:40.545 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:15:40.545 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:15:40.546 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:15:40.546 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:15:40.546 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:15:40.546 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:15:40.546 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:15:40.546 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:40.546 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:15:40.546 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:15:40.546 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:15:40.546 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:15:40.546 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:15:40.546 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:15:40.546 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:15:40.546 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:15:40.547 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:15:40.547 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:15:40.547 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:15:40.547 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:15:40.547 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:40.547 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:40.547 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:15:40.547 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:15:40.547 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:15:40.547 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:15:40.547 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:15:40.547 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:15:40.547 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:15:40.547 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:15:40.548 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:15:40.548 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:15:40.548 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:15:40.548 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:15:40.548 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:15:40.548 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:15:40.548 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:15:40.548 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:15:40.548 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:15:40.548 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:15:40.548 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:15:40.548 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:15:40.548 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:15:40.548 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:15:40.549 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:15:40.549 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:15:40.549 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:15:40.549 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:15:40.549 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:15:40.549 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:40.549 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:15:40.549 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:40.549 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:15:40.549 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:40.549 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:15:40.549 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:15:40.549 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:15:40.549 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:15:40.549 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:40.550 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:15:40.550 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:15:40.550 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:40.550 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:40.550 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:40.550 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:15:40.550 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:15:40.550 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:40.550 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:40.550 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:40.550 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:40.550 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:15:40.550 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:15:40.550 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:15:40.551 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:15:40.551 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:15:40.551 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:40.551 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:40.551 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:15:40.551 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:15:40.551 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:15:40.551 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:15:40.551 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:40.551 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:15:40.551 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:15:40.551 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:40.551 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:40.551 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:40.551 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:15:40.552 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:15:40.552 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:40.552 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:40.552 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:40.552 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:40.552 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:15:40.552 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:15:40.552 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:15:40.552 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:15:40.552 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:15:40.552 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:40.552 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:40.552 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:15:40.553 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:15:40.553 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:15:40.553 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:15:40.553 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:40.553 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:15:40.553 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:15:40.553 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:40.553 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:40.553 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:40.553 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:15:40.553 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:15:40.553 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:40.553 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:40.553 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:40.553 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:40.554 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:15:40.554 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:15:40.554 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:15:40.554 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:15:40.554 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:15:40.554 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:40.554 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:40.554 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:40.554 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:15:40.554 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:15:40.554 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:40.554 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:15:40.554 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:15:40.554 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:15:40.555 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:40.555 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:40.555 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:15:40.555 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:40.555 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:15:40.555 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:15:40.555 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 4 2024-04-22T08:15:40.555 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:40.555 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:40.555 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:15:40.555 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:40.555 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:15:40.555 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:15:40.555 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 2 2024-04-22T08:15:40.555 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:40.556 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:40.556 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:40.556 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:15:40.556 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:15:40.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.534+0000 7f96b700b640 1 -- 172.21.15.40:0/2988329375 >> v2:172.21.15.189:6800/30168 conn(0x7f9694037820 msgr2=0x7f9694039ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:40.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.534+0000 7f96b700b640 1 --2- 172.21.15.40:0/2988329375 >> v2:172.21.15.189:6800/30168 conn(0x7f9694037820 0x7f9694039ce0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f96b00a4500 tx=0x7f96a8030000 comp rx=0 tx=0).stop 2024-04-22T08:15:40.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.534+0000 7f96b700b640 10 monclient: shutdown 2024-04-22T08:15:40.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.534+0000 7f96b700b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:15:40.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.534+0000 7f96b700b640 1 -- 172.21.15.40:0/2988329375 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f96b00a1ba0 msgr2=0x7f96b011e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:40.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.534+0000 7f96b700b640 1 --2- 172.21.15.40:0/2988329375 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f96b00a1ba0 0x7f96b011e9d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f96a400e760 tx=0x7f96a400ec20 comp rx=0 tx=0).stop 2024-04-22T08:15:40.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.534+0000 7f96b55a8640 1 -- 172.21.15.40:0/2988329375 reap_dead start 2024-04-22T08:15:40.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.534+0000 7f96b55a8640 1 -- 172.21.15.40:0/2988329375 reap_dead start 2024-04-22T08:15:40.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.534+0000 7f96b700b640 1 -- 172.21.15.40:0/2988329375 shutdown_connections 2024-04-22T08:15:40.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.534+0000 7f96b700b640 1 -- 172.21.15.40:0/2988329375 >> 172.21.15.40:0/2988329375 conn(0x7f96b00a4e80 msgr2=0x7f96b00a6e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:15:40.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.534+0000 7f96b700b640 1 -- 172.21.15.40:0/2988329375 shutdown_connections 2024-04-22T08:15:40.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.534+0000 7f96b700b640 1 -- 172.21.15.40:0/2988329375 wait complete. 2024-04-22T08:15:40.558 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:15:40.559 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-22T08:15:40.559 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-22T08:15:40.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.659+0000 7fdc74f34640 10 monclient: get_monmap_and_config 2024-04-22T08:15:40.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.659+0000 7fdc74f34640 10 monclient: build_initial_monmap 2024-04-22T08:15:40.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.659+0000 7fdc74f34640 10 monclient: monmap: 2024-04-22T08:15:40.658 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:15:40.659 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:40.659 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:15:40.659871+0000 2024-04-22T08:15:40.659 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:15:40.659871+0000 2024-04-22T08:15:40.659 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:15:40.659 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:40.659 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:15:40.659 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:15:40.659 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:15:40.659 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:15:40.659 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:15:40.659 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:40.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.659+0000 7fdc74f34640 1 Processor -- start 2024-04-22T08:15:40.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.659+0000 7fdc74f34640 1 -- start start 2024-04-22T08:15:40.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.659+0000 7fdc74f34640 10 monclient: init 2024-04-22T08:15:40.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.660+0000 7fdc74f34640 10 monclient: _reopen_session rank -1 2024-04-22T08:15:40.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.660+0000 7fdc74f34640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-04-22T08:15:40.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.660+0000 7fdc74f34640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdc700a1c60 0x7fdc700a31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:40.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.660+0000 7fdc74f34640 10 monclient(hunting): picked mon.noname-b con 0x7fdc700a1c60 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:15:40.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.660+0000 7fdc74f34640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdc700a2040 addr v1:172.21.15.189:6789/0 2024-04-22T08:15:40.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.660+0000 7fdc74f34640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdc700a24a0 0x7fdc701362c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:40.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.660+0000 7fdc74f34640 10 monclient(hunting): picked mon.noname-c con 0x7fdc700a24a0 addr v2:172.21.15.189:3300/0 2024-04-22T08:15:40.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.660+0000 7fdc74f34640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc700a41f0 con 0x7fdc700a2040 2024-04-22T08:15:40.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.660+0000 7fdc74f34640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:40.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.660+0000 7fdc74f34640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc7009fe90 con 0x7fdc700a24a0 2024-04-22T08:15:40.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.660+0000 7fdc74f34640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:40.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.660+0000 7fdc74f34640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc700a0170 con 0x7fdc700a1c60 2024-04-22T08:15:40.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.660+0000 7fdc74f34640 10 monclient(hunting): _renew_subs 2024-04-22T08:15:40.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.660+0000 7fdc74f34640 10 monclient(hunting): authenticate will time out at 1014.654113s 2024-04-22T08:15:40.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.660+0000 7fdc6e59c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdc700a1c60 0x7fdc700a31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:40.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.660+0000 7fdc6e59c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdc700a1c60 0x7fdc700a31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:44756/0 (socket says 172.21.15.40:44756) 2024-04-22T08:15:40.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.660+0000 7fdc6e59c640 1 -- 172.21.15.40:0/1438215878 learned_addr learned my addr 172.21.15.40:0/1438215878 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:15:40.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.660+0000 7fdc6e59c640 10 monclient(hunting): get_auth_request con 0x7fdc700a1c60 auth_method 0 2024-04-22T08:15:40.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.660+0000 7fdc6e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:40.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.660+0000 7fdc6e59c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:40.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.660+0000 7fdc6e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:40.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.660+0000 7fdc6e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:40.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.660+0000 7fdc6e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:40.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.660+0000 7fdc6e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:40.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.660+0000 7fdc6e59c640 10 monclient(hunting): handle_auth_done global_id 44138 payload 274 2024-04-22T08:15:40.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.660+0000 7fdc6e59c640 1 -- 172.21.15.40:0/1438215878 >> v2:172.21.15.189:3300/0 conn(0x7fdc700a24a0 msgr2=0x7fdc701362c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:15:40.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.660+0000 7fdc6e59c640 1 --2- 172.21.15.40:0/1438215878 >> v2:172.21.15.189:3300/0 conn(0x7fdc700a24a0 0x7fdc701362c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:40.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.660+0000 7fdc6e59c640 1 -- 172.21.15.40:0/1438215878 >> v1:172.21.15.189:6789/0 conn(0x7fdc700a2040 legacy=0x7fdc70135bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:15:40.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.660+0000 7fdc6e59c640 10 monclient: _finish_hunting 0 2024-04-22T08:15:40.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.660+0000 7fdc6e59c640 1 monclient: found mon.noname-b 2024-04-22T08:15:40.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.660+0000 7fdc6e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:15:40.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.660+0000 7fdc6e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:15:40.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.660+0000 7fdc6e59c640 1 -- 172.21.15.40:0/1438215878 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc700a0450 con 0x7fdc700a1c60 2024-04-22T08:15:40.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.663+0000 7fdc6e59c640 1 -- 172.21.15.40:0/1438215878 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdc700a1c60 msgr2=0x7fdc700a31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:40.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.663+0000 7fdc6e59c640 1 --2- 172.21.15.40:0/1438215878 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdc700a1c60 0x7fdc700a31d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fdc68009e50 tx=0x7fdc6802bdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47ba8edab3086ad5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:40.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.663+0000 7fdc6d59a640 1 -- 172.21.15.40:0/1438215878 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc6802cb60 con 0x7fdc700a1c60 2024-04-22T08:15:40.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.663+0000 7fdc6d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:40.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.663+0000 7fdc6d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:15:40.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.663+0000 7fdc6d59a640 10 monclient: dump: 2024-04-22T08:15:40.669 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:40.669 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:40.669 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:40.669 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:40.669 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:40.669 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:40.670 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:40.670 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:40.670 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:40.670 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:40.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.663+0000 7fdc6d59a640 10 monclient: _finish_auth 0 2024-04-22T08:15:40.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.663+0000 7fdc6d59a640 10 monclient: _check_auth_tickets 2024-04-22T08:15:40.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.663+0000 7fdc6d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:15:40.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.663+0000 7fdc6d59a640 1 -- 172.21.15.40:0/1438215878 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc6802ccc0 con 0x7fdc700a1c60 2024-04-22T08:15:40.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.663+0000 7fdc6d59a640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:15:40.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.663+0000 7fdc6d59a640 1 -- 172.21.15.40:0/1438215878 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc68034610 con 0x7fdc700a1c60 2024-04-22T08:15:40.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.663+0000 7fdc74f34640 5 monclient: authenticate success, global_id 44138 2024-04-22T08:15:40.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.663+0000 7fdc74f34640 10 monclient: get_monmap_and_config success 2024-04-22T08:15:40.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.663+0000 7fdc74f34640 4 set_mon_vals no callback set 2024-04-22T08:15:40.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.663+0000 7fdc6d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:40.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.663+0000 7fdc6d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:15:40.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.663+0000 7fdc6d59a640 10 monclient: dump: 2024-04-22T08:15:40.671 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:40.671 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:40.671 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:40.671 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:40.671 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:40.671 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:40.671 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:40.671 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:40.671 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:40.671 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:40.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.664+0000 7fdc74f34640 10 monclient: shutdown 2024-04-22T08:15:40.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.664+0000 7fdc74f34640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:15:40.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.664+0000 7fdc74f34640 1 -- 172.21.15.40:0/1438215878 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdc700a1c60 msgr2=0x7fdc700a31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:40.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.664+0000 7fdc74f34640 1 --2- 172.21.15.40:0/1438215878 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdc700a1c60 0x7fdc700a31d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fdc68009e50 tx=0x7fdc6802bdb0 comp rx=0 tx=0).stop 2024-04-22T08:15:40.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.664+0000 7fdc74f34640 1 -- 172.21.15.40:0/1438215878 shutdown_connections 2024-04-22T08:15:40.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.664+0000 7fdc74f34640 1 --2- 172.21.15.40:0/1438215878 >> v2:172.21.15.189:3300/0 conn(0x7fdc700a24a0 0x7fdc701362c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:40.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.664+0000 7fdc74f34640 1 --2- 172.21.15.40:0/1438215878 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdc700a1c60 0x7fdc700a31d0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:40.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.664+0000 7fdc74f34640 1 -- 172.21.15.40:0/1438215878 >> 172.21.15.40:0/1438215878 conn(0x7fdc700a89a0 msgr2=0x7fdc700a4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:15:40.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.664+0000 7fdc74f34640 1 -- 172.21.15.40:0/1438215878 shutdown_connections 2024-04-22T08:15:40.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.664+0000 7fdc74f34640 1 -- 172.21.15.40:0/1438215878 wait complete. 2024-04-22T08:15:40.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.665+0000 7fdc74f34640 10 monclient: build_initial_monmap 2024-04-22T08:15:40.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.665+0000 7fdc74f34640 10 monclient: monmap: 2024-04-22T08:15:40.672 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:15:40.672 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:15:40.672 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:15:40.672 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:15:40.673 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:15:40.673 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:40.673 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:15:40.673 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:15:40.673 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:15:40.673 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:40.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.665+0000 7fdc74f34640 1 Processor -- start 2024-04-22T08:15:40.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.665+0000 7fdc74f34640 1 -- start start 2024-04-22T08:15:40.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.665+0000 7fdc74f34640 10 monclient: init 2024-04-22T08:15:40.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.665+0000 7fdc74f34640 10 monclient: _reopen_session rank -1 2024-04-22T08:15:40.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.665+0000 7fdc74f34640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-22T08:15:40.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.665+0000 7fdc74f34640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdc700a1c60 0x7fdc701caf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:40.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.666+0000 7fdc74f34640 10 monclient(hunting): picked mon.noname-c con 0x7fdc700a1c60 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:15:40.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.666+0000 7fdc74f34640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc700a2040 0x7fdc701cb440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:40.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.666+0000 7fdc74f34640 10 monclient(hunting): picked mon.noname-a con 0x7fdc700a2040 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:15:40.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.666+0000 7fdc74f34640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc700a24a0 0x7fdc70122270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:40.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.666+0000 7fdc74f34640 10 monclient(hunting): picked mon.noname-b con 0x7fdc700a24a0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:15:40.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.666+0000 7fdc74f34640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:40.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.666+0000 7fdc74f34640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc70126b70 con 0x7fdc700a2040 2024-04-22T08:15:40.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.666+0000 7fdc74f34640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:40.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.666+0000 7fdc74f34640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc70124ed0 con 0x7fdc700a24a0 2024-04-22T08:15:40.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.666+0000 7fdc74f34640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:40.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.666+0000 7fdc74f34640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc701261a0 con 0x7fdc700a1c60 2024-04-22T08:15:40.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.666+0000 7fdc74f34640 10 monclient(hunting): _renew_subs 2024-04-22T08:15:40.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.666+0000 7fdc74f34640 10 monclient(hunting): authenticate will time out at 1014.659851s 2024-04-22T08:15:40.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.666+0000 7fdc6e59c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdc700a1c60 0x7fdc701caf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:40.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.666+0000 7fdc6e59c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdc700a1c60 0x7fdc701caf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:44760/0 (socket says 172.21.15.40:44760) 2024-04-22T08:15:40.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.666+0000 7fdc6e59c640 1 -- 172.21.15.40:0/700400462 learned_addr learned my addr 172.21.15.40:0/700400462 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:15:40.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.666+0000 7fdc6e59c640 10 monclient(hunting): get_auth_request con 0x7fdc700a1c60 auth_method 0 2024-04-22T08:15:40.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.666+0000 7fdc6e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:40.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.666+0000 7fdc6e59c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:40.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.666+0000 7fdc6e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:40.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.666+0000 7fdc6dd9b640 1 --2- 172.21.15.40:0/700400462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc700a2040 0x7fdc701cb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:40.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.666+0000 7fdc6dd9b640 10 monclient(hunting): get_auth_request con 0x7fdc700a2040 auth_method 0 2024-04-22T08:15:40.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.666+0000 7fdc6dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:40.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.666+0000 7fdc6dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:40.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.666+0000 7fdc6dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:40.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.666+0000 7fdc6e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:40.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.666+0000 7fdc6e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:40.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.666+0000 7fdc6e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:40.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.666+0000 7fdc6dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:40.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.666+0000 7fdc6dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:40.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.666+0000 7fdc6dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:40.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.666+0000 7fdc6ed9d640 1 --2- 172.21.15.40:0/700400462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc700a24a0 0x7fdc70122270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:40.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.666+0000 7fdc6ed9d640 10 monclient(hunting): get_auth_request con 0x7fdc700a24a0 auth_method 0 2024-04-22T08:15:40.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.666+0000 7fdc6ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:40.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.666+0000 7fdc6ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:40.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.666+0000 7fdc6ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:40.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.666+0000 7fdc6e59c640 10 monclient(hunting): handle_auth_done global_id 44141 payload 824 2024-04-22T08:15:40.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.666+0000 7fdc6e59c640 1 -- 172.21.15.40:0/700400462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc700a24a0 msgr2=0x7fdc70122270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:40.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.666+0000 7fdc6e59c640 1 --2- 172.21.15.40:0/700400462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc700a24a0 0x7fdc70122270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:40.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.666+0000 7fdc6e59c640 1 -- 172.21.15.40:0/700400462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc700a2040 msgr2=0x7fdc701cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:40.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.666+0000 7fdc6e59c640 1 --2- 172.21.15.40:0/700400462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc700a2040 0x7fdc701cb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:40.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.666+0000 7fdc6e59c640 10 monclient: _finish_hunting 0 2024-04-22T08:15:40.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.666+0000 7fdc6e59c640 1 monclient: found mon.noname-c 2024-04-22T08:15:40.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.666+0000 7fdc6e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:15:40.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.666+0000 7fdc6e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:15:40.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.666+0000 7fdc6e59c640 1 -- 172.21.15.40:0/700400462 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc68009b30 con 0x7fdc700a1c60 2024-04-22T08:15:40.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.666+0000 7fdc6dd9b640 1 --2- 172.21.15.40:0/700400462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc700a2040 0x7fdc701cb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:15:40.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.668+0000 7fdc6e59c640 1 -- 172.21.15.40:0/700400462 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdc700a1c60 msgr2=0x7fdc701caf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:40.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.668+0000 7fdc6e59c640 1 --2- 172.21.15.40:0/700400462 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdc700a1c60 0x7fdc701caf00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fdc68036040 tx=0x7fdc68004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:40.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.668+0000 7fdc5effd640 1 -- 172.21.15.40:0/700400462 DispatchQueue::entry inject delay of 0.002000 2024-04-22T08:15:40.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.670+0000 7fdc5effd640 1 -- 172.21.15.40:0/700400462 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc6803e660 con 0x7fdc700a1c60 2024-04-22T08:15:40.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.670+0000 7fdc5effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:40.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.670+0000 7fdc5effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:15:40.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.670+0000 7fdc5effd640 10 monclient: dump: 2024-04-22T08:15:40.681 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:40.681 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:40.681 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:40.681 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:40.682 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:40.682 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:40.682 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:40.682 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:40.682 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:40.682 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:40.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.670+0000 7fdc5effd640 10 monclient: _finish_auth 0 2024-04-22T08:15:40.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.670+0000 7fdc5effd640 10 monclient: _check_auth_tickets 2024-04-22T08:15:40.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.670+0000 7fdc5effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:15:40.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.670+0000 7fdc5effd640 1 -- 172.21.15.40:0/700400462 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc6803e7e0 con 0x7fdc700a1c60 2024-04-22T08:15:40.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.670+0000 7fdc5effd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:15:40.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.670+0000 7fdc5effd640 1 -- 172.21.15.40:0/700400462 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc6803eb90 con 0x7fdc700a1c60 2024-04-22T08:15:40.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.670+0000 7fdc5effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:40.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.670+0000 7fdc5effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:15:40.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.670+0000 7fdc5effd640 10 monclient: dump: 2024-04-22T08:15:40.683 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:40.683 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:40.683 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:40.683 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:40.683 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:40.683 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:40.683 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:40.683 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:40.683 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:40.683 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:40.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.670+0000 7fdc74f34640 5 monclient: authenticate success, global_id 44141 2024-04-22T08:15:40.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.671+0000 7fdc74f34640 10 monclient: _renew_subs 2024-04-22T08:15:40.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.671+0000 7fdc74f34640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:15:40.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.671+0000 7fdc74f34640 1 -- 172.21.15.40:0/700400462 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc701249a0 con 0x7fdc700a1c60 2024-04-22T08:15:40.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.670+0000 7fdc6cd99640 4 set_mon_vals no callback set 2024-04-22T08:15:40.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.671+0000 7fdc74f34640 10 monclient: _renew_subs 2024-04-22T08:15:40.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.671+0000 7fdc74f34640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:15:40.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.671+0000 7fdc74f34640 1 -- 172.21.15.40:0/700400462 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc700a4790 con 0x7fdc700a1c60 2024-04-22T08:15:40.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.671+0000 7fdc5effd640 1 -- 172.21.15.40:0/700400462 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fdc6803ecf0 con 0x7fdc700a1c60 2024-04-22T08:15:40.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.671+0000 7fdc74f34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:40.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.671+0000 7fdc74f34640 1 --2- 172.21.15.40:0/700400462 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdc701227b0 0x7fdc70122b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:40.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.671+0000 7fdc74f34640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:40.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.672+0000 7fdc74f34640 1 -- 172.21.15.40:0/700400462 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc70066650 con 0x7fdc701227b0 2024-04-22T08:15:40.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.672+0000 7fdc6dd9b640 1 --2- 172.21.15.40:0/700400462 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdc701227b0 0x7fdc70122b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:40.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.672+0000 7fdc6dd9b640 10 monclient: get_auth_request con 0x7fdc701227b0 auth_method 0 2024-04-22T08:15:40.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.672+0000 7fdc6dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:40.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.672+0000 7fdc6dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:40.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.672+0000 7fdc6dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:40.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.672+0000 7fdc5effd640 1 --2- 172.21.15.40:0/700400462 >> v2:172.21.15.189:6800/30168 conn(0x7fdc50037820 0x7fdc50039ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:40.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.672+0000 7fdc5effd640 1 -- 172.21.15.40:0/700400462 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 3506+0+0 (secure 0 0 0) 0x7fdc6806cd20 con 0x7fdc700a1c60 2024-04-22T08:15:40.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.672+0000 7fdc6dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:40.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.672+0000 7fdc6dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:40.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.672+0000 7fdc6dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:40.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.672+0000 7fdc6dd9b640 10 monclient(hunting): handle_auth_done global_id 44144 payload 824 2024-04-22T08:15:40.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.672+0000 7fdc6dd9b640 1 -- 172.21.15.40:0/700400462 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdc70125900 con 0x7fdc701227b0 2024-04-22T08:15:40.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.672+0000 7fdc6ed9d640 1 --2- 172.21.15.40:0/700400462 >> v2:172.21.15.189:6800/30168 conn(0x7fdc50037820 0x7fdc50039ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:40.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.672+0000 7fdc6ed9d640 10 monclient: get_auth_request con 0x7fdc50037820 auth_method 0 2024-04-22T08:15:40.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.674+0000 7fdc6dd9b640 1 -- 172.21.15.40:0/700400462 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdc701227b0 msgr2=0x7fdc70122b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:40.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.674+0000 7fdc6dd9b640 1 --2- 172.21.15.40:0/700400462 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdc701227b0 0x7fdc70122b90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fdc701263b0 tx=0x7fdc6400ff70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:40.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.674+0000 7fdc5effd640 1 -- 172.21.15.40:0/700400462 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc6400d7a0 con 0x7fdc701227b0 2024-04-22T08:15:40.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.674+0000 7fdc5effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:40.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.674+0000 7fdc5effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:15:40.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.674+0000 7fdc5effd640 10 monclient: dump: 2024-04-22T08:15:40.686 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:40.686 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:40.686 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:40.686 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:40.686 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:40.687 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:40.687 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:40.688 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:40.688 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:40.688 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:40.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.674+0000 7fdc5effd640 1 -- 172.21.15.40:0/700400462 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7fdc6400d900 con 0x7fdc701227b0 2024-04-22T08:15:40.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.674+0000 7fdc5effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:40.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.674+0000 7fdc5effd640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:15:40.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.674+0000 7fdc5effd640 1 -- 172.21.15.40:0/700400462 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdc701227b0 msgr2=0x7fdc70122b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:40.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.674+0000 7fdc5effd640 1 --2- 172.21.15.40:0/700400462 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdc701227b0 0x7fdc70122b90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fdc701263b0 tx=0x7fdc6400ff70 comp rx=0 tx=0).stop 2024-04-22T08:15:40.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.674+0000 7fdc5effd640 1 -- 172.21.15.40:0/700400462 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdc701227b0 msgr2=0x7fdc70122b90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:15:40.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.674+0000 7fdc5effd640 1 --2- 172.21.15.40:0/700400462 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdc701227b0 0x7fdc70122b90 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fdc701263b0 tx=0x7fdc6400ff70 comp rx=0 tx=0).stop 2024-04-22T08:15:40.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.675+0000 7fdc6ed9d640 1 --2- 172.21.15.40:0/700400462 >> v2:172.21.15.189:6800/30168 conn(0x7fdc50037820 0x7fdc50039ce0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fdc700a66a0 tx=0x7fdc60008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:40.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.681+0000 7fdc74f34640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:15:40.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.681+0000 7fdc74f34640 1 --2- 172.21.15.40:0/700400462 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdc701230d0 0x7fdc7006f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:40.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.681+0000 7fdc74f34640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:40.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.681+0000 7fdc74f34640 1 -- 172.21.15.40:0/700400462 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc70136800 con 0x7fdc701230d0 2024-04-22T08:15:40.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.681+0000 7fdc6dd9b640 1 --2- 172.21.15.40:0/700400462 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdc701230d0 0x7fdc7006f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:40.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.681+0000 7fdc6dd9b640 10 monclient: get_auth_request con 0x7fdc701230d0 auth_method 0 2024-04-22T08:15:40.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.681+0000 7fdc6dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:40.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.681+0000 7fdc6dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:40.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.681+0000 7fdc6dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:40.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.681+0000 7fdc6dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:40.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.681+0000 7fdc6dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:40.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.681+0000 7fdc6dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:40.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.681+0000 7fdc6dd9b640 10 monclient(hunting): handle_auth_done global_id 44147 payload 824 2024-04-22T08:15:40.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.681+0000 7fdc6dd9b640 1 -- 172.21.15.40:0/700400462 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fdc7004e9b0 con 0x7fdc701230d0 2024-04-22T08:15:40.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.683+0000 7fdc6dd9b640 1 -- 172.21.15.40:0/700400462 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdc701230d0 msgr2=0x7fdc7006f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:40.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.683+0000 7fdc6dd9b640 1 --2- 172.21.15.40:0/700400462 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdc701230d0 0x7fdc7006f590 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fdc64017000 tx=0x7fdc64001020 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:40.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.683+0000 7fdc5effd640 1 -- 172.21.15.40:0/700400462 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc64024070 con 0x7fdc701230d0 2024-04-22T08:15:40.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.683+0000 7fdc5effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:40.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.683+0000 7fdc5effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:15:40.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.683+0000 7fdc5effd640 10 monclient: dump: 2024-04-22T08:15:40.691 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:40.691 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:40.691 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:40.691 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:40.691 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:40.691 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:40.691 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:40.691 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:40.691 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:40.692 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:40.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.684+0000 7fdc5effd640 1 -- 172.21.15.40:0/700400462 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4461 (secure 0 0 0) 0x7fdc6401d470 con 0x7fdc701230d0 2024-04-22T08:15:40.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.684+0000 7fdc5effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:15:40.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.684+0000 7fdc5effd640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:15:40.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.684+0000 7fdc5effd640 1 -- 172.21.15.40:0/700400462 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdc701230d0 msgr2=0x7fdc7006f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:40.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.684+0000 7fdc5effd640 1 --2- 172.21.15.40:0/700400462 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdc701230d0 0x7fdc7006f590 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fdc64017000 tx=0x7fdc64001020 comp rx=0 tx=0).stop 2024-04-22T08:15:40.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.684+0000 7fdc5effd640 1 -- 172.21.15.40:0/700400462 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdc701230d0 msgr2=0x7fdc7006f590 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:15:40.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.684+0000 7fdc5effd640 1 --2- 172.21.15.40:0/700400462 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdc701230d0 0x7fdc7006f590 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fdc64017000 tx=0x7fdc64001020 comp rx=0 tx=0).stop 2024-04-22T08:15:40.692 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:15:40.692 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:15:40.692 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:15:40.693 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:15:40.693 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 36, 2024-04-22T08:15:40.693 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:15:40.693 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:15:40.693 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:15:40.693 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:15:40.693 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:40.693 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 7, 2024-04-22T08:15:40.693 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:15:40.693 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:15:40.693 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:15:40.693 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:15:40.693 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:15:40.694 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:15:40.694 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:15:40.694 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:15:40.694 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:15:40.694 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:15:40.694 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:15:40.694 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:15:40.694 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:15:40.694 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:40.694 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:15:40.694 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:15:40.694 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:15:40.694 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:15:40.694 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:15:40.695 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:15:40.695 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:15:40.695 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:15:40.695 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:15:40.695 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:15:40.695 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:15:40.695 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:15:40.695 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:40.695 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:40.695 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:15:40.695 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:15:40.695 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:15:40.695 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:15:40.695 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:15:40.695 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:15:40.696 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:15:40.696 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:15:40.696 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:15:40.696 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:15:40.696 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:15:40.696 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:15:40.696 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:15:40.696 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:15:40.696 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:15:40.696 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:15:40.696 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:15:40.696 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:15:40.696 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:15:40.696 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:15:40.696 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:15:40.697 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:15:40.697 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:15:40.697 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:15:40.697 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:15:40.697 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:15:40.697 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:15:40.697 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:40.697 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:15:40.697 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:40.697 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:15:40.697 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:40.697 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:15:40.698 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:15:40.698 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:15:40.698 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:15:40.698 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:40.698 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:15:40.698 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:15:40.698 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:40.698 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:40.698 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:40.698 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:15:40.698 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:15:40.698 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:40.698 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:40.698 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:40.698 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:40.699 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:15:40.699 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:15:40.699 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:15:40.699 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:15:40.699 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:15:40.699 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:40.699 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:40.699 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:15:40.699 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:15:40.699 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:15:40.699 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:15:40.699 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:40.699 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:15:40.699 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:15:40.700 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:40.700 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:40.700 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:40.700 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:15:40.700 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:15:40.700 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:40.700 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:40.700 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:40.700 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:40.700 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:15:40.700 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:15:40.700 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:15:40.700 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:15:40.700 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:15:40.700 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:40.701 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:40.701 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:15:40.701 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:15:40.701 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:15:40.701 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:15:40.701 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:40.701 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:15:40.701 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:15:40.701 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:40.701 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:40.701 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:40.701 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:15:40.701 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:15:40.701 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:40.701 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:40.702 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:40.702 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:40.702 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:15:40.702 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:15:40.702 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:15:40.702 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:15:40.702 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:15:40.702 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:40.702 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:40.702 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:40.702 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:15:40.702 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:15:40.702 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:40.702 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:15:40.703 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:15:40.703 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:15:40.703 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:40.703 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:40.703 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:15:40.703 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:40.703 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:15:40.703 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:15:40.703 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 2 2024-04-22T08:15:40.703 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:40.704 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:40.704 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:40.704 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:15:40.704 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:15:40.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.684+0000 7fdc74f34640 1 -- 172.21.15.40:0/700400462 >> v2:172.21.15.189:6800/30168 conn(0x7fdc50037820 msgr2=0x7fdc50039ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:40.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.684+0000 7fdc74f34640 1 --2- 172.21.15.40:0/700400462 >> v2:172.21.15.189:6800/30168 conn(0x7fdc50037820 0x7fdc50039ce0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fdc700a66a0 tx=0x7fdc60008040 comp rx=0 tx=0).stop 2024-04-22T08:15:40.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.684+0000 7fdc74f34640 10 monclient: shutdown 2024-04-22T08:15:40.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.684+0000 7fdc74f34640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:15:40.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.684+0000 7fdc74f34640 1 -- 172.21.15.40:0/700400462 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdc700a1c60 msgr2=0x7fdc701caf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:40.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.684+0000 7fdc74f34640 1 --2- 172.21.15.40:0/700400462 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdc700a1c60 0x7fdc701caf00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fdc68036040 tx=0x7fdc68004a40 comp rx=0 tx=0).stop 2024-04-22T08:15:40.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.684+0000 7fdc6ed9d640 1 -- 172.21.15.40:0/700400462 reap_dead start 2024-04-22T08:15:40.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.684+0000 7fdc6ed9d640 1 -- 172.21.15.40:0/700400462 reap_dead start 2024-04-22T08:15:40.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.684+0000 7fdc74f34640 1 -- 172.21.15.40:0/700400462 shutdown_connections 2024-04-22T08:15:40.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.684+0000 7fdc74f34640 1 -- 172.21.15.40:0/700400462 >> 172.21.15.40:0/700400462 conn(0x7fdc700a89a0 msgr2=0x7fdc700a6e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:15:40.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.684+0000 7fdc74f34640 1 -- 172.21.15.40:0/700400462 shutdown_connections 2024-04-22T08:15:40.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.684+0000 7fdc74f34640 1 -- 172.21.15.40:0/700400462 wait complete. 2024-04-22T08:15:40.707 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:15:40.707 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-22T08:15:40.707 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-22T08:15:40.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.808+0000 7f6ed3a77640 10 monclient: get_monmap_and_config 2024-04-22T08:15:40.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.808+0000 7f6ed3a77640 10 monclient: build_initial_monmap 2024-04-22T08:15:40.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.808+0000 7f6ed3a77640 10 monclient: monmap: 2024-04-22T08:15:40.808 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:15:40.808 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:40.808 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:15:40.809389+0000 2024-04-22T08:15:40.808 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:15:40.809389+0000 2024-04-22T08:15:40.808 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:15:40.808 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:40.808 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:15:40.809 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:15:40.809 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:15:40.809 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:15:40.809 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:15:40.809 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:40.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.809+0000 7f6ed3a77640 1 Processor -- start 2024-04-22T08:15:40.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.809+0000 7f6ed3a77640 1 -- start start 2024-04-22T08:15:40.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.809+0000 7f6ed3a77640 10 monclient: init 2024-04-22T08:15:40.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.809+0000 7f6ed3a77640 10 monclient: _reopen_session rank -1 2024-04-22T08:15:40.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.809+0000 7f6ed3a77640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-04-22T08:15:40.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.809+0000 7f6ed3a77640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6ecc0a1a80 0x7f6ecc0a01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:40.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.809+0000 7f6ed3a77640 10 monclient(hunting): picked mon.noname-c con 0x7f6ecc0a1a80 addr v2:172.21.15.189:3300/0 2024-04-22T08:15:40.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.809+0000 7f6ed3a77640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6ecc0a1e60 addr v1:172.21.15.189:6789/0 2024-04-22T08:15:40.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.809+0000 7f6ed3a77640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6ecc13b0e0 addr v1:172.21.15.40:6789/0 2024-04-22T08:15:40.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.809+0000 7f6ed3a77640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ecc09d860 con 0x7f6ecc13b0e0 2024-04-22T08:15:40.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.809+0000 7f6ed3a77640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ecc09ef50 con 0x7f6ecc0a1e60 2024-04-22T08:15:40.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.809+0000 7f6ed3a77640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:40.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.809+0000 7f6ed3a77640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ecc09f880 con 0x7f6ecc0a1a80 2024-04-22T08:15:40.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.809+0000 7f6ed3a77640 10 monclient(hunting): _renew_subs 2024-04-22T08:15:40.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.809+0000 7f6ed3a77640 10 monclient(hunting): authenticate will time out at 1014.803605s 2024-04-22T08:15:40.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.809+0000 7f6ed2014640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f6ecc13b0e0 0x7f6ecc13d4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41020/0 (socket says 172.21.15.40:41020) 2024-04-22T08:15:40.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.809+0000 7f6ed2014640 1 -- 172.21.15.40:0/3773567251 learned_addr learned my addr 172.21.15.40:0/3773567251 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:15:40.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.810+0000 7f6ed1813640 1 --2- 172.21.15.40:0/3773567251 >> v2:172.21.15.189:3300/0 conn(0x7f6ecc0a1a80 0x7f6ecc0a01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:40.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.810+0000 7f6ed1813640 10 monclient(hunting): get_auth_request con 0x7f6ecc0a1a80 auth_method 0 2024-04-22T08:15:40.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.810+0000 7f6ed1813640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:40.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.810+0000 7f6ed1813640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:40.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.810+0000 7f6ed1813640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:40.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.810+0000 7f6ed1813640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:40.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.810+0000 7f6ed1813640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:40.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.810+0000 7f6ed1813640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:40.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.810+0000 7f6ed1813640 10 monclient(hunting): handle_auth_done global_id 24619 payload 274 2024-04-22T08:15:40.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.810+0000 7f6ed1813640 1 -- 172.21.15.40:0/3773567251 >> v1:172.21.15.189:6789/0 conn(0x7f6ecc0a1e60 legacy=0x7f6ecc0a2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:15:40.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.810+0000 7f6ed1813640 1 -- 172.21.15.40:0/3773567251 >> v1:172.21.15.40:6789/0 conn(0x7f6ecc13b0e0 legacy=0x7f6ecc13d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:15:40.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.810+0000 7f6ed1813640 10 monclient: _finish_hunting 0 2024-04-22T08:15:40.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.810+0000 7f6ed1813640 1 monclient: found mon.noname-c 2024-04-22T08:15:40.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.810+0000 7f6ed1813640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:15:40.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.810+0000 7f6ed1813640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.189:3300/0 2024-04-22T08:15:40.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.810+0000 7f6ed1813640 1 -- 172.21.15.40:0/3773567251 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ecc072160 con 0x7f6ecc0a1a80 2024-04-22T08:15:40.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.813+0000 7f6ed1813640 1 -- 172.21.15.40:0/3773567251 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ecc0a1a80 msgr2=0x7f6ecc0a01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:40.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.813+0000 7f6ed1813640 1 --2- 172.21.15.40:0/3773567251 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ecc0a1a80 0x7f6ecc0a01d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f6ec80097a0 tx=0x7f6ec802b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0ee2cca05c819a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:40.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.814+0000 7f6ed0811640 1 -- 172.21.15.40:0/3773567251 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec8030a00 con 0x7f6ecc0a1a80 2024-04-22T08:15:40.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.814+0000 7f6ed0811640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:40.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.814+0000 7f6ed0811640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:15:40.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.814+0000 7f6ed0811640 10 monclient: dump: 2024-04-22T08:15:40.820 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:40.821 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:40.821 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:40.821 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:40.821 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:40.821 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:40.821 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:40.821 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:40.821 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:40.821 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:40.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.814+0000 7f6ed0811640 10 monclient: _finish_auth 0 2024-04-22T08:15:40.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.814+0000 7f6ed0811640 10 monclient: _check_auth_tickets 2024-04-22T08:15:40.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.814+0000 7f6ed0811640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:15:40.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.814+0000 7f6ed0811640 1 -- 172.21.15.40:0/3773567251 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ec802e050 con 0x7f6ecc0a1a80 2024-04-22T08:15:40.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.814+0000 7f6ed0811640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:15:40.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.814+0000 7f6ed0811640 1 -- 172.21.15.40:0/3773567251 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec8030d80 con 0x7f6ecc0a1a80 2024-04-22T08:15:40.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.814+0000 7f6ed3a77640 5 monclient: authenticate success, global_id 24619 2024-04-22T08:15:40.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.814+0000 7f6ed3a77640 10 monclient: get_monmap_and_config success 2024-04-22T08:15:40.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.814+0000 7f6ed3a77640 4 set_mon_vals no callback set 2024-04-22T08:15:40.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.814+0000 7f6ed0811640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:40.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.814+0000 7f6ed0811640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:15:40.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.814+0000 7f6ed0811640 10 monclient: dump: 2024-04-22T08:15:40.822 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:40.823 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:40.823 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:40.823 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:40.823 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:40.823 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:40.823 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:40.823 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:40.823 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:40.823 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:40.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.814+0000 7f6ed3a77640 10 monclient: shutdown 2024-04-22T08:15:40.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.814+0000 7f6ed3a77640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:15:40.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.814+0000 7f6ed3a77640 1 -- 172.21.15.40:0/3773567251 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ecc0a1a80 msgr2=0x7f6ecc0a01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:40.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.814+0000 7f6ed3a77640 1 --2- 172.21.15.40:0/3773567251 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ecc0a1a80 0x7f6ecc0a01d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f6ec80097a0 tx=0x7f6ec802b710 comp rx=0 tx=0).stop 2024-04-22T08:15:40.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.814+0000 7f6ed3a77640 1 -- 172.21.15.40:0/3773567251 shutdown_connections 2024-04-22T08:15:40.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.814+0000 7f6ed3a77640 1 --2- 172.21.15.40:0/3773567251 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ecc0a1a80 0x7f6ecc0a01d0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:40.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.814+0000 7f6ed3a77640 1 -- 172.21.15.40:0/3773567251 >> 172.21.15.40:0/3773567251 conn(0x7f6ecc0a66a0 msgr2=0x7f6ecc0a2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:15:40.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.814+0000 7f6ed3a77640 1 -- 172.21.15.40:0/3773567251 shutdown_connections 2024-04-22T08:15:40.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.814+0000 7f6ed3a77640 1 -- 172.21.15.40:0/3773567251 wait complete. 2024-04-22T08:15:40.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.815+0000 7f6ed3a77640 10 monclient: build_initial_monmap 2024-04-22T08:15:40.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.815+0000 7f6ed3a77640 10 monclient: monmap: 2024-04-22T08:15:40.824 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:15:40.825 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:15:40.825 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:15:40.825 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:15:40.825 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:15:40.825 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:40.825 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:15:40.825 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:15:40.825 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:15:40.825 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:40.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.815+0000 7f6ed3a77640 1 Processor -- start 2024-04-22T08:15:40.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.815+0000 7f6ed3a77640 1 -- start start 2024-04-22T08:15:40.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.815+0000 7f6ed3a77640 10 monclient: init 2024-04-22T08:15:40.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed3a77640 10 monclient: _reopen_session rank -1 2024-04-22T08:15:40.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed3a77640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-22T08:15:40.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed3a77640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6ecc0a1a80 0x7f6ecc120ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:40.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed3a77640 10 monclient(hunting): picked mon.noname-c con 0x7f6ecc0a1a80 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:15:40.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed3a77640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ecc0a1e60 0x7f6ecc121010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:40.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed3a77640 10 monclient(hunting): picked mon.noname-b con 0x7f6ecc0a1e60 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:15:40.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed3a77640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ecc13b0e0 0x7f6ecc121550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:40.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed3a77640 10 monclient(hunting): picked mon.noname-a con 0x7f6ecc13b0e0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:15:40.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed3a77640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:40.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed3a77640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ecc1231a0 con 0x7f6ecc13b0e0 2024-04-22T08:15:40.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed3a77640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:40.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed3a77640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ecc09cc20 con 0x7f6ecc0a1e60 2024-04-22T08:15:40.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed3a77640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:40.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed3a77640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ecc099d10 con 0x7f6ecc0a1a80 2024-04-22T08:15:40.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed3a77640 10 monclient(hunting): _renew_subs 2024-04-22T08:15:40.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed3a77640 10 monclient(hunting): authenticate will time out at 1014.810134s 2024-04-22T08:15:40.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed1813640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6ecc0a1a80 0x7f6ecc120ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:40.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed2014640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ecc13b0e0 0x7f6ecc121550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:40.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed1813640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6ecc0a1a80 0x7f6ecc120ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:44790/0 (socket says 172.21.15.40:44790) 2024-04-22T08:15:40.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed1813640 1 -- 172.21.15.40:0/1060984775 learned_addr learned my addr 172.21.15.40:0/1060984775 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:15:40.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed1813640 10 monclient(hunting): get_auth_request con 0x7f6ecc0a1a80 auth_method 0 2024-04-22T08:15:40.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed1813640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:40.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed1813640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:40.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed1813640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:40.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed2014640 10 monclient(hunting): get_auth_request con 0x7f6ecc13b0e0 auth_method 0 2024-04-22T08:15:40.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed2014640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:40.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed2014640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:40.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed2014640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:40.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed1813640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:40.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed1813640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:40.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed1813640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:40.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed1012640 1 --2- 172.21.15.40:0/1060984775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ecc0a1e60 0x7f6ecc121010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:40.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed2014640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:40.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed2014640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:40.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed2014640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:40.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed1012640 10 monclient(hunting): get_auth_request con 0x7f6ecc0a1e60 auth_method 0 2024-04-22T08:15:40.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed1012640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:40.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed1012640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:40.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed1012640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:40.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed1012640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:40.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed1012640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:40.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed1012640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:40.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed1813640 10 monclient(hunting): handle_auth_done global_id 44150 payload 824 2024-04-22T08:15:40.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed1813640 1 -- 172.21.15.40:0/1060984775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ecc0a1e60 msgr2=0x7f6ecc121010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:40.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed1813640 1 --2- 172.21.15.40:0/1060984775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ecc0a1e60 0x7f6ecc121010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:40.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed1813640 1 -- 172.21.15.40:0/1060984775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ecc13b0e0 msgr2=0x7f6ecc121550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:40.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed1813640 1 --2- 172.21.15.40:0/1060984775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ecc13b0e0 0x7f6ecc121550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:40.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed1813640 10 monclient: _finish_hunting 0 2024-04-22T08:15:40.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed1813640 1 monclient: found mon.noname-c 2024-04-22T08:15:40.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed1813640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:15:40.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed1813640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:15:40.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed1813640 1 -- 172.21.15.40:0/1060984775 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ec8009400 con 0x7f6ecc0a1a80 2024-04-22T08:15:40.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.816+0000 7f6ed2014640 1 --2- 172.21.15.40:0/1060984775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ecc13b0e0 0x7f6ecc121550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:15:40.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.819+0000 7f6ed1813640 1 -- 172.21.15.40:0/1060984775 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6ecc0a1a80 msgr2=0x7f6ecc120ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:40.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.819+0000 7f6ed1813640 1 --2- 172.21.15.40:0/1060984775 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6ecc0a1a80 0x7f6ecc120ad0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6ec80098c0 tx=0x7f6ec80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:40.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.819+0000 7f6ec27fc640 1 -- 172.21.15.40:0/1060984775 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec8032be0 con 0x7f6ecc0a1a80 2024-04-22T08:15:40.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.819+0000 7f6ec27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:40.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.819+0000 7f6ec27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:15:40.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.819+0000 7f6ec27fc640 10 monclient: dump: 2024-04-22T08:15:40.831 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:40.831 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:40.831 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:40.831 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:40.831 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:40.831 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:40.831 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:40.831 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:40.831 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:40.831 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:40.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.819+0000 7f6ec27fc640 10 monclient: _finish_auth 0 2024-04-22T08:15:40.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.819+0000 7f6ec27fc640 10 monclient: _check_auth_tickets 2024-04-22T08:15:40.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.819+0000 7f6ec27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:15:40.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.819+0000 7f6ec27fc640 1 -- 172.21.15.40:0/1060984775 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ec8032d40 con 0x7f6ecc0a1a80 2024-04-22T08:15:40.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.819+0000 7f6ed3a77640 5 monclient: authenticate success, global_id 44150 2024-04-22T08:15:40.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.819+0000 7f6ed3a77640 10 monclient: _renew_subs 2024-04-22T08:15:40.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.819+0000 7f6ed3a77640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:15:40.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.819+0000 7f6ed3a77640 1 -- 172.21.15.40:0/1060984775 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ecc1222a0 con 0x7f6ecc0a1a80 2024-04-22T08:15:40.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.819+0000 7f6ec27fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:15:40.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.819+0000 7f6ec27fc640 1 -- 172.21.15.40:0/1060984775 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec803d6a0 con 0x7f6ecc0a1a80 2024-04-22T08:15:40.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.819+0000 7f6ec27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:40.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.819+0000 7f6ec27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:15:40.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.819+0000 7f6ec27fc640 10 monclient: dump: 2024-04-22T08:15:40.832 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:40.833 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:40.833 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:40.833 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:40.833 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:40.833 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:40.833 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:40.833 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:40.833 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:40.833 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:40.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.819+0000 7f6ec3fff640 4 set_mon_vals no callback set 2024-04-22T08:15:40.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.819+0000 7f6ed3a77640 10 monclient: _renew_subs 2024-04-22T08:15:40.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.819+0000 7f6ed3a77640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:15:40.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.819+0000 7f6ed3a77640 1 -- 172.21.15.40:0/1060984775 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ecc071640 con 0x7f6ecc0a1a80 2024-04-22T08:15:40.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.819+0000 7f6ec27fc640 1 -- 172.21.15.40:0/1060984775 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f6ec8039040 con 0x7f6ecc0a1a80 2024-04-22T08:15:40.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.820+0000 7f6ec27fc640 1 --2- 172.21.15.40:0/1060984775 >> v2:172.21.15.189:6800/30168 conn(0x7f6eb00377d0 0x7f6eb0039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:40.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.820+0000 7f6ed3a77640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:40.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.820+0000 7f6ed3a77640 1 --2- 172.21.15.40:0/1060984775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ecc1233e0 0x7f6ecc139540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:40.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.820+0000 7f6ed3a77640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:40.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.820+0000 7f6ed3a77640 1 -- 172.21.15.40:0/1060984775 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ecc066770 con 0x7f6ecc1233e0 2024-04-22T08:15:40.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.820+0000 7f6ec27fc640 1 -- 172.21.15.40:0/1060984775 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f6ec806c610 con 0x7f6ecc0a1a80 2024-04-22T08:15:40.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.820+0000 7f6ed2014640 1 --2- 172.21.15.40:0/1060984775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ecc1233e0 0x7f6ecc139540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:40.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.820+0000 7f6ed1012640 1 --2- 172.21.15.40:0/1060984775 >> v2:172.21.15.189:6800/30168 conn(0x7f6eb00377d0 0x7f6eb0039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:40.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.820+0000 7f6ed2014640 10 monclient: get_auth_request con 0x7f6ecc1233e0 auth_method 0 2024-04-22T08:15:40.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.820+0000 7f6ed2014640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:40.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.820+0000 7f6ed2014640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:40.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.820+0000 7f6ed2014640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:40.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.820+0000 7f6ed1012640 10 monclient: get_auth_request con 0x7f6eb00377d0 auth_method 0 2024-04-22T08:15:40.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.820+0000 7f6ed2014640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:40.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.820+0000 7f6ed2014640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:40.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.820+0000 7f6ed2014640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:40.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.820+0000 7f6ed2014640 10 monclient(hunting): handle_auth_done global_id 24625 payload 824 2024-04-22T08:15:40.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.820+0000 7f6ed2014640 1 -- 172.21.15.40:0/1060984775 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6ecc09dc10 con 0x7f6ecc1233e0 2024-04-22T08:15:40.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.822+0000 7f6ed1012640 1 --2- 172.21.15.40:0/1060984775 >> v2:172.21.15.189:6800/30168 conn(0x7f6eb00377d0 0x7f6eb0039c90 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc071950 tx=0x7f6ebc007e30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:40.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.823+0000 7f6ed2014640 1 -- 172.21.15.40:0/1060984775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ecc1233e0 msgr2=0x7f6ecc139540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:40.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.823+0000 7f6ed2014640 1 --2- 172.21.15.40:0/1060984775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ecc1233e0 0x7f6ecc139540 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6ec400d380 tx=0x7f6ec400d840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:40.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.823+0000 7f6ec27fc640 1 -- 172.21.15.40:0/1060984775 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec400fed0 con 0x7f6ecc1233e0 2024-04-22T08:15:40.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.823+0000 7f6ec27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:40.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.823+0000 7f6ec27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:15:40.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.823+0000 7f6ec27fc640 10 monclient: dump: 2024-04-22T08:15:40.838 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:40.838 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:40.838 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:40.838 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:40.838 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:40.838 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:40.838 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:40.839 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:40.839 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:40.839 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:40.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.823+0000 7f6ec27fc640 1 -- 172.21.15.40:0/1060984775 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f6ec40074f0 con 0x7f6ecc1233e0 2024-04-22T08:15:40.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.823+0000 7f6ec27fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:40.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.823+0000 7f6ec27fc640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:15:40.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.823+0000 7f6ec27fc640 1 -- 172.21.15.40:0/1060984775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ecc1233e0 msgr2=0x7f6ecc139540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:40.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.823+0000 7f6ec27fc640 1 --2- 172.21.15.40:0/1060984775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ecc1233e0 0x7f6ecc139540 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6ec400d380 tx=0x7f6ec400d840 comp rx=0 tx=0).stop 2024-04-22T08:15:40.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.823+0000 7f6ec27fc640 1 -- 172.21.15.40:0/1060984775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ecc1233e0 msgr2=0x7f6ecc139540 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:15:40.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.823+0000 7f6ec27fc640 1 --2- 172.21.15.40:0/1060984775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ecc1233e0 0x7f6ecc139540 secure :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6ec400d380 tx=0x7f6ec400d840 comp rx=0 tx=0).stop 2024-04-22T08:15:40.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.829+0000 7f6ed3a77640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:15:40.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.830+0000 7f6ed3a77640 1 --2- 172.21.15.40:0/1060984775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ecc139a80 0x7f6ecc139e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:40.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.830+0000 7f6ed3a77640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:40.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.830+0000 7f6ed3a77640 1 -- 172.21.15.40:0/1060984775 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ecc13a3d0 con 0x7f6ecc139a80 2024-04-22T08:15:40.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.830+0000 7f6ed2014640 1 --2- 172.21.15.40:0/1060984775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ecc139a80 0x7f6ecc139e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:40.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.830+0000 7f6ed2014640 10 monclient: get_auth_request con 0x7f6ecc139a80 auth_method 0 2024-04-22T08:15:40.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.830+0000 7f6ed2014640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:40.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.830+0000 7f6ed2014640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:40.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.830+0000 7f6ed2014640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:40.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.830+0000 7f6ed2014640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:40.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.830+0000 7f6ed2014640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:40.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.830+0000 7f6ed2014640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:40.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.830+0000 7f6ed2014640 10 monclient(hunting): handle_auth_done global_id 24628 payload 824 2024-04-22T08:15:40.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.830+0000 7f6ed2014640 1 -- 172.21.15.40:0/1060984775 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6ecc04e9b0 con 0x7f6ecc139a80 2024-04-22T08:15:40.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.832+0000 7f6ed2014640 1 -- 172.21.15.40:0/1060984775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ecc139a80 msgr2=0x7f6ecc139e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:40.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.832+0000 7f6ed2014640 1 --2- 172.21.15.40:0/1060984775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ecc139a80 0x7f6ecc139e90 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6ec400d380 tx=0x7f6ec4004cc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:40.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.833+0000 7f6ec27fc640 1 -- 172.21.15.40:0/1060984775 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec4024070 con 0x7f6ecc139a80 2024-04-22T08:15:40.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.833+0000 7f6ec27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:40.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.833+0000 7f6ec27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:15:40.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.833+0000 7f6ec27fc640 10 monclient: dump: 2024-04-22T08:15:40.841 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:40.841 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:40.841 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:40.841 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:40.841 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:40.841 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:40.841 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:40.841 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:40.841 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:40.841 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:40.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.833+0000 7f6ec27fc640 1 -- 172.21.15.40:0/1060984775 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4801 (secure 0 0 0) 0x7f6ec401da30 con 0x7f6ecc139a80 2024-04-22T08:15:40.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.833+0000 7f6ec27fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:15:40.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.833+0000 7f6ec27fc640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:15:40.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.833+0000 7f6ec27fc640 1 -- 172.21.15.40:0/1060984775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ecc139a80 msgr2=0x7f6ecc139e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:40.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.833+0000 7f6ec27fc640 1 --2- 172.21.15.40:0/1060984775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ecc139a80 0x7f6ecc139e90 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6ec400d380 tx=0x7f6ec4004cc0 comp rx=0 tx=0).stop 2024-04-22T08:15:40.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.833+0000 7f6ec27fc640 1 -- 172.21.15.40:0/1060984775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ecc139a80 msgr2=0x7f6ecc139e90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:15:40.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.833+0000 7f6ec27fc640 1 --2- 172.21.15.40:0/1060984775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ecc139a80 0x7f6ecc139e90 secure :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6ec400d380 tx=0x7f6ec4004cc0 comp rx=0 tx=0).stop 2024-04-22T08:15:40.842 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:15:40.842 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:15:40.842 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:15:40.842 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:15:40.842 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 36, 2024-04-22T08:15:40.842 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:15:40.842 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:15:40.843 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:15:40.843 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:15:40.843 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:40.843 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 7, 2024-04-22T08:15:40.843 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:15:40.843 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:15:40.843 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:15:40.843 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:15:40.843 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:15:40.843 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:15:40.843 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:15:40.843 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:15:40.843 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:15:40.843 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:15:40.843 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:15:40.844 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:15:40.844 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:15:40.844 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:40.844 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:15:40.844 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:15:40.844 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:15:40.844 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:15:40.844 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:15:40.844 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:15:40.844 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:15:40.844 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:15:40.844 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:15:40.844 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:15:40.844 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:15:40.844 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:15:40.845 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:40.845 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:40.845 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:15:40.845 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:15:40.845 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:15:40.845 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:15:40.845 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:15:40.845 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:15:40.845 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:15:40.845 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:15:40.845 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:15:40.845 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:15:40.845 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:15:40.845 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:15:40.846 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:15:40.846 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:15:40.846 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:15:40.846 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:15:40.846 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:15:40.846 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:15:40.846 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:15:40.846 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:15:40.846 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:15:40.846 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:15:40.846 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:15:40.846 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:15:40.846 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:15:40.846 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:15:40.847 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:15:40.847 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:40.847 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:15:40.847 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:40.847 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:15:40.847 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:40.847 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:15:40.847 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:15:40.847 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:15:40.847 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:15:40.847 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:40.847 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:15:40.847 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:15:40.847 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:40.848 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:40.848 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:40.848 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:15:40.848 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:15:40.848 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:40.848 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:40.848 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:40.848 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:40.848 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:15:40.848 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:15:40.848 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:15:40.848 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:15:40.848 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:15:40.848 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:40.848 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:40.849 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:15:40.849 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:15:40.849 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:15:40.849 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:15:40.849 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:40.849 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:15:40.849 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:15:40.849 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:40.849 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:40.849 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:40.849 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:15:40.849 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:15:40.849 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:40.850 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:40.850 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:40.850 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:40.850 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:15:40.850 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:15:40.850 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:15:40.850 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:15:40.850 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:15:40.850 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:40.850 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:40.850 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:15:40.850 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:15:40.850 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:15:40.850 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:15:40.851 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:40.851 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:15:40.851 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:15:40.851 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:40.851 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:40.851 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:40.851 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:15:40.851 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:15:40.851 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:40.851 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:40.851 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:40.851 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:40.851 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:15:40.851 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:15:40.851 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:15:40.852 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:15:40.852 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:15:40.852 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:40.852 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:40.852 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:40.852 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:15:40.852 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:15:40.852 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:40.852 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:15:40.852 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:15:40.852 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:15:40.852 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:40.852 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:40.852 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:15:40.853 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:40.853 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:15:40.853 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:15:40.853 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 2 2024-04-22T08:15:40.853 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:40.853 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:40.853 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:15:40.853 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:40.853 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:15:40.853 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:15:40.853 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 7 2024-04-22T08:15:40.853 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:40.853 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:40.853 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": [ 2024-04-22T08:15:40.853 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:40.854 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:15:40.854 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:15:40.854 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:15:40.854 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:40.854 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:40.854 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:40.854 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:15:40.854 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:15:40.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.833+0000 7f6ed3a77640 1 -- 172.21.15.40:0/1060984775 >> v2:172.21.15.189:6800/30168 conn(0x7f6eb00377d0 msgr2=0x7f6eb0039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:40.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.833+0000 7f6ed3a77640 1 --2- 172.21.15.40:0/1060984775 >> v2:172.21.15.189:6800/30168 conn(0x7f6eb00377d0 0x7f6eb0039c90 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc071950 tx=0x7f6ebc007e30 comp rx=0 tx=0).stop 2024-04-22T08:15:40.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.833+0000 7f6ed3a77640 10 monclient: shutdown 2024-04-22T08:15:40.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.833+0000 7f6ed3a77640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:15:40.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.833+0000 7f6ed3a77640 1 -- 172.21.15.40:0/1060984775 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6ecc0a1a80 msgr2=0x7f6ecc120ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:40.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.833+0000 7f6ed3a77640 1 --2- 172.21.15.40:0/1060984775 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6ecc0a1a80 0x7f6ecc120ad0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6ec80098c0 tx=0x7f6ec80046c0 comp rx=0 tx=0).stop 2024-04-22T08:15:40.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.833+0000 7f6ed2014640 1 -- 172.21.15.40:0/1060984775 reap_dead start 2024-04-22T08:15:40.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.833+0000 7f6ed2014640 1 -- 172.21.15.40:0/1060984775 reap_dead start 2024-04-22T08:15:40.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.833+0000 7f6ed3a77640 1 -- 172.21.15.40:0/1060984775 shutdown_connections 2024-04-22T08:15:40.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.833+0000 7f6ed3a77640 1 -- 172.21.15.40:0/1060984775 >> 172.21.15.40:0/1060984775 conn(0x7f6ecc0a66a0 msgr2=0x7f6ecc09b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:15:40.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.833+0000 7f6ed3a77640 1 -- 172.21.15.40:0/1060984775 shutdown_connections 2024-04-22T08:15:40.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:40.833+0000 7f6ed3a77640 1 -- 172.21.15.40:0/1060984775 wait complete. 2024-04-22T08:15:40.857 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:15:40.857 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-04-22T08:15:40.857 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-04-22T08:15:40.857 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-04-22T08:15:40.857 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-04-22T08:15:40.857 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-22T08:15:40.857 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T08:15:40.871 INFO:tasks.ceph.mon.c:Stopped 2024-04-22T08:15:40.871 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-22T08:15:40.871 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:15:41.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.010+0000 7fde5bb55640 10 monclient: get_monmap_and_config 2024-04-22T08:15:41.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.010+0000 7fde5bb55640 10 monclient: build_initial_monmap 2024-04-22T08:15:41.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.010+0000 7fde5bb55640 10 monclient: monmap: 2024-04-22T08:15:41.009 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:15:41.010 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:41.010 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:15:41.010864+0000 2024-04-22T08:15:41.010 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:15:41.010864+0000 2024-04-22T08:15:41.010 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:15:41.010 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:41.010 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:15:41.010 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:15:41.010 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:15:41.010 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:15:41.010 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:15:41.010 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:41.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.010+0000 7fde5bb55640 1 Processor -- start 2024-04-22T08:15:41.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.010+0000 7fde5bb55640 1 -- start start 2024-04-22T08:15:41.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.010+0000 7fde5bb55640 10 monclient: init 2024-04-22T08:15:41.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.011+0000 7fde5bb55640 10 monclient: _reopen_session rank -1 2024-04-22T08:15:41.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.011+0000 7fde5bb55640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-04-22T08:15:41.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.011+0000 7fde5bb55640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fde5409ee10 addr v1:172.21.15.189:6789/0 2024-04-22T08:15:41.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.011+0000 7fde5bb55640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde540a32b0 0x7fde5413b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:41.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.011+0000 7fde5bb55640 10 monclient(hunting): picked mon.noname-b con 0x7fde540a32b0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:15:41.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.011+0000 7fde5bb55640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fde5413b5f0 0x7fde5413d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:41.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.011+0000 7fde5bb55640 10 monclient(hunting): picked mon.noname-c con 0x7fde5413b5f0 addr v2:172.21.15.189:3300/0 2024-04-22T08:15:41.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.011+0000 7fde5bb55640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde5409f1f0 con 0x7fde5409ee10 2024-04-22T08:15:41.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.011+0000 7fde5bb55640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:41.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.011+0000 7fde5bb55640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde540a39c0 con 0x7fde5413b5f0 2024-04-22T08:15:41.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.011+0000 7fde5bb55640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:41.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.011+0000 7fde5bb55640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde5409fd30 con 0x7fde540a32b0 2024-04-22T08:15:41.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.011+0000 7fde5bb55640 10 monclient(hunting): _renew_subs 2024-04-22T08:15:41.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.011+0000 7fde5bb55640 10 monclient(hunting): authenticate will time out at 1015.005105s 2024-04-22T08:15:41.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.011+0000 7fde590f0640 1 -- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde540a32b0 msgr2=0x7fde5413b0b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:15:41.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.011+0000 7fde590f0640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde540a32b0 0x7fde5413b0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:15:41.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.011+0000 7fde5a0f2640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fde5413b5f0 0x7fde5413d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:41.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.011+0000 7fde598f1640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fde5409ee10 0x7fde540a1d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.40:43896/0 (socket says 172.21.15.40:43896) 2024-04-22T08:15:41.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.011+0000 7fde598f1640 1 -- 172.21.15.40:0/321363305 learned_addr learned my addr 172.21.15.40:0/321363305 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:15:41.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.011+0000 7fde5a0f2640 10 monclient(hunting): get_auth_request con 0x7fde5413b5f0 auth_method 0 2024-04-22T08:15:41.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.011+0000 7fde5a0f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:41.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.011+0000 7fde5a0f2640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:41.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.011+0000 7fde5a0f2640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:41.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.011+0000 7fde5a0f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:41.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.011+0000 7fde5a0f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:41.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.011+0000 7fde5a0f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:41.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.012+0000 7fde5a0f2640 10 monclient(hunting): handle_auth_done global_id 24631 payload 274 2024-04-22T08:15:41.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.012+0000 7fde5a0f2640 1 -- 172.21.15.40:0/321363305 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde540a32b0 msgr2=0x7fde5413b0b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:15:41.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.012+0000 7fde5a0f2640 1 --2- 172.21.15.40:0/321363305 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde540a32b0 0x7fde5413b0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:41.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.012+0000 7fde5a0f2640 1 -- 172.21.15.40:0/321363305 >> v1:172.21.15.189:6789/0 conn(0x7fde5409ee10 legacy=0x7fde540a1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:15:41.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.012+0000 7fde5a0f2640 10 monclient: _finish_hunting 0 2024-04-22T08:15:41.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.012+0000 7fde5a0f2640 1 monclient: found mon.noname-c 2024-04-22T08:15:41.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.012+0000 7fde5a0f2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:15:41.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.012+0000 7fde5a0f2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.189:3300/0 2024-04-22T08:15:41.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.012+0000 7fde5a0f2640 1 -- 172.21.15.40:0/321363305 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde540a02f0 con 0x7fde5413b5f0 2024-04-22T08:15:41.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.015+0000 7fde5a0f2640 1 -- 172.21.15.40:0/321363305 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde5413b5f0 msgr2=0x7fde5413d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:41.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.015+0000 7fde5a0f2640 1 --2- 172.21.15.40:0/321363305 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde5413b5f0 0x7fde5413d9e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fde54074690 tx=0x7fde4402bce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ca52d3c0de5ca7f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:41.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.015+0000 7fde588ef640 1 -- 172.21.15.40:0/321363305 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde4402cd80 con 0x7fde5413b5f0 2024-04-22T08:15:41.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.015+0000 7fde588ef640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:41.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.015+0000 7fde588ef640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:15:41.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.015+0000 7fde588ef640 10 monclient: dump: 2024-04-22T08:15:41.023 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:41.023 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:41.023 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:41.023 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:41.023 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:41.023 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:41.023 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:41.023 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:41.023 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:41.024 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:41.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.015+0000 7fde588ef640 10 monclient: _finish_auth 0 2024-04-22T08:15:41.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.015+0000 7fde588ef640 10 monclient: _check_auth_tickets 2024-04-22T08:15:41.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.015+0000 7fde588ef640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:15:41.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.015+0000 7fde588ef640 1 -- 172.21.15.40:0/321363305 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde44034440 con 0x7fde5413b5f0 2024-04-22T08:15:41.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.015+0000 7fde588ef640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:15:41.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.015+0000 7fde588ef640 1 -- 172.21.15.40:0/321363305 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde440347d0 con 0x7fde5413b5f0 2024-04-22T08:15:41.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.015+0000 7fde5bb55640 5 monclient: authenticate success, global_id 24631 2024-04-22T08:15:41.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.015+0000 7fde5bb55640 10 monclient: get_monmap_and_config success 2024-04-22T08:15:41.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.015+0000 7fde5bb55640 4 set_mon_vals no callback set 2024-04-22T08:15:41.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.015+0000 7fde588ef640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:41.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.015+0000 7fde588ef640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:15:41.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.015+0000 7fde588ef640 10 monclient: dump: 2024-04-22T08:15:41.024 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:41.024 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:41.025 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:41.025 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:41.025 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:41.025 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:41.025 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:41.025 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:41.025 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:41.025 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:41.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.015+0000 7fde5bb55640 10 monclient: shutdown 2024-04-22T08:15:41.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.015+0000 7fde5bb55640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:15:41.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.015+0000 7fde5bb55640 1 -- 172.21.15.40:0/321363305 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde5413b5f0 msgr2=0x7fde5413d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:41.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.015+0000 7fde5bb55640 1 --2- 172.21.15.40:0/321363305 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde5413b5f0 0x7fde5413d9e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fde54074690 tx=0x7fde4402bce0 comp rx=0 tx=0).stop 2024-04-22T08:15:41.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.015+0000 7fde5bb55640 1 -- 172.21.15.40:0/321363305 shutdown_connections 2024-04-22T08:15:41.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.015+0000 7fde5bb55640 1 --2- 172.21.15.40:0/321363305 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde5413b5f0 0x7fde5413d9e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:41.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.015+0000 7fde5bb55640 1 --2- 172.21.15.40:0/321363305 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde540a32b0 0x7fde5413b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:41.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.015+0000 7fde5bb55640 1 -- 172.21.15.40:0/321363305 >> 172.21.15.40:0/321363305 conn(0x7fde540a4ef0 msgr2=0x7fde540a5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:15:41.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.016+0000 7fde5bb55640 1 -- 172.21.15.40:0/321363305 shutdown_connections 2024-04-22T08:15:41.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.016+0000 7fde5bb55640 1 -- 172.21.15.40:0/321363305 wait complete. 2024-04-22T08:15:41.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.016+0000 7fde5bb55640 10 monclient: build_initial_monmap 2024-04-22T08:15:41.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.016+0000 7fde5bb55640 10 monclient: monmap: 2024-04-22T08:15:41.026 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:15:41.026 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:15:41.026 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:15:41.026 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:15:41.026 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:15:41.026 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:41.026 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:15:41.027 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:15:41.027 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:15:41.027 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:41.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.016+0000 7fde5bb55640 1 Processor -- start 2024-04-22T08:15:41.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.016+0000 7fde5bb55640 1 -- start start 2024-04-22T08:15:41.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.016+0000 7fde5bb55640 10 monclient: init 2024-04-22T08:15:41.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.017+0000 7fde5bb55640 10 monclient: _reopen_session rank -1 2024-04-22T08:15:41.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.017+0000 7fde5bb55640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-22T08:15:41.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.017+0000 7fde5bb55640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde5409ee10 0x7fde5411ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:41.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.017+0000 7fde5bb55640 10 monclient(hunting): picked mon.noname-a con 0x7fde5409ee10 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:15:41.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.017+0000 7fde5bb55640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde540a32b0 0x7fde541cb200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:41.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.017+0000 7fde5bb55640 10 monclient(hunting): picked mon.noname-c con 0x7fde540a32b0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:15:41.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.017+0000 7fde5bb55640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde5413b5f0 0x7fde541cb740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:41.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.017+0000 7fde5bb55640 10 monclient(hunting): picked mon.noname-b con 0x7fde5413b5f0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:15:41.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.017+0000 7fde5bb55640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:41.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.017+0000 7fde5bb55640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde541c2cc0 con 0x7fde5409ee10 2024-04-22T08:15:41.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.017+0000 7fde5bb55640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:41.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.017+0000 7fde5bb55640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde540a2730 con 0x7fde5413b5f0 2024-04-22T08:15:41.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.017+0000 7fde5bb55640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:41.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.017+0000 7fde5bb55640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde5411f880 con 0x7fde540a32b0 2024-04-22T08:15:41.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.017+0000 7fde5bb55640 10 monclient(hunting): _renew_subs 2024-04-22T08:15:41.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.017+0000 7fde5bb55640 10 monclient(hunting): authenticate will time out at 1015.011140s 2024-04-22T08:15:41.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.017+0000 7fde590f0640 1 -- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde540a32b0 msgr2=0x7fde541cb200 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:15:41.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.017+0000 7fde590f0640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde540a32b0 0x7fde541cb200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:15:41.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.017+0000 7fde598f1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde5409ee10 0x7fde5411ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:41.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.017+0000 7fde598f1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde5409ee10 0x7fde5411ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33592/0 (socket says 172.21.15.40:33592) 2024-04-22T08:15:41.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.017+0000 7fde598f1640 1 -- 172.21.15.40:0/611002615 learned_addr learned my addr 172.21.15.40:0/611002615 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:15:41.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.017+0000 7fde598f1640 10 monclient(hunting): get_auth_request con 0x7fde5409ee10 auth_method 0 2024-04-22T08:15:41.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.017+0000 7fde598f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:41.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.017+0000 7fde598f1640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:41.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.017+0000 7fde598f1640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:41.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.017+0000 7fde598f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:41.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.017+0000 7fde598f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:41.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.017+0000 7fde5a0f2640 1 --2- 172.21.15.40:0/611002615 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde5413b5f0 0x7fde541cb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:41.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.017+0000 7fde598f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:41.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.017+0000 7fde5a0f2640 10 monclient(hunting): get_auth_request con 0x7fde5413b5f0 auth_method 0 2024-04-22T08:15:41.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.017+0000 7fde5a0f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:41.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.017+0000 7fde5a0f2640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:41.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.017+0000 7fde5a0f2640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:41.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.017+0000 7fde5a0f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:41.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.017+0000 7fde5a0f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:41.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.017+0000 7fde5a0f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:41.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.017+0000 7fde598f1640 10 monclient(hunting): handle_auth_done global_id 34416 payload 824 2024-04-22T08:15:41.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.017+0000 7fde598f1640 1 -- 172.21.15.40:0/611002615 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde540a32b0 msgr2=0x7fde541cb200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:15:41.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.017+0000 7fde598f1640 1 --2- 172.21.15.40:0/611002615 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde540a32b0 0x7fde541cb200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:41.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.017+0000 7fde598f1640 1 -- 172.21.15.40:0/611002615 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde5413b5f0 msgr2=0x7fde541cb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:41.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.017+0000 7fde598f1640 1 --2- 172.21.15.40:0/611002615 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde5413b5f0 0x7fde541cb740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:41.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.017+0000 7fde598f1640 10 monclient: _finish_hunting 0 2024-04-22T08:15:41.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.017+0000 7fde598f1640 1 monclient: found mon.noname-a 2024-04-22T08:15:41.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.017+0000 7fde598f1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:15:41.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.017+0000 7fde598f1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:15:41.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.017+0000 7fde598f1640 1 -- 172.21.15.40:0/611002615 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde44009a70 con 0x7fde5409ee10 2024-04-22T08:15:41.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.020+0000 7fde598f1640 1 -- 172.21.15.40:0/611002615 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde5409ee10 msgr2=0x7fde5411ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:41.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.020+0000 7fde598f1640 1 --2- 172.21.15.40:0/611002615 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde5409ee10 0x7fde5411ea40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fde4c00b460 tx=0x7fde4c00b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:41.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.020+0000 7fde527fc640 1 -- 172.21.15.40:0/611002615 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde4c011aa0 con 0x7fde5409ee10 2024-04-22T08:15:41.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.020+0000 7fde527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:41.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.020+0000 7fde527fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:15:41.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.020+0000 7fde527fc640 10 monclient: dump: 2024-04-22T08:15:41.031 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:41.031 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:41.031 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:41.031 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:41.031 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:41.031 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:41.031 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:41.031 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:41.031 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:41.032 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:41.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.020+0000 7fde527fc640 10 monclient: _finish_auth 0 2024-04-22T08:15:41.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.020+0000 7fde527fc640 10 monclient: _check_auth_tickets 2024-04-22T08:15:41.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.020+0000 7fde527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:15:41.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.020+0000 7fde527fc640 1 -- 172.21.15.40:0/611002615 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde4c011c20 con 0x7fde5409ee10 2024-04-22T08:15:41.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.020+0000 7fde5bb55640 5 monclient: authenticate success, global_id 34416 2024-04-22T08:15:41.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.020+0000 7fde5bb55640 10 monclient: _renew_subs 2024-04-22T08:15:41.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.020+0000 7fde5bb55640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:41.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.020+0000 7fde5bb55640 1 -- 172.21.15.40:0/611002615 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde5411f3c0 con 0x7fde5409ee10 2024-04-22T08:15:41.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.020+0000 7fde527fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:15:41.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.020+0000 7fde527fc640 1 -- 172.21.15.40:0/611002615 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde4c0103e0 con 0x7fde5409ee10 2024-04-22T08:15:41.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.020+0000 7fde527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:41.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.020+0000 7fde527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:15:41.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.020+0000 7fde527fc640 10 monclient: dump: 2024-04-22T08:15:41.032 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:41.032 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:41.033 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:41.033 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:41.033 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:41.033 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:41.033 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:41.033 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:41.033 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:41.033 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:41.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.020+0000 7fde5bb55640 10 monclient: _renew_subs 2024-04-22T08:15:41.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.020+0000 7fde53fff640 4 set_mon_vals no callback set 2024-04-22T08:15:41.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.020+0000 7fde5bb55640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:41.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.020+0000 7fde5bb55640 1 -- 172.21.15.40:0/611002615 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde54135de0 con 0x7fde5409ee10 2024-04-22T08:15:41.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.021+0000 7fde527fc640 1 -- 172.21.15.40:0/611002615 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fde4c005020 con 0x7fde5409ee10 2024-04-22T08:15:41.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.021+0000 7fde5bb55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:41.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.021+0000 7fde5bb55640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:41.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.021+0000 7fde5bb55640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:41.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.021+0000 7fde5bb55640 1 -- 172.21.15.40:0/611002615 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde540a81d0 con 0x7fde5409ee10 2024-04-22T08:15:41.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.021+0000 7fde527fc640 1 --2- 172.21.15.40:0/611002615 >> v2:172.21.15.189:6800/30168 conn(0x7fde400377b0 0x7fde40039c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:41.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.021+0000 7fde527fc640 1 -- 172.21.15.40:0/611002615 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 3898+0+0 (secure 0 0 0) 0x7fde4c04a3e0 con 0x7fde5409ee10 2024-04-22T08:15:41.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.021+0000 7fde590f0640 1 --2- 172.21.15.40:0/611002615 >> v2:172.21.15.189:6800/30168 conn(0x7fde400377b0 0x7fde40039c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:41.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.022+0000 7fde590f0640 10 monclient: get_auth_request con 0x7fde400377b0 auth_method 0 2024-04-22T08:15:41.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.024+0000 7fde590f0640 1 --2- 172.21.15.40:0/611002615 >> v2:172.21.15.189:6800/30168 conn(0x7fde400377b0 0x7fde40039c70 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fde48006fd0 tx=0x7fde48006d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:41.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.026+0000 7fde527fc640 1 -- 172.21.15.40:0/611002615 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7fde4c021710 con 0x7fde5409ee10 2024-04-22T08:15:41.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.026+0000 7fde527fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:41.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.026+0000 7fde527fc640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:15:41.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.275+0000 7fde5bb55640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:15:41.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.275+0000 7fde5bb55640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:15:41.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.275+0000 7fde5bb55640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:41.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.275+0000 7fde5bb55640 1 -- 172.21.15.40:0/611002615 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fde5404e9b0 con 0x7fde5409ee10 2024-04-22T08:15:41.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.275+0000 7fde527fc640 1 -- 172.21.15.40:0/611002615 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7fde4c015030 con 0x7fde5409ee10 2024-04-22T08:15:41.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.275+0000 7fde527fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:15:41.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.275+0000 7fde527fc640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:15:41.275 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":36,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:15:41.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.277+0000 7fde5bb55640 1 -- 172.21.15.40:0/611002615 >> v2:172.21.15.189:6800/30168 conn(0x7fde400377b0 msgr2=0x7fde40039c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:41.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.277+0000 7fde5bb55640 1 --2- 172.21.15.40:0/611002615 >> v2:172.21.15.189:6800/30168 conn(0x7fde400377b0 0x7fde40039c70 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fde48006fd0 tx=0x7fde48006d80 comp rx=0 tx=0).stop 2024-04-22T08:15:41.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.277+0000 7fde5bb55640 10 monclient: shutdown 2024-04-22T08:15:41.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.277+0000 7fde5bb55640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:15:41.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.277+0000 7fde5bb55640 1 -- 172.21.15.40:0/611002615 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde5409ee10 msgr2=0x7fde5411ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:41.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.277+0000 7fde5bb55640 1 --2- 172.21.15.40:0/611002615 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde5409ee10 0x7fde5411ea40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fde4c00b460 tx=0x7fde4c00b920 comp rx=0 tx=0).stop 2024-04-22T08:15:41.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.277+0000 7fde5bb55640 1 -- 172.21.15.40:0/611002615 shutdown_connections 2024-04-22T08:15:41.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.277+0000 7fde5bb55640 1 --2- 172.21.15.40:0/611002615 >> v2:172.21.15.189:6800/30168 conn(0x7fde400377b0 0x7fde40039c70 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:41.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.277+0000 7fde5bb55640 1 --2- 172.21.15.40:0/611002615 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde5413b5f0 0x7fde541cb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:41.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.277+0000 7fde5bb55640 1 --2- 172.21.15.40:0/611002615 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde540a32b0 0x7fde541cb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:41.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.277+0000 7fde5bb55640 1 --2- 172.21.15.40:0/611002615 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde5409ee10 0x7fde5411ea40 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:41.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.277+0000 7fde5bb55640 1 -- 172.21.15.40:0/611002615 >> 172.21.15.40:0/611002615 conn(0x7fde540a4ef0 msgr2=0x7fde540a6c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:15:41.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.277+0000 7fde5bb55640 1 -- 172.21.15.40:0/611002615 shutdown_connections 2024-04-22T08:15:41.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:41.277+0000 7fde5bb55640 1 -- 172.21.15.40:0/611002615 wait complete. 2024-04-22T08:15:44.288 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:15:44.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.399+0000 7ff59a408640 10 monclient: get_monmap_and_config 2024-04-22T08:15:44.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.399+0000 7ff59a408640 10 monclient: build_initial_monmap 2024-04-22T08:15:44.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.399+0000 7ff59a408640 10 monclient: monmap: 2024-04-22T08:15:44.400 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:15:44.400 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:44.400 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:15:44.400642+0000 2024-04-22T08:15:44.400 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:15:44.400642+0000 2024-04-22T08:15:44.400 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:15:44.400 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:44.400 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:15:44.400 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:15:44.400 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:15:44.400 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:15:44.400 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:15:44.400 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:44.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.400+0000 7ff59a408640 1 Processor -- start 2024-04-22T08:15:44.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.400+0000 7ff59a408640 1 -- start start 2024-04-22T08:15:44.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.400+0000 7ff59a408640 10 monclient: init 2024-04-22T08:15:44.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.401+0000 7ff59a408640 10 monclient: _reopen_session rank -1 2024-04-22T08:15:44.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.401+0000 7ff59a408640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-04-22T08:15:44.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.401+0000 7ff59a408640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff59409ee10 0x7ff5940a3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:44.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.401+0000 7ff59a408640 10 monclient(hunting): picked mon.noname-c con 0x7ff59409ee10 addr v2:172.21.15.189:3300/0 2024-04-22T08:15:44.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.401+0000 7ff59a408640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff5940a1820 0x7ff5940a1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:44.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.401+0000 7ff59a408640 10 monclient(hunting): picked mon.noname-a con 0x7ff5940a1820 addr v2:172.21.15.40:3300/0 2024-04-22T08:15:44.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.401+0000 7ff59a408640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff594135dc0 addr v1:172.21.15.40:6789/0 2024-04-22T08:15:44.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.401+0000 7ff59a408640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5940a4470 con 0x7ff594135dc0 2024-04-22T08:15:44.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.401+0000 7ff59a408640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:44.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.401+0000 7ff59a408640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff59409f1f0 con 0x7ff5940a1820 2024-04-22T08:15:44.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.401+0000 7ff59a408640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:44.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.401+0000 7ff59a408640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff59409fd30 con 0x7ff59409ee10 2024-04-22T08:15:44.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.401+0000 7ff59a408640 10 monclient(hunting): _renew_subs 2024-04-22T08:15:44.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.401+0000 7ff59a408640 10 monclient(hunting): authenticate will time out at 1018.394982s 2024-04-22T08:15:44.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.401+0000 7ff5937fe640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff5940a1820 0x7ff5940a1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:44.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.401+0000 7ff5937fe640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff5940a1820 0x7ff5940a1c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33594/0 (socket says 172.21.15.40:33594) 2024-04-22T08:15:44.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.401+0000 7ff5937fe640 1 -- 172.21.15.40:0/591168725 learned_addr learned my addr 172.21.15.40:0/591168725 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:15:44.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.401+0000 7ff5937fe640 10 monclient(hunting): get_auth_request con 0x7ff5940a1820 auth_method 0 2024-04-22T08:15:44.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.401+0000 7ff5989a5640 1 --1- 172.21.15.40:0/591168725 >> v1:172.21.15.40:6789/0 conn(0x7ff594135dc0 0x7ff594136210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41022/0 (socket says 172.21.15.40:41022) 2024-04-22T08:15:44.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.401+0000 7ff5937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:44.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.401+0000 7ff5937fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:44.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.401+0000 7ff5937fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:44.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.401+0000 7ff5937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:44.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.401+0000 7ff5937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:44.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.401+0000 7ff5937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:44.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.401+0000 7ff593fff640 1 --2- 172.21.15.40:0/591168725 >> v2:172.21.15.189:3300/0 conn(0x7ff59409ee10 0x7ff5940a3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:44.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.401+0000 7ff593fff640 10 monclient(hunting): get_auth_request con 0x7ff59409ee10 auth_method 0 2024-04-22T08:15:44.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.401+0000 7ff593fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:44.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.401+0000 7ff593fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:44.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.401+0000 7ff593fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:44.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.401+0000 7ff5937fe640 10 monclient(hunting): handle_auth_done global_id 34419 payload 274 2024-04-22T08:15:44.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.401+0000 7ff5937fe640 1 -- 172.21.15.40:0/591168725 >> v2:172.21.15.189:3300/0 conn(0x7ff59409ee10 msgr2=0x7ff5940a3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:15:44.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.401+0000 7ff5937fe640 1 --2- 172.21.15.40:0/591168725 >> v2:172.21.15.189:3300/0 conn(0x7ff59409ee10 0x7ff5940a3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:44.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.401+0000 7ff5937fe640 1 -- 172.21.15.40:0/591168725 >> v1:172.21.15.40:6789/0 conn(0x7ff594135dc0 legacy=0x7ff594136210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:15:44.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.401+0000 7ff5937fe640 10 monclient: _finish_hunting 0 2024-04-22T08:15:44.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.401+0000 7ff5937fe640 1 monclient: found mon.noname-a 2024-04-22T08:15:44.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.401+0000 7ff5937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:15:44.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.401+0000 7ff5937fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:15:44.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.401+0000 7ff5937fe640 1 -- 172.21.15.40:0/591168725 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5940a0010 con 0x7ff5940a1820 2024-04-22T08:15:44.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.404+0000 7ff5937fe640 1 -- 172.21.15.40:0/591168725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5940a1820 msgr2=0x7ff5940a1c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:44.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.404+0000 7ff5937fe640 1 --2- 172.21.15.40:0/591168725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5940a1820 0x7ff5940a1c30 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff5940a5ea0 tx=0x7ff58402b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62090df178c2491d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:44.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.404+0000 7ff592ffd640 1 -- 172.21.15.40:0/591168725 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff584031960 con 0x7ff5940a1820 2024-04-22T08:15:44.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.404+0000 7ff592ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:44.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.404+0000 7ff592ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:15:44.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.404+0000 7ff592ffd640 10 monclient: dump: 2024-04-22T08:15:44.412 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:44.412 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:44.412 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:44.412 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:44.412 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:44.412 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:44.413 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:44.413 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:44.413 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:44.413 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:44.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.404+0000 7ff592ffd640 10 monclient: _finish_auth 0 2024-04-22T08:15:44.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.404+0000 7ff592ffd640 10 monclient: _check_auth_tickets 2024-04-22T08:15:44.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.404+0000 7ff592ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:15:44.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.404+0000 7ff592ffd640 1 -- 172.21.15.40:0/591168725 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff584031ac0 con 0x7ff5940a1820 2024-04-22T08:15:44.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.404+0000 7ff592ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:15:44.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.404+0000 7ff592ffd640 1 -- 172.21.15.40:0/591168725 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff584031e50 con 0x7ff5940a1820 2024-04-22T08:15:44.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.404+0000 7ff59a408640 5 monclient: authenticate success, global_id 34419 2024-04-22T08:15:44.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.404+0000 7ff59a408640 10 monclient: get_monmap_and_config success 2024-04-22T08:15:44.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.404+0000 7ff59a408640 4 set_mon_vals no callback set 2024-04-22T08:15:44.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.404+0000 7ff592ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:44.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.404+0000 7ff592ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:15:44.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.404+0000 7ff592ffd640 10 monclient: dump: 2024-04-22T08:15:44.414 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:44.414 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:44.414 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:44.414 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:44.414 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:44.414 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:44.414 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:44.414 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:44.414 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:44.414 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:44.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.405+0000 7ff59a408640 10 monclient: shutdown 2024-04-22T08:15:44.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.405+0000 7ff59a408640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:15:44.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.405+0000 7ff59a408640 1 -- 172.21.15.40:0/591168725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5940a1820 msgr2=0x7ff5940a1c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:44.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.405+0000 7ff59a408640 1 --2- 172.21.15.40:0/591168725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5940a1820 0x7ff5940a1c30 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff5940a5ea0 tx=0x7ff58402b7f0 comp rx=0 tx=0).stop 2024-04-22T08:15:44.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.405+0000 7ff59a408640 1 -- 172.21.15.40:0/591168725 shutdown_connections 2024-04-22T08:15:44.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.405+0000 7ff59a408640 1 --2- 172.21.15.40:0/591168725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5940a1820 0x7ff5940a1c30 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:44.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.405+0000 7ff59a408640 1 --2- 172.21.15.40:0/591168725 >> v2:172.21.15.189:3300/0 conn(0x7ff59409ee10 0x7ff5940a3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:44.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.405+0000 7ff59a408640 1 -- 172.21.15.40:0/591168725 >> 172.21.15.40:0/591168725 conn(0x7ff5940a4e80 msgr2=0x7ff5940a5290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:15:44.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.405+0000 7ff59a408640 1 -- 172.21.15.40:0/591168725 shutdown_connections 2024-04-22T08:15:44.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.405+0000 7ff59a408640 1 -- 172.21.15.40:0/591168725 wait complete. 2024-04-22T08:15:44.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.406+0000 7ff59a408640 10 monclient: build_initial_monmap 2024-04-22T08:15:44.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.406+0000 7ff59a408640 10 monclient: monmap: 2024-04-22T08:15:44.415 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:15:44.415 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:15:44.415 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:15:44.415 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:15:44.416 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:15:44.416 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:44.416 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:15:44.416 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:15:44.416 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:15:44.416 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:44.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.406+0000 7ff59a408640 1 Processor -- start 2024-04-22T08:15:44.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.406+0000 7ff59a408640 1 -- start start 2024-04-22T08:15:44.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.406+0000 7ff59a408640 10 monclient: init 2024-04-22T08:15:44.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.407+0000 7ff59a408640 10 monclient: _reopen_session rank -1 2024-04-22T08:15:44.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.407+0000 7ff59a408640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-22T08:15:44.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.407+0000 7ff59a408640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff59409ee10 0x7ff5941caa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:44.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.407+0000 7ff59a408640 10 monclient(hunting): picked mon.noname-c con 0x7ff59409ee10 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:15:44.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.407+0000 7ff59a408640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5940a1820 0x7ff5941331e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:44.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.407+0000 7ff59a408640 10 monclient(hunting): picked mon.noname-b con 0x7ff5940a1820 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:15:44.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.407+0000 7ff59a408640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff594135dc0 0x7ff594133720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:44.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.407+0000 7ff59a408640 10 monclient(hunting): picked mon.noname-a con 0x7ff594135dc0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:15:44.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.407+0000 7ff59a408640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:44.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.407+0000 7ff59a408640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5941cb720 con 0x7ff594135dc0 2024-04-22T08:15:44.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.407+0000 7ff59a408640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:44.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.407+0000 7ff59a408640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5940a4260 con 0x7ff5940a1820 2024-04-22T08:15:44.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.407+0000 7ff59a408640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:44.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.407+0000 7ff59a408640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5941bdf80 con 0x7ff59409ee10 2024-04-22T08:15:44.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.407+0000 7ff59a408640 10 monclient(hunting): _renew_subs 2024-04-22T08:15:44.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.407+0000 7ff59a408640 10 monclient(hunting): authenticate will time out at 1018.400882s 2024-04-22T08:15:44.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.407+0000 7ff593fff640 1 -- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff59409ee10 msgr2=0x7ff5941caa50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:15:44.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.407+0000 7ff593fff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff59409ee10 0x7ff5941caa50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:15:44.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.407+0000 7ff5989a5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff594135dc0 0x7ff594133720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:44.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.407+0000 7ff5989a5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff594135dc0 0x7ff594133720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33606/0 (socket says 172.21.15.40:33606) 2024-04-22T08:15:44.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.407+0000 7ff5989a5640 1 -- 172.21.15.40:0/889147038 learned_addr learned my addr 172.21.15.40:0/889147038 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:15:44.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.407+0000 7ff5989a5640 10 monclient(hunting): get_auth_request con 0x7ff594135dc0 auth_method 0 2024-04-22T08:15:44.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.407+0000 7ff5989a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:44.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.407+0000 7ff5989a5640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:44.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.407+0000 7ff5989a5640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:44.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.407+0000 7ff5989a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:44.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.407+0000 7ff5989a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:44.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.407+0000 7ff5989a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:44.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.407+0000 7ff5937fe640 1 --2- 172.21.15.40:0/889147038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5940a1820 0x7ff5941331e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:44.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.407+0000 7ff5937fe640 10 monclient(hunting): get_auth_request con 0x7ff5940a1820 auth_method 0 2024-04-22T08:15:44.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.407+0000 7ff5937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:44.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.407+0000 7ff5937fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:44.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.407+0000 7ff5937fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:44.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.407+0000 7ff5937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:44.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.407+0000 7ff5937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:44.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.407+0000 7ff5937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:44.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.407+0000 7ff5989a5640 10 monclient(hunting): handle_auth_done global_id 34422 payload 824 2024-04-22T08:15:44.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.407+0000 7ff5989a5640 1 -- 172.21.15.40:0/889147038 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff59409ee10 msgr2=0x7ff5941caa50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:15:44.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.407+0000 7ff5989a5640 1 --2- 172.21.15.40:0/889147038 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff59409ee10 0x7ff5941caa50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:44.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.407+0000 7ff5989a5640 1 -- 172.21.15.40:0/889147038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5940a1820 msgr2=0x7ff5941331e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:44.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.407+0000 7ff5989a5640 1 --2- 172.21.15.40:0/889147038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5940a1820 0x7ff5941331e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:44.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.407+0000 7ff5989a5640 10 monclient: _finish_hunting 0 2024-04-22T08:15:44.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.407+0000 7ff5989a5640 1 monclient: found mon.noname-a 2024-04-22T08:15:44.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.407+0000 7ff5989a5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:15:44.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.407+0000 7ff5989a5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:15:44.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.407+0000 7ff5989a5640 1 -- 172.21.15.40:0/889147038 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5840094d0 con 0x7ff594135dc0 2024-04-22T08:15:44.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.409+0000 7ff5989a5640 1 -- 172.21.15.40:0/889147038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff594135dc0 msgr2=0x7ff594133720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:44.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.409+0000 7ff5989a5640 1 --2- 172.21.15.40:0/889147038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff594135dc0 0x7ff594133720 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff58c00c750 tx=0x7ff58c00cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:44.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.409+0000 7ff590ff9640 1 -- 172.21.15.40:0/889147038 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff58c00a840 con 0x7ff594135dc0 2024-04-22T08:15:44.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.409+0000 7ff590ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:44.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.409+0000 7ff590ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:15:44.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.409+0000 7ff590ff9640 10 monclient: dump: 2024-04-22T08:15:44.420 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:44.420 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:44.420 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:44.420 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:44.420 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:44.420 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:44.421 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:44.421 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:44.421 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:44.421 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:44.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.409+0000 7ff590ff9640 10 monclient: _finish_auth 0 2024-04-22T08:15:44.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.409+0000 7ff590ff9640 10 monclient: _check_auth_tickets 2024-04-22T08:15:44.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.409+0000 7ff590ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:15:44.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.409+0000 7ff590ff9640 1 -- 172.21.15.40:0/889147038 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff58c00a9c0 con 0x7ff594135dc0 2024-04-22T08:15:44.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.409+0000 7ff590ff9640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:15:44.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.409+0000 7ff590ff9640 1 -- 172.21.15.40:0/889147038 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff58c00ad70 con 0x7ff594135dc0 2024-04-22T08:15:44.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.409+0000 7ff59a408640 5 monclient: authenticate success, global_id 34422 2024-04-22T08:15:44.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.409+0000 7ff590ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:44.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.409+0000 7ff590ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:15:44.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.409+0000 7ff590ff9640 10 monclient: dump: 2024-04-22T08:15:44.422 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:44.422 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:44.422 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:44.422 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:44.422 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:44.422 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:44.422 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:44.422 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:44.422 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:44.422 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:44.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.409+0000 7ff5927fc640 4 set_mon_vals no callback set 2024-04-22T08:15:44.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.409+0000 7ff59a408640 10 monclient: _renew_subs 2024-04-22T08:15:44.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.409+0000 7ff59a408640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:44.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.409+0000 7ff59a408640 1 -- 172.21.15.40:0/889147038 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff594120970 con 0x7ff594135dc0 2024-04-22T08:15:44.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.410+0000 7ff59a408640 10 monclient: _renew_subs 2024-04-22T08:15:44.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.410+0000 7ff59a408640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:44.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.410+0000 7ff59a408640 1 -- 172.21.15.40:0/889147038 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff594071a00 con 0x7ff594135dc0 2024-04-22T08:15:44.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.410+0000 7ff590ff9640 1 -- 172.21.15.40:0/889147038 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7ff58c007070 con 0x7ff594135dc0 2024-04-22T08:15:44.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.411+0000 7ff59a408640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:44.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.411+0000 7ff59a408640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:44.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.411+0000 7ff59a408640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:44.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.411+0000 7ff59a408640 1 -- 172.21.15.40:0/889147038 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff59404e9b0 con 0x7ff594135dc0 2024-04-22T08:15:44.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.411+0000 7ff590ff9640 1 --2- 172.21.15.40:0/889147038 >> v2:172.21.15.189:6800/30168 conn(0x7ff574037780 0x7ff574039c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:44.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.411+0000 7ff590ff9640 1 -- 172.21.15.40:0/889147038 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 3898+0+0 (secure 0 0 0) 0x7ff58c049ce0 con 0x7ff594135dc0 2024-04-22T08:15:44.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.411+0000 7ff593fff640 1 --2- 172.21.15.40:0/889147038 >> v2:172.21.15.189:6800/30168 conn(0x7ff574037780 0x7ff574039c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:44.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.411+0000 7ff593fff640 10 monclient: get_auth_request con 0x7ff574037780 auth_method 0 2024-04-22T08:15:44.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.414+0000 7ff593fff640 1 --2- 172.21.15.40:0/889147038 >> v2:172.21.15.189:6800/30168 conn(0x7ff574037780 0x7ff574039c40 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff57c006fd0 tx=0x7ff57c0092e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:44.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.416+0000 7ff590ff9640 1 -- 172.21.15.40:0/889147038 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7ff58c04f020 con 0x7ff594135dc0 2024-04-22T08:15:44.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.416+0000 7ff590ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:44.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.416+0000 7ff590ff9640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:15:44.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.667+0000 7ff59a408640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:15:44.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.667+0000 7ff59a408640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:15:44.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.667+0000 7ff59a408640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:44.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.667+0000 7ff59a408640 1 -- 172.21.15.40:0/889147038 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff59409e550 con 0x7ff594135dc0 2024-04-22T08:15:44.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.668+0000 7ff590ff9640 1 -- 172.21.15.40:0/889147038 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7ff58c017020 con 0x7ff594135dc0 2024-04-22T08:15:44.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.668+0000 7ff590ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:15:44.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.668+0000 7ff590ff9640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:15:44.668 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":36,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:15:44.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.669+0000 7ff59a408640 1 -- 172.21.15.40:0/889147038 >> v2:172.21.15.189:6800/30168 conn(0x7ff574037780 msgr2=0x7ff574039c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:44.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.669+0000 7ff59a408640 1 --2- 172.21.15.40:0/889147038 >> v2:172.21.15.189:6800/30168 conn(0x7ff574037780 0x7ff574039c40 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff57c006fd0 tx=0x7ff57c0092e0 comp rx=0 tx=0).stop 2024-04-22T08:15:44.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.669+0000 7ff59a408640 10 monclient: shutdown 2024-04-22T08:15:44.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.669+0000 7ff59a408640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:15:44.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.669+0000 7ff59a408640 1 -- 172.21.15.40:0/889147038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff594135dc0 msgr2=0x7ff594133720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:44.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.669+0000 7ff59a408640 1 --2- 172.21.15.40:0/889147038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff594135dc0 0x7ff594133720 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff58c00c750 tx=0x7ff58c00cc10 comp rx=0 tx=0).stop 2024-04-22T08:15:44.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.669+0000 7ff59a408640 1 -- 172.21.15.40:0/889147038 shutdown_connections 2024-04-22T08:15:44.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.669+0000 7ff59a408640 1 --2- 172.21.15.40:0/889147038 >> v2:172.21.15.189:6800/30168 conn(0x7ff574037780 0x7ff574039c40 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:44.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.669+0000 7ff59a408640 1 --2- 172.21.15.40:0/889147038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5940a1820 0x7ff5941331e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:44.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.669+0000 7ff59a408640 1 --2- 172.21.15.40:0/889147038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff594135dc0 0x7ff594133720 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:44.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.669+0000 7ff59a408640 1 --2- 172.21.15.40:0/889147038 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff59409ee10 0x7ff5941caa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:44.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.669+0000 7ff59a408640 1 -- 172.21.15.40:0/889147038 >> 172.21.15.40:0/889147038 conn(0x7ff5940a4e80 msgr2=0x7ff5940a6b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:15:44.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.670+0000 7ff59a408640 1 -- 172.21.15.40:0/889147038 shutdown_connections 2024-04-22T08:15:44.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:44.670+0000 7ff59a408640 1 -- 172.21.15.40:0/889147038 wait complete. 2024-04-22T08:15:47.680 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:15:47.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.792+0000 7f8ab260b640 10 monclient: get_monmap_and_config 2024-04-22T08:15:47.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.792+0000 7f8ab260b640 10 monclient: build_initial_monmap 2024-04-22T08:15:47.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.792+0000 7f8ab260b640 10 monclient: monmap: 2024-04-22T08:15:47.791 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:15:47.792 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:47.792 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:15:47.792807+0000 2024-04-22T08:15:47.792 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:15:47.792807+0000 2024-04-22T08:15:47.792 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:15:47.792 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:47.792 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:15:47.792 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:15:47.792 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:15:47.792 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:15:47.792 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:15:47.792 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:47.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.792+0000 7f8ab260b640 1 Processor -- start 2024-04-22T08:15:47.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.792+0000 7f8ab260b640 1 -- start start 2024-04-22T08:15:47.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.792+0000 7f8ab260b640 10 monclient: init 2024-04-22T08:15:47.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.793+0000 7f8ab260b640 10 monclient: _reopen_session rank -1 2024-04-22T08:15:47.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.793+0000 7f8ab260b640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-04-22T08:15:47.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.793+0000 7f8ab260b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8aac135e30 addr v1:172.21.15.40:6789/0 2024-04-22T08:15:47.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.793+0000 7f8ab260b640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8aac071c90 0x7f8aac13aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:47.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.793+0000 7f8ab260b640 10 monclient(hunting): picked mon.noname-b con 0x7f8aac071c90 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:15:47.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.793+0000 7f8ab260b640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8aac13b410 0x7f8aac13d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:47.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.793+0000 7f8ab260b640 10 monclient(hunting): picked mon.noname-a con 0x7f8aac13b410 addr v2:172.21.15.40:3300/0 2024-04-22T08:15:47.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.793+0000 7f8ab260b640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aac09e310 con 0x7f8aac135e30 2024-04-22T08:15:47.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.793+0000 7f8ab260b640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:47.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.793+0000 7f8ab260b640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8aac09de80 con 0x7f8aac13b410 2024-04-22T08:15:47.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.793+0000 7f8ab260b640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:47.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.793+0000 7f8ab260b640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac09f130 con 0x7f8aac071c90 2024-04-22T08:15:47.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.793+0000 7f8ab260b640 10 monclient(hunting): _renew_subs 2024-04-22T08:15:47.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.793+0000 7f8ab260b640 10 monclient(hunting): authenticate will time out at 1021.787150s 2024-04-22T08:15:47.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.793+0000 7f8aab7fe640 1 -- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8aac071c90 msgr2=0x7f8aac13aed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:15:47.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.793+0000 7f8aab7fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8aac071c90 0x7f8aac13aed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:15:47.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.793+0000 7f8aabfff640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f8aac135e30 0x7f8aac136210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41034/0 (socket says 172.21.15.40:41034) 2024-04-22T08:15:47.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.793+0000 7f8ab0ba8640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8aac13b410 0x7f8aac13d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:47.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.793+0000 7f8aabfff640 1 -- 172.21.15.40:0/2618684958 learned_addr learned my addr 172.21.15.40:0/2618684958 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:15:47.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.793+0000 7f8ab0ba8640 10 monclient(hunting): get_auth_request con 0x7f8aac13b410 auth_method 0 2024-04-22T08:15:47.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.793+0000 7f8ab0ba8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:47.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.793+0000 7f8ab0ba8640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:47.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.793+0000 7f8ab0ba8640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:47.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.793+0000 7f8ab0ba8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:47.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.793+0000 7f8ab0ba8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:47.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.793+0000 7f8ab0ba8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:47.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.793+0000 7f8ab0ba8640 10 monclient(hunting): handle_auth_done global_id 34425 payload 274 2024-04-22T08:15:47.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.793+0000 7f8ab0ba8640 1 -- 172.21.15.40:0/2618684958 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8aac071c90 msgr2=0x7f8aac13aed0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:15:47.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.793+0000 7f8ab0ba8640 1 --2- 172.21.15.40:0/2618684958 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8aac071c90 0x7f8aac13aed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:47.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.793+0000 7f8ab0ba8640 1 -- 172.21.15.40:0/2618684958 >> v1:172.21.15.40:6789/0 conn(0x7f8aac135e30 legacy=0x7f8aac136210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:15:47.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.793+0000 7f8ab0ba8640 10 monclient: _finish_hunting 0 2024-04-22T08:15:47.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.793+0000 7f8ab0ba8640 1 monclient: found mon.noname-a 2024-04-22T08:15:47.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.793+0000 7f8ab0ba8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:15:47.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.793+0000 7f8ab0ba8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:15:47.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.793+0000 7f8ab0ba8640 1 -- 172.21.15.40:0/2618684958 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aac09f410 con 0x7f8aac13b410 2024-04-22T08:15:47.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.796+0000 7f8ab0ba8640 1 -- 172.21.15.40:0/2618684958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8aac13b410 msgr2=0x7f8aac13d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:47.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.796+0000 7f8ab0ba8640 1 --2- 172.21.15.40:0/2618684958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8aac13b410 0x7f8aac13d800 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8aa4009d90 tx=0x7f8aa402bd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14e68cbd04a9bcb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:47.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.796+0000 7f8aaaffd640 1 -- 172.21.15.40:0/2618684958 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa402cdc0 con 0x7f8aac13b410 2024-04-22T08:15:47.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.796+0000 7f8aaaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:47.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.796+0000 7f8aaaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:15:47.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.796+0000 7f8aaaffd640 10 monclient: dump: 2024-04-22T08:15:47.804 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:47.804 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:47.804 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:47.804 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:47.804 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:47.804 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:47.804 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:47.804 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:47.804 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:47.804 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:47.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.796+0000 7f8aaaffd640 10 monclient: _finish_auth 0 2024-04-22T08:15:47.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.796+0000 7f8aaaffd640 10 monclient: _check_auth_tickets 2024-04-22T08:15:47.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.796+0000 7f8aaaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:15:47.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.796+0000 7f8aaaffd640 1 -- 172.21.15.40:0/2618684958 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aa4034440 con 0x7f8aac13b410 2024-04-22T08:15:47.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.796+0000 7f8ab260b640 5 monclient: authenticate success, global_id 34425 2024-04-22T08:15:47.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.796+0000 7f8ab260b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:15:47.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.796+0000 7f8aaaffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:15:47.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.796+0000 7f8ab260b640 10 monclient: get_monmap_and_config success 2024-04-22T08:15:47.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.796+0000 7f8aaaffd640 1 -- 172.21.15.40:0/2618684958 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa40347d0 con 0x7f8aac13b410 2024-04-22T08:15:47.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.796+0000 7f8aaaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:47.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.796+0000 7f8ab260b640 4 set_mon_vals no callback set 2024-04-22T08:15:47.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.796+0000 7f8aaaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:15:47.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.796+0000 7f8aaaffd640 10 monclient: dump: 2024-04-22T08:15:47.805 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:47.806 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:47.806 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:47.806 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:47.806 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:47.806 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:47.806 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:47.806 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:47.806 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:47.806 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:47.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.797+0000 7f8ab260b640 10 monclient: shutdown 2024-04-22T08:15:47.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.797+0000 7f8ab260b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:15:47.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.797+0000 7f8ab260b640 1 -- 172.21.15.40:0/2618684958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8aac13b410 msgr2=0x7f8aac13d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:47.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.797+0000 7f8ab260b640 1 --2- 172.21.15.40:0/2618684958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8aac13b410 0x7f8aac13d800 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8aa4009d90 tx=0x7f8aa402bd20 comp rx=0 tx=0).stop 2024-04-22T08:15:47.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.797+0000 7f8ab260b640 1 -- 172.21.15.40:0/2618684958 shutdown_connections 2024-04-22T08:15:47.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.797+0000 7f8ab260b640 1 --2- 172.21.15.40:0/2618684958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8aac13b410 0x7f8aac13d800 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:47.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.797+0000 7f8ab260b640 1 --2- 172.21.15.40:0/2618684958 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8aac071c90 0x7f8aac13aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:47.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.797+0000 7f8ab260b640 1 -- 172.21.15.40:0/2618684958 >> 172.21.15.40:0/2618684958 conn(0x7f8aac0a65c0 msgr2=0x7f8aac0a2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:15:47.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.797+0000 7f8ab260b640 1 -- 172.21.15.40:0/2618684958 shutdown_connections 2024-04-22T08:15:47.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.797+0000 7f8ab260b640 1 -- 172.21.15.40:0/2618684958 wait complete. 2024-04-22T08:15:47.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.798+0000 7f8ab260b640 10 monclient: build_initial_monmap 2024-04-22T08:15:47.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.798+0000 7f8ab260b640 10 monclient: monmap: 2024-04-22T08:15:47.807 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:15:47.807 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:15:47.807 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:15:47.807 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:15:47.807 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:15:47.807 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:47.807 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:15:47.807 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:15:47.808 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:15:47.808 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:47.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.798+0000 7f8ab260b640 1 Processor -- start 2024-04-22T08:15:47.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.798+0000 7f8ab260b640 1 -- start start 2024-04-22T08:15:47.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.798+0000 7f8ab260b640 10 monclient: init 2024-04-22T08:15:47.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.799+0000 7f8ab260b640 10 monclient: _reopen_session rank -1 2024-04-22T08:15:47.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.799+0000 7f8ab260b640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:15:47.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.799+0000 7f8ab260b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8aac071c90 0x7f8aac1ca820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:47.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.799+0000 7f8ab260b640 10 monclient(hunting): picked mon.noname-a con 0x7f8aac071c90 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:15:47.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.799+0000 7f8ab260b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8aac135e30 0x7f8aac134c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:47.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.799+0000 7f8ab260b640 10 monclient(hunting): picked mon.noname-b con 0x7f8aac135e30 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:15:47.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.799+0000 7f8ab260b640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8aac13b410 0x7f8aac135150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:47.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.799+0000 7f8ab260b640 10 monclient(hunting): picked mon.noname-c con 0x7f8aac13b410 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:15:47.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.799+0000 7f8ab260b640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:47.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.799+0000 7f8ab260b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac0a22a0 con 0x7f8aac071c90 2024-04-22T08:15:47.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.799+0000 7f8ab260b640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:47.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.799+0000 7f8ab260b640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac0720e0 con 0x7f8aac135e30 2024-04-22T08:15:47.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.799+0000 7f8ab260b640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:47.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.799+0000 7f8ab260b640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac070500 con 0x7f8aac13b410 2024-04-22T08:15:47.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.799+0000 7f8ab260b640 10 monclient(hunting): _renew_subs 2024-04-22T08:15:47.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.799+0000 7f8ab260b640 10 monclient(hunting): authenticate will time out at 1021.792900s 2024-04-22T08:15:47.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.799+0000 7f8ab0ba8640 1 -- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8aac13b410 msgr2=0x7f8aac135150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:15:47.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.799+0000 7f8ab0ba8640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8aac13b410 0x7f8aac135150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:15:47.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.799+0000 7f8aabfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8aac071c90 0x7f8aac1ca820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:47.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.799+0000 7f8aabfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8aac071c90 0x7f8aac1ca820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33626/0 (socket says 172.21.15.40:33626) 2024-04-22T08:15:47.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.799+0000 7f8aabfff640 1 -- 172.21.15.40:0/2421588031 learned_addr learned my addr 172.21.15.40:0/2421588031 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:15:47.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.799+0000 7f8aabfff640 10 monclient(hunting): get_auth_request con 0x7f8aac071c90 auth_method 0 2024-04-22T08:15:47.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.799+0000 7f8aabfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:47.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.799+0000 7f8aabfff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:47.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.799+0000 7f8aabfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:47.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.799+0000 7f8aabfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:47.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.799+0000 7f8aabfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:47.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.799+0000 7f8aabfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:47.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.799+0000 7f8aab7fe640 1 --2- 172.21.15.40:0/2421588031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8aac135e30 0x7f8aac134c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:47.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.799+0000 7f8aabfff640 10 monclient(hunting): handle_auth_done global_id 34428 payload 824 2024-04-22T08:15:47.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.799+0000 7f8aabfff640 1 -- 172.21.15.40:0/2421588031 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8aac13b410 msgr2=0x7f8aac135150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:15:47.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.799+0000 7f8aabfff640 1 --2- 172.21.15.40:0/2421588031 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8aac13b410 0x7f8aac135150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:47.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.799+0000 7f8aabfff640 1 -- 172.21.15.40:0/2421588031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8aac135e30 msgr2=0x7f8aac134c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:47.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.799+0000 7f8aabfff640 1 --2- 172.21.15.40:0/2421588031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8aac135e30 0x7f8aac134c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:47.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.799+0000 7f8aabfff640 10 monclient: _finish_hunting 0 2024-04-22T08:15:47.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.799+0000 7f8aabfff640 1 monclient: found mon.noname-a 2024-04-22T08:15:47.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.799+0000 7f8aabfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:15:47.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.799+0000 7f8aabfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:15:47.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.799+0000 7f8aabfff640 1 -- 172.21.15.40:0/2421588031 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aa4009a70 con 0x7f8aac071c90 2024-04-22T08:15:47.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.799+0000 7f8aab7fe640 10 monclient: get_auth_request con 0x7f8aac135e30 auth_method 0 2024-04-22T08:15:47.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.799+0000 7f8aab7fe640 1 --2- 172.21.15.40:0/2421588031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8aac135e30 0x7f8aac134c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T08:15:47.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.801+0000 7f8aabfff640 1 -- 172.21.15.40:0/2421588031 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8aac071c90 msgr2=0x7f8aac1ca820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:47.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.801+0000 7f8aabfff640 1 --2- 172.21.15.40:0/2421588031 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8aac071c90 0x7f8aac1ca820 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8a9400c460 tx=0x7f8a9400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:47.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.801+0000 7f8aabfff640 1 --2- 172.21.15.40:0/2421588031 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8aac071c90 0x7f8aac1ca820 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8a9400c460 tx=0x7f8a9400c920 comp rx=0 tx=0).queue_received will delay after 1.71377e+09 on 0x7f8a940071f0 config(0 keys) v1 2024-04-22T08:15:47.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.801+0000 7f8aa8ff9640 1 -- 172.21.15.40:0/2421588031 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a94007070 con 0x7f8aac071c90 2024-04-22T08:15:47.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.801+0000 7f8aa8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:47.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.801+0000 7f8aa8ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:15:47.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.801+0000 7f8aa8ff9640 10 monclient: dump: 2024-04-22T08:15:47.811 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:47.811 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:47.811 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:47.812 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:47.812 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:47.812 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:47.812 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:47.812 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:47.812 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:47.812 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:47.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.801+0000 7f8aa8ff9640 10 monclient: _finish_auth 0 2024-04-22T08:15:47.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.801+0000 7f8aa8ff9640 10 monclient: _check_auth_tickets 2024-04-22T08:15:47.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.801+0000 7f8aa8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:15:47.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.801+0000 7f8aa8ff9640 1 -- 172.21.15.40:0/2421588031 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a940071f0 con 0x7f8aac071c90 2024-04-22T08:15:47.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.801+0000 7f8aa8ff9640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:15:47.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.801+0000 7f8ab260b640 5 monclient: authenticate success, global_id 34428 2024-04-22T08:15:47.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.802+0000 7f8ab260b640 10 monclient: _renew_subs 2024-04-22T08:15:47.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.802+0000 7f8ab260b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:47.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.802+0000 7f8ab260b640 1 -- 172.21.15.40:0/2421588031 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aac1215c0 con 0x7f8aac071c90 2024-04-22T08:15:47.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.801+0000 7f8aaa7fc640 4 set_mon_vals no callback set 2024-04-22T08:15:47.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.802+0000 7f8ab260b640 10 monclient: _renew_subs 2024-04-22T08:15:47.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.802+0000 7f8ab260b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:47.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.802+0000 7f8ab260b640 1 -- 172.21.15.40:0/2421588031 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aac1209f0 con 0x7f8aac071c90 2024-04-22T08:15:47.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.802+0000 7f8aa8ff9640 1 -- 172.21.15.40:0/2421588031 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a940075a0 con 0x7f8aac071c90 2024-04-22T08:15:47.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.802+0000 7f8aa8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:47.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.802+0000 7f8aa8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:15:47.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.802+0000 7f8aa8ff9640 10 monclient: dump: 2024-04-22T08:15:47.813 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:47.813 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:47.813 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:47.813 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:47.814 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:47.814 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:47.814 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:47.814 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:47.814 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:47.814 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:47.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.802+0000 7f8aa8ff9640 1 -- 172.21.15.40:0/2421588031 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f8a9401b400 con 0x7f8aac071c90 2024-04-22T08:15:47.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.802+0000 7f8ab260b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:47.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.803+0000 7f8ab260b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:47.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.803+0000 7f8ab260b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:47.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.803+0000 7f8ab260b640 1 -- 172.21.15.40:0/2421588031 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aac04e9b0 con 0x7f8aac071c90 2024-04-22T08:15:47.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.803+0000 7f8aa8ff9640 1 --2- 172.21.15.40:0/2421588031 >> v2:172.21.15.189:6800/30168 conn(0x7f8a8c0377d0 0x7f8a8c039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:47.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.803+0000 7f8aab7fe640 1 --2- 172.21.15.40:0/2421588031 >> v2:172.21.15.189:6800/30168 conn(0x7f8a8c0377d0 0x7f8a8c039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:47.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.803+0000 7f8aab7fe640 10 monclient: get_auth_request con 0x7f8a8c0377d0 auth_method 0 2024-04-22T08:15:47.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.806+0000 7f8aab7fe640 1 --2- 172.21.15.40:0/2421588031 >> v2:172.21.15.189:6800/30168 conn(0x7f8a8c0377d0 0x7f8a8c039c90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c006fd0 tx=0x7f8a9c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:47.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:47.807+0000 7f8aa8ff9640 1 -- 172.21.15.40:0/2421588031 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f8a9404a120 con 0x7f8aac071c90 2024-04-22T08:15:48.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:48.424+0000 7f8aa8ff9640 1 -- 172.21.15.40:0/2421588031 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f8a9407d7f0 con 0x7f8aac071c90 2024-04-22T08:15:48.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:48.424+0000 7f8aa8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:48.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:48.424+0000 7f8aa8ff9640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:15:48.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:48.678+0000 7f8ab260b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:15:48.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:48.678+0000 7f8ab260b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:15:48.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:48.678+0000 7f8ab260b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:48.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:48.678+0000 7f8ab260b640 1 -- 172.21.15.40:0/2421588031 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8aac099a90 con 0x7f8aac071c90 2024-04-22T08:15:48.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:48.678+0000 7f8aa8ff9640 1 -- 172.21.15.40:0/2421588031 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7f8a94010710 con 0x7f8aac071c90 2024-04-22T08:15:48.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:48.678+0000 7f8aa8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:15:48.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:48.678+0000 7f8aa8ff9640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:15:48.679 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":36,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:15:48.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:48.680+0000 7f8ab260b640 1 -- 172.21.15.40:0/2421588031 >> v2:172.21.15.189:6800/30168 conn(0x7f8a8c0377d0 msgr2=0x7f8a8c039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:48.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:48.680+0000 7f8ab260b640 1 --2- 172.21.15.40:0/2421588031 >> v2:172.21.15.189:6800/30168 conn(0x7f8a8c0377d0 0x7f8a8c039c90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c006fd0 tx=0x7f8a9c008040 comp rx=0 tx=0).stop 2024-04-22T08:15:48.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:48.680+0000 7f8ab260b640 10 monclient: shutdown 2024-04-22T08:15:48.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:48.680+0000 7f8ab260b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:15:48.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:48.680+0000 7f8ab260b640 1 -- 172.21.15.40:0/2421588031 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8aac071c90 msgr2=0x7f8aac1ca820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:48.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:48.680+0000 7f8ab260b640 1 --2- 172.21.15.40:0/2421588031 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8aac071c90 0x7f8aac1ca820 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8a9400c460 tx=0x7f8a9400c920 comp rx=0 tx=0).stop 2024-04-22T08:15:48.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:48.681+0000 7f8ab260b640 1 -- 172.21.15.40:0/2421588031 shutdown_connections 2024-04-22T08:15:48.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:48.681+0000 7f8ab260b640 1 --2- 172.21.15.40:0/2421588031 >> v2:172.21.15.189:6800/30168 conn(0x7f8a8c0377d0 0x7f8a8c039c90 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:48.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:48.681+0000 7f8ab260b640 1 --2- 172.21.15.40:0/2421588031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8aac135e30 0x7f8aac134c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:48.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:48.681+0000 7f8ab260b640 1 --2- 172.21.15.40:0/2421588031 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8aac13b410 0x7f8aac135150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:48.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:48.681+0000 7f8ab260b640 1 --2- 172.21.15.40:0/2421588031 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8aac071c90 0x7f8aac1ca820 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:48.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:48.681+0000 7f8ab260b640 1 -- 172.21.15.40:0/2421588031 >> 172.21.15.40:0/2421588031 conn(0x7f8aac0a65c0 msgr2=0x7f8aac0a0990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:15:48.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:48.681+0000 7f8ab260b640 1 -- 172.21.15.40:0/2421588031 shutdown_connections 2024-04-22T08:15:48.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:48.681+0000 7f8ab260b640 1 -- 172.21.15.40:0/2421588031 wait complete. 2024-04-22T08:15:51.692 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:15:51.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:51.803+0000 7f4f99608640 10 monclient: get_monmap_and_config 2024-04-22T08:15:51.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:51.803+0000 7f4f99608640 10 monclient: build_initial_monmap 2024-04-22T08:15:51.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:51.803+0000 7f4f99608640 10 monclient: monmap: 2024-04-22T08:15:51.803 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:15:51.803 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:51.803 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:15:51.803937+0000 2024-04-22T08:15:51.803 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:15:51.803937+0000 2024-04-22T08:15:51.803 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:15:51.803 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:51.803 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:15:51.803 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:15:51.803 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:15:51.803 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:15:51.803 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:15:51.803 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:51.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:51.803+0000 7f4f99608640 1 Processor -- start 2024-04-22T08:15:51.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:51.803+0000 7f4f99608640 1 -- start start 2024-04-22T08:15:51.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:51.803+0000 7f4f99608640 10 monclient: init 2024-04-22T08:15:51.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:51.804+0000 7f4f99608640 10 monclient: _reopen_session rank -1 2024-04-22T08:15:51.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:51.804+0000 7f4f99608640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-04-22T08:15:51.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:51.804+0000 7f4f99608640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4f9409ee10 0x7f4f940a32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:51.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:51.804+0000 7f4f99608640 10 monclient(hunting): picked mon.noname-c con 0x7f4f9409ee10 addr v2:172.21.15.189:3300/0 2024-04-22T08:15:51.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:51.804+0000 7f4f99608640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4f940a1d30 0x7f4f940a2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:51.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:51.804+0000 7f4f99608640 10 monclient(hunting): picked mon.noname-b con 0x7f4f940a1d30 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:15:51.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:51.804+0000 7f4f99608640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4f94135d90 addr v1:172.21.15.40:6789/0 2024-04-22T08:15:51.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:51.804+0000 7f4f99608640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f9409f1f0 con 0x7f4f94135d90 2024-04-22T08:15:51.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:51.804+0000 7f4f99608640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:51.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:51.804+0000 7f4f99608640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f940a39c0 con 0x7f4f9409ee10 2024-04-22T08:15:51.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:51.804+0000 7f4f99608640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:51.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:51.804+0000 7f4f99608640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f9409fd30 con 0x7f4f940a1d30 2024-04-22T08:15:51.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:51.804+0000 7f4f99608640 10 monclient(hunting): _renew_subs 2024-04-22T08:15:51.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:51.804+0000 7f4f99608640 10 monclient(hunting): authenticate will time out at 1025.798075s 2024-04-22T08:15:51.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:51.804+0000 7f4f8a7fc640 1 -- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4f940a1d30 msgr2=0x7f4f940a2150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:15:51.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:51.804+0000 7f4f8a7fc640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4f940a1d30 0x7f4f940a2150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:15:51.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:51.804+0000 7f4f937fe640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f4f94135d90 0x7f4f941361c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:43958/0 (socket says 172.21.15.40:43958) 2024-04-22T08:15:51.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:51.804+0000 7f4f937fe640 1 -- 172.21.15.40:0/2639466349 learned_addr learned my addr 172.21.15.40:0/2639466349 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:15:51.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:51.804+0000 7f4f92ffd640 1 --2- 172.21.15.40:0/2639466349 >> v2:172.21.15.189:3300/0 conn(0x7f4f9409ee10 0x7f4f940a32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:51.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:51.804+0000 7f4f92ffd640 10 monclient(hunting): get_auth_request con 0x7f4f9409ee10 auth_method 0 2024-04-22T08:15:51.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:51.804+0000 7f4f92ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:51.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:51.804+0000 7f4f92ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:51.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:51.804+0000 7f4f92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:51.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:51.804+0000 7f4f92ffd640 1 -- 172.21.15.40:0/2639466349 >> v2:172.21.15.189:3300/0 conn(0x7f4f9409ee10 msgr2=0x7f4f940a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:15:51.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:51.804+0000 7f4f92ffd640 1 -- 172.21.15.40:0/2639466349 >> v2:172.21.15.189:3300/0 conn(0x7f4f9409ee10 msgr2=0x7f4f940a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:15:51.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:51.804+0000 7f4f92ffd640 1 --2- 172.21.15.40:0/2639466349 >> v2:172.21.15.189:3300/0 conn(0x7f4f9409ee10 0x7f4f940a32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:15:51.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:51.804+0000 7f4f92ffd640 1 --2- 172.21.15.40:0/2639466349 >> v2:172.21.15.189:3300/0 conn(0x7f4f9409ee10 0x7f4f940a32b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:15:51.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:51.806+0000 7f4f937fe640 1 -- 172.21.15.40:0/2639466349 >> v1:172.21.15.40:6789/0 conn(0x7f4f94135d90 legacy=0x7f4f941361c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:52.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:52.005+0000 7f4f8a7fc640 1 -- 172.21.15.40:0/2639466349 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4f940a1d30 msgr2=0x7f4f940a2150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:15:52.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:52.005+0000 7f4f8a7fc640 1 --2- 172.21.15.40:0/2639466349 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4f940a1d30 0x7f4f940a2150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:15:52.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:52.005+0000 7f4f92ffd640 1 -- 172.21.15.40:0/2639466349 >> v2:172.21.15.189:3300/0 conn(0x7f4f9409ee10 msgr2=0x7f4f940a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:15:52.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:52.005+0000 7f4f92ffd640 1 -- 172.21.15.40:0/2639466349 >> v2:172.21.15.189:3300/0 conn(0x7f4f9409ee10 msgr2=0x7f4f940a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:15:52.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:52.005+0000 7f4f92ffd640 1 --2- 172.21.15.40:0/2639466349 >> v2:172.21.15.189:3300/0 conn(0x7f4f9409ee10 0x7f4f940a32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) 2024-04-22T08:15:52.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:52.005+0000 7f4f92ffd640 1 --2- 172.21.15.40:0/2639466349 >> v2:172.21.15.189:3300/0 conn(0x7f4f9409ee10 0x7f4f940a32b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:15:52.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:52.406+0000 7f4f8a7fc640 1 -- 172.21.15.40:0/2639466349 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4f940a1d30 msgr2=0x7f4f940a2150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:15:52.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:52.406+0000 7f4f8a7fc640 1 --2- 172.21.15.40:0/2639466349 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4f940a1d30 0x7f4f940a2150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:15:52.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:52.406+0000 7f4f92ffd640 1 --2- 172.21.15.40:0/2639466349 >> v2:172.21.15.189:3300/0 conn(0x7f4f9409ee10 0x7f4f940a32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:52.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:52.406+0000 7f4f92ffd640 10 monclient(hunting): get_auth_request con 0x7f4f9409ee10 auth_method 0 2024-04-22T08:15:52.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:52.406+0000 7f4f92ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:52.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:52.406+0000 7f4f92ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:52.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:52.406+0000 7f4f92ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:15:52.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:52.406+0000 7f4f92ffd640 1 -- 172.21.15.40:0/2639466349 >> v2:172.21.15.189:3300/0 conn(0x7f4f9409ee10 msgr2=0x7f4f940a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:15:52.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:52.406+0000 7f4f92ffd640 1 -- 172.21.15.40:0/2639466349 >> v2:172.21.15.189:3300/0 conn(0x7f4f9409ee10 msgr2=0x7f4f940a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:15:52.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:52.406+0000 7f4f92ffd640 1 --2- 172.21.15.40:0/2639466349 >> v2:172.21.15.189:3300/0 conn(0x7f4f9409ee10 0x7f4f940a32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:15:52.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:52.406+0000 7f4f92ffd640 1 --2- 172.21.15.40:0/2639466349 >> v2:172.21.15.189:3300/0 conn(0x7f4f9409ee10 0x7f4f940a32b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:15:53.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:53.207+0000 7f4f8a7fc640 1 -- 172.21.15.40:0/2639466349 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4f940a1d30 msgr2=0x7f4f940a2150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:15:53.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:53.207+0000 7f4f8a7fc640 1 --2- 172.21.15.40:0/2639466349 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4f940a1d30 0x7f4f940a2150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T08:15:53.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:53.207+0000 7f4f92ffd640 1 --2- 172.21.15.40:0/2639466349 >> v2:172.21.15.189:3300/0 conn(0x7f4f9409ee10 0x7f4f940a32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:53.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:53.207+0000 7f4f92ffd640 10 monclient(hunting): get_auth_request con 0x7f4f9409ee10 auth_method 0 2024-04-22T08:15:53.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:53.207+0000 7f4f92ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:53.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:53.207+0000 7f4f92ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:53.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:53.207+0000 7f4f92ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:15:53.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:53.208+0000 7f4f92ffd640 1 -- 172.21.15.40:0/2639466349 >> v2:172.21.15.189:3300/0 conn(0x7f4f9409ee10 msgr2=0x7f4f940a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:15:53.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:53.208+0000 7f4f92ffd640 1 -- 172.21.15.40:0/2639466349 >> v2:172.21.15.189:3300/0 conn(0x7f4f9409ee10 msgr2=0x7f4f940a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:15:53.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:53.208+0000 7f4f92ffd640 1 --2- 172.21.15.40:0/2639466349 >> v2:172.21.15.189:3300/0 conn(0x7f4f9409ee10 0x7f4f940a32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:15:53.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:53.208+0000 7f4f92ffd640 1 --2- 172.21.15.40:0/2639466349 >> v2:172.21.15.189:3300/0 conn(0x7f4f9409ee10 0x7f4f940a32b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T08:15:54.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:54.804+0000 7f4f917fa640 10 monclient(hunting): tick 2024-04-22T08:15:54.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:54.804+0000 7f4f917fa640 10 monclient(hunting): _check_auth_tickets 2024-04-22T08:15:54.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:54.804+0000 7f4f917fa640 1 monclient(hunting): continuing hunt 2024-04-22T08:15:54.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:54.804+0000 7f4f917fa640 10 monclient(hunting): _reopen_session rank -1 2024-04-22T08:15:54.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:54.804+0000 7f4f917fa640 1 -- 172.21.15.40:0/2639466349 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4f940a1d30 msgr2=0x7f4f940a2150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:15:54.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:54.804+0000 7f4f917fa640 1 --2- 172.21.15.40:0/2639466349 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4f940a1d30 0x7f4f940a2150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:54.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:54.804+0000 7f4f917fa640 1 -- 172.21.15.40:0/2639466349 >> v2:172.21.15.189:3300/0 conn(0x7f4f9409ee10 msgr2=0x7f4f940a32b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:15:54.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:54.804+0000 7f4f917fa640 1 --2- 172.21.15.40:0/2639466349 >> v2:172.21.15.189:3300/0 conn(0x7f4f9409ee10 0x7f4f940a32b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:54.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:54.804+0000 7f4f917fa640 1 -- 172.21.15.40:0/2639466349 >> v1:172.21.15.40:6789/0 conn(0x7f4f94135d90 legacy=0x7f4f941361c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:54.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:54.804+0000 7f4f917fa640 10 monclient: _add_conns ranks=[1,3] 2024-04-22T08:15:54.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:54.804+0000 7f4f917fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4f74003760 addr v1:172.21.15.189:6789/0 2024-04-22T08:15:54.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:54.804+0000 7f4f917fa640 1 --2- 172.21.15.40:0/2639466349 >> v2:172.21.15.40:3300/0 conn(0x7f4f740074b0 0x7f4f74009970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:54.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:54.804+0000 7f4f917fa640 10 monclient(hunting): picked mon.noname-a con 0x7f4f740074b0 addr v2:172.21.15.40:3300/0 2024-04-22T08:15:54.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:54.804+0000 7f4f917fa640 1 -- 172.21.15.40:0/2639466349 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f940a02f0 con 0x7f4f74003760 2024-04-22T08:15:54.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:54.804+0000 7f4f917fa640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:54.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:54.804+0000 7f4f917fa640 1 -- 172.21.15.40:0/2639466349 --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f940a39c0 con 0x7f4f740074b0 2024-04-22T08:15:54.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:54.804+0000 7f4f917fa640 10 monclient(hunting): _renew_subs 2024-04-22T08:15:54.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:54.804+0000 7f4f8a7fc640 1 --2- 172.21.15.40:0/2639466349 >> v2:172.21.15.40:3300/0 conn(0x7f4f740074b0 0x7f4f74009970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:54.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:54.804+0000 7f4f8a7fc640 10 monclient(hunting): get_auth_request con 0x7f4f740074b0 auth_method 0 2024-04-22T08:15:54.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:54.804+0000 7f4f8a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:54.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:54.804+0000 7f4f8a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:54.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:54.804+0000 7f4f8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:54.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:54.804+0000 7f4f8a7fc640 1 -- 172.21.15.40:0/2639466349 >> v2:172.21.15.40:3300/0 conn(0x7f4f740074b0 msgr2=0x7f4f74009970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:15:54.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:54.804+0000 7f4f8a7fc640 1 -- 172.21.15.40:0/2639466349 >> v2:172.21.15.40:3300/0 conn(0x7f4f740074b0 msgr2=0x7f4f74009970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:15:54.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:54.804+0000 7f4f8a7fc640 1 --2- 172.21.15.40:0/2639466349 >> v2:172.21.15.40:3300/0 conn(0x7f4f740074b0 0x7f4f74009970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:15:54.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:54.804+0000 7f4f8a7fc640 1 --2- 172.21.15.40:0/2639466349 >> v2:172.21.15.40:3300/0 conn(0x7f4f740074b0 0x7f4f74009970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:15:54.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:54.807+0000 7f4f92ffd640 1 -- 172.21.15.40:0/2639466349 >> v1:172.21.15.189:6789/0 conn(0x7f4f74003760 legacy=0x7f4f74005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:55.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.005+0000 7f4f8a7fc640 1 --2- 172.21.15.40:0/2639466349 >> v2:172.21.15.40:3300/0 conn(0x7f4f740074b0 0x7f4f74009970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:55.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.005+0000 7f4f8a7fc640 10 monclient(hunting): get_auth_request con 0x7f4f740074b0 auth_method 0 2024-04-22T08:15:55.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.005+0000 7f4f8a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:55.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.005+0000 7f4f8a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:55.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.005+0000 7f4f8a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:15:55.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.005+0000 7f4f8a7fc640 1 -- 172.21.15.40:0/2639466349 >> v2:172.21.15.40:3300/0 conn(0x7f4f740074b0 msgr2=0x7f4f74009970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:15:55.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.005+0000 7f4f8a7fc640 1 -- 172.21.15.40:0/2639466349 >> v2:172.21.15.40:3300/0 conn(0x7f4f740074b0 msgr2=0x7f4f74009970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:15:55.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.005+0000 7f4f8a7fc640 1 --2- 172.21.15.40:0/2639466349 >> v2:172.21.15.40:3300/0 conn(0x7f4f740074b0 0x7f4f74009970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:15:55.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.005+0000 7f4f8a7fc640 1 --2- 172.21.15.40:0/2639466349 >> v2:172.21.15.40:3300/0 conn(0x7f4f740074b0 0x7f4f74009970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:15:55.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.406+0000 7f4f8a7fc640 1 --2- 172.21.15.40:0/2639466349 >> v2:172.21.15.40:3300/0 conn(0x7f4f740074b0 0x7f4f74009970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:55.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.406+0000 7f4f8a7fc640 10 monclient(hunting): get_auth_request con 0x7f4f740074b0 auth_method 0 2024-04-22T08:15:55.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.406+0000 7f4f8a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:55.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.406+0000 7f4f8a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:55.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.406+0000 7f4f8a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:15:55.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.406+0000 7f4f8a7fc640 1 -- 172.21.15.40:0/2639466349 >> v2:172.21.15.40:3300/0 conn(0x7f4f740074b0 msgr2=0x7f4f74009970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:15:55.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.406+0000 7f4f8a7fc640 1 -- 172.21.15.40:0/2639466349 >> v2:172.21.15.40:3300/0 conn(0x7f4f740074b0 msgr2=0x7f4f74009970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:15:55.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.406+0000 7f4f8a7fc640 1 --2- 172.21.15.40:0/2639466349 >> v2:172.21.15.40:3300/0 conn(0x7f4f740074b0 0x7f4f74009970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:15:55.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.406+0000 7f4f8a7fc640 1 --2- 172.21.15.40:0/2639466349 >> v2:172.21.15.40:3300/0 conn(0x7f4f740074b0 0x7f4f74009970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:15:55.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.873+0000 7f4f927fc640 1 -- 172.21.15.40:0/2639466349 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 350583948 0 0) 0x7f4f940a02f0 con 0x7f4f74003760 2024-04-22T08:15:55.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.873+0000 7f4f927fc640 10 monclient(hunting): _negotiate 2024-04-22T08:15:55.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.873+0000 7f4f927fc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:55.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.873+0000 7f4f927fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:55.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.873+0000 7f4f927fc640 10 monclient(hunting): my global_id is 24643 2024-04-22T08:15:55.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.873+0000 7f4f927fc640 1 -- 172.21.15.40:0/2639466349 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f80003520 con 0x7f4f74003760 2024-04-22T08:15:55.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f927fc640 1 -- 172.21.15.40:0/2639466349 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3958858719 0 0) 0x7f4f80003520 con 0x7f4f74003760 2024-04-22T08:15:55.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f927fc640 1 -- 172.21.15.40:0/2639466349 >> v2:172.21.15.40:3300/0 conn(0x7f4f740074b0 msgr2=0x7f4f74009970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:15:55.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f927fc640 1 --2- 172.21.15.40:0/2639466349 >> v2:172.21.15.40:3300/0 conn(0x7f4f740074b0 0x7f4f74009970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:55.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f927fc640 10 monclient: _finish_hunting 0 2024-04-22T08:15:55.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f927fc640 1 monclient: found mon.noname-c-legacy 2024-04-22T08:15:55.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:15:55.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f927fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.189:6789/0 2024-04-22T08:15:55.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f927fc640 1 -- 172.21.15.40:0/2639466349 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f7400b0e0 con 0x7f4f74003760 2024-04-22T08:15:55.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f927fc640 10 monclient: _finish_auth 0 2024-04-22T08:15:55.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f927fc640 10 monclient: _check_auth_tickets 2024-04-22T08:15:55.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:15:55.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f927fc640 1 -- 172.21.15.40:0/2639466349 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1087006497 0 0) 0x7f4f8400b690 con 0x7f4f74003760 2024-04-22T08:15:55.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:55.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f927fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-04-22T08:15:55.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f927fc640 10 monclient: dump: 2024-04-22T08:15:55.885 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:55.885 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:55.885 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:55.885 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:55.885 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:55.885 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:55.885 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:55.885 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:55.885 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:55.886 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:55.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f927fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] but i'm connected to v1:172.21.15.189:6789/0, reconnecting 2024-04-22T08:15:55.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f927fc640 10 monclient: _reopen_session rank -1 2024-04-22T08:15:55.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f927fc640 1 -- 172.21.15.40:0/2639466349 >> v1:172.21.15.189:6789/0 conn(0x7f4f74003760 legacy=0x7f4f74005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:55.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f927fc640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-22T08:15:55.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f937fe640 1 -- 172.21.15.40:0/2639466349 reap_dead start 2024-04-22T08:15:55.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f927fc640 1 --2- 172.21.15.40:0/2639466349 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4f80003970 0x7f4f80003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:55.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f927fc640 10 monclient(hunting): picked mon.c con 0x7f4f80003970 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:15:55.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f927fc640 1 --2- 172.21.15.40:0/2639466349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f800047c0 0x7f4f80004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:55.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f927fc640 10 monclient(hunting): picked mon.a con 0x7f4f800047c0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:15:55.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f927fc640 1 --2- 172.21.15.40:0/2639466349 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f800051d0 0x7f4f8000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:55.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f927fc640 10 monclient(hunting): picked mon.b con 0x7f4f800051d0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:15:55.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f927fc640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:55.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f927fc640 1 -- 172.21.15.40:0/2639466349 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f940a39c0 con 0x7f4f800047c0 2024-04-22T08:15:55.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f927fc640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:55.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f927fc640 1 -- 172.21.15.40:0/2639466349 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f8000c100 con 0x7f4f800051d0 2024-04-22T08:15:55.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f927fc640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:55.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f927fc640 1 -- 172.21.15.40:0/2639466349 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f8000c2c0 con 0x7f4f80003970 2024-04-22T08:15:55.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f927fc640 10 monclient(hunting): _renew_subs 2024-04-22T08:15:55.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f92ffd640 1 -- 172.21.15.40:0/2639466349 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4f80003970 msgr2=0x7f4f80003d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:15:55.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f927fc640 10 monclient(hunting): _finish_auth 0 2024-04-22T08:15:55.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f92ffd640 1 --2- 172.21.15.40:0/2639466349 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4f80003970 0x7f4f80003d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:15:55.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f8a7fc640 1 --2- 172.21.15.40:0/2639466349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f800047c0 0x7f4f80004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:55.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f937fe640 1 --2- 172.21.15.40:0/2639466349 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f800051d0 0x7f4f8000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:55.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f8a7fc640 10 monclient(hunting): get_auth_request con 0x7f4f800047c0 auth_method 0 2024-04-22T08:15:55.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f8a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:55.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f8a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:55.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f8a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:15:55.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f937fe640 10 monclient(hunting): get_auth_request con 0x7f4f800051d0 auth_method 0 2024-04-22T08:15:55.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:55.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f937fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:55.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f937fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:15:55.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f8a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:55.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f8a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:55.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f8a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-22T08:15:55.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:55.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:55.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.882+0000 7f4f937fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-22T08:15:55.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.883+0000 7f4f937fe640 10 monclient(hunting): handle_auth_done global_id 24643 payload 293 2024-04-22T08:15:55.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.883+0000 7f4f937fe640 1 -- 172.21.15.40:0/2639466349 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4f80003970 msgr2=0x7f4f80003d70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:15:55.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.883+0000 7f4f937fe640 1 --2- 172.21.15.40:0/2639466349 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4f80003970 0x7f4f80003d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:55.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.883+0000 7f4f937fe640 1 -- 172.21.15.40:0/2639466349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f800047c0 msgr2=0x7f4f80004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:15:55.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.883+0000 7f4f937fe640 1 --2- 172.21.15.40:0/2639466349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f800047c0 0x7f4f80004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:55.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.883+0000 7f4f937fe640 10 monclient: _finish_hunting 0 2024-04-22T08:15:55.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.883+0000 7f4f937fe640 1 monclient: found mon.b 2024-04-22T08:15:55.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.883+0000 7f4f937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:15:55.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.883+0000 7f4f937fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.189:3300/0 2024-04-22T08:15:55.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.883+0000 7f4f937fe640 1 -- 172.21.15.40:0/2639466349 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f80003520 con 0x7f4f800051d0 2024-04-22T08:15:55.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.883+0000 7f4f937fe640 10 monclient: _finish_auth 0 2024-04-22T08:15:55.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.883+0000 7f4f937fe640 10 monclient: _check_auth_tickets 2024-04-22T08:15:55.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.883+0000 7f4f937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:15:55.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.883+0000 7f4f99608640 5 monclient: authenticate success, global_id 24643 2024-04-22T08:15:55.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.883+0000 7f4f8a7fc640 1 --2- 172.21.15.40:0/2639466349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f800047c0 0x7f4f80004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:15:55.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.883+0000 7f4f99608640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:15:55.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.886+0000 7f4f937fe640 1 -- 172.21.15.40:0/2639466349 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f800051d0 msgr2=0x7f4f8000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:55.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.886+0000 7f4f937fe640 1 --2- 172.21.15.40:0/2639466349 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f800051d0 0x7f4f8000b980 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4f94136170 tx=0x7f4f800057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df5f93fcfa5eb3bf server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:55.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.886+0000 7f4f927fc640 1 -- 172.21.15.40:0/2639466349 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f8c02c880 con 0x7f4f800051d0 2024-04-22T08:15:55.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.886+0000 7f4f927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:55.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.886+0000 7f4f927fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:15:55.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.886+0000 7f4f927fc640 10 monclient: dump: 2024-04-22T08:15:55.895 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:55.896 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:55.896 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:55.896 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:55.896 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:55.896 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:55.896 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:55.896 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:55.896 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:55.896 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:55.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.886+0000 7f4f927fc640 1 -- 172.21.15.40:0/2639466349 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f8c02c9e0 con 0x7f4f800051d0 2024-04-22T08:15:55.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.886+0000 7f4f927fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:15:55.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.886+0000 7f4f927fc640 1 -- 172.21.15.40:0/2639466349 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f8c02cd70 con 0x7f4f800051d0 2024-04-22T08:15:55.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.886+0000 7f4f99608640 10 monclient: get_monmap_and_config success 2024-04-22T08:15:55.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.886+0000 7f4f99608640 4 set_mon_vals no callback set 2024-04-22T08:15:55.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.886+0000 7f4f927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:55.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.886+0000 7f4f927fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:15:55.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.886+0000 7f4f927fc640 10 monclient: dump: 2024-04-22T08:15:55.897 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:55.897 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:55.897 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:55.897 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:55.897 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:55.897 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:55.897 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:55.897 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:55.897 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:55.897 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:55.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.887+0000 7f4f99608640 10 monclient: shutdown 2024-04-22T08:15:55.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.887+0000 7f4f99608640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:15:55.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.887+0000 7f4f99608640 1 -- 172.21.15.40:0/2639466349 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f800051d0 msgr2=0x7f4f8000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:55.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.887+0000 7f4f99608640 1 --2- 172.21.15.40:0/2639466349 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f800051d0 0x7f4f8000b980 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4f94136170 tx=0x7f4f800057f0 comp rx=0 tx=0).stop 2024-04-22T08:15:55.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.887+0000 7f4f99608640 1 -- 172.21.15.40:0/2639466349 shutdown_connections 2024-04-22T08:15:55.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.887+0000 7f4f99608640 1 --2- 172.21.15.40:0/2639466349 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f800051d0 0x7f4f8000b980 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:55.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.887+0000 7f4f99608640 1 --2- 172.21.15.40:0/2639466349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f800047c0 0x7f4f80004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:55.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.887+0000 7f4f99608640 1 --2- 172.21.15.40:0/2639466349 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4f80003970 0x7f4f80003d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:55.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.887+0000 7f4f99608640 1 -- 172.21.15.40:0/2639466349 >> 172.21.15.40:0/2639466349 conn(0x7f4f940a4ef0 msgr2=0x7f4f940a5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:15:55.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.887+0000 7f4f99608640 1 -- 172.21.15.40:0/2639466349 shutdown_connections 2024-04-22T08:15:55.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.887+0000 7f4f99608640 1 -- 172.21.15.40:0/2639466349 wait complete. 2024-04-22T08:15:55.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.888+0000 7f4f99608640 10 monclient: build_initial_monmap 2024-04-22T08:15:55.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.888+0000 7f4f99608640 10 monclient: monmap: 2024-04-22T08:15:55.898 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:15:55.899 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:15:55.899 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:15:55.899 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:15:55.899 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:15:55.899 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:55.899 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:15:55.899 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:15:55.899 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:15:55.899 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:55.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.888+0000 7f4f99608640 1 Processor -- start 2024-04-22T08:15:55.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.889+0000 7f4f99608640 1 -- start start 2024-04-22T08:15:55.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.889+0000 7f4f99608640 10 monclient: init 2024-04-22T08:15:55.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.889+0000 7f4f99608640 10 monclient: _reopen_session rank -1 2024-04-22T08:15:55.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.889+0000 7f4f99608640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-22T08:15:55.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.889+0000 7f4f99608640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f80003970 0x7f4f941ca560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:55.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.889+0000 7f4f99608640 10 monclient(hunting): picked mon.noname-a con 0x7f4f80003970 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:15:55.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.889+0000 7f4f99608640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4f800047c0 0x7f4f94134f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:55.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.889+0000 7f4f99608640 10 monclient(hunting): picked mon.noname-c con 0x7f4f800047c0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:15:55.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.889+0000 7f4f99608640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f800051d0 0x7f4f94135480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:55.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.889+0000 7f4f99608640 10 monclient(hunting): picked mon.noname-b con 0x7f4f800051d0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:15:55.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.889+0000 7f4f99608640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:55.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.889+0000 7f4f99608640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f941cb1a0 con 0x7f4f80003970 2024-04-22T08:15:55.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.889+0000 7f4f99608640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:55.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.889+0000 7f4f99608640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f94071d20 con 0x7f4f800051d0 2024-04-22T08:15:55.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.889+0000 7f4f99608640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:55.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.889+0000 7f4f99608640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f940a2730 con 0x7f4f800047c0 2024-04-22T08:15:55.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.889+0000 7f4f99608640 10 monclient(hunting): _renew_subs 2024-04-22T08:15:55.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.889+0000 7f4f99608640 10 monclient(hunting): authenticate will time out at 1029.883538s 2024-04-22T08:15:55.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.889+0000 7f4f8a7fc640 1 -- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4f800047c0 msgr2=0x7f4f94134f40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:15:55.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.889+0000 7f4f8a7fc640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4f800047c0 0x7f4f94134f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:15:55.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.889+0000 7f4f92ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f80003970 0x7f4f941ca560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:55.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.889+0000 7f4f92ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f80003970 0x7f4f941ca560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37370/0 (socket says 172.21.15.40:37370) 2024-04-22T08:15:55.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.889+0000 7f4f92ffd640 1 -- 172.21.15.40:0/3752586972 learned_addr learned my addr 172.21.15.40:0/3752586972 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:15:55.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.889+0000 7f4f92ffd640 10 monclient(hunting): get_auth_request con 0x7f4f80003970 auth_method 0 2024-04-22T08:15:55.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.889+0000 7f4f92ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:55.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.889+0000 7f4f92ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:55.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.889+0000 7f4f92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:55.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.890+0000 7f4f92ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:55.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.890+0000 7f4f92ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:55.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.890+0000 7f4f92ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:55.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.890+0000 7f4f937fe640 1 --2- 172.21.15.40:0/3752586972 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f800051d0 0x7f4f94135480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:55.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.890+0000 7f4f937fe640 10 monclient(hunting): get_auth_request con 0x7f4f800051d0 auth_method 0 2024-04-22T08:15:55.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.890+0000 7f4f937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:55.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.890+0000 7f4f937fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:55.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.890+0000 7f4f937fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:55.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.890+0000 7f4f92ffd640 10 monclient(hunting): handle_auth_done global_id 34431 payload 824 2024-04-22T08:15:55.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.890+0000 7f4f92ffd640 1 -- 172.21.15.40:0/3752586972 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4f800047c0 msgr2=0x7f4f94134f40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:15:55.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.890+0000 7f4f92ffd640 1 --2- 172.21.15.40:0/3752586972 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4f800047c0 0x7f4f94134f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:55.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.890+0000 7f4f92ffd640 1 -- 172.21.15.40:0/3752586972 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f800051d0 msgr2=0x7f4f94135480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:55.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.890+0000 7f4f92ffd640 1 --2- 172.21.15.40:0/3752586972 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f800051d0 0x7f4f94135480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:55.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.890+0000 7f4f92ffd640 10 monclient: _finish_hunting 0 2024-04-22T08:15:55.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.890+0000 7f4f92ffd640 1 monclient: found mon.noname-a 2024-04-22T08:15:55.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.890+0000 7f4f92ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:15:55.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.890+0000 7f4f92ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:15:55.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.890+0000 7f4f92ffd640 1 -- 172.21.15.40:0/3752586972 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f941226f0 con 0x7f4f80003970 2024-04-22T08:15:55.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.892+0000 7f4f92ffd640 1 -- 172.21.15.40:0/3752586972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f80003970 msgr2=0x7f4f941ca560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:55.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.892+0000 7f4f92ffd640 1 --2- 172.21.15.40:0/3752586972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f80003970 0x7f4f941ca560 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4f840116e0 tx=0x7f4f84011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:55.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.892+0000 7f4f8bfff640 1 -- 172.21.15.40:0/3752586972 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f8400fb70 con 0x7f4f80003970 2024-04-22T08:15:55.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.892+0000 7f4f8bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:55.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.892+0000 7f4f8bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:15:55.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.892+0000 7f4f8bfff640 10 monclient: dump: 2024-04-22T08:15:55.903 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:55.903 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:55.903 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:55.903 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:55.903 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:55.903 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:55.904 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:55.904 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:55.904 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:55.904 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:55.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.892+0000 7f4f8bfff640 10 monclient: _finish_auth 0 2024-04-22T08:15:55.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.892+0000 7f4f8bfff640 10 monclient: _check_auth_tickets 2024-04-22T08:15:55.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.892+0000 7f4f8bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:15:55.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.892+0000 7f4f99608640 5 monclient: authenticate success, global_id 34431 2024-04-22T08:15:55.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.892+0000 7f4f8bfff640 1 -- 172.21.15.40:0/3752586972 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f8400fcd0 con 0x7f4f80003970 2024-04-22T08:15:55.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.892+0000 7f4f8bfff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:15:55.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.892+0000 7f4f99608640 10 monclient: _renew_subs 2024-04-22T08:15:55.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.892+0000 7f4f8bfff640 1 -- 172.21.15.40:0/3752586972 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f84015680 con 0x7f4f80003970 2024-04-22T08:15:55.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.892+0000 7f4f99608640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:55.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.892+0000 7f4f99608640 1 -- 172.21.15.40:0/3752586972 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f94121e50 con 0x7f4f80003970 2024-04-22T08:15:55.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.892+0000 7f4f8bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:55.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.892+0000 7f4f8bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:15:55.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.892+0000 7f4f8bfff640 10 monclient: dump: 2024-04-22T08:15:55.905 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:55.905 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:55.905 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:55.905 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:55.905 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:55.905 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:55.905 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:55.905 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:55.905 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:55.905 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:55.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.892+0000 7f4f91ffb640 4 set_mon_vals no callback set 2024-04-22T08:15:55.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.893+0000 7f4f99608640 10 monclient: _renew_subs 2024-04-22T08:15:55.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.893+0000 7f4f99608640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:55.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.893+0000 7f4f99608640 1 -- 172.21.15.40:0/3752586972 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f94120c20 con 0x7f4f80003970 2024-04-22T08:15:55.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.893+0000 7f4f8bfff640 1 -- 172.21.15.40:0/3752586972 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f4f84015820 con 0x7f4f80003970 2024-04-22T08:15:55.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.894+0000 7f4f8bfff640 1 --2- 172.21.15.40:0/3752586972 >> v2:172.21.15.189:6800/30168 conn(0x7f4f74038060 0x7f4f7403a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:55.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.894+0000 7f4f8bfff640 1 -- 172.21.15.40:0/3752586972 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f4f8404ab80 con 0x7f4f80003970 2024-04-22T08:15:55.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.894+0000 7f4f8a7fc640 1 --2- 172.21.15.40:0/3752586972 >> v2:172.21.15.189:6800/30168 conn(0x7f4f74038060 0x7f4f7403a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:55.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.894+0000 7f4f99608640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:55.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.894+0000 7f4f99608640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:55.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.894+0000 7f4f99608640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:55.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.894+0000 7f4f99608640 1 -- 172.21.15.40:0/3752586972 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f9404e9b0 con 0x7f4f80003970 2024-04-22T08:15:55.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.894+0000 7f4f8a7fc640 10 monclient: get_auth_request con 0x7f4f74038060 auth_method 0 2024-04-22T08:15:55.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.897+0000 7f4f8a7fc640 1 --2- 172.21.15.40:0/3752586972 >> v2:172.21.15.189:6800/30168 conn(0x7f4f74038060 0x7f4f7403a520 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4f940a49a0 tx=0x7f4f7c005040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:55.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.899+0000 7f4f8bfff640 1 -- 172.21.15.40:0/3752586972 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f4f840143e0 con 0x7f4f80003970 2024-04-22T08:15:55.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.899+0000 7f4f8bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:55.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:55.899+0000 7f4f8bfff640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:15:56.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.150+0000 7f4f99608640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:15:56.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.150+0000 7f4f99608640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:15:56.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.150+0000 7f4f99608640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:56.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.150+0000 7f4f99608640 1 -- 172.21.15.40:0/3752586972 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4f94122090 con 0x7f4f80003970 2024-04-22T08:15:56.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.150+0000 7f4f8bfff640 1 -- 172.21.15.40:0/3752586972 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1537 (secure 0 0 0) 0x7f4f84013340 con 0x7f4f80003970 2024-04-22T08:15:56.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.150+0000 7f4f8bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:15:56.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.150+0000 7f4f8bfff640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:15:56.150 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":40,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:15:56.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.152+0000 7f4f99608640 1 -- 172.21.15.40:0/3752586972 >> v2:172.21.15.189:6800/30168 conn(0x7f4f74038060 msgr2=0x7f4f7403a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:56.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.152+0000 7f4f99608640 1 --2- 172.21.15.40:0/3752586972 >> v2:172.21.15.189:6800/30168 conn(0x7f4f74038060 0x7f4f7403a520 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4f940a49a0 tx=0x7f4f7c005040 comp rx=0 tx=0).stop 2024-04-22T08:15:56.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.152+0000 7f4f99608640 10 monclient: shutdown 2024-04-22T08:15:56.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.152+0000 7f4f99608640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:15:56.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.152+0000 7f4f99608640 1 -- 172.21.15.40:0/3752586972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f80003970 msgr2=0x7f4f941ca560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:56.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.152+0000 7f4f99608640 1 --2- 172.21.15.40:0/3752586972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f80003970 0x7f4f941ca560 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4f840116e0 tx=0x7f4f84011ba0 comp rx=0 tx=0).stop 2024-04-22T08:15:56.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.152+0000 7f4f99608640 1 -- 172.21.15.40:0/3752586972 shutdown_connections 2024-04-22T08:15:56.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.152+0000 7f4f99608640 1 --2- 172.21.15.40:0/3752586972 >> v2:172.21.15.189:6800/30168 conn(0x7f4f74038060 0x7f4f7403a520 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:56.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.152+0000 7f4f99608640 1 --2- 172.21.15.40:0/3752586972 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f800051d0 0x7f4f94135480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:56.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.152+0000 7f4f99608640 1 --2- 172.21.15.40:0/3752586972 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4f800047c0 0x7f4f94134f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:56.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.152+0000 7f4f99608640 1 --2- 172.21.15.40:0/3752586972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f80003970 0x7f4f941ca560 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:56.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.152+0000 7f4f99608640 1 -- 172.21.15.40:0/3752586972 >> 172.21.15.40:0/3752586972 conn(0x7f4f940a4ef0 msgr2=0x7f4f940a7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:15:56.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.152+0000 7f4f99608640 1 -- 172.21.15.40:0/3752586972 shutdown_connections 2024-04-22T08:15:56.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.152+0000 7f4f99608640 1 -- 172.21.15.40:0/3752586972 wait complete. 2024-04-22T08:15:56.162 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-22T08:15:56.162 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-04-22T08:15:56.163 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-22T08:15:56.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.263+0000 7fc17fc2f640 10 monclient: get_monmap_and_config 2024-04-22T08:15:56.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.263+0000 7fc17fc2f640 10 monclient: build_initial_monmap 2024-04-22T08:15:56.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.263+0000 7fc17fc2f640 10 monclient: monmap: 2024-04-22T08:15:56.263 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:15:56.263 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:56.263 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:15:56.264249+0000 2024-04-22T08:15:56.263 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:15:56.264249+0000 2024-04-22T08:15:56.263 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:15:56.263 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:56.263 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:15:56.263 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:15:56.263 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:15:56.264 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:15:56.264 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:15:56.264 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:56.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.264+0000 7fc17fc2f640 1 Processor -- start 2024-04-22T08:15:56.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.264+0000 7fc17fc2f640 1 -- start start 2024-04-22T08:15:56.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.264+0000 7fc17fc2f640 10 monclient: init 2024-04-22T08:15:56.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.264+0000 7fc17fc2f640 10 monclient: _reopen_session rank -1 2024-04-22T08:15:56.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.264+0000 7fc17fc2f640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-04-22T08:15:56.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.264+0000 7fc17fc2f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc178136070 addr v1:172.21.15.189:6789/0 2024-04-22T08:15:56.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.264+0000 7fc17fc2f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc178136450 addr v1:172.21.15.40:6789/0 2024-04-22T08:15:56.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.264+0000 7fc17fc2f640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1781368b0 0x7fc17813e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:56.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.264+0000 7fc17fc2f640 10 monclient(hunting): picked mon.noname-b con 0x7fc1781368b0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:15:56.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.264+0000 7fc17fc2f640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc17806bb50 con 0x7fc178136450 2024-04-22T08:15:56.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.264+0000 7fc17fc2f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc178099dd0 con 0x7fc178136070 2024-04-22T08:15:56.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.264+0000 7fc17fc2f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:56.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.264+0000 7fc17fc2f640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc17809a390 con 0x7fc1781368b0 2024-04-22T08:15:56.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.264+0000 7fc17fc2f640 10 monclient(hunting): _renew_subs 2024-04-22T08:15:56.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.264+0000 7fc17fc2f640 10 monclient(hunting): authenticate will time out at 1030.258530s 2024-04-22T08:15:56.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.264+0000 7fc17e1cc640 1 -- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1781368b0 msgr2=0x7fc17813e600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:15:56.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.264+0000 7fc17e1cc640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1781368b0 0x7fc17813e600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:15:56.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.264+0000 7fc17d1ca640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fc178136450 0x7fc17813aed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:43964/0 (socket says 172.21.15.40:43964) 2024-04-22T08:15:56.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.264+0000 7fc17d1ca640 1 -- 172.21.15.40:0/987376482 learned_addr learned my addr 172.21.15.40:0/987376482 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:15:56.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17d1ca640 1 -- 172.21.15.40:0/987376482 >> v1:172.21.15.40:6789/0 conn(0x7fc178136450 legacy=0x7fc17813aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:56.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17c9c9640 1 -- 172.21.15.40:0/987376482 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4189242161 0 0) 0x7fc17806bb50 con 0x7fc178136450 2024-04-22T08:15:56.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17c9c9640 10 monclient(hunting): _negotiate 2024-04-22T08:15:56.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17c9c9640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:56.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17c9c9640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:56.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17c9c9640 10 monclient(hunting): my global_id is 34434 2024-04-22T08:15:56.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17c9c9640 1 -- 172.21.15.40:0/987376482 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc164003520 con 0x7fc178136450 2024-04-22T08:15:56.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17d9cb640 1 -- 172.21.15.40:0/987376482 >> v1:172.21.15.189:6789/0 conn(0x7fc178136070 legacy=0x7fc17809aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:56.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17c9c9640 1 -- 172.21.15.40:0/987376482 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 206864114 0 0) 0x7fc178099dd0 con 0x7fc178136070 2024-04-22T08:15:56.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17c9c9640 10 monclient(hunting): _negotiate 2024-04-22T08:15:56.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17c9c9640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:56.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17c9c9640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:56.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17c9c9640 10 monclient(hunting): my global_id is 24649 2024-04-22T08:15:56.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17c9c9640 1 -- 172.21.15.40:0/987376482 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc17806bb50 con 0x7fc178136070 2024-04-22T08:15:56.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17c9c9640 1 -- 172.21.15.40:0/987376482 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3191395209 0 0) 0x7fc164003520 con 0x7fc178136450 2024-04-22T08:15:56.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17c9c9640 1 -- 172.21.15.40:0/987376482 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1781368b0 msgr2=0x7fc17813e600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:15:56.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17c9c9640 1 --2- 172.21.15.40:0/987376482 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1781368b0 0x7fc17813e600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:56.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17c9c9640 1 -- 172.21.15.40:0/987376482 >> v1:172.21.15.189:6789/0 conn(0x7fc178136070 legacy=0x7fc17809aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:56.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17c9c9640 10 monclient: _finish_hunting 0 2024-04-22T08:15:56.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17c9c9640 1 monclient: found mon.noname-a-legacy 2024-04-22T08:15:56.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17c9c9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:15:56.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17c9c9640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.40:6789/0 2024-04-22T08:15:56.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17c9c9640 1 -- 172.21.15.40:0/987376482 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc17809a0b0 con 0x7fc178136450 2024-04-22T08:15:56.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17c9c9640 10 monclient: _finish_auth 0 2024-04-22T08:15:56.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17c9c9640 10 monclient: _check_auth_tickets 2024-04-22T08:15:56.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17c9c9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:15:56.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17fc2f640 5 monclient: authenticate success, global_id 34434 2024-04-22T08:15:56.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17c9c9640 1 -- 172.21.15.40:0/987376482 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1087006497 0 0) 0x7fc168001920 con 0x7fc178136450 2024-04-22T08:15:56.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17fc2f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:15:56.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17c9c9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:56.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17c9c9640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-22T08:15:56.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17c9c9640 10 monclient: dump: 2024-04-22T08:15:56.279 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:56.280 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:56.280 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:56.280 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:56.280 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:56.280 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:56.280 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:56.280 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:56.280 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:56.281 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:56.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17c9c9640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] but i'm connected to v1:172.21.15.40:6789/0, reconnecting 2024-04-22T08:15:56.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17c9c9640 10 monclient: _reopen_session rank -1 2024-04-22T08:15:56.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17c9c9640 1 -- 172.21.15.40:0/987376482 >> v1:172.21.15.40:6789/0 conn(0x7fc178136450 legacy=0x7fc17813aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:56.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17c9c9640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-22T08:15:56.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17c9c9640 1 --2- 172.21.15.40:0/987376482 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc164003b30 0x7fc164003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:56.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17c9c9640 10 monclient(hunting): picked mon.a con 0x7fc164003b30 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:15:56.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17c9c9640 1 --2- 172.21.15.40:0/987376482 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1640047c0 0x7fc164004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:56.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17c9c9640 10 monclient(hunting): picked mon.c con 0x7fc1640047c0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:15:56.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17c9c9640 1 --2- 172.21.15.40:0/987376482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc164005160 0x7fc16400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:56.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17c9c9640 10 monclient(hunting): picked mon.b con 0x7fc164005160 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:15:56.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17c9c9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:56.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17c9c9640 1 -- 172.21.15.40:0/987376482 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc17809a390 con 0x7fc164003b30 2024-04-22T08:15:56.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17c9c9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:56.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17c9c9640 1 -- 172.21.15.40:0/987376482 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc16400c050 con 0x7fc164005160 2024-04-22T08:15:56.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17c9c9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:56.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17c9c9640 1 -- 172.21.15.40:0/987376482 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc16400c1e0 con 0x7fc1640047c0 2024-04-22T08:15:56.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17c9c9640 10 monclient(hunting): _renew_subs 2024-04-22T08:15:56.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17d1ca640 1 -- 172.21.15.40:0/987376482 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1640047c0 msgr2=0x7fc164004c20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:15:56.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17c9c9640 10 monclient(hunting): _finish_auth 0 2024-04-22T08:15:56.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17fc2f640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-04-22T08:15:56.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17d1ca640 1 --2- 172.21.15.40:0/987376482 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1640047c0 0x7fc164004c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:15:56.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17d9cb640 1 --2- 172.21.15.40:0/987376482 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc164003b30 0x7fc164003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:56.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17d9cb640 10 monclient(hunting): get_auth_request con 0x7fc164003b30 auth_method 0 2024-04-22T08:15:56.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17d9cb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:56.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17d9cb640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:56.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17d9cb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:15:56.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17d9cb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:56.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17d9cb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:56.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17d9cb640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-22T08:15:56.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.267+0000 7fc17e1cc640 1 --2- 172.21.15.40:0/987376482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc164005160 0x7fc16400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:56.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.268+0000 7fc17e1cc640 10 monclient(hunting): get_auth_request con 0x7fc164005160 auth_method 0 2024-04-22T08:15:56.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.268+0000 7fc17e1cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:56.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.268+0000 7fc17e1cc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:56.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.268+0000 7fc17e1cc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:15:56.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.268+0000 7fc17d9cb640 10 monclient(hunting): handle_auth_done global_id 34434 payload 293 2024-04-22T08:15:56.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.268+0000 7fc17d9cb640 1 -- 172.21.15.40:0/987376482 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1640047c0 msgr2=0x7fc164004c20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:15:56.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.268+0000 7fc17d9cb640 1 --2- 172.21.15.40:0/987376482 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1640047c0 0x7fc164004c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:56.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.268+0000 7fc17d9cb640 1 -- 172.21.15.40:0/987376482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc164005160 msgr2=0x7fc16400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:15:56.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.268+0000 7fc17d9cb640 1 --2- 172.21.15.40:0/987376482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc164005160 0x7fc16400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:56.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.268+0000 7fc17d9cb640 10 monclient: _finish_hunting 0 2024-04-22T08:15:56.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.268+0000 7fc17d9cb640 1 monclient: found mon.a 2024-04-22T08:15:56.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.268+0000 7fc17d9cb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:15:56.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.268+0000 7fc17d9cb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:56.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.268+0000 7fc17d9cb640 1 -- 172.21.15.40:0/987376482 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc164003520 con 0x7fc164003b30 2024-04-22T08:15:56.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.268+0000 7fc17d9cb640 10 monclient: _finish_auth 0 2024-04-22T08:15:56.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.268+0000 7fc17d9cb640 10 monclient: _check_auth_tickets 2024-04-22T08:15:56.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.268+0000 7fc17d9cb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:15:56.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.270+0000 7fc17d9cb640 1 -- 172.21.15.40:0/987376482 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc164003b30 msgr2=0x7fc164003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:56.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.270+0000 7fc17d9cb640 1 --2- 172.21.15.40:0/987376482 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc164003b30 0x7fc164003f30 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc174007c40 tx=0x7fc1640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b15ae574c7b3f5ef server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:56.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.271+0000 7fc17c9c9640 1 -- 172.21.15.40:0/987376482 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc174007270 con 0x7fc164003b30 2024-04-22T08:15:56.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.271+0000 7fc17c9c9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:56.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.271+0000 7fc17c9c9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:15:56.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.271+0000 7fc17c9c9640 10 monclient: dump: 2024-04-22T08:15:56.285 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:56.285 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:56.286 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:56.286 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:56.286 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:56.286 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:56.286 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:56.286 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:56.286 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:56.286 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:56.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.271+0000 7fc17fc2f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:15:56.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.271+0000 7fc17c9c9640 1 -- 172.21.15.40:0/987376482 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1740073f0 con 0x7fc164003b30 2024-04-22T08:15:56.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.271+0000 7fc17c9c9640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:15:56.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.271+0000 7fc17fc2f640 10 monclient: get_monmap_and_config success 2024-04-22T08:15:56.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.271+0000 7fc17c9c9640 1 -- 172.21.15.40:0/987376482 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1740057c0 con 0x7fc164003b30 2024-04-22T08:15:56.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.271+0000 7fc17fc2f640 4 set_mon_vals no callback set 2024-04-22T08:15:56.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.271+0000 7fc17c9c9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:56.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.271+0000 7fc17c9c9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:15:56.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.271+0000 7fc17c9c9640 10 monclient: dump: 2024-04-22T08:15:56.287 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:56.287 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:56.287 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:56.287 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:56.287 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:56.287 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:56.287 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:56.287 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:56.287 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:56.287 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:56.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.271+0000 7fc17fc2f640 10 monclient: shutdown 2024-04-22T08:15:56.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.271+0000 7fc17fc2f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:15:56.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.271+0000 7fc17fc2f640 1 -- 172.21.15.40:0/987376482 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc164003b30 msgr2=0x7fc164003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:56.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.271+0000 7fc17fc2f640 1 --2- 172.21.15.40:0/987376482 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc164003b30 0x7fc164003f30 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc174007c40 tx=0x7fc1640057f0 comp rx=0 tx=0).stop 2024-04-22T08:15:56.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.271+0000 7fc17e1cc640 1 -- 172.21.15.40:0/987376482 reap_dead start 2024-04-22T08:15:56.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.271+0000 7fc17fc2f640 1 -- 172.21.15.40:0/987376482 shutdown_connections 2024-04-22T08:15:56.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.271+0000 7fc17fc2f640 1 -- 172.21.15.40:0/987376482 >> 172.21.15.40:0/987376482 conn(0x7fc1780a1b60 msgr2=0x7fc1780a1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:15:56.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.271+0000 7fc17fc2f640 1 -- 172.21.15.40:0/987376482 shutdown_connections 2024-04-22T08:15:56.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.271+0000 7fc17fc2f640 1 -- 172.21.15.40:0/987376482 wait complete. 2024-04-22T08:15:56.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.272+0000 7fc17fc2f640 10 monclient: build_initial_monmap 2024-04-22T08:15:56.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.272+0000 7fc17fc2f640 10 monclient: monmap: 2024-04-22T08:15:56.288 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:15:56.288 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:15:56.289 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:15:56.289 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:15:56.289 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:15:56.289 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:56.289 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:15:56.289 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:15:56.289 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:15:56.289 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:56.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.272+0000 7fc17fc2f640 1 Processor -- start 2024-04-22T08:15:56.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.272+0000 7fc17fc2f640 1 -- start start 2024-04-22T08:15:56.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.272+0000 7fc17fc2f640 10 monclient: init 2024-04-22T08:15:56.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.273+0000 7fc17fc2f640 10 monclient: _reopen_session rank -1 2024-04-22T08:15:56.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.273+0000 7fc17fc2f640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-22T08:15:56.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.273+0000 7fc17fc2f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc164003b30 0x7fc178134a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:56.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.273+0000 7fc17fc2f640 10 monclient(hunting): picked mon.noname-b con 0x7fc164003b30 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:15:56.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.273+0000 7fc17fc2f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc178138350 0x7fc178138750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:56.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.273+0000 7fc17fc2f640 10 monclient(hunting): picked mon.noname-a con 0x7fc178138350 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:15:56.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.273+0000 7fc17fc2f640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc178138c90 0x7fc178139140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:56.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.273+0000 7fc17fc2f640 10 monclient(hunting): picked mon.noname-c con 0x7fc178138c90 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:15:56.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.273+0000 7fc17fc2f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:56.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.273+0000 7fc17fc2f640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc178120400 con 0x7fc178138350 2024-04-22T08:15:56.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.273+0000 7fc17fc2f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:56.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.273+0000 7fc17fc2f640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc178139fa0 con 0x7fc164003b30 2024-04-22T08:15:56.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.273+0000 7fc17fc2f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:56.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.273+0000 7fc17fc2f640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc17809b470 con 0x7fc178138c90 2024-04-22T08:15:56.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.273+0000 7fc17fc2f640 10 monclient(hunting): _renew_subs 2024-04-22T08:15:56.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.273+0000 7fc17fc2f640 10 monclient(hunting): authenticate will time out at 1030.266888s 2024-04-22T08:15:56.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.273+0000 7fc17e1cc640 1 -- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc178138c90 msgr2=0x7fc178139140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:15:56.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.273+0000 7fc17e1cc640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc178138c90 0x7fc178139140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:15:56.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.273+0000 7fc17d1ca640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc178138350 0x7fc178138750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:56.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.273+0000 7fc17d1ca640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc178138350 0x7fc178138750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37386/0 (socket says 172.21.15.40:37386) 2024-04-22T08:15:56.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.273+0000 7fc17d1ca640 1 -- 172.21.15.40:0/2787858521 learned_addr learned my addr 172.21.15.40:0/2787858521 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:15:56.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.273+0000 7fc17d1ca640 10 monclient(hunting): get_auth_request con 0x7fc178138350 auth_method 0 2024-04-22T08:15:56.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.273+0000 7fc17d1ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:56.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.273+0000 7fc17d1ca640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:56.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.273+0000 7fc17d1ca640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:56.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.273+0000 7fc17d1ca640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:56.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.273+0000 7fc17d1ca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:56.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.273+0000 7fc17d1ca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:56.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.273+0000 7fc17d9cb640 1 --2- 172.21.15.40:0/2787858521 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc164003b30 0x7fc178134a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:56.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.273+0000 7fc17d9cb640 10 monclient(hunting): get_auth_request con 0x7fc164003b30 auth_method 0 2024-04-22T08:15:56.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.273+0000 7fc17d9cb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:56.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.273+0000 7fc17d9cb640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:56.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.273+0000 7fc17d9cb640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:56.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.273+0000 7fc17d1ca640 10 monclient(hunting): handle_auth_done global_id 34437 payload 824 2024-04-22T08:15:56.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.273+0000 7fc17d1ca640 1 -- 172.21.15.40:0/2787858521 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc178138c90 msgr2=0x7fc178139140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:15:56.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.273+0000 7fc17d1ca640 1 --2- 172.21.15.40:0/2787858521 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc178138c90 0x7fc178139140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:56.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.273+0000 7fc17d1ca640 1 -- 172.21.15.40:0/2787858521 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc164003b30 msgr2=0x7fc178134a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:56.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.273+0000 7fc17d1ca640 1 --2- 172.21.15.40:0/2787858521 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc164003b30 0x7fc178134a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:56.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.273+0000 7fc17d1ca640 10 monclient: _finish_hunting 0 2024-04-22T08:15:56.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.273+0000 7fc17d1ca640 1 monclient: found mon.noname-a 2024-04-22T08:15:56.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.273+0000 7fc17d1ca640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:15:56.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.273+0000 7fc17d1ca640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:15:56.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.273+0000 7fc17d1ca640 1 -- 172.21.15.40:0/2787858521 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc17813bd30 con 0x7fc178138350 2024-04-22T08:15:56.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.275+0000 7fc17d1ca640 1 -- 172.21.15.40:0/2787858521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc178138350 msgr2=0x7fc178138750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:56.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.275+0000 7fc17d1ca640 1 --2- 172.21.15.40:0/2787858521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc178138350 0x7fc178138750 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc16800c700 tx=0x7fc16800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:56.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.275+0000 7fc16e7fc640 1 -- 172.21.15.40:0/2787858521 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1680074e0 con 0x7fc178138350 2024-04-22T08:15:56.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.275+0000 7fc16e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:56.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.275+0000 7fc16e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:15:56.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.275+0000 7fc16e7fc640 10 monclient: dump: 2024-04-22T08:15:56.293 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:56.293 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:56.293 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:56.293 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:56.293 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:56.293 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:56.293 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:56.294 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:56.294 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:56.294 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:56.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.275+0000 7fc16e7fc640 10 monclient: _finish_auth 0 2024-04-22T08:15:56.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.275+0000 7fc16e7fc640 10 monclient: _check_auth_tickets 2024-04-22T08:15:56.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.275+0000 7fc16e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:15:56.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.275+0000 7fc16e7fc640 1 -- 172.21.15.40:0/2787858521 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc168007640 con 0x7fc178138350 2024-04-22T08:15:56.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.275+0000 7fc17fc2f640 5 monclient: authenticate success, global_id 34437 2024-04-22T08:15:56.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.275+0000 7fc17fc2f640 10 monclient: _renew_subs 2024-04-22T08:15:56.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.275+0000 7fc17fc2f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:56.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.275+0000 7fc17fc2f640 1 -- 172.21.15.40:0/2787858521 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1781cb400 con 0x7fc178138350 2024-04-22T08:15:56.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.275+0000 7fc16e7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:15:56.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.275+0000 7fc16e7fc640 1 -- 172.21.15.40:0/2787858521 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc168004a20 con 0x7fc178138350 2024-04-22T08:15:56.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.275+0000 7fc16e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:56.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.275+0000 7fc16e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:15:56.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.275+0000 7fc16e7fc640 10 monclient: dump: 2024-04-22T08:15:56.295 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:56.295 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:56.295 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:56.295 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:56.295 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:56.295 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:56.295 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:56.295 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:56.295 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:56.295 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:56.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.275+0000 7fc16ffff640 4 set_mon_vals no callback set 2024-04-22T08:15:56.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.276+0000 7fc17fc2f640 10 monclient: _renew_subs 2024-04-22T08:15:56.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.276+0000 7fc17fc2f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:56.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.276+0000 7fc17fc2f640 1 -- 172.21.15.40:0/2787858521 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc17811fb60 con 0x7fc178138350 2024-04-22T08:15:56.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.276+0000 7fc17fc2f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:56.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.276+0000 7fc17fc2f640 1 --2- 172.21.15.40:0/2787858521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc17802c380 0x7fc178060ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:56.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.276+0000 7fc17fc2f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:56.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.276+0000 7fc17fc2f640 1 -- 172.21.15.40:0/2787858521 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1780667a0 con 0x7fc17802c380 2024-04-22T08:15:56.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.276+0000 7fc16e7fc640 1 -- 172.21.15.40:0/2787858521 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fc168004b80 con 0x7fc178138350 2024-04-22T08:15:56.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.277+0000 7fc17d9cb640 1 --2- 172.21.15.40:0/2787858521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc17802c380 0x7fc178060ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:56.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.277+0000 7fc17d9cb640 10 monclient: get_auth_request con 0x7fc17802c380 auth_method 0 2024-04-22T08:15:56.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.277+0000 7fc17d9cb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:56.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.277+0000 7fc17d9cb640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:56.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.277+0000 7fc17d9cb640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:56.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.277+0000 7fc17d9cb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:56.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.277+0000 7fc17d9cb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:56.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.277+0000 7fc16e7fc640 1 --2- 172.21.15.40:0/2787858521 >> v2:172.21.15.189:6800/30168 conn(0x7fc15c0377d0 0x7fc15c039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:56.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.277+0000 7fc17d9cb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:56.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.277+0000 7fc16e7fc640 1 -- 172.21.15.40:0/2787858521 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 3898+0+0 (secure 0 0 0) 0x7fc168049c60 con 0x7fc178138350 2024-04-22T08:15:56.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.277+0000 7fc17d9cb640 10 monclient(hunting): handle_auth_done global_id 34440 payload 824 2024-04-22T08:15:56.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.277+0000 7fc17d9cb640 1 -- 172.21.15.40:0/2787858521 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc17813c5f0 con 0x7fc17802c380 2024-04-22T08:15:56.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.277+0000 7fc17e1cc640 1 --2- 172.21.15.40:0/2787858521 >> v2:172.21.15.189:6800/30168 conn(0x7fc15c0377d0 0x7fc15c039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:56.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.277+0000 7fc17e1cc640 10 monclient: get_auth_request con 0x7fc15c0377d0 auth_method 0 2024-04-22T08:15:56.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.279+0000 7fc17d9cb640 1 -- 172.21.15.40:0/2787858521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc17802c380 msgr2=0x7fc178060ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:56.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.279+0000 7fc17d9cb640 1 --2- 172.21.15.40:0/2787858521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc17802c380 0x7fc178060ff0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc174033000 tx=0x7fc174001690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:56.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.279+0000 7fc16e7fc640 1 -- 172.21.15.40:0/2787858521 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc174040070 con 0x7fc17802c380 2024-04-22T08:15:56.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.279+0000 7fc16e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:56.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.279+0000 7fc16e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:15:56.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.279+0000 7fc16e7fc640 10 monclient: dump: 2024-04-22T08:15:56.300 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:56.300 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:56.300 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:56.300 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:56.300 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:56.300 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:56.300 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:56.300 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:56.301 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:56.301 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:56.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.279+0000 7fc16e7fc640 1 -- 172.21.15.40:0/2787858521 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7fc17402f440 con 0x7fc17802c380 2024-04-22T08:15:56.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.279+0000 7fc16e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:56.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.279+0000 7fc16e7fc640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:15:56.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.279+0000 7fc16e7fc640 1 -- 172.21.15.40:0/2787858521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc17802c380 msgr2=0x7fc178060ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:56.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.279+0000 7fc16e7fc640 1 --2- 172.21.15.40:0/2787858521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc17802c380 0x7fc178060ff0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc174033000 tx=0x7fc174001690 comp rx=0 tx=0).stop 2024-04-22T08:15:56.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.279+0000 7fc16e7fc640 1 -- 172.21.15.40:0/2787858521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc17802c380 msgr2=0x7fc178060ff0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:15:56.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.279+0000 7fc16e7fc640 1 --2- 172.21.15.40:0/2787858521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc17802c380 0x7fc178060ff0 secure :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc174033000 tx=0x7fc174001690 comp rx=0 tx=0).stop 2024-04-22T08:15:56.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.280+0000 7fc17e1cc640 1 --2- 172.21.15.40:0/2787858521 >> v2:172.21.15.189:6800/30168 conn(0x7fc15c0377d0 0x7fc15c039c90 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc178066a10 tx=0x7fc1700092d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:56.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.286+0000 7fc17fc2f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:15:56.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.286+0000 7fc17fc2f640 1 --2- 172.21.15.40:0/2787858521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc178061530 0x7fc1781d58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:56.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.286+0000 7fc17fc2f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:56.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.286+0000 7fc17fc2f640 1 -- 172.21.15.40:0/2787858521 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc178120080 con 0x7fc178061530 2024-04-22T08:15:56.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.286+0000 7fc17d9cb640 1 --2- 172.21.15.40:0/2787858521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc178061530 0x7fc1781d58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:56.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.286+0000 7fc17d9cb640 10 monclient: get_auth_request con 0x7fc178061530 auth_method 0 2024-04-22T08:15:56.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.286+0000 7fc17d9cb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:56.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.286+0000 7fc17d9cb640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:56.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.286+0000 7fc17d9cb640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:56.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.286+0000 7fc17d9cb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:56.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.286+0000 7fc17d9cb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:56.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.286+0000 7fc17d9cb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:56.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.286+0000 7fc17d9cb640 10 monclient(hunting): handle_auth_done global_id 34443 payload 824 2024-04-22T08:15:56.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.286+0000 7fc17d9cb640 1 -- 172.21.15.40:0/2787858521 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc17802c760 con 0x7fc178061530 2024-04-22T08:15:56.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.288+0000 7fc17d9cb640 1 -- 172.21.15.40:0/2787858521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc178061530 msgr2=0x7fc1781d58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:56.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.288+0000 7fc17d9cb640 1 --2- 172.21.15.40:0/2787858521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc178061530 0x7fc1781d58a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc174007c40 tx=0x7fc174009030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:56.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.289+0000 7fc16e7fc640 1 -- 172.21.15.40:0/2787858521 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17404e650 con 0x7fc178061530 2024-04-22T08:15:56.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.289+0000 7fc16e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:56.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.289+0000 7fc16e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:15:56.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.289+0000 7fc16e7fc640 10 monclient: dump: 2024-04-22T08:15:56.303 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:56.303 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:56.303 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:56.303 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:56.303 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:56.303 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:56.303 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:56.303 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:56.303 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:56.303 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:56.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.289+0000 7fc16e7fc640 1 -- 172.21.15.40:0/2787858521 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4622 (secure 0 0 0) 0x7fc17404e7b0 con 0x7fc178061530 2024-04-22T08:15:56.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.289+0000 7fc16e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:15:56.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.289+0000 7fc16e7fc640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:15:56.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.289+0000 7fc16e7fc640 1 -- 172.21.15.40:0/2787858521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc178061530 msgr2=0x7fc1781d58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:56.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.289+0000 7fc16e7fc640 1 --2- 172.21.15.40:0/2787858521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc178061530 0x7fc1781d58a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc174007c40 tx=0x7fc174009030 comp rx=0 tx=0).stop 2024-04-22T08:15:56.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.289+0000 7fc16e7fc640 1 -- 172.21.15.40:0/2787858521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc178061530 msgr2=0x7fc1781d58a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:15:56.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.289+0000 7fc16e7fc640 1 --2- 172.21.15.40:0/2787858521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc178061530 0x7fc1781d58a0 secure :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc174007c40 tx=0x7fc174009030 comp rx=0 tx=0).stop 2024-04-22T08:15:56.304 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:15:56.304 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:15:56.304 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:15:56.304 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "leader", 2024-04-22T08:15:56.304 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 40, 2024-04-22T08:15:56.304 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:15:56.305 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:15:56.305 INFO:teuthology.orchestra.run.smithi040.stdout: 1 2024-04-22T08:15:56.305 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:56.305 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 0, 2024-04-22T08:15:56.305 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:15:56.305 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:15:56.305 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:15:56.305 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:15:56.305 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:15:56.305 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:15:56.305 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:15:56.305 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:15:56.305 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:15:56.306 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:15:56.306 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:15:56.306 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:15:56.306 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:15:56.306 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:56.306 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:15:56.306 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:15:56.306 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:15:56.306 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:15:56.306 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:15:56.306 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:15:56.306 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:15:56.307 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:15:56.307 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:15:56.307 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:15:56.307 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:15:56.307 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:15:56.307 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:56.307 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:56.307 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:15:56.307 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:15:56.307 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:15:56.307 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:15:56.307 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:15:56.307 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:15:56.307 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:15:56.308 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:15:56.308 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:15:56.308 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:15:56.308 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:15:56.308 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:15:56.308 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:15:56.308 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:15:56.308 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:15:56.308 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:15:56.308 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:15:56.308 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:15:56.308 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:15:56.308 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:15:56.308 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:15:56.309 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:15:56.309 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:15:56.309 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:15:56.309 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:15:56.309 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:15:56.309 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:15:56.309 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:56.309 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:15:56.309 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:56.309 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:15:56.309 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:56.309 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:15:56.309 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:15:56.309 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:15:56.310 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:15:56.310 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:56.310 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:15:56.310 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:15:56.310 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:56.310 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:56.310 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:56.310 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:15:56.310 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:15:56.310 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:56.310 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:56.310 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:56.310 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:56.310 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:15:56.311 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:15:56.311 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:15:56.311 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:15:56.311 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:15:56.311 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:56.311 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:56.311 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:15:56.311 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:15:56.311 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:15:56.311 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:15:56.311 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:56.311 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:15:56.311 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:15:56.312 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:56.312 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:56.312 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:56.312 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:15:56.312 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:15:56.312 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:56.312 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:56.312 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:56.312 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:56.312 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:15:56.312 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:15:56.312 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:15:56.312 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:15:56.312 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:15:56.313 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:56.313 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:56.313 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:15:56.313 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:15:56.313 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:15:56.313 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:15:56.313 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:56.313 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:15:56.313 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:15:56.313 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:56.313 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:56.313 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:56.313 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:15:56.313 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:15:56.314 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:56.314 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:56.314 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:56.314 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:56.314 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:15:56.314 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:15:56.314 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:15:56.314 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:15:56.314 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:15:56.314 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:56.314 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:56.314 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:56.314 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:15:56.315 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:15:56.315 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:56.315 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:15:56.315 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:15:56.315 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:15:56.315 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:56.315 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:56.315 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:15:56.315 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:56.315 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:15:56.315 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:15:56.315 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 4 2024-04-22T08:15:56.315 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:56.315 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:56.315 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:15:56.316 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:56.316 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:15:56.316 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:15:56.316 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 3 2024-04-22T08:15:56.316 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:56.316 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:56.316 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:56.316 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:15:56.316 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:15:56.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.289+0000 7fc17fc2f640 1 -- 172.21.15.40:0/2787858521 >> v2:172.21.15.189:6800/30168 conn(0x7fc15c0377d0 msgr2=0x7fc15c039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:56.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.289+0000 7fc17fc2f640 1 --2- 172.21.15.40:0/2787858521 >> v2:172.21.15.189:6800/30168 conn(0x7fc15c0377d0 0x7fc15c039c90 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc178066a10 tx=0x7fc1700092d0 comp rx=0 tx=0).stop 2024-04-22T08:15:56.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.289+0000 7fc17fc2f640 10 monclient: shutdown 2024-04-22T08:15:56.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.289+0000 7fc17fc2f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:15:56.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.289+0000 7fc17fc2f640 1 -- 172.21.15.40:0/2787858521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc178138350 msgr2=0x7fc178138750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:56.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.289+0000 7fc17fc2f640 1 --2- 172.21.15.40:0/2787858521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc178138350 0x7fc178138750 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc16800c700 tx=0x7fc16800cbc0 comp rx=0 tx=0).stop 2024-04-22T08:15:56.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.289+0000 7fc17e1cc640 1 -- 172.21.15.40:0/2787858521 reap_dead start 2024-04-22T08:15:56.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.289+0000 7fc17e1cc640 1 -- 172.21.15.40:0/2787858521 reap_dead start 2024-04-22T08:15:56.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.289+0000 7fc17fc2f640 1 -- 172.21.15.40:0/2787858521 shutdown_connections 2024-04-22T08:15:56.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.289+0000 7fc17fc2f640 1 -- 172.21.15.40:0/2787858521 >> 172.21.15.40:0/2787858521 conn(0x7fc1780a1b60 msgr2=0x7fc1780a02f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:15:56.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.289+0000 7fc17fc2f640 1 -- 172.21.15.40:0/2787858521 shutdown_connections 2024-04-22T08:15:56.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.289+0000 7fc17fc2f640 1 -- 172.21.15.40:0/2787858521 wait complete. 2024-04-22T08:15:56.319 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-22T08:15:56.319 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-04-22T08:15:56.319 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-22T08:15:56.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.419+0000 7fe791c5f640 10 monclient: get_monmap_and_config 2024-04-22T08:15:56.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.419+0000 7fe791c5f640 10 monclient: build_initial_monmap 2024-04-22T08:15:56.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.419+0000 7fe791c5f640 10 monclient: monmap: 2024-04-22T08:15:56.419 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:15:56.419 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:56.419 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:15:56.420321+0000 2024-04-22T08:15:56.419 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:15:56.420321+0000 2024-04-22T08:15:56.419 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:15:56.419 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:56.419 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:15:56.420 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:15:56.420 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:15:56.420 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:15:56.420 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:15:56.420 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:56.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.420+0000 7fe791c5f640 1 Processor -- start 2024-04-22T08:15:56.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.420+0000 7fe791c5f640 1 -- start start 2024-04-22T08:15:56.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.420+0000 7fe791c5f640 10 monclient: init 2024-04-22T08:15:56.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.420+0000 7fe791c5f640 10 monclient: _reopen_session rank -1 2024-04-22T08:15:56.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.420+0000 7fe791c5f640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-04-22T08:15:56.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.420+0000 7fe791c5f640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe78c0a1c60 0x7fe78c0a31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:56.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.420+0000 7fe791c5f640 10 monclient(hunting): picked mon.noname-b con 0x7fe78c0a1c60 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:15:56.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.420+0000 7fe791c5f640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe78c0a2040 0x7fe78c135bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:56.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.420+0000 7fe791c5f640 10 monclient(hunting): picked mon.noname-a con 0x7fe78c0a2040 addr v2:172.21.15.40:3300/0 2024-04-22T08:15:56.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.420+0000 7fe791c5f640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe78c0a24a0 0x7fe78c1360f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:56.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.420+0000 7fe791c5f640 10 monclient(hunting): picked mon.noname-c con 0x7fe78c0a24a0 addr v2:172.21.15.189:3300/0 2024-04-22T08:15:56.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.420+0000 7fe791c5f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:56.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.420+0000 7fe791c5f640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe78c0a41f0 con 0x7fe78c0a2040 2024-04-22T08:15:56.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.420+0000 7fe791c5f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:56.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.420+0000 7fe791c5f640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe78c09fe90 con 0x7fe78c0a24a0 2024-04-22T08:15:56.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.420+0000 7fe791c5f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:56.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.420+0000 7fe791c5f640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe78c0a0170 con 0x7fe78c0a1c60 2024-04-22T08:15:56.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.420+0000 7fe791c5f640 10 monclient(hunting): _renew_subs 2024-04-22T08:15:56.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.420+0000 7fe791c5f640 10 monclient(hunting): authenticate will time out at 1030.414540s 2024-04-22T08:15:56.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.420+0000 7fe78b7fe640 1 -- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe78c0a1c60 msgr2=0x7fe78c0a31d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:15:56.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.420+0000 7fe78b7fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe78c0a1c60 0x7fe78c0a31d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:15:56.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.420+0000 7fe78affd640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe78c0a2040 0x7fe78c135bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:56.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.420+0000 7fe78affd640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe78c0a2040 0x7fe78c135bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37410/0 (socket says 172.21.15.40:37410) 2024-04-22T08:15:56.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.420+0000 7fe78affd640 1 -- 172.21.15.40:0/3635485743 learned_addr learned my addr 172.21.15.40:0/3635485743 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:15:56.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.420+0000 7fe78affd640 10 monclient(hunting): get_auth_request con 0x7fe78c0a2040 auth_method 0 2024-04-22T08:15:56.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.420+0000 7fe78affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:56.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.420+0000 7fe78affd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:56.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.420+0000 7fe78affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:56.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.421+0000 7fe78affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:56.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.421+0000 7fe78affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:56.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.421+0000 7fe78affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:56.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.421+0000 7fe78bfff640 1 --2- 172.21.15.40:0/3635485743 >> v2:172.21.15.189:3300/0 conn(0x7fe78c0a24a0 0x7fe78c1360f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:56.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.421+0000 7fe78affd640 10 monclient(hunting): handle_auth_done global_id 34446 payload 274 2024-04-22T08:15:56.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.421+0000 7fe78affd640 1 -- 172.21.15.40:0/3635485743 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe78c0a1c60 msgr2=0x7fe78c0a31d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:15:56.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.421+0000 7fe78affd640 1 --2- 172.21.15.40:0/3635485743 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe78c0a1c60 0x7fe78c0a31d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:56.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.421+0000 7fe78affd640 1 -- 172.21.15.40:0/3635485743 >> v2:172.21.15.189:3300/0 conn(0x7fe78c0a24a0 msgr2=0x7fe78c1360f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:15:56.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.421+0000 7fe78affd640 1 --2- 172.21.15.40:0/3635485743 >> v2:172.21.15.189:3300/0 conn(0x7fe78c0a24a0 0x7fe78c1360f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:56.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.421+0000 7fe78affd640 10 monclient: _finish_hunting 0 2024-04-22T08:15:56.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.421+0000 7fe78affd640 1 monclient: found mon.noname-a 2024-04-22T08:15:56.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.421+0000 7fe78affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:15:56.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.421+0000 7fe78affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:15:56.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.421+0000 7fe78affd640 1 -- 172.21.15.40:0/3635485743 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe78c0a0450 con 0x7fe78c0a2040 2024-04-22T08:15:56.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.424+0000 7fe78affd640 1 -- 172.21.15.40:0/3635485743 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe78c0a2040 msgr2=0x7fe78c135bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:56.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.424+0000 7fe78affd640 1 --2- 172.21.15.40:0/3635485743 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe78c0a2040 0x7fe78c135bb0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe78c071b30 tx=0x7fe78002b800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69fe8500e1a3aa42 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:56.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.424+0000 7fe78a7fc640 1 -- 172.21.15.40:0/3635485743 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe780031a50 con 0x7fe78c0a2040 2024-04-22T08:15:56.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.424+0000 7fe78a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:56.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.424+0000 7fe78a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:15:56.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.424+0000 7fe78a7fc640 10 monclient: dump: 2024-04-22T08:15:56.432 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:56.432 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:56.432 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:56.432 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:56.432 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:56.432 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:56.432 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:56.432 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:56.432 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:56.432 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:56.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.424+0000 7fe78a7fc640 10 monclient: _finish_auth 0 2024-04-22T08:15:56.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.424+0000 7fe78a7fc640 10 monclient: _check_auth_tickets 2024-04-22T08:15:56.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.424+0000 7fe78a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:15:56.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.424+0000 7fe78a7fc640 1 -- 172.21.15.40:0/3635485743 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe780031bb0 con 0x7fe78c0a2040 2024-04-22T08:15:56.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.424+0000 7fe791c5f640 5 monclient: authenticate success, global_id 34446 2024-04-22T08:15:56.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.424+0000 7fe791c5f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:15:56.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.424+0000 7fe78a7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:15:56.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.424+0000 7fe78a7fc640 1 -- 172.21.15.40:0/3635485743 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7800323f0 con 0x7fe78c0a2040 2024-04-22T08:15:56.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.424+0000 7fe791c5f640 10 monclient: get_monmap_and_config success 2024-04-22T08:15:56.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.424+0000 7fe791c5f640 4 set_mon_vals no callback set 2024-04-22T08:15:56.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.424+0000 7fe78a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:56.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.424+0000 7fe78a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:15:56.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.424+0000 7fe78a7fc640 10 monclient: dump: 2024-04-22T08:15:56.433 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:56.433 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:56.433 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:56.433 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:56.433 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:56.434 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:56.434 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:56.434 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:56.434 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:56.434 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:56.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.424+0000 7fe791c5f640 10 monclient: shutdown 2024-04-22T08:15:56.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.424+0000 7fe791c5f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:15:56.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.424+0000 7fe791c5f640 1 -- 172.21.15.40:0/3635485743 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe78c0a2040 msgr2=0x7fe78c135bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:56.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.424+0000 7fe791c5f640 1 --2- 172.21.15.40:0/3635485743 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe78c0a2040 0x7fe78c135bb0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe78c071b30 tx=0x7fe78002b800 comp rx=0 tx=0).stop 2024-04-22T08:15:56.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.424+0000 7fe791c5f640 1 -- 172.21.15.40:0/3635485743 shutdown_connections 2024-04-22T08:15:56.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.424+0000 7fe791c5f640 1 --2- 172.21.15.40:0/3635485743 >> v2:172.21.15.189:3300/0 conn(0x7fe78c0a24a0 0x7fe78c1360f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:56.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.424+0000 7fe791c5f640 1 --2- 172.21.15.40:0/3635485743 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe78c0a2040 0x7fe78c135bb0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:56.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.424+0000 7fe791c5f640 1 --2- 172.21.15.40:0/3635485743 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe78c0a1c60 0x7fe78c0a31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:56.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.424+0000 7fe791c5f640 1 -- 172.21.15.40:0/3635485743 >> 172.21.15.40:0/3635485743 conn(0x7fe78c0a89a0 msgr2=0x7fe78c0a4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:15:56.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.424+0000 7fe791c5f640 1 -- 172.21.15.40:0/3635485743 shutdown_connections 2024-04-22T08:15:56.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.424+0000 7fe791c5f640 1 -- 172.21.15.40:0/3635485743 wait complete. 2024-04-22T08:15:56.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.425+0000 7fe791c5f640 10 monclient: build_initial_monmap 2024-04-22T08:15:56.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.425+0000 7fe791c5f640 10 monclient: monmap: 2024-04-22T08:15:56.435 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:15:56.435 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:15:56.435 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:15:56.435 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:15:56.435 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:15:56.435 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:56.435 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:15:56.435 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:15:56.435 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:15:56.435 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:56.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.425+0000 7fe791c5f640 1 Processor -- start 2024-04-22T08:15:56.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.425+0000 7fe791c5f640 1 -- start start 2024-04-22T08:15:56.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.425+0000 7fe791c5f640 10 monclient: init 2024-04-22T08:15:56.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.426+0000 7fe791c5f640 10 monclient: _reopen_session rank -1 2024-04-22T08:15:56.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.426+0000 7fe791c5f640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-22T08:15:56.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.426+0000 7fe791c5f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe78c0a1c60 0x7fe78c1202f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:56.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.426+0000 7fe791c5f640 10 monclient(hunting): picked mon.noname-b con 0x7fe78c0a1c60 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:15:56.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.426+0000 7fe791c5f640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe78c0a2040 0x7fe78c120830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:56.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.426+0000 7fe791c5f640 10 monclient(hunting): picked mon.noname-c con 0x7fe78c0a2040 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:15:56.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.426+0000 7fe791c5f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe78c0a24a0 0x7fe78c120d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:56.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.426+0000 7fe791c5f640 10 monclient(hunting): picked mon.noname-a con 0x7fe78c0a24a0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:15:56.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.426+0000 7fe791c5f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:56.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.426+0000 7fe791c5f640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe78c1227e0 con 0x7fe78c0a24a0 2024-04-22T08:15:56.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.426+0000 7fe791c5f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:56.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.426+0000 7fe791c5f640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe78c124390 con 0x7fe78c0a1c60 2024-04-22T08:15:56.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.426+0000 7fe791c5f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:56.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.426+0000 7fe791c5f640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe78c123f70 con 0x7fe78c0a2040 2024-04-22T08:15:56.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.426+0000 7fe791c5f640 10 monclient(hunting): _renew_subs 2024-04-22T08:15:56.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.426+0000 7fe791c5f640 10 monclient(hunting): authenticate will time out at 1030.420060s 2024-04-22T08:15:56.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.426+0000 7fe78affd640 1 -- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe78c0a2040 msgr2=0x7fe78c120830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:15:56.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.426+0000 7fe78affd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe78c0a2040 0x7fe78c120830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:15:56.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.426+0000 7fe78bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe78c0a24a0 0x7fe78c120d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:56.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.426+0000 7fe78bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe78c0a24a0 0x7fe78c120d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37422/0 (socket says 172.21.15.40:37422) 2024-04-22T08:15:56.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.426+0000 7fe78bfff640 1 -- 172.21.15.40:0/1805786646 learned_addr learned my addr 172.21.15.40:0/1805786646 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:15:56.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.426+0000 7fe78bfff640 10 monclient(hunting): get_auth_request con 0x7fe78c0a24a0 auth_method 0 2024-04-22T08:15:56.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.426+0000 7fe78bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:56.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.426+0000 7fe78bfff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:56.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.426+0000 7fe78bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:56.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.426+0000 7fe78bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:56.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.426+0000 7fe78bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:56.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.426+0000 7fe78bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:56.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.426+0000 7fe78b7fe640 1 --2- 172.21.15.40:0/1805786646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe78c0a1c60 0x7fe78c1202f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:56.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.426+0000 7fe78b7fe640 10 monclient(hunting): get_auth_request con 0x7fe78c0a1c60 auth_method 0 2024-04-22T08:15:56.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.426+0000 7fe78b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:56.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.426+0000 7fe78b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:56.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.426+0000 7fe78b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:56.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.426+0000 7fe78bfff640 10 monclient(hunting): handle_auth_done global_id 34449 payload 824 2024-04-22T08:15:56.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.426+0000 7fe78bfff640 1 -- 172.21.15.40:0/1805786646 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe78c0a2040 msgr2=0x7fe78c120830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:15:56.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.426+0000 7fe78bfff640 1 --2- 172.21.15.40:0/1805786646 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe78c0a2040 0x7fe78c120830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:56.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.426+0000 7fe78bfff640 1 -- 172.21.15.40:0/1805786646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe78c0a1c60 msgr2=0x7fe78c1202f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:56.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.426+0000 7fe78bfff640 1 --2- 172.21.15.40:0/1805786646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe78c0a1c60 0x7fe78c1202f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:56.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.426+0000 7fe78bfff640 10 monclient: _finish_hunting 0 2024-04-22T08:15:56.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.426+0000 7fe78bfff640 1 monclient: found mon.noname-a 2024-04-22T08:15:56.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.426+0000 7fe78bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:15:56.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.426+0000 7fe78bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:15:56.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.426+0000 7fe78bfff640 1 -- 172.21.15.40:0/1805786646 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7800094d0 con 0x7fe78c0a24a0 2024-04-22T08:15:56.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.426+0000 7fe78b7fe640 1 --2- 172.21.15.40:0/1805786646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe78c0a1c60 0x7fe78c1202f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T08:15:56.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.428+0000 7fe78bfff640 1 -- 172.21.15.40:0/1805786646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe78c0a24a0 msgr2=0x7fe78c120d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:56.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.428+0000 7fe78bfff640 1 --2- 172.21.15.40:0/1805786646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe78c0a24a0 0x7fe78c120d70 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe78400cd70 tx=0x7fe78400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:56.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.429+0000 7fe78bfff640 1 --2- 172.21.15.40:0/1805786646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe78c0a24a0 0x7fe78c120d70 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe78400cd70 tx=0x7fe78400a660 comp rx=0 tx=0).queue_received will delay after 1.71377e+09 on 0x7fe784003b90 mon_map magic: 0 v1 2024-04-22T08:15:56.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.429+0000 7fe773fff640 1 -- 172.21.15.40:0/1805786646 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe784003b90 con 0x7fe78c0a24a0 2024-04-22T08:15:56.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.429+0000 7fe773fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:56.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.429+0000 7fe773fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:15:56.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.429+0000 7fe773fff640 10 monclient: dump: 2024-04-22T08:15:56.440 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:56.440 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:56.440 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:56.440 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:56.440 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:56.440 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:56.440 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:56.440 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:56.440 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:56.440 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:56.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.429+0000 7fe773fff640 10 monclient: _finish_auth 0 2024-04-22T08:15:56.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.429+0000 7fe773fff640 10 monclient: _check_auth_tickets 2024-04-22T08:15:56.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.429+0000 7fe773fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:15:56.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.429+0000 7fe773fff640 1 -- 172.21.15.40:0/1805786646 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe784003cf0 con 0x7fe78c0a24a0 2024-04-22T08:15:56.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.429+0000 7fe791c5f640 5 monclient: authenticate success, global_id 34449 2024-04-22T08:15:56.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.429+0000 7fe791c5f640 10 monclient: _renew_subs 2024-04-22T08:15:56.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.429+0000 7fe791c5f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:56.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.429+0000 7fe791c5f640 1 -- 172.21.15.40:0/1805786646 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe78c0a4790 con 0x7fe78c0a24a0 2024-04-22T08:15:56.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.429+0000 7fe773fff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:15:56.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.429+0000 7fe791c5f640 10 monclient: _renew_subs 2024-04-22T08:15:56.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.429+0000 7fe791c5f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:15:56.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.429+0000 7fe791c5f640 1 -- 172.21.15.40:0/1805786646 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe78c123c90 con 0x7fe78c0a24a0 2024-04-22T08:15:56.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.429+0000 7fe789ffb640 4 set_mon_vals no callback set 2024-04-22T08:15:56.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.430+0000 7fe791c5f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:56.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.430+0000 7fe791c5f640 1 --2- 172.21.15.40:0/1805786646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe78c1212b0 0x7fe78c06e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:56.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.430+0000 7fe791c5f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:56.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.430+0000 7fe791c5f640 1 -- 172.21.15.40:0/1805786646 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe78c066650 con 0x7fe78c1212b0 2024-04-22T08:15:56.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.430+0000 7fe773fff640 1 -- 172.21.15.40:0/1805786646 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe784004240 con 0x7fe78c0a24a0 2024-04-22T08:15:56.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.430+0000 7fe773fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:56.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.430+0000 7fe773fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:15:56.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.430+0000 7fe773fff640 10 monclient: dump: 2024-04-22T08:15:56.442 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:56.442 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:56.442 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:56.442 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:56.443 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:56.443 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:56.443 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:56.443 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:56.443 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:56.443 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:56.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.430+0000 7fe773fff640 1 -- 172.21.15.40:0/1805786646 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fe784004420 con 0x7fe78c0a24a0 2024-04-22T08:15:56.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.430+0000 7fe773fff640 1 --2- 172.21.15.40:0/1805786646 >> v2:172.21.15.189:6800/30168 conn(0x7fe7780377d0 0x7fe778039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:56.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.430+0000 7fe78b7fe640 1 --2- 172.21.15.40:0/1805786646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe78c1212b0 0x7fe78c06e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:56.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.430+0000 7fe78b7fe640 10 monclient: get_auth_request con 0x7fe78c1212b0 auth_method 0 2024-04-22T08:15:56.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.430+0000 7fe78b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:56.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.430+0000 7fe78b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:56.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.430+0000 7fe78b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:56.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.430+0000 7fe78b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:56.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.430+0000 7fe78b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:56.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.430+0000 7fe78b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:56.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.430+0000 7fe78affd640 1 --2- 172.21.15.40:0/1805786646 >> v2:172.21.15.189:6800/30168 conn(0x7fe7780377d0 0x7fe778039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:56.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.430+0000 7fe78affd640 10 monclient: get_auth_request con 0x7fe7780377d0 auth_method 0 2024-04-22T08:15:56.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.430+0000 7fe78b7fe640 10 monclient(hunting): handle_auth_done global_id 24658 payload 824 2024-04-22T08:15:56.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.430+0000 7fe78b7fe640 1 -- 172.21.15.40:0/1805786646 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe78c123760 con 0x7fe78c1212b0 2024-04-22T08:15:56.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.433+0000 7fe78b7fe640 1 -- 172.21.15.40:0/1805786646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe78c1212b0 msgr2=0x7fe78c06e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:56.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.433+0000 7fe78b7fe640 1 --2- 172.21.15.40:0/1805786646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe78c1212b0 0x7fe78c06e580 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe77c00d700 tx=0x7fe77c00dbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:56.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.433+0000 7fe78affd640 1 --2- 172.21.15.40:0/1805786646 >> v2:172.21.15.189:6800/30168 conn(0x7fe7780377d0 0x7fe778039c90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe780036000 tx=0x7fe780008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:56.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.433+0000 7fe773fff640 1 -- 172.21.15.40:0/1805786646 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe77c003bc0 con 0x7fe78c1212b0 2024-04-22T08:15:56.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.433+0000 7fe773fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:56.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.433+0000 7fe773fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:15:56.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.433+0000 7fe773fff640 10 monclient: dump: 2024-04-22T08:15:56.447 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:56.447 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:56.447 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:56.447 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:56.447 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:56.447 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:56.447 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:56.447 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:56.447 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:56.447 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:56.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.433+0000 7fe773fff640 1 -- 172.21.15.40:0/1805786646 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7fe77c003d20 con 0x7fe78c1212b0 2024-04-22T08:15:56.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.433+0000 7fe773fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:15:56.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.433+0000 7fe773fff640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:15:56.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.433+0000 7fe773fff640 1 -- 172.21.15.40:0/1805786646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe78c1212b0 msgr2=0x7fe78c06e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:56.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.433+0000 7fe773fff640 1 --2- 172.21.15.40:0/1805786646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe78c1212b0 0x7fe78c06e580 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe77c00d700 tx=0x7fe77c00dbc0 comp rx=0 tx=0).stop 2024-04-22T08:15:56.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.433+0000 7fe773fff640 1 -- 172.21.15.40:0/1805786646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe78c1212b0 msgr2=0x7fe78c06e580 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:15:56.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.433+0000 7fe773fff640 1 --2- 172.21.15.40:0/1805786646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe78c1212b0 0x7fe78c06e580 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:56.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.439+0000 7fe791c5f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:15:56.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.439+0000 7fe791c5f640 1 --2- 172.21.15.40:0/1805786646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe78c06eac0 0x7fe78c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:15:56.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.439+0000 7fe791c5f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:15:56.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.439+0000 7fe791c5f640 1 -- 172.21.15.40:0/1805786646 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe78c136630 con 0x7fe78c06eac0 2024-04-22T08:15:56.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.440+0000 7fe78b7fe640 1 --2- 172.21.15.40:0/1805786646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe78c06eac0 0x7fe78c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:15:56.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.440+0000 7fe78b7fe640 10 monclient: get_auth_request con 0x7fe78c06eac0 auth_method 0 2024-04-22T08:15:56.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.440+0000 7fe78b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:15:56.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.440+0000 7fe78b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:15:56.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.440+0000 7fe78b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:15:56.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.440+0000 7fe78b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:15:56.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.440+0000 7fe78b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:15:56.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.440+0000 7fe78b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:15:56.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.440+0000 7fe78b7fe640 10 monclient(hunting): handle_auth_done global_id 24661 payload 824 2024-04-22T08:15:56.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.440+0000 7fe78b7fe640 1 -- 172.21.15.40:0/1805786646 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe78c04e9b0 con 0x7fe78c06eac0 2024-04-22T08:15:56.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.442+0000 7fe78b7fe640 1 -- 172.21.15.40:0/1805786646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe78c06eac0 msgr2=0x7fe78c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:15:56.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.442+0000 7fe78b7fe640 1 --2- 172.21.15.40:0/1805786646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe78c06eac0 0x7fe78c06eed0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fe77c016000 tx=0x7fe77c0059d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:15:56.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.442+0000 7fe773fff640 1 -- 172.21.15.40:0/1805786646 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe77c022070 con 0x7fe78c06eac0 2024-04-22T08:15:56.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.442+0000 7fe773fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:15:56.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.442+0000 7fe773fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:15:56.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.442+0000 7fe773fff640 10 monclient: dump: 2024-04-22T08:15:56.449 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:15:56.449 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:15:56.449 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:56.450 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:15:56.450 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:15:56.450 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:15:56.450 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:15:56.450 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:15:56.450 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:15:56.450 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:15:56.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.443+0000 7fe773fff640 1 -- 172.21.15.40:0/1805786646 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4790 (secure 0 0 0) 0x7fe77c0054f0 con 0x7fe78c06eac0 2024-04-22T08:15:56.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.443+0000 7fe773fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:15:56.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.443+0000 7fe773fff640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:15:56.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.443+0000 7fe773fff640 1 -- 172.21.15.40:0/1805786646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe78c06eac0 msgr2=0x7fe78c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:56.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.443+0000 7fe773fff640 1 --2- 172.21.15.40:0/1805786646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe78c06eac0 0x7fe78c06eed0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fe77c016000 tx=0x7fe77c0059d0 comp rx=0 tx=0).stop 2024-04-22T08:15:56.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.443+0000 7fe773fff640 1 -- 172.21.15.40:0/1805786646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe78c06eac0 msgr2=0x7fe78c06eed0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:15:56.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.443+0000 7fe773fff640 1 --2- 172.21.15.40:0/1805786646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe78c06eac0 0x7fe78c06eed0 secure :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fe77c016000 tx=0x7fe77c0059d0 comp rx=0 tx=0).stop 2024-04-22T08:15:56.451 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:15:56.451 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:15:56.451 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:15:56.451 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:15:56.451 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 40, 2024-04-22T08:15:56.451 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:15:56.451 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:15:56.451 INFO:teuthology.orchestra.run.smithi040.stdout: 1 2024-04-22T08:15:56.451 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:56.451 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 0, 2024-04-22T08:15:56.451 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:15:56.452 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:15:56.452 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:15:56.452 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:15:56.452 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:15:56.452 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:15:56.452 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:15:56.452 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:15:56.452 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:15:56.452 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:15:56.452 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:15:56.452 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:15:56.452 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:15:56.452 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:56.453 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:15:56.453 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:15:56.453 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:15:56.453 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:15:56.453 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:15:56.453 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:15:56.453 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:15:56.453 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:15:56.453 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:15:56.453 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:15:56.453 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:15:56.453 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:15:56.453 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:56.453 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:56.454 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:15:56.454 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:15:56.454 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:15:56.454 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:15:56.454 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:15:56.454 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:15:56.454 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:15:56.454 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:15:56.454 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:15:56.454 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:15:56.454 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:15:56.454 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:15:56.454 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:15:56.454 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:15:56.455 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:15:56.455 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:15:56.455 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:15:56.455 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:15:56.455 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:15:56.455 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:15:56.455 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:15:56.455 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:15:56.455 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:15:56.455 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:15:56.455 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:15:56.455 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:15:56.455 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:15:56.455 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:56.455 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:15:56.456 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:56.456 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:15:56.456 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:56.456 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:15:56.456 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:15:56.456 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:15:56.456 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:15:56.456 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:56.456 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:15:56.456 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:15:56.456 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:56.456 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:56.456 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:56.456 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:15:56.456 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:15:56.457 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:56.457 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:56.457 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:56.457 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:56.457 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:15:56.457 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:15:56.457 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:15:56.457 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:15:56.457 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:15:56.457 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:56.457 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:56.457 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:15:56.457 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:15:56.457 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:15:56.458 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:15:56.458 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:56.458 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:15:56.458 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:15:56.458 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:56.458 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:56.458 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:56.458 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:15:56.458 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:15:56.458 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:56.458 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:56.458 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:56.458 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:56.458 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:15:56.459 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:15:56.459 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:15:56.459 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:15:56.459 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:15:56.459 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:56.459 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:56.459 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:15:56.459 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:15:56.459 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:15:56.459 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:15:56.459 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:56.459 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:15:56.459 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:15:56.459 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:56.460 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:56.460 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:56.460 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:15:56.460 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:15:56.460 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:15:56.460 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:56.460 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:56.460 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:56.460 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:15:56.460 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:15:56.460 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:15:56.460 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:15:56.460 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:15:56.460 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:56.461 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:56.461 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:56.461 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:15:56.461 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:15:56.461 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:56.461 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:15:56.461 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:15:56.461 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:15:56.461 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:56.461 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:56.461 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:15:56.461 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:56.461 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:15:56.461 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:15:56.462 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 2 2024-04-22T08:15:56.462 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:56.462 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:56.462 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:15:56.462 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:56.462 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:15:56.462 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:15:56.462 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 7 2024-04-22T08:15:56.462 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:56.462 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:15:56.462 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": [ 2024-04-22T08:15:56.462 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:15:56.462 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:15:56.463 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:15:56.463 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:15:56.463 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:15:56.463 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:15:56.463 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:15:56.463 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:15:56.463 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:15:56.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.443+0000 7fe791c5f640 1 -- 172.21.15.40:0/1805786646 >> v2:172.21.15.189:6800/30168 conn(0x7fe7780377d0 msgr2=0x7fe778039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:56.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.443+0000 7fe791c5f640 1 --2- 172.21.15.40:0/1805786646 >> v2:172.21.15.189:6800/30168 conn(0x7fe7780377d0 0x7fe778039c90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe780036000 tx=0x7fe780008030 comp rx=0 tx=0).stop 2024-04-22T08:15:56.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.443+0000 7fe791c5f640 10 monclient: shutdown 2024-04-22T08:15:56.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.443+0000 7fe791c5f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:15:56.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.443+0000 7fe791c5f640 1 -- 172.21.15.40:0/1805786646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe78c0a24a0 msgr2=0x7fe78c120d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:15:56.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.443+0000 7fe791c5f640 1 --2- 172.21.15.40:0/1805786646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe78c0a24a0 0x7fe78c120d70 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe78400cd70 tx=0x7fe78400a660 comp rx=0 tx=0).stop 2024-04-22T08:15:56.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.443+0000 7fe78bfff640 1 -- 172.21.15.40:0/1805786646 reap_dead start 2024-04-22T08:15:56.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.443+0000 7fe773fff640 1 -- 172.21.15.40:0/1805786646 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 3898+0+0 (unknown 0 0 0) 0x7fe784012070 con 0x7fe78c0a24a0 2024-04-22T08:15:56.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.443+0000 7fe791c5f640 1 -- 172.21.15.40:0/1805786646 shutdown_connections 2024-04-22T08:15:56.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.443+0000 7fe791c5f640 1 --2- 172.21.15.40:0/1805786646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe78c0a24a0 0x7fe78c120d70 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:15:56.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.443+0000 7fe791c5f640 1 -- 172.21.15.40:0/1805786646 >> 172.21.15.40:0/1805786646 conn(0x7fe78c0a89a0 msgr2=0x7fe78c0a6cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:15:56.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.443+0000 7fe791c5f640 1 -- 172.21.15.40:0/1805786646 shutdown_connections 2024-04-22T08:15:56.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:15:56.443+0000 7fe791c5f640 1 -- 172.21.15.40:0/1805786646 wait complete. 2024-04-22T08:15:56.466 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-22T08:15:56.466 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-04-22T08:15:56.991 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:15:56.991 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36036]: starting. 2024-04-22T08:15:56.991 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36036]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-36001 2024-04-22T08:15:56.991 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36036]: created object 0... 2024-04-22T08:15:56.991 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36036]: created object 25... 2024-04-22T08:15:56.991 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36036]: created object 49... 2024-04-22T08:15:56.991 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36036]: finishing. 2024-04-22T08:15:56.991 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36036]: shutting down. 2024-04-22T08:15:57.030 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:15:57.031 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36038]: starting. 2024-04-22T08:15:57.031 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36038]: listing objects. 2024-04-22T08:15:57.031 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36038]: listed object 0... 2024-04-22T08:15:57.031 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36038]: listed object 25... 2024-04-22T08:15:57.031 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36038]: saw 50 objects 2024-04-22T08:15:57.031 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36038]: shutting down. 2024-04-22T08:15:57.883 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:15:57.883 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[36037]: starting. 2024-04-22T08:15:57.883 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[36037]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-36001 2024-04-22T08:15:57.883 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[36037]: shutting down. 2024-04-22T08:15:57.884 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:15:57.885 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:15:57.885 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:15:57.887 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:15:59.998 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36105]: starting. 2024-04-22T08:15:59.999 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36105]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-36104 2024-04-22T08:15:59.999 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36105]: created object 0... 2024-04-22T08:15:59.999 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36105]: created object 25... 2024-04-22T08:15:59.999 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36105]: created object 49... 2024-04-22T08:15:59.999 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36105]: finishing. 2024-04-22T08:15:59.999 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36105]: shutting down. 2024-04-22T08:16:00.904 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[36106]: starting. 2024-04-22T08:16:00.904 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[36106]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-36104 2024-04-22T08:16:00.904 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[36106]: shutting down. 2024-04-22T08:16:02.973 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:16:02.974 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36139]: starting. 2024-04-22T08:16:02.974 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36139]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-36104 2024-04-22T08:16:02.974 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36139]: created object 0... 2024-04-22T08:16:02.974 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36139]: created object 25... 2024-04-22T08:16:02.974 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36139]: created object 49... 2024-04-22T08:16:02.974 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36139]: finishing. 2024-04-22T08:16:02.974 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36139]: shutting down. 2024-04-22T08:16:03.014 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:16:03.014 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36141]: starting. 2024-04-22T08:16:03.014 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36141]: listing objects. 2024-04-22T08:16:03.014 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36141]: listed object 0... 2024-04-22T08:16:03.014 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36141]: listed object 25... 2024-04-22T08:16:03.014 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36141]: saw 50 objects 2024-04-22T08:16:03.014 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36141]: shutting down. 2024-04-22T08:16:03.860 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:16:03.860 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[36140]: starting. 2024-04-22T08:16:03.860 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[36140]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-36104 2024-04-22T08:16:03.860 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[36140]: shutting down. 2024-04-22T08:16:03.862 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:16:03.862 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:16:03.862 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:16:03.865 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:16:06.202 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36208]: starting. 2024-04-22T08:16:06.202 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36208]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-36207 2024-04-22T08:16:06.203 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36208]: created object 0... 2024-04-22T08:16:06.203 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36208]: created object 25... 2024-04-22T08:16:06.203 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36208]: created object 49... 2024-04-22T08:16:06.203 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36208]: finishing. 2024-04-22T08:16:06.203 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36208]: shutting down. 2024-04-22T08:16:07.095 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[36209]: starting. 2024-04-22T08:16:07.095 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[36209]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-36207 2024-04-22T08:16:07.095 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[36209]: shutting down. 2024-04-22T08:16:09.223 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:16:09.223 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36242]: starting. 2024-04-22T08:16:09.223 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36242]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-36207 2024-04-22T08:16:09.223 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36242]: created object 0... 2024-04-22T08:16:09.223 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36242]: created object 25... 2024-04-22T08:16:09.223 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36242]: created object 49... 2024-04-22T08:16:09.223 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36242]: finishing. 2024-04-22T08:16:09.223 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36242]: shutting down. 2024-04-22T08:16:09.262 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:16:09.263 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36244]: starting. 2024-04-22T08:16:09.263 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36244]: listing objects. 2024-04-22T08:16:09.263 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36244]: listed object 0... 2024-04-22T08:16:09.263 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36244]: listed object 25... 2024-04-22T08:16:09.263 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36244]: saw 50 objects 2024-04-22T08:16:09.263 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36244]: shutting down. 2024-04-22T08:16:10.114 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:16:10.115 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[36243]: starting. 2024-04-22T08:16:10.115 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[36243]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-36207 2024-04-22T08:16:10.115 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[36243]: shutting down. 2024-04-22T08:16:10.116 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:16:10.116 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:16:10.116 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:16:10.118 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:16:13.328 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36311]: starting. 2024-04-22T08:16:13.328 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36311]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-36310 2024-04-22T08:16:13.329 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36311]: created object 0... 2024-04-22T08:16:13.329 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36311]: created object 25... 2024-04-22T08:16:13.329 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36311]: created object 49... 2024-04-22T08:16:13.329 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36311]: finishing. 2024-04-22T08:16:13.329 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36311]: shutting down. 2024-04-22T08:16:13.863 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[36312]: starting. 2024-04-22T08:16:13.863 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[36312]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-36310 2024-04-22T08:16:13.864 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[36312]: shutting down. 2024-04-22T08:16:15.992 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:16:15.993 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36345]: starting. 2024-04-22T08:16:15.993 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36345]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-36310 2024-04-22T08:16:15.993 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36345]: created object 0... 2024-04-22T08:16:15.993 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36345]: created object 25... 2024-04-22T08:16:15.993 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36345]: created object 49... 2024-04-22T08:16:15.993 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36345]: finishing. 2024-04-22T08:16:15.993 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36345]: shutting down. 2024-04-22T08:16:16.034 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:16:16.034 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36347]: starting. 2024-04-22T08:16:16.034 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36347]: listing objects. 2024-04-22T08:16:16.034 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36347]: listed object 0... 2024-04-22T08:16:16.034 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36347]: listed object 25... 2024-04-22T08:16:16.034 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36347]: saw 50 objects 2024-04-22T08:16:16.034 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36347]: shutting down. 2024-04-22T08:16:16.467 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-04-22T08:16:16.467 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-04-22T08:16:16.467 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-22T08:16:16.467 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-22T08:16:16.470 INFO:tasks.ceph.mon.c:Started 2024-04-22T08:16:16.470 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-22T08:16:16.470 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:16:16.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.581+0000 7f784da21640 10 monclient: get_monmap_and_config 2024-04-22T08:16:16.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.581+0000 7f784da21640 10 monclient: build_initial_monmap 2024-04-22T08:16:16.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.581+0000 7f784da21640 10 monclient: monmap: 2024-04-22T08:16:16.581 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:16:16.581 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:16.582 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:16:16.582802+0000 2024-04-22T08:16:16.582 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:16:16.582802+0000 2024-04-22T08:16:16.582 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:16:16.582 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:16.582 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:16:16.582 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:16:16.582 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:16:16.582 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:16:16.582 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:16:16.582 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:16.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.581+0000 7f784da21640 1 Processor -- start 2024-04-22T08:16:16.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.581+0000 7f784da21640 1 -- start start 2024-04-22T08:16:16.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.581+0000 7f784da21640 10 monclient: init 2024-04-22T08:16:16.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.582+0000 7f784da21640 10 monclient: _reopen_session rank -1 2024-04-22T08:16:16.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.582+0000 7f784da21640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-04-22T08:16:16.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.582+0000 7f784da21640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f784809ee10 0x7f78480a32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:16.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.582+0000 7f784da21640 10 monclient(hunting): picked mon.noname-b con 0x7f784809ee10 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:16:16.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.582+0000 7f784da21640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f78480a1d30 addr v1:172.21.15.40:6789/0 2024-04-22T08:16:16.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.582+0000 7f784da21640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f78480a2150 addr v1:172.21.15.189:6789/0 2024-04-22T08:16:16.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.582+0000 7f784da21640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78480a2580 con 0x7f78480a1d30 2024-04-22T08:16:16.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.582+0000 7f784da21640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f784809f1f0 con 0x7f78480a2150 2024-04-22T08:16:16.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.582+0000 7f784da21640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:16.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.582+0000 7f784da21640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7848136bb0 con 0x7f784809ee10 2024-04-22T08:16:16.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.582+0000 7f784da21640 10 monclient(hunting): _renew_subs 2024-04-22T08:16:16.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.582+0000 7f784da21640 10 monclient(hunting): authenticate will time out at 1050.577063s 2024-04-22T08:16:16.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.582+0000 7f7846ffd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f784809ee10 0x7f78480a32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:16.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.582+0000 7f78467fc640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f78480a1d30 0x7f7848135d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:51202/0 (socket says 172.21.15.40:51202) 2024-04-22T08:16:16.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.582+0000 7f78467fc640 1 -- 172.21.15.40:0/1479313193 learned_addr learned my addr 172.21.15.40:0/1479313193 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:16:16.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.582+0000 7f7846ffd640 10 monclient(hunting): get_auth_request con 0x7f784809ee10 auth_method 0 2024-04-22T08:16:16.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.582+0000 7f7846ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:16.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.582+0000 7f7846ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:16.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.582+0000 7f7846ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:16.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.582+0000 7f7846ffd640 1 -- 172.21.15.40:0/1479313193 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f784809ee10 msgr2=0x7f78480a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:16:16.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.582+0000 7f7846ffd640 1 -- 172.21.15.40:0/1479313193 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f784809ee10 msgr2=0x7f78480a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:16:16.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.582+0000 7f7846ffd640 1 --2- 172.21.15.40:0/1479313193 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f784809ee10 0x7f78480a32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:16:16.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.582+0000 7f7846ffd640 1 --2- 172.21.15.40:0/1479313193 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f784809ee10 0x7f78480a32b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:16:16.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.584+0000 7f78467fc640 1 -- 172.21.15.40:0/1479313193 >> v1:172.21.15.40:6789/0 conn(0x7f78480a1d30 legacy=0x7f7848135d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:16:16.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.584+0000 7f7845ffb640 1 -- 172.21.15.40:0/1479313193 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3104432782 0 0) 0x7f78480a2580 con 0x7f78480a1d30 2024-04-22T08:16:16.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.584+0000 7f7845ffb640 10 monclient(hunting): _negotiate 2024-04-22T08:16:16.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.584+0000 7f7845ffb640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:16.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.584+0000 7f7845ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:16.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.584+0000 7f7845ffb640 10 monclient(hunting): my global_id is 34521 2024-04-22T08:16:16.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.584+0000 7f78477fe640 1 -- 172.21.15.40:0/1479313193 >> v1:172.21.15.189:6789/0 conn(0x7f78480a2150 legacy=0x7f78481364a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:16:16.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.584+0000 7f7845ffb640 1 -- 172.21.15.40:0/1479313193 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7830003520 con 0x7f78480a1d30 2024-04-22T08:16:16.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.584+0000 7f7845ffb640 1 -- 172.21.15.40:0/1479313193 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2756664441 0 0) 0x7f7830003520 con 0x7f78480a1d30 2024-04-22T08:16:16.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f7845ffb640 1 -- 172.21.15.40:0/1479313193 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f784809ee10 msgr2=0x7f78480a32b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:16:16.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f7845ffb640 1 --2- 172.21.15.40:0/1479313193 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f784809ee10 0x7f78480a32b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:16.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f7845ffb640 1 -- 172.21.15.40:0/1479313193 >> v1:172.21.15.189:6789/0 conn(0x7f78480a2150 legacy=0x7f78481364a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:16.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f7845ffb640 10 monclient: _finish_hunting 0 2024-04-22T08:16:16.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f7845ffb640 1 monclient: found mon.noname-a-legacy 2024-04-22T08:16:16.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f7845ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:16:16.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f7845ffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.40:6789/0 2024-04-22T08:16:16.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f7845ffb640 1 -- 172.21.15.40:0/1479313193 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78480a39c0 con 0x7f78480a1d30 2024-04-22T08:16:16.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f7845ffb640 10 monclient: _finish_auth 0 2024-04-22T08:16:16.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f7845ffb640 10 monclient: _check_auth_tickets 2024-04-22T08:16:16.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f7845ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:16:16.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f7845ffb640 1 -- 172.21.15.40:0/1479313193 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1087006497 0 0) 0x7f7838001920 con 0x7f78480a1d30 2024-04-22T08:16:16.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f784da21640 5 monclient: authenticate success, global_id 34521 2024-04-22T08:16:16.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f784da21640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:16:16.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f7845ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:16.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f7845ffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-22T08:16:16.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f7845ffb640 10 monclient: dump: 2024-04-22T08:16:16.595 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:16.595 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:16.595 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:16.595 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:16.595 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:16.595 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:16.595 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:16.595 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:16.595 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:16.595 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:16.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f7845ffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] but i'm connected to v1:172.21.15.40:6789/0, reconnecting 2024-04-22T08:16:16.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f7845ffb640 10 monclient: _reopen_session rank -1 2024-04-22T08:16:16.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f7845ffb640 1 -- 172.21.15.40:0/1479313193 >> v1:172.21.15.40:6789/0 conn(0x7f78480a1d30 legacy=0x7f7848135d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:16.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f7845ffb640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-22T08:16:16.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f7845ffb640 1 --2- 172.21.15.40:0/1479313193 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7830003a20 0x7f7830003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:16.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f7845ffb640 10 monclient(hunting): picked mon.b con 0x7f7830003a20 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:16:16.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f7845ffb640 1 --2- 172.21.15.40:0/1479313193 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f784809ee10 0x7f78300047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:16.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f7845ffb640 10 monclient(hunting): picked mon.c con 0x7f784809ee10 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:16:16.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f7845ffb640 1 --2- 172.21.15.40:0/1479313193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7830004f00 0x7f783000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:16.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f7845ffb640 10 monclient(hunting): picked mon.a con 0x7f7830004f00 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:16:16.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f7845ffb640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:16.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f7845ffb640 1 -- 172.21.15.40:0/1479313193 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7848136bb0 con 0x7f7830004f00 2024-04-22T08:16:16.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f7845ffb640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:16.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f7845ffb640 1 -- 172.21.15.40:0/1479313193 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7830005600 con 0x7f7830003a20 2024-04-22T08:16:16.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f7845ffb640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:16.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f7845ffb640 1 -- 172.21.15.40:0/1479313193 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f783000bec0 con 0x7f784809ee10 2024-04-22T08:16:16.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f7845ffb640 10 monclient(hunting): _renew_subs 2024-04-22T08:16:16.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f7845ffb640 10 monclient(hunting): _finish_auth 0 2024-04-22T08:16:16.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f784da21640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-04-22T08:16:16.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f78467fc640 1 --2- 172.21.15.40:0/1479313193 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f784809ee10 0x7f78300047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:16.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f78477fe640 1 --2- 172.21.15.40:0/1479313193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7830004f00 0x7f783000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:16.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f78467fc640 10 monclient(hunting): get_auth_request con 0x7f784809ee10 auth_method 0 2024-04-22T08:16:16.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f78467fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:16.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f78467fc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:16.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f78467fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:16:16.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f7846ffd640 1 --2- 172.21.15.40:0/1479313193 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7830003a20 0x7f7830003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:16.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f78477fe640 10 monclient(hunting): get_auth_request con 0x7f7830004f00 auth_method 0 2024-04-22T08:16:16.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f78477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:16.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f78477fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:16.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f78477fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:16:16.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f7846ffd640 10 monclient(hunting): get_auth_request con 0x7f7830003a20 auth_method 0 2024-04-22T08:16:16.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f7846ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:16.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f7846ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:16.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f7846ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:16:16.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f78467fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:16.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f78467fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:16.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f78467fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-22T08:16:16.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f78477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:16.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f78477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:16.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f78477fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-22T08:16:16.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f7846ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:16.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f7846ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:16.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f7846ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-22T08:16:16.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f78467fc640 10 monclient(hunting): handle_auth_done global_id 34521 payload 293 2024-04-22T08:16:16.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f78467fc640 1 -- 172.21.15.40:0/1479313193 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7830003a20 msgr2=0x7f7830003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:16:16.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f78467fc640 1 --2- 172.21.15.40:0/1479313193 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7830003a20 0x7f7830003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:16.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f78467fc640 1 -- 172.21.15.40:0/1479313193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7830004f00 msgr2=0x7f783000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:16:16.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f78467fc640 1 --2- 172.21.15.40:0/1479313193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7830004f00 0x7f783000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:16.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f78467fc640 10 monclient: _finish_hunting 0 2024-04-22T08:16:16.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f78467fc640 1 monclient: found mon.c 2024-04-22T08:16:16.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f78467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:16:16.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f78467fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:16:16.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f78467fc640 1 -- 172.21.15.40:0/1479313193 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7830003520 con 0x7f784809ee10 2024-04-22T08:16:16.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f78467fc640 10 monclient: _finish_auth 0 2024-04-22T08:16:16.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f78467fc640 10 monclient: _check_auth_tickets 2024-04-22T08:16:16.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f78467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:16:16.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.585+0000 7f78477fe640 1 --2- 172.21.15.40:0/1479313193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7830004f00 0x7f783000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:16:16.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.588+0000 7f78467fc640 1 -- 172.21.15.40:0/1479313193 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f784809ee10 msgr2=0x7f78300047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:16:16.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.588+0000 7f78467fc640 1 --2- 172.21.15.40:0/1479313193 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f784809ee10 0x7f78300047c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f783800c3b0 tx=0x7f78300057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc292b3ef50b9200 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:16.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.588+0000 7f7845ffb640 1 -- 172.21.15.40:0/1479313193 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f783802cb60 con 0x7f784809ee10 2024-04-22T08:16:16.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.588+0000 7f7845ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:16.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.588+0000 7f7845ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:16:16.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.588+0000 7f7845ffb640 10 monclient: dump: 2024-04-22T08:16:16.600 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:16.600 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:16.600 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:16.600 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:16.600 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:16.600 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:16.600 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:16.600 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:16.600 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:16.600 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:16.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.588+0000 7f784da21640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:16:16.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.588+0000 7f7845ffb640 1 -- 172.21.15.40:0/1479313193 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f783802cce0 con 0x7f784809ee10 2024-04-22T08:16:16.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.588+0000 7f7845ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:16:16.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.588+0000 7f784da21640 10 monclient: get_monmap_and_config success 2024-04-22T08:16:16.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.588+0000 7f7845ffb640 1 -- 172.21.15.40:0/1479313193 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7838005b10 con 0x7f784809ee10 2024-04-22T08:16:16.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.588+0000 7f784da21640 4 set_mon_vals no callback set 2024-04-22T08:16:16.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.588+0000 7f7845ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:16.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.588+0000 7f7845ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:16:16.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.588+0000 7f7845ffb640 10 monclient: dump: 2024-04-22T08:16:16.601 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:16.601 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:16.601 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:16.601 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:16.601 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:16.601 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:16.601 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:16.601 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:16.601 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:16.602 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:16.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.589+0000 7f784da21640 10 monclient: shutdown 2024-04-22T08:16:16.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.589+0000 7f784da21640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:16:16.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.589+0000 7f784da21640 1 -- 172.21.15.40:0/1479313193 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f784809ee10 msgr2=0x7f78300047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:16.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.589+0000 7f784da21640 1 --2- 172.21.15.40:0/1479313193 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f784809ee10 0x7f78300047c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f783800c3b0 tx=0x7f78300057f0 comp rx=0 tx=0).stop 2024-04-22T08:16:16.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.589+0000 7f78477fe640 1 -- 172.21.15.40:0/1479313193 reap_dead start 2024-04-22T08:16:16.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.589+0000 7f784da21640 1 -- 172.21.15.40:0/1479313193 shutdown_connections 2024-04-22T08:16:16.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.589+0000 7f784da21640 1 -- 172.21.15.40:0/1479313193 >> 172.21.15.40:0/1479313193 conn(0x7f78480a4ef0 msgr2=0x7f78480a5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:16:16.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.589+0000 7f784da21640 1 -- 172.21.15.40:0/1479313193 shutdown_connections 2024-04-22T08:16:16.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.589+0000 7f784da21640 1 -- 172.21.15.40:0/1479313193 wait complete. 2024-04-22T08:16:16.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.589+0000 7f784da21640 10 monclient: build_initial_monmap 2024-04-22T08:16:16.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.589+0000 7f784da21640 10 monclient: monmap: 2024-04-22T08:16:16.602 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:16:16.602 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:16:16.603 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:16:16.603 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:16:16.603 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:16:16.603 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:16.603 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:16:16.603 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:16:16.603 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:16:16.603 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:16.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.590+0000 7f784da21640 1 Processor -- start 2024-04-22T08:16:16.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.590+0000 7f784da21640 1 -- start start 2024-04-22T08:16:16.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.590+0000 7f784da21640 10 monclient: init 2024-04-22T08:16:16.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.590+0000 7f784da21640 10 monclient: _reopen_session rank -1 2024-04-22T08:16:16.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.590+0000 7f784da21640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-22T08:16:16.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.590+0000 7f784da21640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7848133b20 0x7f78481ceec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:16.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.590+0000 7f784da21640 10 monclient(hunting): picked mon.noname-b con 0x7f7848133b20 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:16:16.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.590+0000 7f784da21640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7848133f00 0x7f78481342e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:16.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.590+0000 7f784da21640 10 monclient(hunting): picked mon.noname-a con 0x7f7848133f00 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:16:16.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.590+0000 7f784da21640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f784808afb0 0x7f784808b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:16.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.590+0000 7f784da21640 10 monclient(hunting): picked mon.noname-c con 0x7f784808afb0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:16:16.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.590+0000 7f784da21640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:16.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.590+0000 7f784da21640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78481cfbc0 con 0x7f7848133f00 2024-04-22T08:16:16.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.590+0000 7f784da21640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:16.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.590+0000 7f784da21640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f784808ef20 con 0x7f7848133b20 2024-04-22T08:16:16.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.590+0000 7f784da21640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:16.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.590+0000 7f784da21640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f784808cbc0 con 0x7f784808afb0 2024-04-22T08:16:16.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.590+0000 7f784da21640 10 monclient(hunting): _renew_subs 2024-04-22T08:16:16.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.590+0000 7f784da21640 10 monclient(hunting): authenticate will time out at 1050.585757s 2024-04-22T08:16:16.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.591+0000 7f78467fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7848133f00 0x7f78481342e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:16.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.591+0000 7f78477fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f784808afb0 0x7f784808b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:16.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.591+0000 7f78467fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7848133f00 0x7f78481342e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58132/0 (socket says 172.21.15.40:58132) 2024-04-22T08:16:16.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.591+0000 7f78467fc640 1 -- 172.21.15.40:0/2675021308 learned_addr learned my addr 172.21.15.40:0/2675021308 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:16:16.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.591+0000 7f78477fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f784808afb0 0x7f784808b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:51844/0 (socket says 172.21.15.40:51844) 2024-04-22T08:16:16.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.591+0000 7f78467fc640 10 monclient(hunting): get_auth_request con 0x7f7848133f00 auth_method 0 2024-04-22T08:16:16.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.591+0000 7f78467fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:16.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.591+0000 7f78467fc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:16.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.591+0000 7f78467fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:16.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.591+0000 7f78477fe640 10 monclient(hunting): get_auth_request con 0x7f784808afb0 auth_method 0 2024-04-22T08:16:16.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.591+0000 7f78477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:16.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.591+0000 7f78477fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:16.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.591+0000 7f78477fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:16.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.591+0000 7f78477fe640 1 -- 172.21.15.40:0/2675021308 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f784808afb0 msgr2=0x7f784808b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-22T08:16:16.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.591+0000 7f78477fe640 1 -- 172.21.15.40:0/2675021308 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f784808afb0 msgr2=0x7f784808b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:16:16.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.591+0000 7f78477fe640 1 --2- 172.21.15.40:0/2675021308 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f784808afb0 0x7f784808b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:16:16.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.591+0000 7f78477fe640 1 --2- 172.21.15.40:0/2675021308 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f784808afb0 0x7f784808b3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:16:16.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.591+0000 7f78467fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:16.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.591+0000 7f78467fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:16.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.591+0000 7f7846ffd640 1 --2- 172.21.15.40:0/2675021308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7848133b20 0x7f78481ceec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:16.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.591+0000 7f78467fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:16.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.591+0000 7f7846ffd640 10 monclient(hunting): get_auth_request con 0x7f7848133b20 auth_method 0 2024-04-22T08:16:16.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.591+0000 7f7846ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:16.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.591+0000 7f7846ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:16.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.591+0000 7f7846ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:16.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.591+0000 7f7846ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:16.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.591+0000 7f7846ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:16.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.591+0000 7f7846ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:16.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.591+0000 7f78467fc640 10 monclient(hunting): handle_auth_done global_id 34524 payload 824 2024-04-22T08:16:16.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.591+0000 7f78467fc640 1 -- 172.21.15.40:0/2675021308 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f784808afb0 msgr2=0x7f784808b3f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:16:16.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.591+0000 7f78467fc640 1 --2- 172.21.15.40:0/2675021308 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f784808afb0 0x7f784808b3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:16.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.591+0000 7f78467fc640 1 -- 172.21.15.40:0/2675021308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7848133b20 msgr2=0x7f78481ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:16.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.591+0000 7f78467fc640 1 --2- 172.21.15.40:0/2675021308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7848133b20 0x7f78481ceec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:16.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.591+0000 7f78467fc640 10 monclient: _finish_hunting 0 2024-04-22T08:16:16.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.591+0000 7f78467fc640 1 monclient: found mon.noname-a 2024-04-22T08:16:16.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.591+0000 7f78467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:16:16.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.591+0000 7f78467fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:16:16.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.591+0000 7f78467fc640 1 -- 172.21.15.40:0/2675021308 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78480a2730 con 0x7f7848133f00 2024-04-22T08:16:16.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.593+0000 7f78467fc640 1 -- 172.21.15.40:0/2675021308 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7848133f00 msgr2=0x7f78481342e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:16:16.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.593+0000 7f78467fc640 1 --2- 172.21.15.40:0/2675021308 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7848133f00 0x7f78481342e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f783802c500 tx=0x7f783802e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:16.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.594+0000 7f782ffff640 1 -- 172.21.15.40:0/2675021308 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f783803c040 con 0x7f7848133f00 2024-04-22T08:16:16.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.594+0000 7f782ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:16.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.594+0000 7f782ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:16:16.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.594+0000 7f782ffff640 10 monclient: dump: 2024-04-22T08:16:16.610 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:16.610 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:16.610 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:16.610 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:16.610 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:16.611 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:16.611 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:16.611 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:16.611 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:16.611 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:16.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.594+0000 7f782ffff640 10 monclient: _finish_auth 0 2024-04-22T08:16:16.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.594+0000 7f782ffff640 10 monclient: _check_auth_tickets 2024-04-22T08:16:16.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.594+0000 7f782ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:16:16.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.594+0000 7f782ffff640 1 -- 172.21.15.40:0/2675021308 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f783802cb70 con 0x7f7848133f00 2024-04-22T08:16:16.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.594+0000 7f784da21640 5 monclient: authenticate success, global_id 34524 2024-04-22T08:16:16.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.594+0000 7f784da21640 10 monclient: _renew_subs 2024-04-22T08:16:16.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.594+0000 7f784da21640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:16:16.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.594+0000 7f784da21640 1 -- 172.21.15.40:0/2675021308 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78481360e0 con 0x7f7848133f00 2024-04-22T08:16:16.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.594+0000 7f782ffff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:16:16.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.594+0000 7f782ffff640 1 -- 172.21.15.40:0/2675021308 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7838007470 con 0x7f7848133f00 2024-04-22T08:16:16.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.594+0000 7f782ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:16.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.594+0000 7f782ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:16:16.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.594+0000 7f782ffff640 10 monclient: dump: 2024-04-22T08:16:16.612 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:16.612 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:16.612 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:16.612 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:16.612 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:16.612 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:16.612 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:16.612 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:16.612 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:16.612 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:16.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.594+0000 7f78457fa640 4 set_mon_vals no callback set 2024-04-22T08:16:16.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.594+0000 7f784da21640 10 monclient: _renew_subs 2024-04-22T08:16:16.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.594+0000 7f784da21640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:16:16.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.594+0000 7f784da21640 1 -- 172.21.15.40:0/2675021308 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f784808ecf0 con 0x7f7848133f00 2024-04-22T08:16:16.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.595+0000 7f784da21640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:16:16.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.595+0000 7f784da21640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:16:16.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.595+0000 7f784da21640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:16:16.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.595+0000 7f784da21640 1 -- 172.21.15.40:0/2675021308 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f784804e9b0 con 0x7f7848133f00 2024-04-22T08:16:16.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.595+0000 7f782ffff640 1 -- 172.21.15.40:0/2675021308 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f7838032070 con 0x7f7848133f00 2024-04-22T08:16:16.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.595+0000 7f782ffff640 1 --2- 172.21.15.40:0/2675021308 >> v2:172.21.15.189:6800/30168 conn(0x7f78340377d0 0x7f7834039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:16.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.595+0000 7f782ffff640 1 -- 172.21.15.40:0/2675021308 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f783806b690 con 0x7f7848133f00 2024-04-22T08:16:16.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.596+0000 7f7846ffd640 1 --2- 172.21.15.40:0/2675021308 >> v2:172.21.15.189:6800/30168 conn(0x7f78340377d0 0x7f7834039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:16.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.596+0000 7f7846ffd640 10 monclient: get_auth_request con 0x7f78340377d0 auth_method 0 2024-04-22T08:16:16.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.598+0000 7f7846ffd640 1 --2- 172.21.15.40:0/2675021308 >> v2:172.21.15.189:6800/30168 conn(0x7f78340377d0 0x7f7834039c90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f783c006fd0 tx=0x7f783c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:16.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.600+0000 7f782ffff640 1 -- 172.21.15.40:0/2675021308 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f78380343d0 con 0x7f7848133f00 2024-04-22T08:16:16.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.600+0000 7f782ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:16:16.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.600+0000 7f782ffff640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:16:16.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.851+0000 7f784da21640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:16:16.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.851+0000 7f784da21640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:16:16.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.851+0000 7f784da21640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:16:16.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.851+0000 7f784da21640 1 -- 172.21.15.40:0/2675021308 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f78480713e0 con 0x7f7848133f00 2024-04-22T08:16:16.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.852+0000 7f782ffff640 1 -- 172.21.15.40:0/2675021308 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7f7838033b60 con 0x7f7848133f00 2024-04-22T08:16:16.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.852+0000 7f782ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:16:16.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.852+0000 7f782ffff640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:16:16.853 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":40,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:16:16.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.854+0000 7f784da21640 1 -- 172.21.15.40:0/2675021308 >> v2:172.21.15.189:6800/30168 conn(0x7f78340377d0 msgr2=0x7f7834039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:16.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.854+0000 7f784da21640 1 --2- 172.21.15.40:0/2675021308 >> v2:172.21.15.189:6800/30168 conn(0x7f78340377d0 0x7f7834039c90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f783c006fd0 tx=0x7f783c008040 comp rx=0 tx=0).stop 2024-04-22T08:16:16.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.854+0000 7f784da21640 10 monclient: shutdown 2024-04-22T08:16:16.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.854+0000 7f784da21640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:16:16.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.854+0000 7f784da21640 1 -- 172.21.15.40:0/2675021308 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7848133f00 msgr2=0x7f78481342e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:16.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.854+0000 7f784da21640 1 --2- 172.21.15.40:0/2675021308 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7848133f00 0x7f78481342e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f783802c500 tx=0x7f783802e070 comp rx=0 tx=0).stop 2024-04-22T08:16:16.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.854+0000 7f784da21640 1 -- 172.21.15.40:0/2675021308 shutdown_connections 2024-04-22T08:16:16.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.854+0000 7f784da21640 1 --2- 172.21.15.40:0/2675021308 >> v2:172.21.15.189:6800/30168 conn(0x7f78340377d0 0x7f7834039c90 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:16.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.854+0000 7f784da21640 1 --2- 172.21.15.40:0/2675021308 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f784808afb0 0x7f784808b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:16.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.854+0000 7f784da21640 1 --2- 172.21.15.40:0/2675021308 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7848133f00 0x7f78481342e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:16.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.854+0000 7f784da21640 1 --2- 172.21.15.40:0/2675021308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7848133b20 0x7f78481ceec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:16.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.854+0000 7f784da21640 1 -- 172.21.15.40:0/2675021308 >> 172.21.15.40:0/2675021308 conn(0x7f78480a4ef0 msgr2=0x7f7848135420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:16:16.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.854+0000 7f784da21640 1 -- 172.21.15.40:0/2675021308 shutdown_connections 2024-04-22T08:16:16.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:16.854+0000 7f784da21640 1 -- 172.21.15.40:0/2675021308 wait complete. 2024-04-22T08:16:16.878 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:16:16.879 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[36346]: starting. 2024-04-22T08:16:16.879 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[36346]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-36310 2024-04-22T08:16:16.879 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[36346]: shutting down. 2024-04-22T08:16:16.880 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:16:16.880 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:16:16.880 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:16:16.883 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:16:19.866 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:16:19.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.976+0000 7f32d0c1c640 10 monclient: get_monmap_and_config 2024-04-22T08:16:19.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.976+0000 7f32d0c1c640 10 monclient: build_initial_monmap 2024-04-22T08:16:19.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.976+0000 7f32d0c1c640 10 monclient: monmap: 2024-04-22T08:16:19.977 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:16:19.977 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:19.977 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:16:19.978163+0000 2024-04-22T08:16:19.977 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:16:19.978163+0000 2024-04-22T08:16:19.977 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:16:19.977 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:19.977 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:16:19.977 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:16:19.977 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:16:19.977 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:16:19.977 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:16:19.978 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:19.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.976+0000 7f32d0c1c640 1 Processor -- start 2024-04-22T08:16:19.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.977+0000 7f32d0c1c640 1 -- start start 2024-04-22T08:16:19.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.977+0000 7f32d0c1c640 10 monclient: init 2024-04-22T08:16:19.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.977+0000 7f32d0c1c640 10 monclient: _reopen_session rank -1 2024-04-22T08:16:19.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.977+0000 7f32d0c1c640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-04-22T08:16:19.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.977+0000 7f32d0c1c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32cc12fc40 0x7f32cc130020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:19.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.977+0000 7f32d0c1c640 10 monclient(hunting): picked mon.noname-b con 0x7f32cc12fc40 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:16:19.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.977+0000 7f32d0c1c640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f32cc09e210 0x7f32cc132990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:19.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.977+0000 7f32d0c1c640 10 monclient(hunting): picked mon.noname-a con 0x7f32cc09e210 addr v2:172.21.15.40:3300/0 2024-04-22T08:16:19.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.977+0000 7f32d0c1c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f32cc132ed0 addr v1:172.21.15.40:6789/0 2024-04-22T08:16:19.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.977+0000 7f32d0c1c640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32cc09ea50 con 0x7f32cc132ed0 2024-04-22T08:16:19.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.977+0000 7f32d0c1c640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:19.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.977+0000 7f32d0c1c640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32cc09cf90 con 0x7f32cc09e210 2024-04-22T08:16:19.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.977+0000 7f32d0c1c640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:19.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.977+0000 7f32d0c1c640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32cc09d270 con 0x7f32cc12fc40 2024-04-22T08:16:19.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.977+0000 7f32d0c1c640 10 monclient(hunting): _renew_subs 2024-04-22T08:16:19.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.977+0000 7f32d0c1c640 10 monclient(hunting): authenticate will time out at 1053.972439s 2024-04-22T08:16:19.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.977+0000 7f32c9d9b640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f32cc09e210 0x7f32cc132990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:19.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.977+0000 7f32ca59c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32cc12fc40 0x7f32cc130020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:19.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.977+0000 7f32cad9d640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f32cc132ed0 0x7f32cc139cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57430/0 (socket says 172.21.15.40:57430) 2024-04-22T08:16:19.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.977+0000 7f32cad9d640 1 -- 172.21.15.40:0/3145655432 learned_addr learned my addr 172.21.15.40:0/3145655432 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:16:19.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.977+0000 7f32c9d9b640 10 monclient(hunting): get_auth_request con 0x7f32cc09e210 auth_method 0 2024-04-22T08:16:19.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.977+0000 7f32c9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:19.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.977+0000 7f32c9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:19.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.977+0000 7f32c9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:19.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.977+0000 7f32ca59c640 10 monclient(hunting): get_auth_request con 0x7f32cc12fc40 auth_method 0 2024-04-22T08:16:19.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.977+0000 7f32ca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:19.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.977+0000 7f32ca59c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:19.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.977+0000 7f32ca59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:19.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.977+0000 7f32c9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:19.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.977+0000 7f32c9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:19.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.977+0000 7f32ca59c640 1 -- 172.21.15.40:0/3145655432 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32cc12fc40 msgr2=0x7f32cc130020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:16:19.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.977+0000 7f32ca59c640 1 -- 172.21.15.40:0/3145655432 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32cc12fc40 msgr2=0x7f32cc130020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:16:19.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.977+0000 7f32ca59c640 1 --2- 172.21.15.40:0/3145655432 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32cc12fc40 0x7f32cc130020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:16:19.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.977+0000 7f32c9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:19.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.978+0000 7f32ca59c640 1 --2- 172.21.15.40:0/3145655432 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32cc12fc40 0x7f32cc130020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:16:19.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.978+0000 7f32c9d9b640 10 monclient(hunting): handle_auth_done global_id 34536 payload 274 2024-04-22T08:16:19.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.978+0000 7f32c9d9b640 1 -- 172.21.15.40:0/3145655432 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32cc12fc40 msgr2=0x7f32cc130020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:16:19.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.978+0000 7f32c9d9b640 1 --2- 172.21.15.40:0/3145655432 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32cc12fc40 0x7f32cc130020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:19.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.978+0000 7f32c9d9b640 1 -- 172.21.15.40:0/3145655432 >> v1:172.21.15.40:6789/0 conn(0x7f32cc132ed0 legacy=0x7f32cc139cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:16:19.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.978+0000 7f32c9d9b640 10 monclient: _finish_hunting 0 2024-04-22T08:16:19.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.978+0000 7f32c9d9b640 1 monclient: found mon.noname-a 2024-04-22T08:16:19.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.978+0000 7f32c9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:16:19.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.978+0000 7f32c9d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:16:19.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.978+0000 7f32c9d9b640 1 -- 172.21.15.40:0/3145655432 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32cc09d550 con 0x7f32cc09e210 2024-04-22T08:16:19.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.980+0000 7f32c9d9b640 1 -- 172.21.15.40:0/3145655432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32cc09e210 msgr2=0x7f32cc132990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:16:19.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.980+0000 7f32c9d9b640 1 --2- 172.21.15.40:0/3145655432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32cc09e210 0x7f32cc132990 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f32c00097a0 tx=0x7f32c002b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=526cde4d893b87c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:19.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.981+0000 7f32c959a640 1 -- 172.21.15.40:0/3145655432 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c0030920 con 0x7f32cc09e210 2024-04-22T08:16:19.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.981+0000 7f32c959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:19.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.981+0000 7f32c959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:16:19.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.981+0000 7f32c959a640 10 monclient: dump: 2024-04-22T08:16:19.989 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:19.989 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:19.989 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:19.989 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:19.989 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:19.989 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:19.989 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:19.990 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:19.990 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:19.990 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:19.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.981+0000 7f32c959a640 10 monclient: _finish_auth 0 2024-04-22T08:16:19.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.981+0000 7f32c959a640 10 monclient: _check_auth_tickets 2024-04-22T08:16:19.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.981+0000 7f32c959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:16:19.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.981+0000 7f32d0c1c640 5 monclient: authenticate success, global_id 34536 2024-04-22T08:16:19.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.981+0000 7f32c959a640 1 -- 172.21.15.40:0/3145655432 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32c002e050 con 0x7f32cc09e210 2024-04-22T08:16:19.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.981+0000 7f32d0c1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:16:19.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.981+0000 7f32c959a640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:16:19.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.981+0000 7f32d0c1c640 10 monclient: get_monmap_and_config success 2024-04-22T08:16:19.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.981+0000 7f32d0c1c640 4 set_mon_vals no callback set 2024-04-22T08:16:19.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.981+0000 7f32c959a640 1 -- 172.21.15.40:0/3145655432 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c0030ca0 con 0x7f32cc09e210 2024-04-22T08:16:19.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.981+0000 7f32c959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:19.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.981+0000 7f32c959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:16:19.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.981+0000 7f32c959a640 10 monclient: dump: 2024-04-22T08:16:19.991 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:19.991 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:19.991 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:19.991 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:19.991 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:19.991 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:19.991 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:19.991 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:19.991 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:19.991 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:19.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.981+0000 7f32d0c1c640 10 monclient: shutdown 2024-04-22T08:16:19.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.981+0000 7f32d0c1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:16:19.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.981+0000 7f32d0c1c640 1 -- 172.21.15.40:0/3145655432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32cc09e210 msgr2=0x7f32cc132990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:19.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.981+0000 7f32d0c1c640 1 --2- 172.21.15.40:0/3145655432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32cc09e210 0x7f32cc132990 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f32c00097a0 tx=0x7f32c002b710 comp rx=0 tx=0).stop 2024-04-22T08:16:19.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.981+0000 7f32d0c1c640 1 -- 172.21.15.40:0/3145655432 shutdown_connections 2024-04-22T08:16:19.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.981+0000 7f32d0c1c640 1 --2- 172.21.15.40:0/3145655432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32cc09e210 0x7f32cc132990 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:19.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.981+0000 7f32d0c1c640 1 --2- 172.21.15.40:0/3145655432 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32cc12fc40 0x7f32cc130020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:19.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.981+0000 7f32d0c1c640 1 -- 172.21.15.40:0/3145655432 >> 172.21.15.40:0/3145655432 conn(0x7f32cc0a0b50 msgr2=0x7f32cc0a0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:16:19.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.981+0000 7f32d0c1c640 1 -- 172.21.15.40:0/3145655432 shutdown_connections 2024-04-22T08:16:19.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.981+0000 7f32d0c1c640 1 -- 172.21.15.40:0/3145655432 wait complete. 2024-04-22T08:16:19.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.982+0000 7f32d0c1c640 10 monclient: build_initial_monmap 2024-04-22T08:16:19.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.982+0000 7f32d0c1c640 10 monclient: monmap: 2024-04-22T08:16:19.992 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:16:19.992 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:16:19.992 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:16:19.992 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:16:19.992 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:16:19.993 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:19.993 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:16:19.993 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:16:19.993 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:16:19.993 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:19.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.982+0000 7f32d0c1c640 1 Processor -- start 2024-04-22T08:16:19.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.983+0000 7f32d0c1c640 1 -- start start 2024-04-22T08:16:19.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.983+0000 7f32d0c1c640 10 monclient: init 2024-04-22T08:16:19.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.983+0000 7f32d0c1c640 10 monclient: _reopen_session rank -1 2024-04-22T08:16:19.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.983+0000 7f32d0c1c640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-22T08:16:19.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.983+0000 7f32d0c1c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32cc09e210 0x7f32cc1cabb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:19.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.983+0000 7f32d0c1c640 10 monclient(hunting): picked mon.noname-a con 0x7f32cc09e210 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:16:19.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.983+0000 7f32d0c1c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32cc12fc40 0x7f32cc1cb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:19.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.983+0000 7f32d0c1c640 10 monclient(hunting): picked mon.noname-c con 0x7f32cc12fc40 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:16:19.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.983+0000 7f32d0c1c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32cc132ed0 0x7f32cc06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:19.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.983+0000 7f32d0c1c640 10 monclient(hunting): picked mon.noname-b con 0x7f32cc132ed0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:16:19.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.983+0000 7f32d0c1c640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:19.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.983+0000 7f32d0c1c640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32cc11efa0 con 0x7f32cc09e210 2024-04-22T08:16:19.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.983+0000 7f32d0c1c640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:19.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.983+0000 7f32d0c1c640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32cc071830 con 0x7f32cc132ed0 2024-04-22T08:16:19.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.983+0000 7f32d0c1c640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:19.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.983+0000 7f32d0c1c640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32cc071450 con 0x7f32cc12fc40 2024-04-22T08:16:19.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.983+0000 7f32d0c1c640 10 monclient(hunting): _renew_subs 2024-04-22T08:16:19.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.983+0000 7f32d0c1c640 10 monclient(hunting): authenticate will time out at 1053.978343s 2024-04-22T08:16:19.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.983+0000 7f32ca59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32cc09e210 0x7f32cc1cabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:19.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.983+0000 7f32ca59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32cc09e210 0x7f32cc1cabb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41526/0 (socket says 172.21.15.40:41526) 2024-04-22T08:16:19.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.983+0000 7f32c9d9b640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32cc12fc40 0x7f32cc1cb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:19.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.983+0000 7f32ca59c640 1 -- 172.21.15.40:0/3505642351 learned_addr learned my addr 172.21.15.40:0/3505642351 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:16:19.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.983+0000 7f32ca59c640 10 monclient(hunting): get_auth_request con 0x7f32cc09e210 auth_method 0 2024-04-22T08:16:19.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.983+0000 7f32ca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:19.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.983+0000 7f32ca59c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:19.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.983+0000 7f32ca59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:19.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.983+0000 7f32c9d9b640 10 monclient(hunting): get_auth_request con 0x7f32cc12fc40 auth_method 0 2024-04-22T08:16:19.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.983+0000 7f32c9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:19.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.983+0000 7f32c9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:19.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.983+0000 7f32c9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:19.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.983+0000 7f32ca59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:19.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.983+0000 7f32ca59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:19.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.983+0000 7f32ca59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:19.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.983+0000 7f32c9d9b640 1 -- 172.21.15.40:0/3505642351 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32cc12fc40 msgr2=0x7f32cc1cb0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-22T08:16:19.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.983+0000 7f32c9d9b640 1 -- 172.21.15.40:0/3505642351 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32cc12fc40 msgr2=0x7f32cc1cb0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:16:19.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.983+0000 7f32c9d9b640 1 --2- 172.21.15.40:0/3505642351 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32cc12fc40 0x7f32cc1cb0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:16:19.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.983+0000 7f32c9d9b640 1 --2- 172.21.15.40:0/3505642351 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32cc12fc40 0x7f32cc1cb0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:16:19.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.983+0000 7f32cad9d640 1 --2- 172.21.15.40:0/3505642351 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32cc132ed0 0x7f32cc06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:19.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.983+0000 7f32cad9d640 10 monclient(hunting): get_auth_request con 0x7f32cc132ed0 auth_method 0 2024-04-22T08:16:19.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.983+0000 7f32cad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:19.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.983+0000 7f32cad9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:19.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.983+0000 7f32cad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:19.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.984+0000 7f32ca59c640 10 monclient(hunting): handle_auth_done global_id 34539 payload 824 2024-04-22T08:16:19.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.984+0000 7f32ca59c640 1 -- 172.21.15.40:0/3505642351 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32cc12fc40 msgr2=0x7f32cc1cb0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:16:19.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.984+0000 7f32ca59c640 1 --2- 172.21.15.40:0/3505642351 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32cc12fc40 0x7f32cc1cb0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:19.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.984+0000 7f32ca59c640 1 -- 172.21.15.40:0/3505642351 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32cc132ed0 msgr2=0x7f32cc06dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:19.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.984+0000 7f32ca59c640 1 --2- 172.21.15.40:0/3505642351 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32cc132ed0 0x7f32cc06dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:19.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.984+0000 7f32ca59c640 10 monclient: _finish_hunting 0 2024-04-22T08:16:19.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.984+0000 7f32ca59c640 1 monclient: found mon.noname-a 2024-04-22T08:16:19.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.984+0000 7f32ca59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:16:19.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.984+0000 7f32ca59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:16:19.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.984+0000 7f32ca59c640 1 -- 172.21.15.40:0/3505642351 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32c0009400 con 0x7f32cc09e210 2024-04-22T08:16:19.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.984+0000 7f32cad9d640 1 --2- 172.21.15.40:0/3505642351 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32cc132ed0 0x7f32cc06dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T08:16:19.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.986+0000 7f32ca59c640 1 -- 172.21.15.40:0/3505642351 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32cc09e210 msgr2=0x7f32cc1cabb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:16:19.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.986+0000 7f32ca59c640 1 --2- 172.21.15.40:0/3505642351 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32cc09e210 0x7f32cc1cabb0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f32bc00d7d0 tx=0x7f32bc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:19.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.986+0000 7f32baffd640 1 -- 172.21.15.40:0/3505642351 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32bc00b8d0 con 0x7f32cc09e210 2024-04-22T08:16:19.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.986+0000 7f32baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:19.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.986+0000 7f32baffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:16:19.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.986+0000 7f32baffd640 10 monclient: dump: 2024-04-22T08:16:19.997 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:19.997 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:19.997 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:19.997 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:19.997 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:19.997 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:19.997 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:19.998 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:19.998 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:19.998 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:19.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.986+0000 7f32baffd640 10 monclient: _finish_auth 0 2024-04-22T08:16:19.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.986+0000 7f32baffd640 10 monclient: _check_auth_tickets 2024-04-22T08:16:19.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.986+0000 7f32baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:16:19.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.986+0000 7f32baffd640 1 -- 172.21.15.40:0/3505642351 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32bc00ba50 con 0x7f32cc09e210 2024-04-22T08:16:19.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.986+0000 7f32d0c1c640 5 monclient: authenticate success, global_id 34539 2024-04-22T08:16:19.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.986+0000 7f32baffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:16:19.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.986+0000 7f32baffd640 1 -- 172.21.15.40:0/3505642351 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32bc00be00 con 0x7f32cc09e210 2024-04-22T08:16:19.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.986+0000 7f32baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:19.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.986+0000 7f32baffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:16:19.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.986+0000 7f32baffd640 10 monclient: dump: 2024-04-22T08:16:19.998 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:19.999 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:19.999 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:19.999 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:19.999 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:19.999 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:19.999 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:19.999 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:19.999 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:19.999 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:19.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.986+0000 7f32d0c1c640 10 monclient: _renew_subs 2024-04-22T08:16:19.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.986+0000 7f32d0c1c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:16:19.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.986+0000 7f32c8d99640 4 set_mon_vals no callback set 2024-04-22T08:16:19.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.986+0000 7f32d0c1c640 1 -- 172.21.15.40:0/3505642351 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32cc11ecc0 con 0x7f32cc09e210 2024-04-22T08:16:19.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.986+0000 7f32d0c1c640 10 monclient: _renew_subs 2024-04-22T08:16:20.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.986+0000 7f32d0c1c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:16:20.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.986+0000 7f32d0c1c640 1 -- 172.21.15.40:0/3505642351 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32cc1cb810 con 0x7f32cc09e210 2024-04-22T08:16:20.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.987+0000 7f32baffd640 1 -- 172.21.15.40:0/3505642351 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f32bc01e460 con 0x7f32cc09e210 2024-04-22T08:16:20.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.987+0000 7f32d0c1c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:16:20.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.987+0000 7f32d0c1c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:16:20.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.987+0000 7f32d0c1c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:16:20.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.987+0000 7f32d0c1c640 1 -- 172.21.15.40:0/3505642351 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32cc04e9b0 con 0x7f32cc09e210 2024-04-22T08:16:20.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.987+0000 7f32baffd640 1 --2- 172.21.15.40:0/3505642351 >> v2:172.21.15.189:6800/30168 conn(0x7f32ac037830 0x7f32ac039cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:20.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.987+0000 7f32baffd640 1 -- 172.21.15.40:0/3505642351 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f32bc04a9d0 con 0x7f32cc09e210 2024-04-22T08:16:20.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.988+0000 7f32c9d9b640 1 --2- 172.21.15.40:0/3505642351 >> v2:172.21.15.189:6800/30168 conn(0x7f32ac037830 0x7f32ac039cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:20.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.988+0000 7f32c9d9b640 10 monclient: get_auth_request con 0x7f32ac037830 auth_method 0 2024-04-22T08:16:20.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.990+0000 7f32c9d9b640 1 --2- 172.21.15.40:0/3505642351 >> v2:172.21.15.189:6800/30168 conn(0x7f32ac037830 0x7f32ac039cf0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f32c0034000 tx=0x7f32c002f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:20.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.992+0000 7f32baffd640 1 -- 172.21.15.40:0/3505642351 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f32bc01c720 con 0x7f32cc09e210 2024-04-22T08:16:20.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.992+0000 7f32baffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:16:20.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:19.992+0000 7f32baffd640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:16:20.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:20.241+0000 7f32d0c1c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:16:20.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:20.241+0000 7f32d0c1c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:16:20.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:20.241+0000 7f32d0c1c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:16:20.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:20.241+0000 7f32d0c1c640 1 -- 172.21.15.40:0/3505642351 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f32cc06a290 con 0x7f32cc09e210 2024-04-22T08:16:20.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:20.242+0000 7f32baffd640 1 -- 172.21.15.40:0/3505642351 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7f32bc03c620 con 0x7f32cc09e210 2024-04-22T08:16:20.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:20.242+0000 7f32baffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:16:20.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:20.242+0000 7f32baffd640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:16:20.243 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":40,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:16:20.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:20.243+0000 7f32d0c1c640 1 -- 172.21.15.40:0/3505642351 >> v2:172.21.15.189:6800/30168 conn(0x7f32ac037830 msgr2=0x7f32ac039cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:20.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:20.243+0000 7f32d0c1c640 1 --2- 172.21.15.40:0/3505642351 >> v2:172.21.15.189:6800/30168 conn(0x7f32ac037830 0x7f32ac039cf0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f32c0034000 tx=0x7f32c002f000 comp rx=0 tx=0).stop 2024-04-22T08:16:20.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:20.243+0000 7f32d0c1c640 10 monclient: shutdown 2024-04-22T08:16:20.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:20.243+0000 7f32d0c1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:16:20.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:20.243+0000 7f32d0c1c640 1 -- 172.21.15.40:0/3505642351 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32cc09e210 msgr2=0x7f32cc1cabb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:20.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:20.243+0000 7f32d0c1c640 1 --2- 172.21.15.40:0/3505642351 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32cc09e210 0x7f32cc1cabb0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f32bc00d7d0 tx=0x7f32bc00dc90 comp rx=0 tx=0).stop 2024-04-22T08:16:20.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:20.244+0000 7f32d0c1c640 1 -- 172.21.15.40:0/3505642351 shutdown_connections 2024-04-22T08:16:20.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:20.244+0000 7f32d0c1c640 1 --2- 172.21.15.40:0/3505642351 >> v2:172.21.15.189:6800/30168 conn(0x7f32ac037830 0x7f32ac039cf0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:20.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:20.244+0000 7f32d0c1c640 1 --2- 172.21.15.40:0/3505642351 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32cc132ed0 0x7f32cc06dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:20.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:20.244+0000 7f32d0c1c640 1 --2- 172.21.15.40:0/3505642351 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32cc12fc40 0x7f32cc1cb0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:20.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:20.244+0000 7f32d0c1c640 1 --2- 172.21.15.40:0/3505642351 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32cc09e210 0x7f32cc1cabb0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:20.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:20.244+0000 7f32d0c1c640 1 -- 172.21.15.40:0/3505642351 >> 172.21.15.40:0/3505642351 conn(0x7f32cc0a0b50 msgr2=0x7f32cc09b630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:16:20.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:20.244+0000 7f32d0c1c640 1 -- 172.21.15.40:0/3505642351 shutdown_connections 2024-04-22T08:16:20.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:20.244+0000 7f32d0c1c640 1 -- 172.21.15.40:0/3505642351 wait complete. 2024-04-22T08:16:22.033 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36414]: starting. 2024-04-22T08:16:22.033 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36414]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-36413 2024-04-22T08:16:22.033 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36414]: created object 0... 2024-04-22T08:16:22.033 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36414]: created object 25... 2024-04-22T08:16:22.033 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36414]: created object 49... 2024-04-22T08:16:22.033 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36414]: finishing. 2024-04-22T08:16:22.034 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36414]: shutting down. 2024-04-22T08:16:22.920 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[36415]: starting. 2024-04-22T08:16:22.920 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[36415]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-36413 2024-04-22T08:16:22.921 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[36415]: shutting down. 2024-04-22T08:16:23.256 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:16:23.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.366+0000 7fa84b63a640 10 monclient: get_monmap_and_config 2024-04-22T08:16:23.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.366+0000 7fa84b63a640 10 monclient: build_initial_monmap 2024-04-22T08:16:23.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.366+0000 7fa84b63a640 10 monclient: monmap: 2024-04-22T08:16:23.367 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:16:23.367 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:23.367 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:16:23.367884+0000 2024-04-22T08:16:23.367 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:16:23.367884+0000 2024-04-22T08:16:23.367 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:16:23.367 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:23.367 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:16:23.367 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:16:23.367 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:16:23.367 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:16:23.368 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:16:23.368 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:23.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.366+0000 7fa84b63a640 1 Processor -- start 2024-04-22T08:16:23.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.366+0000 7fa84b63a640 1 -- start start 2024-04-22T08:16:23.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.366+0000 7fa84b63a640 10 monclient: init 2024-04-22T08:16:23.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.367+0000 7fa84b63a640 10 monclient: _reopen_session rank -1 2024-04-22T08:16:23.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.367+0000 7fa84b63a640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-04-22T08:16:23.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.367+0000 7fa84b63a640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa84409ee10 0x7fa8440a32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:23.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.367+0000 7fa84b63a640 10 monclient(hunting): picked mon.noname-a con 0x7fa84409ee10 addr v2:172.21.15.40:3300/0 2024-04-22T08:16:23.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.367+0000 7fa84b63a640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa8440a1d30 0x7fa8440a2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:23.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.367+0000 7fa84b63a640 10 monclient(hunting): picked mon.noname-b con 0x7fa8440a1d30 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:16:23.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.367+0000 7fa84b63a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa844135d90 addr v1:172.21.15.40:6789/0 2024-04-22T08:16:23.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.367+0000 7fa84b63a640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa84409f1f0 con 0x7fa844135d90 2024-04-22T08:16:23.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.367+0000 7fa84b63a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:23.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.367+0000 7fa84b63a640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8440a39c0 con 0x7fa84409ee10 2024-04-22T08:16:23.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.367+0000 7fa84b63a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:23.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.367+0000 7fa84b63a640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa84409fd30 con 0x7fa8440a1d30 2024-04-22T08:16:23.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.367+0000 7fa84b63a640 10 monclient(hunting): _renew_subs 2024-04-22T08:16:23.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.367+0000 7fa84b63a640 10 monclient(hunting): authenticate will time out at 1057.362204s 2024-04-22T08:16:23.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.367+0000 7fa848bd5640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa8440a1d30 0x7fa8440a2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:23.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.367+0000 7fa8493d6640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa84409ee10 0x7fa8440a32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:23.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.367+0000 7fa848bd5640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa8440a1d30 0x7fa8440a2150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:33126/0 (socket says 172.21.15.40:33126) 2024-04-22T08:16:23.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.367+0000 7fa848bd5640 1 -- 172.21.15.40:0/3931335886 learned_addr learned my addr 172.21.15.40:0/3931335886 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:16:23.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.367+0000 7fa8493d6640 10 monclient(hunting): get_auth_request con 0x7fa84409ee10 auth_method 0 2024-04-22T08:16:23.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.367+0000 7fa8493d6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:23.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.367+0000 7fa8493d6640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:23.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.367+0000 7fa8493d6640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:23.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.367+0000 7fa848bd5640 10 monclient(hunting): get_auth_request con 0x7fa8440a1d30 auth_method 0 2024-04-22T08:16:23.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.367+0000 7fa848bd5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:23.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.367+0000 7fa848bd5640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:23.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.367+0000 7fa848bd5640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:23.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.367+0000 7fa848bd5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:23.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.367+0000 7fa848bd5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:23.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.367+0000 7fa848bd5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:23.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.367+0000 7fa8493d6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:23.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.367+0000 7fa8493d6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:23.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.367+0000 7fa8493d6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:23.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.367+0000 7fa848bd5640 10 monclient(hunting): handle_auth_done global_id 54119 payload 274 2024-04-22T08:16:23.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.367+0000 7fa848bd5640 1 -- 172.21.15.40:0/3931335886 >> v2:172.21.15.40:3300/0 conn(0x7fa84409ee10 msgr2=0x7fa8440a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:16:23.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.367+0000 7fa848bd5640 1 --2- 172.21.15.40:0/3931335886 >> v2:172.21.15.40:3300/0 conn(0x7fa84409ee10 0x7fa8440a32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:23.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.367+0000 7fa848bd5640 1 -- 172.21.15.40:0/3931335886 >> v1:172.21.15.40:6789/0 conn(0x7fa844135d90 legacy=0x7fa8441361c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:16:23.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.367+0000 7fa848bd5640 10 monclient: _finish_hunting 0 2024-04-22T08:16:23.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.367+0000 7fa848bd5640 1 monclient: found mon.noname-b 2024-04-22T08:16:23.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.367+0000 7fa848bd5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:16:23.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.367+0000 7fa848bd5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:16:23.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.367+0000 7fa848bd5640 1 -- 172.21.15.40:0/3931335886 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8440a02f0 con 0x7fa8440a1d30 2024-04-22T08:16:23.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.367+0000 7fa8493d6640 1 --2- 172.21.15.40:0/3931335886 >> v2:172.21.15.40:3300/0 conn(0x7fa84409ee10 0x7fa8440a32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:16:23.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.370+0000 7fa848bd5640 1 -- 172.21.15.40:0/3931335886 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa8440a1d30 msgr2=0x7fa8440a2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:16:23.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.370+0000 7fa848bd5640 1 --2- 172.21.15.40:0/3931335886 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa8440a1d30 0x7fa8440a2150 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa838009870 tx=0x7fa83802b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f301e6e937ba0766 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:23.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.370+0000 7fa843fff640 1 -- 172.21.15.40:0/3931335886 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa838031950 con 0x7fa8440a1d30 2024-04-22T08:16:23.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.370+0000 7fa843fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:23.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.370+0000 7fa843fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:16:23.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.370+0000 7fa843fff640 10 monclient: dump: 2024-04-22T08:16:23.379 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:23.379 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:23.379 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:23.379 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:23.379 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:23.379 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:23.379 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:23.379 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:23.380 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:23.380 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:23.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.370+0000 7fa843fff640 10 monclient: _finish_auth 0 2024-04-22T08:16:23.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.370+0000 7fa843fff640 10 monclient: _check_auth_tickets 2024-04-22T08:16:23.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.370+0000 7fa843fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:16:23.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.370+0000 7fa843fff640 1 -- 172.21.15.40:0/3931335886 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa838031ab0 con 0x7fa8440a1d30 2024-04-22T08:16:23.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.370+0000 7fa84b63a640 5 monclient: authenticate success, global_id 54119 2024-04-22T08:16:23.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.370+0000 7fa84b63a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:16:23.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.370+0000 7fa843fff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:16:23.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.370+0000 7fa843fff640 1 -- 172.21.15.40:0/3931335886 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa838031e40 con 0x7fa8440a1d30 2024-04-22T08:16:23.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.370+0000 7fa84b63a640 10 monclient: get_monmap_and_config success 2024-04-22T08:16:23.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.370+0000 7fa84b63a640 4 set_mon_vals no callback set 2024-04-22T08:16:23.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.370+0000 7fa843fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:23.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.371+0000 7fa843fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:16:23.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.371+0000 7fa843fff640 10 monclient: dump: 2024-04-22T08:16:23.381 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:23.381 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:23.381 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:23.381 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:23.381 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:23.381 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:23.381 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:23.381 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:23.381 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:23.381 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:23.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.371+0000 7fa84b63a640 10 monclient: shutdown 2024-04-22T08:16:23.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.371+0000 7fa84b63a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:16:23.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.371+0000 7fa84b63a640 1 -- 172.21.15.40:0/3931335886 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa8440a1d30 msgr2=0x7fa8440a2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:23.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.371+0000 7fa84b63a640 1 --2- 172.21.15.40:0/3931335886 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa8440a1d30 0x7fa8440a2150 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa838009870 tx=0x7fa83802b7e0 comp rx=0 tx=0).stop 2024-04-22T08:16:23.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.371+0000 7fa84b63a640 1 -- 172.21.15.40:0/3931335886 shutdown_connections 2024-04-22T08:16:23.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.371+0000 7fa84b63a640 1 --2- 172.21.15.40:0/3931335886 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa8440a1d30 0x7fa8440a2150 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:23.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.371+0000 7fa84b63a640 1 --2- 172.21.15.40:0/3931335886 >> v2:172.21.15.40:3300/0 conn(0x7fa84409ee10 0x7fa8440a32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:23.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.371+0000 7fa84b63a640 1 -- 172.21.15.40:0/3931335886 >> 172.21.15.40:0/3931335886 conn(0x7fa8440a4ef0 msgr2=0x7fa8440a5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:16:23.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.371+0000 7fa84b63a640 1 -- 172.21.15.40:0/3931335886 shutdown_connections 2024-04-22T08:16:23.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.371+0000 7fa84b63a640 1 -- 172.21.15.40:0/3931335886 wait complete. 2024-04-22T08:16:23.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.372+0000 7fa84b63a640 10 monclient: build_initial_monmap 2024-04-22T08:16:23.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.372+0000 7fa84b63a640 10 monclient: monmap: 2024-04-22T08:16:23.382 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:16:23.382 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:16:23.382 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:16:23.382 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:16:23.382 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:16:23.382 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:23.383 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:16:23.383 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:16:23.383 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:16:23.383 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:23.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.372+0000 7fa84b63a640 1 Processor -- start 2024-04-22T08:16:23.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.372+0000 7fa84b63a640 1 -- start start 2024-04-22T08:16:23.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.372+0000 7fa84b63a640 10 monclient: init 2024-04-22T08:16:23.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa84b63a640 10 monclient: _reopen_session rank -1 2024-04-22T08:16:23.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa84b63a640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-22T08:16:23.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa84b63a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa84409ee10 0x7fa8441cae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:23.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa84b63a640 10 monclient(hunting): picked mon.noname-b con 0x7fa84409ee10 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:16:23.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa84b63a640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa8440a1d30 0x7fa8441cb380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:23.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa84b63a640 10 monclient(hunting): picked mon.noname-c con 0x7fa8440a1d30 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:16:23.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa84b63a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa844135d90 0x7fa844122250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:23.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa84b63a640 10 monclient(hunting): picked mon.noname-a con 0x7fa844135d90 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:16:23.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa84b63a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:23.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa84b63a640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa844125890 con 0x7fa844135d90 2024-04-22T08:16:23.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa84b63a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:23.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa84b63a640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8441249a0 con 0x7fa84409ee10 2024-04-22T08:16:23.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa84b63a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:23.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa84b63a640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8440a2730 con 0x7fa8440a1d30 2024-04-22T08:16:23.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa84b63a640 10 monclient(hunting): _renew_subs 2024-04-22T08:16:23.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa84b63a640 10 monclient(hunting): authenticate will time out at 1057.368115s 2024-04-22T08:16:23.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa848bd5640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa8440a1d30 0x7fa8441cb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:23.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa849bd7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa844135d90 0x7fa844122250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:23.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa848bd5640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa8440a1d30 0x7fa8441cb380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:33132/0 (socket says 172.21.15.40:33132) 2024-04-22T08:16:23.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa848bd5640 1 -- 172.21.15.40:0/3239885306 learned_addr learned my addr 172.21.15.40:0/3239885306 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:16:23.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa848bd5640 10 monclient(hunting): get_auth_request con 0x7fa8440a1d30 auth_method 0 2024-04-22T08:16:23.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa848bd5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:23.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa848bd5640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:23.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa848bd5640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:23.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa849bd7640 10 monclient(hunting): get_auth_request con 0x7fa844135d90 auth_method 0 2024-04-22T08:16:23.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa849bd7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:23.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa849bd7640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:23.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa849bd7640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:23.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa849bd7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:23.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa849bd7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:23.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa849bd7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:23.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa848bd5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:23.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa848bd5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:23.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa848bd5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:23.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa8493d6640 1 --2- 172.21.15.40:0/3239885306 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa84409ee10 0x7fa8441cae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:23.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa8493d6640 10 monclient(hunting): get_auth_request con 0x7fa84409ee10 auth_method 0 2024-04-22T08:16:23.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa8493d6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:23.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa8493d6640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:23.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa8493d6640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:23.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa8493d6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:23.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa8493d6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:23.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa8493d6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:23.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa848bd5640 10 monclient(hunting): handle_auth_done global_id 54122 payload 824 2024-04-22T08:16:23.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa848bd5640 1 -- 172.21.15.40:0/3239885306 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa84409ee10 msgr2=0x7fa8441cae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:23.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa848bd5640 1 --2- 172.21.15.40:0/3239885306 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa84409ee10 0x7fa8441cae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:23.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa848bd5640 1 -- 172.21.15.40:0/3239885306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa844135d90 msgr2=0x7fa844122250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:23.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa848bd5640 1 --2- 172.21.15.40:0/3239885306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa844135d90 0x7fa844122250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:23.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa848bd5640 10 monclient: _finish_hunting 0 2024-04-22T08:16:23.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa848bd5640 1 monclient: found mon.noname-c 2024-04-22T08:16:23.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa848bd5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:16:23.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa848bd5640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:16:23.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa848bd5640 1 -- 172.21.15.40:0/3239885306 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa83c009400 con 0x7fa8440a1d30 2024-04-22T08:16:23.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.373+0000 7fa849bd7640 1 --2- 172.21.15.40:0/3239885306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa844135d90 0x7fa844122250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:16:23.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.376+0000 7fa848bd5640 1 -- 172.21.15.40:0/3239885306 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa8440a1d30 msgr2=0x7fa8441cb380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:16:23.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.376+0000 7fa848bd5640 1 --2- 172.21.15.40:0/3239885306 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa8440a1d30 0x7fa8441cb380 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa8380090b0 tx=0x7fa8380046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:23.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.376+0000 7fa841ffb640 1 -- 172.21.15.40:0/3239885306 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa838033c30 con 0x7fa8440a1d30 2024-04-22T08:16:23.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.376+0000 7fa841ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:23.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.376+0000 7fa841ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:16:23.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.376+0000 7fa841ffb640 10 monclient: dump: 2024-04-22T08:16:23.387 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:23.387 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:23.387 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:23.387 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:23.388 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:23.388 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:23.388 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:23.388 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:23.388 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:23.388 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:23.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.376+0000 7fa841ffb640 10 monclient: _finish_auth 0 2024-04-22T08:16:23.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.376+0000 7fa841ffb640 10 monclient: _check_auth_tickets 2024-04-22T08:16:23.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.376+0000 7fa841ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:16:23.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.376+0000 7fa841ffb640 1 -- 172.21.15.40:0/3239885306 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa838033d90 con 0x7fa8440a1d30 2024-04-22T08:16:23.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.376+0000 7fa84b63a640 5 monclient: authenticate success, global_id 54122 2024-04-22T08:16:23.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.376+0000 7fa84b63a640 10 monclient: _renew_subs 2024-04-22T08:16:23.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.376+0000 7fa84b63a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:16:23.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.376+0000 7fa84b63a640 1 -- 172.21.15.40:0/3239885306 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8380094d0 con 0x7fa8440a1d30 2024-04-22T08:16:23.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.376+0000 7fa841ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:16:23.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.376+0000 7fa841ffb640 1 -- 172.21.15.40:0/3239885306 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa83803c640 con 0x7fa8440a1d30 2024-04-22T08:16:23.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.376+0000 7fa841ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:23.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.376+0000 7fa841ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:16:23.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.376+0000 7fa841ffb640 10 monclient: dump: 2024-04-22T08:16:23.389 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:23.389 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:23.389 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:23.389 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:23.389 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:23.389 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:23.389 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:23.389 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:23.389 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:23.389 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:23.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.376+0000 7fa84b63a640 10 monclient: _renew_subs 2024-04-22T08:16:23.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.376+0000 7fa84b63a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:16:23.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.376+0000 7fa84b63a640 1 -- 172.21.15.40:0/3239885306 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa844126130 con 0x7fa8440a1d30 2024-04-22T08:16:23.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.376+0000 7fa8437fe640 4 set_mon_vals no callback set 2024-04-22T08:16:23.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.377+0000 7fa841ffb640 1 -- 172.21.15.40:0/3239885306 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fa83803a040 con 0x7fa8440a1d30 2024-04-22T08:16:23.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.377+0000 7fa84b63a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:16:23.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.377+0000 7fa84b63a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:16:23.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.377+0000 7fa84b63a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:16:23.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.377+0000 7fa841ffb640 1 --2- 172.21.15.40:0/3239885306 >> v2:172.21.15.189:6800/30168 conn(0x7fa8300377d0 0x7fa830039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:23.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.377+0000 7fa84b63a640 1 -- 172.21.15.40:0/3239885306 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa844123e20 con 0x7fa8440a1d30 2024-04-22T08:16:23.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.377+0000 7fa841ffb640 1 -- 172.21.15.40:0/3239885306 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 3506+0+0 (secure 0 0 0) 0x7fa83806bcb0 con 0x7fa8440a1d30 2024-04-22T08:16:23.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.377+0000 7fa8493d6640 1 --2- 172.21.15.40:0/3239885306 >> v2:172.21.15.189:6800/30168 conn(0x7fa8300377d0 0x7fa830039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:23.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.377+0000 7fa8493d6640 10 monclient: get_auth_request con 0x7fa8300377d0 auth_method 0 2024-04-22T08:16:23.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.380+0000 7fa8493d6640 1 --2- 172.21.15.40:0/3239885306 >> v2:172.21.15.189:6800/30168 conn(0x7fa8300377d0 0x7fa830039c90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa83c009c30 tx=0x7fa83c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:23.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.382+0000 7fa841ffb640 1 -- 172.21.15.40:0/3239885306 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7fa844123e20 con 0x7fa8440a1d30 2024-04-22T08:16:23.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.382+0000 7fa841ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:16:23.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.382+0000 7fa841ffb640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:16:23.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.629+0000 7fa84b63a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:16:23.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.630+0000 7fa84b63a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:16:23.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.630+0000 7fa84b63a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:16:23.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.630+0000 7fa84b63a640 1 -- 172.21.15.40:0/3239885306 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa844123e20 con 0x7fa8440a1d30 2024-04-22T08:16:23.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.630+0000 7fa841ffb640 1 -- 172.21.15.40:0/3239885306 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7fa844123e20 con 0x7fa8440a1d30 2024-04-22T08:16:23.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.630+0000 7fa841ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:16:23.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.630+0000 7fa841ffb640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:16:23.631 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":42,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:16:23.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.632+0000 7fa84b63a640 1 -- 172.21.15.40:0/3239885306 >> v2:172.21.15.189:6800/30168 conn(0x7fa8300377d0 msgr2=0x7fa830039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:23.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.632+0000 7fa84b63a640 1 --2- 172.21.15.40:0/3239885306 >> v2:172.21.15.189:6800/30168 conn(0x7fa8300377d0 0x7fa830039c90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa83c009c30 tx=0x7fa83c009190 comp rx=0 tx=0).stop 2024-04-22T08:16:23.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.632+0000 7fa84b63a640 10 monclient: shutdown 2024-04-22T08:16:23.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.632+0000 7fa84b63a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:16:23.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.632+0000 7fa84b63a640 1 -- 172.21.15.40:0/3239885306 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa8440a1d30 msgr2=0x7fa8441cb380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:23.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.632+0000 7fa84b63a640 1 --2- 172.21.15.40:0/3239885306 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa8440a1d30 0x7fa8441cb380 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa8380090b0 tx=0x7fa8380046c0 comp rx=0 tx=0).stop 2024-04-22T08:16:23.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.632+0000 7fa84b63a640 1 -- 172.21.15.40:0/3239885306 shutdown_connections 2024-04-22T08:16:23.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.632+0000 7fa84b63a640 1 --2- 172.21.15.40:0/3239885306 >> v2:172.21.15.189:6800/30168 conn(0x7fa8300377d0 0x7fa830039c90 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:23.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.632+0000 7fa84b63a640 1 --2- 172.21.15.40:0/3239885306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa844135d90 0x7fa844122250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:23.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.632+0000 7fa84b63a640 1 --2- 172.21.15.40:0/3239885306 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa8440a1d30 0x7fa8441cb380 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:23.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.632+0000 7fa84b63a640 1 --2- 172.21.15.40:0/3239885306 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa84409ee10 0x7fa8441cae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:23.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.632+0000 7fa84b63a640 1 -- 172.21.15.40:0/3239885306 >> 172.21.15.40:0/3239885306 conn(0x7fa8440a4ef0 msgr2=0x7fa84406eea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:16:23.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.632+0000 7fa84b63a640 1 -- 172.21.15.40:0/3239885306 shutdown_connections 2024-04-22T08:16:23.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.632+0000 7fa84b63a640 1 -- 172.21.15.40:0/3239885306 wait complete. 2024-04-22T08:16:23.643 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-22T08:16:23.643 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-22T08:16:23.643 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-22T08:16:23.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.743+0000 7ff8f99be640 10 monclient: get_monmap_and_config 2024-04-22T08:16:23.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.743+0000 7ff8f99be640 10 monclient: build_initial_monmap 2024-04-22T08:16:23.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.743+0000 7ff8f99be640 10 monclient: monmap: 2024-04-22T08:16:23.744 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:16:23.744 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:23.744 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:16:23.744716+0000 2024-04-22T08:16:23.744 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:16:23.744716+0000 2024-04-22T08:16:23.744 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:16:23.744 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:23.744 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:16:23.744 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:16:23.744 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:16:23.744 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:16:23.744 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:16:23.744 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:23.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.743+0000 7ff8f99be640 1 Processor -- start 2024-04-22T08:16:23.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.743+0000 7ff8f99be640 1 -- start start 2024-04-22T08:16:23.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.743+0000 7ff8f99be640 10 monclient: init 2024-04-22T08:16:23.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.743+0000 7ff8f99be640 10 monclient: _reopen_session rank -1 2024-04-22T08:16:23.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.743+0000 7ff8f99be640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-04-22T08:16:23.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.743+0000 7ff8f99be640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff8f4135fe0 0x7ff8f40a01c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:23.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.743+0000 7ff8f99be640 10 monclient(hunting): picked mon.noname-c con 0x7ff8f4135fe0 addr v2:172.21.15.189:3300/0 2024-04-22T08:16:23.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.743+0000 7ff8f99be640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff8f41363c0 addr v1:172.21.15.40:6789/0 2024-04-22T08:16:23.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.744+0000 7ff8f99be640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff8f41367d0 addr v1:172.21.15.189:6789/0 2024-04-22T08:16:23.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.744+0000 7ff8f99be640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f40a28a0 con 0x7ff8f41363c0 2024-04-22T08:16:23.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.744+0000 7ff8f99be640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f409d7f0 con 0x7ff8f41367d0 2024-04-22T08:16:23.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.744+0000 7ff8f99be640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:23.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.744+0000 7ff8f99be640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8f409f330 con 0x7ff8f4135fe0 2024-04-22T08:16:23.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.744+0000 7ff8f99be640 10 monclient(hunting): _renew_subs 2024-04-22T08:16:23.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.744+0000 7ff8f99be640 10 monclient(hunting): authenticate will time out at 1057.738823s 2024-04-22T08:16:23.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.744+0000 7ff8f27fc640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7ff8f41363c0 0x7ff8f40a1a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57448/0 (socket says 172.21.15.40:57448) 2024-04-22T08:16:23.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.744+0000 7ff8f27fc640 1 -- 172.21.15.40:0/4177375761 learned_addr learned my addr 172.21.15.40:0/4177375761 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:16:23.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.744+0000 7ff8f2ffd640 1 --2- 172.21.15.40:0/4177375761 >> v2:172.21.15.189:3300/0 conn(0x7ff8f4135fe0 0x7ff8f40a01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:23.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.744+0000 7ff8f2ffd640 10 monclient(hunting): get_auth_request con 0x7ff8f4135fe0 auth_method 0 2024-04-22T08:16:23.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.744+0000 7ff8f2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:23.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.744+0000 7ff8f2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:23.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.744+0000 7ff8f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:23.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.744+0000 7ff8f2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:23.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.744+0000 7ff8f2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:23.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.744+0000 7ff8f2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:23.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.744+0000 7ff8f2ffd640 10 monclient(hunting): handle_auth_done global_id 24778 payload 274 2024-04-22T08:16:23.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.744+0000 7ff8f2ffd640 1 -- 172.21.15.40:0/4177375761 >> v1:172.21.15.189:6789/0 conn(0x7ff8f41367d0 legacy=0x7ff8f40a2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:16:23.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.744+0000 7ff8f2ffd640 1 -- 172.21.15.40:0/4177375761 >> v1:172.21.15.40:6789/0 conn(0x7ff8f41363c0 legacy=0x7ff8f40a1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:16:23.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.744+0000 7ff8f2ffd640 10 monclient: _finish_hunting 0 2024-04-22T08:16:23.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.744+0000 7ff8f2ffd640 1 monclient: found mon.noname-c 2024-04-22T08:16:23.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.744+0000 7ff8f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:16:23.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.744+0000 7ff8f2ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.189:3300/0 2024-04-22T08:16:23.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.744+0000 7ff8f2ffd640 1 -- 172.21.15.40:0/4177375761 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f40999f0 con 0x7ff8f4135fe0 2024-04-22T08:16:23.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.748+0000 7ff8f2ffd640 1 -- 172.21.15.40:0/4177375761 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8f4135fe0 msgr2=0x7ff8f40a01c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:16:23.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.748+0000 7ff8f2ffd640 1 --2- 172.21.15.40:0/4177375761 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8f4135fe0 0x7ff8f40a01c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec0097a0 tx=0x7ff8ec02b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fef8c7a48216bab9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:23.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.748+0000 7ff8f1ffb640 1 -- 172.21.15.40:0/4177375761 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ec030a00 con 0x7ff8f4135fe0 2024-04-22T08:16:23.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.748+0000 7ff8f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:23.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.748+0000 7ff8f1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:16:23.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.748+0000 7ff8f1ffb640 10 monclient: dump: 2024-04-22T08:16:23.756 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:23.756 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:23.756 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:23.756 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:23.756 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:23.756 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:23.756 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:23.756 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:23.756 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:23.756 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:23.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.748+0000 7ff8f1ffb640 10 monclient: _finish_auth 0 2024-04-22T08:16:23.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.748+0000 7ff8f1ffb640 10 monclient: _check_auth_tickets 2024-04-22T08:16:23.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.748+0000 7ff8f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:16:23.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.748+0000 7ff8f1ffb640 1 -- 172.21.15.40:0/4177375761 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8ec02e050 con 0x7ff8f4135fe0 2024-04-22T08:16:23.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.748+0000 7ff8f99be640 5 monclient: authenticate success, global_id 24778 2024-04-22T08:16:23.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.748+0000 7ff8f99be640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:16:23.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.748+0000 7ff8f1ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:16:23.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.748+0000 7ff8f99be640 10 monclient: get_monmap_and_config success 2024-04-22T08:16:23.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.748+0000 7ff8f1ffb640 1 -- 172.21.15.40:0/4177375761 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ec030d80 con 0x7ff8f4135fe0 2024-04-22T08:16:23.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.748+0000 7ff8f99be640 4 set_mon_vals no callback set 2024-04-22T08:16:23.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.748+0000 7ff8f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:23.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.748+0000 7ff8f1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:16:23.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.748+0000 7ff8f1ffb640 10 monclient: dump: 2024-04-22T08:16:23.757 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:23.757 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:23.757 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:23.758 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:23.758 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:23.758 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:23.758 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:23.758 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:23.758 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:23.758 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:23.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.748+0000 7ff8f99be640 10 monclient: shutdown 2024-04-22T08:16:23.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.748+0000 7ff8f99be640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:16:23.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.748+0000 7ff8f99be640 1 -- 172.21.15.40:0/4177375761 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8f4135fe0 msgr2=0x7ff8f40a01c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:23.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.748+0000 7ff8f99be640 1 --2- 172.21.15.40:0/4177375761 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8f4135fe0 0x7ff8f40a01c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec0097a0 tx=0x7ff8ec02b710 comp rx=0 tx=0).stop 2024-04-22T08:16:23.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.748+0000 7ff8f99be640 1 -- 172.21.15.40:0/4177375761 shutdown_connections 2024-04-22T08:16:23.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.748+0000 7ff8f99be640 1 --2- 172.21.15.40:0/4177375761 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8f4135fe0 0x7ff8f40a01c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:23.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.748+0000 7ff8f99be640 1 -- 172.21.15.40:0/4177375761 >> 172.21.15.40:0/4177375761 conn(0x7ff8f40a66a0 msgr2=0x7ff8f40a2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:16:23.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.748+0000 7ff8f99be640 1 -- 172.21.15.40:0/4177375761 shutdown_connections 2024-04-22T08:16:23.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.748+0000 7ff8f99be640 1 -- 172.21.15.40:0/4177375761 wait complete. 2024-04-22T08:16:23.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.749+0000 7ff8f99be640 10 monclient: build_initial_monmap 2024-04-22T08:16:23.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.749+0000 7ff8f99be640 10 monclient: monmap: 2024-04-22T08:16:23.759 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:16:23.759 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:16:23.759 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:16:23.759 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:16:23.759 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:16:23.759 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:23.760 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:16:23.760 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:16:23.760 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:16:23.760 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:23.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.749+0000 7ff8f99be640 1 Processor -- start 2024-04-22T08:16:23.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.749+0000 7ff8f99be640 1 -- start start 2024-04-22T08:16:23.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.749+0000 7ff8f99be640 10 monclient: init 2024-04-22T08:16:23.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f99be640 10 monclient: _reopen_session rank -1 2024-04-22T08:16:23.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f99be640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-22T08:16:23.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f99be640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8f4135fe0 0x7ff8f41caca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:23.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f99be640 10 monclient(hunting): picked mon.noname-a con 0x7ff8f4135fe0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:16:23.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f99be640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8f41363c0 0x7ff8f41cb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:23.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f99be640 10 monclient(hunting): picked mon.noname-c con 0x7ff8f41363c0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:16:23.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f99be640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8f41367d0 0x7ff8f4122200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:23.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f99be640 10 monclient(hunting): picked mon.noname-b con 0x7ff8f41367d0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:16:23.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f99be640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:23.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f99be640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f4125e90 con 0x7ff8f4135fe0 2024-04-22T08:16:23.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f99be640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:23.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f99be640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f41260e0 con 0x7ff8f41367d0 2024-04-22T08:16:23.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f99be640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:23.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f99be640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f41248e0 con 0x7ff8f41363c0 2024-04-22T08:16:23.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f99be640 10 monclient(hunting): _renew_subs 2024-04-22T08:16:23.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f99be640 10 monclient(hunting): authenticate will time out at 1057.744906s 2024-04-22T08:16:23.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f2ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8f4135fe0 0x7ff8f41caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:23.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f27fc640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8f41363c0 0x7ff8f41cb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:23.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f2ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8f4135fe0 0x7ff8f41caca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41546/0 (socket says 172.21.15.40:41546) 2024-04-22T08:16:23.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f2ffd640 1 -- 172.21.15.40:0/1616761528 learned_addr learned my addr 172.21.15.40:0/1616761528 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:16:23.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f2ffd640 10 monclient(hunting): get_auth_request con 0x7ff8f4135fe0 auth_method 0 2024-04-22T08:16:23.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:23.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:23.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:23.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f27fc640 10 monclient(hunting): get_auth_request con 0x7ff8f41363c0 auth_method 0 2024-04-22T08:16:23.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:23.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f27fc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:23.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:23.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:23.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:23.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:23.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:23.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:23.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:23.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f37fe640 1 --2- 172.21.15.40:0/1616761528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8f41367d0 0x7ff8f4122200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:23.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f37fe640 10 monclient(hunting): get_auth_request con 0x7ff8f41367d0 auth_method 0 2024-04-22T08:16:23.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:23.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f37fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:23.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:23.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f27fc640 10 monclient(hunting): handle_auth_done global_id 54125 payload 824 2024-04-22T08:16:23.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f27fc640 1 -- 172.21.15.40:0/1616761528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8f41367d0 msgr2=0x7ff8f4122200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:23.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f27fc640 1 --2- 172.21.15.40:0/1616761528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8f41367d0 0x7ff8f4122200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:23.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f27fc640 1 -- 172.21.15.40:0/1616761528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8f4135fe0 msgr2=0x7ff8f41caca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:23.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f27fc640 1 --2- 172.21.15.40:0/1616761528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8f4135fe0 0x7ff8f41caca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:23.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f27fc640 10 monclient: _finish_hunting 0 2024-04-22T08:16:23.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f27fc640 1 monclient: found mon.noname-c 2024-04-22T08:16:23.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:16:23.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f27fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:16:23.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f27fc640 1 -- 172.21.15.40:0/1616761528 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8ec009400 con 0x7ff8f41363c0 2024-04-22T08:16:23.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.750+0000 7ff8f2ffd640 1 --2- 172.21.15.40:0/1616761528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8f4135fe0 0x7ff8f41caca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:16:23.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.752+0000 7ff8f27fc640 1 -- 172.21.15.40:0/1616761528 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8f41363c0 msgr2=0x7ff8f41cb1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:16:23.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.752+0000 7ff8f27fc640 1 --2- 172.21.15.40:0/1616761528 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8f41363c0 0x7ff8f41cb1e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff8e800cb20 tx=0x7ff8e800a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:23.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.752+0000 7ff8dbfff640 1 -- 172.21.15.40:0/1616761528 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e8012400 con 0x7ff8f41363c0 2024-04-22T08:16:23.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.753+0000 7ff8dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:23.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.753+0000 7ff8dbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:16:23.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.753+0000 7ff8dbfff640 10 monclient: dump: 2024-04-22T08:16:23.764 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:23.764 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:23.764 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:23.764 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:23.764 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:23.764 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:23.765 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:23.765 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:23.765 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:23.765 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:23.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.753+0000 7ff8dbfff640 10 monclient: _finish_auth 0 2024-04-22T08:16:23.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.753+0000 7ff8dbfff640 10 monclient: _check_auth_tickets 2024-04-22T08:16:23.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.753+0000 7ff8dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:16:23.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.753+0000 7ff8f99be640 5 monclient: authenticate success, global_id 54125 2024-04-22T08:16:23.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.753+0000 7ff8dbfff640 1 -- 172.21.15.40:0/1616761528 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8e8012560 con 0x7ff8f41363c0 2024-04-22T08:16:23.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.753+0000 7ff8f99be640 10 monclient: _renew_subs 2024-04-22T08:16:23.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.753+0000 7ff8f99be640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:16:23.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.753+0000 7ff8f99be640 1 -- 172.21.15.40:0/1616761528 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8f409c2d0 con 0x7ff8f41363c0 2024-04-22T08:16:23.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.753+0000 7ff8dbfff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:16:23.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.753+0000 7ff8dbfff640 1 -- 172.21.15.40:0/1616761528 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e8012910 con 0x7ff8f41363c0 2024-04-22T08:16:23.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.753+0000 7ff8dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:23.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.753+0000 7ff8dbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:16:23.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.753+0000 7ff8dbfff640 10 monclient: dump: 2024-04-22T08:16:23.766 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:23.766 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:23.766 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:23.766 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:23.766 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:23.766 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:23.766 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:23.766 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:23.766 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:23.766 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:23.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.753+0000 7ff8f99be640 10 monclient: _renew_subs 2024-04-22T08:16:23.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.753+0000 7ff8f17fa640 4 set_mon_vals no callback set 2024-04-22T08:16:23.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.753+0000 7ff8f99be640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:16:23.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.753+0000 7ff8f99be640 1 -- 172.21.15.40:0/1616761528 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8f4123d10 con 0x7ff8f41363c0 2024-04-22T08:16:23.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.753+0000 7ff8f99be640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:16:23.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.753+0000 7ff8f99be640 1 --2- 172.21.15.40:0/1616761528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8f4122f00 0x7ff8f402b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:23.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.753+0000 7ff8f99be640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:23.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.753+0000 7ff8f99be640 1 -- 172.21.15.40:0/1616761528 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f4066770 con 0x7ff8f4122f00 2024-04-22T08:16:23.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.753+0000 7ff8dbfff640 1 -- 172.21.15.40:0/1616761528 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7ff8e8012b30 con 0x7ff8f41363c0 2024-04-22T08:16:23.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.753+0000 7ff8f2ffd640 1 --2- 172.21.15.40:0/1616761528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8f4122f00 0x7ff8f402b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:23.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.753+0000 7ff8f2ffd640 10 monclient: get_auth_request con 0x7ff8f4122f00 auth_method 0 2024-04-22T08:16:23.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.753+0000 7ff8f2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:23.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.753+0000 7ff8f2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:23.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.753+0000 7ff8f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:23.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.754+0000 7ff8f2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:23.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.754+0000 7ff8f2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:23.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.754+0000 7ff8f2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:23.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.754+0000 7ff8dbfff640 1 --2- 172.21.15.40:0/1616761528 >> v2:172.21.15.189:6800/30168 conn(0x7ff8e0037800 0x7ff8e0039cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:23.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.754+0000 7ff8dbfff640 1 -- 172.21.15.40:0/1616761528 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 3506+0+0 (secure 0 0 0) 0x7ff8e804a7d0 con 0x7ff8f41363c0 2024-04-22T08:16:23.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.754+0000 7ff8f2ffd640 10 monclient(hunting): handle_auth_done global_id 34563 payload 824 2024-04-22T08:16:23.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.754+0000 7ff8f2ffd640 1 -- 172.21.15.40:0/1616761528 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff8f4126250 con 0x7ff8f4122f00 2024-04-22T08:16:23.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.754+0000 7ff8f37fe640 1 --2- 172.21.15.40:0/1616761528 >> v2:172.21.15.189:6800/30168 conn(0x7ff8e0037800 0x7ff8e0039cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:23.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.754+0000 7ff8f37fe640 10 monclient: get_auth_request con 0x7ff8e0037800 auth_method 0 2024-04-22T08:16:23.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.756+0000 7ff8f2ffd640 1 -- 172.21.15.40:0/1616761528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8f4122f00 msgr2=0x7ff8f402b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:16:23.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.756+0000 7ff8f2ffd640 1 --2- 172.21.15.40:0/1616761528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8f4122f00 0x7ff8f402b200 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec000c00 tx=0x7ff8ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:23.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.758+0000 7ff8f37fe640 1 --2- 172.21.15.40:0/1616761528 >> v2:172.21.15.189:6800/30168 conn(0x7ff8e0037800 0x7ff8e0039cc0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff8f4125000 tx=0x7ff8e4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:23.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.759+0000 7ff8dbfff640 1 -- 172.21.15.40:0/1616761528 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ec032d80 con 0x7ff8f4122f00 2024-04-22T08:16:23.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.759+0000 7ff8dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:23.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.759+0000 7ff8dbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:16:23.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.759+0000 7ff8dbfff640 10 monclient: dump: 2024-04-22T08:16:23.771 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:23.771 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:23.771 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:23.771 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:23.771 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:23.771 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:23.771 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:23.771 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:23.772 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:23.772 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:23.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.759+0000 7ff8dbfff640 1 -- 172.21.15.40:0/1616761528 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7ff8ec03d440 con 0x7ff8f4122f00 2024-04-22T08:16:23.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.759+0000 7ff8dbfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:16:23.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.759+0000 7ff8dbfff640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:16:23.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.759+0000 7ff8dbfff640 1 -- 172.21.15.40:0/1616761528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8f4122f00 msgr2=0x7ff8f402b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:23.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.759+0000 7ff8dbfff640 1 --2- 172.21.15.40:0/1616761528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8f4122f00 0x7ff8f402b200 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec000c00 tx=0x7ff8ec0046c0 comp rx=0 tx=0).stop 2024-04-22T08:16:23.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.759+0000 7ff8dbfff640 1 -- 172.21.15.40:0/1616761528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8f4122f00 msgr2=0x7ff8f402b200 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:16:23.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.759+0000 7ff8dbfff640 1 --2- 172.21.15.40:0/1616761528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8f4122f00 0x7ff8f402b200 secure :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec000c00 tx=0x7ff8ec0046c0 comp rx=0 tx=0).stop 2024-04-22T08:16:23.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.765+0000 7ff8f99be640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:16:23.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.765+0000 7ff8f99be640 1 --2- 172.21.15.40:0/1616761528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8f41232e0 0x7ff8f402b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:23.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.765+0000 7ff8f99be640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:23.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.765+0000 7ff8f99be640 1 -- 172.21.15.40:0/1616761528 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f409dba0 con 0x7ff8f41232e0 2024-04-22T08:16:23.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.765+0000 7ff8f2ffd640 1 --2- 172.21.15.40:0/1616761528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8f41232e0 0x7ff8f402b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:23.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.765+0000 7ff8f2ffd640 10 monclient: get_auth_request con 0x7ff8f41232e0 auth_method 0 2024-04-22T08:16:23.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.765+0000 7ff8f2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:23.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.765+0000 7ff8f2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:23.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.765+0000 7ff8f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:23.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.765+0000 7ff8f2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:23.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.765+0000 7ff8f2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:23.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.765+0000 7ff8f2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:23.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.765+0000 7ff8f2ffd640 10 monclient(hunting): handle_auth_done global_id 34566 payload 824 2024-04-22T08:16:23.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.765+0000 7ff8f2ffd640 1 -- 172.21.15.40:0/1616761528 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff8f40a34f0 con 0x7ff8f41232e0 2024-04-22T08:16:23.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.767+0000 7ff8f2ffd640 1 -- 172.21.15.40:0/1616761528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8f41232e0 msgr2=0x7ff8f402b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:16:23.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.767+0000 7ff8f2ffd640 1 --2- 172.21.15.40:0/1616761528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8f41232e0 0x7ff8f402b740 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec02b750 tx=0x7ff8ec008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:23.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.768+0000 7ff8dbfff640 1 -- 172.21.15.40:0/1616761528 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ec003dd0 con 0x7ff8f41232e0 2024-04-22T08:16:23.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.768+0000 7ff8dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:23.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.768+0000 7ff8dbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:16:23.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.768+0000 7ff8dbfff640 10 monclient: dump: 2024-04-22T08:16:23.774 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:23.774 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:23.774 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:23.774 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:23.774 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:23.774 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:23.774 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:23.774 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:23.774 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:23.774 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:23.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.768+0000 7ff8dbfff640 1 -- 172.21.15.40:0/1616761528 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4633 (secure 0 0 0) 0x7ff8ec03d5b0 con 0x7ff8f41232e0 2024-04-22T08:16:23.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.768+0000 7ff8dbfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:16:23.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.768+0000 7ff8dbfff640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:16:23.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.768+0000 7ff8dbfff640 1 -- 172.21.15.40:0/1616761528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8f41232e0 msgr2=0x7ff8f402b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:23.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.768+0000 7ff8dbfff640 1 --2- 172.21.15.40:0/1616761528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8f41232e0 0x7ff8f402b740 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec02b750 tx=0x7ff8ec008030 comp rx=0 tx=0).stop 2024-04-22T08:16:23.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.768+0000 7ff8dbfff640 1 -- 172.21.15.40:0/1616761528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8f41232e0 msgr2=0x7ff8f402b740 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:16:23.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.768+0000 7ff8dbfff640 1 --2- 172.21.15.40:0/1616761528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8f41232e0 0x7ff8f402b740 secure :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec02b750 tx=0x7ff8ec008030 comp rx=0 tx=0).stop 2024-04-22T08:16:23.775 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:16:23.775 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:16:23.776 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:16:23.776 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "leader", 2024-04-22T08:16:23.776 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 42, 2024-04-22T08:16:23.776 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:16:23.776 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:16:23.776 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:16:23.776 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:16:23.776 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:16:23.776 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 2, 2024-04-22T08:16:23.776 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:16:23.776 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:16:23.776 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:16:23.776 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:16:23.777 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:16:23.777 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:16:23.777 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:16:23.777 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:16:23.777 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:16:23.777 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:16:23.777 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:16:23.777 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:16:23.777 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:16:23.777 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:16:23.777 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:16:23.777 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:16:23.777 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:16:23.777 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:16:23.778 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:16:23.778 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:16:23.778 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:16:23.778 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:16:23.778 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:16:23.778 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:16:23.778 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:16:23.778 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:16:23.778 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:23.778 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:23.778 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:16:23.778 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:16:23.778 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:16:23.778 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:16:23.779 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:16:23.779 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:16:23.779 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:16:23.779 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:16:23.779 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:16:23.779 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:16:23.779 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:16:23.779 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:16:23.779 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:16:23.779 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:16:23.779 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:16:23.779 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:16:23.779 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:16:23.779 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:16:23.779 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:16:23.780 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:16:23.780 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:16:23.780 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:16:23.780 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:16:23.780 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:16:23.780 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:16:23.780 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:16:23.780 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:16:23.780 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:16:23.780 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:16:23.780 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:23.780 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:16:23.780 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:23.780 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:16:23.780 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:16:23.781 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:16:23.781 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:16:23.781 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:23.781 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:16:23.781 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:16:23.781 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:23.781 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:23.781 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:23.781 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:16:23.781 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:16:23.781 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:23.781 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:23.781 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:23.781 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:23.782 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:16:23.782 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:16:23.782 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:16:23.782 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:16:23.782 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:16:23.782 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:23.782 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:23.782 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:16:23.782 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:16:23.782 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:16:23.782 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:16:23.782 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:23.782 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:16:23.782 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:16:23.782 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:23.783 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:23.783 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:23.783 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:16:23.783 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:16:23.783 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:23.783 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:23.783 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:23.783 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:23.783 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:16:23.783 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:16:23.783 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:16:23.783 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:16:23.783 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:16:23.783 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:23.783 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:23.784 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:16:23.784 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:16:23.784 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:16:23.784 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:16:23.784 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:23.784 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:16:23.784 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:16:23.784 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:23.784 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:23.784 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:23.784 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:16:23.784 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:16:23.784 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:23.784 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:23.784 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:23.785 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:23.785 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:16:23.785 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:16:23.785 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:16:23.785 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:16:23.785 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:16:23.785 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:23.785 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:23.785 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:23.785 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:16:23.785 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:16:23.785 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:23.785 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:16:23.785 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:16:23.785 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:16:23.786 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:23.786 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:16:23.786 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:16:23.786 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:23.786 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:16:23.786 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:16:23.786 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 3 2024-04-22T08:16:23.786 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:23.786 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:16:23.786 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:16:23.787 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:23.787 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:16:23.787 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:16:23.787 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:16:23.787 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:23.787 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:23.787 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:23.787 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:16:23.787 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:16:23.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.768+0000 7ff8f99be640 1 -- 172.21.15.40:0/1616761528 >> v2:172.21.15.189:6800/30168 conn(0x7ff8e0037800 msgr2=0x7ff8e0039cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:23.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.768+0000 7ff8f99be640 1 --2- 172.21.15.40:0/1616761528 >> v2:172.21.15.189:6800/30168 conn(0x7ff8e0037800 0x7ff8e0039cc0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff8f4125000 tx=0x7ff8e4009040 comp rx=0 tx=0).stop 2024-04-22T08:16:23.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.768+0000 7ff8f99be640 10 monclient: shutdown 2024-04-22T08:16:23.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.768+0000 7ff8f99be640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:16:23.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.768+0000 7ff8f99be640 1 -- 172.21.15.40:0/1616761528 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8f41363c0 msgr2=0x7ff8f41cb1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:23.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.768+0000 7ff8f99be640 1 --2- 172.21.15.40:0/1616761528 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8f41363c0 0x7ff8f41cb1e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff8e800cb20 tx=0x7ff8e800a060 comp rx=0 tx=0).stop 2024-04-22T08:16:23.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.768+0000 7ff8f37fe640 1 -- 172.21.15.40:0/1616761528 reap_dead start 2024-04-22T08:16:23.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.768+0000 7ff8f37fe640 1 -- 172.21.15.40:0/1616761528 reap_dead start 2024-04-22T08:16:23.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.768+0000 7ff8f99be640 1 -- 172.21.15.40:0/1616761528 shutdown_connections 2024-04-22T08:16:23.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.768+0000 7ff8f99be640 1 -- 172.21.15.40:0/1616761528 >> 172.21.15.40:0/1616761528 conn(0x7ff8f40a66a0 msgr2=0x7ff8f409c510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:16:23.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.768+0000 7ff8f99be640 1 -- 172.21.15.40:0/1616761528 shutdown_connections 2024-04-22T08:16:23.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.768+0000 7ff8f99be640 1 -- 172.21.15.40:0/1616761528 wait complete. 2024-04-22T08:16:23.789 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:16:23.790 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-22T08:16:23.790 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-22T08:16:23.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.890+0000 7f7a8b9be640 10 monclient: get_monmap_and_config 2024-04-22T08:16:23.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.890+0000 7f7a8b9be640 10 monclient: build_initial_monmap 2024-04-22T08:16:23.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.890+0000 7f7a8b9be640 10 monclient: monmap: 2024-04-22T08:16:23.890 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:16:23.890 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:23.890 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:16:23.891130+0000 2024-04-22T08:16:23.890 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:16:23.891130+0000 2024-04-22T08:16:23.891 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:16:23.891 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:23.891 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:16:23.891 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:16:23.891 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:16:23.891 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:16:23.891 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:16:23.891 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:23.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.890+0000 7f7a8b9be640 1 Processor -- start 2024-04-22T08:16:23.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.890+0000 7f7a8b9be640 1 -- start start 2024-04-22T08:16:23.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.890+0000 7f7a8b9be640 10 monclient: init 2024-04-22T08:16:23.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.890+0000 7f7a8b9be640 10 monclient: _reopen_session rank -1 2024-04-22T08:16:23.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.890+0000 7f7a8b9be640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-04-22T08:16:23.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.890+0000 7f7a8b9be640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7a84135fe0 addr v1:172.21.15.40:6789/0 2024-04-22T08:16:23.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.890+0000 7f7a8b9be640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7a840a01c0 0x7f7a8413ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:23.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.890+0000 7f7a8b9be640 10 monclient(hunting): picked mon.noname-a con 0x7f7a840a01c0 addr v2:172.21.15.40:3300/0 2024-04-22T08:16:23.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.890+0000 7f7a8b9be640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7a8413b3a0 addr v1:172.21.15.189:6789/0 2024-04-22T08:16:23.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.890+0000 7f7a8b9be640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a840a05d0 con 0x7f7a84135fe0 2024-04-22T08:16:23.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.890+0000 7f7a8b9be640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a8409d7f0 con 0x7f7a8413b3a0 2024-04-22T08:16:23.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.890+0000 7f7a8b9be640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:23.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.890+0000 7f7a8b9be640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a8409f330 con 0x7f7a840a01c0 2024-04-22T08:16:23.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.890+0000 7f7a8b9be640 10 monclient(hunting): _renew_subs 2024-04-22T08:16:23.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.890+0000 7f7a8b9be640 10 monclient(hunting): authenticate will time out at 1057.885417s 2024-04-22T08:16:23.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.890+0000 7f7a88f59640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7a840a01c0 0x7f7a8413ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:23.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.890+0000 7f7a88f59640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7a840a01c0 0x7f7a8413ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41564/0 (socket says 172.21.15.40:41564) 2024-04-22T08:16:23.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.890+0000 7f7a8975a640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f7a84135fe0 0x7f7a841363c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57450/0 (socket says 172.21.15.40:57450) 2024-04-22T08:16:23.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.890+0000 7f7a88f59640 1 -- 172.21.15.40:0/2894011068 learned_addr learned my addr 172.21.15.40:0/2894011068 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:16:23.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.890+0000 7f7a88f59640 10 monclient(hunting): get_auth_request con 0x7f7a840a01c0 auth_method 0 2024-04-22T08:16:23.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.890+0000 7f7a88f59640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:23.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.890+0000 7f7a88f59640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:23.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.890+0000 7f7a88f59640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:23.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.890+0000 7f7a88f59640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:23.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.890+0000 7f7a88f59640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:23.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.890+0000 7f7a88f59640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:23.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.891+0000 7f7a88f59640 10 monclient(hunting): handle_auth_done global_id 34569 payload 274 2024-04-22T08:16:23.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.891+0000 7f7a88f59640 1 -- 172.21.15.40:0/2894011068 >> v1:172.21.15.189:6789/0 conn(0x7f7a8413b3a0 legacy=0x7f7a8413d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:16:23.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.891+0000 7f7a88f59640 1 -- 172.21.15.40:0/2894011068 >> v1:172.21.15.40:6789/0 conn(0x7f7a84135fe0 legacy=0x7f7a841363c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:16:23.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.891+0000 7f7a88f59640 10 monclient: _finish_hunting 0 2024-04-22T08:16:23.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.891+0000 7f7a88f59640 1 monclient: found mon.noname-a 2024-04-22T08:16:23.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.891+0000 7f7a88f59640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:16:23.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.891+0000 7f7a88f59640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:16:23.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.891+0000 7f7a88f59640 1 -- 172.21.15.40:0/2894011068 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a840999f0 con 0x7f7a840a01c0 2024-04-22T08:16:23.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.893+0000 7f7a88f59640 1 -- 172.21.15.40:0/2894011068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a840a01c0 msgr2=0x7f7a8413ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:16:23.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.893+0000 7f7a88f59640 1 --2- 172.21.15.40:0/2894011068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a840a01c0 0x7f7a8413ae60 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7a74009870 tx=0x7f7a7402b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4d9b453ee38871a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:23.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.894+0000 7f7a7bfff640 1 -- 172.21.15.40:0/2894011068 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a74031a40 con 0x7f7a840a01c0 2024-04-22T08:16:23.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.894+0000 7f7a7bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:23.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.894+0000 7f7a7bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:16:23.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.894+0000 7f7a7bfff640 10 monclient: dump: 2024-04-22T08:16:23.902 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:23.903 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:23.903 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:23.903 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:23.903 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:23.903 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:23.903 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:23.903 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:23.903 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:23.903 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:23.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.894+0000 7f7a7bfff640 10 monclient: _finish_auth 0 2024-04-22T08:16:23.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.894+0000 7f7a7bfff640 10 monclient: _check_auth_tickets 2024-04-22T08:16:23.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.894+0000 7f7a7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:16:23.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.894+0000 7f7a8b9be640 5 monclient: authenticate success, global_id 34569 2024-04-22T08:16:23.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.894+0000 7f7a8b9be640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:16:23.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.894+0000 7f7a7bfff640 1 -- 172.21.15.40:0/2894011068 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a74031ba0 con 0x7f7a840a01c0 2024-04-22T08:16:23.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.894+0000 7f7a7bfff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:16:23.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.894+0000 7f7a8b9be640 10 monclient: get_monmap_and_config success 2024-04-22T08:16:23.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.894+0000 7f7a7bfff640 1 -- 172.21.15.40:0/2894011068 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a74033440 con 0x7f7a840a01c0 2024-04-22T08:16:23.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.894+0000 7f7a7bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:23.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.894+0000 7f7a8b9be640 4 set_mon_vals no callback set 2024-04-22T08:16:23.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.894+0000 7f7a7bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:16:23.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.894+0000 7f7a7bfff640 10 monclient: dump: 2024-04-22T08:16:23.904 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:23.904 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:23.904 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:23.904 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:23.904 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:23.905 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:23.905 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:23.905 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:23.905 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:23.905 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:23.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.894+0000 7f7a8b9be640 10 monclient: shutdown 2024-04-22T08:16:23.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.894+0000 7f7a8b9be640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:16:23.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.894+0000 7f7a8b9be640 1 -- 172.21.15.40:0/2894011068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a840a01c0 msgr2=0x7f7a8413ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:23.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.894+0000 7f7a8b9be640 1 --2- 172.21.15.40:0/2894011068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a840a01c0 0x7f7a8413ae60 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7a74009870 tx=0x7f7a7402b7f0 comp rx=0 tx=0).stop 2024-04-22T08:16:23.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.894+0000 7f7a8b9be640 1 -- 172.21.15.40:0/2894011068 shutdown_connections 2024-04-22T08:16:23.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.894+0000 7f7a8b9be640 1 --2- 172.21.15.40:0/2894011068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a840a01c0 0x7f7a8413ae60 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:23.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.894+0000 7f7a8b9be640 1 -- 172.21.15.40:0/2894011068 >> 172.21.15.40:0/2894011068 conn(0x7f7a840a66a0 msgr2=0x7f7a840a2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:16:23.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.894+0000 7f7a8b9be640 1 -- 172.21.15.40:0/2894011068 shutdown_connections 2024-04-22T08:16:23.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.894+0000 7f7a8b9be640 1 -- 172.21.15.40:0/2894011068 wait complete. 2024-04-22T08:16:23.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.895+0000 7f7a8b9be640 10 monclient: build_initial_monmap 2024-04-22T08:16:23.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.895+0000 7f7a8b9be640 10 monclient: monmap: 2024-04-22T08:16:23.906 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:16:23.906 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:16:23.906 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:16:23.906 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:16:23.906 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:16:23.906 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:23.906 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:16:23.906 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:16:23.906 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:16:23.906 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:23.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.895+0000 7f7a8b9be640 1 Processor -- start 2024-04-22T08:16:23.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.895+0000 7f7a8b9be640 1 -- start start 2024-04-22T08:16:23.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.895+0000 7f7a8b9be640 10 monclient: init 2024-04-22T08:16:23.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.895+0000 7f7a8b9be640 10 monclient: _reopen_session rank -1 2024-04-22T08:16:23.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.895+0000 7f7a8b9be640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-22T08:16:23.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.895+0000 7f7a8b9be640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a840a01c0 0x7f7a841ca7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:23.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.895+0000 7f7a8b9be640 10 monclient(hunting): picked mon.noname-b con 0x7f7a840a01c0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:16:23.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.896+0000 7f7a8b9be640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a84135fe0 0x7f7a84134c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:23.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.896+0000 7f7a8b9be640 10 monclient(hunting): picked mon.noname-c con 0x7f7a84135fe0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:16:23.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.896+0000 7f7a8b9be640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a8413b3a0 0x7f7a841351c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:23.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.896+0000 7f7a8b9be640 10 monclient(hunting): picked mon.noname-a con 0x7f7a8413b3a0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:16:23.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.896+0000 7f7a8b9be640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:23.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.896+0000 7f7a8b9be640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8409cce0 con 0x7f7a8413b3a0 2024-04-22T08:16:23.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.896+0000 7f7a8b9be640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:23.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.896+0000 7f7a8b9be640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a840a51b0 con 0x7f7a840a01c0 2024-04-22T08:16:23.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.896+0000 7f7a8b9be640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:23.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.896+0000 7f7a8b9be640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a84071060 con 0x7f7a84135fe0 2024-04-22T08:16:23.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.896+0000 7f7a8b9be640 10 monclient(hunting): _renew_subs 2024-04-22T08:16:23.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.896+0000 7f7a8b9be640 10 monclient(hunting): authenticate will time out at 1057.890823s 2024-04-22T08:16:23.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.896+0000 7f7a89f5b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a8413b3a0 0x7f7a841351c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:23.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.896+0000 7f7a88f59640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a84135fe0 0x7f7a84134c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:23.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.896+0000 7f7a88f59640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a84135fe0 0x7f7a84134c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:33152/0 (socket says 172.21.15.40:33152) 2024-04-22T08:16:23.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.896+0000 7f7a89f5b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a8413b3a0 0x7f7a841351c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41572/0 (socket says 172.21.15.40:41572) 2024-04-22T08:16:23.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.896+0000 7f7a88f59640 1 -- 172.21.15.40:0/4211047008 learned_addr learned my addr 172.21.15.40:0/4211047008 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:16:23.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.896+0000 7f7a88f59640 10 monclient(hunting): get_auth_request con 0x7f7a84135fe0 auth_method 0 2024-04-22T08:16:23.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.896+0000 7f7a88f59640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:23.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.896+0000 7f7a88f59640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:23.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.896+0000 7f7a88f59640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:23.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.896+0000 7f7a89f5b640 10 monclient(hunting): get_auth_request con 0x7f7a8413b3a0 auth_method 0 2024-04-22T08:16:23.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.896+0000 7f7a89f5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:23.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.896+0000 7f7a89f5b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:23.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.896+0000 7f7a89f5b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:23.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.896+0000 7f7a88f59640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:23.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.896+0000 7f7a88f59640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:23.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.896+0000 7f7a88f59640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:23.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.896+0000 7f7a89f5b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:23.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.896+0000 7f7a89f5b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:23.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.896+0000 7f7a89f5b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:23.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.896+0000 7f7a8975a640 1 --2- 172.21.15.40:0/4211047008 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a840a01c0 0x7f7a841ca7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:23.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.896+0000 7f7a89f5b640 10 monclient(hunting): handle_auth_done global_id 34572 payload 824 2024-04-22T08:16:23.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.896+0000 7f7a89f5b640 1 -- 172.21.15.40:0/4211047008 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a84135fe0 msgr2=0x7f7a84134c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:23.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.896+0000 7f7a89f5b640 1 --2- 172.21.15.40:0/4211047008 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a84135fe0 0x7f7a84134c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:23.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.896+0000 7f7a89f5b640 1 -- 172.21.15.40:0/4211047008 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a840a01c0 msgr2=0x7f7a841ca7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:23.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.896+0000 7f7a89f5b640 1 --2- 172.21.15.40:0/4211047008 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a840a01c0 0x7f7a841ca7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:23.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.896+0000 7f7a89f5b640 10 monclient: _finish_hunting 0 2024-04-22T08:16:23.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.896+0000 7f7a89f5b640 1 monclient: found mon.noname-a 2024-04-22T08:16:23.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.896+0000 7f7a89f5b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:16:23.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.896+0000 7f7a89f5b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:16:23.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.896+0000 7f7a89f5b640 1 -- 172.21.15.40:0/4211047008 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a740094d0 con 0x7f7a8413b3a0 2024-04-22T08:16:23.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.896+0000 7f7a8975a640 10 monclient: get_auth_request con 0x7f7a840a01c0 auth_method 0 2024-04-22T08:16:23.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.896+0000 7f7a88f59640 1 --2- 172.21.15.40:0/4211047008 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a84135fe0 0x7f7a84134c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:16:23.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.896+0000 7f7a8975a640 1 --2- 172.21.15.40:0/4211047008 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a840a01c0 0x7f7a841ca7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T08:16:23.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.898+0000 7f7a89f5b640 1 -- 172.21.15.40:0/4211047008 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a8413b3a0 msgr2=0x7f7a841351c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:16:23.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.898+0000 7f7a89f5b640 1 --2- 172.21.15.40:0/4211047008 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a8413b3a0 0x7f7a841351c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c00c780 tx=0x7f7a7c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:23.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.898+0000 7f7a79ffb640 1 -- 172.21.15.40:0/4211047008 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a7c00ac60 con 0x7f7a8413b3a0 2024-04-22T08:16:23.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.898+0000 7f7a79ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:23.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.898+0000 7f7a79ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:16:23.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.898+0000 7f7a79ffb640 10 monclient: dump: 2024-04-22T08:16:23.911 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:23.911 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:23.911 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:23.911 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:23.911 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:23.911 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:23.911 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:23.912 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:23.912 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:23.912 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:23.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.898+0000 7f7a79ffb640 10 monclient: _finish_auth 0 2024-04-22T08:16:23.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.898+0000 7f7a79ffb640 10 monclient: _check_auth_tickets 2024-04-22T08:16:23.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.898+0000 7f7a79ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:16:23.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.898+0000 7f7a79ffb640 1 -- 172.21.15.40:0/4211047008 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a7c00adc0 con 0x7f7a8413b3a0 2024-04-22T08:16:23.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.898+0000 7f7a8b9be640 5 monclient: authenticate success, global_id 34572 2024-04-22T08:16:23.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.898+0000 7f7a8b9be640 10 monclient: _renew_subs 2024-04-22T08:16:23.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.899+0000 7f7a8b9be640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:16:23.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.899+0000 7f7a8b9be640 1 -- 172.21.15.40:0/4211047008 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a84120ad0 con 0x7f7a8413b3a0 2024-04-22T08:16:23.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.899+0000 7f7a79ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:16:23.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.899+0000 7f7a79ffb640 1 -- 172.21.15.40:0/4211047008 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a7c012660 con 0x7f7a8413b3a0 2024-04-22T08:16:23.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.899+0000 7f7a79ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:23.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.899+0000 7f7a79ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:16:23.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.899+0000 7f7a79ffb640 10 monclient: dump: 2024-04-22T08:16:23.913 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:23.913 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:23.913 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:23.913 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:23.913 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:23.913 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:23.913 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:23.913 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:23.913 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:23.913 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:23.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.899+0000 7f7a7b7fe640 4 set_mon_vals no callback set 2024-04-22T08:16:23.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.899+0000 7f7a8b9be640 10 monclient: _renew_subs 2024-04-22T08:16:23.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.899+0000 7f7a8b9be640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:16:23.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.899+0000 7f7a8b9be640 1 -- 172.21.15.40:0/4211047008 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a84070490 con 0x7f7a8413b3a0 2024-04-22T08:16:23.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.900+0000 7f7a8b9be640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:16:23.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.900+0000 7f7a8b9be640 1 --2- 172.21.15.40:0/4211047008 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a8406e810 0x7f7a8406ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:23.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.900+0000 7f7a8b9be640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:23.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.900+0000 7f7a8b9be640 1 -- 172.21.15.40:0/4211047008 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a84120e50 con 0x7f7a8406e810 2024-04-22T08:16:23.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.900+0000 7f7a79ffb640 1 -- 172.21.15.40:0/4211047008 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f7a7c012880 con 0x7f7a8413b3a0 2024-04-22T08:16:23.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.900+0000 7f7a8975a640 1 --2- 172.21.15.40:0/4211047008 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a8406e810 0x7f7a8406ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:23.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.900+0000 7f7a8975a640 10 monclient: get_auth_request con 0x7f7a8406e810 auth_method 0 2024-04-22T08:16:23.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.900+0000 7f7a8975a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:23.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.900+0000 7f7a8975a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:23.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.900+0000 7f7a8975a640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:23.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.900+0000 7f7a8975a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:23.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.900+0000 7f7a8975a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:23.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.900+0000 7f7a8975a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:23.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.900+0000 7f7a79ffb640 1 --2- 172.21.15.40:0/4211047008 >> v2:172.21.15.189:6800/30168 conn(0x7f7a68037820 0x7f7a68039ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:23.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.900+0000 7f7a8975a640 10 monclient(hunting): handle_auth_done global_id 54131 payload 824 2024-04-22T08:16:23.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.900+0000 7f7a79ffb640 1 -- 172.21.15.40:0/4211047008 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f7a7c04a2c0 con 0x7f7a8413b3a0 2024-04-22T08:16:23.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.900+0000 7f7a8975a640 1 -- 172.21.15.40:0/4211047008 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7a8409dba0 con 0x7f7a8406e810 2024-04-22T08:16:23.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.900+0000 7f7a88f59640 1 --2- 172.21.15.40:0/4211047008 >> v2:172.21.15.189:6800/30168 conn(0x7f7a68037820 0x7f7a68039ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:23.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.900+0000 7f7a88f59640 10 monclient: get_auth_request con 0x7f7a68037820 auth_method 0 2024-04-22T08:16:23.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.902+0000 7f7a8975a640 1 -- 172.21.15.40:0/4211047008 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a8406e810 msgr2=0x7f7a8406ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:16:23.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.902+0000 7f7a8975a640 1 --2- 172.21.15.40:0/4211047008 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a8406e810 0x7f7a8406ebf0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7a8000d4c0 tx=0x7f7a8000d980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:23.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.902+0000 7f7a79ffb640 1 -- 172.21.15.40:0/4211047008 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a80005e30 con 0x7f7a8406e810 2024-04-22T08:16:23.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.902+0000 7f7a79ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:23.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.902+0000 7f7a79ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:16:23.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.902+0000 7f7a79ffb640 10 monclient: dump: 2024-04-22T08:16:23.918 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:23.918 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:23.918 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:23.918 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:23.918 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:23.918 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:23.918 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:23.918 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:23.919 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:23.919 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:23.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.903+0000 7f7a79ffb640 1 -- 172.21.15.40:0/4211047008 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f7a80007070 con 0x7f7a8406e810 2024-04-22T08:16:23.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.903+0000 7f7a79ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:16:23.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.903+0000 7f7a79ffb640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:16:23.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.903+0000 7f7a79ffb640 1 -- 172.21.15.40:0/4211047008 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a8406e810 msgr2=0x7f7a8406ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:23.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.903+0000 7f7a79ffb640 1 --2- 172.21.15.40:0/4211047008 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a8406e810 0x7f7a8406ebf0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7a8000d4c0 tx=0x7f7a8000d980 comp rx=0 tx=0).stop 2024-04-22T08:16:23.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.903+0000 7f7a79ffb640 1 -- 172.21.15.40:0/4211047008 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a8406e810 msgr2=0x7f7a8406ebf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:16:23.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.903+0000 7f7a79ffb640 1 --2- 172.21.15.40:0/4211047008 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a8406e810 0x7f7a8406ebf0 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7a8000d4c0 tx=0x7f7a8000d980 comp rx=0 tx=0).stop 2024-04-22T08:16:23.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.903+0000 7f7a88f59640 1 --2- 172.21.15.40:0/4211047008 >> v2:172.21.15.189:6800/30168 conn(0x7f7a68037820 0x7f7a68039ce0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7a74000c00 tx=0x7f7a74008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:23.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.910+0000 7f7a8b9be640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:16:23.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.910+0000 7f7a8b9be640 1 --2- 172.21.15.40:0/4211047008 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a8406f130 0x7f7a84138bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:23.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.910+0000 7f7a8b9be640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:23.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.910+0000 7f7a8b9be640 1 -- 172.21.15.40:0/4211047008 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a84066770 con 0x7f7a8406f130 2024-04-22T08:16:23.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.910+0000 7f7a8975a640 1 --2- 172.21.15.40:0/4211047008 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a8406f130 0x7f7a84138bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:23.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.910+0000 7f7a8975a640 10 monclient: get_auth_request con 0x7f7a8406f130 auth_method 0 2024-04-22T08:16:23.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.910+0000 7f7a8975a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:23.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.910+0000 7f7a8975a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:23.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.910+0000 7f7a8975a640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:23.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.910+0000 7f7a8975a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:23.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.910+0000 7f7a8975a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:23.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.910+0000 7f7a8975a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:23.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.910+0000 7f7a8975a640 10 monclient(hunting): handle_auth_done global_id 54134 payload 824 2024-04-22T08:16:23.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.910+0000 7f7a8975a640 1 -- 172.21.15.40:0/4211047008 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7a840a34f0 con 0x7f7a8406f130 2024-04-22T08:16:23.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.913+0000 7f7a8975a640 1 -- 172.21.15.40:0/4211047008 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a8406f130 msgr2=0x7f7a84138bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:16:23.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.913+0000 7f7a8975a640 1 --2- 172.21.15.40:0/4211047008 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a8406f130 0x7f7a84138bb0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7a8000d4c0 tx=0x7f7a80004930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:23.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.913+0000 7f7a79ffb640 1 -- 172.21.15.40:0/4211047008 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a80022070 con 0x7f7a8406f130 2024-04-22T08:16:23.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.913+0000 7f7a79ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:23.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.913+0000 7f7a79ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:16:23.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.913+0000 7f7a79ffb640 10 monclient: dump: 2024-04-22T08:16:23.921 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:23.921 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:23.921 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:23.921 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:23.921 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:23.921 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:23.921 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:23.921 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:23.921 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:23.921 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:23.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.913+0000 7f7a79ffb640 1 -- 172.21.15.40:0/4211047008 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4631 (secure 0 0 0) 0x7f7a80012410 con 0x7f7a8406f130 2024-04-22T08:16:23.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.913+0000 7f7a79ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:16:23.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.913+0000 7f7a79ffb640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:16:23.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.913+0000 7f7a79ffb640 1 -- 172.21.15.40:0/4211047008 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a8406f130 msgr2=0x7f7a84138bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:23.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.913+0000 7f7a79ffb640 1 --2- 172.21.15.40:0/4211047008 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a8406f130 0x7f7a84138bb0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7a8000d4c0 tx=0x7f7a80004930 comp rx=0 tx=0).stop 2024-04-22T08:16:23.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.913+0000 7f7a79ffb640 1 -- 172.21.15.40:0/4211047008 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a8406f130 msgr2=0x7f7a84138bb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:16:23.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.913+0000 7f7a79ffb640 1 --2- 172.21.15.40:0/4211047008 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a8406f130 0x7f7a84138bb0 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7a8000d4c0 tx=0x7f7a80004930 comp rx=0 tx=0).stop 2024-04-22T08:16:23.922 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:16:23.922 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:16:23.922 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:16:23.922 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:16:23.922 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 42, 2024-04-22T08:16:23.922 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:16:23.922 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:16:23.923 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:16:23.923 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:16:23.923 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:16:23.923 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 2, 2024-04-22T08:16:23.923 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:16:23.923 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:16:23.923 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:16:23.923 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:16:23.923 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:16:23.923 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:16:23.923 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:16:23.923 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:16:23.923 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:16:23.923 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:16:23.924 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:16:23.924 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:16:23.924 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:16:23.924 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:16:23.924 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:16:23.924 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:16:23.924 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:16:23.924 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:16:23.924 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:16:23.924 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:16:23.924 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:16:23.924 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:16:23.924 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:16:23.925 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:16:23.925 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:16:23.925 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:16:23.925 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:23.925 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:23.925 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:16:23.925 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:16:23.925 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:16:23.925 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:16:23.925 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:16:23.925 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:16:23.925 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:16:23.925 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:16:23.925 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:16:23.926 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:16:23.926 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:16:23.926 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:16:23.926 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:16:23.926 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:16:23.926 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:16:23.926 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:16:23.926 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:16:23.926 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:16:23.926 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:16:23.926 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:16:23.926 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:16:23.926 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:16:23.926 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:16:23.927 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:16:23.927 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:16:23.927 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:16:23.927 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:16:23.927 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:16:23.927 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:16:23.927 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:23.927 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:16:23.927 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:23.927 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:16:23.927 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:16:23.927 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:16:23.927 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:16:23.927 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:23.927 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:16:23.928 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:16:23.928 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:23.928 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:23.928 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:23.928 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:16:23.928 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:16:23.928 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:23.928 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:23.928 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:23.928 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:23.928 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:16:23.928 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:16:23.928 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:16:23.928 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:16:23.929 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:16:23.929 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:23.929 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:23.929 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:16:23.929 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:16:23.929 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:16:23.929 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:16:23.929 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:23.929 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:16:23.929 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:16:23.929 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:23.929 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:23.929 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:23.929 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:16:23.930 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:16:23.930 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:23.930 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:23.930 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:23.930 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:23.930 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:16:23.930 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:16:23.930 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:16:23.930 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:16:23.930 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:16:23.930 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:23.930 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:23.930 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:16:23.931 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:16:23.931 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:16:23.931 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:16:23.931 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:23.931 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:16:23.931 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:16:23.931 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:23.931 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:23.931 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:23.931 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:16:23.931 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:16:23.931 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:23.931 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:23.931 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:23.932 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:23.932 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:16:23.932 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:16:23.932 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:16:23.932 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:16:23.932 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:16:23.932 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:23.932 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:23.932 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:23.932 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:16:23.932 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:16:23.932 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:23.933 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:16:23.933 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:16:23.933 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:16:23.933 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:23.933 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:16:23.933 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:16:23.933 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:23.933 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:16:23.933 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:16:23.933 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:16:23.933 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:23.933 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:16:23.933 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:16:23.933 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:23.933 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:16:23.934 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:16:23.934 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 2 2024-04-22T08:16:23.934 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:23.934 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:23.934 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:23.934 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:16:23.934 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:16:23.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.913+0000 7f7a8b9be640 1 -- 172.21.15.40:0/4211047008 >> v2:172.21.15.189:6800/30168 conn(0x7f7a68037820 msgr2=0x7f7a68039ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:23.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.913+0000 7f7a8b9be640 1 --2- 172.21.15.40:0/4211047008 >> v2:172.21.15.189:6800/30168 conn(0x7f7a68037820 0x7f7a68039ce0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7a74000c00 tx=0x7f7a74008030 comp rx=0 tx=0).stop 2024-04-22T08:16:23.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.913+0000 7f7a8b9be640 10 monclient: shutdown 2024-04-22T08:16:23.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.913+0000 7f7a8b9be640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:16:23.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.913+0000 7f7a8b9be640 1 -- 172.21.15.40:0/4211047008 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a8413b3a0 msgr2=0x7f7a841351c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:23.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.913+0000 7f7a8b9be640 1 --2- 172.21.15.40:0/4211047008 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a8413b3a0 0x7f7a841351c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c00c780 tx=0x7f7a7c00cc40 comp rx=0 tx=0).stop 2024-04-22T08:16:23.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.913+0000 7f7a89f5b640 1 -- 172.21.15.40:0/4211047008 reap_dead start 2024-04-22T08:16:23.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.913+0000 7f7a89f5b640 1 -- 172.21.15.40:0/4211047008 reap_dead start 2024-04-22T08:16:23.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.913+0000 7f7a8b9be640 1 -- 172.21.15.40:0/4211047008 shutdown_connections 2024-04-22T08:16:23.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.913+0000 7f7a8b9be640 1 -- 172.21.15.40:0/4211047008 >> 172.21.15.40:0/4211047008 conn(0x7f7a840a66a0 msgr2=0x7f7a840a3f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:16:23.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.913+0000 7f7a8b9be640 1 -- 172.21.15.40:0/4211047008 shutdown_connections 2024-04-22T08:16:23.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:23.913+0000 7f7a8b9be640 1 -- 172.21.15.40:0/4211047008 wait complete. 2024-04-22T08:16:23.936 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:16:23.937 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-22T08:16:23.937 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-22T08:16:24.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.037+0000 7f8481ea1640 10 monclient: get_monmap_and_config 2024-04-22T08:16:24.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.037+0000 7f8481ea1640 10 monclient: build_initial_monmap 2024-04-22T08:16:24.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.037+0000 7f8481ea1640 10 monclient: monmap: 2024-04-22T08:16:24.040 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:16:24.040 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:24.040 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:16:24.038330+0000 2024-04-22T08:16:24.040 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:16:24.038330+0000 2024-04-22T08:16:24.040 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:16:24.040 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:24.040 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:16:24.040 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:16:24.041 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:16:24.041 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:16:24.041 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:16:24.041 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:24.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.037+0000 7f8481ea1640 1 Processor -- start 2024-04-22T08:16:24.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.037+0000 7f8481ea1640 1 -- start start 2024-04-22T08:16:24.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.037+0000 7f8481ea1640 10 monclient: init 2024-04-22T08:16:24.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.037+0000 7f8481ea1640 10 monclient: _reopen_session rank -1 2024-04-22T08:16:24.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.037+0000 7f8481ea1640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-04-22T08:16:24.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.037+0000 7f8481ea1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f847c0a1a80 addr v1:172.21.15.40:6789/0 2024-04-22T08:16:24.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.037+0000 7f8481ea1640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f847c0a2570 0x7f847c0a01b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:24.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.037+0000 7f8481ea1640 10 monclient(hunting): picked mon.noname-b con 0x7f847c0a2570 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:16:24.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.037+0000 7f8481ea1640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f847c13b0e0 0x7f847c13d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:24.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.037+0000 7f8481ea1640 10 monclient(hunting): picked mon.noname-c con 0x7f847c13b0e0 addr v2:172.21.15.189:3300/0 2024-04-22T08:16:24.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.037+0000 7f8481ea1640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f847c09d7f0 con 0x7f847c0a1a80 2024-04-22T08:16:24.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.037+0000 7f8481ea1640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:24.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.037+0000 7f8481ea1640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f847c09f1c0 con 0x7f847c13b0e0 2024-04-22T08:16:24.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.037+0000 7f8481ea1640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:24.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.037+0000 7f8481ea1640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f847c09dc30 con 0x7f847c0a2570 2024-04-22T08:16:24.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.037+0000 7f8481ea1640 10 monclient(hunting): _renew_subs 2024-04-22T08:16:24.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.037+0000 7f8481ea1640 10 monclient(hunting): authenticate will time out at 1058.032571s 2024-04-22T08:16:24.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.037+0000 7f847affd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f847c0a2570 0x7f847c0a01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:24.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.037+0000 7f847b7fe640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f847c0a1a80 0x7f847c0a1e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57456/0 (socket says 172.21.15.40:57456) 2024-04-22T08:16:24.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.037+0000 7f847b7fe640 1 -- 172.21.15.40:0/4058518515 learned_addr learned my addr 172.21.15.40:0/4058518515 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:16:24.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.037+0000 7f847affd640 10 monclient(hunting): get_auth_request con 0x7f847c0a2570 auth_method 0 2024-04-22T08:16:24.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.037+0000 7f847affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:24.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.037+0000 7f847affd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:24.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.037+0000 7f847affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:24.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.038+0000 7f847affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:24.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.038+0000 7f847affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:24.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.038+0000 7f847affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:24.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.038+0000 7f847bfff640 1 --2- 172.21.15.40:0/4058518515 >> v2:172.21.15.189:3300/0 conn(0x7f847c13b0e0 0x7f847c13d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:24.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.038+0000 7f847bfff640 10 monclient(hunting): get_auth_request con 0x7f847c13b0e0 auth_method 0 2024-04-22T08:16:24.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.038+0000 7f847bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:24.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.038+0000 7f847bfff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:24.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.038+0000 7f847bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:24.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.038+0000 7f847affd640 10 monclient(hunting): handle_auth_done global_id 54137 payload 274 2024-04-22T08:16:24.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.038+0000 7f847affd640 1 -- 172.21.15.40:0/4058518515 >> v2:172.21.15.189:3300/0 conn(0x7f847c13b0e0 msgr2=0x7f847c13d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:16:24.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.038+0000 7f847affd640 1 --2- 172.21.15.40:0/4058518515 >> v2:172.21.15.189:3300/0 conn(0x7f847c13b0e0 0x7f847c13d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:24.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.038+0000 7f847affd640 1 -- 172.21.15.40:0/4058518515 >> v1:172.21.15.40:6789/0 conn(0x7f847c0a1a80 legacy=0x7f847c0a1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:16:24.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.038+0000 7f847affd640 10 monclient: _finish_hunting 0 2024-04-22T08:16:24.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.038+0000 7f847affd640 1 monclient: found mon.noname-b 2024-04-22T08:16:24.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.038+0000 7f847affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:16:24.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.038+0000 7f847affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:16:24.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.038+0000 7f847affd640 1 -- 172.21.15.40:0/4058518515 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f847c09eee0 con 0x7f847c0a2570 2024-04-22T08:16:24.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.041+0000 7f847affd640 1 -- 172.21.15.40:0/4058518515 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f847c0a2570 msgr2=0x7f847c0a01b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:16:24.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.041+0000 7f847affd640 1 --2- 172.21.15.40:0/4058518515 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f847c0a2570 0x7f847c0a01b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f847c09f300 tx=0x7f846c02b6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=985ec20308ae05b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:24.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.041+0000 7f847a7fc640 1 -- 172.21.15.40:0/4058518515 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846c030920 con 0x7f847c0a2570 2024-04-22T08:16:24.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.041+0000 7f847a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:24.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.041+0000 7f847a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:16:24.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.041+0000 7f847a7fc640 10 monclient: dump: 2024-04-22T08:16:24.056 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:24.057 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:24.057 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:24.057 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:24.058 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:24.058 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:24.058 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:24.058 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:24.058 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:24.058 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:24.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.041+0000 7f847a7fc640 10 monclient: _finish_auth 0 2024-04-22T08:16:24.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.041+0000 7f847a7fc640 10 monclient: _check_auth_tickets 2024-04-22T08:16:24.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.041+0000 7f847a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:16:24.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.041+0000 7f847a7fc640 1 -- 172.21.15.40:0/4058518515 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f846c02e050 con 0x7f847c0a2570 2024-04-22T08:16:24.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.041+0000 7f847a7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:16:24.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.041+0000 7f847a7fc640 1 -- 172.21.15.40:0/4058518515 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846c030ca0 con 0x7f847c0a2570 2024-04-22T08:16:24.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.041+0000 7f8481ea1640 5 monclient: authenticate success, global_id 54137 2024-04-22T08:16:24.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.041+0000 7f8481ea1640 10 monclient: get_monmap_and_config success 2024-04-22T08:16:24.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.041+0000 7f8481ea1640 4 set_mon_vals no callback set 2024-04-22T08:16:24.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.041+0000 7f847a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:24.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.041+0000 7f847a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:16:24.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.041+0000 7f847a7fc640 10 monclient: dump: 2024-04-22T08:16:24.061 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:24.062 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:24.062 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:24.062 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:24.062 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:24.062 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:24.062 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:24.062 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:24.062 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:24.062 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:24.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.041+0000 7f8481ea1640 10 monclient: shutdown 2024-04-22T08:16:24.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.041+0000 7f8481ea1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:16:24.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.041+0000 7f8481ea1640 1 -- 172.21.15.40:0/4058518515 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f847c0a2570 msgr2=0x7f847c0a01b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:24.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.041+0000 7f8481ea1640 1 --2- 172.21.15.40:0/4058518515 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f847c0a2570 0x7f847c0a01b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f847c09f300 tx=0x7f846c02b6f0 comp rx=0 tx=0).stop 2024-04-22T08:16:24.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.041+0000 7f8481ea1640 1 -- 172.21.15.40:0/4058518515 shutdown_connections 2024-04-22T08:16:24.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.041+0000 7f8481ea1640 1 --2- 172.21.15.40:0/4058518515 >> v2:172.21.15.189:3300/0 conn(0x7f847c13b0e0 0x7f847c13d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:24.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.041+0000 7f8481ea1640 1 --2- 172.21.15.40:0/4058518515 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f847c0a2570 0x7f847c0a01b0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:24.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.041+0000 7f8481ea1640 1 -- 172.21.15.40:0/4058518515 >> 172.21.15.40:0/4058518515 conn(0x7f847c0a66a0 msgr2=0x7f847c0a2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:16:24.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.041+0000 7f8481ea1640 1 -- 172.21.15.40:0/4058518515 shutdown_connections 2024-04-22T08:16:24.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.041+0000 7f8481ea1640 1 -- 172.21.15.40:0/4058518515 wait complete. 2024-04-22T08:16:24.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.042+0000 7f8481ea1640 10 monclient: build_initial_monmap 2024-04-22T08:16:24.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.042+0000 7f8481ea1640 10 monclient: monmap: 2024-04-22T08:16:24.063 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:16:24.063 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:16:24.063 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:16:24.064 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:16:24.064 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:16:24.064 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:24.064 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:16:24.064 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:16:24.064 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:16:24.064 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:24.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.042+0000 7f8481ea1640 1 Processor -- start 2024-04-22T08:16:24.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.042+0000 7f8481ea1640 1 -- start start 2024-04-22T08:16:24.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.042+0000 7f8481ea1640 10 monclient: init 2024-04-22T08:16:24.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f8481ea1640 10 monclient: _reopen_session rank -1 2024-04-22T08:16:24.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f8481ea1640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-22T08:16:24.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f8481ea1640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f847c0a1a80 0x7f847c1cad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:24.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f8481ea1640 10 monclient(hunting): picked mon.noname-b con 0x7f847c0a1a80 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:16:24.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f8481ea1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f847c0a2570 0x7f847c1cb240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:24.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f8481ea1640 10 monclient(hunting): picked mon.noname-a con 0x7f847c0a2570 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:16:24.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f8481ea1640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f847c13b0e0 0x7f847c120ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:24.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f8481ea1640 10 monclient(hunting): picked mon.noname-c con 0x7f847c13b0e0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:16:24.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f8481ea1640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:24.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f8481ea1640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847c122d30 con 0x7f847c0a2570 2024-04-22T08:16:24.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f8481ea1640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:24.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f8481ea1640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847c122110 con 0x7f847c0a1a80 2024-04-22T08:16:24.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f8481ea1640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:24.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f8481ea1640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f847c09cc20 con 0x7f847c13b0e0 2024-04-22T08:16:24.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f8481ea1640 10 monclient(hunting): _renew_subs 2024-04-22T08:16:24.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f8481ea1640 10 monclient(hunting): authenticate will time out at 1058.038179s 2024-04-22T08:16:24.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f847bfff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f847c13b0e0 0x7f847c120ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:24.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f847affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f847c0a2570 0x7f847c1cb240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:24.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f847bfff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f847c13b0e0 0x7f847c120ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:33196/0 (socket says 172.21.15.40:33196) 2024-04-22T08:16:24.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f847affd640 10 monclient(hunting): get_auth_request con 0x7f847c0a2570 auth_method 0 2024-04-22T08:16:24.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f847bfff640 1 -- 172.21.15.40:0/4210252437 learned_addr learned my addr 172.21.15.40:0/4210252437 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:16:24.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f847affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:24.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f847affd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:24.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f847affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:24.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f847bfff640 10 monclient(hunting): get_auth_request con 0x7f847c13b0e0 auth_method 0 2024-04-22T08:16:24.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f847bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:24.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f847bfff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:24.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f847bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:24.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f847b7fe640 1 --2- 172.21.15.40:0/4210252437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f847c0a1a80 0x7f847c1cad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:24.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f847b7fe640 10 monclient(hunting): get_auth_request con 0x7f847c0a1a80 auth_method 0 2024-04-22T08:16:24.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f847b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:24.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f847b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:24.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f847b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:24.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f847affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:24.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f847affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:24.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f847affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:24.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f847bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:24.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f847bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:24.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f847bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:24.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f847b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:24.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f847b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:24.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f847b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:24.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f847affd640 10 monclient(hunting): handle_auth_done global_id 34575 payload 824 2024-04-22T08:16:24.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f847affd640 1 -- 172.21.15.40:0/4210252437 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f847c13b0e0 msgr2=0x7f847c120ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:24.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f847affd640 1 --2- 172.21.15.40:0/4210252437 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f847c13b0e0 0x7f847c120ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:24.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f847affd640 1 -- 172.21.15.40:0/4210252437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f847c0a1a80 msgr2=0x7f847c1cad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:24.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f847affd640 1 --2- 172.21.15.40:0/4210252437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f847c0a1a80 0x7f847c1cad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:24.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f847affd640 10 monclient: _finish_hunting 0 2024-04-22T08:16:24.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f847affd640 1 monclient: found mon.noname-a 2024-04-22T08:16:24.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f847affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:16:24.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f847affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:16:24.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f847affd640 1 -- 172.21.15.40:0/4210252437 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f846c009400 con 0x7f847c0a2570 2024-04-22T08:16:24.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.043+0000 7f847bfff640 1 --2- 172.21.15.40:0/4210252437 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f847c13b0e0 0x7f847c120ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:16:24.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.046+0000 7f847affd640 1 -- 172.21.15.40:0/4210252437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f847c0a2570 msgr2=0x7f847c1cb240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:16:24.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.046+0000 7f847affd640 1 --2- 172.21.15.40:0/4210252437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f847c0a2570 0x7f847c1cb240 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f846c032000 tx=0x7f846c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:24.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.046+0000 7f8463fff640 1 -- 172.21.15.40:0/4210252437 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846c03bae0 con 0x7f847c0a2570 2024-04-22T08:16:24.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.046+0000 7f8463fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:24.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.046+0000 7f8463fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:16:24.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.046+0000 7f8463fff640 10 monclient: dump: 2024-04-22T08:16:24.068 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:24.069 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:24.069 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:24.069 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:24.069 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:24.069 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:24.069 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:24.069 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:24.069 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:24.069 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:24.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.046+0000 7f8463fff640 10 monclient: _finish_auth 0 2024-04-22T08:16:24.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.046+0000 7f8463fff640 10 monclient: _check_auth_tickets 2024-04-22T08:16:24.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.046+0000 7f8463fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:16:24.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.046+0000 7f8463fff640 1 -- 172.21.15.40:0/4210252437 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f846c03bc60 con 0x7f847c0a2570 2024-04-22T08:16:24.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.046+0000 7f8481ea1640 5 monclient: authenticate success, global_id 34575 2024-04-22T08:16:24.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.046+0000 7f8481ea1640 10 monclient: _renew_subs 2024-04-22T08:16:24.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.046+0000 7f8481ea1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:16:24.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.046+0000 7f8481ea1640 1 -- 172.21.15.40:0/4210252437 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f847c13b650 con 0x7f847c0a2570 2024-04-22T08:16:24.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.046+0000 7f8463fff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:16:24.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.046+0000 7f8463fff640 1 -- 172.21.15.40:0/4210252437 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846c032400 con 0x7f847c0a2570 2024-04-22T08:16:24.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.046+0000 7f8463fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:24.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.046+0000 7f8463fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:16:24.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.046+0000 7f8463fff640 10 monclient: dump: 2024-04-22T08:16:24.070 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:24.070 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:24.070 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:24.070 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:24.070 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:24.071 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:24.071 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:24.071 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:24.071 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:24.071 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:24.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.046+0000 7f8479ffb640 4 set_mon_vals no callback set 2024-04-22T08:16:24.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.046+0000 7f8481ea1640 10 monclient: _renew_subs 2024-04-22T08:16:24.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.046+0000 7f8481ea1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:16:24.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.046+0000 7f8481ea1640 1 -- 172.21.15.40:0/4210252437 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f847c124450 con 0x7f847c0a2570 2024-04-22T08:16:24.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.047+0000 7f8481ea1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:16:24.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.047+0000 7f8481ea1640 1 --2- 172.21.15.40:0/4210252437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f847c121010 0x7f847c1213f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:24.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.047+0000 7f8463fff640 1 -- 172.21.15.40:0/4210252437 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f846c039040 con 0x7f847c0a2570 2024-04-22T08:16:24.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.047+0000 7f8481ea1640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:24.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.047+0000 7f8481ea1640 1 -- 172.21.15.40:0/4210252437 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847c066770 con 0x7f847c121010 2024-04-22T08:16:24.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.047+0000 7f8463fff640 1 --2- 172.21.15.40:0/4210252437 >> v2:172.21.15.189:6800/30168 conn(0x7f8468037800 0x7f8468039cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:24.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.047+0000 7f8463fff640 1 -- 172.21.15.40:0/4210252437 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f846c06b950 con 0x7f847c0a2570 2024-04-22T08:16:24.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.047+0000 7f847b7fe640 1 --2- 172.21.15.40:0/4210252437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f847c121010 0x7f847c1213f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:24.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.047+0000 7f847b7fe640 10 monclient: get_auth_request con 0x7f847c121010 auth_method 0 2024-04-22T08:16:24.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.047+0000 7f847b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:24.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.047+0000 7f847b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:24.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.047+0000 7f847b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:24.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.047+0000 7f847bfff640 1 --2- 172.21.15.40:0/4210252437 >> v2:172.21.15.189:6800/30168 conn(0x7f8468037800 0x7f8468039cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:24.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.047+0000 7f847bfff640 10 monclient: get_auth_request con 0x7f8468037800 auth_method 0 2024-04-22T08:16:24.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.047+0000 7f847b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:24.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.047+0000 7f847b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:24.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.047+0000 7f847b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:24.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.047+0000 7f847b7fe640 10 monclient(hunting): handle_auth_done global_id 24790 payload 824 2024-04-22T08:16:24.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.047+0000 7f847b7fe640 1 -- 172.21.15.40:0/4210252437 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f847c13da10 con 0x7f847c121010 2024-04-22T08:16:24.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.050+0000 7f847bfff640 1 --2- 172.21.15.40:0/4210252437 >> v2:172.21.15.189:6800/30168 conn(0x7f8468037800 0x7f8468039cc0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8474007880 tx=0x7f84740098d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:24.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.050+0000 7f847b7fe640 1 -- 172.21.15.40:0/4210252437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f847c121010 msgr2=0x7f847c1213f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:16:24.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.050+0000 7f847b7fe640 1 --2- 172.21.15.40:0/4210252437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f847c121010 0x7f847c1213f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f847000e7a0 tx=0x7f847000ec60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:24.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.050+0000 7f8463fff640 1 -- 172.21.15.40:0/4210252437 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f847000ee70 con 0x7f847c121010 2024-04-22T08:16:24.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.050+0000 7f8463fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:24.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.050+0000 7f8463fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:16:24.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.050+0000 7f8463fff640 10 monclient: dump: 2024-04-22T08:16:24.073 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:24.073 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:24.073 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:24.073 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:24.073 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:24.073 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:24.073 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:24.074 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:24.074 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:24.074 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:24.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.050+0000 7f8463fff640 1 -- 172.21.15.40:0/4210252437 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f847000cb40 con 0x7f847c121010 2024-04-22T08:16:24.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.050+0000 7f8463fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:16:24.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.050+0000 7f8463fff640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:16:24.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.050+0000 7f8463fff640 1 -- 172.21.15.40:0/4210252437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f847c121010 msgr2=0x7f847c1213f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:24.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.050+0000 7f8463fff640 1 --2- 172.21.15.40:0/4210252437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f847c121010 0x7f847c1213f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f847000e7a0 tx=0x7f847000ec60 comp rx=0 tx=0).stop 2024-04-22T08:16:24.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.050+0000 7f8463fff640 1 -- 172.21.15.40:0/4210252437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f847c121010 msgr2=0x7f847c1213f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:16:24.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.050+0000 7f8463fff640 1 --2- 172.21.15.40:0/4210252437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f847c121010 0x7f847c1213f0 secure :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f847000e7a0 tx=0x7f847000ec60 comp rx=0 tx=0).stop 2024-04-22T08:16:24.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.058+0000 7f8481ea1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:16:24.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.058+0000 7f8481ea1640 1 --2- 172.21.15.40:0/4210252437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f847c121930 0x7f847c06db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:24.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.058+0000 7f8481ea1640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:24.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.058+0000 7f8481ea1640 1 -- 172.21.15.40:0/4210252437 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847c123c40 con 0x7f847c121930 2024-04-22T08:16:24.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.058+0000 7f847b7fe640 1 --2- 172.21.15.40:0/4210252437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f847c121930 0x7f847c06db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:24.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.058+0000 7f847b7fe640 10 monclient: get_auth_request con 0x7f847c121930 auth_method 0 2024-04-22T08:16:24.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.058+0000 7f847b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:24.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.058+0000 7f847b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:24.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.058+0000 7f847b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:24.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.058+0000 7f847b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:24.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.058+0000 7f847b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:24.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.058+0000 7f847b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:24.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.058+0000 7f847b7fe640 10 monclient(hunting): handle_auth_done global_id 24793 payload 824 2024-04-22T08:16:24.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.058+0000 7f847b7fe640 1 -- 172.21.15.40:0/4210252437 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f847c04e9b0 con 0x7f847c121930 2024-04-22T08:16:24.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.060+0000 7f847b7fe640 1 -- 172.21.15.40:0/4210252437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f847c121930 msgr2=0x7f847c06db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:16:24.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.060+0000 7f847b7fe640 1 --2- 172.21.15.40:0/4210252437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f847c121930 0x7f847c06db80 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8470019000 tx=0x7f84700058d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:24.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.060+0000 7f8463fff640 1 -- 172.21.15.40:0/4210252437 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8470022070 con 0x7f847c121930 2024-04-22T08:16:24.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.060+0000 7f8463fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:24.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.060+0000 7f8463fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:16:24.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.060+0000 7f8463fff640 10 monclient: dump: 2024-04-22T08:16:24.077 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:24.077 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:24.078 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:24.078 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:24.078 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:24.078 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:24.078 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:24.078 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:24.078 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:24.078 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:24.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.060+0000 7f8463fff640 1 -- 172.21.15.40:0/4210252437 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4801 (secure 0 0 0) 0x7f8470006660 con 0x7f847c121930 2024-04-22T08:16:24.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.060+0000 7f8463fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:16:24.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.060+0000 7f8463fff640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:16:24.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.060+0000 7f8463fff640 1 -- 172.21.15.40:0/4210252437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f847c121930 msgr2=0x7f847c06db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:24.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.060+0000 7f8463fff640 1 --2- 172.21.15.40:0/4210252437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f847c121930 0x7f847c06db80 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8470019000 tx=0x7f84700058d0 comp rx=0 tx=0).stop 2024-04-22T08:16:24.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.060+0000 7f8463fff640 1 -- 172.21.15.40:0/4210252437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f847c121930 msgr2=0x7f847c06db80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:16:24.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.060+0000 7f8463fff640 1 --2- 172.21.15.40:0/4210252437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f847c121930 0x7f847c06db80 secure :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8470019000 tx=0x7f84700058d0 comp rx=0 tx=0).stop 2024-04-22T08:16:24.079 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:16:24.079 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:16:24.079 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:16:24.079 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:16:24.079 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 42, 2024-04-22T08:16:24.079 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:16:24.079 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:16:24.079 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:16:24.079 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:16:24.080 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:16:24.080 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 2, 2024-04-22T08:16:24.080 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:16:24.080 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:16:24.080 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:16:24.080 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:16:24.080 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:16:24.080 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:16:24.080 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:16:24.080 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:16:24.080 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:16:24.080 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:16:24.080 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:16:24.081 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:16:24.081 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:16:24.081 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:16:24.081 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:16:24.081 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:16:24.081 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:16:24.081 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:16:24.081 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:16:24.081 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:16:24.081 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:16:24.081 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:16:24.081 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:16:24.081 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:16:24.081 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:16:24.082 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:16:24.082 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:24.082 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:24.082 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:16:24.082 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:16:24.082 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:16:24.082 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:16:24.082 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:16:24.082 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:16:24.082 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:16:24.082 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:16:24.082 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:16:24.082 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:16:24.082 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:16:24.083 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:16:24.083 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:16:24.083 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:16:24.083 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:16:24.083 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:16:24.083 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:16:24.083 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:16:24.083 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:16:24.083 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:16:24.083 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:16:24.083 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:16:24.083 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:16:24.083 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:16:24.083 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:16:24.084 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:16:24.084 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:16:24.084 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:16:24.084 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:16:24.084 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:24.084 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:16:24.084 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:24.084 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:16:24.084 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:16:24.084 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:16:24.084 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:16:24.084 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:24.084 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:16:24.084 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:16:24.085 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:24.085 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:24.085 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:24.085 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:16:24.085 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:16:24.085 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:24.085 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:24.085 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:24.085 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:24.085 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:16:24.085 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:16:24.086 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:16:24.086 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:16:24.086 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:16:24.086 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:24.086 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:24.086 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:16:24.086 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:16:24.086 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:16:24.086 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:16:24.086 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:24.086 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:16:24.086 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:16:24.086 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:24.087 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:24.087 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:24.087 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:16:24.087 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:16:24.087 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:24.087 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:24.087 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:24.087 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:24.087 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:16:24.087 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:16:24.087 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:16:24.087 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:16:24.087 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:16:24.087 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:24.088 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:24.088 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:16:24.088 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:16:24.088 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:16:24.088 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:16:24.088 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:24.088 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:16:24.088 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:16:24.088 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:24.088 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:24.088 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:24.088 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:16:24.088 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:16:24.088 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:24.089 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:24.089 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:24.089 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:24.089 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:16:24.089 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:16:24.089 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:16:24.089 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:16:24.089 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:16:24.089 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:24.089 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:24.089 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:24.089 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:16:24.089 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:16:24.089 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:24.090 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:16:24.090 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:16:24.090 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:16:24.090 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:24.090 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:16:24.090 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:16:24.090 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:24.090 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:16:24.090 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:16:24.090 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 2 2024-04-22T08:16:24.090 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:24.090 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:16:24.090 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:16:24.090 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:24.091 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:16:24.091 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:16:24.091 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 7 2024-04-22T08:16:24.091 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:24.091 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:16:24.091 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": [ 2024-04-22T08:16:24.091 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:24.091 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:16:24.091 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:16:24.091 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:16:24.091 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:24.091 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:24.091 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:24.091 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:16:24.092 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:16:24.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.060+0000 7f8481ea1640 1 -- 172.21.15.40:0/4210252437 >> v2:172.21.15.189:6800/30168 conn(0x7f8468037800 msgr2=0x7f8468039cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:24.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.060+0000 7f8481ea1640 1 --2- 172.21.15.40:0/4210252437 >> v2:172.21.15.189:6800/30168 conn(0x7f8468037800 0x7f8468039cc0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8474007880 tx=0x7f84740098d0 comp rx=0 tx=0).stop 2024-04-22T08:16:24.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.060+0000 7f8481ea1640 10 monclient: shutdown 2024-04-22T08:16:24.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.060+0000 7f8481ea1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:16:24.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.060+0000 7f8481ea1640 1 -- 172.21.15.40:0/4210252437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f847c0a2570 msgr2=0x7f847c1cb240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:24.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.060+0000 7f847bfff640 1 -- 172.21.15.40:0/4210252437 reap_dead start 2024-04-22T08:16:24.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.060+0000 7f8481ea1640 1 --2- 172.21.15.40:0/4210252437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f847c0a2570 0x7f847c1cb240 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f846c032000 tx=0x7f846c0046c0 comp rx=0 tx=0).stop 2024-04-22T08:16:24.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.061+0000 7f8481ea1640 1 -- 172.21.15.40:0/4210252437 shutdown_connections 2024-04-22T08:16:24.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.061+0000 7f8481ea1640 1 --2- 172.21.15.40:0/4210252437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f847c0a2570 0x7f847c1cb240 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:24.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.061+0000 7f8481ea1640 1 -- 172.21.15.40:0/4210252437 >> 172.21.15.40:0/4210252437 conn(0x7f847c0a66a0 msgr2=0x7f847c0a3f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:16:24.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.061+0000 7f8481ea1640 1 -- 172.21.15.40:0/4210252437 shutdown_connections 2024-04-22T08:16:24.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.061+0000 7f8481ea1640 1 -- 172.21.15.40:0/4210252437 wait complete. 2024-04-22T08:16:24.094 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:16:24.094 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-22T08:16:24.094 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-22T08:16:24.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.193+0000 7f7e091ca640 10 monclient: get_monmap_and_config 2024-04-22T08:16:24.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.193+0000 7f7e091ca640 10 monclient: build_initial_monmap 2024-04-22T08:16:24.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.193+0000 7f7e091ca640 10 monclient: monmap: 2024-04-22T08:16:24.193 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:16:24.193 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:24.193 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:16:24.194237+0000 2024-04-22T08:16:24.193 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:16:24.194237+0000 2024-04-22T08:16:24.193 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:16:24.193 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:24.193 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:16:24.193 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:16:24.193 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:16:24.194 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:16:24.194 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:16:24.194 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:24.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.193+0000 7f7e091ca640 1 Processor -- start 2024-04-22T08:16:24.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.193+0000 7f7e091ca640 1 -- start start 2024-04-22T08:16:24.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.193+0000 7f7e091ca640 10 monclient: init 2024-04-22T08:16:24.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.193+0000 7f7e091ca640 10 monclient: _reopen_session rank -1 2024-04-22T08:16:24.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.193+0000 7f7e091ca640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-04-22T08:16:24.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.193+0000 7f7e091ca640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7e0409f2e0 0x7f7e0409f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:24.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.193+0000 7f7e091ca640 10 monclient(hunting): picked mon.noname-b con 0x7f7e0409f2e0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:16:24.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.193+0000 7f7e091ca640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7e0409b760 0x7f7e0409fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:24.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.193+0000 7f7e091ca640 10 monclient(hunting): picked mon.noname-c con 0x7f7e0409b760 addr v2:172.21.15.189:3300/0 2024-04-22T08:16:24.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.193+0000 7f7e091ca640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7e04136840 addr v1:172.21.15.189:6789/0 2024-04-22T08:16:24.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.193+0000 7f7e091ca640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e0409cf70 con 0x7f7e04136840 2024-04-22T08:16:24.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.193+0000 7f7e091ca640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:24.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.193+0000 7f7e091ca640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e040a0140 con 0x7f7e0409b760 2024-04-22T08:16:24.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.193+0000 7f7e091ca640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:24.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.193+0000 7f7e091ca640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0409c6d0 con 0x7f7e0409f2e0 2024-04-22T08:16:24.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.193+0000 7f7e091ca640 10 monclient(hunting): _renew_subs 2024-04-22T08:16:24.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.193+0000 7f7e091ca640 10 monclient(hunting): authenticate will time out at 1058.188313s 2024-04-22T08:16:24.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.193+0000 7f7e02d9d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7e0409f2e0 0x7f7e0409f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:24.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.193+0000 7f7e02d9d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7e0409f2e0 0x7f7e0409f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:33206/0 (socket says 172.21.15.40:33206) 2024-04-22T08:16:24.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.193+0000 7f7e02d9d640 1 -- 172.21.15.40:0/2253624033 learned_addr learned my addr 172.21.15.40:0/2253624033 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:16:24.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.193+0000 7f7e02d9d640 10 monclient(hunting): get_auth_request con 0x7f7e0409f2e0 auth_method 0 2024-04-22T08:16:24.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.193+0000 7f7e02d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:24.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.193+0000 7f7e02d9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:24.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.193+0000 7f7e02d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:24.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.193+0000 7f7e02d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:24.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.193+0000 7f7e02d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:24.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.193+0000 7f7e02d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:24.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.193+0000 7f7e0259c640 1 --2- 172.21.15.40:0/2253624033 >> v2:172.21.15.189:3300/0 conn(0x7f7e0409b760 0x7f7e0409fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:24.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.193+0000 7f7e02d9d640 10 monclient(hunting): handle_auth_done global_id 54143 payload 274 2024-04-22T08:16:24.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.193+0000 7f7e02d9d640 1 -- 172.21.15.40:0/2253624033 >> v2:172.21.15.189:3300/0 conn(0x7f7e0409b760 msgr2=0x7f7e0409fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:16:24.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.193+0000 7f7e02d9d640 1 --2- 172.21.15.40:0/2253624033 >> v2:172.21.15.189:3300/0 conn(0x7f7e0409b760 0x7f7e0409fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:24.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.193+0000 7f7e02d9d640 1 -- 172.21.15.40:0/2253624033 >> v1:172.21.15.189:6789/0 conn(0x7f7e04136840 legacy=0x7f7e04138c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:16:24.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.193+0000 7f7e02d9d640 10 monclient: _finish_hunting 0 2024-04-22T08:16:24.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.193+0000 7f7e02d9d640 1 monclient: found mon.noname-b 2024-04-22T08:16:24.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.193+0000 7f7e02d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:16:24.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.193+0000 7f7e02d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:16:24.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.193+0000 7f7e02d9d640 1 -- 172.21.15.40:0/2253624033 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e0409c9b0 con 0x7f7e0409f2e0 2024-04-22T08:16:24.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.193+0000 7f7e0259c640 10 monclient: get_auth_request con 0x7f7e0409b760 auth_method 0 2024-04-22T08:16:24.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.194+0000 7f7e0259c640 1 --2- 172.21.15.40:0/2253624033 >> v2:172.21.15.189:3300/0 conn(0x7f7e0409b760 0x7f7e0409fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T08:16:24.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.196+0000 7f7e02d9d640 1 -- 172.21.15.40:0/2253624033 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7e0409f2e0 msgr2=0x7f7e0409f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:16:24.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.196+0000 7f7e02d9d640 1 --2- 172.21.15.40:0/2253624033 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7e0409f2e0 0x7f7e0409f6c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc00bb10 tx=0x7f7dfc02dfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ca8fbc8c1b208bf server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:24.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.196+0000 7f7e01d9b640 1 -- 172.21.15.40:0/2253624033 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dfc036640 con 0x7f7e0409f2e0 2024-04-22T08:16:24.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.196+0000 7f7e01d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:24.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.196+0000 7f7e01d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:16:24.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.196+0000 7f7e01d9b640 10 monclient: dump: 2024-04-22T08:16:24.206 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:24.206 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:24.206 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:24.206 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:24.206 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:24.206 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:24.206 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:24.206 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:24.206 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:24.206 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:24.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.197+0000 7f7e01d9b640 10 monclient: _finish_auth 0 2024-04-22T08:16:24.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.197+0000 7f7e01d9b640 10 monclient: _check_auth_tickets 2024-04-22T08:16:24.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.197+0000 7f7e01d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:16:24.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.197+0000 7f7e091ca640 5 monclient: authenticate success, global_id 54143 2024-04-22T08:16:24.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.197+0000 7f7e091ca640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:16:24.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.197+0000 7f7e01d9b640 1 -- 172.21.15.40:0/2253624033 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dfc0367a0 con 0x7f7e0409f2e0 2024-04-22T08:16:24.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.197+0000 7f7e01d9b640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:16:24.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.197+0000 7f7e091ca640 10 monclient: get_monmap_and_config success 2024-04-22T08:16:24.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.197+0000 7f7e091ca640 4 set_mon_vals no callback set 2024-04-22T08:16:24.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.197+0000 7f7e01d9b640 1 -- 172.21.15.40:0/2253624033 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dfc036b30 con 0x7f7e0409f2e0 2024-04-22T08:16:24.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.197+0000 7f7e01d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:24.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.197+0000 7f7e01d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:16:24.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.197+0000 7f7e01d9b640 10 monclient: dump: 2024-04-22T08:16:24.207 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:24.207 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:24.208 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:24.208 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:24.208 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:24.208 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:24.208 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:24.208 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:24.208 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:24.208 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:24.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.197+0000 7f7e091ca640 10 monclient: shutdown 2024-04-22T08:16:24.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.197+0000 7f7e091ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:16:24.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.197+0000 7f7e091ca640 1 -- 172.21.15.40:0/2253624033 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7e0409f2e0 msgr2=0x7f7e0409f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:24.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.197+0000 7f7e091ca640 1 --2- 172.21.15.40:0/2253624033 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7e0409f2e0 0x7f7e0409f6c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc00bb10 tx=0x7f7dfc02dfa0 comp rx=0 tx=0).stop 2024-04-22T08:16:24.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.197+0000 7f7e091ca640 1 -- 172.21.15.40:0/2253624033 shutdown_connections 2024-04-22T08:16:24.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.197+0000 7f7e091ca640 1 --2- 172.21.15.40:0/2253624033 >> v2:172.21.15.189:3300/0 conn(0x7f7e0409b760 0x7f7e0409fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:24.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.197+0000 7f7e091ca640 1 --2- 172.21.15.40:0/2253624033 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7e0409f2e0 0x7f7e0409f6c0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:24.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.197+0000 7f7e091ca640 1 -- 172.21.15.40:0/2253624033 >> 172.21.15.40:0/2253624033 conn(0x7f7e040a0b50 msgr2=0x7f7e040a0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:16:24.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.197+0000 7f7e091ca640 1 -- 172.21.15.40:0/2253624033 shutdown_connections 2024-04-22T08:16:24.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.197+0000 7f7e091ca640 1 -- 172.21.15.40:0/2253624033 wait complete. 2024-04-22T08:16:24.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.198+0000 7f7e091ca640 10 monclient: build_initial_monmap 2024-04-22T08:16:24.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.198+0000 7f7e091ca640 10 monclient: monmap: 2024-04-22T08:16:24.209 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:16:24.209 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:16:24.209 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:16:24.209 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:16:24.209 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:16:24.209 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:24.209 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:16:24.209 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:16:24.210 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:16:24.210 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:24.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.198+0000 7f7e091ca640 1 Processor -- start 2024-04-22T08:16:24.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.198+0000 7f7e091ca640 1 -- start start 2024-04-22T08:16:24.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.198+0000 7f7e091ca640 10 monclient: init 2024-04-22T08:16:24.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e091ca640 10 monclient: _reopen_session rank -1 2024-04-22T08:16:24.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e091ca640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-22T08:16:24.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e091ca640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e0409f2e0 0x7f7e0411a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:24.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e091ca640 10 monclient(hunting): picked mon.noname-a con 0x7f7e0409f2e0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:16:24.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e091ca640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7e04136840 0x7f7e0411abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:24.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e091ca640 10 monclient(hunting): picked mon.noname-c con 0x7f7e04136840 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:16:24.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e091ca640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e0411b120 0x7f7e041c69e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:24.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e091ca640 10 monclient(hunting): picked mon.noname-b con 0x7f7e0411b120 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:16:24.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e091ca640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:24.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e091ca640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0409bb50 con 0x7f7e0409f2e0 2024-04-22T08:16:24.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e091ca640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:24.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e091ca640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e04134d00 con 0x7f7e0411b120 2024-04-22T08:16:24.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e091ca640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:24.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e091ca640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0411c0f0 con 0x7f7e04136840 2024-04-22T08:16:24.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e091ca640 10 monclient(hunting): _renew_subs 2024-04-22T08:16:24.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e091ca640 10 monclient(hunting): authenticate will time out at 1058.194012s 2024-04-22T08:16:24.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e02d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e0409f2e0 0x7f7e0411a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:24.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e0259c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7e04136840 0x7f7e0411abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:24.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e02d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e0409f2e0 0x7f7e0411a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41576/0 (socket says 172.21.15.40:41576) 2024-04-22T08:16:24.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e02d9d640 1 -- 172.21.15.40:0/632689982 learned_addr learned my addr 172.21.15.40:0/632689982 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:16:24.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e02d9d640 10 monclient(hunting): get_auth_request con 0x7f7e0409f2e0 auth_method 0 2024-04-22T08:16:24.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e0259c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7e04136840 0x7f7e0411abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:33212/0 (socket says 172.21.15.40:33212) 2024-04-22T08:16:24.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e02d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:24.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e02d9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:24.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e02d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:24.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e0259c640 10 monclient(hunting): get_auth_request con 0x7f7e04136840 auth_method 0 2024-04-22T08:16:24.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e0259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:24.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e0259c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:24.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e0259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:24.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e02d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:24.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e02d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:24.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e02d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:24.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e0259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:24.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e0259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:24.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e0259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:24.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e0359e640 1 --2- 172.21.15.40:0/632689982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e0411b120 0x7f7e041c69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:24.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e0259c640 10 monclient(hunting): handle_auth_done global_id 54146 payload 824 2024-04-22T08:16:24.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e0259c640 1 -- 172.21.15.40:0/632689982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e0411b120 msgr2=0x7f7e041c69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:24.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e0259c640 1 --2- 172.21.15.40:0/632689982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e0411b120 0x7f7e041c69e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:24.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e0259c640 1 -- 172.21.15.40:0/632689982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e0409f2e0 msgr2=0x7f7e0411a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:24.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e0259c640 1 --2- 172.21.15.40:0/632689982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e0409f2e0 0x7f7e0411a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:24.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e0259c640 10 monclient: _finish_hunting 0 2024-04-22T08:16:24.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e0259c640 1 monclient: found mon.noname-c 2024-04-22T08:16:24.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e0259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:16:24.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e0259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:16:24.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e0259c640 1 -- 172.21.15.40:0/632689982 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dfc00b770 con 0x7f7e04136840 2024-04-22T08:16:24.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e0359e640 10 monclient: get_auth_request con 0x7f7e0411b120 auth_method 0 2024-04-22T08:16:24.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e0359e640 1 --2- 172.21.15.40:0/632689982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e0411b120 0x7f7e041c69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T08:16:24.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.199+0000 7f7e02d9d640 1 --2- 172.21.15.40:0/632689982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e0409f2e0 0x7f7e0411a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:16:24.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.201+0000 7f7e0259c640 1 -- 172.21.15.40:0/632689982 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7e04136840 msgr2=0x7f7e0411abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:16:24.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.201+0000 7f7e0259c640 1 --2- 172.21.15.40:0/632689982 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7e04136840 0x7f7e0411abe0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7df800c7f0 tx=0x7f7df800ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:24.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.202+0000 7f7deb7fe640 1 -- 172.21.15.40:0/632689982 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df8004220 con 0x7f7e04136840 2024-04-22T08:16:24.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.202+0000 7f7deb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:24.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.202+0000 7f7deb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:16:24.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.202+0000 7f7deb7fe640 10 monclient: dump: 2024-04-22T08:16:24.214 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:24.214 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:24.214 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:24.214 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:24.214 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:24.214 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:24.214 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:24.214 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:24.214 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:24.215 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:24.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.202+0000 7f7deb7fe640 10 monclient: _finish_auth 0 2024-04-22T08:16:24.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.202+0000 7f7deb7fe640 10 monclient: _check_auth_tickets 2024-04-22T08:16:24.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.202+0000 7f7deb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:16:24.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.202+0000 7f7deb7fe640 1 -- 172.21.15.40:0/632689982 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7df8004380 con 0x7f7e04136840 2024-04-22T08:16:24.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.202+0000 7f7deb7fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:16:24.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.202+0000 7f7deb7fe640 1 -- 172.21.15.40:0/632689982 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df8007a50 con 0x7f7e04136840 2024-04-22T08:16:24.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.202+0000 7f7deb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:24.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.202+0000 7f7deb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:16:24.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.202+0000 7f7deb7fe640 10 monclient: dump: 2024-04-22T08:16:24.215 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:24.215 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:24.215 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:24.215 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:24.216 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:24.216 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:24.216 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:24.216 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:24.216 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:24.216 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:24.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.202+0000 7f7e091ca640 5 monclient: authenticate success, global_id 54146 2024-04-22T08:16:24.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.202+0000 7f7e091ca640 10 monclient: _renew_subs 2024-04-22T08:16:24.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.202+0000 7f7e091ca640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:16:24.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.202+0000 7f7e091ca640 1 -- 172.21.15.40:0/632689982 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e0409cd40 con 0x7f7e04136840 2024-04-22T08:16:24.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.202+0000 7f7e00d99640 4 set_mon_vals no callback set 2024-04-22T08:16:24.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.202+0000 7f7e091ca640 10 monclient: _renew_subs 2024-04-22T08:16:24.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.202+0000 7f7e091ca640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:16:24.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.202+0000 7f7e091ca640 1 -- 172.21.15.40:0/632689982 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e0411d2a0 con 0x7f7e04136840 2024-04-22T08:16:24.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.202+0000 7f7deb7fe640 1 -- 172.21.15.40:0/632689982 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f7df8007bf0 con 0x7f7e04136840 2024-04-22T08:16:24.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.203+0000 7f7e091ca640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:16:24.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.203+0000 7f7e091ca640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:16:24.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.203+0000 7f7e091ca640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:16:24.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.203+0000 7f7e091ca640 1 -- 172.21.15.40:0/632689982 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e0404e9b0 con 0x7f7e04136840 2024-04-22T08:16:24.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.203+0000 7f7deb7fe640 1 --2- 172.21.15.40:0/632689982 >> v2:172.21.15.189:6800/30168 conn(0x7f7df00377d0 0x7f7df0039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:24.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.203+0000 7f7deb7fe640 1 -- 172.21.15.40:0/632689982 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f7df804a1b0 con 0x7f7e04136840 2024-04-22T08:16:24.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.203+0000 7f7e02d9d640 1 --2- 172.21.15.40:0/632689982 >> v2:172.21.15.189:6800/30168 conn(0x7f7df00377d0 0x7f7df0039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:24.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.203+0000 7f7e02d9d640 10 monclient: get_auth_request con 0x7f7df00377d0 auth_method 0 2024-04-22T08:16:24.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.207+0000 7f7e02d9d640 1 --2- 172.21.15.40:0/632689982 >> v2:172.21.15.189:6800/30168 conn(0x7f7df00377d0 0x7f7df0039c90 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc000c00 tx=0x7f7dfc032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:24.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.208+0000 7f7deb7fe640 1 -- 172.21.15.40:0/632689982 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f7df8020300 con 0x7f7e04136840 2024-04-22T08:16:24.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.208+0000 7f7deb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:16:24.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.208+0000 7f7deb7fe640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:16:24.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.462+0000 7f7e091ca640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-22T08:16:24.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.462+0000 7f7e091ca640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-22T08:16:24.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.462+0000 7f7e091ca640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:16:24.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.462+0000 7f7e091ca640 1 -- 172.21.15.40:0/632689982 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f7e0411b530 con 0x7f7e04136840 2024-04-22T08:16:24.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.464+0000 7f7deb7fe640 1 -- 172.21.15.40:0/632689982 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f7df8017b00 con 0x7f7e04136840 2024-04-22T08:16:24.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.464+0000 7f7deb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-22T08:16:24.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.464+0000 7f7deb7fe640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:16:24.465 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.464+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:16:24.465 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.464+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=2,health=12,kv=9,logm=65} crc {auth=1924667659,config=1618278320,health=478386949,kv=1350343662,logm=816210257}) 2024-04-22T08:16:24.465 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.464+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=12,config=2,health=11,kv=9,logm=66} crc {auth=1924667659,config=1618278320,health=4122615360,kv=1350343662,logm=1556297425}) 2024-04-22T08:16:24.465 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.464+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:16:24.465 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.464+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=2,health=12,kv=9,logm=65} crc {auth=1924667659,config=1618278320,health=478386949,kv=1350343662,logm=816210257}) 2024-04-22T08:16:24.465 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.464+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=12,config=2,health=12,kv=9,logm=65} crc {auth=1924667659,config=1618278320,health=478386949,kv=1350343662,logm=2155609613}) 2024-04-22T08:16:24.466 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.465+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:16:24.466 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.465+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=185511793}) 2024-04-22T08:16:24.466 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.465+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1955880736}) 2024-04-22T08:16:24.466 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.465+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:16:24.466 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.465+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=185511793}) 2024-04-22T08:16:24.466 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.465+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4223532406}) 2024-04-22T08:16:24.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.466+0000 7f7e091ca640 1 -- 172.21.15.40:0/632689982 >> v2:172.21.15.189:6800/30168 conn(0x7f7df00377d0 msgr2=0x7f7df0039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:24.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.466+0000 7f7e091ca640 1 --2- 172.21.15.40:0/632689982 >> v2:172.21.15.189:6800/30168 conn(0x7f7df00377d0 0x7f7df0039c90 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc000c00 tx=0x7f7dfc032000 comp rx=0 tx=0).stop 2024-04-22T08:16:24.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.466+0000 7f7e091ca640 10 monclient: shutdown 2024-04-22T08:16:24.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.466+0000 7f7e091ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:16:24.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.466+0000 7f7e091ca640 1 -- 172.21.15.40:0/632689982 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7e04136840 msgr2=0x7f7e0411abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:24.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.466+0000 7f7e091ca640 1 --2- 172.21.15.40:0/632689982 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7e04136840 0x7f7e0411abe0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7df800c7f0 tx=0x7f7df800ccb0 comp rx=0 tx=0).stop 2024-04-22T08:16:24.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.466+0000 7f7e091ca640 1 -- 172.21.15.40:0/632689982 shutdown_connections 2024-04-22T08:16:24.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.466+0000 7f7e091ca640 1 --2- 172.21.15.40:0/632689982 >> v2:172.21.15.189:6800/30168 conn(0x7f7df00377d0 0x7f7df0039c90 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:24.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.466+0000 7f7e091ca640 1 --2- 172.21.15.40:0/632689982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e0411b120 0x7f7e041c69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:24.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.466+0000 7f7e091ca640 1 --2- 172.21.15.40:0/632689982 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7e04136840 0x7f7e0411abe0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:24.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.466+0000 7f7e091ca640 1 --2- 172.21.15.40:0/632689982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e0409f2e0 0x7f7e0411a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:24.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.466+0000 7f7e091ca640 1 -- 172.21.15.40:0/632689982 >> 172.21.15.40:0/632689982 conn(0x7f7e040a0b50 msgr2=0x7f7e04136130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:16:24.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.466+0000 7f7e091ca640 1 -- 172.21.15.40:0/632689982 shutdown_connections 2024-04-22T08:16:24.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:24.466+0000 7f7e091ca640 1 -- 172.21.15.40:0/632689982 wait complete. 2024-04-22T08:16:24.473 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.466+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:16:24.473 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.466+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1928667968}) 2024-04-22T08:16:24.474 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.466+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=210343493}) 2024-04-22T08:16:24.474 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.466+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:16:24.474 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.466+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1928667968}) 2024-04-22T08:16:24.474 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.466+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1753522218}) 2024-04-22T08:16:24.474 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.467+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:16:24.474 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.467+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=625694147}) 2024-04-22T08:16:24.474 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.467+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2365192797}) 2024-04-22T08:16:24.474 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.467+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:16:24.474 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.467+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=625694147}) 2024-04-22T08:16:24.474 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.467+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1531693196}) 2024-04-22T08:16:24.474 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.468+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:16:24.474 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.468+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=155037242}) 2024-04-22T08:16:24.474 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.468+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2937179651}) 2024-04-22T08:16:24.475 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.468+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:16:24.475 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.468+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=155037242}) 2024-04-22T08:16:24.475 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.468+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=198363288}) 2024-04-22T08:16:24.475 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.469+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:16:24.475 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.469+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3214734160}) 2024-04-22T08:16:24.475 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.469+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1498702193}) 2024-04-22T08:16:24.475 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.469+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:16:24.475 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.469+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3214734160}) 2024-04-22T08:16:24.475 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.469+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3366863592}) 2024-04-22T08:16:24.475 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.470+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:16:24.475 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.470+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3892513947}) 2024-04-22T08:16:24.475 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.470+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2826441396}) 2024-04-22T08:16:24.475 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.470+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:16:24.476 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.470+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3892513947}) 2024-04-22T08:16:24.476 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.470+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2942259904}) 2024-04-22T08:16:24.476 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.471+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:16:24.476 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.471+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3956989552}) 2024-04-22T08:16:24.476 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.471+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=489846100}) 2024-04-22T08:16:24.476 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.471+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:16:24.476 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.471+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3956989552}) 2024-04-22T08:16:24.476 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.471+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4133790715}) 2024-04-22T08:16:24.476 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.472+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:16:24.476 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.472+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4069477188}) 2024-04-22T08:16:24.476 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.472+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1045038363}) 2024-04-22T08:16:24.476 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.472+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:16:24.476 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.472+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4069477188}) 2024-04-22T08:16:24.476 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.472+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1154068407}) 2024-04-22T08:16:24.476 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.473+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:16:24.477 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.473+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=3,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=7,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=57} crc {logm=3041649643,mdsmap=1531151339,mgr=1219260174,mgr_command_descs=470045235,mgr_metadata=4222303120,mgrstat=746587459,mon_config_key=1718430553,monmap=3469086636,osd_metadata=4151711904,osd_pg_creating=281692744,osdmap=2695319336}) 2024-04-22T08:16:24.477 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.473+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=3,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=7,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=56} crc {logm=3041649643,mdsmap=1531151339,mgr=878880162,mgr_command_descs=470045235,mgr_metadata=4222303120,mgrstat=939059172,mon_config_key=3255128821,monmap=3469086636,osd_metadata=4151711904,osd_pg_creating=281692744,osdmap=4014090665}) 2024-04-22T08:16:24.477 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.473+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:16:24.477 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.473+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=3,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=7,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=57} crc {logm=3041649643,mdsmap=1531151339,mgr=1219260174,mgr_command_descs=470045235,mgr_metadata=4222303120,mgrstat=746587459,mon_config_key=1718430553,monmap=3469086636,osd_metadata=4151711904,osd_pg_creating=281692744,osdmap=2695319336}) 2024-04-22T08:16:24.477 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.473+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=3,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=8,monmap=3,osd_metadata=5,osd_pg_creating=1,osdmap=55} crc {logm=3041649643,mdsmap=1531151339,mgr=878880162,mgr_command_descs=470045235,mgr_metadata=4222303120,mgrstat=1159716734,mon_config_key=4255067121,monmap=3469086636,osd_metadata=206245670,osd_pg_creating=281692744,osdmap=1116880111}) 2024-04-22T08:16:24.478 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.474+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:16:24.478 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.474+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2122181234}) 2024-04-22T08:16:24.478 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.474+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=65256455}) 2024-04-22T08:16:24.478 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.474+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:16:24.478 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.474+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2122181234}) 2024-04-22T08:16:24.478 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.474+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1272584466}) 2024-04-22T08:16:24.478 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.477+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:16:24.478 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.477+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=56} crc {osdmap=2006418806}) 2024-04-22T08:16:24.478 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.477+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=56} crc {osdmap=3551675345}) 2024-04-22T08:16:24.478 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.477+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:16:24.479 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.477+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=56} crc {osdmap=2006418806}) 2024-04-22T08:16:24.479 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:16:24.477+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=58} crc {osdmap=1592840646}) 2024-04-22T08:16:24.479 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-22T08:16:25.039 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:16:25.039 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36448]: starting. 2024-04-22T08:16:25.039 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36448]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-36413 2024-04-22T08:16:25.039 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36448]: created object 0... 2024-04-22T08:16:25.039 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36448]: created object 25... 2024-04-22T08:16:25.039 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36448]: created object 49... 2024-04-22T08:16:25.039 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36448]: finishing. 2024-04-22T08:16:25.039 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36448]: shutting down. 2024-04-22T08:16:25.079 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:16:25.079 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36450]: starting. 2024-04-22T08:16:25.081 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36450]: listing objects. 2024-04-22T08:16:25.081 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36450]: listed object 0... 2024-04-22T08:16:25.081 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36450]: listed object 25... 2024-04-22T08:16:25.081 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36450]: saw 50 objects 2024-04-22T08:16:25.081 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36450]: shutting down. 2024-04-22T08:16:25.480 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-22T08:16:25.480 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:16:25.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.587+0000 7f62e3c2f640 10 monclient: get_monmap_and_config 2024-04-22T08:16:25.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.587+0000 7f62e3c2f640 10 monclient: build_initial_monmap 2024-04-22T08:16:25.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.587+0000 7f62e3c2f640 10 monclient: monmap: 2024-04-22T08:16:25.587 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:16:25.587 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:25.587 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:16:25.588529+0000 2024-04-22T08:16:25.587 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:16:25.588529+0000 2024-04-22T08:16:25.587 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:16:25.587 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:25.588 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:16:25.588 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:16:25.588 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:16:25.588 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:16:25.588 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:16:25.588 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:25.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.588+0000 7f62e3c2f640 1 Processor -- start 2024-04-22T08:16:25.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.588+0000 7f62e3c2f640 1 -- start start 2024-04-22T08:16:25.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.588+0000 7f62e3c2f640 10 monclient: init 2024-04-22T08:16:25.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.588+0000 7f62e3c2f640 10 monclient: _reopen_session rank -1 2024-04-22T08:16:25.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.588+0000 7f62e3c2f640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-04-22T08:16:25.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e3c2f640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f62dc135e30 0x7f62dc136210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:25.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e3c2f640 10 monclient(hunting): picked mon.noname-a con 0x7f62dc135e30 addr v2:172.21.15.40:3300/0 2024-04-22T08:16:25.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e3c2f640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f62dc071c90 0x7f62dc136750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:25.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e3c2f640 10 monclient(hunting): picked mon.noname-c con 0x7f62dc071c90 addr v2:172.21.15.189:3300/0 2024-04-22T08:16:25.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e3c2f640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f62dc0a19a0 0x7f62dc0a1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:25.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e3c2f640 10 monclient(hunting): picked mon.noname-b con 0x7f62dc0a19a0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:16:25.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e3c2f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:25.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e3c2f640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62dc09e310 con 0x7f62dc135e30 2024-04-22T08:16:25.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e3c2f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:25.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e3c2f640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62dc09de80 con 0x7f62dc071c90 2024-04-22T08:16:25.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e3c2f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:25.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e3c2f640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc09f130 con 0x7f62dc0a19a0 2024-04-22T08:16:25.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e3c2f640 10 monclient(hunting): _renew_subs 2024-04-22T08:16:25.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e3c2f640 10 monclient(hunting): authenticate will time out at 1059.582833s 2024-04-22T08:16:25.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e21cc640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f62dc0a19a0 0x7f62dc0a1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:25.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e19cb640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f62dc135e30 0x7f62dc136210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:25.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e21cc640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f62dc0a19a0 0x7f62dc0a1de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:33214/0 (socket says 172.21.15.40:33214) 2024-04-22T08:16:25.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e21cc640 1 -- 172.21.15.40:0/1511513137 learned_addr learned my addr 172.21.15.40:0/1511513137 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:16:25.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e21cc640 10 monclient(hunting): get_auth_request con 0x7f62dc0a19a0 auth_method 0 2024-04-22T08:16:25.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e21cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:25.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e21cc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:25.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e21cc640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:25.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e19cb640 10 monclient(hunting): get_auth_request con 0x7f62dc135e30 auth_method 0 2024-04-22T08:16:25.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e19cb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:25.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e19cb640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:25.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e19cb640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:25.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e21cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:25.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e21cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:25.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e21cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:25.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e11ca640 1 --2- 172.21.15.40:0/1511513137 >> v2:172.21.15.189:3300/0 conn(0x7f62dc071c90 0x7f62dc136750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:25.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e19cb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:25.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e19cb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:25.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e19cb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:25.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e11ca640 10 monclient(hunting): get_auth_request con 0x7f62dc071c90 auth_method 0 2024-04-22T08:16:25.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e11ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:25.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e11ca640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:25.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e11ca640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:25.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e21cc640 10 monclient(hunting): handle_auth_done global_id 54149 payload 274 2024-04-22T08:16:25.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e21cc640 1 -- 172.21.15.40:0/1511513137 >> v2:172.21.15.189:3300/0 conn(0x7f62dc071c90 msgr2=0x7f62dc136750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:16:25.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e21cc640 1 --2- 172.21.15.40:0/1511513137 >> v2:172.21.15.189:3300/0 conn(0x7f62dc071c90 0x7f62dc136750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:25.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e21cc640 1 -- 172.21.15.40:0/1511513137 >> v2:172.21.15.40:3300/0 conn(0x7f62dc135e30 msgr2=0x7f62dc136210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:16:25.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e21cc640 1 --2- 172.21.15.40:0/1511513137 >> v2:172.21.15.40:3300/0 conn(0x7f62dc135e30 0x7f62dc136210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:25.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e21cc640 10 monclient: _finish_hunting 0 2024-04-22T08:16:25.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e21cc640 1 monclient: found mon.noname-b 2024-04-22T08:16:25.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e21cc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:16:25.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e21cc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:16:25.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e21cc640 1 -- 172.21.15.40:0/1511513137 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62dc09f410 con 0x7f62dc0a19a0 2024-04-22T08:16:25.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e11ca640 1 --2- 172.21.15.40:0/1511513137 >> v2:172.21.15.189:3300/0 conn(0x7f62dc071c90 0x7f62dc136750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T08:16:25.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.589+0000 7f62e19cb640 1 --2- 172.21.15.40:0/1511513137 >> v2:172.21.15.40:3300/0 conn(0x7f62dc135e30 0x7f62dc136210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:16:25.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.592+0000 7f62e21cc640 1 -- 172.21.15.40:0/1511513137 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f62dc0a19a0 msgr2=0x7f62dc0a1de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:16:25.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.592+0000 7f62e21cc640 1 --2- 172.21.15.40:0/1511513137 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f62dc0a19a0 0x7f62dc0a1de0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f62d4009e50 tx=0x7f62d402bdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3375ec28381a055 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:25.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.592+0000 7f62e09c9640 1 -- 172.21.15.40:0/1511513137 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d402cb60 con 0x7f62dc0a19a0 2024-04-22T08:16:25.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.592+0000 7f62e09c9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:25.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.592+0000 7f62e09c9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:16:25.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.592+0000 7f62e09c9640 10 monclient: dump: 2024-04-22T08:16:25.601 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:25.601 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:25.601 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:25.601 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:25.601 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:25.601 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:25.601 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:25.601 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:25.602 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:25.602 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:25.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.592+0000 7f62e09c9640 10 monclient: _finish_auth 0 2024-04-22T08:16:25.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.592+0000 7f62e09c9640 10 monclient: _check_auth_tickets 2024-04-22T08:16:25.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.592+0000 7f62e09c9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:16:25.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.592+0000 7f62e09c9640 1 -- 172.21.15.40:0/1511513137 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62d402ccc0 con 0x7f62dc0a19a0 2024-04-22T08:16:25.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.592+0000 7f62e09c9640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:16:25.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.592+0000 7f62e09c9640 1 -- 172.21.15.40:0/1511513137 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d4034610 con 0x7f62dc0a19a0 2024-04-22T08:16:25.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.592+0000 7f62e3c2f640 5 monclient: authenticate success, global_id 54149 2024-04-22T08:16:25.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.592+0000 7f62e3c2f640 10 monclient: get_monmap_and_config success 2024-04-22T08:16:25.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.592+0000 7f62e3c2f640 4 set_mon_vals no callback set 2024-04-22T08:16:25.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.592+0000 7f62e09c9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:25.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.592+0000 7f62e09c9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:16:25.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.592+0000 7f62e09c9640 10 monclient: dump: 2024-04-22T08:16:25.603 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:25.603 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:25.603 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:25.603 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:25.603 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:25.603 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:25.603 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:25.603 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:25.603 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:25.603 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:25.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.593+0000 7f62e3c2f640 10 monclient: shutdown 2024-04-22T08:16:25.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.593+0000 7f62e3c2f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:16:25.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.593+0000 7f62e3c2f640 1 -- 172.21.15.40:0/1511513137 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f62dc0a19a0 msgr2=0x7f62dc0a1de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:25.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.593+0000 7f62e3c2f640 1 --2- 172.21.15.40:0/1511513137 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f62dc0a19a0 0x7f62dc0a1de0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f62d4009e50 tx=0x7f62d402bdb0 comp rx=0 tx=0).stop 2024-04-22T08:16:25.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.593+0000 7f62e3c2f640 1 -- 172.21.15.40:0/1511513137 shutdown_connections 2024-04-22T08:16:25.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.593+0000 7f62e3c2f640 1 --2- 172.21.15.40:0/1511513137 >> v2:172.21.15.189:3300/0 conn(0x7f62dc071c90 0x7f62dc136750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:25.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.593+0000 7f62e3c2f640 1 --2- 172.21.15.40:0/1511513137 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f62dc0a19a0 0x7f62dc0a1de0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:25.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.593+0000 7f62e3c2f640 1 --2- 172.21.15.40:0/1511513137 >> v2:172.21.15.40:3300/0 conn(0x7f62dc135e30 0x7f62dc136210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:25.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.593+0000 7f62e3c2f640 1 -- 172.21.15.40:0/1511513137 >> 172.21.15.40:0/1511513137 conn(0x7f62dc0a65c0 msgr2=0x7f62dc0a2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:16:25.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.593+0000 7f62e3c2f640 1 -- 172.21.15.40:0/1511513137 shutdown_connections 2024-04-22T08:16:25.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.593+0000 7f62e3c2f640 1 -- 172.21.15.40:0/1511513137 wait complete. 2024-04-22T08:16:25.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.593+0000 7f62e3c2f640 10 monclient: build_initial_monmap 2024-04-22T08:16:25.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.593+0000 7f62e3c2f640 10 monclient: monmap: 2024-04-22T08:16:25.604 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:16:25.604 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:16:25.604 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:16:25.604 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:16:25.605 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:16:25.605 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:25.605 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:16:25.605 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:16:25.605 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:16:25.605 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:25.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e3c2f640 1 Processor -- start 2024-04-22T08:16:25.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e3c2f640 1 -- start start 2024-04-22T08:16:25.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e3c2f640 10 monclient: init 2024-04-22T08:16:25.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e3c2f640 10 monclient: _reopen_session rank -1 2024-04-22T08:16:25.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e3c2f640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-22T08:16:25.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e3c2f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62dc071c90 0x7f62dc1211d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:25.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e3c2f640 10 monclient(hunting): picked mon.noname-a con 0x7f62dc071c90 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:16:25.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e3c2f640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f62dc0a19a0 0x7f62dc1c7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:25.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e3c2f640 10 monclient(hunting): picked mon.noname-c con 0x7f62dc0a19a0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:16:25.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e3c2f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62dc135e30 0x7f62dc135010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:25.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e3c2f640 10 monclient(hunting): picked mon.noname-b con 0x7f62dc135e30 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:16:25.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e3c2f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:25.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e3c2f640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc1c8350 con 0x7f62dc071c90 2024-04-22T08:16:25.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e3c2f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:25.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e3c2f640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc09c930 con 0x7f62dc135e30 2024-04-22T08:16:25.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e3c2f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:25.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e3c2f640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc1226f0 con 0x7f62dc0a19a0 2024-04-22T08:16:25.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e3c2f640 10 monclient(hunting): _renew_subs 2024-04-22T08:16:25.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e3c2f640 10 monclient(hunting): authenticate will time out at 1059.588349s 2024-04-22T08:16:25.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e19cb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62dc071c90 0x7f62dc1211d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:25.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e11ca640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f62dc0a19a0 0x7f62dc1c7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:25.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e19cb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62dc071c90 0x7f62dc1211d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41580/0 (socket says 172.21.15.40:41580) 2024-04-22T08:16:25.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e19cb640 1 -- 172.21.15.40:0/2425164189 learned_addr learned my addr 172.21.15.40:0/2425164189 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:16:25.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e19cb640 10 monclient(hunting): get_auth_request con 0x7f62dc071c90 auth_method 0 2024-04-22T08:16:25.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e19cb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:25.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e19cb640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:25.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e19cb640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:25.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e11ca640 10 monclient(hunting): get_auth_request con 0x7f62dc0a19a0 auth_method 0 2024-04-22T08:16:25.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e11ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:25.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e11ca640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:25.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e11ca640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:25.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e19cb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:25.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e19cb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:25.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e19cb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:25.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e11ca640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:25.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e11ca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:25.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e11ca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:25.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e19cb640 1 -- 172.21.15.40:0/2425164189 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62dc071c90 msgr2=0x7f62dc1211d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-22T08:16:25.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e19cb640 1 -- 172.21.15.40:0/2425164189 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62dc071c90 msgr2=0x7f62dc1211d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:16:25.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e19cb640 1 --2- 172.21.15.40:0/2425164189 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62dc071c90 0x7f62dc1211d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:16:25.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e21cc640 1 --2- 172.21.15.40:0/2425164189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62dc135e30 0x7f62dc135010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:25.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e21cc640 10 monclient(hunting): get_auth_request con 0x7f62dc135e30 auth_method 0 2024-04-22T08:16:25.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e21cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:25.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e21cc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:25.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e21cc640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:25.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e19cb640 1 --2- 172.21.15.40:0/2425164189 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62dc071c90 0x7f62dc1211d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:16:25.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e21cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:25.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e21cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:25.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.594+0000 7f62e21cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:25.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.595+0000 7f62e11ca640 10 monclient(hunting): handle_auth_done global_id 54152 payload 824 2024-04-22T08:16:25.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.595+0000 7f62e11ca640 1 -- 172.21.15.40:0/2425164189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62dc135e30 msgr2=0x7f62dc135010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:25.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.595+0000 7f62e11ca640 1 --2- 172.21.15.40:0/2425164189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62dc135e30 0x7f62dc135010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:25.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.595+0000 7f62e11ca640 1 -- 172.21.15.40:0/2425164189 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62dc071c90 msgr2=0x7f62dc1211d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:16:25.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.595+0000 7f62e11ca640 1 --2- 172.21.15.40:0/2425164189 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62dc071c90 0x7f62dc1211d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:25.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.595+0000 7f62e11ca640 10 monclient: _finish_hunting 0 2024-04-22T08:16:25.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.595+0000 7f62e11ca640 1 monclient: found mon.noname-c 2024-04-22T08:16:25.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.595+0000 7f62e11ca640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:16:25.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.595+0000 7f62e11ca640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:16:25.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.595+0000 7f62e11ca640 1 -- 172.21.15.40:0/2425164189 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d8009400 con 0x7f62dc0a19a0 2024-04-22T08:16:25.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.597+0000 7f62e11ca640 1 -- 172.21.15.40:0/2425164189 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f62dc0a19a0 msgr2=0x7f62dc1c7890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:16:25.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.597+0000 7f62e11ca640 1 --2- 172.21.15.40:0/2425164189 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f62dc0a19a0 0x7f62dc1c7890 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f62dc0a2a60 tx=0x7f62cc00ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:25.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.597+0000 7f62d27fc640 1 -- 172.21.15.40:0/2425164189 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62cc00c8d0 con 0x7f62dc0a19a0 2024-04-22T08:16:25.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.597+0000 7f62d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:25.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.597+0000 7f62d27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:16:25.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.597+0000 7f62d27fc640 10 monclient: dump: 2024-04-22T08:16:25.610 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:25.610 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:25.610 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:25.610 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:25.610 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:25.610 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:25.610 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:25.610 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:25.610 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:25.610 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:25.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.597+0000 7f62d27fc640 10 monclient: _finish_auth 0 2024-04-22T08:16:25.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.597+0000 7f62d27fc640 10 monclient: _check_auth_tickets 2024-04-22T08:16:25.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.597+0000 7f62d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:16:25.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.597+0000 7f62e3c2f640 5 monclient: authenticate success, global_id 54152 2024-04-22T08:16:25.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.597+0000 7f62d27fc640 1 -- 172.21.15.40:0/2425164189 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62cc00ca30 con 0x7f62dc0a19a0 2024-04-22T08:16:25.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.597+0000 7f62d27fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:16:25.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.597+0000 7f62e3c2f640 10 monclient: _renew_subs 2024-04-22T08:16:25.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.597+0000 7f62d27fc640 1 -- 172.21.15.40:0/2425164189 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62cc00cde0 con 0x7f62dc0a19a0 2024-04-22T08:16:25.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.597+0000 7f62e3c2f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:16:25.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.597+0000 7f62e3c2f640 1 -- 172.21.15.40:0/2425164189 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62d4009b30 con 0x7f62dc0a19a0 2024-04-22T08:16:25.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.597+0000 7f62d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:25.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.597+0000 7f62d27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:16:25.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.597+0000 7f62d27fc640 10 monclient: dump: 2024-04-22T08:16:25.611 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:25.612 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:25.612 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:25.612 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:25.612 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:25.612 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:25.612 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:25.612 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:25.612 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:25.612 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:25.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.597+0000 7f62d37fe640 4 set_mon_vals no callback set 2024-04-22T08:16:25.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.597+0000 7f62e3c2f640 10 monclient: _renew_subs 2024-04-22T08:16:25.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.597+0000 7f62e3c2f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:16:25.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.597+0000 7f62e3c2f640 1 -- 172.21.15.40:0/2425164189 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62dc1359d0 con 0x7f62dc0a19a0 2024-04-22T08:16:25.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.598+0000 7f62e11ca640 1 --2- 172.21.15.40:0/2425164189 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f62dc0a19a0 0x7f62dc1c7890 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f62dc0a2a60 tx=0x7f62cc00ec70 comp rx=0 tx=0).queue_received will delay after 1.71377e+09 on 0x7f62cc013410 mgrmap(e 5) v1 2024-04-22T08:16:25.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.598+0000 7f62d27fc640 1 -- 172.21.15.40:0/2425164189 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f62cc013410 con 0x7f62dc0a19a0 2024-04-22T08:16:25.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.598+0000 7f62d27fc640 1 --2- 172.21.15.40:0/2425164189 >> v2:172.21.15.189:6800/30168 conn(0x7f62c00377d0 0x7f62c0039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:25.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.598+0000 7f62e3c2f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:16:25.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.598+0000 7f62e3c2f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:16:25.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.598+0000 7f62e3c2f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:16:25.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.598+0000 7f62e3c2f640 1 -- 172.21.15.40:0/2425164189 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62dc0999f0 con 0x7f62dc0a19a0 2024-04-22T08:16:25.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.598+0000 7f62e19cb640 1 --2- 172.21.15.40:0/2425164189 >> v2:172.21.15.189:6800/30168 conn(0x7f62c00377d0 0x7f62c0039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:25.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.598+0000 7f62e19cb640 10 monclient: get_auth_request con 0x7f62c00377d0 auth_method 0 2024-04-22T08:16:25.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.601+0000 7f62e19cb640 1 --2- 172.21.15.40:0/2425164189 >> v2:172.21.15.189:6800/30168 conn(0x7f62c00377d0 0x7f62c0039c90 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f62d80098c0 tx=0x7f62d8009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:25.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:25.603+0000 7f62d27fc640 1 -- 172.21.15.40:0/2425164189 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f62cc04a190 con 0x7f62dc0a19a0 2024-04-22T08:16:25.931 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:16:25.931 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[36449]: starting. 2024-04-22T08:16:25.931 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[36449]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-36413 2024-04-22T08:16:25.931 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[36449]: shutting down. 2024-04-22T08:16:25.933 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:16:25.933 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:16:25.933 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:16:25.936 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:16:26.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.426+0000 7f62d27fc640 1 -- 172.21.15.40:0/2425164189 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f62cc0136c0 con 0x7f62dc0a19a0 2024-04-22T08:16:26.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.426+0000 7f62d27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:16:26.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.426+0000 7f62d27fc640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:16:26.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.681+0000 7f62e3c2f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:16:26.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.681+0000 7f62e3c2f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:16:26.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.681+0000 7f62e3c2f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:16:26.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.681+0000 7f62e3c2f640 1 -- 172.21.15.40:0/2425164189 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f62dc04e9b0 con 0x7f62dc0a19a0 2024-04-22T08:16:26.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.681+0000 7f62d27fc640 1 -- 172.21.15.40:0/2425164189 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f62cc01bba0 con 0x7f62dc0a19a0 2024-04-22T08:16:26.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.681+0000 7f62d27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:16:26.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.681+0000 7f62d27fc640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:16:26.681 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":42,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:16:26.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.683+0000 7f62e3c2f640 1 -- 172.21.15.40:0/2425164189 >> v2:172.21.15.189:6800/30168 conn(0x7f62c00377d0 msgr2=0x7f62c0039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:26.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.683+0000 7f62e3c2f640 1 --2- 172.21.15.40:0/2425164189 >> v2:172.21.15.189:6800/30168 conn(0x7f62c00377d0 0x7f62c0039c90 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f62d80098c0 tx=0x7f62d8009190 comp rx=0 tx=0).stop 2024-04-22T08:16:26.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.683+0000 7f62e3c2f640 10 monclient: shutdown 2024-04-22T08:16:26.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.683+0000 7f62e3c2f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:16:26.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.683+0000 7f62e3c2f640 1 -- 172.21.15.40:0/2425164189 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f62dc0a19a0 msgr2=0x7f62dc1c7890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:26.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.683+0000 7f62e3c2f640 1 --2- 172.21.15.40:0/2425164189 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f62dc0a19a0 0x7f62dc1c7890 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f62dc0a2a60 tx=0x7f62cc00ec70 comp rx=0 tx=0).stop 2024-04-22T08:16:26.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.683+0000 7f62e3c2f640 1 -- 172.21.15.40:0/2425164189 shutdown_connections 2024-04-22T08:16:26.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.683+0000 7f62e3c2f640 1 --2- 172.21.15.40:0/2425164189 >> v2:172.21.15.189:6800/30168 conn(0x7f62c00377d0 0x7f62c0039c90 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:26.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.683+0000 7f62e3c2f640 1 --2- 172.21.15.40:0/2425164189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62dc135e30 0x7f62dc135010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:26.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.683+0000 7f62e3c2f640 1 --2- 172.21.15.40:0/2425164189 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f62dc0a19a0 0x7f62dc1c7890 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:26.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.683+0000 7f62e3c2f640 1 --2- 172.21.15.40:0/2425164189 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62dc071c90 0x7f62dc1211d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:26.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.683+0000 7f62e3c2f640 1 -- 172.21.15.40:0/2425164189 >> 172.21.15.40:0/2425164189 conn(0x7f62dc0a65c0 msgr2=0x7f62dc0a55b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:16:26.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.683+0000 7f62e3c2f640 1 -- 172.21.15.40:0/2425164189 shutdown_connections 2024-04-22T08:16:26.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.683+0000 7f62e3c2f640 1 -- 172.21.15.40:0/2425164189 wait complete. 2024-04-22T08:16:26.693 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-22T08:16:26.693 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-22T08:16:26.693 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-22T08:16:26.693 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-22T08:16:26.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.793+0000 7f7e77fcd640 10 monclient: get_monmap_and_config 2024-04-22T08:16:26.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.793+0000 7f7e77fcd640 10 monclient: build_initial_monmap 2024-04-22T08:16:26.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.793+0000 7f7e77fcd640 10 monclient: monmap: 2024-04-22T08:16:26.793 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:16:26.793 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:26.793 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:16:26.794568+0000 2024-04-22T08:16:26.793 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:16:26.794568+0000 2024-04-22T08:16:26.793 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:16:26.793 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:26.794 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:16:26.794 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:16:26.794 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:16:26.794 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:16:26.794 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:16:26.794 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:26.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.794+0000 7f7e77fcd640 1 Processor -- start 2024-04-22T08:16:26.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.794+0000 7f7e77fcd640 1 -- start start 2024-04-22T08:16:26.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.794+0000 7f7e77fcd640 10 monclient: init 2024-04-22T08:16:26.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.794+0000 7f7e77fcd640 10 monclient: _reopen_session rank -1 2024-04-22T08:16:26.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.794+0000 7f7e77fcd640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-04-22T08:16:26.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.794+0000 7f7e77fcd640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7e700a5a10 0x7f7e700a32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:26.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.794+0000 7f7e77fcd640 10 monclient(hunting): picked mon.noname-a con 0x7f7e700a5a10 addr v2:172.21.15.40:3300/0 2024-04-22T08:16:26.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.794+0000 7f7e77fcd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7e700a1970 addr v1:172.21.15.189:6789/0 2024-04-22T08:16:26.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.794+0000 7f7e77fcd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7e700a1dc0 0x7f7e70136330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:26.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.795+0000 7f7e77fcd640 10 monclient(hunting): picked mon.noname-b con 0x7f7e700a1dc0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:16:26.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.795+0000 7f7e77fcd640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e7009fe80 con 0x7f7e700a1970 2024-04-22T08:16:26.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.795+0000 7f7e77fcd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:26.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.795+0000 7f7e77fcd640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e700a37f0 con 0x7f7e700a5a10 2024-04-22T08:16:26.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.795+0000 7f7e77fcd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:26.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.795+0000 7f7e77fcd640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e700a25b0 con 0x7f7e700a1dc0 2024-04-22T08:16:26.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.795+0000 7f7e77fcd640 10 monclient(hunting): _renew_subs 2024-04-22T08:16:26.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.795+0000 7f7e77fcd640 10 monclient(hunting): authenticate will time out at 1060.788808s 2024-04-22T08:16:26.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.795+0000 7f7e7656a640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7e700a1dc0 0x7f7e70136330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:26.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.795+0000 7f7e7656a640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7e700a1dc0 0x7f7e70136330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:33228/0 (socket says 172.21.15.40:33228) 2024-04-22T08:16:26.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.795+0000 7f7e75d69640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7e700a5a10 0x7f7e700a32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:26.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.795+0000 7f7e7656a640 1 -- 172.21.15.40:0/362669474 learned_addr learned my addr 172.21.15.40:0/362669474 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:16:26.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.795+0000 7f7e7656a640 10 monclient(hunting): get_auth_request con 0x7f7e700a1dc0 auth_method 0 2024-04-22T08:16:26.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.795+0000 7f7e7656a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:26.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.795+0000 7f7e7656a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:26.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.795+0000 7f7e7656a640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:26.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.795+0000 7f7e75d69640 10 monclient(hunting): get_auth_request con 0x7f7e700a5a10 auth_method 0 2024-04-22T08:16:26.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.795+0000 7f7e75d69640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:26.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.795+0000 7f7e75d69640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:26.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.795+0000 7f7e75d69640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:26.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.795+0000 7f7e7656a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:26.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.795+0000 7f7e7656a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:26.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.795+0000 7f7e7656a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:26.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.795+0000 7f7e75d69640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:26.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.795+0000 7f7e75d69640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:26.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.795+0000 7f7e75d69640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:26.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.795+0000 7f7e7656a640 10 monclient(hunting): handle_auth_done global_id 54167 payload 274 2024-04-22T08:16:26.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.795+0000 7f7e7656a640 1 -- 172.21.15.40:0/362669474 >> v2:172.21.15.40:3300/0 conn(0x7f7e700a5a10 msgr2=0x7f7e700a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:16:26.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.795+0000 7f7e7656a640 1 --2- 172.21.15.40:0/362669474 >> v2:172.21.15.40:3300/0 conn(0x7f7e700a5a10 0x7f7e700a32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:26.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.795+0000 7f7e7656a640 1 -- 172.21.15.40:0/362669474 >> v1:172.21.15.189:6789/0 conn(0x7f7e700a1970 legacy=0x7f7e70135c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:16:26.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.795+0000 7f7e7656a640 10 monclient: _finish_hunting 0 2024-04-22T08:16:26.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.795+0000 7f7e7656a640 1 monclient: found mon.noname-b 2024-04-22T08:16:26.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.795+0000 7f7e7656a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:16:26.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.795+0000 7f7e7656a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:16:26.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.795+0000 7f7e7656a640 1 -- 172.21.15.40:0/362669474 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e7009fba0 con 0x7f7e700a1dc0 2024-04-22T08:16:26.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.795+0000 7f7e75d69640 1 --2- 172.21.15.40:0/362669474 >> v2:172.21.15.40:3300/0 conn(0x7f7e700a5a10 0x7f7e700a32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:16:26.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.798+0000 7f7e7656a640 1 -- 172.21.15.40:0/362669474 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7e700a1dc0 msgr2=0x7f7e70136330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:16:26.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.798+0000 7f7e7656a640 1 --2- 172.21.15.40:0/362669474 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7e700a1dc0 0x7f7e70136330 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7e7006c3a0 tx=0x7f7e6802bd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6155b065fc0adb92 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:26.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.798+0000 7f7e74d67640 1 -- 172.21.15.40:0/362669474 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6802cb20 con 0x7f7e700a1dc0 2024-04-22T08:16:26.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.798+0000 7f7e74d67640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:26.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.798+0000 7f7e74d67640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:16:26.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.798+0000 7f7e74d67640 10 monclient: dump: 2024-04-22T08:16:26.808 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:26.808 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:26.808 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:26.808 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:26.808 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:26.808 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:26.808 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:26.808 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:26.808 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:26.808 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:26.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.798+0000 7f7e74d67640 10 monclient: _finish_auth 0 2024-04-22T08:16:26.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.798+0000 7f7e74d67640 10 monclient: _check_auth_tickets 2024-04-22T08:16:26.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.798+0000 7f7e74d67640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:16:26.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.798+0000 7f7e74d67640 1 -- 172.21.15.40:0/362669474 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e6802cc80 con 0x7f7e700a1dc0 2024-04-22T08:16:26.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.798+0000 7f7e77fcd640 5 monclient: authenticate success, global_id 54167 2024-04-22T08:16:26.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.798+0000 7f7e77fcd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:16:26.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.798+0000 7f7e74d67640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:16:26.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.798+0000 7f7e74d67640 1 -- 172.21.15.40:0/362669474 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e680343f0 con 0x7f7e700a1dc0 2024-04-22T08:16:26.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.798+0000 7f7e77fcd640 10 monclient: get_monmap_and_config success 2024-04-22T08:16:26.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.798+0000 7f7e74d67640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:26.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.798+0000 7f7e77fcd640 4 set_mon_vals no callback set 2024-04-22T08:16:26.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.798+0000 7f7e74d67640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:16:26.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.798+0000 7f7e74d67640 10 monclient: dump: 2024-04-22T08:16:26.809 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:26.809 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:26.809 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:26.809 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:26.810 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:26.810 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:26.810 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:26.810 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:26.810 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:26.810 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:26.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.798+0000 7f7e77fcd640 10 monclient: shutdown 2024-04-22T08:16:26.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.798+0000 7f7e77fcd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:16:26.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.798+0000 7f7e77fcd640 1 -- 172.21.15.40:0/362669474 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7e700a1dc0 msgr2=0x7f7e70136330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:26.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.798+0000 7f7e77fcd640 1 --2- 172.21.15.40:0/362669474 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7e700a1dc0 0x7f7e70136330 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7e7006c3a0 tx=0x7f7e6802bd70 comp rx=0 tx=0).stop 2024-04-22T08:16:26.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.799+0000 7f7e77fcd640 1 -- 172.21.15.40:0/362669474 shutdown_connections 2024-04-22T08:16:26.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.799+0000 7f7e77fcd640 1 --2- 172.21.15.40:0/362669474 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7e700a1dc0 0x7f7e70136330 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:26.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.799+0000 7f7e77fcd640 1 --2- 172.21.15.40:0/362669474 >> v2:172.21.15.40:3300/0 conn(0x7f7e700a5a10 0x7f7e700a32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:26.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.799+0000 7f7e77fcd640 1 -- 172.21.15.40:0/362669474 >> 172.21.15.40:0/362669474 conn(0x7f7e700a4ef0 msgr2=0x7f7e700a5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:16:26.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.799+0000 7f7e77fcd640 1 -- 172.21.15.40:0/362669474 shutdown_connections 2024-04-22T08:16:26.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.799+0000 7f7e77fcd640 1 -- 172.21.15.40:0/362669474 wait complete. 2024-04-22T08:16:26.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.799+0000 7f7e77fcd640 10 monclient: build_initial_monmap 2024-04-22T08:16:26.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.799+0000 7f7e77fcd640 10 monclient: monmap: 2024-04-22T08:16:26.811 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:16:26.811 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:16:26.811 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:16:26.811 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:16:26.811 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:16:26.811 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:26.811 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:16:26.811 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:16:26.811 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:16:26.812 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:26.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.799+0000 7f7e77fcd640 1 Processor -- start 2024-04-22T08:16:26.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e77fcd640 1 -- start start 2024-04-22T08:16:26.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e77fcd640 10 monclient: init 2024-04-22T08:16:26.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e77fcd640 10 monclient: _reopen_session rank -1 2024-04-22T08:16:26.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e77fcd640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-22T08:16:26.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e77fcd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e700a1970 0x7f7e701caaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:26.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e77fcd640 10 monclient(hunting): picked mon.noname-a con 0x7f7e700a1970 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:16:26.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e77fcd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7e700a1dc0 0x7f7e701cb030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:26.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e77fcd640 10 monclient(hunting): picked mon.noname-c con 0x7f7e700a1dc0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:16:26.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e77fcd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e700a5a10 0x7f7e701cb570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:26.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e77fcd640 10 monclient(hunting): picked mon.noname-b con 0x7f7e700a5a10 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:16:26.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e77fcd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:26.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e77fcd640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e70120c90 con 0x7f7e700a1970 2024-04-22T08:16:26.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e77fcd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:26.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e77fcd640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e70122aa0 con 0x7f7e700a5a10 2024-04-22T08:16:26.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e77fcd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:26.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e77fcd640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e700a27c0 con 0x7f7e700a1dc0 2024-04-22T08:16:26.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e77fcd640 10 monclient(hunting): _renew_subs 2024-04-22T08:16:26.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e77fcd640 10 monclient(hunting): authenticate will time out at 1060.794238s 2024-04-22T08:16:26.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e75568640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7e700a1dc0 0x7f7e701cb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:26.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e75d69640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e700a1970 0x7f7e701caaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:26.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e75d69640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e700a1970 0x7f7e701caaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41602/0 (socket says 172.21.15.40:41602) 2024-04-22T08:16:26.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e75d69640 1 -- 172.21.15.40:0/897633986 learned_addr learned my addr 172.21.15.40:0/897633986 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:16:26.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e75d69640 10 monclient(hunting): get_auth_request con 0x7f7e700a1970 auth_method 0 2024-04-22T08:16:26.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e75d69640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:26.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e75d69640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:26.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e75d69640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:26.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e75568640 10 monclient(hunting): get_auth_request con 0x7f7e700a1dc0 auth_method 0 2024-04-22T08:16:26.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e75568640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:26.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e75568640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:26.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e75568640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:26.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e75568640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:26.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e75568640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:26.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e75568640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:26.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e75d69640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:26.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e75d69640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:26.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e75d69640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:26.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e7656a640 1 --2- 172.21.15.40:0/897633986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e700a5a10 0x7f7e701cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:26.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e75568640 10 monclient(hunting): handle_auth_done global_id 54170 payload 824 2024-04-22T08:16:26.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e75568640 1 -- 172.21.15.40:0/897633986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e700a5a10 msgr2=0x7f7e701cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:26.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e75568640 1 --2- 172.21.15.40:0/897633986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e700a5a10 0x7f7e701cb570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:26.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e75568640 1 -- 172.21.15.40:0/897633986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e700a1970 msgr2=0x7f7e701caaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:26.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e75568640 1 --2- 172.21.15.40:0/897633986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e700a1970 0x7f7e701caaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:26.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e75568640 10 monclient: _finish_hunting 0 2024-04-22T08:16:26.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e75568640 1 monclient: found mon.noname-c 2024-04-22T08:16:26.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e75568640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:16:26.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e75568640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:16:26.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e75568640 1 -- 172.21.15.40:0/897633986 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e6c009400 con 0x7f7e700a1dc0 2024-04-22T08:16:26.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.800+0000 7f7e75d69640 1 --2- 172.21.15.40:0/897633986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e700a1970 0x7f7e701caaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:16:26.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.803+0000 7f7e75568640 1 -- 172.21.15.40:0/897633986 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7e700a1dc0 msgr2=0x7f7e701cb030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:16:26.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.803+0000 7f7e75568640 1 --2- 172.21.15.40:0/897633986 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7e700a1dc0 0x7f7e701cb030 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7e6000c410 tx=0x7f7e6000c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:26.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.803+0000 7f7e667fc640 1 -- 172.21.15.40:0/897633986 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e60005230 con 0x7f7e700a1dc0 2024-04-22T08:16:26.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.803+0000 7f7e667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:26.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.803+0000 7f7e667fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:16:26.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.803+0000 7f7e667fc640 10 monclient: dump: 2024-04-22T08:16:26.816 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:26.816 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:26.816 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:26.816 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:26.816 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:26.816 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:26.816 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:26.816 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:26.816 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:26.816 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:26.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.803+0000 7f7e667fc640 10 monclient: _finish_auth 0 2024-04-22T08:16:26.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.803+0000 7f7e667fc640 10 monclient: _check_auth_tickets 2024-04-22T08:16:26.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.803+0000 7f7e667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:16:26.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.803+0000 7f7e667fc640 1 -- 172.21.15.40:0/897633986 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e60005390 con 0x7f7e700a1dc0 2024-04-22T08:16:26.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.803+0000 7f7e77fcd640 5 monclient: authenticate success, global_id 54170 2024-04-22T08:16:26.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.803+0000 7f7e77fcd640 10 monclient: _renew_subs 2024-04-22T08:16:26.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.803+0000 7f7e77fcd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:16:26.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.803+0000 7f7e77fcd640 1 -- 172.21.15.40:0/897633986 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e68009b30 con 0x7f7e700a1dc0 2024-04-22T08:16:26.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.803+0000 7f7e667fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:16:26.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.803+0000 7f7e667fc640 1 -- 172.21.15.40:0/897633986 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e600119e0 con 0x7f7e700a1dc0 2024-04-22T08:16:26.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.803+0000 7f7e667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:26.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.803+0000 7f7e667fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:16:26.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.803+0000 7f7e667fc640 10 monclient: dump: 2024-04-22T08:16:26.817 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:26.817 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:26.817 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:26.817 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:26.817 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:26.817 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:26.818 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:26.818 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:26.818 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:26.818 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:26.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.803+0000 7f7e67fff640 4 set_mon_vals no callback set 2024-04-22T08:16:26.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.803+0000 7f7e77fcd640 10 monclient: _renew_subs 2024-04-22T08:16:26.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.803+0000 7f7e77fcd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:16:26.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.803+0000 7f7e77fcd640 1 -- 172.21.15.40:0/897633986 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e70121da0 con 0x7f7e700a1dc0 2024-04-22T08:16:26.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.804+0000 7f7e667fc640 1 -- 172.21.15.40:0/897633986 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f7e60011bc0 con 0x7f7e700a1dc0 2024-04-22T08:16:26.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.804+0000 7f7e77fcd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:16:26.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.804+0000 7f7e77fcd640 1 --2- 172.21.15.40:0/897633986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e7006e280 0x7f7e7006e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:26.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.804+0000 7f7e77fcd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:26.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.804+0000 7f7e77fcd640 1 -- 172.21.15.40:0/897633986 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e700667a0 con 0x7f7e7006e280 2024-04-22T08:16:26.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.804+0000 7f7e667fc640 1 --2- 172.21.15.40:0/897633986 >> v2:172.21.15.189:6800/30168 conn(0x7f7e54037820 0x7f7e54039ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:26.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.804+0000 7f7e667fc640 1 -- 172.21.15.40:0/897633986 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f7e60049860 con 0x7f7e700a1dc0 2024-04-22T08:16:26.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.804+0000 7f7e75d69640 1 --2- 172.21.15.40:0/897633986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e7006e280 0x7f7e7006e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:26.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.804+0000 7f7e75d69640 10 monclient: get_auth_request con 0x7f7e7006e280 auth_method 0 2024-04-22T08:16:26.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.804+0000 7f7e75d69640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:26.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.804+0000 7f7e75d69640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:26.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.804+0000 7f7e75d69640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:26.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.804+0000 7f7e75d69640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:26.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.804+0000 7f7e75d69640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:26.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.804+0000 7f7e75d69640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:26.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.804+0000 7f7e75d69640 10 monclient(hunting): handle_auth_done global_id 34599 payload 824 2024-04-22T08:16:26.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.804+0000 7f7e75d69640 1 -- 172.21.15.40:0/897633986 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7e7009ee10 con 0x7f7e7006e280 2024-04-22T08:16:26.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.804+0000 7f7e7656a640 1 --2- 172.21.15.40:0/897633986 >> v2:172.21.15.189:6800/30168 conn(0x7f7e54037820 0x7f7e54039ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:26.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.804+0000 7f7e7656a640 10 monclient: get_auth_request con 0x7f7e54037820 auth_method 0 2024-04-22T08:16:26.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.806+0000 7f7e75d69640 1 -- 172.21.15.40:0/897633986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e7006e280 msgr2=0x7f7e7006e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:16:26.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.806+0000 7f7e75d69640 1 --2- 172.21.15.40:0/897633986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e7006e280 0x7f7e7006e660 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c00f4a0 tx=0x7f7e6c00f960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:26.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.806+0000 7f7e667fc640 1 -- 172.21.15.40:0/897633986 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c004030 con 0x7f7e7006e280 2024-04-22T08:16:26.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.806+0000 7f7e667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:26.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.806+0000 7f7e667fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:16:26.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.806+0000 7f7e667fc640 10 monclient: dump: 2024-04-22T08:16:26.820 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:26.820 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:26.820 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:26.820 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:26.820 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:26.820 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:26.820 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:26.820 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:26.820 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:26.820 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:26.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.807+0000 7f7e7656a640 1 --2- 172.21.15.40:0/897633986 >> v2:172.21.15.189:6800/30168 conn(0x7f7e54037820 0x7f7e54039ce0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f7e68035000 tx=0x7f7e68030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:26.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.807+0000 7f7e667fc640 1 -- 172.21.15.40:0/897633986 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f7e6c004190 con 0x7f7e7006e280 2024-04-22T08:16:26.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.807+0000 7f7e667fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:16:26.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.807+0000 7f7e667fc640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:16:26.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.807+0000 7f7e667fc640 1 -- 172.21.15.40:0/897633986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e7006e280 msgr2=0x7f7e7006e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:26.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.807+0000 7f7e667fc640 1 --2- 172.21.15.40:0/897633986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e7006e280 0x7f7e7006e660 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c00f4a0 tx=0x7f7e6c00f960 comp rx=0 tx=0).stop 2024-04-22T08:16:26.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.807+0000 7f7e667fc640 1 -- 172.21.15.40:0/897633986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e7006e280 msgr2=0x7f7e7006e660 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:16:26.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.807+0000 7f7e667fc640 1 --2- 172.21.15.40:0/897633986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e7006e280 0x7f7e7006e660 secure :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c00f4a0 tx=0x7f7e6c00f960 comp rx=0 tx=0).stop 2024-04-22T08:16:26.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.813+0000 7f7e77fcd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:16:26.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.813+0000 7f7e77fcd640 1 --2- 172.21.15.40:0/897633986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e7006eba0 0x7f7e7011f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:26.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.813+0000 7f7e77fcd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:26.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.813+0000 7f7e77fcd640 1 -- 172.21.15.40:0/897633986 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7004e9b0 con 0x7f7e7006eba0 2024-04-22T08:16:26.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.813+0000 7f7e75d69640 1 --2- 172.21.15.40:0/897633986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e7006eba0 0x7f7e7011f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:26.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.813+0000 7f7e75d69640 10 monclient: get_auth_request con 0x7f7e7006eba0 auth_method 0 2024-04-22T08:16:26.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.813+0000 7f7e75d69640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:26.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.813+0000 7f7e75d69640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:26.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.813+0000 7f7e75d69640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:26.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.813+0000 7f7e75d69640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:26.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.813+0000 7f7e75d69640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:26.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.813+0000 7f7e75d69640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:26.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.813+0000 7f7e75d69640 10 monclient(hunting): handle_auth_done global_id 34602 payload 824 2024-04-22T08:16:26.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.813+0000 7f7e75d69640 1 -- 172.21.15.40:0/897633986 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7e70122880 con 0x7f7e7006eba0 2024-04-22T08:16:26.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.816+0000 7f7e75d69640 1 -- 172.21.15.40:0/897633986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e7006eba0 msgr2=0x7f7e7011f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:16:26.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.816+0000 7f7e75d69640 1 --2- 172.21.15.40:0/897633986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e7006eba0 0x7f7e7011f0e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c00f4a0 tx=0x7f7e6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:26.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.816+0000 7f7e667fc640 1 -- 172.21.15.40:0/897633986 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c003d10 con 0x7f7e7006eba0 2024-04-22T08:16:26.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.816+0000 7f7e667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:26.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.816+0000 7f7e667fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:16:26.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.816+0000 7f7e667fc640 10 monclient: dump: 2024-04-22T08:16:26.825 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:26.825 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:26.825 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:26.825 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:26.825 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:26.825 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:26.825 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:26.825 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:26.825 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:26.825 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:26.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.816+0000 7f7e667fc640 1 -- 172.21.15.40:0/897633986 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4633 (secure 0 0 0) 0x7f7e6c023040 con 0x7f7e7006eba0 2024-04-22T08:16:26.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.816+0000 7f7e667fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:16:26.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.816+0000 7f7e667fc640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:16:26.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.816+0000 7f7e667fc640 1 -- 172.21.15.40:0/897633986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e7006eba0 msgr2=0x7f7e7011f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:26.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.816+0000 7f7e667fc640 1 --2- 172.21.15.40:0/897633986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e7006eba0 0x7f7e7011f0e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c00f4a0 tx=0x7f7e6c0046c0 comp rx=0 tx=0).stop 2024-04-22T08:16:26.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.816+0000 7f7e667fc640 1 -- 172.21.15.40:0/897633986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e7006eba0 msgr2=0x7f7e7011f0e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:16:26.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.816+0000 7f7e667fc640 1 --2- 172.21.15.40:0/897633986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e7006eba0 0x7f7e7011f0e0 secure :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c00f4a0 tx=0x7f7e6c0046c0 comp rx=0 tx=0).stop 2024-04-22T08:16:26.826 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:16:26.826 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:16:26.826 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:16:26.826 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "leader", 2024-04-22T08:16:26.826 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 42, 2024-04-22T08:16:26.827 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:16:26.827 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:16:26.827 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:16:26.827 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:16:26.827 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:16:26.827 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 5, 2024-04-22T08:16:26.827 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:16:26.827 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:16:26.827 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:16:26.827 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:16:26.827 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:16:26.827 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:16:26.827 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:16:26.827 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:16:26.828 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:16:26.828 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:16:26.828 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:16:26.828 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:16:26.828 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:16:26.828 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:16:26.828 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:16:26.828 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:16:26.828 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:16:26.828 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:16:26.828 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:16:26.828 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:16:26.828 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:16:26.828 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:16:26.828 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:16:26.829 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:16:26.829 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:16:26.829 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:16:26.829 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:26.829 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:26.829 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:16:26.829 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:16:26.829 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:16:26.829 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:16:26.829 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:16:26.829 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:16:26.829 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:16:26.829 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:16:26.829 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:16:26.830 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:16:26.830 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:16:26.830 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:16:26.830 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:16:26.830 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:16:26.830 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:16:26.830 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:16:26.830 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:16:26.830 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:16:26.830 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:16:26.830 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:16:26.830 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:16:26.830 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:16:26.830 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:16:26.831 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:16:26.831 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:16:26.831 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:16:26.831 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:16:26.831 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:16:26.831 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:16:26.831 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:26.831 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:16:26.831 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:26.831 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:16:26.831 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:16:26.831 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:16:26.832 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:16:26.832 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:26.832 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:16:26.832 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:16:26.832 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:26.832 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:26.832 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:26.832 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:16:26.832 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:16:26.832 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:26.832 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:26.832 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:26.832 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:26.832 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:16:26.832 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:16:26.833 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:16:26.833 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:16:26.833 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:16:26.833 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:26.833 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:26.833 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:16:26.833 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:16:26.833 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:16:26.833 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:16:26.833 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:26.833 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:16:26.833 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:16:26.833 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:26.833 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:26.834 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:26.834 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:16:26.834 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:16:26.834 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:26.834 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:26.834 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:26.834 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:26.834 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:16:26.834 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:16:26.834 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:16:26.834 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:16:26.834 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:16:26.834 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:26.835 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:26.835 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:16:26.835 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:16:26.835 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:16:26.835 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:16:26.835 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:26.835 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:16:26.835 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:16:26.835 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:26.835 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:26.835 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:26.835 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:16:26.835 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:16:26.835 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:26.835 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:26.836 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:26.836 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:26.836 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:16:26.836 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:16:26.836 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:16:26.836 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:16:26.836 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:16:26.836 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:26.836 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:26.836 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:26.836 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:16:26.836 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:16:26.836 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:26.837 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:16:26.837 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:16:26.837 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:16:26.837 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:26.837 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:16:26.837 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:16:26.837 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:26.837 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:16:26.837 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:16:26.837 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 3 2024-04-22T08:16:26.837 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:26.837 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:16:26.838 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:16:26.838 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:26.838 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:16:26.838 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:16:26.838 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:16:26.838 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:26.838 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:26.838 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:26.838 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:16:26.838 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:16:26.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.816+0000 7f7e77fcd640 1 -- 172.21.15.40:0/897633986 >> v2:172.21.15.189:6800/30168 conn(0x7f7e54037820 msgr2=0x7f7e54039ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:26.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.816+0000 7f7e77fcd640 1 --2- 172.21.15.40:0/897633986 >> v2:172.21.15.189:6800/30168 conn(0x7f7e54037820 0x7f7e54039ce0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f7e68035000 tx=0x7f7e68030000 comp rx=0 tx=0).stop 2024-04-22T08:16:26.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.816+0000 7f7e77fcd640 10 monclient: shutdown 2024-04-22T08:16:26.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.816+0000 7f7e77fcd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:16:26.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.816+0000 7f7e77fcd640 1 -- 172.21.15.40:0/897633986 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7e700a1dc0 msgr2=0x7f7e701cb030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:26.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.816+0000 7f7e77fcd640 1 --2- 172.21.15.40:0/897633986 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7e700a1dc0 0x7f7e701cb030 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7e6000c410 tx=0x7f7e6000c8d0 comp rx=0 tx=0).stop 2024-04-22T08:16:26.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.816+0000 7f7e7656a640 1 -- 172.21.15.40:0/897633986 reap_dead start 2024-04-22T08:16:26.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.816+0000 7f7e7656a640 1 -- 172.21.15.40:0/897633986 reap_dead start 2024-04-22T08:16:26.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.816+0000 7f7e77fcd640 1 -- 172.21.15.40:0/897633986 shutdown_connections 2024-04-22T08:16:26.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.816+0000 7f7e77fcd640 1 -- 172.21.15.40:0/897633986 >> 172.21.15.40:0/897633986 conn(0x7f7e700a4ef0 msgr2=0x7f7e700a61f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:16:26.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.816+0000 7f7e77fcd640 1 -- 172.21.15.40:0/897633986 shutdown_connections 2024-04-22T08:16:26.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.817+0000 7f7e77fcd640 1 -- 172.21.15.40:0/897633986 wait complete. 2024-04-22T08:16:26.841 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:16:26.841 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-22T08:16:26.841 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-22T08:16:26.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.946+0000 7f5bd5c08640 10 monclient: get_monmap_and_config 2024-04-22T08:16:26.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.946+0000 7f5bd5c08640 10 monclient: build_initial_monmap 2024-04-22T08:16:26.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.946+0000 7f5bd5c08640 10 monclient: monmap: 2024-04-22T08:16:26.946 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:16:26.946 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:26.946 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:16:26.947310+0000 2024-04-22T08:16:26.946 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:16:26.947310+0000 2024-04-22T08:16:26.946 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:16:26.946 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:26.946 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:16:26.946 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:16:26.947 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:16:26.947 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:16:26.947 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:16:26.947 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:26.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.947+0000 7f5bd5c08640 1 Processor -- start 2024-04-22T08:16:26.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.947+0000 7f5bd5c08640 1 -- start start 2024-04-22T08:16:26.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.947+0000 7f5bd5c08640 10 monclient: init 2024-04-22T08:16:26.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.947+0000 7f5bd5c08640 10 monclient: _reopen_session rank -1 2024-04-22T08:16:26.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.947+0000 7f5bd5c08640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-04-22T08:16:26.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.947+0000 7f5bd5c08640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5bd00a1ba0 0x7f5bd00a1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:26.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.947+0000 7f5bd5c08640 10 monclient(hunting): picked mon.noname-a con 0x7f5bd00a1ba0 addr v2:172.21.15.40:3300/0 2024-04-22T08:16:26.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.947+0000 7f5bd5c08640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5bd0136230 addr v1:172.21.15.40:6789/0 2024-04-22T08:16:26.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.947+0000 7f5bd5c08640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5bd0136670 addr v1:172.21.15.189:6789/0 2024-04-22T08:16:26.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.947+0000 7f5bd5c08640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd00a3ba0 con 0x7f5bd0136230 2024-04-22T08:16:26.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.947+0000 7f5bd5c08640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd009fe80 con 0x7f5bd0136670 2024-04-22T08:16:26.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.947+0000 7f5bd5c08640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:26.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.947+0000 7f5bd5c08640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bd00722a0 con 0x7f5bd00a1ba0 2024-04-22T08:16:26.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.947+0000 7f5bd5c08640 10 monclient(hunting): _renew_subs 2024-04-22T08:16:26.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.947+0000 7f5bd5c08640 10 monclient(hunting): authenticate will time out at 1060.941643s 2024-04-22T08:16:26.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.947+0000 7f5bcf7fe640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5bd00a1ba0 0x7f5bd00a1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:26.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.947+0000 7f5bceffd640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f5bd0136230 0x7f5bd00a2d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57462/0 (socket says 172.21.15.40:57462) 2024-04-22T08:16:26.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.947+0000 7f5bcf7fe640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5bd00a1ba0 0x7f5bd00a1f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41634/0 (socket says 172.21.15.40:41634) 2024-04-22T08:16:26.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.947+0000 7f5bceffd640 1 -- 172.21.15.40:0/1415623038 learned_addr learned my addr 172.21.15.40:0/1415623038 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:16:26.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.948+0000 7f5bcf7fe640 10 monclient(hunting): get_auth_request con 0x7f5bd00a1ba0 auth_method 0 2024-04-22T08:16:26.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.948+0000 7f5bcf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:26.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.948+0000 7f5bcf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:26.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.948+0000 7f5bcf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:26.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.948+0000 7f5bcf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:26.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.948+0000 7f5bcf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:26.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.948+0000 7f5bcf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:26.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.948+0000 7f5bcf7fe640 10 monclient(hunting): handle_auth_done global_id 34605 payload 274 2024-04-22T08:16:26.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.948+0000 7f5bcf7fe640 1 -- 172.21.15.40:0/1415623038 >> v1:172.21.15.189:6789/0 conn(0x7f5bd0136670 legacy=0x7f5bd00a3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:16:26.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.948+0000 7f5bcf7fe640 1 -- 172.21.15.40:0/1415623038 >> v1:172.21.15.40:6789/0 conn(0x7f5bd0136230 legacy=0x7f5bd00a2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:16:26.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.948+0000 7f5bcf7fe640 10 monclient: _finish_hunting 0 2024-04-22T08:16:26.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.948+0000 7f5bcf7fe640 1 monclient: found mon.noname-a 2024-04-22T08:16:26.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.948+0000 7f5bcf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:16:26.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.948+0000 7f5bcf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:16:26.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.948+0000 7f5bcf7fe640 1 -- 172.21.15.40:0/1415623038 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd00a0160 con 0x7f5bd00a1ba0 2024-04-22T08:16:26.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.951+0000 7f5bcf7fe640 1 -- 172.21.15.40:0/1415623038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bd00a1ba0 msgr2=0x7f5bd00a1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:16:26.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.951+0000 7f5bcf7fe640 1 --2- 172.21.15.40:0/1415623038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bd00a1ba0 0x7f5bd00a1f80 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5bc00097a0 tx=0x7f5bc002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2bb35705678767c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:26.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.951+0000 7f5bce7fc640 1 -- 172.21.15.40:0/1415623038 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc002f050 con 0x7f5bd00a1ba0 2024-04-22T08:16:26.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.951+0000 7f5bce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:26.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.951+0000 7f5bce7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:16:26.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.951+0000 7f5bce7fc640 10 monclient: dump: 2024-04-22T08:16:26.956 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:26.956 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:26.956 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:26.956 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:26.956 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:26.956 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:26.956 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:26.956 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:26.956 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:26.956 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:26.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.951+0000 7f5bce7fc640 10 monclient: _finish_auth 0 2024-04-22T08:16:26.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.951+0000 7f5bce7fc640 10 monclient: _check_auth_tickets 2024-04-22T08:16:26.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.951+0000 7f5bce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:16:26.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.951+0000 7f5bce7fc640 1 -- 172.21.15.40:0/1415623038 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bc00319f0 con 0x7f5bd00a1ba0 2024-04-22T08:16:26.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.951+0000 7f5bd5c08640 5 monclient: authenticate success, global_id 34605 2024-04-22T08:16:26.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.951+0000 7f5bd5c08640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:16:26.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.951+0000 7f5bce7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:16:26.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.951+0000 7f5bd5c08640 10 monclient: get_monmap_and_config success 2024-04-22T08:16:26.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.951+0000 7f5bd5c08640 4 set_mon_vals no callback set 2024-04-22T08:16:26.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.951+0000 7f5bce7fc640 1 -- 172.21.15.40:0/1415623038 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc0031d80 con 0x7f5bd00a1ba0 2024-04-22T08:16:26.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.951+0000 7f5bce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:26.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.951+0000 7f5bce7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:16:26.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.951+0000 7f5bce7fc640 10 monclient: dump: 2024-04-22T08:16:26.957 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:26.957 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:26.958 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:26.958 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:26.958 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:26.958 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:26.958 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:26.958 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:26.958 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:26.958 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:26.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.951+0000 7f5bd5c08640 10 monclient: shutdown 2024-04-22T08:16:26.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.951+0000 7f5bd5c08640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:16:26.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.951+0000 7f5bd5c08640 1 -- 172.21.15.40:0/1415623038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bd00a1ba0 msgr2=0x7f5bd00a1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:26.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.951+0000 7f5bd5c08640 1 --2- 172.21.15.40:0/1415623038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bd00a1ba0 0x7f5bd00a1f80 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5bc00097a0 tx=0x7f5bc002b7a0 comp rx=0 tx=0).stop 2024-04-22T08:16:26.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.951+0000 7f5bd5c08640 1 -- 172.21.15.40:0/1415623038 shutdown_connections 2024-04-22T08:16:26.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.951+0000 7f5bd5c08640 1 --2- 172.21.15.40:0/1415623038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bd00a1ba0 0x7f5bd00a1f80 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:26.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.951+0000 7f5bd5c08640 1 -- 172.21.15.40:0/1415623038 >> 172.21.15.40:0/1415623038 conn(0x7f5bd00a4e80 msgr2=0x7f5bd00a5290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:16:26.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.951+0000 7f5bd5c08640 1 -- 172.21.15.40:0/1415623038 shutdown_connections 2024-04-22T08:16:26.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.951+0000 7f5bd5c08640 1 -- 172.21.15.40:0/1415623038 wait complete. 2024-04-22T08:16:26.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.952+0000 7f5bd5c08640 10 monclient: build_initial_monmap 2024-04-22T08:16:26.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.952+0000 7f5bd5c08640 10 monclient: monmap: 2024-04-22T08:16:26.959 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:16:26.959 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:16:26.959 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:16:26.959 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:16:26.959 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:16:26.959 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:26.959 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:16:26.959 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:16:26.959 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:16:26.959 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:26.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.952+0000 7f5bd5c08640 1 Processor -- start 2024-04-22T08:16:26.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bd5c08640 1 -- start start 2024-04-22T08:16:26.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bd5c08640 10 monclient: init 2024-04-22T08:16:26.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bd5c08640 10 monclient: _reopen_session rank -1 2024-04-22T08:16:26.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bd5c08640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-22T08:16:26.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bd5c08640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bd00a1ba0 0x7f5bd01caff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:26.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bd5c08640 10 monclient(hunting): picked mon.noname-c con 0x7f5bd00a1ba0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:16:26.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bd5c08640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5bd0136230 0x7f5bd01cb530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:26.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bd5c08640 10 monclient(hunting): picked mon.noname-b con 0x7f5bd0136230 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:16:26.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bd5c08640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bd0136670 0x7f5bd006dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:26.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bd5c08640 10 monclient(hunting): picked mon.noname-a con 0x7f5bd0136670 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:16:26.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bd5c08640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:26.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bd5c08640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd0131b90 con 0x7f5bd0136670 2024-04-22T08:16:26.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bd5c08640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:26.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bd5c08640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd0070520 con 0x7f5bd0136230 2024-04-22T08:16:26.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bd5c08640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:26.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bd5c08640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd0132410 con 0x7f5bd00a1ba0 2024-04-22T08:16:26.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bd5c08640 10 monclient(hunting): _renew_subs 2024-04-22T08:16:26.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bd5c08640 10 monclient(hunting): authenticate will time out at 1060.947308s 2024-04-22T08:16:26.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bcf7fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bd00a1ba0 0x7f5bd01caff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:26.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bcf7fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bd00a1ba0 0x7f5bd01caff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:33242/0 (socket says 172.21.15.40:33242) 2024-04-22T08:16:26.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bcffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bd0136670 0x7f5bd006dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:26.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bcf7fe640 1 -- 172.21.15.40:0/4153778549 learned_addr learned my addr 172.21.15.40:0/4153778549 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:16:26.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bcf7fe640 10 monclient(hunting): get_auth_request con 0x7f5bd00a1ba0 auth_method 0 2024-04-22T08:16:26.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bcf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:26.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bcf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:26.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bcf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:26.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bcffff640 10 monclient(hunting): get_auth_request con 0x7f5bd0136670 auth_method 0 2024-04-22T08:16:26.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bcffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:26.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bcffff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:26.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bcffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:26.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bcffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:26.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bcffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:26.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bcffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:26.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bcf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:26.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bcf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:26.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bcf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:26.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bceffd640 1 --2- 172.21.15.40:0/4153778549 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5bd0136230 0x7f5bd01cb530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:26.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bceffd640 10 monclient(hunting): get_auth_request con 0x7f5bd0136230 auth_method 0 2024-04-22T08:16:26.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:26.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bceffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:26.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bceffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:26.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bcffff640 10 monclient(hunting): handle_auth_done global_id 34608 payload 824 2024-04-22T08:16:26.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bcffff640 1 -- 172.21.15.40:0/4153778549 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bd00a1ba0 msgr2=0x7f5bd01caff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:26.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bcffff640 1 --2- 172.21.15.40:0/4153778549 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bd00a1ba0 0x7f5bd01caff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:26.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bcffff640 1 -- 172.21.15.40:0/4153778549 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5bd0136230 msgr2=0x7f5bd01cb530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:26.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bcffff640 1 --2- 172.21.15.40:0/4153778549 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5bd0136230 0x7f5bd01cb530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:26.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bcffff640 10 monclient: _finish_hunting 0 2024-04-22T08:16:26.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bcffff640 1 monclient: found mon.noname-a 2024-04-22T08:16:26.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bcffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:16:26.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bcffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:16:26.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bcffff640 1 -- 172.21.15.40:0/4153778549 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bc0009400 con 0x7f5bd0136670 2024-04-22T08:16:26.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.953+0000 7f5bceffd640 1 --2- 172.21.15.40:0/4153778549 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5bd0136230 0x7f5bd01cb530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T08:16:26.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.956+0000 7f5bcffff640 1 -- 172.21.15.40:0/4153778549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bd0136670 msgr2=0x7f5bd006dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:16:26.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.956+0000 7f5bcffff640 1 --2- 172.21.15.40:0/4153778549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bd0136670 0x7f5bd006dfe0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5bc800c750 tx=0x7f5bc800cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:26.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.958+0000 7f5bb7fff640 1 -- 172.21.15.40:0/4153778549 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc800aa00 con 0x7f5bd0136670 2024-04-22T08:16:26.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.958+0000 7f5bb7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:26.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.958+0000 7f5bb7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:16:26.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.958+0000 7f5bb7fff640 10 monclient: dump: 2024-04-22T08:16:26.966 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:26.967 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:26.967 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:26.967 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:26.967 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:26.967 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:26.967 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:26.967 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:26.967 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:26.967 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:26.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.958+0000 7f5bb7fff640 10 monclient: _finish_auth 0 2024-04-22T08:16:26.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.958+0000 7f5bb7fff640 10 monclient: _check_auth_tickets 2024-04-22T08:16:26.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.958+0000 7f5bb7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:16:26.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.958+0000 7f5bb7fff640 1 -- 172.21.15.40:0/4153778549 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bc800ab60 con 0x7f5bd0136670 2024-04-22T08:16:26.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.958+0000 7f5bd5c08640 5 monclient: authenticate success, global_id 34608 2024-04-22T08:16:26.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.958+0000 7f5bd5c08640 10 monclient: _renew_subs 2024-04-22T08:16:26.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.958+0000 7f5bd5c08640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:16:26.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.958+0000 7f5bd5c08640 1 -- 172.21.15.40:0/4153778549 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bd00a2710 con 0x7f5bd0136670 2024-04-22T08:16:26.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.958+0000 7f5bb7fff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:16:26.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.958+0000 7f5bb7fff640 1 -- 172.21.15.40:0/4153778549 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc8012420 con 0x7f5bd0136670 2024-04-22T08:16:26.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.958+0000 7f5bb7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:26.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.958+0000 7f5bb7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:16:26.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.958+0000 7f5bb7fff640 10 monclient: dump: 2024-04-22T08:16:26.968 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:26.968 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:26.968 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:26.968 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:26.968 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:26.968 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:26.969 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:26.969 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:26.969 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:26.969 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:26.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.958+0000 7f5bd5c08640 10 monclient: _renew_subs 2024-04-22T08:16:26.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.958+0000 7f5bcdffb640 4 set_mon_vals no callback set 2024-04-22T08:16:26.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.958+0000 7f5bd5c08640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:16:26.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.958+0000 7f5bd5c08640 1 -- 172.21.15.40:0/4153778549 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bd00712f0 con 0x7f5bd0136670 2024-04-22T08:16:26.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.959+0000 7f5bd5c08640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:16:26.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.959+0000 7f5bd5c08640 1 --2- 172.21.15.40:0/4153778549 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bd006e520 0x7f5bd006e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:26.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.959+0000 7f5bd5c08640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:26.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.959+0000 7f5bd5c08640 1 -- 172.21.15.40:0/4153778549 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd0066650 con 0x7f5bd006e520 2024-04-22T08:16:26.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.959+0000 7f5bb7fff640 1 -- 172.21.15.40:0/4153778549 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f5bc8012640 con 0x7f5bd0136670 2024-04-22T08:16:26.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.959+0000 7f5bcf7fe640 1 --2- 172.21.15.40:0/4153778549 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bd006e520 0x7f5bd006e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:26.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.959+0000 7f5bcf7fe640 10 monclient: get_auth_request con 0x7f5bd006e520 auth_method 0 2024-04-22T08:16:26.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.959+0000 7f5bcf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:26.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.959+0000 7f5bcf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:26.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.959+0000 7f5bcf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:26.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.959+0000 7f5bcf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:26.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.959+0000 7f5bcf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:26.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.959+0000 7f5bcf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:26.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.959+0000 7f5bb7fff640 1 --2- 172.21.15.40:0/4153778549 >> v2:172.21.15.189:6800/30168 conn(0x7f5bbc037820 0x7f5bbc039ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:26.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.959+0000 7f5bb7fff640 1 -- 172.21.15.40:0/4153778549 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f5bc804a480 con 0x7f5bd0136670 2024-04-22T08:16:26.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.959+0000 7f5bcf7fe640 10 monclient(hunting): handle_auth_done global_id 54176 payload 824 2024-04-22T08:16:26.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.959+0000 7f5bcf7fe640 1 -- 172.21.15.40:0/4153778549 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5bd00a24c0 con 0x7f5bd006e520 2024-04-22T08:16:26.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.959+0000 7f5bceffd640 1 --2- 172.21.15.40:0/4153778549 >> v2:172.21.15.189:6800/30168 conn(0x7f5bbc037820 0x7f5bbc039ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:26.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.959+0000 7f5bceffd640 10 monclient: get_auth_request con 0x7f5bbc037820 auth_method 0 2024-04-22T08:16:26.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.961+0000 7f5bcf7fe640 1 -- 172.21.15.40:0/4153778549 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bd006e520 msgr2=0x7f5bd006e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:16:26.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.961+0000 7f5bcf7fe640 1 --2- 172.21.15.40:0/4153778549 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bd006e520 0x7f5bd006e900 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5bc0000c00 tx=0x7f5bc00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:26.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.961+0000 7f5bb7fff640 1 -- 172.21.15.40:0/4153778549 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc003ddd0 con 0x7f5bd006e520 2024-04-22T08:16:26.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.961+0000 7f5bb7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:26.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.962+0000 7f5bb7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:16:26.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.962+0000 7f5bb7fff640 10 monclient: dump: 2024-04-22T08:16:26.971 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:26.971 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:26.971 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:26.971 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:26.971 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:26.971 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:26.971 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:26.971 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:26.971 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:26.972 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:26.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.962+0000 7f5bb7fff640 1 -- 172.21.15.40:0/4153778549 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f5bc00333f0 con 0x7f5bd006e520 2024-04-22T08:16:26.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.962+0000 7f5bb7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:16:26.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.962+0000 7f5bb7fff640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:16:26.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.962+0000 7f5bb7fff640 1 -- 172.21.15.40:0/4153778549 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bd006e520 msgr2=0x7f5bd006e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:26.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.962+0000 7f5bb7fff640 1 --2- 172.21.15.40:0/4153778549 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bd006e520 0x7f5bd006e900 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5bc0000c00 tx=0x7f5bc00046c0 comp rx=0 tx=0).stop 2024-04-22T08:16:26.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.962+0000 7f5bb7fff640 1 -- 172.21.15.40:0/4153778549 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bd006e520 msgr2=0x7f5bd006e900 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:16:26.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.962+0000 7f5bb7fff640 1 --2- 172.21.15.40:0/4153778549 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bd006e520 0x7f5bd006e900 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:26.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.962+0000 7f5bceffd640 1 --2- 172.21.15.40:0/4153778549 >> v2:172.21.15.189:6800/30168 conn(0x7f5bbc037820 0x7f5bbc039ce0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f5bd013ed90 tx=0x7f5bc4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:26.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.968+0000 7f5bd5c08640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:16:26.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.968+0000 7f5bd5c08640 1 --2- 172.21.15.40:0/4153778549 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bd006ee40 0x7f5bd012ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:26.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.968+0000 7f5bd5c08640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:26.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.968+0000 7f5bd5c08640 1 -- 172.21.15.40:0/4153778549 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd0131cd0 con 0x7f5bd006ee40 2024-04-22T08:16:26.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.968+0000 7f5bcf7fe640 1 --2- 172.21.15.40:0/4153778549 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bd006ee40 0x7f5bd012ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:26.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.968+0000 7f5bcf7fe640 10 monclient: get_auth_request con 0x7f5bd006ee40 auth_method 0 2024-04-22T08:16:26.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.968+0000 7f5bcf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:26.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.968+0000 7f5bcf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:26.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.968+0000 7f5bcf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:26.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.968+0000 7f5bcf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:26.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.968+0000 7f5bcf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:26.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.968+0000 7f5bcf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:26.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.968+0000 7f5bcf7fe640 10 monclient(hunting): handle_auth_done global_id 54179 payload 824 2024-04-22T08:16:26.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.968+0000 7f5bcf7fe640 1 -- 172.21.15.40:0/4153778549 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5bd0075260 con 0x7f5bd006ee40 2024-04-22T08:16:26.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.971+0000 7f5bcf7fe640 1 -- 172.21.15.40:0/4153778549 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bd006ee40 msgr2=0x7f5bd012ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:16:26.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.971+0000 7f5bcf7fe640 1 --2- 172.21.15.40:0/4153778549 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bd006ee40 0x7f5bd012ab90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5bd0065ad0 tx=0x7f5bc0031000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:26.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.971+0000 7f5bb7fff640 1 -- 172.21.15.40:0/4153778549 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc002f040 con 0x7f5bd006ee40 2024-04-22T08:16:26.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.971+0000 7f5bb7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:26.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.971+0000 7f5bb7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:16:26.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.971+0000 7f5bb7fff640 10 monclient: dump: 2024-04-22T08:16:26.975 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:26.975 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:26.975 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:26.976 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:26.976 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:26.976 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:26.976 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:26.976 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:26.976 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:26.976 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:26.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.971+0000 7f5bb7fff640 1 -- 172.21.15.40:0/4153778549 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4631 (secure 0 0 0) 0x7f5bc0052450 con 0x7f5bd006ee40 2024-04-22T08:16:26.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.971+0000 7f5bb7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:16:26.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.971+0000 7f5bb7fff640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:16:26.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.971+0000 7f5bb7fff640 1 -- 172.21.15.40:0/4153778549 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bd006ee40 msgr2=0x7f5bd012ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:26.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.971+0000 7f5bb7fff640 1 --2- 172.21.15.40:0/4153778549 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bd006ee40 0x7f5bd012ab90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5bd0065ad0 tx=0x7f5bc0031000 comp rx=0 tx=0).stop 2024-04-22T08:16:26.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.971+0000 7f5bb7fff640 1 -- 172.21.15.40:0/4153778549 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bd006ee40 msgr2=0x7f5bd012ab90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:16:26.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.971+0000 7f5bb7fff640 1 --2- 172.21.15.40:0/4153778549 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bd006ee40 0x7f5bd012ab90 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5bd0065ad0 tx=0x7f5bc0031000 comp rx=0 tx=0).stop 2024-04-22T08:16:26.977 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:16:26.977 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:16:26.977 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:16:26.977 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:16:26.977 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 42, 2024-04-22T08:16:26.977 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:16:26.977 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:16:26.977 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:16:26.977 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:16:26.978 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:16:26.978 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 5, 2024-04-22T08:16:26.978 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:16:26.978 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:16:26.978 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:16:26.978 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:16:26.978 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:16:26.978 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:16:26.978 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:16:26.978 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:16:26.978 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:16:26.978 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:16:26.978 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:16:26.978 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:16:26.978 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:16:26.979 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:16:26.979 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:16:26.979 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:16:26.979 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:16:26.979 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:16:26.979 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:16:26.979 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:16:26.979 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:16:26.979 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:16:26.979 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:16:26.979 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:16:26.979 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:16:26.979 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:16:26.979 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:26.980 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:26.980 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:16:26.980 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:16:26.980 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:16:26.980 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:16:26.980 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:16:26.980 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:16:26.980 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:16:26.980 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:16:26.980 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:16:26.980 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:16:26.980 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:16:26.980 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:16:26.980 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:16:26.981 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:16:26.981 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:16:26.981 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:16:26.981 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:16:26.981 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:16:26.981 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:16:26.981 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:16:26.981 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:16:26.981 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:16:26.981 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:16:26.981 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:16:26.981 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:16:26.981 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:16:26.981 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:16:26.981 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:16:26.982 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:16:26.982 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:26.982 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:16:26.982 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:26.982 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:16:26.982 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:16:26.982 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:16:26.982 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:16:26.982 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:26.982 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:16:26.982 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:16:26.982 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:26.982 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:26.982 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:26.983 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:16:26.983 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:16:26.983 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:26.983 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:26.983 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:26.983 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:26.983 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:16:26.983 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:16:26.983 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:16:26.983 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:16:26.983 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:16:26.983 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:26.983 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:26.983 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:16:26.983 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:16:26.984 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:16:26.984 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:16:26.984 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:26.984 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:16:26.984 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:16:26.984 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:26.984 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:26.984 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:26.984 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:16:26.984 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:16:26.984 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:26.984 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:26.984 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:26.984 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:26.985 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:16:26.985 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:16:26.985 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:16:26.985 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:16:26.985 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:16:26.985 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:26.985 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:26.985 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:16:26.985 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:16:26.985 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:16:26.985 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:16:26.985 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:26.985 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:16:26.985 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:16:26.985 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:26.986 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:26.986 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:26.986 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:16:26.986 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:16:26.986 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:26.986 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:26.986 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:26.986 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:26.986 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:16:26.986 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:16:26.986 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:16:26.986 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:16:26.986 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:16:26.986 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:26.986 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:26.987 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:26.987 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:16:26.987 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:16:26.987 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:26.987 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:16:26.987 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:16:26.987 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:16:26.987 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:26.987 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:16:26.987 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:16:26.987 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:26.987 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:16:26.987 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:16:26.987 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:16:26.988 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:26.988 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:16:26.988 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:16:26.988 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:26.988 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:16:26.988 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:16:26.988 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:16:26.988 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:26.988 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:26.988 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:26.988 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:16:26.988 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:16:26.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.971+0000 7f5bd5c08640 1 -- 172.21.15.40:0/4153778549 >> v2:172.21.15.189:6800/30168 conn(0x7f5bbc037820 msgr2=0x7f5bbc039ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:26.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.971+0000 7f5bd5c08640 1 --2- 172.21.15.40:0/4153778549 >> v2:172.21.15.189:6800/30168 conn(0x7f5bbc037820 0x7f5bbc039ce0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f5bd013ed90 tx=0x7f5bc4009040 comp rx=0 tx=0).stop 2024-04-22T08:16:26.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.971+0000 7f5bd5c08640 10 monclient: shutdown 2024-04-22T08:16:26.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.971+0000 7f5bd5c08640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:16:26.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.971+0000 7f5bd5c08640 1 -- 172.21.15.40:0/4153778549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bd0136670 msgr2=0x7f5bd006dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:26.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.971+0000 7f5bd5c08640 1 --2- 172.21.15.40:0/4153778549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bd0136670 0x7f5bd006dfe0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5bc800c750 tx=0x7f5bc800cc10 comp rx=0 tx=0).stop 2024-04-22T08:16:26.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.971+0000 7f5bcffff640 1 -- 172.21.15.40:0/4153778549 reap_dead start 2024-04-22T08:16:26.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.971+0000 7f5bd5c08640 1 -- 172.21.15.40:0/4153778549 shutdown_connections 2024-04-22T08:16:26.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.971+0000 7f5bd5c08640 1 --2- 172.21.15.40:0/4153778549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bd0136670 0x7f5bd006dfe0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:26.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.971+0000 7f5bd5c08640 1 -- 172.21.15.40:0/4153778549 >> 172.21.15.40:0/4153778549 conn(0x7f5bd00a4e80 msgr2=0x7f5bd00a0920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:16:26.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.971+0000 7f5bd5c08640 1 -- 172.21.15.40:0/4153778549 shutdown_connections 2024-04-22T08:16:26.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:26.971+0000 7f5bd5c08640 1 -- 172.21.15.40:0/4153778549 wait complete. 2024-04-22T08:16:26.991 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:16:26.991 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-22T08:16:26.991 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-22T08:16:27.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.091+0000 7f46c0937640 10 monclient: get_monmap_and_config 2024-04-22T08:16:27.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.091+0000 7f46c0937640 10 monclient: build_initial_monmap 2024-04-22T08:16:27.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.091+0000 7f46c0937640 10 monclient: monmap: 2024-04-22T08:16:27.091 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:16:27.091 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:27.091 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:16:27.092617+0000 2024-04-22T08:16:27.092 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:16:27.092617+0000 2024-04-22T08:16:27.092 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:16:27.092 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:27.092 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:16:27.092 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:16:27.092 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:16:27.092 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:16:27.092 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:16:27.092 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:27.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.092+0000 7f46c0937640 1 Processor -- start 2024-04-22T08:16:27.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.092+0000 7f46c0937640 1 -- start start 2024-04-22T08:16:27.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.092+0000 7f46c0937640 10 monclient: init 2024-04-22T08:16:27.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.092+0000 7f46c0937640 10 monclient: _reopen_session rank -1 2024-04-22T08:16:27.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.092+0000 7f46c0937640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-04-22T08:16:27.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.092+0000 7f46c0937640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f46bc0a1a80 0x7f46bc0a01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:27.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.092+0000 7f46c0937640 10 monclient(hunting): picked mon.noname-a con 0x7f46bc0a1a80 addr v2:172.21.15.40:3300/0 2024-04-22T08:16:27.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.092+0000 7f46c0937640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f46bc0a1e60 addr v1:172.21.15.189:6789/0 2024-04-22T08:16:27.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.093+0000 7f46c0937640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f46bc13b0e0 addr v1:172.21.15.40:6789/0 2024-04-22T08:16:27.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.093+0000 7f46c0937640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46bc09d860 con 0x7f46bc13b0e0 2024-04-22T08:16:27.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.093+0000 7f46c0937640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46bc09ef50 con 0x7f46bc0a1e60 2024-04-22T08:16:27.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.093+0000 7f46c0937640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:27.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.093+0000 7f46c0937640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46bc09f880 con 0x7f46bc0a1a80 2024-04-22T08:16:27.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.093+0000 7f46c0937640 10 monclient(hunting): _renew_subs 2024-04-22T08:16:27.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.093+0000 7f46c0937640 10 monclient(hunting): authenticate will time out at 1061.086819s 2024-04-22T08:16:27.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.093+0000 7f46ba59c640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f46bc0a1a80 0x7f46bc0a01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:27.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.093+0000 7f46bad9d640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f46bc13b0e0 0x7f46bc13d4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57470/0 (socket says 172.21.15.40:57470) 2024-04-22T08:16:27.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.093+0000 7f46bad9d640 1 -- 172.21.15.40:0/851561628 learned_addr learned my addr 172.21.15.40:0/851561628 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:16:27.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.093+0000 7f46ba59c640 10 monclient(hunting): get_auth_request con 0x7f46bc0a1a80 auth_method 0 2024-04-22T08:16:27.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.093+0000 7f46ba59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:27.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.093+0000 7f46ba59c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:27.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.093+0000 7f46ba59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:27.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.093+0000 7f46ba59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:27.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.093+0000 7f46ba59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:27.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.093+0000 7f46ba59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:27.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.093+0000 7f46ba59c640 10 monclient(hunting): handle_auth_done global_id 34611 payload 274 2024-04-22T08:16:27.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.093+0000 7f46ba59c640 1 -- 172.21.15.40:0/851561628 >> v1:172.21.15.189:6789/0 conn(0x7f46bc0a1e60 legacy=0x7f46bc0a2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:16:27.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.093+0000 7f46ba59c640 1 -- 172.21.15.40:0/851561628 >> v1:172.21.15.40:6789/0 conn(0x7f46bc13b0e0 legacy=0x7f46bc13d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:16:27.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.093+0000 7f46ba59c640 10 monclient: _finish_hunting 0 2024-04-22T08:16:27.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.093+0000 7f46ba59c640 1 monclient: found mon.noname-a 2024-04-22T08:16:27.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.093+0000 7f46ba59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:16:27.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.093+0000 7f46ba59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:16:27.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.093+0000 7f46ba59c640 1 -- 172.21.15.40:0/851561628 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46bc072160 con 0x7f46bc0a1a80 2024-04-22T08:16:27.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.096+0000 7f46ba59c640 1 -- 172.21.15.40:0/851561628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46bc0a1a80 msgr2=0x7f46bc0a01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:16:27.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.096+0000 7f46ba59c640 1 --2- 172.21.15.40:0/851561628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46bc0a1a80 0x7f46bc0a01d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f46b40097a0 tx=0x7f46b402b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df5f5fae5f8b698e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:27.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.096+0000 7f46b959a640 1 -- 172.21.15.40:0/851561628 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b4030a00 con 0x7f46bc0a1a80 2024-04-22T08:16:27.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.096+0000 7f46b959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:27.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.096+0000 7f46b959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:16:27.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.096+0000 7f46b959a640 10 monclient: dump: 2024-04-22T08:16:27.101 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:27.102 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:27.102 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:27.102 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:27.102 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:27.102 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:27.102 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:27.102 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:27.102 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:27.102 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:27.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.096+0000 7f46b959a640 10 monclient: _finish_auth 0 2024-04-22T08:16:27.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.096+0000 7f46b959a640 10 monclient: _check_auth_tickets 2024-04-22T08:16:27.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.096+0000 7f46b959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:16:27.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.096+0000 7f46b959a640 1 -- 172.21.15.40:0/851561628 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46b402e050 con 0x7f46bc0a1a80 2024-04-22T08:16:27.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.096+0000 7f46c0937640 5 monclient: authenticate success, global_id 34611 2024-04-22T08:16:27.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.096+0000 7f46c0937640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:16:27.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.096+0000 7f46b959a640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:16:27.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.096+0000 7f46c0937640 10 monclient: get_monmap_and_config success 2024-04-22T08:16:27.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.096+0000 7f46c0937640 4 set_mon_vals no callback set 2024-04-22T08:16:27.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.096+0000 7f46b959a640 1 -- 172.21.15.40:0/851561628 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b4030d80 con 0x7f46bc0a1a80 2024-04-22T08:16:27.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.096+0000 7f46b959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:27.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.096+0000 7f46b959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:16:27.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.096+0000 7f46b959a640 10 monclient: dump: 2024-04-22T08:16:27.103 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:27.103 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:27.103 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:27.103 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:27.103 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:27.103 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:27.103 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:27.104 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:27.104 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:27.104 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:27.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.096+0000 7f46c0937640 10 monclient: shutdown 2024-04-22T08:16:27.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.096+0000 7f46c0937640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:16:27.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.096+0000 7f46c0937640 1 -- 172.21.15.40:0/851561628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46bc0a1a80 msgr2=0x7f46bc0a01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:27.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.096+0000 7f46c0937640 1 --2- 172.21.15.40:0/851561628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46bc0a1a80 0x7f46bc0a01d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f46b40097a0 tx=0x7f46b402b710 comp rx=0 tx=0).stop 2024-04-22T08:16:27.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.097+0000 7f46c0937640 1 -- 172.21.15.40:0/851561628 shutdown_connections 2024-04-22T08:16:27.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.097+0000 7f46c0937640 1 --2- 172.21.15.40:0/851561628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46bc0a1a80 0x7f46bc0a01d0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:27.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.097+0000 7f46c0937640 1 -- 172.21.15.40:0/851561628 >> 172.21.15.40:0/851561628 conn(0x7f46bc0a66a0 msgr2=0x7f46bc0a2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:16:27.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.097+0000 7f46c0937640 1 -- 172.21.15.40:0/851561628 shutdown_connections 2024-04-22T08:16:27.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.097+0000 7f46c0937640 1 -- 172.21.15.40:0/851561628 wait complete. 2024-04-22T08:16:27.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.097+0000 7f46c0937640 10 monclient: build_initial_monmap 2024-04-22T08:16:27.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.097+0000 7f46c0937640 10 monclient: monmap: 2024-04-22T08:16:27.105 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:16:27.105 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:16:27.105 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:16:27.105 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:16:27.105 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:16:27.105 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:27.105 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:16:27.105 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:16:27.105 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:16:27.105 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:27.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.098+0000 7f46c0937640 1 Processor -- start 2024-04-22T08:16:27.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.098+0000 7f46c0937640 1 -- start start 2024-04-22T08:16:27.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.098+0000 7f46c0937640 10 monclient: init 2024-04-22T08:16:27.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.098+0000 7f46c0937640 10 monclient: _reopen_session rank -1 2024-04-22T08:16:27.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.098+0000 7f46c0937640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-22T08:16:27.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.098+0000 7f46c0937640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46bc0a1a80 0x7f46bc120ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:27.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.098+0000 7f46c0937640 10 monclient(hunting): picked mon.noname-a con 0x7f46bc0a1a80 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:16:27.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.098+0000 7f46c0937640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f46bc0a1e60 0x7f46bc121010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:27.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.098+0000 7f46c0937640 10 monclient(hunting): picked mon.noname-c con 0x7f46bc0a1e60 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:16:27.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.098+0000 7f46c0937640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46bc13b0e0 0x7f46bc121550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:27.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.098+0000 7f46c0937640 10 monclient(hunting): picked mon.noname-b con 0x7f46bc13b0e0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:16:27.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.098+0000 7f46c0937640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:27.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.098+0000 7f46c0937640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46bc1231a0 con 0x7f46bc0a1a80 2024-04-22T08:16:27.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.098+0000 7f46c0937640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:27.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.098+0000 7f46c0937640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46bc09cc20 con 0x7f46bc13b0e0 2024-04-22T08:16:27.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.098+0000 7f46c0937640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:27.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.098+0000 7f46c0937640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46bc099d10 con 0x7f46bc0a1e60 2024-04-22T08:16:27.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.098+0000 7f46c0937640 10 monclient(hunting): _renew_subs 2024-04-22T08:16:27.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.098+0000 7f46c0937640 10 monclient(hunting): authenticate will time out at 1061.092303s 2024-04-22T08:16:27.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.098+0000 7f46ba59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46bc0a1a80 0x7f46bc120ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:27.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.098+0000 7f46b9d9b640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f46bc0a1e60 0x7f46bc121010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:27.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.098+0000 7f46ba59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46bc0a1a80 0x7f46bc120ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41664/0 (socket says 172.21.15.40:41664) 2024-04-22T08:16:27.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.098+0000 7f46ba59c640 1 -- 172.21.15.40:0/1584443090 learned_addr learned my addr 172.21.15.40:0/1584443090 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:16:27.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.098+0000 7f46ba59c640 10 monclient(hunting): get_auth_request con 0x7f46bc0a1a80 auth_method 0 2024-04-22T08:16:27.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.098+0000 7f46ba59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:27.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.098+0000 7f46ba59c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:27.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.098+0000 7f46ba59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:27.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.098+0000 7f46b9d9b640 10 monclient(hunting): get_auth_request con 0x7f46bc0a1e60 auth_method 0 2024-04-22T08:16:27.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.098+0000 7f46b9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:27.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.098+0000 7f46b9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:27.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.098+0000 7f46b9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:27.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.098+0000 7f46b9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:27.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.098+0000 7f46b9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:27.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.098+0000 7f46b9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:27.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.098+0000 7f46ba59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:27.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.098+0000 7f46ba59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:27.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.098+0000 7f46ba59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:27.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.098+0000 7f46bad9d640 1 --2- 172.21.15.40:0/1584443090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46bc13b0e0 0x7f46bc121550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:27.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.098+0000 7f46b9d9b640 10 monclient(hunting): handle_auth_done global_id 54182 payload 824 2024-04-22T08:16:27.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.098+0000 7f46b9d9b640 1 -- 172.21.15.40:0/1584443090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46bc13b0e0 msgr2=0x7f46bc121550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:27.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.098+0000 7f46b9d9b640 1 --2- 172.21.15.40:0/1584443090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46bc13b0e0 0x7f46bc121550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:27.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.098+0000 7f46b9d9b640 1 -- 172.21.15.40:0/1584443090 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46bc0a1a80 msgr2=0x7f46bc120ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:27.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.098+0000 7f46b9d9b640 1 --2- 172.21.15.40:0/1584443090 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46bc0a1a80 0x7f46bc120ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:27.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.098+0000 7f46b9d9b640 10 monclient: _finish_hunting 0 2024-04-22T08:16:27.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.098+0000 7f46b9d9b640 1 monclient: found mon.noname-c 2024-04-22T08:16:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.099+0000 7f46b9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:16:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.099+0000 7f46b9d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:16:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.099+0000 7f46b9d9b640 1 -- 172.21.15.40:0/1584443090 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b4009400 con 0x7f46bc0a1e60 2024-04-22T08:16:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.099+0000 7f46bad9d640 10 monclient: get_auth_request con 0x7f46bc13b0e0 auth_method 0 2024-04-22T08:16:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.099+0000 7f46bad9d640 1 --2- 172.21.15.40:0/1584443090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46bc13b0e0 0x7f46bc121550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T08:16:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.099+0000 7f46ba59c640 1 --2- 172.21.15.40:0/1584443090 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46bc0a1a80 0x7f46bc120ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:16:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.101+0000 7f46b9d9b640 1 -- 172.21.15.40:0/1584443090 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f46bc0a1e60 msgr2=0x7f46bc121010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:16:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.101+0000 7f46b9d9b640 1 --2- 172.21.15.40:0/1584443090 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f46bc0a1e60 0x7f46bc121010 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f46b000c410 tx=0x7f46b000c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.101+0000 7f46aaffd640 1 -- 172.21.15.40:0/1584443090 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b0005230 con 0x7f46bc0a1e60 2024-04-22T08:16:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.101+0000 7f46aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.101+0000 7f46aaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:16:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.101+0000 7f46aaffd640 10 monclient: dump: 2024-04-22T08:16:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:27.110 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:27.110 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:27.110 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:27.110 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:27.110 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:27.110 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:27.110 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:27.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.101+0000 7f46aaffd640 10 monclient: _finish_auth 0 2024-04-22T08:16:27.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.101+0000 7f46aaffd640 10 monclient: _check_auth_tickets 2024-04-22T08:16:27.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.101+0000 7f46aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:16:27.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.101+0000 7f46c0937640 5 monclient: authenticate success, global_id 54182 2024-04-22T08:16:27.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.101+0000 7f46c0937640 10 monclient: _renew_subs 2024-04-22T08:16:27.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.101+0000 7f46c0937640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:16:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.101+0000 7f46aaffd640 1 -- 172.21.15.40:0/1584443090 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46b0005390 con 0x7f46bc0a1e60 2024-04-22T08:16:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.101+0000 7f46c0937640 1 -- 172.21.15.40:0/1584443090 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46bc1222a0 con 0x7f46bc0a1e60 2024-04-22T08:16:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.101+0000 7f46aaffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:16:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.101+0000 7f46aaffd640 1 -- 172.21.15.40:0/1584443090 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b00119e0 con 0x7f46bc0a1e60 2024-04-22T08:16:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.101+0000 7f46aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.101+0000 7f46aaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:16:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.101+0000 7f46aaffd640 10 monclient: dump: 2024-04-22T08:16:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:27.112 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:27.112 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:27.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.101+0000 7f46c0937640 10 monclient: _renew_subs 2024-04-22T08:16:27.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.101+0000 7f46c0937640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:16:27.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.101+0000 7f46c0937640 1 -- 172.21.15.40:0/1584443090 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46bc071640 con 0x7f46bc0a1e60 2024-04-22T08:16:27.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.101+0000 7f46abfff640 4 set_mon_vals no callback set 2024-04-22T08:16:27.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.102+0000 7f46c0937640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:16:27.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.102+0000 7f46c0937640 1 --2- 172.21.15.40:0/1584443090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46bc139540 0x7f46bc139920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:27.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.102+0000 7f46c0937640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:27.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.102+0000 7f46c0937640 1 -- 172.21.15.40:0/1584443090 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46bc066770 con 0x7f46bc139540 2024-04-22T08:16:27.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.102+0000 7f46aaffd640 1 -- 172.21.15.40:0/1584443090 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f46b0011bc0 con 0x7f46bc0a1e60 2024-04-22T08:16:27.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.102+0000 7f46ba59c640 1 --2- 172.21.15.40:0/1584443090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46bc139540 0x7f46bc139920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:27.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.102+0000 7f46aaffd640 1 --2- 172.21.15.40:0/1584443090 >> v2:172.21.15.189:6800/30168 conn(0x7f469c0377d0 0x7f469c039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:27.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.102+0000 7f46ba59c640 10 monclient: get_auth_request con 0x7f46bc139540 auth_method 0 2024-04-22T08:16:27.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.102+0000 7f46ba59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:27.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.102+0000 7f46ba59c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:27.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.102+0000 7f46ba59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:27.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.102+0000 7f46aaffd640 1 -- 172.21.15.40:0/1584443090 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f46b00499a0 con 0x7f46bc0a1e60 2024-04-22T08:16:27.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.102+0000 7f46ba59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:27.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.102+0000 7f46ba59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:27.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.102+0000 7f46ba59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:27.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.102+0000 7f46bad9d640 1 --2- 172.21.15.40:0/1584443090 >> v2:172.21.15.189:6800/30168 conn(0x7f469c0377d0 0x7f469c039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:27.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.102+0000 7f46bad9d640 10 monclient: get_auth_request con 0x7f469c0377d0 auth_method 0 2024-04-22T08:16:27.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.102+0000 7f46ba59c640 10 monclient(hunting): handle_auth_done global_id 24811 payload 824 2024-04-22T08:16:27.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.102+0000 7f46ba59c640 1 -- 172.21.15.40:0/1584443090 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f46bc09dc10 con 0x7f46bc139540 2024-04-22T08:16:27.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.105+0000 7f46ba59c640 1 -- 172.21.15.40:0/1584443090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46bc139540 msgr2=0x7f46bc139920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:16:27.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.105+0000 7f46ba59c640 1 --2- 172.21.15.40:0/1584443090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46bc139540 0x7f46bc139920 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f46b4000c00 tx=0x7f46b40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:27.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.105+0000 7f46bad9d640 1 --2- 172.21.15.40:0/1584443090 >> v2:172.21.15.189:6800/30168 conn(0x7f469c0377d0 0x7f469c039c90 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f46bc122b80 tx=0x7f46ac009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:27.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.105+0000 7f46aaffd640 1 -- 172.21.15.40:0/1584443090 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b4032d80 con 0x7f46bc139540 2024-04-22T08:16:27.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.105+0000 7f46aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:27.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.105+0000 7f46aaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:16:27.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.105+0000 7f46aaffd640 10 monclient: dump: 2024-04-22T08:16:27.117 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:27.117 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:27.117 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:27.117 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:27.117 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:27.117 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:27.118 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:27.118 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:27.118 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:27.118 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:27.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.105+0000 7f46aaffd640 1 -- 172.21.15.40:0/1584443090 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f46b403d440 con 0x7f46bc139540 2024-04-22T08:16:27.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.105+0000 7f46aaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:16:27.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.105+0000 7f46aaffd640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:16:27.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.105+0000 7f46aaffd640 1 -- 172.21.15.40:0/1584443090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46bc139540 msgr2=0x7f46bc139920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:27.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.105+0000 7f46aaffd640 1 --2- 172.21.15.40:0/1584443090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46bc139540 0x7f46bc139920 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f46b4000c00 tx=0x7f46b40046c0 comp rx=0 tx=0).stop 2024-04-22T08:16:27.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.105+0000 7f46aaffd640 1 -- 172.21.15.40:0/1584443090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46bc139540 msgr2=0x7f46bc139920 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:16:27.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.105+0000 7f46aaffd640 1 --2- 172.21.15.40:0/1584443090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46bc139540 0x7f46bc139920 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:27.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.112+0000 7f46c0937640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:16:27.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.112+0000 7f46c0937640 1 --2- 172.21.15.40:0/1584443090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46bc139e60 0x7f46bc06cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:27.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.112+0000 7f46c0937640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:27.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.112+0000 7f46c0937640 1 -- 172.21.15.40:0/1584443090 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46bc1236c0 con 0x7f46bc139e60 2024-04-22T08:16:27.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.112+0000 7f46ba59c640 1 --2- 172.21.15.40:0/1584443090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46bc139e60 0x7f46bc06cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:27.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.112+0000 7f46ba59c640 10 monclient: get_auth_request con 0x7f46bc139e60 auth_method 0 2024-04-22T08:16:27.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.112+0000 7f46ba59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:27.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.112+0000 7f46ba59c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:27.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.112+0000 7f46ba59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:27.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.112+0000 7f46ba59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:27.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.112+0000 7f46ba59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:27.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.112+0000 7f46ba59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:27.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.112+0000 7f46ba59c640 10 monclient(hunting): handle_auth_done global_id 24814 payload 824 2024-04-22T08:16:27.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.113+0000 7f46ba59c640 1 -- 172.21.15.40:0/1584443090 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f46bc04e9b0 con 0x7f46bc139e60 2024-04-22T08:16:27.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.115+0000 7f46ba59c640 1 -- 172.21.15.40:0/1584443090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46bc139e60 msgr2=0x7f46bc06cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:16:27.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.115+0000 7f46ba59c640 1 --2- 172.21.15.40:0/1584443090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46bc139e60 0x7f46bc06cb70 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f46b402b750 tx=0x7f46b4008030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:27.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.115+0000 7f46aaffd640 1 -- 172.21.15.40:0/1584443090 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b403d5b0 con 0x7f46bc139e60 2024-04-22T08:16:27.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.115+0000 7f46aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:27.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.115+0000 7f46aaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:16:27.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.115+0000 7f46aaffd640 10 monclient: dump: 2024-04-22T08:16:27.120 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:27.120 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:27.120 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:27.120 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:27.120 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:27.120 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:27.120 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:27.120 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:27.121 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:27.121 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:27.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.115+0000 7f46aaffd640 1 -- 172.21.15.40:0/1584443090 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4801 (secure 0 0 0) 0x7f46b403d710 con 0x7f46bc139e60 2024-04-22T08:16:27.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.115+0000 7f46aaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:16:27.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.115+0000 7f46aaffd640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:16:27.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.115+0000 7f46aaffd640 1 -- 172.21.15.40:0/1584443090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46bc139e60 msgr2=0x7f46bc06cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:27.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.115+0000 7f46aaffd640 1 --2- 172.21.15.40:0/1584443090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46bc139e60 0x7f46bc06cb70 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f46b402b750 tx=0x7f46b4008030 comp rx=0 tx=0).stop 2024-04-22T08:16:27.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.115+0000 7f46aaffd640 1 -- 172.21.15.40:0/1584443090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46bc139e60 msgr2=0x7f46bc06cb70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:16:27.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.115+0000 7f46aaffd640 1 --2- 172.21.15.40:0/1584443090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46bc139e60 0x7f46bc06cb70 secure :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f46b402b750 tx=0x7f46b4008030 comp rx=0 tx=0).stop 2024-04-22T08:16:27.121 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:16:27.122 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:16:27.122 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:16:27.122 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:16:27.122 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 42, 2024-04-22T08:16:27.122 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:16:27.122 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:16:27.122 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:16:27.122 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:16:27.122 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:16:27.122 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 5, 2024-04-22T08:16:27.122 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:16:27.122 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:16:27.123 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:16:27.123 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:16:27.123 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:16:27.123 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:16:27.123 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:16:27.123 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:16:27.123 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:16:27.123 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:16:27.123 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:16:27.123 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:16:27.123 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:16:27.123 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:16:27.123 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:16:27.123 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:16:27.124 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:16:27.124 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:16:27.124 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:16:27.124 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:16:27.124 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:16:27.124 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:16:27.124 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:16:27.124 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:16:27.124 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:16:27.124 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:16:27.124 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:27.124 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:27.124 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:16:27.124 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:16:27.125 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:16:27.125 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:16:27.125 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:16:27.125 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:16:27.125 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:16:27.125 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:16:27.125 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:16:27.125 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:16:27.125 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:16:27.125 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:16:27.125 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:16:27.125 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:16:27.125 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:16:27.125 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:16:27.125 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:16:27.126 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:16:27.126 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:16:27.126 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:16:27.126 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:16:27.126 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:16:27.126 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:16:27.126 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:16:27.126 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:16:27.126 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:16:27.126 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:16:27.126 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:16:27.126 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:16:27.126 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:27.126 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:16:27.127 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:27.127 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:16:27.127 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:16:27.127 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:16:27.127 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:16:27.127 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:27.127 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:16:27.127 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:16:27.127 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:27.127 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:27.127 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:27.127 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:16:27.127 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:16:27.127 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:27.128 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:27.128 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:27.128 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:27.128 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:16:27.128 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:16:27.128 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:16:27.128 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:16:27.129 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:16:27.129 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:27.129 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:27.129 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:16:27.129 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:16:27.129 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:16:27.129 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:16:27.129 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:27.129 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:16:27.129 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:16:27.129 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:27.130 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:27.130 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:27.130 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:16:27.130 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:16:27.130 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:27.130 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:27.130 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:27.130 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:27.130 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:16:27.130 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:16:27.130 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:16:27.130 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:16:27.130 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:16:27.130 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:27.131 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:27.131 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:16:27.131 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:16:27.131 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:16:27.131 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:16:27.131 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:27.131 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:16:27.131 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:16:27.131 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:27.131 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:27.131 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:27.131 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:16:27.131 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:16:27.131 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:27.132 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:27.132 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:27.132 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:27.132 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:16:27.132 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:16:27.132 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:16:27.132 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:16:27.132 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:16:27.132 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:27.132 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:27.132 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:27.132 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:16:27.132 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:16:27.133 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:27.133 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:16:27.133 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:16:27.133 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:16:27.133 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:27.133 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:16:27.133 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:16:27.133 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:27.133 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:16:27.133 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:16:27.133 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 2 2024-04-22T08:16:27.133 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:27.134 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:16:27.134 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:16:27.134 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:27.134 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:16:27.134 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:16:27.134 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 7 2024-04-22T08:16:27.134 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:27.134 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:16:27.134 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": [ 2024-04-22T08:16:27.134 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:27.134 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:16:27.134 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:16:27.134 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:16:27.135 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:27.135 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:27.135 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:27.135 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:16:27.135 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:16:27.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.116+0000 7f46c0937640 1 -- 172.21.15.40:0/1584443090 >> v2:172.21.15.189:6800/30168 conn(0x7f469c0377d0 msgr2=0x7f469c039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:27.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.116+0000 7f46c0937640 1 --2- 172.21.15.40:0/1584443090 >> v2:172.21.15.189:6800/30168 conn(0x7f469c0377d0 0x7f469c039c90 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f46bc122b80 tx=0x7f46ac009040 comp rx=0 tx=0).stop 2024-04-22T08:16:27.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.116+0000 7f46c0937640 10 monclient: shutdown 2024-04-22T08:16:27.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.116+0000 7f46c0937640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:16:27.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.116+0000 7f46c0937640 1 -- 172.21.15.40:0/1584443090 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f46bc0a1e60 msgr2=0x7f46bc121010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:27.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.116+0000 7f46c0937640 1 --2- 172.21.15.40:0/1584443090 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f46bc0a1e60 0x7f46bc121010 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f46b000c410 tx=0x7f46b000c8d0 comp rx=0 tx=0).stop 2024-04-22T08:16:27.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.116+0000 7f46bad9d640 1 -- 172.21.15.40:0/1584443090 reap_dead start 2024-04-22T08:16:27.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.116+0000 7f46c0937640 1 -- 172.21.15.40:0/1584443090 shutdown_connections 2024-04-22T08:16:27.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.116+0000 7f46c0937640 1 --2- 172.21.15.40:0/1584443090 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f46bc0a1e60 0x7f46bc121010 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:27.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.116+0000 7f46c0937640 1 -- 172.21.15.40:0/1584443090 >> 172.21.15.40:0/1584443090 conn(0x7f46bc0a66a0 msgr2=0x7f46bc09b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:16:27.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.116+0000 7f46c0937640 1 -- 172.21.15.40:0/1584443090 shutdown_connections 2024-04-22T08:16:27.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:27.116+0000 7f46c0937640 1 -- 172.21.15.40:0/1584443090 wait complete. 2024-04-22T08:16:27.137 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:16:27.137 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-04-22T08:16:27.137 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2024-04-22T08:16:27.138 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-04-22T08:16:27.138 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-04-22T08:16:27.138 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-22T08:16:27.138 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T08:16:27.154 INFO:tasks.ceph.mon.b:Stopped 2024-04-22T08:16:27.154 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-04-22T08:16:27.155 INFO:tasks.ceph.mon.c:Sent signal 19 2024-04-22T08:16:27.155 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-04-22T08:16:42.156 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-04-22T08:16:42.157 INFO:tasks.ceph.mon.c:Sent signal 18 2024-04-22T08:16:42.157 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-22T08:16:42.157 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:16:42.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.268+0000 7fd87f6bd640 10 monclient: get_monmap_and_config 2024-04-22T08:16:42.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.268+0000 7fd87f6bd640 10 monclient: build_initial_monmap 2024-04-22T08:16:42.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.268+0000 7fd87f6bd640 10 monclient: monmap: 2024-04-22T08:16:42.268 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:16:42.268 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:42.268 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:16:42.269275+0000 2024-04-22T08:16:42.268 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:16:42.269275+0000 2024-04-22T08:16:42.268 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:16:42.268 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:42.268 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:16:42.268 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:16:42.269 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:16:42.269 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:16:42.269 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:16:42.269 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:42.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.269+0000 7fd87f6bd640 1 Processor -- start 2024-04-22T08:16:42.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.269+0000 7fd87f6bd640 1 -- start start 2024-04-22T08:16:42.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.269+0000 7fd87f6bd640 10 monclient: init 2024-04-22T08:16:42.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.269+0000 7fd87f6bd640 10 monclient: _reopen_session rank -1 2024-04-22T08:16:42.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.269+0000 7fd87f6bd640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-04-22T08:16:42.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.269+0000 7fd87f6bd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd878135e30 addr v1:172.21.15.40:6789/0 2024-04-22T08:16:42.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.269+0000 7fd87f6bd640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd878071c90 0x7fd87813aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:42.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.269+0000 7fd87f6bd640 10 monclient(hunting): picked mon.noname-a con 0x7fd878071c90 addr v2:172.21.15.40:3300/0 2024-04-22T08:16:42.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.269+0000 7fd87f6bd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd87813b410 0x7fd87813d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:42.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.269+0000 7fd87f6bd640 10 monclient(hunting): picked mon.noname-b con 0x7fd87813b410 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:16:42.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.269+0000 7fd87f6bd640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd87809e340 con 0x7fd878135e30 2024-04-22T08:16:42.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.269+0000 7fd87f6bd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:42.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.269+0000 7fd87f6bd640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd87809de80 con 0x7fd878071c90 2024-04-22T08:16:42.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.269+0000 7fd87f6bd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:42.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.269+0000 7fd87f6bd640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd87809f130 con 0x7fd87813b410 2024-04-22T08:16:42.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.269+0000 7fd87f6bd640 10 monclient(hunting): _renew_subs 2024-04-22T08:16:42.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.269+0000 7fd87f6bd640 10 monclient(hunting): authenticate will time out at 1076.263601s 2024-04-22T08:16:42.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.269+0000 7fd87cc58640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd878071c90 0x7fd87813aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:42.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.269+0000 7fd87dc5a640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd87813b410 0x7fd87813d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:42.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.269+0000 7fd87d459640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fd878135e30 0x7fd878136210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:34474/0 (socket says 172.21.15.40:34474) 2024-04-22T08:16:42.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.269+0000 7fd87d459640 1 -- 172.21.15.40:0/1899618789 learned_addr learned my addr 172.21.15.40:0/1899618789 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:16:42.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.270+0000 7fd87cc58640 10 monclient(hunting): get_auth_request con 0x7fd878071c90 auth_method 0 2024-04-22T08:16:42.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.270+0000 7fd87cc58640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:42.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.270+0000 7fd87cc58640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:42.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.270+0000 7fd87cc58640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:42.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.270+0000 7fd87dc5a640 10 monclient(hunting): get_auth_request con 0x7fd87813b410 auth_method 0 2024-04-22T08:16:42.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.270+0000 7fd87dc5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:42.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.270+0000 7fd87dc5a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:42.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.270+0000 7fd87dc5a640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:42.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.270+0000 7fd87cc58640 1 -- 172.21.15.40:0/1899618789 >> v2:172.21.15.40:3300/0 conn(0x7fd878071c90 msgr2=0x7fd87813aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:16:42.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.270+0000 7fd87cc58640 1 -- 172.21.15.40:0/1899618789 >> v2:172.21.15.40:3300/0 conn(0x7fd878071c90 msgr2=0x7fd87813aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:16:42.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.270+0000 7fd87cc58640 1 --2- 172.21.15.40:0/1899618789 >> v2:172.21.15.40:3300/0 conn(0x7fd878071c90 0x7fd87813aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:16:42.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.270+0000 7fd87dc5a640 1 -- 172.21.15.40:0/1899618789 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd87813b410 msgr2=0x7fd87813d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-22T08:16:42.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.270+0000 7fd87dc5a640 1 -- 172.21.15.40:0/1899618789 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd87813b410 msgr2=0x7fd87813d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:16:42.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.270+0000 7fd87dc5a640 1 --2- 172.21.15.40:0/1899618789 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd87813b410 0x7fd87813d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:16:42.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.270+0000 7fd87cc58640 1 --2- 172.21.15.40:0/1899618789 >> v2:172.21.15.40:3300/0 conn(0x7fd878071c90 0x7fd87813aed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:16:42.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.270+0000 7fd87dc5a640 1 --2- 172.21.15.40:0/1899618789 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd87813b410 0x7fd87813d800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:16:42.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.272+0000 7fd87d459640 1 -- 172.21.15.40:0/1899618789 >> v1:172.21.15.40:6789/0 conn(0x7fd878135e30 legacy=0x7fd878136210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:16:42.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.471+0000 7fd87cc58640 1 --2- 172.21.15.40:0/1899618789 >> v2:172.21.15.40:3300/0 conn(0x7fd878071c90 0x7fd87813aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:42.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.471+0000 7fd87dc5a640 1 --2- 172.21.15.40:0/1899618789 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd87813b410 0x7fd87813d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:42.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.471+0000 7fd87cc58640 10 monclient(hunting): get_auth_request con 0x7fd878071c90 auth_method 0 2024-04-22T08:16:42.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.471+0000 7fd87cc58640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:42.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.471+0000 7fd87cc58640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:42.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.471+0000 7fd87cc58640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:16:42.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.471+0000 7fd87dc5a640 10 monclient(hunting): get_auth_request con 0x7fd87813b410 auth_method 0 2024-04-22T08:16:42.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.471+0000 7fd87dc5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:42.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.471+0000 7fd87dc5a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:42.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.471+0000 7fd87dc5a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:16:42.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.471+0000 7fd87cc58640 1 -- 172.21.15.40:0/1899618789 >> v2:172.21.15.40:3300/0 conn(0x7fd878071c90 msgr2=0x7fd87813aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:16:42.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.471+0000 7fd87cc58640 1 -- 172.21.15.40:0/1899618789 >> v2:172.21.15.40:3300/0 conn(0x7fd878071c90 msgr2=0x7fd87813aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:16:42.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.471+0000 7fd87cc58640 1 --2- 172.21.15.40:0/1899618789 >> v2:172.21.15.40:3300/0 conn(0x7fd878071c90 0x7fd87813aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:16:42.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.471+0000 7fd87dc5a640 1 -- 172.21.15.40:0/1899618789 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd87813b410 msgr2=0x7fd87813d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-22T08:16:42.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.471+0000 7fd87dc5a640 1 -- 172.21.15.40:0/1899618789 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd87813b410 msgr2=0x7fd87813d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:16:42.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.471+0000 7fd87dc5a640 1 --2- 172.21.15.40:0/1899618789 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd87813b410 0x7fd87813d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:16:42.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.471+0000 7fd87cc58640 1 --2- 172.21.15.40:0/1899618789 >> v2:172.21.15.40:3300/0 conn(0x7fd878071c90 0x7fd87813aed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:16:42.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.471+0000 7fd87dc5a640 1 --2- 172.21.15.40:0/1899618789 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd87813b410 0x7fd87813d800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:16:42.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.872+0000 7fd87cc58640 1 --2- 172.21.15.40:0/1899618789 >> v2:172.21.15.40:3300/0 conn(0x7fd878071c90 0x7fd87813aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:42.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.872+0000 7fd87dc5a640 1 --2- 172.21.15.40:0/1899618789 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd87813b410 0x7fd87813d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:42.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.872+0000 7fd87cc58640 10 monclient(hunting): get_auth_request con 0x7fd878071c90 auth_method 0 2024-04-22T08:16:42.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.872+0000 7fd87cc58640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:42.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.872+0000 7fd87cc58640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:42.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.872+0000 7fd87cc58640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:16:42.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.872+0000 7fd87dc5a640 10 monclient(hunting): get_auth_request con 0x7fd87813b410 auth_method 0 2024-04-22T08:16:42.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.872+0000 7fd87dc5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:42.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.872+0000 7fd87dc5a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:42.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.872+0000 7fd87dc5a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:16:42.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.872+0000 7fd87cc58640 1 -- 172.21.15.40:0/1899618789 >> v2:172.21.15.40:3300/0 conn(0x7fd878071c90 msgr2=0x7fd87813aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-22T08:16:42.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.872+0000 7fd87cc58640 1 -- 172.21.15.40:0/1899618789 >> v2:172.21.15.40:3300/0 conn(0x7fd878071c90 msgr2=0x7fd87813aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:16:42.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.872+0000 7fd87cc58640 1 --2- 172.21.15.40:0/1899618789 >> v2:172.21.15.40:3300/0 conn(0x7fd878071c90 0x7fd87813aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:16:42.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.872+0000 7fd87dc5a640 1 -- 172.21.15.40:0/1899618789 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd87813b410 msgr2=0x7fd87813d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:16:42.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.872+0000 7fd87dc5a640 1 -- 172.21.15.40:0/1899618789 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd87813b410 msgr2=0x7fd87813d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:16:42.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.872+0000 7fd87dc5a640 1 --2- 172.21.15.40:0/1899618789 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd87813b410 0x7fd87813d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:16:42.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.872+0000 7fd87cc58640 1 --2- 172.21.15.40:0/1899618789 >> v2:172.21.15.40:3300/0 conn(0x7fd878071c90 0x7fd87813aed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:16:42.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:42.872+0000 7fd87dc5a640 1 --2- 172.21.15.40:0/1899618789 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd87813b410 0x7fd87813d800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:16:43.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:43.673+0000 7fd87dc5a640 1 --2- 172.21.15.40:0/1899618789 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd87813b410 0x7fd87813d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:43.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:43.673+0000 7fd87dc5a640 10 monclient(hunting): get_auth_request con 0x7fd87813b410 auth_method 0 2024-04-22T08:16:43.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:43.673+0000 7fd87dc5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:43.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:43.673+0000 7fd87dc5a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:43.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:43.673+0000 7fd87dc5a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:16:43.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:43.673+0000 7fd87cc58640 1 --2- 172.21.15.40:0/1899618789 >> v2:172.21.15.40:3300/0 conn(0x7fd878071c90 0x7fd87813aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:43.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:43.673+0000 7fd87cc58640 10 monclient(hunting): get_auth_request con 0x7fd878071c90 auth_method 0 2024-04-22T08:16:43.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:43.673+0000 7fd87cc58640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:43.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:43.673+0000 7fd87cc58640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:43.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:43.673+0000 7fd87cc58640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:16:43.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:43.673+0000 7fd87dc5a640 1 -- 172.21.15.40:0/1899618789 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd87813b410 msgr2=0x7fd87813d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:16:43.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:43.673+0000 7fd87dc5a640 1 -- 172.21.15.40:0/1899618789 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd87813b410 msgr2=0x7fd87813d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:16:43.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:43.673+0000 7fd87dc5a640 1 --2- 172.21.15.40:0/1899618789 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd87813b410 0x7fd87813d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:16:43.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:43.673+0000 7fd87cc58640 1 -- 172.21.15.40:0/1899618789 >> v2:172.21.15.40:3300/0 conn(0x7fd878071c90 msgr2=0x7fd87813aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-22T08:16:43.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:43.673+0000 7fd87cc58640 1 -- 172.21.15.40:0/1899618789 >> v2:172.21.15.40:3300/0 conn(0x7fd878071c90 msgr2=0x7fd87813aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:16:43.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:43.673+0000 7fd87cc58640 1 --2- 172.21.15.40:0/1899618789 >> v2:172.21.15.40:3300/0 conn(0x7fd878071c90 0x7fd87813aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:16:43.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:43.673+0000 7fd87dc5a640 1 --2- 172.21.15.40:0/1899618789 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd87813b410 0x7fd87813d800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T08:16:43.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:43.673+0000 7fd87cc58640 1 --2- 172.21.15.40:0/1899618789 >> v2:172.21.15.40:3300/0 conn(0x7fd878071c90 0x7fd87813aed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T08:16:45.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:45.270+0000 7fd86effd640 10 monclient(hunting): tick 2024-04-22T08:16:45.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:45.270+0000 7fd86effd640 10 monclient(hunting): _check_auth_tickets 2024-04-22T08:16:45.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:45.270+0000 7fd86effd640 1 monclient(hunting): continuing hunt 2024-04-22T08:16:45.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:45.270+0000 7fd86effd640 10 monclient(hunting): _reopen_session rank -1 2024-04-22T08:16:45.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:45.270+0000 7fd86effd640 1 -- 172.21.15.40:0/1899618789 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd87813b410 msgr2=0x7fd87813d800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:16:45.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:45.270+0000 7fd86effd640 1 --2- 172.21.15.40:0/1899618789 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd87813b410 0x7fd87813d800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:45.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:45.270+0000 7fd86effd640 1 -- 172.21.15.40:0/1899618789 >> v2:172.21.15.40:3300/0 conn(0x7fd878071c90 msgr2=0x7fd87813aed0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:16:45.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:45.270+0000 7fd86effd640 1 --2- 172.21.15.40:0/1899618789 >> v2:172.21.15.40:3300/0 conn(0x7fd878071c90 0x7fd87813aed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:45.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:45.270+0000 7fd86effd640 1 -- 172.21.15.40:0/1899618789 >> v1:172.21.15.40:6789/0 conn(0x7fd878135e30 legacy=0x7fd878136210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:45.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:45.270+0000 7fd86effd640 10 monclient: _add_conns ranks=[1,4] 2024-04-22T08:16:45.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:45.270+0000 7fd86effd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd85c003790 addr v1:172.21.15.189:6789/0 2024-04-22T08:16:45.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:45.270+0000 7fd86effd640 1 --2- 172.21.15.40:0/1899618789 >> v2:172.21.15.189:3300/0 conn(0x7fd85c0074a0 0x7fd85c009960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:45.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:45.270+0000 7fd86effd640 10 monclient(hunting): picked mon.noname-c con 0x7fd85c0074a0 addr v2:172.21.15.189:3300/0 2024-04-22T08:16:45.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:45.270+0000 7fd86effd640 1 -- 172.21.15.40:0/1899618789 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd87809f410 con 0x7fd85c003790 2024-04-22T08:16:45.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:45.270+0000 7fd86effd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:45.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:45.270+0000 7fd86effd640 1 -- 172.21.15.40:0/1899618789 --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd87809de80 con 0x7fd85c0074a0 2024-04-22T08:16:45.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:45.270+0000 7fd86effd640 10 monclient(hunting): _renew_subs 2024-04-22T08:16:45.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:45.270+0000 7fd87d459640 1 -- 172.21.15.40:0/1899618789 >> v1:172.21.15.189:6789/0 conn(0x7fd85c003790 legacy=0x7fd85c005bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-04-22T08:16:45.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:45.270+0000 7fd87cc58640 1 -- 172.21.15.40:0/1899618789 >> v2:172.21.15.189:3300/0 conn(0x7fd85c0074a0 msgr2=0x7fd85c009960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.189:3300/0 2024-04-22T08:16:45.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:45.270+0000 7fd87cc58640 1 --2- 172.21.15.40:0/1899618789 >> v2:172.21.15.189:3300/0 conn(0x7fd85c0074a0 0x7fd85c009960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:16:45.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:45.471+0000 7fd87d459640 1 -- 172.21.15.40:0/1899618789 >> v1:172.21.15.189:6789/0 conn(0x7fd85c003790 legacy=0x7fd85c005bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-04-22T08:16:45.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:45.471+0000 7fd87cc58640 1 -- 172.21.15.40:0/1899618789 >> v2:172.21.15.189:3300/0 conn(0x7fd85c0074a0 msgr2=0x7fd85c009960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.189:3300/0 2024-04-22T08:16:45.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:45.471+0000 7fd87cc58640 1 --2- 172.21.15.40:0/1899618789 >> v2:172.21.15.189:3300/0 conn(0x7fd85c0074a0 0x7fd85c009960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:16:45.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:45.872+0000 7fd87cc58640 1 -- 172.21.15.40:0/1899618789 >> v2:172.21.15.189:3300/0 conn(0x7fd85c0074a0 msgr2=0x7fd85c009960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.189:3300/0 2024-04-22T08:16:45.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:45.872+0000 7fd87d459640 1 -- 172.21.15.40:0/1899618789 >> v1:172.21.15.189:6789/0 conn(0x7fd85c003790 legacy=0x7fd85c005bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-04-22T08:16:45.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:45.872+0000 7fd87cc58640 1 --2- 172.21.15.40:0/1899618789 >> v2:172.21.15.189:3300/0 conn(0x7fd85c0074a0 0x7fd85c009960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:16:46.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:46.673+0000 7fd87d459640 1 -- 172.21.15.40:0/1899618789 >> v1:172.21.15.189:6789/0 conn(0x7fd85c003790 legacy=0x7fd85c005bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-04-22T08:16:46.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:46.673+0000 7fd87cc58640 1 -- 172.21.15.40:0/1899618789 >> v2:172.21.15.189:3300/0 conn(0x7fd85c0074a0 msgr2=0x7fd85c009960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.189:3300/0 2024-04-22T08:16:46.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:46.673+0000 7fd87cc58640 1 --2- 172.21.15.40:0/1899618789 >> v2:172.21.15.189:3300/0 conn(0x7fd85c0074a0 0x7fd85c009960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T08:16:47.281 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36517]: starting. 2024-04-22T08:16:47.281 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36517]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-36516 2024-04-22T08:16:47.281 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36517]: created object 0... 2024-04-22T08:16:47.281 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36517]: created object 25... 2024-04-22T08:16:47.281 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36517]: created object 49... 2024-04-22T08:16:47.281 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36517]: finishing. 2024-04-22T08:16:47.281 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36517]: shutting down. 2024-04-22T08:16:48.175 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[36518]: starting. 2024-04-22T08:16:48.175 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[36518]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-36516 2024-04-22T08:16:48.176 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[36518]: shutting down. 2024-04-22T08:16:48.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.270+0000 7fd86effd640 10 monclient(hunting): tick 2024-04-22T08:16:48.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.270+0000 7fd86effd640 10 monclient(hunting): _check_auth_tickets 2024-04-22T08:16:48.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.270+0000 7fd86effd640 1 monclient(hunting): continuing hunt 2024-04-22T08:16:48.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.270+0000 7fd86effd640 10 monclient(hunting): _reopen_session rank -1 2024-04-22T08:16:48.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.270+0000 7fd86effd640 1 -- 172.21.15.40:0/1899618789 >> v2:172.21.15.189:3300/0 conn(0x7fd85c0074a0 msgr2=0x7fd85c009960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:16:48.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.270+0000 7fd86effd640 1 --2- 172.21.15.40:0/1899618789 >> v2:172.21.15.189:3300/0 conn(0x7fd85c0074a0 0x7fd85c009960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:48.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.270+0000 7fd86effd640 1 -- 172.21.15.40:0/1899618789 >> v1:172.21.15.189:6789/0 conn(0x7fd85c003790 legacy=0x7fd85c005bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:16:48.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.270+0000 7fd86effd640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-04-22T08:16:48.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.270+0000 7fd87dc5a640 1 -- 172.21.15.40:0/1899618789 reap_dead start 2024-04-22T08:16:48.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.270+0000 7fd86effd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd878135e30 addr v1:172.21.15.40:6789/0 2024-04-22T08:16:48.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.270+0000 7fd86effd640 1 --2- 172.21.15.40:0/1899618789 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd85c00a740 0x7fd85c009960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:48.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.270+0000 7fd86effd640 10 monclient(hunting): picked mon.noname-b con 0x7fd85c00a740 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:16:48.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.270+0000 7fd86effd640 1 --2- 172.21.15.40:0/1899618789 >> v2:172.21.15.40:3300/0 conn(0x7fd85c00ac30 0x7fd85c005bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:48.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.270+0000 7fd86effd640 10 monclient(hunting): picked mon.noname-a con 0x7fd85c00ac30 addr v2:172.21.15.40:3300/0 2024-04-22T08:16:48.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.270+0000 7fd86effd640 1 -- 172.21.15.40:0/1899618789 --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd85c00b030 con 0x7fd878135e30 2024-04-22T08:16:48.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.270+0000 7fd86effd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:48.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.270+0000 7fd86effd640 1 -- 172.21.15.40:0/1899618789 --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd87809de80 con 0x7fd85c00ac30 2024-04-22T08:16:48.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.270+0000 7fd86effd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:48.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.270+0000 7fd86effd640 1 -- 172.21.15.40:0/1899618789 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd87809f130 con 0x7fd85c00a740 2024-04-22T08:16:48.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.270+0000 7fd86effd640 10 monclient(hunting): _renew_subs 2024-04-22T08:16:48.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.270+0000 7fd87cc58640 1 --2- 172.21.15.40:0/1899618789 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd85c00a740 0x7fd85c009960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:48.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.270+0000 7fd87dc5a640 1 --2- 172.21.15.40:0/1899618789 >> v2:172.21.15.40:3300/0 conn(0x7fd85c00ac30 0x7fd85c005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:48.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.270+0000 7fd87cc58640 10 monclient(hunting): get_auth_request con 0x7fd85c00a740 auth_method 0 2024-04-22T08:16:48.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.270+0000 7fd87cc58640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:48.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.270+0000 7fd87cc58640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:48.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.270+0000 7fd87cc58640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:48.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.270+0000 7fd87dc5a640 10 monclient(hunting): get_auth_request con 0x7fd85c00ac30 auth_method 0 2024-04-22T08:16:48.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.270+0000 7fd87dc5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:48.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.270+0000 7fd87dc5a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:48.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.270+0000 7fd87dc5a640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:48.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.270+0000 7fd87cc58640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:48.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.270+0000 7fd87cc58640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:48.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.270+0000 7fd87cc58640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:48.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.270+0000 7fd87dc5a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:48.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.270+0000 7fd87dc5a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:48.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.270+0000 7fd87dc5a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:48.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.270+0000 7fd87cc58640 10 monclient(hunting): handle_auth_done global_id 54194 payload 274 2024-04-22T08:16:48.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.270+0000 7fd87cc58640 1 -- 172.21.15.40:0/1899618789 >> v2:172.21.15.40:3300/0 conn(0x7fd85c00ac30 msgr2=0x7fd85c005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:16:48.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.270+0000 7fd87cc58640 1 --2- 172.21.15.40:0/1899618789 >> v2:172.21.15.40:3300/0 conn(0x7fd85c00ac30 0x7fd85c005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:48.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.270+0000 7fd87cc58640 1 -- 172.21.15.40:0/1899618789 >> v1:172.21.15.40:6789/0 conn(0x7fd878135e30 legacy=0x7fd85c00a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:16:48.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.270+0000 7fd87cc58640 10 monclient: _finish_hunting 0 2024-04-22T08:16:48.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.270+0000 7fd87cc58640 1 monclient: found mon.noname-b 2024-04-22T08:16:48.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.270+0000 7fd87cc58640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:16:48.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.270+0000 7fd87cc58640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:16:48.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.270+0000 7fd87cc58640 1 -- 172.21.15.40:0/1899618789 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd87809f410 con 0x7fd85c00a740 2024-04-22T08:16:48.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.270+0000 7fd87dc5a640 1 --2- 172.21.15.40:0/1899618789 >> v2:172.21.15.40:3300/0 conn(0x7fd85c00ac30 0x7fd85c005bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:16:48.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.273+0000 7fd87cc58640 1 -- 172.21.15.40:0/1899618789 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd85c00a740 msgr2=0x7fd85c009960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:16:48.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.273+0000 7fd87cc58640 1 --2- 172.21.15.40:0/1899618789 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd85c00a740 0x7fd85c009960 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd868006e60 tx=0x7fd86802c390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d66358c94f2a3325 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:48.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.274+0000 7fd86ffff640 1 -- 172.21.15.40:0/1899618789 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd868032480 con 0x7fd85c00a740 2024-04-22T08:16:48.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.274+0000 7fd86ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:48.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.274+0000 7fd86ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:16:48.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.274+0000 7fd86ffff640 10 monclient: dump: 2024-04-22T08:16:48.285 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:48.285 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:48.285 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:48.285 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:48.285 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:48.285 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:48.285 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:48.285 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:48.286 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:48.286 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:48.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.274+0000 7fd86ffff640 10 monclient: _finish_auth 0 2024-04-22T08:16:48.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.274+0000 7fd86ffff640 10 monclient: _check_auth_tickets 2024-04-22T08:16:48.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.274+0000 7fd86ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:16:48.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.274+0000 7fd86ffff640 1 -- 172.21.15.40:0/1899618789 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd86802f030 con 0x7fd85c00a740 2024-04-22T08:16:48.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.274+0000 7fd86ffff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:16:48.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.274+0000 7fd86ffff640 1 -- 172.21.15.40:0/1899618789 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd868032840 con 0x7fd85c00a740 2024-04-22T08:16:48.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.274+0000 7fd87f6bd640 5 monclient: authenticate success, global_id 54194 2024-04-22T08:16:48.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.274+0000 7fd87f6bd640 10 monclient: get_monmap_and_config success 2024-04-22T08:16:48.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.274+0000 7fd87f6bd640 4 set_mon_vals no callback set 2024-04-22T08:16:48.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.274+0000 7fd86ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:48.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.274+0000 7fd86ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:16:48.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.274+0000 7fd86ffff640 10 monclient: dump: 2024-04-22T08:16:48.287 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:48.287 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:48.287 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:48.287 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:48.287 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:48.287 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:48.287 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:48.287 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:48.287 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:48.287 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:48.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.274+0000 7fd87f6bd640 10 monclient: shutdown 2024-04-22T08:16:48.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.274+0000 7fd87f6bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:16:48.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.274+0000 7fd87f6bd640 1 -- 172.21.15.40:0/1899618789 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd85c00a740 msgr2=0x7fd85c009960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:48.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.274+0000 7fd87f6bd640 1 --2- 172.21.15.40:0/1899618789 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd85c00a740 0x7fd85c009960 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd868006e60 tx=0x7fd86802c390 comp rx=0 tx=0).stop 2024-04-22T08:16:48.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.274+0000 7fd87f6bd640 1 -- 172.21.15.40:0/1899618789 shutdown_connections 2024-04-22T08:16:48.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.274+0000 7fd87f6bd640 1 --2- 172.21.15.40:0/1899618789 >> v2:172.21.15.40:3300/0 conn(0x7fd85c00ac30 0x7fd85c005bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:48.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.274+0000 7fd87f6bd640 1 --2- 172.21.15.40:0/1899618789 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd85c00a740 0x7fd85c009960 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:48.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.274+0000 7fd87f6bd640 1 -- 172.21.15.40:0/1899618789 >> 172.21.15.40:0/1899618789 conn(0x7fd8780a65c0 msgr2=0x7fd8780a2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:16:48.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.274+0000 7fd87f6bd640 1 -- 172.21.15.40:0/1899618789 shutdown_connections 2024-04-22T08:16:48.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.275+0000 7fd87f6bd640 1 -- 172.21.15.40:0/1899618789 wait complete. 2024-04-22T08:16:48.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.275+0000 7fd87f6bd640 10 monclient: build_initial_monmap 2024-04-22T08:16:48.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.275+0000 7fd87f6bd640 10 monclient: monmap: 2024-04-22T08:16:48.289 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:16:48.289 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:16:48.289 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:16:48.289 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:16:48.289 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:16:48.289 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:48.289 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:16:48.289 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:16:48.290 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:16:48.290 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:48.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.276+0000 7fd87f6bd640 1 Processor -- start 2024-04-22T08:16:48.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.276+0000 7fd87f6bd640 1 -- start start 2024-04-22T08:16:48.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.276+0000 7fd87f6bd640 10 monclient: init 2024-04-22T08:16:48.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.276+0000 7fd87f6bd640 10 monclient: _reopen_session rank -1 2024-04-22T08:16:48.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.276+0000 7fd87f6bd640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-22T08:16:48.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.276+0000 7fd87f6bd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd85c00a740 0x7fd8781cac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:48.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.276+0000 7fd87f6bd640 10 monclient(hunting): picked mon.noname-b con 0x7fd85c00a740 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:16:48.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.276+0000 7fd87f6bd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd85c00ac30 0x7fd87806e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:48.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.276+0000 7fd87f6bd640 10 monclient(hunting): picked mon.noname-a con 0x7fd85c00ac30 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:16:48.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.276+0000 7fd87f6bd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd878135e30 0x7fd87806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:48.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.276+0000 7fd87f6bd640 10 monclient(hunting): picked mon.noname-c con 0x7fd878135e30 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:16:48.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.276+0000 7fd87f6bd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:48.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.276+0000 7fd87f6bd640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd878120a90 con 0x7fd85c00ac30 2024-04-22T08:16:48.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.276+0000 7fd87f6bd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:48.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.276+0000 7fd87f6bd640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8780720e0 con 0x7fd85c00a740 2024-04-22T08:16:48.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.276+0000 7fd87f6bd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:48.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.276+0000 7fd87f6bd640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd87813aab0 con 0x7fd878135e30 2024-04-22T08:16:48.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.276+0000 7fd87f6bd640 10 monclient(hunting): _renew_subs 2024-04-22T08:16:48.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.276+0000 7fd87f6bd640 10 monclient(hunting): authenticate will time out at 1082.270470s 2024-04-22T08:16:48.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.276+0000 7fd87cc58640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd85c00ac30 0x7fd87806e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:48.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.276+0000 7fd87d459640 1 -- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd85c00a740 msgr2=0x7fd8781cac90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:3300/0 2024-04-22T08:16:48.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.276+0000 7fd87dc5a640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd878135e30 0x7fd87806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:48.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.276+0000 7fd87d459640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd85c00a740 0x7fd8781cac90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:16:48.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.276+0000 7fd87cc58640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd85c00ac30 0x7fd87806e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52420/0 (socket says 172.21.15.40:52420) 2024-04-22T08:16:48.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.276+0000 7fd87dc5a640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd878135e30 0x7fd87806eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:43142/0 (socket says 172.21.15.40:43142) 2024-04-22T08:16:48.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.276+0000 7fd87cc58640 1 -- 172.21.15.40:0/1666455926 learned_addr learned my addr 172.21.15.40:0/1666455926 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:16:48.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.276+0000 7fd87cc58640 10 monclient(hunting): get_auth_request con 0x7fd85c00ac30 auth_method 0 2024-04-22T08:16:48.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.276+0000 7fd87cc58640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:48.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.276+0000 7fd87cc58640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:48.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.276+0000 7fd87cc58640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:48.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.276+0000 7fd87dc5a640 10 monclient(hunting): get_auth_request con 0x7fd878135e30 auth_method 0 2024-04-22T08:16:48.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.276+0000 7fd87dc5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:48.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.276+0000 7fd87dc5a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:48.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.276+0000 7fd87dc5a640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:48.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.276+0000 7fd87dc5a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:48.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.276+0000 7fd87dc5a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:48.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.276+0000 7fd87dc5a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:48.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.276+0000 7fd87cc58640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:48.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.276+0000 7fd87cc58640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:48.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.276+0000 7fd87cc58640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:48.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.277+0000 7fd87dc5a640 10 monclient(hunting): handle_auth_done global_id 54197 payload 824 2024-04-22T08:16:48.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.277+0000 7fd87dc5a640 1 -- 172.21.15.40:0/1666455926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd85c00a740 msgr2=0x7fd8781cac90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:16:48.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.277+0000 7fd87dc5a640 1 --2- 172.21.15.40:0/1666455926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd85c00a740 0x7fd8781cac90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:48.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.277+0000 7fd87dc5a640 1 -- 172.21.15.40:0/1666455926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd85c00ac30 msgr2=0x7fd87806e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:48.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.277+0000 7fd87dc5a640 1 --2- 172.21.15.40:0/1666455926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd85c00ac30 0x7fd87806e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:48.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.277+0000 7fd87dc5a640 10 monclient: _finish_hunting 0 2024-04-22T08:16:48.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.277+0000 7fd87dc5a640 1 monclient: found mon.noname-c 2024-04-22T08:16:48.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.277+0000 7fd87dc5a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:16:48.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.277+0000 7fd87dc5a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:16:48.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.277+0000 7fd87dc5a640 1 -- 172.21.15.40:0/1666455926 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd870003d50 con 0x7fd878135e30 2024-04-22T08:16:48.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.277+0000 7fd87cc58640 1 --2- 172.21.15.40:0/1666455926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd85c00ac30 0x7fd87806e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:16:48.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.279+0000 7fd87dc5a640 1 -- 172.21.15.40:0/1666455926 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd878135e30 msgr2=0x7fd87806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:16:48.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.279+0000 7fd87dc5a640 1 --2- 172.21.15.40:0/1666455926 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd878135e30 0x7fd87806eb30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd8780a3e20 tx=0x7fd870016710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:48.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.279+0000 7fd86dffb640 1 -- 172.21.15.40:0/1666455926 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd870018630 con 0x7fd878135e30 2024-04-22T08:16:48.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.279+0000 7fd86dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:48.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.279+0000 7fd86dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:16:48.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.279+0000 7fd86dffb640 10 monclient: dump: 2024-04-22T08:16:48.295 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:48.295 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:48.295 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:48.295 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:48.296 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:48.296 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:48.296 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:48.296 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:48.296 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:48.296 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:48.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.279+0000 7fd86dffb640 10 monclient: _finish_auth 0 2024-04-22T08:16:48.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.279+0000 7fd86dffb640 10 monclient: _check_auth_tickets 2024-04-22T08:16:48.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.279+0000 7fd86dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:16:48.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.279+0000 7fd86dffb640 1 -- 172.21.15.40:0/1666455926 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8700187d0 con 0x7fd878135e30 2024-04-22T08:16:48.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.279+0000 7fd87f6bd640 5 monclient: authenticate success, global_id 54197 2024-04-22T08:16:48.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.279+0000 7fd87f6bd640 10 monclient: _renew_subs 2024-04-22T08:16:48.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.279+0000 7fd87f6bd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:16:48.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.279+0000 7fd87f6bd640 1 -- 172.21.15.40:0/1666455926 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd86800e040 con 0x7fd878135e30 2024-04-22T08:16:48.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.279+0000 7fd86dffb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:16:48.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.279+0000 7fd86dffb640 1 -- 172.21.15.40:0/1666455926 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd870018b80 con 0x7fd878135e30 2024-04-22T08:16:48.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.279+0000 7fd86dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:48.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.279+0000 7fd86dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:16:48.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.279+0000 7fd86dffb640 10 monclient: dump: 2024-04-22T08:16:48.297 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:48.297 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:48.297 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:48.297 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:48.298 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:48.298 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:48.298 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:48.298 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:48.298 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:48.298 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:48.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.280+0000 7fd87f6bd640 10 monclient: _renew_subs 2024-04-22T08:16:48.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.280+0000 7fd87f6bd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:16:48.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.280+0000 7fd87f6bd640 1 -- 172.21.15.40:0/1666455926 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8780701a0 con 0x7fd878135e30 2024-04-22T08:16:48.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.279+0000 7fd86f7fe640 4 set_mon_vals no callback set 2024-04-22T08:16:48.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.281+0000 7fd86dffb640 1 -- 172.21.15.40:0/1666455926 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fd870018ce0 con 0x7fd878135e30 2024-04-22T08:16:48.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.281+0000 7fd87f6bd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:16:48.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.281+0000 7fd87f6bd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:16:48.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.281+0000 7fd87f6bd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:16:48.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.281+0000 7fd87f6bd640 1 -- 172.21.15.40:0/1666455926 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd87804e9b0 con 0x7fd878135e30 2024-04-22T08:16:48.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.281+0000 7fd86dffb640 1 --2- 172.21.15.40:0/1666455926 >> v2:172.21.15.189:6800/30168 conn(0x7fd85c039060 0x7fd85c03b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:48.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.281+0000 7fd86dffb640 1 -- 172.21.15.40:0/1666455926 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 3506+0+0 (secure 0 0 0) 0x7fd87004b4c0 con 0x7fd878135e30 2024-04-22T08:16:48.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.281+0000 7fd87d459640 1 --2- 172.21.15.40:0/1666455926 >> v2:172.21.15.189:6800/30168 conn(0x7fd85c039060 0x7fd85c03b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:48.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.281+0000 7fd87d459640 10 monclient: get_auth_request con 0x7fd85c039060 auth_method 0 2024-04-22T08:16:48.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.284+0000 7fd87d459640 1 --2- 172.21.15.40:0/1666455926 >> v2:172.21.15.189:6800/30168 conn(0x7fd85c039060 0x7fd85c03b520 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd874001460 tx=0x7fd8740079c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:48.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.286+0000 7fd86dffb640 1 -- 172.21.15.40:0/1666455926 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7fd870028ae0 con 0x7fd878135e30 2024-04-22T08:16:48.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.286+0000 7fd86dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:16:48.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.286+0000 7fd86dffb640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:16:48.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.534+0000 7fd87f6bd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:16:48.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.534+0000 7fd87f6bd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:16:48.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.534+0000 7fd87f6bd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:16:48.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.534+0000 7fd87f6bd640 1 -- 172.21.15.40:0/1666455926 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd878120fb0 con 0x7fd878135e30 2024-04-22T08:16:48.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.534+0000 7fd86dffb640 1 -- 172.21.15.40:0/1666455926 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1537 (secure 0 0 0) 0x7fd870028900 con 0x7fd878135e30 2024-04-22T08:16:48.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.534+0000 7fd86dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:16:48.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.534+0000 7fd86dffb640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:16:48.534 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":44,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:16:48.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.536+0000 7fd87f6bd640 1 -- 172.21.15.40:0/1666455926 >> v2:172.21.15.189:6800/30168 conn(0x7fd85c039060 msgr2=0x7fd85c03b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:48.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.536+0000 7fd87f6bd640 1 --2- 172.21.15.40:0/1666455926 >> v2:172.21.15.189:6800/30168 conn(0x7fd85c039060 0x7fd85c03b520 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd874001460 tx=0x7fd8740079c0 comp rx=0 tx=0).stop 2024-04-22T08:16:48.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.536+0000 7fd87f6bd640 10 monclient: shutdown 2024-04-22T08:16:48.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.536+0000 7fd87f6bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:16:48.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.536+0000 7fd87f6bd640 1 -- 172.21.15.40:0/1666455926 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd878135e30 msgr2=0x7fd87806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:48.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.536+0000 7fd87f6bd640 1 --2- 172.21.15.40:0/1666455926 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd878135e30 0x7fd87806eb30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd8780a3e20 tx=0x7fd870016710 comp rx=0 tx=0).stop 2024-04-22T08:16:48.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.536+0000 7fd87f6bd640 1 -- 172.21.15.40:0/1666455926 shutdown_connections 2024-04-22T08:16:48.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.536+0000 7fd87f6bd640 1 --2- 172.21.15.40:0/1666455926 >> v2:172.21.15.189:6800/30168 conn(0x7fd85c039060 0x7fd85c03b520 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:48.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.536+0000 7fd87f6bd640 1 --2- 172.21.15.40:0/1666455926 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd878135e30 0x7fd87806eb30 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:48.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.536+0000 7fd87f6bd640 1 --2- 172.21.15.40:0/1666455926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd85c00ac30 0x7fd87806e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:48.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.536+0000 7fd87f6bd640 1 --2- 172.21.15.40:0/1666455926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd85c00a740 0x7fd8781cac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:48.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.536+0000 7fd87f6bd640 1 -- 172.21.15.40:0/1666455926 >> 172.21.15.40:0/1666455926 conn(0x7fd8780a65c0 msgr2=0x7fd8780a1120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:16:48.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.536+0000 7fd87f6bd640 1 -- 172.21.15.40:0/1666455926 shutdown_connections 2024-04-22T08:16:48.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.536+0000 7fd87f6bd640 1 -- 172.21.15.40:0/1666455926 wait complete. 2024-04-22T08:16:48.546 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-22T08:16:48.546 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-04-22T08:16:48.546 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-22T08:16:48.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.646+0000 7fce90de4640 10 monclient: get_monmap_and_config 2024-04-22T08:16:48.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.646+0000 7fce90de4640 10 monclient: build_initial_monmap 2024-04-22T08:16:48.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.646+0000 7fce90de4640 10 monclient: monmap: 2024-04-22T08:16:48.646 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:16:48.647 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:48.647 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:16:48.647355+0000 2024-04-22T08:16:48.647 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:16:48.647355+0000 2024-04-22T08:16:48.647 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:16:48.647 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:48.647 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:16:48.647 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:16:48.647 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:16:48.647 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:16:48.647 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:16:48.647 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:48.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.647+0000 7fce90de4640 1 Processor -- start 2024-04-22T08:16:48.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.647+0000 7fce90de4640 1 -- start start 2024-04-22T08:16:48.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.647+0000 7fce90de4640 10 monclient: init 2024-04-22T08:16:48.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.647+0000 7fce90de4640 10 monclient: _reopen_session rank -1 2024-04-22T08:16:48.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.647+0000 7fce90de4640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-04-22T08:16:48.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.647+0000 7fce90de4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fce8c0a1a80 addr v1:172.21.15.40:6789/0 2024-04-22T08:16:48.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.647+0000 7fce90de4640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce8c0a2570 0x7fce8c0a01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:48.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.647+0000 7fce90de4640 10 monclient(hunting): picked mon.noname-b con 0x7fce8c0a2570 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:16:48.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.647+0000 7fce90de4640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fce8c13b0e0 0x7fce8c13d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:48.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.647+0000 7fce90de4640 10 monclient(hunting): picked mon.noname-a con 0x7fce8c13b0e0 addr v2:172.21.15.40:3300/0 2024-04-22T08:16:48.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.647+0000 7fce90de4640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce8c09d860 con 0x7fce8c0a1a80 2024-04-22T08:16:48.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.647+0000 7fce90de4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:48.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.647+0000 7fce90de4640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce8c09f880 con 0x7fce8c13b0e0 2024-04-22T08:16:48.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.647+0000 7fce90de4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:48.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.647+0000 7fce90de4640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce8c09ef50 con 0x7fce8c0a2570 2024-04-22T08:16:48.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.647+0000 7fce90de4640 10 monclient(hunting): _renew_subs 2024-04-22T08:16:48.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.647+0000 7fce90de4640 10 monclient(hunting): authenticate will time out at 1082.641567s 2024-04-22T08:16:48.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.647+0000 7fce8a59c640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fce8c0a1a80 0x7fce8c0a1e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:34494/0 (socket says 172.21.15.40:34494) 2024-04-22T08:16:48.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.647+0000 7fce8a59c640 1 -- 172.21.15.40:0/351036637 learned_addr learned my addr 172.21.15.40:0/351036637 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:16:48.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.647+0000 7fce83fff640 1 --2- 172.21.15.40:0/351036637 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce8c0a2570 0x7fce8c0a01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:48.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.647+0000 7fce8ad9d640 1 --2- 172.21.15.40:0/351036637 >> v2:172.21.15.40:3300/0 conn(0x7fce8c13b0e0 0x7fce8c13d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:48.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.647+0000 7fce83fff640 10 monclient(hunting): get_auth_request con 0x7fce8c0a2570 auth_method 0 2024-04-22T08:16:48.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.647+0000 7fce83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:48.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.647+0000 7fce83fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:48.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.647+0000 7fce83fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:48.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.647+0000 7fce8ad9d640 10 monclient(hunting): get_auth_request con 0x7fce8c13b0e0 auth_method 0 2024-04-22T08:16:48.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.647+0000 7fce8ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:48.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.647+0000 7fce8ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:48.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.647+0000 7fce8ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:48.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.648+0000 7fce83fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:48.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.648+0000 7fce83fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:48.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.648+0000 7fce83fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:48.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.648+0000 7fce8ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:48.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.648+0000 7fce8ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:48.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.648+0000 7fce8ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:48.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.648+0000 7fce83fff640 10 monclient(hunting): handle_auth_done global_id 54200 payload 274 2024-04-22T08:16:48.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.648+0000 7fce83fff640 1 -- 172.21.15.40:0/351036637 >> v2:172.21.15.40:3300/0 conn(0x7fce8c13b0e0 msgr2=0x7fce8c13d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:16:48.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.648+0000 7fce83fff640 1 --2- 172.21.15.40:0/351036637 >> v2:172.21.15.40:3300/0 conn(0x7fce8c13b0e0 0x7fce8c13d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:48.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.648+0000 7fce83fff640 1 -- 172.21.15.40:0/351036637 >> v1:172.21.15.40:6789/0 conn(0x7fce8c0a1a80 legacy=0x7fce8c0a1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:16:48.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.648+0000 7fce83fff640 10 monclient: _finish_hunting 0 2024-04-22T08:16:48.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.648+0000 7fce83fff640 1 monclient: found mon.noname-b 2024-04-22T08:16:48.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.648+0000 7fce83fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:16:48.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.648+0000 7fce83fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:16:48.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.648+0000 7fce83fff640 1 -- 172.21.15.40:0/351036637 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce8c072160 con 0x7fce8c0a2570 2024-04-22T08:16:48.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.648+0000 7fce8ad9d640 1 --2- 172.21.15.40:0/351036637 >> v2:172.21.15.40:3300/0 conn(0x7fce8c13b0e0 0x7fce8c13d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:16:48.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.651+0000 7fce83fff640 1 -- 172.21.15.40:0/351036637 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce8c0a2570 msgr2=0x7fce8c0a01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:16:48.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.651+0000 7fce83fff640 1 --2- 172.21.15.40:0/351036637 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce8c0a2570 0x7fce8c0a01d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fce740097a0 tx=0x7fce7402b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb9e77dad3278a8a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:48.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.651+0000 7fce89d9b640 1 -- 172.21.15.40:0/351036637 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce74030920 con 0x7fce8c0a2570 2024-04-22T08:16:48.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.651+0000 7fce89d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:48.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.651+0000 7fce89d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:16:48.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.651+0000 7fce89d9b640 10 monclient: dump: 2024-04-22T08:16:48.658 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:48.658 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:48.658 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:48.658 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:48.659 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:48.659 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:48.659 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:48.659 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:48.659 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:48.659 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:48.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.651+0000 7fce89d9b640 10 monclient: _finish_auth 0 2024-04-22T08:16:48.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.651+0000 7fce89d9b640 10 monclient: _check_auth_tickets 2024-04-22T08:16:48.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.651+0000 7fce89d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:16:48.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.651+0000 7fce89d9b640 1 -- 172.21.15.40:0/351036637 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce7402e050 con 0x7fce8c0a2570 2024-04-22T08:16:48.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.651+0000 7fce90de4640 5 monclient: authenticate success, global_id 54200 2024-04-22T08:16:48.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.651+0000 7fce90de4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:16:48.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.651+0000 7fce89d9b640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:16:48.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.651+0000 7fce90de4640 10 monclient: get_monmap_and_config success 2024-04-22T08:16:48.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.651+0000 7fce89d9b640 1 -- 172.21.15.40:0/351036637 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce74030ca0 con 0x7fce8c0a2570 2024-04-22T08:16:48.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.651+0000 7fce89d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:48.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.651+0000 7fce90de4640 4 set_mon_vals no callback set 2024-04-22T08:16:48.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.651+0000 7fce89d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:16:48.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.651+0000 7fce89d9b640 10 monclient: dump: 2024-04-22T08:16:48.660 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:48.660 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:48.660 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:48.660 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:48.660 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:48.660 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:48.660 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:48.660 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:48.660 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:48.660 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:48.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.651+0000 7fce90de4640 10 monclient: shutdown 2024-04-22T08:16:48.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.651+0000 7fce90de4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:16:48.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.651+0000 7fce90de4640 1 -- 172.21.15.40:0/351036637 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce8c0a2570 msgr2=0x7fce8c0a01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:48.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.651+0000 7fce90de4640 1 --2- 172.21.15.40:0/351036637 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce8c0a2570 0x7fce8c0a01d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fce740097a0 tx=0x7fce7402b710 comp rx=0 tx=0).stop 2024-04-22T08:16:48.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.651+0000 7fce90de4640 1 -- 172.21.15.40:0/351036637 shutdown_connections 2024-04-22T08:16:48.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.651+0000 7fce90de4640 1 --2- 172.21.15.40:0/351036637 >> v2:172.21.15.40:3300/0 conn(0x7fce8c13b0e0 0x7fce8c13d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:48.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.651+0000 7fce90de4640 1 --2- 172.21.15.40:0/351036637 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce8c0a2570 0x7fce8c0a01d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:48.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.651+0000 7fce90de4640 1 -- 172.21.15.40:0/351036637 >> 172.21.15.40:0/351036637 conn(0x7fce8c0a66a0 msgr2=0x7fce8c0a2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:16:48.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.651+0000 7fce90de4640 1 -- 172.21.15.40:0/351036637 shutdown_connections 2024-04-22T08:16:48.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.651+0000 7fce90de4640 1 -- 172.21.15.40:0/351036637 wait complete. 2024-04-22T08:16:48.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.652+0000 7fce90de4640 10 monclient: build_initial_monmap 2024-04-22T08:16:48.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.652+0000 7fce90de4640 10 monclient: monmap: 2024-04-22T08:16:48.661 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:16:48.661 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:16:48.661 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:16:48.662 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:16:48.662 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:16:48.662 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:48.662 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:16:48.662 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:16:48.662 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:16:48.662 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:48.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.652+0000 7fce90de4640 1 Processor -- start 2024-04-22T08:16:48.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.652+0000 7fce90de4640 1 -- start start 2024-04-22T08:16:48.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.652+0000 7fce90de4640 10 monclient: init 2024-04-22T08:16:48.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.653+0000 7fce90de4640 10 monclient: _reopen_session rank -1 2024-04-22T08:16:48.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.653+0000 7fce90de4640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-22T08:16:48.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.653+0000 7fce90de4640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce8c0a1a80 0x7fce8c120d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:48.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.653+0000 7fce90de4640 10 monclient(hunting): picked mon.noname-c con 0x7fce8c0a1a80 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:16:48.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.653+0000 7fce90de4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce8c0a2570 0x7fce8c121280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:48.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.653+0000 7fce90de4640 10 monclient(hunting): picked mon.noname-a con 0x7fce8c0a2570 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:16:48.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.653+0000 7fce90de4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce8c13b0e0 0x7fce8c1217c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:48.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.653+0000 7fce90de4640 10 monclient(hunting): picked mon.noname-b con 0x7fce8c13b0e0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:16:48.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.653+0000 7fce90de4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:48.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.653+0000 7fce90de4640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce8c122110 con 0x7fce8c0a2570 2024-04-22T08:16:48.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.653+0000 7fce90de4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:48.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.653+0000 7fce90de4640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce8c124080 con 0x7fce8c13b0e0 2024-04-22T08:16:48.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.653+0000 7fce90de4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:48.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.653+0000 7fce90de4640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce8c09cc20 con 0x7fce8c0a1a80 2024-04-22T08:16:48.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.653+0000 7fce90de4640 10 monclient(hunting): _renew_subs 2024-04-22T08:16:48.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.653+0000 7fce90de4640 10 monclient(hunting): authenticate will time out at 1082.647092s 2024-04-22T08:16:48.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.653+0000 7fce83fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce8c0a2570 0x7fce8c121280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:48.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.653+0000 7fce8a59c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce8c0a1a80 0x7fce8c120d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:48.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.653+0000 7fce83fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce8c0a2570 0x7fce8c121280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52432/0 (socket says 172.21.15.40:52432) 2024-04-22T08:16:48.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.653+0000 7fce8a59c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce8c0a1a80 0x7fce8c120d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:43160/0 (socket says 172.21.15.40:43160) 2024-04-22T08:16:48.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.653+0000 7fce83fff640 1 -- 172.21.15.40:0/4157163184 learned_addr learned my addr 172.21.15.40:0/4157163184 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:16:48.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.653+0000 7fce83fff640 10 monclient(hunting): get_auth_request con 0x7fce8c0a2570 auth_method 0 2024-04-22T08:16:48.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.653+0000 7fce83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:48.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.653+0000 7fce83fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:48.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.653+0000 7fce83fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:48.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.653+0000 7fce8a59c640 10 monclient(hunting): get_auth_request con 0x7fce8c0a1a80 auth_method 0 2024-04-22T08:16:48.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.653+0000 7fce8a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:48.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.653+0000 7fce8a59c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:48.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.653+0000 7fce8ad9d640 1 -- 172.21.15.40:0/4157163184 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce8c13b0e0 msgr2=0x7fce8c1217c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:3300/0 2024-04-22T08:16:48.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.653+0000 7fce8a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:48.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.653+0000 7fce8ad9d640 1 --2- 172.21.15.40:0/4157163184 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce8c13b0e0 0x7fce8c1217c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:16:48.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.653+0000 7fce83fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:48.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.653+0000 7fce83fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:48.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.653+0000 7fce83fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:48.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.653+0000 7fce8a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:48.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.653+0000 7fce8a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:48.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.653+0000 7fce8a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:48.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.653+0000 7fce83fff640 10 monclient(hunting): handle_auth_done global_id 34644 payload 824 2024-04-22T08:16:48.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.653+0000 7fce83fff640 1 -- 172.21.15.40:0/4157163184 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce8c0a1a80 msgr2=0x7fce8c120d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:48.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.653+0000 7fce83fff640 1 --2- 172.21.15.40:0/4157163184 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce8c0a1a80 0x7fce8c120d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:48.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.653+0000 7fce83fff640 1 -- 172.21.15.40:0/4157163184 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce8c13b0e0 msgr2=0x7fce8c1217c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:16:48.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.653+0000 7fce83fff640 1 --2- 172.21.15.40:0/4157163184 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce8c13b0e0 0x7fce8c1217c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:48.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.653+0000 7fce83fff640 10 monclient: _finish_hunting 0 2024-04-22T08:16:48.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.653+0000 7fce83fff640 1 monclient: found mon.noname-a 2024-04-22T08:16:48.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.653+0000 7fce83fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:16:48.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.653+0000 7fce83fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:16:48.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.653+0000 7fce83fff640 1 -- 172.21.15.40:0/4157163184 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce84009a70 con 0x7fce8c0a2570 2024-04-22T08:16:48.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.655+0000 7fce83fff640 1 -- 172.21.15.40:0/4157163184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce8c0a2570 msgr2=0x7fce8c121280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:16:48.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.655+0000 7fce83fff640 1 --2- 172.21.15.40:0/4157163184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce8c0a2570 0x7fce8c121280 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fce74000c00 tx=0x7fce740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:48.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.656+0000 7fce82ffd640 1 -- 172.21.15.40:0/4157163184 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce7403bae0 con 0x7fce8c0a2570 2024-04-22T08:16:48.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.656+0000 7fce82ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:48.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.656+0000 7fce82ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:16:48.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.656+0000 7fce82ffd640 10 monclient: dump: 2024-04-22T08:16:48.666 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:48.666 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:48.666 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:48.666 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:48.666 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:48.666 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:48.666 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:48.666 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:48.667 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:48.667 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:48.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.656+0000 7fce82ffd640 10 monclient: _finish_auth 0 2024-04-22T08:16:48.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.656+0000 7fce82ffd640 10 monclient: _check_auth_tickets 2024-04-22T08:16:48.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.656+0000 7fce82ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:16:48.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.656+0000 7fce90de4640 5 monclient: authenticate success, global_id 34644 2024-04-22T08:16:48.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.656+0000 7fce90de4640 10 monclient: _renew_subs 2024-04-22T08:16:48.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.656+0000 7fce90de4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:16:48.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.656+0000 7fce90de4640 1 -- 172.21.15.40:0/4157163184 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce74009400 con 0x7fce8c0a2570 2024-04-22T08:16:48.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.656+0000 7fce82ffd640 1 -- 172.21.15.40:0/4157163184 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce7403bc80 con 0x7fce8c0a2570 2024-04-22T08:16:48.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.656+0000 7fce82ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:16:48.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.656+0000 7fce82ffd640 1 -- 172.21.15.40:0/4157163184 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce74032400 con 0x7fce8c0a2570 2024-04-22T08:16:48.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.656+0000 7fce82ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:48.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.656+0000 7fce82ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:16:48.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.656+0000 7fce82ffd640 10 monclient: dump: 2024-04-22T08:16:48.668 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:48.668 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:48.668 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:48.668 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:48.668 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:48.668 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:48.668 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:48.668 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:48.668 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:48.668 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:48.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.656+0000 7fce8959a640 4 set_mon_vals no callback set 2024-04-22T08:16:48.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.656+0000 7fce90de4640 10 monclient: _renew_subs 2024-04-22T08:16:48.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.656+0000 7fce90de4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:16:48.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.656+0000 7fce90de4640 1 -- 172.21.15.40:0/4157163184 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce8c124360 con 0x7fce8c0a2570 2024-04-22T08:16:48.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.657+0000 7fce90de4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:16:48.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.657+0000 7fce90de4640 1 --2- 172.21.15.40:0/4157163184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce8c121d00 0x7fce8c06db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:48.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.657+0000 7fce90de4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:48.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.657+0000 7fce90de4640 1 -- 172.21.15.40:0/4157163184 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce8c066770 con 0x7fce8c121d00 2024-04-22T08:16:48.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.657+0000 7fce8a59c640 1 --2- 172.21.15.40:0/4157163184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce8c121d00 0x7fce8c06db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:48.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.657+0000 7fce82ffd640 1 -- 172.21.15.40:0/4157163184 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fce74039040 con 0x7fce8c0a2570 2024-04-22T08:16:48.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.657+0000 7fce8a59c640 10 monclient: get_auth_request con 0x7fce8c121d00 auth_method 0 2024-04-22T08:16:48.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.657+0000 7fce8a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:48.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.657+0000 7fce8a59c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:48.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.657+0000 7fce8a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:48.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.657+0000 7fce8a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:48.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.657+0000 7fce8a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:48.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.657+0000 7fce8a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:48.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.657+0000 7fce82ffd640 1 --2- 172.21.15.40:0/4157163184 >> v2:172.21.15.189:6800/30168 conn(0x7fce6c037780 0x7fce6c039c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:48.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.657+0000 7fce8a59c640 10 monclient(hunting): handle_auth_done global_id 34647 payload 824 2024-04-22T08:16:48.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.657+0000 7fce8a59c640 1 -- 172.21.15.40:0/4157163184 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fce8c09dc10 con 0x7fce8c121d00 2024-04-22T08:16:48.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.657+0000 7fce82ffd640 1 -- 172.21.15.40:0/4157163184 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 3506+0+0 (secure 0 0 0) 0x7fce7406bba0 con 0x7fce8c0a2570 2024-04-22T08:16:48.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.657+0000 7fce8ad9d640 1 --2- 172.21.15.40:0/4157163184 >> v2:172.21.15.189:6800/30168 conn(0x7fce6c037780 0x7fce6c039c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:48.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.657+0000 7fce8ad9d640 10 monclient: get_auth_request con 0x7fce6c037780 auth_method 0 2024-04-22T08:16:48.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.659+0000 7fce8a59c640 1 -- 172.21.15.40:0/4157163184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce8c121d00 msgr2=0x7fce8c06db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:16:48.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.659+0000 7fce8a59c640 1 --2- 172.21.15.40:0/4157163184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce8c121d00 0x7fce8c06db80 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fce7c00ea70 tx=0x7fce7c00ef30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:48.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.659+0000 7fce82ffd640 1 -- 172.21.15.40:0/4157163184 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce7c00cc50 con 0x7fce8c121d00 2024-04-22T08:16:48.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.659+0000 7fce82ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:48.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.659+0000 7fce82ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:16:48.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.659+0000 7fce82ffd640 10 monclient: dump: 2024-04-22T08:16:48.673 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:48.673 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:48.673 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:48.673 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:48.673 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:48.673 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:48.673 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:48.673 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:48.673 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:48.674 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:48.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.660+0000 7fce82ffd640 1 -- 172.21.15.40:0/4157163184 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7fce7c00cdb0 con 0x7fce8c121d00 2024-04-22T08:16:48.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.660+0000 7fce82ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:16:48.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.660+0000 7fce82ffd640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:16:48.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.660+0000 7fce82ffd640 1 -- 172.21.15.40:0/4157163184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce8c121d00 msgr2=0x7fce8c06db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:48.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.660+0000 7fce82ffd640 1 --2- 172.21.15.40:0/4157163184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce8c121d00 0x7fce8c06db80 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fce7c00ea70 tx=0x7fce7c00ef30 comp rx=0 tx=0).stop 2024-04-22T08:16:48.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.660+0000 7fce82ffd640 1 -- 172.21.15.40:0/4157163184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce8c121d00 msgr2=0x7fce8c06db80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:16:48.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.660+0000 7fce82ffd640 1 --2- 172.21.15.40:0/4157163184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce8c121d00 0x7fce8c06db80 secure :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fce7c00ea70 tx=0x7fce7c00ef30 comp rx=0 tx=0).stop 2024-04-22T08:16:48.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.660+0000 7fce8ad9d640 1 --2- 172.21.15.40:0/4157163184 >> v2:172.21.15.189:6800/30168 conn(0x7fce6c037780 0x7fce6c039c40 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fce8c09d590 tx=0x7fce8400b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:48.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.666+0000 7fce90de4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:16:48.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.666+0000 7fce90de4640 1 --2- 172.21.15.40:0/4157163184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce8c06e0c0 0x7fce8c06e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:48.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.666+0000 7fce90de4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:48.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.666+0000 7fce90de4640 1 -- 172.21.15.40:0/4157163184 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce8c04e9b0 con 0x7fce8c06e0c0 2024-04-22T08:16:48.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.666+0000 7fce8a59c640 1 --2- 172.21.15.40:0/4157163184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce8c06e0c0 0x7fce8c06e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:48.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.666+0000 7fce8a59c640 10 monclient: get_auth_request con 0x7fce8c06e0c0 auth_method 0 2024-04-22T08:16:48.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.666+0000 7fce8a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:48.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.666+0000 7fce8a59c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:48.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.666+0000 7fce8a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:48.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.666+0000 7fce8a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:48.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.666+0000 7fce8a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:48.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.666+0000 7fce8a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:48.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.666+0000 7fce8a59c640 10 monclient(hunting): handle_auth_done global_id 34650 payload 824 2024-04-22T08:16:48.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.666+0000 7fce8a59c640 1 -- 172.21.15.40:0/4157163184 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fce8c122950 con 0x7fce8c06e0c0 2024-04-22T08:16:48.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.669+0000 7fce8a59c640 1 -- 172.21.15.40:0/4157163184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce8c06e0c0 msgr2=0x7fce8c06e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:16:48.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.669+0000 7fce8a59c640 1 --2- 172.21.15.40:0/4157163184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce8c06e0c0 0x7fce8c06e4d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fce7c017000 tx=0x7fce7c004940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:48.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.669+0000 7fce82ffd640 1 -- 172.21.15.40:0/4157163184 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce7c028480 con 0x7fce8c06e0c0 2024-04-22T08:16:48.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.669+0000 7fce82ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:48.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.669+0000 7fce82ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:16:48.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.669+0000 7fce82ffd640 10 monclient: dump: 2024-04-22T08:16:48.676 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:48.676 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:48.676 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:48.676 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:48.676 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:48.676 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:48.676 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:48.676 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:48.676 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:48.676 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:48.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.669+0000 7fce82ffd640 1 -- 172.21.15.40:0/4157163184 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4622 (secure 0 0 0) 0x7fce7c023070 con 0x7fce8c06e0c0 2024-04-22T08:16:48.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.669+0000 7fce82ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:16:48.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.669+0000 7fce82ffd640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:16:48.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.669+0000 7fce82ffd640 1 -- 172.21.15.40:0/4157163184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce8c06e0c0 msgr2=0x7fce8c06e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:48.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.669+0000 7fce82ffd640 1 --2- 172.21.15.40:0/4157163184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce8c06e0c0 0x7fce8c06e4d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fce7c017000 tx=0x7fce7c004940 comp rx=0 tx=0).stop 2024-04-22T08:16:48.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.669+0000 7fce82ffd640 1 -- 172.21.15.40:0/4157163184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce8c06e0c0 msgr2=0x7fce8c06e4d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:16:48.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.669+0000 7fce82ffd640 1 --2- 172.21.15.40:0/4157163184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce8c06e0c0 0x7fce8c06e4d0 secure :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fce7c017000 tx=0x7fce7c004940 comp rx=0 tx=0).stop 2024-04-22T08:16:48.677 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:16:48.677 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:16:48.677 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:16:48.677 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "leader", 2024-04-22T08:16:48.677 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 44, 2024-04-22T08:16:48.677 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:16:48.677 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:16:48.677 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:16:48.677 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:16:48.678 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 1, 2024-04-22T08:16:48.678 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:16:48.678 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:16:48.678 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:16:48.678 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:16:48.678 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:16:48.678 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:16:48.678 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:16:48.678 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:16:48.678 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:16:48.678 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:16:48.678 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:16:48.678 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:16:48.679 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:16:48.679 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:16:48.679 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:16:48.679 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:16:48.679 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:16:48.679 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:16:48.679 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:16:48.679 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:16:48.679 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:16:48.679 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:16:48.679 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:16:48.679 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:16:48.679 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:16:48.679 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:16:48.680 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:48.680 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:48.680 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:16:48.680 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:16:48.680 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:16:48.680 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:16:48.680 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:16:48.680 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:16:48.680 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:16:48.680 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:16:48.680 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:16:48.680 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:16:48.680 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:16:48.680 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:16:48.680 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:16:48.681 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:16:48.681 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:16:48.681 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:16:48.681 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:16:48.681 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:16:48.681 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:16:48.681 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:16:48.681 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:16:48.681 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:16:48.681 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:16:48.681 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:16:48.681 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:16:48.681 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:16:48.681 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:16:48.681 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:16:48.682 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:16:48.682 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:48.682 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:16:48.682 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:48.682 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:16:48.682 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:16:48.682 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:16:48.682 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:16:48.682 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:48.682 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:16:48.682 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:16:48.682 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:48.682 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:48.682 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:48.683 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:16:48.683 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:16:48.683 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:48.683 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:48.683 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:48.683 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:48.683 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:16:48.683 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:16:48.683 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:16:48.683 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:16:48.683 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:16:48.683 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:48.683 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:48.683 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:16:48.684 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:16:48.684 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:16:48.684 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:16:48.684 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:48.684 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:16:48.684 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:16:48.684 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:48.684 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:48.684 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:48.684 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:16:48.684 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:16:48.684 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:48.684 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:48.684 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:48.685 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:48.685 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:16:48.685 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:16:48.685 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:16:48.685 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:16:48.685 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:16:48.685 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:48.685 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:48.685 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:16:48.685 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:16:48.685 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:16:48.685 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:16:48.685 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:48.685 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:16:48.685 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:16:48.686 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:48.686 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:48.686 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:48.686 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:16:48.686 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:16:48.686 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:48.686 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:48.686 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:48.686 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:48.686 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:16:48.686 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:16:48.686 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:16:48.686 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:16:48.686 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:16:48.687 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:48.687 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:48.687 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:48.687 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:16:48.687 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:16:48.687 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:48.687 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:16:48.687 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:16:48.687 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:16:48.687 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:48.687 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:16:48.687 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:16:48.687 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:48.687 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:16:48.687 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:16:48.688 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 5 2024-04-22T08:16:48.688 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:48.688 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:16:48.688 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:16:48.688 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:48.688 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:16:48.688 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:16:48.688 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 6 2024-04-22T08:16:48.688 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:48.688 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:48.688 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:48.688 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:16:48.688 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:16:48.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.669+0000 7fce90de4640 1 -- 172.21.15.40:0/4157163184 >> v2:172.21.15.189:6800/30168 conn(0x7fce6c037780 msgr2=0x7fce6c039c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:48.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.669+0000 7fce90de4640 1 --2- 172.21.15.40:0/4157163184 >> v2:172.21.15.189:6800/30168 conn(0x7fce6c037780 0x7fce6c039c40 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fce8c09d590 tx=0x7fce8400b040 comp rx=0 tx=0).stop 2024-04-22T08:16:48.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.669+0000 7fce90de4640 10 monclient: shutdown 2024-04-22T08:16:48.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.669+0000 7fce90de4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:16:48.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.669+0000 7fce90de4640 1 -- 172.21.15.40:0/4157163184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce8c0a2570 msgr2=0x7fce8c121280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:48.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.669+0000 7fce90de4640 1 --2- 172.21.15.40:0/4157163184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce8c0a2570 0x7fce8c121280 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fce74000c00 tx=0x7fce740046c0 comp rx=0 tx=0).stop 2024-04-22T08:16:48.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.669+0000 7fce8ad9d640 1 -- 172.21.15.40:0/4157163184 reap_dead start 2024-04-22T08:16:48.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.669+0000 7fce90de4640 1 -- 172.21.15.40:0/4157163184 shutdown_connections 2024-04-22T08:16:48.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.669+0000 7fce90de4640 1 --2- 172.21.15.40:0/4157163184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce8c0a2570 0x7fce8c121280 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:48.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.669+0000 7fce90de4640 1 -- 172.21.15.40:0/4157163184 >> 172.21.15.40:0/4157163184 conn(0x7fce8c0a66a0 msgr2=0x7fce8c0a4dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:16:48.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.669+0000 7fce90de4640 1 -- 172.21.15.40:0/4157163184 shutdown_connections 2024-04-22T08:16:48.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.669+0000 7fce90de4640 1 -- 172.21.15.40:0/4157163184 wait complete. 2024-04-22T08:16:48.691 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-22T08:16:48.691 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-04-22T08:16:48.691 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-22T08:16:48.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.793+0000 7fa34c999640 10 monclient: get_monmap_and_config 2024-04-22T08:16:48.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.793+0000 7fa34c999640 10 monclient: build_initial_monmap 2024-04-22T08:16:48.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.793+0000 7fa34c999640 10 monclient: monmap: 2024-04-22T08:16:48.793 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:16:48.793 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:48.793 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:16:48.793936+0000 2024-04-22T08:16:48.793 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:16:48.793936+0000 2024-04-22T08:16:48.793 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:16:48.793 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:48.793 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:16:48.793 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:16:48.793 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:16:48.793 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:16:48.793 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:16:48.793 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:48.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.793+0000 7fa34c999640 1 Processor -- start 2024-04-22T08:16:48.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.793+0000 7fa34c999640 1 -- start start 2024-04-22T08:16:48.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.793+0000 7fa34c999640 10 monclient: init 2024-04-22T08:16:48.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.794+0000 7fa34c999640 10 monclient: _reopen_session rank -1 2024-04-22T08:16:48.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.794+0000 7fa34c999640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-04-22T08:16:48.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.794+0000 7fa34c999640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa34812fad0 0x7fa34812feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:48.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.794+0000 7fa34c999640 10 monclient(hunting): picked mon.noname-c con 0x7fa34812fad0 addr v2:172.21.15.189:3300/0 2024-04-22T08:16:48.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.794+0000 7fa34c999640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa348132920 addr v1:172.21.15.40:6789/0 2024-04-22T08:16:48.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.794+0000 7fa34c999640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa348133460 0x7fa34813ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:48.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.794+0000 7fa34c999640 10 monclient(hunting): picked mon.noname-a con 0x7fa348133460 addr v2:172.21.15.40:3300/0 2024-04-22T08:16:48.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.794+0000 7fa34c999640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa34809ae50 con 0x7fa348132920 2024-04-22T08:16:48.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.794+0000 7fa34c999640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:48.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.794+0000 7fa34c999640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa34809efc0 con 0x7fa348133460 2024-04-22T08:16:48.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.794+0000 7fa34c999640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:48.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.794+0000 7fa34c999640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa34809d070 con 0x7fa34812fad0 2024-04-22T08:16:48.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.794+0000 7fa34c999640 10 monclient(hunting): _renew_subs 2024-04-22T08:16:48.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.794+0000 7fa34c999640 10 monclient(hunting): authenticate will time out at 1082.788188s 2024-04-22T08:16:48.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.794+0000 7fa346d9d640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa348133460 0x7fa34813ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:48.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.794+0000 7fa345d9b640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa348132920 0x7fa348132d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:55212/0 (socket says 172.21.15.40:55212) 2024-04-22T08:16:48.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.794+0000 7fa346d9d640 10 monclient(hunting): get_auth_request con 0x7fa348133460 auth_method 0 2024-04-22T08:16:48.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.794+0000 7fa345d9b640 1 -- 172.21.15.40:0/3684399241 learned_addr learned my addr 172.21.15.40:0/3684399241 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:16:48.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.794+0000 7fa346d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:48.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.794+0000 7fa346d9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:48.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.794+0000 7fa346d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:48.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.794+0000 7fa34659c640 1 -- >> v2:172.21.15.189:3300/0 conn(0x7fa34812fad0 msgr2=0x7fa34812feb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.189:3300/0 2024-04-22T08:16:48.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.794+0000 7fa34659c640 1 --2- 172.21.15.40:0/3684399241 >> v2:172.21.15.189:3300/0 conn(0x7fa34812fad0 0x7fa34812feb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:16:48.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.794+0000 7fa346d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:48.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.794+0000 7fa346d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:48.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.794+0000 7fa346d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:48.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.794+0000 7fa346d9d640 10 monclient(hunting): handle_auth_done global_id 34653 payload 274 2024-04-22T08:16:48.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.794+0000 7fa346d9d640 1 -- 172.21.15.40:0/3684399241 >> v2:172.21.15.189:3300/0 conn(0x7fa34812fad0 msgr2=0x7fa34812feb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:16:48.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.794+0000 7fa346d9d640 1 --2- 172.21.15.40:0/3684399241 >> v2:172.21.15.189:3300/0 conn(0x7fa34812fad0 0x7fa34812feb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:48.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.794+0000 7fa346d9d640 1 -- 172.21.15.40:0/3684399241 >> v1:172.21.15.40:6789/0 conn(0x7fa348132920 legacy=0x7fa348132d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:16:48.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.794+0000 7fa346d9d640 10 monclient: _finish_hunting 0 2024-04-22T08:16:48.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.794+0000 7fa346d9d640 1 monclient: found mon.noname-a 2024-04-22T08:16:48.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.794+0000 7fa346d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:16:48.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.794+0000 7fa346d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:16:48.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.794+0000 7fa346d9d640 1 -- 172.21.15.40:0/3684399241 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa34809d350 con 0x7fa348133460 2024-04-22T08:16:48.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.797+0000 7fa346d9d640 1 -- 172.21.15.40:0/3684399241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa348133460 msgr2=0x7fa34813ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:16:48.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.797+0000 7fa346d9d640 1 --2- 172.21.15.40:0/3684399241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa348133460 0x7fa34813ac60 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fa34806bb30 tx=0x7fa34002b740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7612947323578399 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:48.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.797+0000 7fa34559a640 1 -- 172.21.15.40:0/3684399241 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa340030a00 con 0x7fa348133460 2024-04-22T08:16:48.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.797+0000 7fa34559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:48.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.797+0000 7fa34559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:16:48.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.797+0000 7fa34559a640 10 monclient: dump: 2024-04-22T08:16:48.805 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:48.805 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:48.805 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:48.805 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:48.805 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:48.805 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:48.805 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:48.805 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:48.805 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:48.805 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:48.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.797+0000 7fa34559a640 10 monclient: _finish_auth 0 2024-04-22T08:16:48.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.797+0000 7fa34559a640 10 monclient: _check_auth_tickets 2024-04-22T08:16:48.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.797+0000 7fa34559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:16:48.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.797+0000 7fa34559a640 1 -- 172.21.15.40:0/3684399241 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa34002e050 con 0x7fa348133460 2024-04-22T08:16:48.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.797+0000 7fa34559a640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:16:48.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.797+0000 7fa34559a640 1 -- 172.21.15.40:0/3684399241 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa340030d80 con 0x7fa348133460 2024-04-22T08:16:48.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.797+0000 7fa34c999640 5 monclient: authenticate success, global_id 34653 2024-04-22T08:16:48.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.797+0000 7fa34c999640 10 monclient: get_monmap_and_config success 2024-04-22T08:16:48.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.797+0000 7fa34c999640 4 set_mon_vals no callback set 2024-04-22T08:16:48.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.797+0000 7fa34559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:48.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.797+0000 7fa34559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:16:48.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.797+0000 7fa34559a640 10 monclient: dump: 2024-04-22T08:16:48.806 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:48.806 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:48.806 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:48.806 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:48.806 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:48.806 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:48.807 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:48.807 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:48.807 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:48.807 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:48.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.798+0000 7fa34c999640 10 monclient: shutdown 2024-04-22T08:16:48.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.798+0000 7fa34c999640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:16:48.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.798+0000 7fa34c999640 1 -- 172.21.15.40:0/3684399241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa348133460 msgr2=0x7fa34813ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:48.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.798+0000 7fa34c999640 1 --2- 172.21.15.40:0/3684399241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa348133460 0x7fa34813ac60 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fa34806bb30 tx=0x7fa34002b740 comp rx=0 tx=0).stop 2024-04-22T08:16:48.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.798+0000 7fa34c999640 1 -- 172.21.15.40:0/3684399241 shutdown_connections 2024-04-22T08:16:48.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.798+0000 7fa34c999640 1 --2- 172.21.15.40:0/3684399241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa348133460 0x7fa34813ac60 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:48.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.798+0000 7fa34c999640 1 --2- 172.21.15.40:0/3684399241 >> v2:172.21.15.189:3300/0 conn(0x7fa34812fad0 0x7fa34812feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:48.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.798+0000 7fa34c999640 1 -- 172.21.15.40:0/3684399241 >> 172.21.15.40:0/3684399241 conn(0x7fa3480a0ae0 msgr2=0x7fa3480a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:16:48.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.798+0000 7fa34c999640 1 -- 172.21.15.40:0/3684399241 shutdown_connections 2024-04-22T08:16:48.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.798+0000 7fa34c999640 1 -- 172.21.15.40:0/3684399241 wait complete. 2024-04-22T08:16:48.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.799+0000 7fa34c999640 10 monclient: build_initial_monmap 2024-04-22T08:16:48.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.799+0000 7fa34c999640 10 monclient: monmap: 2024-04-22T08:16:48.808 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:16:48.808 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:16:48.808 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:16:48.808 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:16:48.808 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:16:48.808 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:48.808 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:16:48.808 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:16:48.808 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:16:48.808 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:48.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.799+0000 7fa34c999640 1 Processor -- start 2024-04-22T08:16:48.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.799+0000 7fa34c999640 1 -- start start 2024-04-22T08:16:48.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.799+0000 7fa34c999640 10 monclient: init 2024-04-22T08:16:48.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa34c999640 10 monclient: _reopen_session rank -1 2024-04-22T08:16:48.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa34c999640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-22T08:16:48.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa34c999640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa34812fad0 0x7fa34813ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:48.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa34c999640 10 monclient(hunting): picked mon.noname-c con 0x7fa34812fad0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:16:48.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa34c999640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa348132920 0x7fa34812e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:48.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa34c999640 10 monclient(hunting): picked mon.noname-a con 0x7fa348132920 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:16:48.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa34c999640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa348133460 0x7fa34812e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:48.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa34c999640 10 monclient(hunting): picked mon.noname-b con 0x7fa348133460 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:16:48.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa34c999640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:48.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa34c999640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa34806eef0 con 0x7fa348132920 2024-04-22T08:16:48.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa34c999640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:48.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa34c999640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3480a22c0 con 0x7fa348133460 2024-04-22T08:16:48.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa34c999640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:48.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa34c999640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa348071870 con 0x7fa34812fad0 2024-04-22T08:16:48.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa34c999640 10 monclient(hunting): _renew_subs 2024-04-22T08:16:48.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa34c999640 10 monclient(hunting): authenticate will time out at 1082.793925s 2024-04-22T08:16:48.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa34659c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa34812fad0 0x7fa34813ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:48.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa345d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa348132920 0x7fa34812e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:48.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa34659c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa34812fad0 0x7fa34813ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:44286/0 (socket says 172.21.15.40:44286) 2024-04-22T08:16:48.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa34659c640 1 -- 172.21.15.40:0/2028248198 learned_addr learned my addr 172.21.15.40:0/2028248198 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:16:48.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa34659c640 10 monclient(hunting): get_auth_request con 0x7fa34812fad0 auth_method 0 2024-04-22T08:16:48.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa34659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:48.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa34659c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:48.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa345d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa348132920 0x7fa34812e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45988/0 (socket says 172.21.15.40:45988) 2024-04-22T08:16:48.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa34659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:48.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa346d9d640 1 -- 172.21.15.40:0/2028248198 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa348133460 msgr2=0x7fa34812e580 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.189:3300/0 2024-04-22T08:16:48.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa345d9b640 10 monclient(hunting): get_auth_request con 0x7fa348132920 auth_method 0 2024-04-22T08:16:48.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa345d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:48.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa345d9b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:48.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa345d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:48.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa346d9d640 1 --2- 172.21.15.40:0/2028248198 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa348133460 0x7fa34812e580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:16:48.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa345d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:48.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa345d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:48.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa345d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:48.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa34659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:48.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa34659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:48.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa34659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:48.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa345d9b640 10 monclient(hunting): handle_auth_done global_id 34656 payload 824 2024-04-22T08:16:48.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa345d9b640 1 -- 172.21.15.40:0/2028248198 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa34812fad0 msgr2=0x7fa34813ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:48.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa345d9b640 1 --2- 172.21.15.40:0/2028248198 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa34812fad0 0x7fa34813ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:48.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa345d9b640 1 -- 172.21.15.40:0/2028248198 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa348133460 msgr2=0x7fa34812e580 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:16:48.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa345d9b640 1 --2- 172.21.15.40:0/2028248198 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa348133460 0x7fa34812e580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:16:48.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa345d9b640 10 monclient: _finish_hunting 0 2024-04-22T08:16:48.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa345d9b640 1 monclient: found mon.noname-a 2024-04-22T08:16:48.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa345d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:16:48.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa345d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:16:48.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa345d9b640 1 -- 172.21.15.40:0/2028248198 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa340009400 con 0x7fa348132920 2024-04-22T08:16:48.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.800+0000 7fa34659c640 1 --2- 172.21.15.40:0/2028248198 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa34812fad0 0x7fa34813ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:16:48.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.802+0000 7fa345d9b640 1 -- 172.21.15.40:0/2028248198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa348132920 msgr2=0x7fa34812e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:16:48.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.802+0000 7fa345d9b640 1 --2- 172.21.15.40:0/2028248198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa348132920 0x7fa34812e040 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa33c00c3e0 tx=0x7fa33c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:48.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.802+0000 7fa336ffd640 1 -- 172.21.15.40:0/2028248198 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa33c0070d0 con 0x7fa348132920 2024-04-22T08:16:48.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.802+0000 7fa336ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:48.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.802+0000 7fa336ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:16:48.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.802+0000 7fa336ffd640 10 monclient: dump: 2024-04-22T08:16:48.812 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:48.812 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:48.813 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:48.813 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:48.813 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:48.813 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:48.813 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:48.813 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:48.813 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:48.813 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:48.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.803+0000 7fa336ffd640 10 monclient: _finish_auth 0 2024-04-22T08:16:48.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.803+0000 7fa336ffd640 10 monclient: _check_auth_tickets 2024-04-22T08:16:48.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.803+0000 7fa336ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:16:48.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.803+0000 7fa336ffd640 1 -- 172.21.15.40:0/2028248198 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa33c007230 con 0x7fa348132920 2024-04-22T08:16:48.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.803+0000 7fa336ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:16:48.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.803+0000 7fa34c999640 5 monclient: authenticate success, global_id 34656 2024-04-22T08:16:48.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.803+0000 7fa336ffd640 1 -- 172.21.15.40:0/2028248198 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa33c0075e0 con 0x7fa348132920 2024-04-22T08:16:48.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.803+0000 7fa34c999640 10 monclient: _renew_subs 2024-04-22T08:16:48.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.803+0000 7fa34c999640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:16:48.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.803+0000 7fa34c999640 1 -- 172.21.15.40:0/2028248198 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa34813d880 con 0x7fa348132920 2024-04-22T08:16:48.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.803+0000 7fa344d99640 4 set_mon_vals no callback set 2024-04-22T08:16:48.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.803+0000 7fa336ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:48.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.803+0000 7fa336ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:16:48.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.803+0000 7fa336ffd640 10 monclient: dump: 2024-04-22T08:16:48.814 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:48.814 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:48.814 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:48.814 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:48.814 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:48.814 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:48.815 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:48.815 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:48.815 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:48.815 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:48.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.803+0000 7fa34c999640 10 monclient: _renew_subs 2024-04-22T08:16:48.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.803+0000 7fa34c999640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:16:48.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.803+0000 7fa34c999640 1 -- 172.21.15.40:0/2028248198 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa34813c280 con 0x7fa348132920 2024-04-22T08:16:48.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.803+0000 7fa34c999640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:16:48.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.803+0000 7fa34c999640 1 --2- 172.21.15.40:0/2028248198 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa34812eac0 0x7fa348130250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:48.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.803+0000 7fa34c999640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:48.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.803+0000 7fa34c999640 1 -- 172.21.15.40:0/2028248198 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3480667a0 con 0x7fa34812eac0 2024-04-22T08:16:48.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.804+0000 7fa34659c640 1 --2- 172.21.15.40:0/2028248198 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa34812eac0 0x7fa348130250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:48.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.804+0000 7fa34659c640 10 monclient: get_auth_request con 0x7fa34812eac0 auth_method 0 2024-04-22T08:16:48.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.804+0000 7fa34659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:48.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.804+0000 7fa336ffd640 1 -- 172.21.15.40:0/2028248198 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fa33c01b3b0 con 0x7fa348132920 2024-04-22T08:16:48.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.804+0000 7fa34659c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:48.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.804+0000 7fa34659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:48.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.804+0000 7fa34659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:48.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.804+0000 7fa34659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:48.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.804+0000 7fa34659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:48.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.804+0000 7fa336ffd640 1 --2- 172.21.15.40:0/2028248198 >> v2:172.21.15.189:6800/30168 conn(0x7fa328037780 0x7fa328039c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:48.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.804+0000 7fa34659c640 10 monclient(hunting): handle_auth_done global_id 54209 payload 824 2024-04-22T08:16:48.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.804+0000 7fa34659c640 1 -- 172.21.15.40:0/2028248198 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa34809e2f0 con 0x7fa34812eac0 2024-04-22T08:16:48.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.804+0000 7fa336ffd640 1 -- 172.21.15.40:0/2028248198 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 3506+0+0 (secure 0 0 0) 0x7fa33c0498a0 con 0x7fa348132920 2024-04-22T08:16:48.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.804+0000 7fa346d9d640 1 --2- 172.21.15.40:0/2028248198 >> v2:172.21.15.189:6800/30168 conn(0x7fa328037780 0x7fa328039c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:48.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.804+0000 7fa346d9d640 10 monclient: get_auth_request con 0x7fa328037780 auth_method 0 2024-04-22T08:16:48.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.806+0000 7fa34659c640 1 -- 172.21.15.40:0/2028248198 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa34812eac0 msgr2=0x7fa348130250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:16:48.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.806+0000 7fa34659c640 1 --2- 172.21.15.40:0/2028248198 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa34812eac0 0x7fa348130250 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa33800e4e0 tx=0x7fa33800e9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:48.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.806+0000 7fa336ffd640 1 -- 172.21.15.40:0/2028248198 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3380057d0 con 0x7fa34812eac0 2024-04-22T08:16:48.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.806+0000 7fa336ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:48.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.806+0000 7fa336ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:16:48.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.806+0000 7fa336ffd640 10 monclient: dump: 2024-04-22T08:16:48.819 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:48.819 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:48.820 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:48.820 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:48.820 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:48.820 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:48.820 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:48.820 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:48.820 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:48.820 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:48.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.806+0000 7fa336ffd640 1 -- 172.21.15.40:0/2028248198 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7fa33800f040 con 0x7fa34812eac0 2024-04-22T08:16:48.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.806+0000 7fa336ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:16:48.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.806+0000 7fa336ffd640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:16:48.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.806+0000 7fa336ffd640 1 -- 172.21.15.40:0/2028248198 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa34812eac0 msgr2=0x7fa348130250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:48.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.806+0000 7fa336ffd640 1 --2- 172.21.15.40:0/2028248198 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa34812eac0 0x7fa348130250 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa33800e4e0 tx=0x7fa33800e9a0 comp rx=0 tx=0).stop 2024-04-22T08:16:48.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.806+0000 7fa336ffd640 1 -- 172.21.15.40:0/2028248198 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa34812eac0 msgr2=0x7fa348130250 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:16:48.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.806+0000 7fa336ffd640 1 --2- 172.21.15.40:0/2028248198 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa34812eac0 0x7fa348130250 secure :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa33800e4e0 tx=0x7fa33800e9a0 comp rx=0 tx=0).stop 2024-04-22T08:16:48.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.807+0000 7fa346d9d640 1 --2- 172.21.15.40:0/2028248198 >> v2:172.21.15.189:6800/30168 conn(0x7fa328037780 0x7fa328039c40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa3480725d0 tx=0x7fa340008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:48.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.813+0000 7fa34c999640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:16:48.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.813+0000 7fa34c999640 1 --2- 172.21.15.40:0/2028248198 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa348130790 0x7fa348130ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:16:48.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.813+0000 7fa34c999640 10 monclient(hunting): start opening mon connection 2024-04-22T08:16:48.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.813+0000 7fa34c999640 1 -- 172.21.15.40:0/2028248198 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa34813dff0 con 0x7fa348130790 2024-04-22T08:16:48.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.813+0000 7fa34659c640 1 --2- 172.21.15.40:0/2028248198 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa348130790 0x7fa348130ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:16:48.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.813+0000 7fa34659c640 10 monclient: get_auth_request con 0x7fa348130790 auth_method 0 2024-04-22T08:16:48.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.813+0000 7fa34659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:16:48.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.813+0000 7fa34659c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:16:48.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.813+0000 7fa34659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:16:48.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.813+0000 7fa34659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:16:48.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.813+0000 7fa34659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:16:48.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.813+0000 7fa34659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:16:48.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.813+0000 7fa34659c640 10 monclient(hunting): handle_auth_done global_id 54212 payload 824 2024-04-22T08:16:48.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.813+0000 7fa34659c640 1 -- 172.21.15.40:0/2028248198 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa34812eea0 con 0x7fa348130790 2024-04-22T08:16:48.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.815+0000 7fa34659c640 1 -- 172.21.15.40:0/2028248198 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa348130790 msgr2=0x7fa348130ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:16:48.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.815+0000 7fa34659c640 1 --2- 172.21.15.40:0/2028248198 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa348130790 0x7fa348130ba0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa33800e4e0 tx=0x7fa338005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:16:48.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.816+0000 7fa336ffd640 1 -- 172.21.15.40:0/2028248198 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa338028600 con 0x7fa348130790 2024-04-22T08:16:48.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.816+0000 7fa336ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:16:48.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.816+0000 7fa336ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:16:48.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.816+0000 7fa336ffd640 10 monclient: dump: 2024-04-22T08:16:48.822 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:16:48.822 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:16:48.822 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:48.822 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:16:48.822 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:16:48.822 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:16:48.822 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:16:48.823 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:16:48.823 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:16:48.823 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:16:48.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.816+0000 7fa336ffd640 1 -- 172.21.15.40:0/2028248198 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4790 (secure 0 0 0) 0x7fa338028760 con 0x7fa348130790 2024-04-22T08:16:48.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.816+0000 7fa336ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:16:48.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.816+0000 7fa336ffd640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:16:48.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.816+0000 7fa336ffd640 1 -- 172.21.15.40:0/2028248198 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa348130790 msgr2=0x7fa348130ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:48.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.816+0000 7fa336ffd640 1 --2- 172.21.15.40:0/2028248198 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa348130790 0x7fa348130ba0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa33800e4e0 tx=0x7fa338005d50 comp rx=0 tx=0).stop 2024-04-22T08:16:48.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.816+0000 7fa336ffd640 1 -- 172.21.15.40:0/2028248198 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa348130790 msgr2=0x7fa348130ba0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:16:48.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.816+0000 7fa336ffd640 1 --2- 172.21.15.40:0/2028248198 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa348130790 0x7fa348130ba0 secure :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa33800e4e0 tx=0x7fa338005d50 comp rx=0 tx=0).stop 2024-04-22T08:16:48.823 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:16:48.824 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:16:48.824 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:16:48.824 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:16:48.824 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 44, 2024-04-22T08:16:48.824 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:16:48.824 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:16:48.824 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:16:48.824 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:16:48.824 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 1, 2024-04-22T08:16:48.824 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:16:48.824 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:16:48.824 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:16:48.824 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:16:48.825 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:16:48.825 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:16:48.825 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:16:48.825 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:16:48.825 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:16:48.825 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:16:48.825 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:16:48.825 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:16:48.825 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:16:48.825 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:16:48.825 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:16:48.825 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:16:48.825 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:16:48.825 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:16:48.825 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:16:48.826 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:16:48.826 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:16:48.826 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:16:48.826 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:16:48.826 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:16:48.826 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:16:48.826 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:16:48.826 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:48.826 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:48.826 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:16:48.826 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:16:48.826 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:16:48.826 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:16:48.826 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:16:48.827 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:16:48.827 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:16:48.827 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:16:48.827 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:16:48.827 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:16:48.827 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:16:48.827 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:16:48.827 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:16:48.827 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:16:48.827 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:16:48.827 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:16:48.827 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:16:48.827 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:16:48.827 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:16:48.828 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:16:48.828 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:16:48.828 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:16:48.828 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:16:48.828 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:16:48.828 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:16:48.828 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:16:48.828 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:16:48.828 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:16:48.828 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:16:48.828 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:48.828 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:16:48.828 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:48.828 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:16:48.828 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:16:48.829 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:16:48.829 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:16:48.829 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:48.829 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:16:48.829 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:16:48.829 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:48.829 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:48.829 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:48.829 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:16:48.829 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:16:48.829 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:48.829 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:48.829 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:48.829 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:48.830 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:16:48.830 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:16:48.830 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:16:48.830 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:16:48.830 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:16:48.830 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:48.830 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:48.830 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:16:48.830 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:16:48.830 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:16:48.830 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:16:48.830 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:48.830 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:16:48.830 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:16:48.830 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:48.831 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:48.831 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:48.831 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:16:48.831 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:16:48.831 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:48.831 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:48.831 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:48.831 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:48.831 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:16:48.831 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:16:48.831 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:16:48.831 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:16:48.831 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:16:48.831 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:48.831 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:48.832 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:16:48.832 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:16:48.832 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:16:48.832 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:16:48.832 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:48.832 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:16:48.832 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:16:48.832 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:48.832 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:48.832 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:48.832 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:16:48.832 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:16:48.832 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:16:48.832 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:48.833 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:48.833 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:48.833 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:16:48.833 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:16:48.833 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:16:48.833 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:16:48.833 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:16:48.833 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:48.833 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:48.833 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:48.833 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:16:48.833 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:16:48.833 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:48.833 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:16:48.833 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:16:48.834 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:16:48.834 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:48.834 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:16:48.834 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:16:48.834 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:48.834 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:16:48.834 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:16:48.834 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:16:48.834 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:48.834 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:16:48.834 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:16:48.834 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:48.834 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:16:48.834 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:16:48.835 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 4 2024-04-22T08:16:48.835 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:48.835 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:16:48.835 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": [ 2024-04-22T08:16:48.835 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:16:48.835 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:16:48.835 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:16:48.835 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:16:48.835 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:16:48.835 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:16:48.835 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:16:48.835 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:16:48.835 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:16:48.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.816+0000 7fa34c999640 1 -- 172.21.15.40:0/2028248198 >> v2:172.21.15.189:6800/30168 conn(0x7fa328037780 msgr2=0x7fa328039c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:48.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.816+0000 7fa34c999640 1 --2- 172.21.15.40:0/2028248198 >> v2:172.21.15.189:6800/30168 conn(0x7fa328037780 0x7fa328039c40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa3480725d0 tx=0x7fa340008030 comp rx=0 tx=0).stop 2024-04-22T08:16:48.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.816+0000 7fa34c999640 10 monclient: shutdown 2024-04-22T08:16:48.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.816+0000 7fa34c999640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:16:48.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.816+0000 7fa34c999640 1 -- 172.21.15.40:0/2028248198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa348132920 msgr2=0x7fa34812e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:16:48.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.816+0000 7fa34c999640 1 --2- 172.21.15.40:0/2028248198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa348132920 0x7fa34812e040 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa33c00c3e0 tx=0x7fa33c00c8a0 comp rx=0 tx=0).stop 2024-04-22T08:16:48.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.816+0000 7fa346d9d640 1 -- 172.21.15.40:0/2028248198 reap_dead start 2024-04-22T08:16:48.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.816+0000 7fa346d9d640 1 -- 172.21.15.40:0/2028248198 reap_dead start 2024-04-22T08:16:48.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.816+0000 7fa34c999640 1 -- 172.21.15.40:0/2028248198 shutdown_connections 2024-04-22T08:16:48.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.816+0000 7fa34c999640 1 -- 172.21.15.40:0/2028248198 >> 172.21.15.40:0/2028248198 conn(0x7fa3480a0ae0 msgr2=0x7fa34809b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:16:48.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.816+0000 7fa34c999640 1 -- 172.21.15.40:0/2028248198 shutdown_connections 2024-04-22T08:16:48.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:16:48.816+0000 7fa34c999640 1 -- 172.21.15.40:0/2028248198 wait complete. 2024-04-22T08:16:48.838 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-22T08:16:48.838 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-04-22T08:16:50.500 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:16:50.500 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36551]: starting. 2024-04-22T08:16:50.500 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36551]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-36516 2024-04-22T08:16:50.500 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36551]: created object 0... 2024-04-22T08:16:50.500 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36551]: created object 25... 2024-04-22T08:16:50.501 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36551]: created object 49... 2024-04-22T08:16:50.501 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36551]: finishing. 2024-04-22T08:16:50.501 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36551]: shutting down. 2024-04-22T08:16:50.543 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:16:50.543 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36553]: starting. 2024-04-22T08:16:50.543 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36553]: listing objects. 2024-04-22T08:16:50.543 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36553]: listed object 0... 2024-04-22T08:16:50.543 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36553]: listed object 25... 2024-04-22T08:16:50.543 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36553]: saw 50 objects 2024-04-22T08:16:50.543 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36553]: shutting down. 2024-04-22T08:16:51.184 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:16:51.184 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[36552]: starting. 2024-04-22T08:16:51.184 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[36552]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-36516 2024-04-22T08:16:51.184 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[36552]: shutting down. 2024-04-22T08:16:51.186 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:16:51.186 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:16:51.186 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:16:51.188 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:16:53.294 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36620]: starting. 2024-04-22T08:16:53.294 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36620]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-36619 2024-04-22T08:16:53.294 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36620]: created object 0... 2024-04-22T08:16:53.294 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36620]: created object 25... 2024-04-22T08:16:53.294 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36620]: created object 49... 2024-04-22T08:16:53.294 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36620]: finishing. 2024-04-22T08:16:53.295 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36620]: shutting down. 2024-04-22T08:16:54.170 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[36621]: starting. 2024-04-22T08:16:54.171 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[36621]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-36619 2024-04-22T08:16:54.171 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[36621]: shutting down. 2024-04-22T08:16:56.302 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:16:56.302 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36654]: starting. 2024-04-22T08:16:56.303 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36654]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-36619 2024-04-22T08:16:56.303 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36654]: created object 0... 2024-04-22T08:16:56.303 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36654]: created object 25... 2024-04-22T08:16:56.303 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36654]: created object 49... 2024-04-22T08:16:56.303 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36654]: finishing. 2024-04-22T08:16:56.303 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36654]: shutting down. 2024-04-22T08:16:56.343 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:16:56.343 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36656]: starting. 2024-04-22T08:16:56.343 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36656]: listing objects. 2024-04-22T08:16:56.344 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36656]: listed object 0... 2024-04-22T08:16:56.344 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36656]: listed object 25... 2024-04-22T08:16:56.344 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36656]: saw 50 objects 2024-04-22T08:16:56.344 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36656]: shutting down. 2024-04-22T08:16:57.185 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:16:57.185 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[36655]: starting. 2024-04-22T08:16:57.185 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[36655]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-36619 2024-04-22T08:16:57.186 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[36655]: shutting down. 2024-04-22T08:16:57.187 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:16:57.187 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:16:57.187 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:16:57.190 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:16:59.325 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36723]: starting. 2024-04-22T08:16:59.325 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36723]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-36722 2024-04-22T08:16:59.325 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36723]: created object 0... 2024-04-22T08:16:59.325 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36723]: created object 25... 2024-04-22T08:16:59.325 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36723]: created object 49... 2024-04-22T08:16:59.325 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36723]: finishing. 2024-04-22T08:16:59.325 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36723]: shutting down. 2024-04-22T08:17:00.215 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[36724]: starting. 2024-04-22T08:17:00.215 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[36724]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-36722 2024-04-22T08:17:00.216 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[36724]: shutting down. 2024-04-22T08:17:01.991 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:17:01.992 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36757]: starting. 2024-04-22T08:17:01.992 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36757]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-36722 2024-04-22T08:17:01.992 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36757]: created object 0... 2024-04-22T08:17:01.992 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36757]: created object 25... 2024-04-22T08:17:01.992 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36757]: created object 49... 2024-04-22T08:17:01.992 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36757]: finishing. 2024-04-22T08:17:01.992 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36757]: shutting down. 2024-04-22T08:17:02.033 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:17:02.033 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36759]: starting. 2024-04-22T08:17:02.033 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36759]: listing objects. 2024-04-22T08:17:02.033 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36759]: listed object 0... 2024-04-22T08:17:02.033 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36759]: listed object 25... 2024-04-22T08:17:02.033 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36759]: saw 50 objects 2024-04-22T08:17:02.033 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36759]: shutting down. 2024-04-22T08:17:02.871 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:17:02.871 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[36758]: starting. 2024-04-22T08:17:02.871 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[36758]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-36722 2024-04-22T08:17:02.871 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[36758]: shutting down. 2024-04-22T08:17:02.872 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:17:02.872 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:17:02.872 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:17:02.875 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:17:04.985 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36826]: starting. 2024-04-22T08:17:04.985 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36826]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-36825 2024-04-22T08:17:04.985 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36826]: created object 0... 2024-04-22T08:17:04.985 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36826]: created object 25... 2024-04-22T08:17:04.985 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36826]: created object 49... 2024-04-22T08:17:04.985 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36826]: finishing. 2024-04-22T08:17:04.986 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36826]: shutting down. 2024-04-22T08:17:05.884 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[36827]: starting. 2024-04-22T08:17:05.884 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[36827]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-36825 2024-04-22T08:17:05.884 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[36827]: shutting down. 2024-04-22T08:17:08.030 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:17:08.031 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36860]: starting. 2024-04-22T08:17:08.031 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36860]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-36825 2024-04-22T08:17:08.031 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36860]: created object 0... 2024-04-22T08:17:08.031 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36860]: created object 25... 2024-04-22T08:17:08.031 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36860]: created object 49... 2024-04-22T08:17:08.031 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36860]: finishing. 2024-04-22T08:17:08.031 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[36860]: shutting down. 2024-04-22T08:17:08.076 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:17:08.076 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36862]: starting. 2024-04-22T08:17:08.076 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36862]: listing objects. 2024-04-22T08:17:08.076 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36862]: listed object 0... 2024-04-22T08:17:08.076 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36862]: listed object 25... 2024-04-22T08:17:08.077 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36862]: saw 50 objects 2024-04-22T08:17:08.077 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[36862]: shutting down. 2024-04-22T08:17:08.839 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-04-22T08:17:08.839 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-04-22T08:17:08.839 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-22T08:17:08.840 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-22T08:17:08.842 INFO:tasks.ceph.mon.b:Started 2024-04-22T08:17:08.842 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-04-22T08:17:08.843 INFO:tasks.ceph.mon.c:Sent signal 19 2024-04-22T08:17:08.843 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-04-22T08:17:23.844 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-04-22T08:17:23.844 INFO:tasks.ceph.mon.c:Sent signal 18 2024-04-22T08:17:23.845 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-22T08:17:23.845 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:17:23.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.954+0000 7ff8aa61c640 10 monclient: get_monmap_and_config 2024-04-22T08:17:23.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.954+0000 7ff8aa61c640 10 monclient: build_initial_monmap 2024-04-22T08:17:23.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.954+0000 7ff8aa61c640 10 monclient: monmap: 2024-04-22T08:17:23.955 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:17:23.955 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:23.955 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:17:23.956430+0000 2024-04-22T08:17:23.955 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:17:23.956430+0000 2024-04-22T08:17:23.955 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:17:23.955 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:23.955 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:17:23.955 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:17:23.956 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:17:23.956 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:17:23.956 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:17:23.956 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:23.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.955+0000 7ff8aa61c640 1 Processor -- start 2024-04-22T08:17:23.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.955+0000 7ff8aa61c640 1 -- start start 2024-04-22T08:17:23.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.955+0000 7ff8aa61c640 10 monclient: init 2024-04-22T08:17:23.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.955+0000 7ff8aa61c640 10 monclient: _reopen_session rank -1 2024-04-22T08:17:23.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.955+0000 7ff8aa61c640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-04-22T08:17:23.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.955+0000 7ff8aa61c640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff8a412fc40 0x7ff8a4130020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:23.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.955+0000 7ff8aa61c640 10 monclient(hunting): picked mon.noname-c con 0x7ff8a412fc40 addr v2:172.21.15.189:3300/0 2024-04-22T08:17:23.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.955+0000 7ff8aa61c640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff8a409e210 0x7ff8a4132990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:23.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.955+0000 7ff8aa61c640 10 monclient(hunting): picked mon.noname-a con 0x7ff8a409e210 addr v2:172.21.15.40:3300/0 2024-04-22T08:17:23.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.955+0000 7ff8aa61c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff8a4132ed0 addr v1:172.21.15.189:6789/0 2024-04-22T08:17:23.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.955+0000 7ff8aa61c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8a409ea50 con 0x7ff8a4132ed0 2024-04-22T08:17:23.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.955+0000 7ff8aa61c640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:23.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.955+0000 7ff8aa61c640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8a409cf90 con 0x7ff8a409e210 2024-04-22T08:17:23.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.955+0000 7ff8aa61c640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:23.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.955+0000 7ff8aa61c640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8a409d270 con 0x7ff8a412fc40 2024-04-22T08:17:23.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.955+0000 7ff8aa61c640 10 monclient(hunting): _renew_subs 2024-04-22T08:17:23.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.955+0000 7ff8aa61c640 10 monclient(hunting): authenticate will time out at 1117.950743s 2024-04-22T08:17:23.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.956+0000 7ff8a37fe640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff8a409e210 0x7ff8a4132990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:23.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.956+0000 7ff8a37fe640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff8a409e210 0x7ff8a4132990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60160/0 (socket says 172.21.15.40:60160) 2024-04-22T08:17:23.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.956+0000 7ff8a37fe640 1 -- 172.21.15.40:0/1960430228 learned_addr learned my addr 172.21.15.40:0/1960430228 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:17:23.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.956+0000 7ff8a37fe640 10 monclient(hunting): get_auth_request con 0x7ff8a409e210 auth_method 0 2024-04-22T08:17:23.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.956+0000 7ff8a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:23.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.956+0000 7ff8a37fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:23.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.956+0000 7ff8a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:23.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.956+0000 7ff8a37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:23.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.956+0000 7ff8a37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:23.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.956+0000 7ff8a37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:23.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.956+0000 7ff8a3fff640 1 --2- 172.21.15.40:0/1960430228 >> v2:172.21.15.189:3300/0 conn(0x7ff8a412fc40 0x7ff8a4130020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:23.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.956+0000 7ff8a37fe640 10 monclient(hunting): handle_auth_done global_id 34737 payload 274 2024-04-22T08:17:23.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.956+0000 7ff8a37fe640 1 -- 172.21.15.40:0/1960430228 >> v2:172.21.15.189:3300/0 conn(0x7ff8a412fc40 msgr2=0x7ff8a4130020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:17:23.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.956+0000 7ff8a37fe640 1 --2- 172.21.15.40:0/1960430228 >> v2:172.21.15.189:3300/0 conn(0x7ff8a412fc40 0x7ff8a4130020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:23.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.956+0000 7ff8a37fe640 1 -- 172.21.15.40:0/1960430228 >> v1:172.21.15.189:6789/0 conn(0x7ff8a4132ed0 legacy=0x7ff8a4139cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:17:23.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.956+0000 7ff8a37fe640 10 monclient: _finish_hunting 0 2024-04-22T08:17:23.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.956+0000 7ff8a37fe640 1 monclient: found mon.noname-a 2024-04-22T08:17:23.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.956+0000 7ff8a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:17:23.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.956+0000 7ff8a37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:17:23.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.956+0000 7ff8a37fe640 1 -- 172.21.15.40:0/1960430228 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a409d550 con 0x7ff8a409e210 2024-04-22T08:17:23.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.956+0000 7ff8a3fff640 10 monclient: get_auth_request con 0x7ff8a412fc40 auth_method 0 2024-04-22T08:17:23.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.956+0000 7ff8a3fff640 1 --2- 172.21.15.40:0/1960430228 >> v2:172.21.15.189:3300/0 conn(0x7ff8a412fc40 0x7ff8a4130020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T08:17:23.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.959+0000 7ff8a37fe640 1 -- 172.21.15.40:0/1960430228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8a409e210 msgr2=0x7ff8a4132990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:17:23.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.959+0000 7ff8a37fe640 1 --2- 172.21.15.40:0/1960430228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8a409e210 0x7ff8a4132990 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff8a40a09c0 tx=0x7ff88c02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc00348712f5b30 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:23.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.959+0000 7ff8a2ffd640 1 -- 172.21.15.40:0/1960430228 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff88c031a40 con 0x7ff8a409e210 2024-04-22T08:17:23.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.959+0000 7ff8a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:23.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.959+0000 7ff8a2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:17:23.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.959+0000 7ff8a2ffd640 10 monclient: dump: 2024-04-22T08:17:23.966 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:23.966 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:23.967 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:23.967 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:23.967 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:23.967 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:23.967 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:23.967 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:23.967 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:23.967 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:23.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.959+0000 7ff8a2ffd640 10 monclient: _finish_auth 0 2024-04-22T08:17:23.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.959+0000 7ff8a2ffd640 10 monclient: _check_auth_tickets 2024-04-22T08:17:23.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.959+0000 7ff8a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:17:23.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.959+0000 7ff8a2ffd640 1 -- 172.21.15.40:0/1960430228 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff88c031ba0 con 0x7ff8a409e210 2024-04-22T08:17:23.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.959+0000 7ff8aa61c640 5 monclient: authenticate success, global_id 34737 2024-04-22T08:17:23.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.959+0000 7ff8aa61c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:17:23.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.959+0000 7ff8a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:17:23.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.959+0000 7ff8aa61c640 10 monclient: get_monmap_and_config success 2024-04-22T08:17:23.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.959+0000 7ff8a2ffd640 1 -- 172.21.15.40:0/1960430228 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff88c033440 con 0x7ff8a409e210 2024-04-22T08:17:23.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.959+0000 7ff8a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:23.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.959+0000 7ff8aa61c640 4 set_mon_vals no callback set 2024-04-22T08:17:23.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.959+0000 7ff8a2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:17:23.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.959+0000 7ff8a2ffd640 10 monclient: dump: 2024-04-22T08:17:23.968 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:23.968 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:23.968 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:23.968 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:23.968 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:23.968 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:23.968 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:23.968 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:23.969 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:23.969 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:23.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.959+0000 7ff8aa61c640 10 monclient: shutdown 2024-04-22T08:17:23.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.959+0000 7ff8aa61c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:17:23.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.959+0000 7ff8aa61c640 1 -- 172.21.15.40:0/1960430228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8a409e210 msgr2=0x7ff8a4132990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:23.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.959+0000 7ff8aa61c640 1 --2- 172.21.15.40:0/1960430228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8a409e210 0x7ff8a4132990 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff8a40a09c0 tx=0x7ff88c02b7f0 comp rx=0 tx=0).stop 2024-04-22T08:17:23.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.960+0000 7ff8aa61c640 1 -- 172.21.15.40:0/1960430228 shutdown_connections 2024-04-22T08:17:23.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.960+0000 7ff8aa61c640 1 --2- 172.21.15.40:0/1960430228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8a409e210 0x7ff8a4132990 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:23.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.960+0000 7ff8aa61c640 1 --2- 172.21.15.40:0/1960430228 >> v2:172.21.15.189:3300/0 conn(0x7ff8a412fc40 0x7ff8a4130020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:23.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.960+0000 7ff8aa61c640 1 -- 172.21.15.40:0/1960430228 >> 172.21.15.40:0/1960430228 conn(0x7ff8a40a0b50 msgr2=0x7ff8a40a0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:17:23.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.960+0000 7ff8aa61c640 1 -- 172.21.15.40:0/1960430228 shutdown_connections 2024-04-22T08:17:23.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.960+0000 7ff8aa61c640 1 -- 172.21.15.40:0/1960430228 wait complete. 2024-04-22T08:17:23.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.960+0000 7ff8aa61c640 10 monclient: build_initial_monmap 2024-04-22T08:17:23.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.960+0000 7ff8aa61c640 10 monclient: monmap: 2024-04-22T08:17:23.969 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:17:23.970 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:17:23.970 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:17:23.970 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:17:23.970 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:17:23.970 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:23.970 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:17:23.970 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:17:23.970 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:17:23.970 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:23.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.960+0000 7ff8aa61c640 1 Processor -- start 2024-04-22T08:17:23.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.961+0000 7ff8aa61c640 1 -- start start 2024-04-22T08:17:23.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.961+0000 7ff8aa61c640 10 monclient: init 2024-04-22T08:17:23.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.961+0000 7ff8aa61c640 10 monclient: _reopen_session rank -1 2024-04-22T08:17:23.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.961+0000 7ff8aa61c640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-22T08:17:23.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.961+0000 7ff8aa61c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8a409e210 0x7ff8a412e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:23.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.961+0000 7ff8aa61c640 10 monclient(hunting): picked mon.noname-a con 0x7ff8a409e210 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:17:23.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.961+0000 7ff8aa61c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8a412fc40 0x7ff8a4133790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:23.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.961+0000 7ff8aa61c640 10 monclient(hunting): picked mon.noname-c con 0x7ff8a412fc40 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:17:23.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.961+0000 7ff8aa61c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8a4132ed0 0x7ff8a4133cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:23.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.961+0000 7ff8aa61c640 10 monclient(hunting): picked mon.noname-b con 0x7ff8a4132ed0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:17:23.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.961+0000 7ff8aa61c640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:23.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.961+0000 7ff8aa61c640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a413be60 con 0x7ff8a409e210 2024-04-22T08:17:23.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.961+0000 7ff8aa61c640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:23.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.961+0000 7ff8aa61c640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a40a3d00 con 0x7ff8a4132ed0 2024-04-22T08:17:23.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.961+0000 7ff8aa61c640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:23.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.961+0000 7ff8aa61c640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a4135860 con 0x7ff8a412fc40 2024-04-22T08:17:23.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.961+0000 7ff8aa61c640 10 monclient(hunting): _renew_subs 2024-04-22T08:17:23.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.961+0000 7ff8aa61c640 10 monclient(hunting): authenticate will time out at 1117.956414s 2024-04-22T08:17:23.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.961+0000 7ff8a3fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8a409e210 0x7ff8a412e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:23.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.961+0000 7ff8a37fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8a412fc40 0x7ff8a4133790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:23.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.961+0000 7ff8a3fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8a409e210 0x7ff8a412e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60164/0 (socket says 172.21.15.40:60164) 2024-04-22T08:17:23.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.961+0000 7ff8a3fff640 1 -- 172.21.15.40:0/576329961 learned_addr learned my addr 172.21.15.40:0/576329961 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:17:23.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.961+0000 7ff8a3fff640 10 monclient(hunting): get_auth_request con 0x7ff8a409e210 auth_method 0 2024-04-22T08:17:23.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.961+0000 7ff8a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:23.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.961+0000 7ff8a3fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:23.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.961+0000 7ff8a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:23.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.961+0000 7ff8a37fe640 10 monclient(hunting): get_auth_request con 0x7ff8a412fc40 auth_method 0 2024-04-22T08:17:23.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.961+0000 7ff8a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:23.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.961+0000 7ff8a37fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:23.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.961+0000 7ff8a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:23.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.961+0000 7ff8a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:23.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.961+0000 7ff8a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:23.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.961+0000 7ff8a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:23.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.961+0000 7ff8a8bb9640 1 --2- 172.21.15.40:0/576329961 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8a4132ed0 0x7ff8a4133cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:23.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.961+0000 7ff8a37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:23.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.961+0000 7ff8a37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:23.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.961+0000 7ff8a37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:23.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.961+0000 7ff8a8bb9640 10 monclient(hunting): get_auth_request con 0x7ff8a4132ed0 auth_method 0 2024-04-22T08:17:23.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.961+0000 7ff8a8bb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:23.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.961+0000 7ff8a8bb9640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:23.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.961+0000 7ff8a8bb9640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:23.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.962+0000 7ff8a8bb9640 1 -- 172.21.15.40:0/576329961 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8a4132ed0 msgr2=0x7ff8a4133cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-22T08:17:23.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.962+0000 7ff8a3fff640 10 monclient(hunting): handle_auth_done global_id 34740 payload 824 2024-04-22T08:17:23.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.962+0000 7ff8a8bb9640 1 -- 172.21.15.40:0/576329961 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8a4132ed0 msgr2=0x7ff8a4133cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:17:23.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.962+0000 7ff8a8bb9640 1 --2- 172.21.15.40:0/576329961 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8a4132ed0 0x7ff8a4133cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:17:23.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.962+0000 7ff8a3fff640 1 -- 172.21.15.40:0/576329961 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8a412fc40 msgr2=0x7ff8a4133790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:23.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.962+0000 7ff8a3fff640 1 --2- 172.21.15.40:0/576329961 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8a412fc40 0x7ff8a4133790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:23.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.962+0000 7ff8a3fff640 1 -- 172.21.15.40:0/576329961 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8a4132ed0 msgr2=0x7ff8a4133cd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:17:23.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.962+0000 7ff8a8bb9640 1 --2- 172.21.15.40:0/576329961 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8a4132ed0 0x7ff8a4133cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:17:23.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.962+0000 7ff8a3fff640 1 --2- 172.21.15.40:0/576329961 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8a4132ed0 0x7ff8a4133cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:23.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.962+0000 7ff8a3fff640 10 monclient: _finish_hunting 0 2024-04-22T08:17:23.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.962+0000 7ff8a3fff640 1 monclient: found mon.noname-a 2024-04-22T08:17:23.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.962+0000 7ff8a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:17:23.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.962+0000 7ff8a3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:17:23.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.962+0000 7ff8a3fff640 1 -- 172.21.15.40:0/576329961 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff88c0094d0 con 0x7ff8a409e210 2024-04-22T08:17:23.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.964+0000 7ff8a3fff640 1 -- 172.21.15.40:0/576329961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8a409e210 msgr2=0x7ff8a412e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:17:23.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.964+0000 7ff8a3fff640 1 --2- 172.21.15.40:0/576329961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8a409e210 0x7ff8a412e7b0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff89400c7f0 tx=0x7ff89400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:23.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.964+0000 7ff8a0ff9640 1 -- 172.21.15.40:0/576329961 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff894007c40 con 0x7ff8a409e210 2024-04-22T08:17:23.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.964+0000 7ff8a0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:23.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.964+0000 7ff8a0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:17:23.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.964+0000 7ff8a0ff9640 10 monclient: dump: 2024-04-22T08:17:23.975 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:23.975 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:23.975 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:23.975 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:23.975 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:23.975 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:23.975 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:23.975 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:23.975 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:23.975 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:23.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.964+0000 7ff8a0ff9640 10 monclient: _finish_auth 0 2024-04-22T08:17:23.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.964+0000 7ff8a0ff9640 10 monclient: _check_auth_tickets 2024-04-22T08:17:23.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.964+0000 7ff8a0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:17:23.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.964+0000 7ff8a0ff9640 1 -- 172.21.15.40:0/576329961 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff894007da0 con 0x7ff8a409e210 2024-04-22T08:17:23.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.964+0000 7ff8aa61c640 5 monclient: authenticate success, global_id 34740 2024-04-22T08:17:23.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.964+0000 7ff8aa61c640 10 monclient: _renew_subs 2024-04-22T08:17:23.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.964+0000 7ff8aa61c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:17:23.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.964+0000 7ff8aa61c640 1 -- 172.21.15.40:0/576329961 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8a4135b40 con 0x7ff8a409e210 2024-04-22T08:17:23.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.964+0000 7ff8a0ff9640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:17:23.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.964+0000 7ff8a0ff9640 1 -- 172.21.15.40:0/576329961 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff894011610 con 0x7ff8a409e210 2024-04-22T08:17:23.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.964+0000 7ff8a0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:23.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.964+0000 7ff8a0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:17:23.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.964+0000 7ff8a0ff9640 10 monclient: dump: 2024-04-22T08:17:23.976 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:23.976 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:23.976 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:23.976 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:23.977 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:23.977 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:23.977 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:23.977 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:23.977 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:23.977 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:23.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.964+0000 7ff8a27fc640 4 set_mon_vals no callback set 2024-04-22T08:17:23.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.964+0000 7ff8aa61c640 10 monclient: _renew_subs 2024-04-22T08:17:23.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.964+0000 7ff8aa61c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:17:23.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.964+0000 7ff8aa61c640 1 -- 172.21.15.40:0/576329961 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8a413c070 con 0x7ff8a409e210 2024-04-22T08:17:23.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.965+0000 7ff8a0ff9640 1 -- 172.21.15.40:0/576329961 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7ff894011770 con 0x7ff8a409e210 2024-04-22T08:17:23.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.965+0000 7ff8aa61c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:17:23.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.965+0000 7ff8aa61c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:17:23.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.965+0000 7ff8aa61c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:17:23.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.965+0000 7ff8a0ff9640 1 --2- 172.21.15.40:0/576329961 >> v2:172.21.15.189:6800/30168 conn(0x7ff8840377d0 0x7ff884039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:23.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.965+0000 7ff8aa61c640 1 -- 172.21.15.40:0/576329961 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8a404e9b0 con 0x7ff8a409e210 2024-04-22T08:17:23.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.965+0000 7ff8a0ff9640 1 -- 172.21.15.40:0/576329961 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 3898+0+0 (secure 0 0 0) 0x7ff89404a690 con 0x7ff8a409e210 2024-04-22T08:17:23.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.965+0000 7ff8a37fe640 1 --2- 172.21.15.40:0/576329961 >> v2:172.21.15.189:6800/30168 conn(0x7ff8840377d0 0x7ff884039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:23.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.966+0000 7ff8a37fe640 10 monclient: get_auth_request con 0x7ff8840377d0 auth_method 0 2024-04-22T08:17:23.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.968+0000 7ff8a37fe640 1 --2- 172.21.15.40:0/576329961 >> v2:172.21.15.189:6800/30168 conn(0x7ff8840377d0 0x7ff884039c90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff8a409af00 tx=0x7ff88c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:23.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.970+0000 7ff8a0ff9640 1 -- 172.21.15.40:0/576329961 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7ff894018360 con 0x7ff8a409e210 2024-04-22T08:17:23.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.970+0000 7ff8a0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:17:23.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:23.970+0000 7ff8a0ff9640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:17:24.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.224+0000 7ff8aa61c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:17:24.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.224+0000 7ff8aa61c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:17:24.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.224+0000 7ff8aa61c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:17:24.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.224+0000 7ff8aa61c640 1 -- 172.21.15.40:0/576329961 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff8a406a290 con 0x7ff8a409e210 2024-04-22T08:17:24.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.225+0000 7ff8a0ff9640 1 -- 172.21.15.40:0/576329961 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7ff894038ab0 con 0x7ff8a409e210 2024-04-22T08:17:24.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.225+0000 7ff8a0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:17:24.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.225+0000 7ff8a0ff9640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:17:24.225 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":46,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:17:24.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.227+0000 7ff8aa61c640 1 -- 172.21.15.40:0/576329961 >> v2:172.21.15.189:6800/30168 conn(0x7ff8840377d0 msgr2=0x7ff884039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:24.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.227+0000 7ff8aa61c640 1 --2- 172.21.15.40:0/576329961 >> v2:172.21.15.189:6800/30168 conn(0x7ff8840377d0 0x7ff884039c90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff8a409af00 tx=0x7ff88c008030 comp rx=0 tx=0).stop 2024-04-22T08:17:24.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.227+0000 7ff8aa61c640 10 monclient: shutdown 2024-04-22T08:17:24.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.227+0000 7ff8aa61c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:17:24.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.227+0000 7ff8aa61c640 1 -- 172.21.15.40:0/576329961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8a409e210 msgr2=0x7ff8a412e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:24.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.227+0000 7ff8aa61c640 1 --2- 172.21.15.40:0/576329961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8a409e210 0x7ff8a412e7b0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff89400c7f0 tx=0x7ff89400cbf0 comp rx=0 tx=0).stop 2024-04-22T08:17:24.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.227+0000 7ff8aa61c640 1 -- 172.21.15.40:0/576329961 shutdown_connections 2024-04-22T08:17:24.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.227+0000 7ff8aa61c640 1 --2- 172.21.15.40:0/576329961 >> v2:172.21.15.189:6800/30168 conn(0x7ff8840377d0 0x7ff884039c90 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:24.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.227+0000 7ff8aa61c640 1 --2- 172.21.15.40:0/576329961 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8a4132ed0 0x7ff8a4133cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:24.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.227+0000 7ff8aa61c640 1 --2- 172.21.15.40:0/576329961 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8a412fc40 0x7ff8a4133790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:24.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.227+0000 7ff8aa61c640 1 --2- 172.21.15.40:0/576329961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8a409e210 0x7ff8a412e7b0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:24.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.227+0000 7ff8aa61c640 1 -- 172.21.15.40:0/576329961 >> 172.21.15.40:0/576329961 conn(0x7ff8a40a0b50 msgr2=0x7ff8a4070e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:17:24.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.227+0000 7ff8aa61c640 1 -- 172.21.15.40:0/576329961 shutdown_connections 2024-04-22T08:17:24.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.227+0000 7ff8aa61c640 1 -- 172.21.15.40:0/576329961 wait complete. 2024-04-22T08:17:24.237 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-22T08:17:24.238 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-22T08:17:24.238 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-22T08:17:24.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.338+0000 7fce68d8a640 10 monclient: get_monmap_and_config 2024-04-22T08:17:24.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.338+0000 7fce68d8a640 10 monclient: build_initial_monmap 2024-04-22T08:17:24.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.338+0000 7fce68d8a640 10 monclient: monmap: 2024-04-22T08:17:24.339 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:17:24.339 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:24.339 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:17:24.340293+0000 2024-04-22T08:17:24.339 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:17:24.340293+0000 2024-04-22T08:17:24.339 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:17:24.339 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:24.339 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:17:24.339 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:17:24.340 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:17:24.340 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:17:24.340 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:17:24.340 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:24.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.339+0000 7fce68d8a640 1 Processor -- start 2024-04-22T08:17:24.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.339+0000 7fce68d8a640 1 -- start start 2024-04-22T08:17:24.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.339+0000 7fce68d8a640 10 monclient: init 2024-04-22T08:17:24.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.339+0000 7fce68d8a640 10 monclient: _reopen_session rank -1 2024-04-22T08:17:24.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.339+0000 7fce68d8a640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-04-22T08:17:24.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.339+0000 7fce68d8a640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fce640a1a80 0x7fce640a01b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:24.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.339+0000 7fce68d8a640 10 monclient(hunting): picked mon.noname-c con 0x7fce640a1a80 addr v2:172.21.15.189:3300/0 2024-04-22T08:17:24.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.339+0000 7fce68d8a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fce640a1e60 addr v1:172.21.15.189:6789/0 2024-04-22T08:17:24.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.339+0000 7fce68d8a640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce6413b0e0 0x7fce6413d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:24.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.339+0000 7fce68d8a640 10 monclient(hunting): picked mon.noname-b con 0x7fce6413b0e0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:17:24.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.339+0000 7fce68d8a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce6409d7f0 con 0x7fce640a1e60 2024-04-22T08:17:24.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.339+0000 7fce68d8a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:24.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.339+0000 7fce68d8a640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce6409f1c0 con 0x7fce640a1a80 2024-04-22T08:17:24.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.339+0000 7fce68d8a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:24.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.339+0000 7fce68d8a640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce6409dc30 con 0x7fce6413b0e0 2024-04-22T08:17:24.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.339+0000 7fce68d8a640 10 monclient(hunting): _renew_subs 2024-04-22T08:17:24.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.339+0000 7fce68d8a640 10 monclient(hunting): authenticate will time out at 1118.334520s 2024-04-22T08:17:24.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.339+0000 7fce62d9d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce6413b0e0 0x7fce6413d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:24.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.339+0000 7fce62d9d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce6413b0e0 0x7fce6413d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:52496/0 (socket says 172.21.15.40:52496) 2024-04-22T08:17:24.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.339+0000 7fce62d9d640 1 -- 172.21.15.40:0/2060840818 learned_addr learned my addr 172.21.15.40:0/2060840818 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:17:24.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.339+0000 7fce62d9d640 10 monclient(hunting): get_auth_request con 0x7fce6413b0e0 auth_method 0 2024-04-22T08:17:24.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.339+0000 7fce62d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:24.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.339+0000 7fce62d9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:24.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.339+0000 7fce62d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:24.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.340+0000 7fce62d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:24.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.340+0000 7fce62d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:24.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.340+0000 7fce62d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:24.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.340+0000 7fce6259c640 1 --2- 172.21.15.40:0/2060840818 >> v2:172.21.15.189:3300/0 conn(0x7fce640a1a80 0x7fce640a01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:24.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.340+0000 7fce62d9d640 10 monclient(hunting): handle_auth_done global_id 54299 payload 274 2024-04-22T08:17:24.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.340+0000 7fce62d9d640 1 -- 172.21.15.40:0/2060840818 >> v2:172.21.15.189:3300/0 conn(0x7fce640a1a80 msgr2=0x7fce640a01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:17:24.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.340+0000 7fce62d9d640 1 --2- 172.21.15.40:0/2060840818 >> v2:172.21.15.189:3300/0 conn(0x7fce640a1a80 0x7fce640a01b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:24.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.340+0000 7fce62d9d640 1 -- 172.21.15.40:0/2060840818 >> v1:172.21.15.189:6789/0 conn(0x7fce640a1e60 legacy=0x7fce640a22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:17:24.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.340+0000 7fce62d9d640 10 monclient: _finish_hunting 0 2024-04-22T08:17:24.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.340+0000 7fce62d9d640 1 monclient: found mon.noname-b 2024-04-22T08:17:24.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.340+0000 7fce62d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:17:24.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.340+0000 7fce62d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:17:24.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.340+0000 7fce62d9d640 1 -- 172.21.15.40:0/2060840818 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce6409eee0 con 0x7fce6413b0e0 2024-04-22T08:17:24.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.340+0000 7fce6259c640 10 monclient: get_auth_request con 0x7fce640a1a80 auth_method 0 2024-04-22T08:17:24.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.340+0000 7fce6259c640 1 --2- 172.21.15.40:0/2060840818 >> v2:172.21.15.189:3300/0 conn(0x7fce640a1a80 0x7fce640a01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T08:17:24.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.343+0000 7fce62d9d640 1 -- 172.21.15.40:0/2060840818 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce6413b0e0 msgr2=0x7fce6413d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:17:24.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.343+0000 7fce62d9d640 1 --2- 172.21.15.40:0/2060840818 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce6413b0e0 0x7fce6413d4d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fce5c009870 tx=0x7fce5c02b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6fb7f3ebed00010f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:24.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.343+0000 7fce6159a640 1 -- 172.21.15.40:0/2060840818 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5c031a30 con 0x7fce6413b0e0 2024-04-22T08:17:24.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.343+0000 7fce6159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:24.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.343+0000 7fce6159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:17:24.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.343+0000 7fce6159a640 10 monclient: dump: 2024-04-22T08:17:24.352 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:24.352 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:24.352 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:24.352 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:24.352 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:24.352 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:24.352 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:24.352 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:24.352 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:24.352 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:24.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.343+0000 7fce6159a640 10 monclient: _finish_auth 0 2024-04-22T08:17:24.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.343+0000 7fce6159a640 10 monclient: _check_auth_tickets 2024-04-22T08:17:24.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.343+0000 7fce6159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:17:24.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.343+0000 7fce6159a640 1 -- 172.21.15.40:0/2060840818 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce5c031b90 con 0x7fce6413b0e0 2024-04-22T08:17:24.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.343+0000 7fce6159a640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:17:24.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.343+0000 7fce68d8a640 5 monclient: authenticate success, global_id 54299 2024-04-22T08:17:24.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.343+0000 7fce68d8a640 10 monclient: get_monmap_and_config success 2024-04-22T08:17:24.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.343+0000 7fce68d8a640 4 set_mon_vals no callback set 2024-04-22T08:17:24.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.343+0000 7fce6159a640 1 -- 172.21.15.40:0/2060840818 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5c033430 con 0x7fce6413b0e0 2024-04-22T08:17:24.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.343+0000 7fce6159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:24.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.343+0000 7fce6159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:17:24.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.343+0000 7fce6159a640 10 monclient: dump: 2024-04-22T08:17:24.353 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:24.353 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:24.353 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:24.353 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:24.354 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:24.354 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:24.354 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:24.354 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:24.354 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:24.354 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:24.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.343+0000 7fce68d8a640 10 monclient: shutdown 2024-04-22T08:17:24.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.343+0000 7fce68d8a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:17:24.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.343+0000 7fce68d8a640 1 -- 172.21.15.40:0/2060840818 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce6413b0e0 msgr2=0x7fce6413d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:24.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.343+0000 7fce68d8a640 1 --2- 172.21.15.40:0/2060840818 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce6413b0e0 0x7fce6413d4d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fce5c009870 tx=0x7fce5c02b7e0 comp rx=0 tx=0).stop 2024-04-22T08:17:24.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.343+0000 7fce68d8a640 1 -- 172.21.15.40:0/2060840818 shutdown_connections 2024-04-22T08:17:24.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.343+0000 7fce68d8a640 1 --2- 172.21.15.40:0/2060840818 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce6413b0e0 0x7fce6413d4d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:24.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.343+0000 7fce68d8a640 1 --2- 172.21.15.40:0/2060840818 >> v2:172.21.15.189:3300/0 conn(0x7fce640a1a80 0x7fce640a01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:24.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.343+0000 7fce68d8a640 1 -- 172.21.15.40:0/2060840818 >> 172.21.15.40:0/2060840818 conn(0x7fce640a66a0 msgr2=0x7fce640a2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:17:24.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.343+0000 7fce68d8a640 1 -- 172.21.15.40:0/2060840818 shutdown_connections 2024-04-22T08:17:24.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.343+0000 7fce68d8a640 1 -- 172.21.15.40:0/2060840818 wait complete. 2024-04-22T08:17:24.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.344+0000 7fce68d8a640 10 monclient: build_initial_monmap 2024-04-22T08:17:24.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.344+0000 7fce68d8a640 10 monclient: monmap: 2024-04-22T08:17:24.355 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:17:24.355 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:17:24.355 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:17:24.355 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:17:24.355 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:17:24.355 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:24.355 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:17:24.355 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:17:24.355 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:17:24.355 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:24.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.344+0000 7fce68d8a640 1 Processor -- start 2024-04-22T08:17:24.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.344+0000 7fce68d8a640 1 -- start start 2024-04-22T08:17:24.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.344+0000 7fce68d8a640 10 monclient: init 2024-04-22T08:17:24.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce68d8a640 10 monclient: _reopen_session rank -1 2024-04-22T08:17:24.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce68d8a640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-22T08:17:24.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce68d8a640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce640a1a80 0x7fce641caf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:24.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce68d8a640 10 monclient(hunting): picked mon.noname-c con 0x7fce640a1a80 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:17:24.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce68d8a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce640a1e60 0x7fce641cb440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:24.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce68d8a640 10 monclient(hunting): picked mon.noname-b con 0x7fce640a1e60 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:17:24.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce68d8a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce6413b0e0 0x7fce64120ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:24.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce68d8a640 10 monclient(hunting): picked mon.noname-a con 0x7fce6413b0e0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:17:24.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce68d8a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:24.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce68d8a640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce64123dd0 con 0x7fce6413b0e0 2024-04-22T08:17:24.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce68d8a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:24.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce68d8a640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce641c2930 con 0x7fce640a1e60 2024-04-22T08:17:24.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce68d8a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:24.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce68d8a640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce64122f50 con 0x7fce640a1a80 2024-04-22T08:17:24.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce68d8a640 10 monclient(hunting): _renew_subs 2024-04-22T08:17:24.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce68d8a640 10 monclient(hunting): authenticate will time out at 1118.340111s 2024-04-22T08:17:24.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce6259c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce640a1a80 0x7fce641caf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:24.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce62d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce6413b0e0 0x7fce64120ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:24.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce6259c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce640a1a80 0x7fce641caf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:52500/0 (socket says 172.21.15.40:52500) 2024-04-22T08:17:24.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce6259c640 1 -- 172.21.15.40:0/774353529 learned_addr learned my addr 172.21.15.40:0/774353529 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:17:24.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce6259c640 10 monclient(hunting): get_auth_request con 0x7fce640a1a80 auth_method 0 2024-04-22T08:17:24.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce6259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:24.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce6259c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:24.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce6259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:24.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce62d9d640 10 monclient(hunting): get_auth_request con 0x7fce6413b0e0 auth_method 0 2024-04-22T08:17:24.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce62d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:24.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce62d9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:24.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce62d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:24.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce61d9b640 1 --2- 172.21.15.40:0/774353529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce640a1e60 0x7fce641cb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:24.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce61d9b640 10 monclient(hunting): get_auth_request con 0x7fce640a1e60 auth_method 0 2024-04-22T08:17:24.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce61d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:24.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce61d9b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:24.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce61d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:24.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce6259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:24.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce6259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:24.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce6259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:24.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce62d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:24.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce62d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:24.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce62d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:24.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce61d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:24.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce61d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:24.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce61d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:24.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce6259c640 10 monclient(hunting): handle_auth_done global_id 54302 payload 824 2024-04-22T08:17:24.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce6259c640 1 -- 172.21.15.40:0/774353529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce640a1e60 msgr2=0x7fce641cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:24.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce6259c640 1 --2- 172.21.15.40:0/774353529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce640a1e60 0x7fce641cb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:24.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce6259c640 1 -- 172.21.15.40:0/774353529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce6413b0e0 msgr2=0x7fce64120ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:24.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce6259c640 1 --2- 172.21.15.40:0/774353529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce6413b0e0 0x7fce64120ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:24.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce6259c640 10 monclient: _finish_hunting 0 2024-04-22T08:17:24.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce6259c640 1 monclient: found mon.noname-c 2024-04-22T08:17:24.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce6259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:17:24.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce6259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:17:24.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce6259c640 1 -- 172.21.15.40:0/774353529 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce5c0094d0 con 0x7fce640a1a80 2024-04-22T08:17:24.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.345+0000 7fce62d9d640 1 --2- 172.21.15.40:0/774353529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce6413b0e0 0x7fce64120ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:17:24.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.348+0000 7fce6259c640 1 -- 172.21.15.40:0/774353529 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce640a1a80 msgr2=0x7fce641caf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:17:24.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.348+0000 7fce6259c640 1 --2- 172.21.15.40:0/774353529 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce640a1a80 0x7fce641caf00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fce5400cf90 tx=0x7fce5400a7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:24.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.348+0000 7fce52ffd640 1 -- 172.21.15.40:0/774353529 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5400da50 con 0x7fce640a1a80 2024-04-22T08:17:24.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.348+0000 7fce52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:24.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.348+0000 7fce52ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:17:24.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.348+0000 7fce52ffd640 10 monclient: dump: 2024-04-22T08:17:24.360 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:24.360 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:24.360 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:24.360 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:24.360 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:24.360 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:24.360 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:24.361 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:24.361 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:24.361 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:24.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.348+0000 7fce52ffd640 10 monclient: _finish_auth 0 2024-04-22T08:17:24.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.348+0000 7fce52ffd640 10 monclient: _check_auth_tickets 2024-04-22T08:17:24.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.348+0000 7fce52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:17:24.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.348+0000 7fce52ffd640 1 -- 172.21.15.40:0/774353529 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce5400dbb0 con 0x7fce640a1a80 2024-04-22T08:17:24.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.348+0000 7fce52ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:17:24.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.348+0000 7fce52ffd640 1 -- 172.21.15.40:0/774353529 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce54015440 con 0x7fce640a1a80 2024-04-22T08:17:24.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.348+0000 7fce68d8a640 5 monclient: authenticate success, global_id 54302 2024-04-22T08:17:24.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.348+0000 7fce68d8a640 10 monclient: _renew_subs 2024-04-22T08:17:24.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.348+0000 7fce68d8a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:17:24.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.348+0000 7fce68d8a640 1 -- 172.21.15.40:0/774353529 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce6409cc20 con 0x7fce640a1a80 2024-04-22T08:17:24.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.348+0000 7fce60d99640 4 set_mon_vals no callback set 2024-04-22T08:17:24.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.348+0000 7fce52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:24.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.348+0000 7fce52ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:17:24.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.348+0000 7fce52ffd640 10 monclient: dump: 2024-04-22T08:17:24.362 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:24.362 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:24.362 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:24.362 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:24.362 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:24.362 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:24.362 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:24.362 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:24.362 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:24.362 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:24.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.348+0000 7fce68d8a640 10 monclient: _renew_subs 2024-04-22T08:17:24.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.348+0000 7fce68d8a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:17:24.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.348+0000 7fce68d8a640 1 -- 172.21.15.40:0/774353529 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce64122c60 con 0x7fce640a1a80 2024-04-22T08:17:24.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.349+0000 7fce68d8a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:17:24.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.349+0000 7fce68d8a640 1 --2- 172.21.15.40:0/774353529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce64121010 0x7fce641213f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:24.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.349+0000 7fce68d8a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:24.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.349+0000 7fce68d8a640 1 -- 172.21.15.40:0/774353529 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce64066770 con 0x7fce64121010 2024-04-22T08:17:24.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.349+0000 7fce52ffd640 1 -- 172.21.15.40:0/774353529 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fce540155a0 con 0x7fce640a1a80 2024-04-22T08:17:24.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.349+0000 7fce61d9b640 1 --2- 172.21.15.40:0/774353529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce64121010 0x7fce641213f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:24.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.349+0000 7fce61d9b640 10 monclient: get_auth_request con 0x7fce64121010 auth_method 0 2024-04-22T08:17:24.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.349+0000 7fce61d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:24.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.349+0000 7fce61d9b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:24.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.349+0000 7fce61d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:24.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.349+0000 7fce61d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:24.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.349+0000 7fce61d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:24.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.349+0000 7fce61d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:24.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.349+0000 7fce52ffd640 1 --2- 172.21.15.40:0/774353529 >> v2:172.21.15.189:6800/30168 conn(0x7fce44037820 0x7fce44039ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:24.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.349+0000 7fce52ffd640 1 -- 172.21.15.40:0/774353529 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 3898+0+0 (secure 0 0 0) 0x7fce5404aea0 con 0x7fce640a1a80 2024-04-22T08:17:24.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.349+0000 7fce61d9b640 10 monclient(hunting): handle_auth_done global_id 34746 payload 824 2024-04-22T08:17:24.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.349+0000 7fce61d9b640 1 -- 172.21.15.40:0/774353529 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fce6413da10 con 0x7fce64121010 2024-04-22T08:17:24.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.349+0000 7fce62d9d640 1 --2- 172.21.15.40:0/774353529 >> v2:172.21.15.189:6800/30168 conn(0x7fce44037820 0x7fce44039ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:24.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.349+0000 7fce62d9d640 10 monclient: get_auth_request con 0x7fce44037820 auth_method 0 2024-04-22T08:17:24.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.351+0000 7fce61d9b640 1 -- 172.21.15.40:0/774353529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce64121010 msgr2=0x7fce641213f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:17:24.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.351+0000 7fce61d9b640 1 --2- 172.21.15.40:0/774353529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce64121010 0x7fce641213f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fce6409cf20 tx=0x7fce5800f9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:24.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.351+0000 7fce52ffd640 1 -- 172.21.15.40:0/774353529 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce580035f0 con 0x7fce64121010 2024-04-22T08:17:24.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.351+0000 7fce52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:24.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.351+0000 7fce52ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:17:24.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.351+0000 7fce52ffd640 10 monclient: dump: 2024-04-22T08:17:24.368 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:24.368 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:24.368 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:24.368 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:24.368 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:24.368 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:24.368 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:24.368 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:24.368 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:24.368 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:24.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.352+0000 7fce52ffd640 1 -- 172.21.15.40:0/774353529 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7fce58003750 con 0x7fce64121010 2024-04-22T08:17:24.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.352+0000 7fce52ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:17:24.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.352+0000 7fce52ffd640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:17:24.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.352+0000 7fce52ffd640 1 -- 172.21.15.40:0/774353529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce64121010 msgr2=0x7fce641213f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:24.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.352+0000 7fce52ffd640 1 --2- 172.21.15.40:0/774353529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce64121010 0x7fce641213f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fce6409cf20 tx=0x7fce5800f9b0 comp rx=0 tx=0).stop 2024-04-22T08:17:24.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.352+0000 7fce52ffd640 1 -- 172.21.15.40:0/774353529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce64121010 msgr2=0x7fce641213f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:17:24.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.352+0000 7fce52ffd640 1 --2- 172.21.15.40:0/774353529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce64121010 0x7fce641213f0 secure :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fce6409cf20 tx=0x7fce5800f9b0 comp rx=0 tx=0).stop 2024-04-22T08:17:24.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.352+0000 7fce62d9d640 1 --2- 172.21.15.40:0/774353529 >> v2:172.21.15.189:6800/30168 conn(0x7fce44037820 0x7fce44039ce0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fce5c035000 tx=0x7fce5c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:24.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.358+0000 7fce68d8a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:17:24.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.358+0000 7fce68d8a640 1 --2- 172.21.15.40:0/774353529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce64121930 0x7fce6406db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:24.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.358+0000 7fce68d8a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:24.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.358+0000 7fce68d8a640 1 -- 172.21.15.40:0/774353529 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce6413d490 con 0x7fce64121930 2024-04-22T08:17:24.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.358+0000 7fce61d9b640 1 --2- 172.21.15.40:0/774353529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce64121930 0x7fce6406db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:24.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.358+0000 7fce61d9b640 10 monclient: get_auth_request con 0x7fce64121930 auth_method 0 2024-04-22T08:17:24.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.358+0000 7fce61d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:24.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.358+0000 7fce61d9b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:24.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.358+0000 7fce61d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:24.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.358+0000 7fce61d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:24.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.358+0000 7fce61d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:24.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.358+0000 7fce61d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:24.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.358+0000 7fce61d9b640 10 monclient(hunting): handle_auth_done global_id 34749 payload 824 2024-04-22T08:17:24.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.359+0000 7fce61d9b640 1 -- 172.21.15.40:0/774353529 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fce6404e9b0 con 0x7fce64121930 2024-04-22T08:17:24.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.361+0000 7fce61d9b640 1 -- 172.21.15.40:0/774353529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce64121930 msgr2=0x7fce6406db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:17:24.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.361+0000 7fce61d9b640 1 --2- 172.21.15.40:0/774353529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce64121930 0x7fce6406db80 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fce6409cf20 tx=0x7fce58001020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:24.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.361+0000 7fce52ffd640 1 -- 172.21.15.40:0/774353529 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce58023070 con 0x7fce64121930 2024-04-22T08:17:24.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.361+0000 7fce52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:24.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.361+0000 7fce52ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:17:24.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.361+0000 7fce52ffd640 10 monclient: dump: 2024-04-22T08:17:24.371 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:24.371 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:24.371 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:24.371 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:24.371 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:24.371 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:24.371 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:24.371 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:24.371 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:24.371 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:24.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.361+0000 7fce52ffd640 1 -- 172.21.15.40:0/774353529 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4633 (secure 0 0 0) 0x7fce58007550 con 0x7fce64121930 2024-04-22T08:17:24.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.361+0000 7fce52ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:17:24.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.361+0000 7fce52ffd640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:17:24.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.361+0000 7fce52ffd640 1 -- 172.21.15.40:0/774353529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce64121930 msgr2=0x7fce6406db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:24.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.361+0000 7fce52ffd640 1 --2- 172.21.15.40:0/774353529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce64121930 0x7fce6406db80 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fce6409cf20 tx=0x7fce58001020 comp rx=0 tx=0).stop 2024-04-22T08:17:24.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.361+0000 7fce52ffd640 1 -- 172.21.15.40:0/774353529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce64121930 msgr2=0x7fce6406db80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:17:24.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.361+0000 7fce52ffd640 1 --2- 172.21.15.40:0/774353529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce64121930 0x7fce6406db80 secure :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fce6409cf20 tx=0x7fce58001020 comp rx=0 tx=0).stop 2024-04-22T08:17:24.372 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:17:24.372 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:17:24.372 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:17:24.372 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "leader", 2024-04-22T08:17:24.372 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 46, 2024-04-22T08:17:24.372 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:17:24.372 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:17:24.373 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:17:24.373 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:17:24.373 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:17:24.373 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 0, 2024-04-22T08:17:24.373 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:17:24.373 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:17:24.373 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:17:24.373 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:17:24.373 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:17:24.373 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:17:24.373 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:17:24.373 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:17:24.373 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:17:24.373 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:17:24.374 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:17:24.374 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:17:24.374 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:17:24.374 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:17:24.374 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:17:24.374 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:17:24.374 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:17:24.374 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:17:24.374 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:17:24.374 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:17:24.374 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:17:24.374 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:17:24.374 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:17:24.375 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:17:24.375 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:17:24.375 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:17:24.375 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:24.375 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:24.375 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:17:24.375 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:17:24.375 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:17:24.375 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:17:24.375 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:17:24.375 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:17:24.375 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:17:24.375 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:17:24.375 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:17:24.376 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:17:24.376 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:17:24.376 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:17:24.376 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:17:24.376 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:17:24.376 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:17:24.376 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:17:24.376 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:17:24.376 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:17:24.376 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:17:24.376 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:17:24.376 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:17:24.376 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:17:24.376 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:17:24.377 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:17:24.377 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:17:24.377 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:17:24.377 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:17:24.377 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:17:24.377 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:17:24.377 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:24.377 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:17:24.377 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:24.377 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:17:24.377 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:17:24.377 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:17:24.377 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:17:24.378 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:24.378 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:17:24.378 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:17:24.378 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:24.378 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:24.378 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:24.378 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:17:24.378 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:17:24.378 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:24.378 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:24.378 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:24.378 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:24.378 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:17:24.379 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:17:24.379 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:17:24.379 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:17:24.379 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:17:24.379 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:24.379 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:24.379 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:17:24.379 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:17:24.379 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:17:24.379 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:17:24.379 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:24.379 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:17:24.379 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:17:24.379 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:24.380 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:24.380 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:24.380 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:17:24.380 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:17:24.380 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:24.380 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:24.380 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:24.380 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:24.380 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:17:24.380 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:17:24.380 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:17:24.380 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:17:24.380 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:17:24.380 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:24.381 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:24.381 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:17:24.381 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:17:24.381 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:17:24.381 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:17:24.381 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:24.381 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:17:24.381 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:17:24.381 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:24.381 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:24.382 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:24.382 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:17:24.382 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:17:24.382 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:24.382 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:24.382 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:24.382 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:24.382 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:17:24.382 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:17:24.382 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:17:24.382 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:17:24.382 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:17:24.382 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:24.383 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:24.383 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:24.383 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:17:24.383 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:17:24.383 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:24.383 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:17:24.383 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:17:24.383 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:17:24.383 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:24.383 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:17:24.383 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:17:24.383 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:24.383 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:17:24.383 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:17:24.384 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 5 2024-04-22T08:17:24.384 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:24.384 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:17:24.384 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:17:24.384 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:24.384 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:17:24.384 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:17:24.384 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 3 2024-04-22T08:17:24.384 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:24.384 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:24.384 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:24.384 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:17:24.384 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:17:24.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.361+0000 7fce68d8a640 1 -- 172.21.15.40:0/774353529 >> v2:172.21.15.189:6800/30168 conn(0x7fce44037820 msgr2=0x7fce44039ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:24.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.361+0000 7fce68d8a640 1 --2- 172.21.15.40:0/774353529 >> v2:172.21.15.189:6800/30168 conn(0x7fce44037820 0x7fce44039ce0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fce5c035000 tx=0x7fce5c008030 comp rx=0 tx=0).stop 2024-04-22T08:17:24.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.361+0000 7fce68d8a640 10 monclient: shutdown 2024-04-22T08:17:24.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.361+0000 7fce68d8a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:17:24.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.361+0000 7fce68d8a640 1 -- 172.21.15.40:0/774353529 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce640a1a80 msgr2=0x7fce641caf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:24.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.361+0000 7fce68d8a640 1 --2- 172.21.15.40:0/774353529 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce640a1a80 0x7fce641caf00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fce5400cf90 tx=0x7fce5400a7d0 comp rx=0 tx=0).stop 2024-04-22T08:17:24.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.361+0000 7fce62d9d640 1 -- 172.21.15.40:0/774353529 reap_dead start 2024-04-22T08:17:24.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.361+0000 7fce62d9d640 1 -- 172.21.15.40:0/774353529 reap_dead start 2024-04-22T08:17:24.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.361+0000 7fce68d8a640 1 -- 172.21.15.40:0/774353529 shutdown_connections 2024-04-22T08:17:24.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.361+0000 7fce68d8a640 1 -- 172.21.15.40:0/774353529 >> 172.21.15.40:0/774353529 conn(0x7fce640a66a0 msgr2=0x7fce640a3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:17:24.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.362+0000 7fce68d8a640 1 -- 172.21.15.40:0/774353529 shutdown_connections 2024-04-22T08:17:24.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.362+0000 7fce68d8a640 1 -- 172.21.15.40:0/774353529 wait complete. 2024-04-22T08:17:24.387 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:17:24.387 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-22T08:17:24.387 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-22T08:17:24.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.486+0000 7f185b737640 10 monclient: get_monmap_and_config 2024-04-22T08:17:24.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.486+0000 7f185b737640 10 monclient: build_initial_monmap 2024-04-22T08:17:24.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.486+0000 7f185b737640 10 monclient: monmap: 2024-04-22T08:17:24.487 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:17:24.487 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:24.487 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:17:24.488282+0000 2024-04-22T08:17:24.487 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:17:24.488282+0000 2024-04-22T08:17:24.487 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:17:24.487 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:24.487 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:17:24.487 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:17:24.488 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:17:24.488 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:17:24.488 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:17:24.488 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:24.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.487+0000 7f185b737640 1 Processor -- start 2024-04-22T08:17:24.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.487+0000 7f185b737640 1 -- start start 2024-04-22T08:17:24.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.487+0000 7f185b737640 10 monclient: init 2024-04-22T08:17:24.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.487+0000 7f185b737640 10 monclient: _reopen_session rank -1 2024-04-22T08:17:24.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.487+0000 7f185b737640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-04-22T08:17:24.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.487+0000 7f185b737640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f185412fad0 0x7f185412feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:24.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.487+0000 7f185b737640 10 monclient(hunting): picked mon.noname-a con 0x7f185412fad0 addr v2:172.21.15.40:3300/0 2024-04-22T08:17:24.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.487+0000 7f185b737640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1854132920 addr v1:172.21.15.189:6789/0 2024-04-22T08:17:24.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.487+0000 7f185b737640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1854133460 0x7f185413ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:24.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.487+0000 7f185b737640 10 monclient(hunting): picked mon.noname-c con 0x7f1854133460 addr v2:172.21.15.189:3300/0 2024-04-22T08:17:24.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.487+0000 7f185b737640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f185409ae50 con 0x7f1854132920 2024-04-22T08:17:24.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.487+0000 7f185b737640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:24.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.487+0000 7f185b737640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f185409efc0 con 0x7f185412fad0 2024-04-22T08:17:24.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.487+0000 7f185b737640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:24.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.487+0000 7f185b737640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f185409d070 con 0x7f1854133460 2024-04-22T08:17:24.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.487+0000 7f185b737640 10 monclient(hunting): _renew_subs 2024-04-22T08:17:24.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.487+0000 7f185b737640 10 monclient(hunting): authenticate will time out at 1118.482621s 2024-04-22T08:17:24.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.487+0000 7f18594d3640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f185412fad0 0x7f185412feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:24.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.487+0000 7f18594d3640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f185412fad0 0x7f185412feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60200/0 (socket says 172.21.15.40:60200) 2024-04-22T08:17:24.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.487+0000 7f18594d3640 1 -- 172.21.15.40:0/3173636949 learned_addr learned my addr 172.21.15.40:0/3173636949 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:17:24.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.487+0000 7f18594d3640 10 monclient(hunting): get_auth_request con 0x7f185412fad0 auth_method 0 2024-04-22T08:17:24.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.487+0000 7f18594d3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:24.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.487+0000 7f18594d3640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:24.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.487+0000 7f18594d3640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:24.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.488+0000 7f18594d3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:24.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.488+0000 7f18594d3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:24.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.488+0000 7f18594d3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:24.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.488+0000 7f1859cd4640 1 --2- 172.21.15.40:0/3173636949 >> v2:172.21.15.189:3300/0 conn(0x7f1854133460 0x7f185413ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:24.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.488+0000 7f1859cd4640 10 monclient(hunting): get_auth_request con 0x7f1854133460 auth_method 0 2024-04-22T08:17:24.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.488+0000 7f1859cd4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:24.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.488+0000 7f1859cd4640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:24.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.488+0000 7f1859cd4640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:24.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.488+0000 7f18594d3640 10 monclient(hunting): handle_auth_done global_id 34752 payload 274 2024-04-22T08:17:24.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.488+0000 7f18594d3640 1 -- 172.21.15.40:0/3173636949 >> v2:172.21.15.189:3300/0 conn(0x7f1854133460 msgr2=0x7f185413ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:17:24.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.488+0000 7f18594d3640 1 --2- 172.21.15.40:0/3173636949 >> v2:172.21.15.189:3300/0 conn(0x7f1854133460 0x7f185413ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:24.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.488+0000 7f18594d3640 1 -- 172.21.15.40:0/3173636949 >> v1:172.21.15.189:6789/0 conn(0x7f1854132920 legacy=0x7f1854132d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:17:24.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.488+0000 7f18594d3640 10 monclient: _finish_hunting 0 2024-04-22T08:17:24.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.488+0000 7f18594d3640 1 monclient: found mon.noname-a 2024-04-22T08:17:24.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.488+0000 7f18594d3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:17:24.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.488+0000 7f18594d3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:17:24.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.488+0000 7f18594d3640 1 -- 172.21.15.40:0/3173636949 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f185409d350 con 0x7f185412fad0 2024-04-22T08:17:24.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.491+0000 7f18594d3640 1 -- 172.21.15.40:0/3173636949 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f185412fad0 msgr2=0x7f185412feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:17:24.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.491+0000 7f18594d3640 1 --2- 172.21.15.40:0/3173636949 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f185412fad0 0x7f185412feb0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f18540a1e70 tx=0x7f185002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76843dab6dca57e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:24.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.491+0000 7f184bfff640 1 -- 172.21.15.40:0/3173636949 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1850031960 con 0x7f185412fad0 2024-04-22T08:17:24.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.491+0000 7f184bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:24.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.491+0000 7f184bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:17:24.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.491+0000 7f184bfff640 10 monclient: dump: 2024-04-22T08:17:24.499 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:24.499 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:24.500 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:24.500 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:24.500 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:24.500 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:24.500 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:24.500 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:24.500 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:24.500 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:24.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.491+0000 7f184bfff640 10 monclient: _finish_auth 0 2024-04-22T08:17:24.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.491+0000 7f184bfff640 10 monclient: _check_auth_tickets 2024-04-22T08:17:24.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.491+0000 7f184bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:17:24.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.491+0000 7f184bfff640 1 -- 172.21.15.40:0/3173636949 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1850031ac0 con 0x7f185412fad0 2024-04-22T08:17:24.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.491+0000 7f184bfff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:17:24.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.491+0000 7f184bfff640 1 -- 172.21.15.40:0/3173636949 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1850031e50 con 0x7f185412fad0 2024-04-22T08:17:24.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.491+0000 7f185b737640 5 monclient: authenticate success, global_id 34752 2024-04-22T08:17:24.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.491+0000 7f185b737640 10 monclient: get_monmap_and_config success 2024-04-22T08:17:24.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.491+0000 7f185b737640 4 set_mon_vals no callback set 2024-04-22T08:17:24.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.491+0000 7f184bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:24.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.491+0000 7f184bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:17:24.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.491+0000 7f184bfff640 10 monclient: dump: 2024-04-22T08:17:24.501 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:24.501 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:24.501 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:24.501 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:24.501 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:24.502 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:24.502 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:24.502 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:24.502 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:24.502 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:24.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.491+0000 7f185b737640 10 monclient: shutdown 2024-04-22T08:17:24.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.491+0000 7f185b737640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:17:24.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.491+0000 7f185b737640 1 -- 172.21.15.40:0/3173636949 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f185412fad0 msgr2=0x7f185412feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:24.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.491+0000 7f185b737640 1 --2- 172.21.15.40:0/3173636949 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f185412fad0 0x7f185412feb0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f18540a1e70 tx=0x7f185002b7f0 comp rx=0 tx=0).stop 2024-04-22T08:17:24.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.491+0000 7f185b737640 1 -- 172.21.15.40:0/3173636949 shutdown_connections 2024-04-22T08:17:24.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.491+0000 7f185b737640 1 --2- 172.21.15.40:0/3173636949 >> v2:172.21.15.189:3300/0 conn(0x7f1854133460 0x7f185413ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:24.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.491+0000 7f185b737640 1 --2- 172.21.15.40:0/3173636949 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f185412fad0 0x7f185412feb0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:24.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.491+0000 7f185b737640 1 -- 172.21.15.40:0/3173636949 >> 172.21.15.40:0/3173636949 conn(0x7f18540a0ae0 msgr2=0x7f18540a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:17:24.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.491+0000 7f185b737640 1 -- 172.21.15.40:0/3173636949 shutdown_connections 2024-04-22T08:17:24.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.491+0000 7f185b737640 1 -- 172.21.15.40:0/3173636949 wait complete. 2024-04-22T08:17:24.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.492+0000 7f185b737640 10 monclient: build_initial_monmap 2024-04-22T08:17:24.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.492+0000 7f185b737640 10 monclient: monmap: 2024-04-22T08:17:24.503 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:17:24.503 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:17:24.503 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:17:24.503 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:17:24.503 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:17:24.503 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:24.503 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:17:24.503 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:17:24.503 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:17:24.504 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:24.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.492+0000 7f185b737640 1 Processor -- start 2024-04-22T08:17:24.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.492+0000 7f185b737640 1 -- start start 2024-04-22T08:17:24.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.492+0000 7f185b737640 10 monclient: init 2024-04-22T08:17:24.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f185b737640 10 monclient: _reopen_session rank -1 2024-04-22T08:17:24.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f185b737640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-22T08:17:24.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f185b737640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f185412fad0 0x7f185409f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:24.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f185b737640 10 monclient(hunting): picked mon.noname-b con 0x7f185412fad0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:17:24.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f185b737640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1854132920 0x7f185412e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:24.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f185b737640 10 monclient(hunting): picked mon.noname-c con 0x7f1854132920 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:17:24.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f185b737640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1854133460 0x7f185412ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:24.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f185b737640 10 monclient(hunting): picked mon.noname-a con 0x7f1854133460 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:17:24.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f185b737640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:24.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f185b737640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1854122f10 con 0x7f1854133460 2024-04-22T08:17:24.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f185b737640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:24.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f185b737640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f185411c460 con 0x7f185412fad0 2024-04-22T08:17:24.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f185b737640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:24.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f185b737640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f185411c6a0 con 0x7f1854132920 2024-04-22T08:17:24.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f185b737640 10 monclient(hunting): _renew_subs 2024-04-22T08:17:24.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f185b737640 10 monclient(hunting): authenticate will time out at 1118.487976s 2024-04-22T08:17:24.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f1859cd4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1854133460 0x7f185412ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:24.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f1858cd2640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1854132920 0x7f185412e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:24.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f1859cd4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1854133460 0x7f185412ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60214/0 (socket says 172.21.15.40:60214) 2024-04-22T08:17:24.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f1859cd4640 1 -- 172.21.15.40:0/2340692708 learned_addr learned my addr 172.21.15.40:0/2340692708 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:17:24.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f1859cd4640 10 monclient(hunting): get_auth_request con 0x7f1854133460 auth_method 0 2024-04-22T08:17:24.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f1859cd4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:24.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f1859cd4640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:24.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f1859cd4640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:24.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f1858cd2640 10 monclient(hunting): get_auth_request con 0x7f1854132920 auth_method 0 2024-04-22T08:17:24.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f1858cd2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:24.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f1858cd2640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:24.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f1858cd2640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:24.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f1859cd4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:24.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f1859cd4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:24.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f18594d3640 1 --2- 172.21.15.40:0/2340692708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f185412fad0 0x7f185409f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:24.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f1859cd4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:24.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f18594d3640 10 monclient(hunting): get_auth_request con 0x7f185412fad0 auth_method 0 2024-04-22T08:17:24.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f18594d3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:24.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f18594d3640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:24.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f18594d3640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:24.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f1858cd2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:24.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f1858cd2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:24.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f1858cd2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:24.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f18594d3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:24.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f18594d3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:24.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f18594d3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:24.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f1859cd4640 10 monclient(hunting): handle_auth_done global_id 34755 payload 824 2024-04-22T08:17:24.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f1859cd4640 1 -- 172.21.15.40:0/2340692708 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1854132920 msgr2=0x7f185412e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:24.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f1859cd4640 1 --2- 172.21.15.40:0/2340692708 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1854132920 0x7f185412e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:24.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f1859cd4640 1 -- 172.21.15.40:0/2340692708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f185412fad0 msgr2=0x7f185409f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:24.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f1859cd4640 1 --2- 172.21.15.40:0/2340692708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f185412fad0 0x7f185409f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:24.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f1859cd4640 10 monclient: _finish_hunting 0 2024-04-22T08:17:24.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f1859cd4640 1 monclient: found mon.noname-a 2024-04-22T08:17:24.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f1859cd4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:17:24.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f1859cd4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:17:24.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.493+0000 7f1859cd4640 1 -- 172.21.15.40:0/2340692708 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18500094d0 con 0x7f1854133460 2024-04-22T08:17:24.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.495+0000 7f1859cd4640 1 -- 172.21.15.40:0/2340692708 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1854133460 msgr2=0x7f185412ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:17:24.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.495+0000 7f1859cd4640 1 --2- 172.21.15.40:0/2340692708 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1854133460 0x7f185412ec00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f184c007e70 tx=0x7f184c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:24.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.496+0000 7f1849ffb640 1 -- 172.21.15.40:0/2340692708 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f184c014620 con 0x7f1854133460 2024-04-22T08:17:24.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.496+0000 7f1849ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:24.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.496+0000 7f1849ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:17:24.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.496+0000 7f1849ffb640 10 monclient: dump: 2024-04-22T08:17:24.508 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:24.508 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:24.508 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:24.508 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:24.508 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:24.508 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:24.509 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:24.509 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:24.509 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:24.509 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:24.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.496+0000 7f1849ffb640 10 monclient: _finish_auth 0 2024-04-22T08:17:24.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.496+0000 7f1849ffb640 10 monclient: _check_auth_tickets 2024-04-22T08:17:24.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.496+0000 7f1849ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:17:24.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.496+0000 7f185b737640 5 monclient: authenticate success, global_id 34755 2024-04-22T08:17:24.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.496+0000 7f1849ffb640 1 -- 172.21.15.40:0/2340692708 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f184c0147a0 con 0x7f1854133460 2024-04-22T08:17:24.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.496+0000 7f1849ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:17:24.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.496+0000 7f1849ffb640 1 -- 172.21.15.40:0/2340692708 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f184c014b50 con 0x7f1854133460 2024-04-22T08:17:24.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.496+0000 7f185b737640 10 monclient: _renew_subs 2024-04-22T08:17:24.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.496+0000 7f185b737640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:17:24.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.496+0000 7f185b737640 1 -- 172.21.15.40:0/2340692708 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1854122590 con 0x7f1854133460 2024-04-22T08:17:24.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.496+0000 7f1849ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:24.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.496+0000 7f1849ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:17:24.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.496+0000 7f1849ffb640 10 monclient: dump: 2024-04-22T08:17:24.510 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:24.510 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:24.510 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:24.510 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:24.510 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:24.510 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:24.510 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:24.510 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:24.510 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:24.510 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:24.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.496+0000 7f184b7fe640 4 set_mon_vals no callback set 2024-04-22T08:17:24.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.496+0000 7f185b737640 10 monclient: _renew_subs 2024-04-22T08:17:24.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.496+0000 7f185b737640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:17:24.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.496+0000 7f185b737640 1 -- 172.21.15.40:0/2340692708 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f185412f140 con 0x7f1854133460 2024-04-22T08:17:24.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.496+0000 7f1849ffb640 1 -- 172.21.15.40:0/2340692708 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f184c014cb0 con 0x7f1854133460 2024-04-22T08:17:24.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.496+0000 7f185b737640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:17:24.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.497+0000 7f185b737640 1 --2- 172.21.15.40:0/2340692708 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f185411d4e0 0x7f18541cb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:24.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.497+0000 7f185b737640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:24.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.497+0000 7f185b737640 1 -- 172.21.15.40:0/2340692708 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18540667a0 con 0x7f185411d4e0 2024-04-22T08:17:24.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.497+0000 7f18594d3640 1 --2- 172.21.15.40:0/2340692708 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f185411d4e0 0x7f18541cb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:24.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.497+0000 7f18594d3640 10 monclient: get_auth_request con 0x7f185411d4e0 auth_method 0 2024-04-22T08:17:24.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.497+0000 7f18594d3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:24.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.497+0000 7f18594d3640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:24.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.497+0000 7f18594d3640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:24.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.497+0000 7f1849ffb640 1 --2- 172.21.15.40:0/2340692708 >> v2:172.21.15.189:6800/30168 conn(0x7f18380377d0 0x7f1838039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:24.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.497+0000 7f18594d3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:24.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.497+0000 7f18594d3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:24.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.497+0000 7f1849ffb640 1 -- 172.21.15.40:0/2340692708 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f184c04ad50 con 0x7f1854133460 2024-04-22T08:17:24.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.497+0000 7f18594d3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:24.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.497+0000 7f18594d3640 10 monclient(hunting): handle_auth_done global_id 54308 payload 824 2024-04-22T08:17:24.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.497+0000 7f18594d3640 1 -- 172.21.15.40:0/2340692708 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f185409e2f0 con 0x7f185411d4e0 2024-04-22T08:17:24.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.497+0000 7f1858cd2640 1 --2- 172.21.15.40:0/2340692708 >> v2:172.21.15.189:6800/30168 conn(0x7f18380377d0 0x7f1838039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:24.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.497+0000 7f1858cd2640 10 monclient: get_auth_request con 0x7f18380377d0 auth_method 0 2024-04-22T08:17:24.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.499+0000 7f18594d3640 1 -- 172.21.15.40:0/2340692708 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f185411d4e0 msgr2=0x7f18541cb0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:17:24.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.499+0000 7f18594d3640 1 --2- 172.21.15.40:0/2340692708 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f185411d4e0 0x7f18541cb0e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1850035000 tx=0x7f18500046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:24.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.499+0000 7f1849ffb640 1 -- 172.21.15.40:0/2340692708 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185003d640 con 0x7f185411d4e0 2024-04-22T08:17:24.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.499+0000 7f1849ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:24.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.499+0000 7f1849ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:17:24.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.499+0000 7f1849ffb640 10 monclient: dump: 2024-04-22T08:17:24.515 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:24.515 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:24.515 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:24.515 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:24.515 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:24.515 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:24.515 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:24.516 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:24.516 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:24.516 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:24.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.499+0000 7f1858cd2640 1 --2- 172.21.15.40:0/2340692708 >> v2:172.21.15.189:6800/30168 conn(0x7f18380377d0 0x7f1838039c90 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f18540a0150 tx=0x7f1844007e30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:24.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.500+0000 7f1849ffb640 1 -- 172.21.15.40:0/2340692708 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f1850041070 con 0x7f185411d4e0 2024-04-22T08:17:24.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.500+0000 7f1849ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:17:24.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.500+0000 7f1849ffb640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:17:24.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.500+0000 7f1849ffb640 1 -- 172.21.15.40:0/2340692708 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f185411d4e0 msgr2=0x7f18541cb0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:24.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.500+0000 7f1849ffb640 1 --2- 172.21.15.40:0/2340692708 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f185411d4e0 0x7f18541cb0e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1850035000 tx=0x7f18500046c0 comp rx=0 tx=0).stop 2024-04-22T08:17:24.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.500+0000 7f1849ffb640 1 -- 172.21.15.40:0/2340692708 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f185411d4e0 msgr2=0x7f18541cb0e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:17:24.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.500+0000 7f1849ffb640 1 --2- 172.21.15.40:0/2340692708 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f185411d4e0 0x7f18541cb0e0 secure :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1850035000 tx=0x7f18500046c0 comp rx=0 tx=0).stop 2024-04-22T08:17:24.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.506+0000 7f185b737640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:17:24.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.506+0000 7f185b737640 1 --2- 172.21.15.40:0/2340692708 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f185402b200 0x7f185402b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:24.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.506+0000 7f185b737640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:24.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.506+0000 7f185b737640 1 -- 172.21.15.40:0/2340692708 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f185404e9b0 con 0x7f185402b200 2024-04-22T08:17:24.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.506+0000 7f18594d3640 1 --2- 172.21.15.40:0/2340692708 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f185402b200 0x7f185402b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:24.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.506+0000 7f18594d3640 10 monclient: get_auth_request con 0x7f185402b200 auth_method 0 2024-04-22T08:17:24.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.506+0000 7f18594d3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:24.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.506+0000 7f18594d3640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:24.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.506+0000 7f18594d3640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:24.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.506+0000 7f18594d3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:24.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.506+0000 7f18594d3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:24.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.506+0000 7f18594d3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:24.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.506+0000 7f18594d3640 10 monclient(hunting): handle_auth_done global_id 54311 payload 824 2024-04-22T08:17:24.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.506+0000 7f18594d3640 1 -- 172.21.15.40:0/2340692708 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f185406fe40 con 0x7f185402b200 2024-04-22T08:17:24.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.508+0000 7f18594d3640 1 -- 172.21.15.40:0/2340692708 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f185402b200 msgr2=0x7f185402b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:17:24.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.508+0000 7f18594d3640 1 --2- 172.21.15.40:0/2340692708 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f185402b200 0x7f185402b5e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1850032040 tx=0x7f1850009240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:24.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.509+0000 7f1849ffb640 1 -- 172.21.15.40:0/2340692708 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1850047af0 con 0x7f185402b200 2024-04-22T08:17:24.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.509+0000 7f1849ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:24.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.509+0000 7f1849ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:17:24.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.509+0000 7f1849ffb640 10 monclient: dump: 2024-04-22T08:17:24.518 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:24.518 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:24.518 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:24.518 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:24.518 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:24.518 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:24.518 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:24.518 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:24.518 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:24.518 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:24.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.509+0000 7f1849ffb640 1 -- 172.21.15.40:0/2340692708 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4801 (secure 0 0 0) 0x7f1850047c50 con 0x7f185402b200 2024-04-22T08:17:24.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.509+0000 7f1849ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:17:24.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.509+0000 7f1849ffb640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:17:24.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.509+0000 7f1849ffb640 1 -- 172.21.15.40:0/2340692708 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f185402b200 msgr2=0x7f185402b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:24.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.509+0000 7f1849ffb640 1 --2- 172.21.15.40:0/2340692708 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f185402b200 0x7f185402b5e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1850032040 tx=0x7f1850009240 comp rx=0 tx=0).stop 2024-04-22T08:17:24.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.509+0000 7f1849ffb640 1 -- 172.21.15.40:0/2340692708 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f185402b200 msgr2=0x7f185402b5e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:17:24.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.509+0000 7f1849ffb640 1 --2- 172.21.15.40:0/2340692708 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f185402b200 0x7f185402b5e0 secure :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1850032040 tx=0x7f1850009240 comp rx=0 tx=0).stop 2024-04-22T08:17:24.519 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:17:24.519 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:17:24.519 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:17:24.520 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:17:24.520 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 46, 2024-04-22T08:17:24.520 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:17:24.520 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:17:24.520 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:17:24.520 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:17:24.520 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:17:24.520 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 0, 2024-04-22T08:17:24.520 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:17:24.520 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:17:24.520 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:17:24.520 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:17:24.520 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:17:24.520 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:17:24.521 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:17:24.521 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:17:24.521 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:17:24.521 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:17:24.521 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:17:24.521 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:17:24.521 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:17:24.521 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:17:24.521 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:17:24.521 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:17:24.521 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:17:24.521 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:17:24.521 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:17:24.521 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:17:24.522 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:17:24.522 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:17:24.522 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:17:24.522 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:17:24.522 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:17:24.522 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:17:24.522 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:24.522 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:24.522 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:17:24.522 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:17:24.522 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:17:24.522 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:17:24.522 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:17:24.523 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:17:24.523 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:17:24.523 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:17:24.523 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:17:24.523 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:17:24.523 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:17:24.523 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:17:24.523 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:17:24.523 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:17:24.523 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:17:24.523 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:17:24.523 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:17:24.523 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:17:24.523 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:17:24.523 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:17:24.524 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:17:24.524 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:17:24.524 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:17:24.524 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:17:24.524 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:17:24.524 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:17:24.524 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:17:24.524 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:17:24.524 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:17:24.524 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:24.524 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:17:24.524 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:24.524 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:17:24.524 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:17:24.524 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:17:24.524 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:17:24.525 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:24.525 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:17:24.525 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:17:24.525 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:24.525 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:24.525 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:24.525 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:17:24.525 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:17:24.525 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:24.525 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:24.525 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:24.525 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:24.525 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:17:24.525 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:17:24.526 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:17:24.526 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:17:24.526 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:17:24.526 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:24.526 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:24.526 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:17:24.526 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:17:24.526 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:17:24.526 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:17:24.526 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:24.526 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:17:24.526 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:17:24.526 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:24.527 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:24.527 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:24.527 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:17:24.527 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:17:24.527 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:24.527 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:24.527 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:24.527 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:24.527 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:17:24.527 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:17:24.527 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:17:24.527 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:17:24.527 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:17:24.527 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:24.528 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:24.528 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:17:24.528 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:17:24.528 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:17:24.528 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:17:24.528 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:24.528 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:17:24.528 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:17:24.528 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:24.528 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:24.528 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:24.528 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:17:24.528 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:17:24.528 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:24.528 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:24.529 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:24.529 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:24.529 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:17:24.529 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:17:24.529 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:17:24.529 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:17:24.529 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:17:24.529 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:24.529 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:24.529 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:24.529 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:17:24.529 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:17:24.529 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:24.529 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:17:24.529 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:17:24.530 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:17:24.530 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:24.530 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:17:24.530 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:17:24.530 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:24.530 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:17:24.530 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:17:24.530 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:17:24.530 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:24.530 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:17:24.530 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:17:24.530 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:24.530 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:17:24.531 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:17:24.531 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 4 2024-04-22T08:17:24.531 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:24.531 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:17:24.531 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": [ 2024-04-22T08:17:24.531 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:24.531 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:17:24.531 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:17:24.531 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:17:24.531 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:24.531 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:24.531 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:24.531 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:17:24.531 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:17:24.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.509+0000 7f185b737640 1 -- 172.21.15.40:0/2340692708 >> v2:172.21.15.189:6800/30168 conn(0x7f18380377d0 msgr2=0x7f1838039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:24.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.509+0000 7f185b737640 1 --2- 172.21.15.40:0/2340692708 >> v2:172.21.15.189:6800/30168 conn(0x7f18380377d0 0x7f1838039c90 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f18540a0150 tx=0x7f1844007e30 comp rx=0 tx=0).stop 2024-04-22T08:17:24.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.509+0000 7f185b737640 10 monclient: shutdown 2024-04-22T08:17:24.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.509+0000 7f185b737640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:17:24.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.509+0000 7f185b737640 1 -- 172.21.15.40:0/2340692708 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1854133460 msgr2=0x7f185412ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:24.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.509+0000 7f185b737640 1 --2- 172.21.15.40:0/2340692708 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1854133460 0x7f185412ec00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f184c007e70 tx=0x7f184c00e4b0 comp rx=0 tx=0).stop 2024-04-22T08:17:24.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.509+0000 7f1859cd4640 1 -- 172.21.15.40:0/2340692708 reap_dead start 2024-04-22T08:17:24.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.509+0000 7f1859cd4640 1 -- 172.21.15.40:0/2340692708 reap_dead start 2024-04-22T08:17:24.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.509+0000 7f185b737640 1 -- 172.21.15.40:0/2340692708 shutdown_connections 2024-04-22T08:17:24.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.509+0000 7f185b737640 1 -- 172.21.15.40:0/2340692708 >> 172.21.15.40:0/2340692708 conn(0x7f18540a0ae0 msgr2=0x7f18540a3640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:17:24.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.509+0000 7f185b737640 1 -- 172.21.15.40:0/2340692708 shutdown_connections 2024-04-22T08:17:24.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.509+0000 7f185b737640 1 -- 172.21.15.40:0/2340692708 wait complete. 2024-04-22T08:17:24.534 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:17:24.534 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-22T08:17:24.534 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-22T08:17:24.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.633+0000 7fd45bcf4640 10 monclient: get_monmap_and_config 2024-04-22T08:17:24.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.633+0000 7fd45bcf4640 10 monclient: build_initial_monmap 2024-04-22T08:17:24.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.633+0000 7fd45bcf4640 10 monclient: monmap: 2024-04-22T08:17:24.634 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:17:24.634 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:24.634 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:17:24.635340+0000 2024-04-22T08:17:24.634 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:17:24.635340+0000 2024-04-22T08:17:24.634 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:17:24.634 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:24.635 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:17:24.635 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:17:24.635 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:17:24.635 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:17:24.635 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:17:24.635 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:24.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.634+0000 7fd45bcf4640 1 Processor -- start 2024-04-22T08:17:24.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.634+0000 7fd45bcf4640 1 -- start start 2024-04-22T08:17:24.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.634+0000 7fd45bcf4640 10 monclient: init 2024-04-22T08:17:24.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.634+0000 7fd45bcf4640 10 monclient: _reopen_session rank -1 2024-04-22T08:17:24.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.634+0000 7fd45bcf4640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-04-22T08:17:24.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.634+0000 7fd45bcf4640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd4540736d0 0x7fd454135f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:24.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.634+0000 7fd45bcf4640 10 monclient(hunting): picked mon.noname-a con 0x7fd4540736d0 addr v2:172.21.15.40:3300/0 2024-04-22T08:17:24.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.634+0000 7fd45bcf4640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd454070eb0 0x7fd454071320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:24.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.634+0000 7fd45bcf4640 10 monclient(hunting): picked mon.noname-c con 0x7fd454070eb0 addr v2:172.21.15.189:3300/0 2024-04-22T08:17:24.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.634+0000 7fd45bcf4640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd45409de00 0x7fd45409e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:24.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.634+0000 7fd45bcf4640 10 monclient(hunting): picked mon.noname-b con 0x7fd45409de00 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:17:24.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.634+0000 7fd45bcf4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:24.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.634+0000 7fd45bcf4640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd454071860 con 0x7fd4540736d0 2024-04-22T08:17:24.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.634+0000 7fd45bcf4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:24.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.634+0000 7fd45bcf4640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd454073ab0 con 0x7fd454070eb0 2024-04-22T08:17:24.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.634+0000 7fd45bcf4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:24.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.634+0000 7fd45bcf4640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd454136510 con 0x7fd45409de00 2024-04-22T08:17:24.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.634+0000 7fd45bcf4640 10 monclient(hunting): _renew_subs 2024-04-22T08:17:24.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.634+0000 7fd45bcf4640 10 monclient(hunting): authenticate will time out at 1118.629571s 2024-04-22T08:17:24.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.634+0000 7fd45a291640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd45409de00 0x7fd45409e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:24.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.634+0000 7fd459a90640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd4540736d0 0x7fd454135f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:24.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.634+0000 7fd45a291640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd45409de00 0x7fd45409e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:52544/0 (socket says 172.21.15.40:52544) 2024-04-22T08:17:24.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.634+0000 7fd459a90640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd4540736d0 0x7fd454135f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60224/0 (socket says 172.21.15.40:60224) 2024-04-22T08:17:24.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.634+0000 7fd45a291640 1 -- 172.21.15.40:0/4018219723 learned_addr learned my addr 172.21.15.40:0/4018219723 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:17:24.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.634+0000 7fd45a291640 10 monclient(hunting): get_auth_request con 0x7fd45409de00 auth_method 0 2024-04-22T08:17:24.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.634+0000 7fd45a291640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:24.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.634+0000 7fd45a291640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:24.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.634+0000 7fd45a291640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:24.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.634+0000 7fd459a90640 10 monclient(hunting): get_auth_request con 0x7fd4540736d0 auth_method 0 2024-04-22T08:17:24.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.634+0000 7fd459a90640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:24.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.634+0000 7fd459a90640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:24.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.634+0000 7fd459a90640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:24.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.635+0000 7fd45a291640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:24.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.635+0000 7fd45a291640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:24.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.635+0000 7fd45a291640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:24.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.635+0000 7fd459a90640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:24.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.635+0000 7fd459a90640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:24.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.635+0000 7fd459a90640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:24.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.635+0000 7fd45928f640 1 --2- 172.21.15.40:0/4018219723 >> v2:172.21.15.189:3300/0 conn(0x7fd454070eb0 0x7fd454071320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:24.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.635+0000 7fd45928f640 10 monclient(hunting): get_auth_request con 0x7fd454070eb0 auth_method 0 2024-04-22T08:17:24.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.635+0000 7fd45928f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:24.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.635+0000 7fd45928f640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:24.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.635+0000 7fd45928f640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:24.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.635+0000 7fd45a291640 10 monclient(hunting): handle_auth_done global_id 54314 payload 274 2024-04-22T08:17:24.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.635+0000 7fd45a291640 1 -- 172.21.15.40:0/4018219723 >> v2:172.21.15.189:3300/0 conn(0x7fd454070eb0 msgr2=0x7fd454071320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:17:24.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.635+0000 7fd45a291640 1 --2- 172.21.15.40:0/4018219723 >> v2:172.21.15.189:3300/0 conn(0x7fd454070eb0 0x7fd454071320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:24.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.635+0000 7fd45a291640 1 -- 172.21.15.40:0/4018219723 >> v2:172.21.15.40:3300/0 conn(0x7fd4540736d0 msgr2=0x7fd454135f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:17:24.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.635+0000 7fd45a291640 1 --2- 172.21.15.40:0/4018219723 >> v2:172.21.15.40:3300/0 conn(0x7fd4540736d0 0x7fd454135f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:24.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.635+0000 7fd45a291640 10 monclient: _finish_hunting 0 2024-04-22T08:17:24.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.635+0000 7fd45a291640 1 monclient: found mon.noname-b 2024-04-22T08:17:24.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.635+0000 7fd45a291640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:17:24.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.635+0000 7fd45a291640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:17:24.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.635+0000 7fd45a291640 1 -- 172.21.15.40:0/4018219723 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd45406ce90 con 0x7fd45409de00 2024-04-22T08:17:24.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.635+0000 7fd459a90640 1 --2- 172.21.15.40:0/4018219723 >> v2:172.21.15.40:3300/0 conn(0x7fd4540736d0 0x7fd454135f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:17:24.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.635+0000 7fd45928f640 1 --2- 172.21.15.40:0/4018219723 >> v2:172.21.15.189:3300/0 conn(0x7fd454070eb0 0x7fd454071320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T08:17:24.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.638+0000 7fd45a291640 1 -- 172.21.15.40:0/4018219723 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd45409de00 msgr2=0x7fd45409e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:17:24.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.638+0000 7fd45a291640 1 --2- 172.21.15.40:0/4018219723 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd45409de00 0x7fd45409e290 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd454073f30 tx=0x7fd44c02bdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=739b05f13edc0b23 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:24.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.638+0000 7fd458a8e640 1 -- 172.21.15.40:0/4018219723 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd44c02cb60 con 0x7fd45409de00 2024-04-22T08:17:24.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.638+0000 7fd458a8e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:24.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.638+0000 7fd458a8e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:17:24.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.638+0000 7fd458a8e640 10 monclient: dump: 2024-04-22T08:17:24.651 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:24.651 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:24.651 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:24.651 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:24.651 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:24.652 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:24.652 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:24.652 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:24.652 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:24.652 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:24.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.638+0000 7fd458a8e640 10 monclient: _finish_auth 0 2024-04-22T08:17:24.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.638+0000 7fd458a8e640 10 monclient: _check_auth_tickets 2024-04-22T08:17:24.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.638+0000 7fd458a8e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:17:24.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.638+0000 7fd458a8e640 1 -- 172.21.15.40:0/4018219723 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd44c02ccc0 con 0x7fd45409de00 2024-04-22T08:17:24.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.638+0000 7fd458a8e640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:17:24.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.638+0000 7fd458a8e640 1 -- 172.21.15.40:0/4018219723 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd44c034610 con 0x7fd45409de00 2024-04-22T08:17:24.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.638+0000 7fd45bcf4640 5 monclient: authenticate success, global_id 54314 2024-04-22T08:17:24.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.638+0000 7fd45bcf4640 10 monclient: get_monmap_and_config success 2024-04-22T08:17:24.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.638+0000 7fd458a8e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:24.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.638+0000 7fd45bcf4640 4 set_mon_vals no callback set 2024-04-22T08:17:24.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.638+0000 7fd458a8e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:17:24.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.638+0000 7fd458a8e640 10 monclient: dump: 2024-04-22T08:17:24.653 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:24.653 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:24.653 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:24.653 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:24.653 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:24.653 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:24.653 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:24.653 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:24.653 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:24.653 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:24.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.638+0000 7fd45bcf4640 10 monclient: shutdown 2024-04-22T08:17:24.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.638+0000 7fd45bcf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:17:24.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.638+0000 7fd45bcf4640 1 -- 172.21.15.40:0/4018219723 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd45409de00 msgr2=0x7fd45409e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:24.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.638+0000 7fd45bcf4640 1 --2- 172.21.15.40:0/4018219723 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd45409de00 0x7fd45409e290 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd454073f30 tx=0x7fd44c02bdb0 comp rx=0 tx=0).stop 2024-04-22T08:17:24.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.638+0000 7fd45bcf4640 1 -- 172.21.15.40:0/4018219723 shutdown_connections 2024-04-22T08:17:24.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.638+0000 7fd45bcf4640 1 --2- 172.21.15.40:0/4018219723 >> v2:172.21.15.189:3300/0 conn(0x7fd454070eb0 0x7fd454071320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:24.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.638+0000 7fd45bcf4640 1 --2- 172.21.15.40:0/4018219723 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd45409de00 0x7fd45409e290 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:24.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.638+0000 7fd45bcf4640 1 --2- 172.21.15.40:0/4018219723 >> v2:172.21.15.40:3300/0 conn(0x7fd4540736d0 0x7fd454135f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:24.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.638+0000 7fd45bcf4640 1 -- 172.21.15.40:0/4018219723 >> 172.21.15.40:0/4018219723 conn(0x7fd4540a1780 msgr2=0x7fd4540a1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:17:24.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.638+0000 7fd45bcf4640 1 -- 172.21.15.40:0/4018219723 shutdown_connections 2024-04-22T08:17:24.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.638+0000 7fd45bcf4640 1 -- 172.21.15.40:0/4018219723 wait complete. 2024-04-22T08:17:24.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.639+0000 7fd45bcf4640 10 monclient: build_initial_monmap 2024-04-22T08:17:24.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.639+0000 7fd45bcf4640 10 monclient: monmap: 2024-04-22T08:17:24.654 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:17:24.654 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:17:24.654 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:17:24.655 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:17:24.655 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:17:24.655 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:24.655 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:17:24.655 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:17:24.655 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:17:24.655 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:24.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.639+0000 7fd45bcf4640 1 Processor -- start 2024-04-22T08:17:24.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.639+0000 7fd45bcf4640 1 -- start start 2024-04-22T08:17:24.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.639+0000 7fd45bcf4640 10 monclient: init 2024-04-22T08:17:24.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd45bcf4640 10 monclient: _reopen_session rank -1 2024-04-22T08:17:24.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd45bcf4640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-22T08:17:24.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd45bcf4640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd454070eb0 0x7fd4541ca9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:24.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd45bcf4640 10 monclient(hunting): picked mon.noname-c con 0x7fd454070eb0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:17:24.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd45bcf4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4540736d0 0x7fd4541caf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:24.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd45bcf4640 10 monclient(hunting): picked mon.noname-a con 0x7fd4540736d0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:17:24.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd45bcf4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd45409de00 0x7fd45411c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:24.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd45bcf4640 10 monclient(hunting): picked mon.noname-b con 0x7fd45409de00 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:17:24.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd45bcf4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:24.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd45bcf4640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd454135140 con 0x7fd4540736d0 2024-04-22T08:17:24.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd45bcf4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:24.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd45bcf4640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd45411e810 con 0x7fd45409de00 2024-04-22T08:17:24.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd45bcf4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:24.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd45bcf4640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4540a4090 con 0x7fd454070eb0 2024-04-22T08:17:24.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd45bcf4640 10 monclient(hunting): _renew_subs 2024-04-22T08:17:24.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd45bcf4640 10 monclient(hunting): authenticate will time out at 1118.635119s 2024-04-22T08:17:24.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd459a90640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd454070eb0 0x7fd4541ca9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:24.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd459a90640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd454070eb0 0x7fd4541ca9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:52560/0 (socket says 172.21.15.40:52560) 2024-04-22T08:17:24.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd459a90640 1 -- 172.21.15.40:0/2391807886 learned_addr learned my addr 172.21.15.40:0/2391807886 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:17:24.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd459a90640 10 monclient(hunting): get_auth_request con 0x7fd454070eb0 auth_method 0 2024-04-22T08:17:24.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd459a90640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:24.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd459a90640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:24.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd459a90640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:24.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd45928f640 1 --2- 172.21.15.40:0/2391807886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4540736d0 0x7fd4541caf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:24.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd45928f640 10 monclient(hunting): get_auth_request con 0x7fd4540736d0 auth_method 0 2024-04-22T08:17:24.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd45928f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:24.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd45928f640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:24.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd45928f640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:24.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd459a90640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:24.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd459a90640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:24.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd459a90640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:24.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd45928f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:24.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd45928f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:24.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd45928f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:24.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd45a291640 1 --2- 172.21.15.40:0/2391807886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd45409de00 0x7fd45411c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:24.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd459a90640 10 monclient(hunting): handle_auth_done global_id 54317 payload 824 2024-04-22T08:17:24.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd459a90640 1 -- 172.21.15.40:0/2391807886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd45409de00 msgr2=0x7fd45411c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:24.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd459a90640 1 --2- 172.21.15.40:0/2391807886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd45409de00 0x7fd45411c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:24.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd459a90640 1 -- 172.21.15.40:0/2391807886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4540736d0 msgr2=0x7fd4541caf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:24.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd459a90640 1 --2- 172.21.15.40:0/2391807886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4540736d0 0x7fd4541caf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:24.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd459a90640 10 monclient: _finish_hunting 0 2024-04-22T08:17:24.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd459a90640 1 monclient: found mon.noname-c 2024-04-22T08:17:24.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd459a90640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:17:24.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd459a90640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:17:24.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd459a90640 1 -- 172.21.15.40:0/2391807886 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd450009400 con 0x7fd454070eb0 2024-04-22T08:17:24.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd45a291640 10 monclient: get_auth_request con 0x7fd45409de00 auth_method 0 2024-04-22T08:17:24.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd45a291640 1 --2- 172.21.15.40:0/2391807886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd45409de00 0x7fd45411c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T08:17:24.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.640+0000 7fd45928f640 1 --2- 172.21.15.40:0/2391807886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4540736d0 0x7fd4541caf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:17:24.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.642+0000 7fd459a90640 1 -- 172.21.15.40:0/2391807886 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd454070eb0 msgr2=0x7fd4541ca9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:17:24.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.642+0000 7fd459a90640 1 --2- 172.21.15.40:0/2391807886 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd454070eb0 0x7fd4541ca9e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd45000ec10 tx=0x7fd45000c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:24.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.643+0000 7fd44a7fc640 1 -- 172.21.15.40:0/2391807886 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd450009760 con 0x7fd454070eb0 2024-04-22T08:17:24.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.643+0000 7fd44a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:24.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.643+0000 7fd44a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:17:24.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.643+0000 7fd44a7fc640 10 monclient: dump: 2024-04-22T08:17:24.659 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:24.659 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:24.659 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:24.659 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:24.659 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:24.659 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:24.659 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:24.659 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:24.660 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:24.660 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:24.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.643+0000 7fd44a7fc640 10 monclient: _finish_auth 0 2024-04-22T08:17:24.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.643+0000 7fd44a7fc640 10 monclient: _check_auth_tickets 2024-04-22T08:17:24.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.643+0000 7fd44a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:17:24.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.643+0000 7fd44a7fc640 1 -- 172.21.15.40:0/2391807886 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd45000ce40 con 0x7fd454070eb0 2024-04-22T08:17:24.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.643+0000 7fd44a7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:17:24.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.643+0000 7fd44a7fc640 1 -- 172.21.15.40:0/2391807886 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd450013690 con 0x7fd454070eb0 2024-04-22T08:17:24.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.643+0000 7fd45bcf4640 5 monclient: authenticate success, global_id 54317 2024-04-22T08:17:24.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.643+0000 7fd45bcf4640 10 monclient: _renew_subs 2024-04-22T08:17:24.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.643+0000 7fd45bcf4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:17:24.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.643+0000 7fd45bcf4640 1 -- 172.21.15.40:0/2391807886 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd44c009b30 con 0x7fd454070eb0 2024-04-22T08:17:24.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.643+0000 7fd44bfff640 4 set_mon_vals no callback set 2024-04-22T08:17:24.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.643+0000 7fd44a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:24.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.643+0000 7fd44a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:17:24.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.643+0000 7fd44a7fc640 10 monclient: dump: 2024-04-22T08:17:24.661 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:24.661 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:24.661 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:24.661 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:24.661 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:24.661 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:24.661 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:24.661 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:24.661 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:24.661 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:24.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.643+0000 7fd45bcf4640 10 monclient: _renew_subs 2024-04-22T08:17:24.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.643+0000 7fd45bcf4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:17:24.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.643+0000 7fd45bcf4640 1 -- 172.21.15.40:0/2391807886 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd45411eeb0 con 0x7fd454070eb0 2024-04-22T08:17:24.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.643+0000 7fd45bcf4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:17:24.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.643+0000 7fd44a7fc640 1 -- 172.21.15.40:0/2391807886 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fd4500137f0 con 0x7fd454070eb0 2024-04-22T08:17:24.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.644+0000 7fd45bcf4640 1 --2- 172.21.15.40:0/2391807886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd45411d3d0 0x7fd45402b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:24.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.644+0000 7fd45bcf4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:24.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.644+0000 7fd45bcf4640 1 -- 172.21.15.40:0/2391807886 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd454066900 con 0x7fd45411d3d0 2024-04-22T08:17:24.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.644+0000 7fd44a7fc640 1 --2- 172.21.15.40:0/2391807886 >> v2:172.21.15.189:6800/30168 conn(0x7fd438037820 0x7fd438039ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:24.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.644+0000 7fd44a7fc640 1 -- 172.21.15.40:0/2391807886 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 3898+0+0 (secure 0 0 0) 0x7fd45004aa40 con 0x7fd454070eb0 2024-04-22T08:17:24.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.644+0000 7fd45928f640 1 --2- 172.21.15.40:0/2391807886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd45411d3d0 0x7fd45402b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:24.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.644+0000 7fd45928f640 10 monclient: get_auth_request con 0x7fd45411d3d0 auth_method 0 2024-04-22T08:17:24.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.644+0000 7fd45928f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:24.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.644+0000 7fd45928f640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:24.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.644+0000 7fd45928f640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:24.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.644+0000 7fd45a291640 1 --2- 172.21.15.40:0/2391807886 >> v2:172.21.15.189:6800/30168 conn(0x7fd438037820 0x7fd438039ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:24.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.644+0000 7fd45a291640 10 monclient: get_auth_request con 0x7fd438037820 auth_method 0 2024-04-22T08:17:24.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.644+0000 7fd45928f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:24.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.644+0000 7fd45928f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:24.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.644+0000 7fd45928f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:24.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.644+0000 7fd45928f640 10 monclient(hunting): handle_auth_done global_id 64111 payload 824 2024-04-22T08:17:24.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.644+0000 7fd45928f640 1 -- 172.21.15.40:0/2391807886 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd4540a3980 con 0x7fd45411d3d0 2024-04-22T08:17:24.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.647+0000 7fd45928f640 1 -- 172.21.15.40:0/2391807886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd45411d3d0 msgr2=0x7fd45402b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:17:24.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.647+0000 7fd45928f640 1 --2- 172.21.15.40:0/2391807886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd45411d3d0 0x7fd45402b200 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd444006fd0 tx=0x7fd444003e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:24.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.647+0000 7fd45a291640 1 --2- 172.21.15.40:0/2391807886 >> v2:172.21.15.189:6800/30168 conn(0x7fd438037820 0x7fd438039ce0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd44c032000 tx=0x7fd44c030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:24.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.647+0000 7fd44a7fc640 1 -- 172.21.15.40:0/2391807886 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd444010c90 con 0x7fd45411d3d0 2024-04-22T08:17:24.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.647+0000 7fd44a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:24.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.647+0000 7fd44a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:17:24.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.647+0000 7fd44a7fc640 10 monclient: dump: 2024-04-22T08:17:24.664 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:24.664 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:24.664 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:24.665 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:24.665 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:24.665 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:24.665 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:24.665 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:24.665 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:24.665 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:24.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.647+0000 7fd44a7fc640 1 -- 172.21.15.40:0/2391807886 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7fd444010df0 con 0x7fd45411d3d0 2024-04-22T08:17:24.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.647+0000 7fd44a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:17:24.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.647+0000 7fd44a7fc640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:17:24.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.647+0000 7fd44a7fc640 1 -- 172.21.15.40:0/2391807886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd45411d3d0 msgr2=0x7fd45402b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:24.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.647+0000 7fd44a7fc640 1 --2- 172.21.15.40:0/2391807886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd45411d3d0 0x7fd45402b200 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd444006fd0 tx=0x7fd444003e10 comp rx=0 tx=0).stop 2024-04-22T08:17:24.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.647+0000 7fd44a7fc640 1 -- 172.21.15.40:0/2391807886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd45411d3d0 msgr2=0x7fd45402b200 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:17:24.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.647+0000 7fd44a7fc640 1 --2- 172.21.15.40:0/2391807886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd45411d3d0 0x7fd45402b200 secure :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd444006fd0 tx=0x7fd444003e10 comp rx=0 tx=0).stop 2024-04-22T08:17:24.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.654+0000 7fd45bcf4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:17:24.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.654+0000 7fd45bcf4640 1 --2- 172.21.15.40:0/2391807886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd45411d7b0 0x7fd45402b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:24.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.654+0000 7fd45bcf4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:24.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.654+0000 7fd45bcf4640 1 -- 172.21.15.40:0/2391807886 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd45406d450 con 0x7fd45411d7b0 2024-04-22T08:17:24.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.654+0000 7fd45928f640 1 --2- 172.21.15.40:0/2391807886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd45411d7b0 0x7fd45402b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:24.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.654+0000 7fd45928f640 10 monclient: get_auth_request con 0x7fd45411d7b0 auth_method 0 2024-04-22T08:17:24.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.654+0000 7fd45928f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:24.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.654+0000 7fd45928f640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:24.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.654+0000 7fd45928f640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:24.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.654+0000 7fd45928f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:24.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.654+0000 7fd45928f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:24.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.654+0000 7fd45928f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:24.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.654+0000 7fd45928f640 10 monclient(hunting): handle_auth_done global_id 64114 payload 824 2024-04-22T08:17:24.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.654+0000 7fd45928f640 1 -- 172.21.15.40:0/2391807886 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd45409e7d0 con 0x7fd45411d7b0 2024-04-22T08:17:24.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.657+0000 7fd45928f640 1 -- 172.21.15.40:0/2391807886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd45411d7b0 msgr2=0x7fd45402b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:17:24.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.657+0000 7fd45928f640 1 --2- 172.21.15.40:0/2391807886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd45411d7b0 0x7fd45402b740 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd444008010 tx=0x7fd444004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:24.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.657+0000 7fd44a7fc640 1 -- 172.21.15.40:0/2391807886 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd444022040 con 0x7fd45411d7b0 2024-04-22T08:17:24.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.657+0000 7fd44a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:24.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.657+0000 7fd44a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:17:24.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.657+0000 7fd44a7fc640 10 monclient: dump: 2024-04-22T08:17:24.669 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:24.669 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:24.669 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:24.669 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:24.669 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:24.669 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:24.669 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:24.669 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:24.669 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:24.669 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:24.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.657+0000 7fd44a7fc640 1 -- 172.21.15.40:0/2391807886 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4461 (secure 0 0 0) 0x7fd444014a50 con 0x7fd45411d7b0 2024-04-22T08:17:24.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.657+0000 7fd44a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:17:24.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.657+0000 7fd44a7fc640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:17:24.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.657+0000 7fd44a7fc640 1 -- 172.21.15.40:0/2391807886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd45411d7b0 msgr2=0x7fd45402b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:24.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.657+0000 7fd44a7fc640 1 --2- 172.21.15.40:0/2391807886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd45411d7b0 0x7fd45402b740 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd444008010 tx=0x7fd444004730 comp rx=0 tx=0).stop 2024-04-22T08:17:24.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.657+0000 7fd44a7fc640 1 -- 172.21.15.40:0/2391807886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd45411d7b0 msgr2=0x7fd45402b740 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:17:24.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.657+0000 7fd44a7fc640 1 --2- 172.21.15.40:0/2391807886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd45411d7b0 0x7fd45402b740 secure :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd444008010 tx=0x7fd444004730 comp rx=0 tx=0).stop 2024-04-22T08:17:24.670 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:17:24.670 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:17:24.670 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:17:24.670 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:17:24.670 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 46, 2024-04-22T08:17:24.670 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:17:24.670 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:17:24.670 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:17:24.671 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:17:24.671 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:17:24.671 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 0, 2024-04-22T08:17:24.671 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:17:24.671 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:17:24.671 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:17:24.671 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:17:24.671 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:17:24.671 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:17:24.671 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:17:24.671 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:17:24.671 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:17:24.671 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:17:24.671 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:17:24.671 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:17:24.672 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:17:24.672 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:17:24.672 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:17:24.672 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:17:24.672 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:17:24.672 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:17:24.672 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:17:24.672 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:17:24.672 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:17:24.672 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:17:24.672 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:17:24.672 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:17:24.672 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:17:24.672 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:17:24.672 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:24.673 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:24.673 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:17:24.673 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:17:24.673 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:17:24.673 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:17:24.673 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:17:24.673 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:17:24.673 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:17:24.673 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:17:24.673 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:17:24.673 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:17:24.673 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:17:24.673 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:17:24.673 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:17:24.674 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:17:24.674 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:17:24.674 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:17:24.674 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:17:24.674 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:17:24.674 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:17:24.674 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:17:24.674 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:17:24.674 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:17:24.674 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:17:24.674 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:17:24.674 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:17:24.674 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:17:24.674 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:17:24.674 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:17:24.675 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:17:24.675 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:24.675 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:17:24.675 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:24.675 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:17:24.675 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:17:24.675 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:17:24.675 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:17:24.675 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:24.675 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:17:24.675 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:17:24.675 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:24.675 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:24.675 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:24.676 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:17:24.676 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:17:24.676 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:24.676 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:24.676 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:24.676 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:24.676 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:17:24.676 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:17:24.676 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:17:24.676 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:17:24.676 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:17:24.676 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:24.676 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:24.676 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:17:24.676 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:17:24.677 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:17:24.677 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:17:24.677 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:24.677 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:17:24.677 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:17:24.677 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:24.677 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:24.677 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:24.677 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:17:24.677 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:17:24.677 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:24.677 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:24.677 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:24.677 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:24.677 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:17:24.678 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:17:24.678 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:17:24.678 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:17:24.678 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:17:24.678 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:24.678 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:24.678 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:17:24.678 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:17:24.678 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:17:24.678 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:17:24.678 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:24.678 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:17:24.678 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:17:24.678 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:24.679 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:24.679 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:24.679 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:17:24.679 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:17:24.679 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:24.679 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:24.679 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:24.679 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:24.679 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:17:24.679 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:17:24.679 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:17:24.679 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:17:24.679 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:17:24.679 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:24.680 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:24.680 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:24.680 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:17:24.680 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:17:24.680 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:24.680 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:17:24.680 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:17:24.680 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:17:24.680 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:24.680 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:17:24.680 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:17:24.680 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:24.680 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:17:24.681 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:17:24.681 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:17:24.681 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:24.681 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:24.681 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:24.681 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:17:24.681 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:17:24.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.658+0000 7fd45bcf4640 1 -- 172.21.15.40:0/2391807886 >> v2:172.21.15.189:6800/30168 conn(0x7fd438037820 msgr2=0x7fd438039ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:24.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.658+0000 7fd45bcf4640 1 --2- 172.21.15.40:0/2391807886 >> v2:172.21.15.189:6800/30168 conn(0x7fd438037820 0x7fd438039ce0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd44c032000 tx=0x7fd44c030000 comp rx=0 tx=0).stop 2024-04-22T08:17:24.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.658+0000 7fd45bcf4640 10 monclient: shutdown 2024-04-22T08:17:24.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.658+0000 7fd45bcf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:17:24.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.658+0000 7fd45bcf4640 1 -- 172.21.15.40:0/2391807886 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd454070eb0 msgr2=0x7fd4541ca9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:24.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.658+0000 7fd45bcf4640 1 --2- 172.21.15.40:0/2391807886 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd454070eb0 0x7fd4541ca9e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd45000ec10 tx=0x7fd45000c520 comp rx=0 tx=0).stop 2024-04-22T08:17:24.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.658+0000 7fd45a291640 1 -- 172.21.15.40:0/2391807886 reap_dead start 2024-04-22T08:17:24.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.658+0000 7fd45bcf4640 1 -- 172.21.15.40:0/2391807886 shutdown_connections 2024-04-22T08:17:24.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.658+0000 7fd45bcf4640 1 --2- 172.21.15.40:0/2391807886 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd454070eb0 0x7fd4541ca9e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:24.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.658+0000 7fd45bcf4640 1 -- 172.21.15.40:0/2391807886 >> 172.21.15.40:0/2391807886 conn(0x7fd4540a1780 msgr2=0x7fd454133060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:17:24.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.658+0000 7fd45bcf4640 1 -- 172.21.15.40:0/2391807886 shutdown_connections 2024-04-22T08:17:24.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.658+0000 7fd45bcf4640 1 -- 172.21.15.40:0/2391807886 wait complete. 2024-04-22T08:17:24.683 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:17:24.684 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-22T08:17:24.684 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-22T08:17:24.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.781+0000 7fdd2e821640 10 monclient: get_monmap_and_config 2024-04-22T08:17:24.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.781+0000 7fdd2e821640 10 monclient: build_initial_monmap 2024-04-22T08:17:24.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.781+0000 7fdd2e821640 10 monclient: monmap: 2024-04-22T08:17:24.782 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:17:24.782 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:24.782 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:17:24.783563+0000 2024-04-22T08:17:24.782 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:17:24.783563+0000 2024-04-22T08:17:24.782 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:17:24.782 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:24.782 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:17:24.783 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:17:24.783 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:17:24.783 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:17:24.783 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:17:24.783 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:24.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.782+0000 7fdd2e821640 1 Processor -- start 2024-04-22T08:17:24.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.782+0000 7fdd2e821640 1 -- start start 2024-04-22T08:17:24.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.782+0000 7fdd2e821640 10 monclient: init 2024-04-22T08:17:24.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.782+0000 7fdd2e821640 10 monclient: _reopen_session rank -1 2024-04-22T08:17:24.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.782+0000 7fdd2e821640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-04-22T08:17:24.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.782+0000 7fdd2e821640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdd28135e30 0x7fdd28136210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:24.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.783+0000 7fdd2e821640 10 monclient(hunting): picked mon.noname-c con 0x7fdd28135e30 addr v2:172.21.15.189:3300/0 2024-04-22T08:17:24.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.783+0000 7fdd2e821640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdd28071c90 addr v1:172.21.15.189:6789/0 2024-04-22T08:17:24.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.783+0000 7fdd2e821640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdd28136750 addr v1:172.21.15.40:6789/0 2024-04-22T08:17:24.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.783+0000 7fdd2e821640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd280a27c0 con 0x7fdd28136750 2024-04-22T08:17:24.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.783+0000 7fdd2e821640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd2809e310 con 0x7fdd28071c90 2024-04-22T08:17:24.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.783+0000 7fdd2e821640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:24.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.783+0000 7fdd2e821640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd2809de80 con 0x7fdd28135e30 2024-04-22T08:17:24.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.783+0000 7fdd2e821640 10 monclient(hunting): _renew_subs 2024-04-22T08:17:24.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.783+0000 7fdd2e821640 10 monclient(hunting): authenticate will time out at 1118.777879s 2024-04-22T08:17:24.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.783+0000 7fdd2cdbe640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fdd28136750 0x7fdd280a20b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:33318/0 (socket says 172.21.15.40:33318) 2024-04-22T08:17:24.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.783+0000 7fdd2cdbe640 1 -- 172.21.15.40:0/2919072242 learned_addr learned my addr 172.21.15.40:0/2919072242 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:17:24.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.783+0000 7fdd27fff640 1 --2- 172.21.15.40:0/2919072242 >> v2:172.21.15.189:3300/0 conn(0x7fdd28135e30 0x7fdd28136210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:24.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.783+0000 7fdd27fff640 10 monclient(hunting): get_auth_request con 0x7fdd28135e30 auth_method 0 2024-04-22T08:17:24.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.783+0000 7fdd27fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:24.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.783+0000 7fdd27fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:24.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.783+0000 7fdd27fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:24.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.783+0000 7fdd27fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:24.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.783+0000 7fdd27fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:24.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.783+0000 7fdd27fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:24.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.785+0000 7fdd2cdbe640 1 -- 172.21.15.40:0/2919072242 >> v1:172.21.15.40:6789/0 conn(0x7fdd28136750 legacy=0x7fdd280a20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:17:24.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.785+0000 7fdd26ffd640 1 -- 172.21.15.40:0/2919072242 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1117089165 0 0) 0x7fdd280a27c0 con 0x7fdd28136750 2024-04-22T08:17:24.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.785+0000 7fdd26ffd640 10 monclient(hunting): _negotiate 2024-04-22T08:17:24.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.785+0000 7fdd26ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:24.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.785+0000 7fdd26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:24.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.785+0000 7fdd26ffd640 10 monclient(hunting): my global_id is 34764 2024-04-22T08:17:24.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.785+0000 7fdd26ffd640 1 -- 172.21.15.40:0/2919072242 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd14003520 con 0x7fdd28136750 2024-04-22T08:17:24.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.785+0000 7fdd277fe640 1 -- 172.21.15.40:0/2919072242 >> v1:172.21.15.189:6789/0 conn(0x7fdd28071c90 legacy=0x7fdd280a19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:17:24.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.785+0000 7fdd26ffd640 1 -- 172.21.15.40:0/2919072242 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4217080400 0 0) 0x7fdd14003520 con 0x7fdd28136750 2024-04-22T08:17:24.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.785+0000 7fdd26ffd640 1 -- 172.21.15.40:0/2919072242 >> v2:172.21.15.189:3300/0 conn(0x7fdd28135e30 msgr2=0x7fdd28136210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:17:24.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.785+0000 7fdd26ffd640 1 --2- 172.21.15.40:0/2919072242 >> v2:172.21.15.189:3300/0 conn(0x7fdd28135e30 0x7fdd28136210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:24.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.785+0000 7fdd26ffd640 1 -- 172.21.15.40:0/2919072242 >> v1:172.21.15.189:6789/0 conn(0x7fdd28071c90 legacy=0x7fdd280a19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:24.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.785+0000 7fdd26ffd640 10 monclient: _finish_hunting 0 2024-04-22T08:17:24.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.785+0000 7fdd26ffd640 1 monclient: found mon.noname-a-legacy 2024-04-22T08:17:24.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.785+0000 7fdd26ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:17:24.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.785+0000 7fdd26ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.40:6789/0 2024-04-22T08:17:24.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.785+0000 7fdd26ffd640 1 -- 172.21.15.40:0/2919072242 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd2809f130 con 0x7fdd28136750 2024-04-22T08:17:24.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.785+0000 7fdd26ffd640 10 monclient: _finish_auth 0 2024-04-22T08:17:24.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.785+0000 7fdd26ffd640 10 monclient: _check_auth_tickets 2024-04-22T08:17:24.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.785+0000 7fdd26ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:17:24.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.785+0000 7fdd2e821640 5 monclient: authenticate success, global_id 34764 2024-04-22T08:17:24.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.785+0000 7fdd26ffd640 1 -- 172.21.15.40:0/2919072242 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1087006497 0 0) 0x7fdd20001ca0 con 0x7fdd28136750 2024-04-22T08:17:24.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.785+0000 7fdd2e821640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:17:24.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.785+0000 7fdd26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:24.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.785+0000 7fdd26ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-22T08:17:24.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.785+0000 7fdd26ffd640 10 monclient: dump: 2024-04-22T08:17:24.795 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:24.795 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:24.795 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:24.795 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:24.795 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:24.795 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:24.795 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:24.795 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:24.795 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:24.795 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:24.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.785+0000 7fdd26ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] but i'm connected to v1:172.21.15.40:6789/0, reconnecting 2024-04-22T08:17:24.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.785+0000 7fdd26ffd640 10 monclient: _reopen_session rank -1 2024-04-22T08:17:24.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.785+0000 7fdd26ffd640 1 -- 172.21.15.40:0/2919072242 >> v1:172.21.15.40:6789/0 conn(0x7fdd28136750 legacy=0x7fdd280a20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:24.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.785+0000 7fdd26ffd640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-22T08:17:24.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.785+0000 7fdd26ffd640 1 --2- 172.21.15.40:0/2919072242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd14003a40 0x7fdd14003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:24.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.785+0000 7fdd26ffd640 10 monclient(hunting): picked mon.a con 0x7fdd14003a40 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:17:24.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.785+0000 7fdd26ffd640 1 --2- 172.21.15.40:0/2919072242 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdd140047c0 0x7fdd14004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:24.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.785+0000 7fdd26ffd640 10 monclient(hunting): picked mon.c con 0x7fdd140047c0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:17:24.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.785+0000 7fdd26ffd640 1 --2- 172.21.15.40:0/2919072242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd14005380 0x7fdd1400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:24.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.785+0000 7fdd26ffd640 10 monclient(hunting): picked mon.b con 0x7fdd14005380 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:17:24.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.785+0000 7fdd26ffd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:24.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.785+0000 7fdd26ffd640 1 -- 172.21.15.40:0/2919072242 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2809de80 con 0x7fdd14003a40 2024-04-22T08:17:24.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.785+0000 7fdd26ffd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:24.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.785+0000 7fdd26ffd640 1 -- 172.21.15.40:0/2919072242 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1400c220 con 0x7fdd14005380 2024-04-22T08:17:24.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.785+0000 7fdd26ffd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:24.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.785+0000 7fdd26ffd640 1 -- 172.21.15.40:0/2919072242 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1400c3e0 con 0x7fdd140047c0 2024-04-22T08:17:24.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.785+0000 7fdd26ffd640 10 monclient(hunting): _renew_subs 2024-04-22T08:17:24.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.785+0000 7fdd26ffd640 10 monclient(hunting): _finish_auth 0 2024-04-22T08:17:24.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.785+0000 7fdd2e821640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-04-22T08:17:24.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.785+0000 7fdd27fff640 1 --2- 172.21.15.40:0/2919072242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd14003a40 0x7fdd14003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:24.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.786+0000 7fdd27fff640 10 monclient(hunting): get_auth_request con 0x7fdd14003a40 auth_method 0 2024-04-22T08:17:24.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.786+0000 7fdd27fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:24.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.786+0000 7fdd27fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:24.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.786+0000 7fdd27fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:17:24.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.786+0000 7fdd277fe640 1 --2- 172.21.15.40:0/2919072242 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdd140047c0 0x7fdd14004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:24.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.786+0000 7fdd277fe640 10 monclient(hunting): get_auth_request con 0x7fdd140047c0 auth_method 0 2024-04-22T08:17:24.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.786+0000 7fdd277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:24.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.786+0000 7fdd277fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:24.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.786+0000 7fdd277fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:17:24.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.786+0000 7fdd27fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:24.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.786+0000 7fdd27fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:24.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.786+0000 7fdd27fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-22T08:17:24.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.786+0000 7fdd277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:24.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.786+0000 7fdd2cdbe640 1 --2- 172.21.15.40:0/2919072242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd14005380 0x7fdd1400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:24.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.786+0000 7fdd277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:24.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.786+0000 7fdd277fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-22T08:17:24.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.786+0000 7fdd2cdbe640 10 monclient(hunting): get_auth_request con 0x7fdd14005380 auth_method 0 2024-04-22T08:17:24.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.786+0000 7fdd2cdbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:24.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.786+0000 7fdd2cdbe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:24.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.786+0000 7fdd2cdbe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:17:24.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.786+0000 7fdd27fff640 10 monclient(hunting): handle_auth_done global_id 34764 payload 293 2024-04-22T08:17:24.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.786+0000 7fdd2cdbe640 1 -- 172.21.15.40:0/2919072242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd14005380 msgr2=0x7fdd1400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-22T08:17:24.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.786+0000 7fdd2cdbe640 1 -- 172.21.15.40:0/2919072242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd14005380 msgr2=0x7fdd1400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:17:24.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.786+0000 7fdd2cdbe640 1 --2- 172.21.15.40:0/2919072242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd14005380 0x7fdd1400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:17:24.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.786+0000 7fdd27fff640 1 -- 172.21.15.40:0/2919072242 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdd140047c0 msgr2=0x7fdd14004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:17:24.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.786+0000 7fdd27fff640 1 --2- 172.21.15.40:0/2919072242 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdd140047c0 0x7fdd14004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:24.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.786+0000 7fdd2cdbe640 1 --2- 172.21.15.40:0/2919072242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd14005380 0x7fdd1400b980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:17:24.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.786+0000 7fdd27fff640 1 -- 172.21.15.40:0/2919072242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd14005380 msgr2=0x7fdd1400b980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:17:24.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.786+0000 7fdd27fff640 1 --2- 172.21.15.40:0/2919072242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd14005380 0x7fdd1400b980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:24.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.786+0000 7fdd27fff640 10 monclient: _finish_hunting 0 2024-04-22T08:17:24.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.786+0000 7fdd27fff640 1 monclient: found mon.a 2024-04-22T08:17:24.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.786+0000 7fdd27fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:17:24.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.786+0000 7fdd27fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:17:24.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.786+0000 7fdd27fff640 1 -- 172.21.15.40:0/2919072242 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd14003520 con 0x7fdd14003a40 2024-04-22T08:17:24.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.786+0000 7fdd27fff640 10 monclient: _finish_auth 0 2024-04-22T08:17:24.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.786+0000 7fdd27fff640 10 monclient: _check_auth_tickets 2024-04-22T08:17:24.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.786+0000 7fdd27fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:17:24.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.786+0000 7fdd2cdbe640 1 -- 172.21.15.40:0/2919072242 reap_dead start 2024-04-22T08:17:24.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.789+0000 7fdd27fff640 1 -- 172.21.15.40:0/2919072242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd14003a40 msgr2=0x7fdd14003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:17:24.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.789+0000 7fdd27fff640 1 --2- 172.21.15.40:0/2919072242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd14003a40 0x7fdd14003e60 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fdd10009ac0 tx=0x7fdd140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5dd69959f0b7193 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:24.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.789+0000 7fdd26ffd640 1 -- 172.21.15.40:0/2919072242 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1002c8c0 con 0x7fdd14003a40 2024-04-22T08:17:24.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.789+0000 7fdd26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:24.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.789+0000 7fdd26ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:17:24.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.789+0000 7fdd26ffd640 10 monclient: dump: 2024-04-22T08:17:24.800 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:24.800 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:24.800 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:24.800 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:24.800 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:24.800 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:24.800 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:24.800 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:24.801 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:24.801 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:24.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.789+0000 7fdd2e821640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:17:24.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.789+0000 7fdd26ffd640 1 -- 172.21.15.40:0/2919072242 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd1002ca20 con 0x7fdd14003a40 2024-04-22T08:17:24.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.789+0000 7fdd26ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:17:24.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.789+0000 7fdd26ffd640 1 -- 172.21.15.40:0/2919072242 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1002cdb0 con 0x7fdd14003a40 2024-04-22T08:17:24.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.789+0000 7fdd2e821640 10 monclient: get_monmap_and_config success 2024-04-22T08:17:24.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.789+0000 7fdd2e821640 4 set_mon_vals no callback set 2024-04-22T08:17:24.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.789+0000 7fdd26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:24.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.789+0000 7fdd26ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:17:24.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.789+0000 7fdd26ffd640 10 monclient: dump: 2024-04-22T08:17:24.801 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:24.801 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:24.801 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:24.801 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:24.802 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:24.802 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:24.802 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:24.802 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:24.802 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:24.802 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:24.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.789+0000 7fdd2e821640 10 monclient: shutdown 2024-04-22T08:17:24.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.789+0000 7fdd2e821640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:17:24.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.789+0000 7fdd2e821640 1 -- 172.21.15.40:0/2919072242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd14003a40 msgr2=0x7fdd14003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:24.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.789+0000 7fdd2e821640 1 --2- 172.21.15.40:0/2919072242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd14003a40 0x7fdd14003e60 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fdd10009ac0 tx=0x7fdd140057f0 comp rx=0 tx=0).stop 2024-04-22T08:17:24.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.789+0000 7fdd2e821640 1 -- 172.21.15.40:0/2919072242 shutdown_connections 2024-04-22T08:17:24.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.789+0000 7fdd2e821640 1 --2- 172.21.15.40:0/2919072242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd14003a40 0x7fdd14003e60 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:24.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.789+0000 7fdd2e821640 1 -- 172.21.15.40:0/2919072242 >> 172.21.15.40:0/2919072242 conn(0x7fdd280a65c0 msgr2=0x7fdd280a2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:17:24.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.789+0000 7fdd2e821640 1 -- 172.21.15.40:0/2919072242 shutdown_connections 2024-04-22T08:17:24.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.789+0000 7fdd2e821640 1 -- 172.21.15.40:0/2919072242 wait complete. 2024-04-22T08:17:24.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.790+0000 7fdd2e821640 10 monclient: build_initial_monmap 2024-04-22T08:17:24.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.790+0000 7fdd2e821640 10 monclient: monmap: 2024-04-22T08:17:24.803 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:17:24.803 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:17:24.803 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:17:24.803 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:17:24.803 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:17:24.803 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:24.803 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:17:24.803 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:17:24.803 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:17:24.804 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:24.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.790+0000 7fdd2e821640 1 Processor -- start 2024-04-22T08:17:24.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd2e821640 1 -- start start 2024-04-22T08:17:24.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd2e821640 10 monclient: init 2024-04-22T08:17:24.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd2e821640 10 monclient: _reopen_session rank -1 2024-04-22T08:17:24.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd2e821640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-22T08:17:24.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd2e821640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdd14003a40 0x7fdd281ca4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:24.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd2e821640 10 monclient(hunting): picked mon.noname-c con 0x7fdd14003a40 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:17:24.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd2e821640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd28133bb0 0x7fdd28133ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:24.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd2e821640 10 monclient(hunting): picked mon.noname-b con 0x7fdd28133bb0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:17:24.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd2e821640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd2806e830 0x7fdd2806ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:24.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd2e821640 10 monclient(hunting): picked mon.noname-a con 0x7fdd2806e830 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:17:24.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd2e821640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:24.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd2e821640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd28120fd0 con 0x7fdd2806e830 2024-04-22T08:17:24.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd2e821640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:24.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd2e821640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd281ca9e0 con 0x7fdd28133bb0 2024-04-22T08:17:24.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd2e821640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:24.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd2e821640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd28070e80 con 0x7fdd14003a40 2024-04-22T08:17:24.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd2e821640 10 monclient(hunting): _renew_subs 2024-04-22T08:17:24.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd2e821640 10 monclient(hunting): authenticate will time out at 1118.786266s 2024-04-22T08:17:24.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd27fff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdd14003a40 0x7fdd281ca4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:24.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd27fff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdd14003a40 0x7fdd281ca4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:52566/0 (socket says 172.21.15.40:52566) 2024-04-22T08:17:24.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd2cdbe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd2806e830 0x7fdd2806ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:24.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd27fff640 1 -- 172.21.15.40:0/3993727180 learned_addr learned my addr 172.21.15.40:0/3993727180 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:17:24.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd27fff640 10 monclient(hunting): get_auth_request con 0x7fdd14003a40 auth_method 0 2024-04-22T08:17:24.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd27fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:24.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd27fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:24.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd27fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:24.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd2cdbe640 10 monclient(hunting): get_auth_request con 0x7fdd2806e830 auth_method 0 2024-04-22T08:17:24.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd2cdbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:24.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd2cdbe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:24.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd2cdbe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:24.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd27fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:24.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd27fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:24.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd27fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:24.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd2cdbe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:24.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd2cdbe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:24.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd2cdbe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:24.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd277fe640 1 --2- 172.21.15.40:0/3993727180 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd28133bb0 0x7fdd28133ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:24.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd277fe640 10 monclient(hunting): get_auth_request con 0x7fdd28133bb0 auth_method 0 2024-04-22T08:17:24.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:24.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd277fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:24.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd277fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:24.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd27fff640 10 monclient(hunting): handle_auth_done global_id 54320 payload 824 2024-04-22T08:17:24.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd27fff640 1 -- 172.21.15.40:0/3993727180 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd28133bb0 msgr2=0x7fdd28133ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:24.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd27fff640 1 --2- 172.21.15.40:0/3993727180 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd28133bb0 0x7fdd28133ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:24.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd27fff640 1 -- 172.21.15.40:0/3993727180 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd2806e830 msgr2=0x7fdd2806ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:24.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd27fff640 1 --2- 172.21.15.40:0/3993727180 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd2806e830 0x7fdd2806ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:24.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd27fff640 10 monclient: _finish_hunting 0 2024-04-22T08:17:24.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd27fff640 1 monclient: found mon.noname-c 2024-04-22T08:17:24.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd27fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:17:24.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd27fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:17:24.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd27fff640 1 -- 172.21.15.40:0/3993727180 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd28120cf0 con 0x7fdd14003a40 2024-04-22T08:17:24.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.791+0000 7fdd2cdbe640 1 --2- 172.21.15.40:0/3993727180 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd2806e830 0x7fdd2806ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:17:24.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.794+0000 7fdd27fff640 1 -- 172.21.15.40:0/3993727180 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdd14003a40 msgr2=0x7fdd281ca4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:17:24.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.794+0000 7fdd27fff640 1 --2- 172.21.15.40:0/3993727180 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdd14003a40 0x7fdd281ca4a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fdd1002c250 tx=0x7fdd10004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:24.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.794+0000 7fdd24ff9640 1 -- 172.21.15.40:0/3993727180 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd10003a20 con 0x7fdd14003a40 2024-04-22T08:17:24.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.794+0000 7fdd24ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:24.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.794+0000 7fdd24ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:17:24.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.794+0000 7fdd24ff9640 10 monclient: dump: 2024-04-22T08:17:24.810 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:24.810 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:24.810 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:24.810 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:24.810 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:24.810 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:24.810 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:24.810 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:24.811 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:24.811 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:24.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.794+0000 7fdd24ff9640 10 monclient: _finish_auth 0 2024-04-22T08:17:24.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.794+0000 7fdd24ff9640 10 monclient: _check_auth_tickets 2024-04-22T08:17:24.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.794+0000 7fdd24ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:17:24.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.794+0000 7fdd24ff9640 1 -- 172.21.15.40:0/3993727180 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd10003b80 con 0x7fdd14003a40 2024-04-22T08:17:24.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.794+0000 7fdd2e821640 5 monclient: authenticate success, global_id 54320 2024-04-22T08:17:24.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.794+0000 7fdd2e821640 10 monclient: _renew_subs 2024-04-22T08:17:24.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.794+0000 7fdd2e821640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:17:24.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.794+0000 7fdd2e821640 1 -- 172.21.15.40:0/3993727180 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd28134ff0 con 0x7fdd14003a40 2024-04-22T08:17:24.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.794+0000 7fdd24ff9640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:17:24.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.794+0000 7fdd24ff9640 1 -- 172.21.15.40:0/3993727180 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd10004490 con 0x7fdd14003a40 2024-04-22T08:17:24.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.794+0000 7fdd24ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:24.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.794+0000 7fdd24ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:17:24.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.794+0000 7fdd24ff9640 10 monclient: dump: 2024-04-22T08:17:24.812 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:24.812 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:24.812 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:24.812 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:24.812 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:24.812 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:24.812 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:24.812 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:24.812 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:24.812 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:24.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.794+0000 7fdd25ffb640 4 set_mon_vals no callback set 2024-04-22T08:17:24.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.794+0000 7fdd2e821640 10 monclient: _renew_subs 2024-04-22T08:17:24.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.794+0000 7fdd2e821640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:17:24.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.794+0000 7fdd2e821640 1 -- 172.21.15.40:0/3993727180 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd28121590 con 0x7fdd14003a40 2024-04-22T08:17:24.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.795+0000 7fdd24ff9640 1 -- 172.21.15.40:0/3993727180 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fdd10034440 con 0x7fdd14003a40 2024-04-22T08:17:24.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.795+0000 7fdd2e821640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:17:24.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.795+0000 7fdd2e821640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:17:24.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.795+0000 7fdd2e821640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:17:24.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.795+0000 7fdd24ff9640 1 --2- 172.21.15.40:0/3993727180 >> v2:172.21.15.189:6800/30168 conn(0x7fdd08037830 0x7fdd08039cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:24.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.795+0000 7fdd2e821640 1 -- 172.21.15.40:0/3993727180 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd2804e9b0 con 0x7fdd14003a40 2024-04-22T08:17:24.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.795+0000 7fdd24ff9640 1 -- 172.21.15.40:0/3993727180 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 3898+0+0 (secure 0 0 0) 0x7fdd1006c4e0 con 0x7fdd14003a40 2024-04-22T08:17:24.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.795+0000 7fdd277fe640 1 --2- 172.21.15.40:0/3993727180 >> v2:172.21.15.189:6800/30168 conn(0x7fdd08037830 0x7fdd08039cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:24.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.795+0000 7fdd277fe640 10 monclient: get_auth_request con 0x7fdd08037830 auth_method 0 2024-04-22T08:17:24.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.798+0000 7fdd277fe640 1 --2- 172.21.15.40:0/3993727180 >> v2:172.21.15.189:6800/30168 conn(0x7fdd08037830 0x7fdd08039cf0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fdd28070080 tx=0x7fdd18009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:24.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.800+0000 7fdd24ff9640 1 -- 172.21.15.40:0/3993727180 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7fdd10039080 con 0x7fdd14003a40 2024-04-22T08:17:24.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.800+0000 7fdd24ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:17:24.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:24.800+0000 7fdd24ff9640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:17:24.865 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:17:24.866 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[36861]: starting. 2024-04-22T08:17:24.866 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[36861]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-36825 2024-04-22T08:17:24.866 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[36861]: shutting down. 2024-04-22T08:17:24.866 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:17:24.866 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:17:24.867 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:17:24.869 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:17:25.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:25.055+0000 7fdd2e821640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-22T08:17:25.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:25.055+0000 7fdd2e821640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-22T08:17:25.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:25.055+0000 7fdd2e821640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:17:25.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:25.055+0000 7fdd2e821640 1 -- 172.21.15.40:0/3993727180 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fdd281209f0 con 0x7fdd14003a40 2024-04-22T08:17:25.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:25.056+0000 7fdd24ff9640 1 -- 172.21.15.40:0/3993727180 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fdd100463b0 con 0x7fdd14003a40 2024-04-22T08:17:25.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:25.056+0000 7fdd24ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-22T08:17:25.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:25.056+0000 7fdd24ff9640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:17:25.058 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.056+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:17:25.058 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.056+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=13,config=2,health=11,kv=9,logm=65} crc {auth=2454428106,config=1618278320,health=539703498,kv=1350343662,logm=3956149705}) 2024-04-22T08:17:25.058 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.056+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=11,config=2,health=10,kv=9,logm=68} crc {auth=470980228,config=1618278320,health=3042652494,kv=2725287440,logm=912061218}) 2024-04-22T08:17:25.058 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.056+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:17:25.058 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.056+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=13,config=2,health=11,kv=9,logm=65} crc {auth=2454428106,config=1618278320,health=539703498,kv=1350343662,logm=3956149705}) 2024-04-22T08:17:25.058 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.056+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=13,config=2,health=12,kv=8,logm=65} crc {auth=2454428106,config=1618278320,health=4015482905,kv=1244876340,logm=3060887320}) 2024-04-22T08:17:25.058 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.057+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:17:25.059 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.057+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3298529552}) 2024-04-22T08:17:25.059 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.057+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=484489044}) 2024-04-22T08:17:25.059 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.057+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:17:25.059 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.057+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3298529552}) 2024-04-22T08:17:25.059 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.057+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3171397217}) 2024-04-22T08:17:25.060 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.058+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:17:25.060 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.058+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3128157348}) 2024-04-22T08:17:25.060 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.058+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2209476368}) 2024-04-22T08:17:25.060 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.058+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:17:25.060 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.058+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3128157348}) 2024-04-22T08:17:25.060 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.058+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1596997100}) 2024-04-22T08:17:25.060 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.059+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:17:25.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:25.059+0000 7fdd2e821640 1 -- 172.21.15.40:0/3993727180 >> v2:172.21.15.189:6800/30168 conn(0x7fdd08037830 msgr2=0x7fdd08039cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:25.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:25.059+0000 7fdd2e821640 1 --2- 172.21.15.40:0/3993727180 >> v2:172.21.15.189:6800/30168 conn(0x7fdd08037830 0x7fdd08039cf0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fdd28070080 tx=0x7fdd18009040 comp rx=0 tx=0).stop 2024-04-22T08:17:25.061 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.059+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4178071152}) 2024-04-22T08:17:25.061 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.059+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=251370664}) 2024-04-22T08:17:25.061 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.059+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:17:25.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:25.059+0000 7fdd2e821640 10 monclient: shutdown 2024-04-22T08:17:25.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:25.059+0000 7fdd2e821640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:17:25.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:25.059+0000 7fdd2e821640 1 -- 172.21.15.40:0/3993727180 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdd14003a40 msgr2=0x7fdd281ca4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:25.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:25.059+0000 7fdd2e821640 1 --2- 172.21.15.40:0/3993727180 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdd14003a40 0x7fdd281ca4a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fdd1002c250 tx=0x7fdd10004730 comp rx=0 tx=0).stop 2024-04-22T08:17:25.062 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.059+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4178071152}) 2024-04-22T08:17:25.062 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.059+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3891722324}) 2024-04-22T08:17:25.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:25.059+0000 7fdd2e821640 1 -- 172.21.15.40:0/3993727180 shutdown_connections 2024-04-22T08:17:25.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:25.059+0000 7fdd2e821640 1 --2- 172.21.15.40:0/3993727180 >> v2:172.21.15.189:6800/30168 conn(0x7fdd08037830 0x7fdd08039cf0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:25.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:25.059+0000 7fdd2e821640 1 --2- 172.21.15.40:0/3993727180 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd28133bb0 0x7fdd28133ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:25.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:25.059+0000 7fdd2e821640 1 --2- 172.21.15.40:0/3993727180 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd2806e830 0x7fdd2806ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:25.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:25.059+0000 7fdd2e821640 1 --2- 172.21.15.40:0/3993727180 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdd14003a40 0x7fdd281ca4a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:25.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:25.059+0000 7fdd2e821640 1 -- 172.21.15.40:0/3993727180 >> 172.21.15.40:0/3993727180 conn(0x7fdd280a65c0 msgr2=0x7fdd281bdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:17:25.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:25.059+0000 7fdd2e821640 1 -- 172.21.15.40:0/3993727180 shutdown_connections 2024-04-22T08:17:25.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:25.059+0000 7fdd2e821640 1 -- 172.21.15.40:0/3993727180 wait complete. 2024-04-22T08:17:25.068 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.059+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:17:25.068 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.059+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1650154494}) 2024-04-22T08:17:25.068 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.059+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=227493576}) 2024-04-22T08:17:25.068 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.059+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:17:25.068 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.059+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1650154494}) 2024-04-22T08:17:25.068 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.059+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=441398921}) 2024-04-22T08:17:25.068 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.060+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:17:25.069 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.060+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3034271501}) 2024-04-22T08:17:25.069 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.060+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1672715330}) 2024-04-22T08:17:25.069 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.060+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:17:25.069 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.060+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3034271501}) 2024-04-22T08:17:25.069 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.060+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1873474153}) 2024-04-22T08:17:25.069 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.061+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:17:25.069 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.061+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=275248076}) 2024-04-22T08:17:25.069 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.061+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=8010273}) 2024-04-22T08:17:25.069 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.061+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:17:25.069 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.061+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=275248076}) 2024-04-22T08:17:25.069 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.061+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3750539403}) 2024-04-22T08:17:25.069 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.062+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:17:25.069 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.062+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=140159642}) 2024-04-22T08:17:25.070 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.062+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=150382738}) 2024-04-22T08:17:25.070 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.062+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:17:25.070 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.062+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=140159642}) 2024-04-22T08:17:25.070 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.062+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=174112404}) 2024-04-22T08:17:25.070 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.062+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:17:25.070 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.062+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3452860017}) 2024-04-22T08:17:25.070 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.062+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=158054901}) 2024-04-22T08:17:25.070 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.062+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:17:25.070 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.062+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3452860017}) 2024-04-22T08:17:25.070 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.062+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2127739400}) 2024-04-22T08:17:25.070 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.063+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:17:25.070 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.063+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=723659930}) 2024-04-22T08:17:25.070 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.063+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2560093451}) 2024-04-22T08:17:25.071 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.063+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:17:25.071 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.063+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=723659930}) 2024-04-22T08:17:25.071 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.063+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3235264395}) 2024-04-22T08:17:25.071 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.064+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:17:25.071 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.064+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=90,mdsmap=3,mgr=7} crc {logm=3451142998,mdsmap=1531151339,mgr=1219260174}) 2024-04-22T08:17:25.071 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.064+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=92,mdsmap=3,mgr=5} crc {logm=517276695,mdsmap=1531151339,mgr=1515034267}) 2024-04-22T08:17:25.071 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.064+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:17:25.071 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.064+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=90,mdsmap=3,mgr=7} crc {logm=3451142998,mdsmap=1531151339,mgr=1219260174}) 2024-04-22T08:17:25.071 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.064+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=96,mdsmap=3,mgr=1} crc {logm=2443010682,mdsmap=1531151339,mgr=3836259515}) 2024-04-22T08:17:25.071 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.065+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:17:25.071 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.065+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=1,mgr_command_descs=1,mgr_metadata=1,mgrstat=15,mon_config_key=8,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=64} crc {mgr=1747058506,mgr_command_descs=470045235,mgr_metadata=4222303120,mgrstat=741411237,mon_config_key=4255067121,monmap=3469086636,osd_metadata=4151711904,osd_pg_creating=1074520031,osdmap=3601934934}) 2024-04-22T08:17:25.071 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.065+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr=1,mgr_command_descs=1,mgr_metadata=1,mgrstat=15,mon_config_key=8,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=64} crc {mgr=1747058506,mgr_command_descs=470045235,mgr_metadata=4222303120,mgrstat=741411237,mon_config_key=4255067121,monmap=3469086636,osd_metadata=4151711904,osd_pg_creating=1074520031,osdmap=41250990}) 2024-04-22T08:17:25.071 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.065+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:17:25.072 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.065+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=1,mgr_command_descs=1,mgr_metadata=1,mgrstat=15,mon_config_key=8,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=64} crc {mgr=1747058506,mgr_command_descs=470045235,mgr_metadata=4222303120,mgrstat=741411237,mon_config_key=4255067121,monmap=3469086636,osd_metadata=4151711904,osd_pg_creating=1074520031,osdmap=3601934934}) 2024-04-22T08:17:25.072 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.065+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=1,mgr_command_descs=1,mgr_metadata=1,mgrstat=15,mon_config_key=8,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=64} crc {mgr=1747058506,mgr_command_descs=470045235,mgr_metadata=4222303120,mgrstat=741411237,mon_config_key=4255067121,monmap=3469086636,osd_metadata=4151711904,osd_pg_creating=1074520031,osdmap=3461588817}) 2024-04-22T08:17:25.072 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.065+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:17:25.072 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.065+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=978086266}) 2024-04-22T08:17:25.072 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.065+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1436092635}) 2024-04-22T08:17:25.072 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.065+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:17:25.072 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.065+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=978086266}) 2024-04-22T08:17:25.072 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.065+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3108573226}) 2024-04-22T08:17:25.073 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.069+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:17:25.073 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.069+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=86} crc {osdmap=4125990725}) 2024-04-22T08:17:25.073 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.069+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=89} crc {osdmap=2514900349}) 2024-04-22T08:17:25.073 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.069+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:17:25.073 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.069+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=86} crc {osdmap=4125990725}) 2024-04-22T08:17:25.073 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:17:25.069+0000 7fb012412640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=90} crc {osdmap=3092481124}) 2024-04-22T08:17:25.074 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-22T08:17:26.075 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-22T08:17:26.076 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:17:26.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.189+0000 7f44d8bfd640 10 monclient: get_monmap_and_config 2024-04-22T08:17:26.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.189+0000 7f44d8bfd640 10 monclient: build_initial_monmap 2024-04-22T08:17:26.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.189+0000 7f44d8bfd640 10 monclient: monmap: 2024-04-22T08:17:26.190 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:17:26.190 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:26.190 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:17:26.191397+0000 2024-04-22T08:17:26.190 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:17:26.191397+0000 2024-04-22T08:17:26.190 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:17:26.190 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:26.190 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:17:26.191 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:17:26.191 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:17:26.191 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:17:26.191 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:17:26.191 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:26.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.190+0000 7f44d8bfd640 1 Processor -- start 2024-04-22T08:17:26.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.190+0000 7f44d8bfd640 1 -- start start 2024-04-22T08:17:26.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.190+0000 7f44d8bfd640 10 monclient: init 2024-04-22T08:17:26.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.190+0000 7f44d8bfd640 10 monclient: _reopen_session rank -1 2024-04-22T08:17:26.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.190+0000 7f44d8bfd640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-04-22T08:17:26.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.190+0000 7f44d8bfd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f44d409f2e0 addr v1:172.21.15.40:6789/0 2024-04-22T08:17:26.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.190+0000 7f44d8bfd640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f44d409b760 0x7f44d4137850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:26.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.190+0000 7f44d8bfd640 10 monclient(hunting): picked mon.noname-a con 0x7f44d409b760 addr v2:172.21.15.40:3300/0 2024-04-22T08:17:26.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.190+0000 7f44d8bfd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f44d409fdd0 addr v1:172.21.15.189:6789/0 2024-04-22T08:17:26.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.190+0000 7f44d8bfd640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44d409cf70 con 0x7f44d409f2e0 2024-04-22T08:17:26.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.190+0000 7f44d8bfd640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44d409c6d0 con 0x7f44d409fdd0 2024-04-22T08:17:26.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.190+0000 7f44d8bfd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:26.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.190+0000 7f44d8bfd640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44d409c9b0 con 0x7f44d409b760 2024-04-22T08:17:26.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.190+0000 7f44d8bfd640 10 monclient(hunting): _renew_subs 2024-04-22T08:17:26.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.190+0000 7f44d8bfd640 10 monclient(hunting): authenticate will time out at 1120.185742s 2024-04-22T08:17:26.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.191+0000 7f44cbfff640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f44d409b760 0x7f44d4137850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:26.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.191+0000 7f44d259c640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f44d409f2e0 0x7f44d409f6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:33330/0 (socket says 172.21.15.40:33330) 2024-04-22T08:17:26.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.191+0000 7f44cbfff640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f44d409b760 0x7f44d4137850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60284/0 (socket says 172.21.15.40:60284) 2024-04-22T08:17:26.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.191+0000 7f44d259c640 1 -- 172.21.15.40:0/503794857 learned_addr learned my addr 172.21.15.40:0/503794857 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:17:26.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.191+0000 7f44cbfff640 10 monclient(hunting): get_auth_request con 0x7f44d409b760 auth_method 0 2024-04-22T08:17:26.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.191+0000 7f44cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:26.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.191+0000 7f44cbfff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:26.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.191+0000 7f44cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:26.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.191+0000 7f44cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:26.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.191+0000 7f44cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:26.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.191+0000 7f44cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:26.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.191+0000 7f44cbfff640 10 monclient(hunting): handle_auth_done global_id 34782 payload 274 2024-04-22T08:17:26.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.191+0000 7f44cbfff640 1 -- 172.21.15.40:0/503794857 >> v1:172.21.15.189:6789/0 conn(0x7f44d409fdd0 legacy=0x7f44d4139da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:17:26.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.191+0000 7f44cbfff640 1 -- 172.21.15.40:0/503794857 >> v1:172.21.15.40:6789/0 conn(0x7f44d409f2e0 legacy=0x7f44d409f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:17:26.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.191+0000 7f44cbfff640 10 monclient: _finish_hunting 0 2024-04-22T08:17:26.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.191+0000 7f44cbfff640 1 monclient: found mon.noname-a 2024-04-22T08:17:26.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.191+0000 7f44cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:17:26.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.191+0000 7f44cbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:17:26.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.191+0000 7f44cbfff640 1 -- 172.21.15.40:0/503794857 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d409cc90 con 0x7f44d409b760 2024-04-22T08:17:26.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.194+0000 7f44cbfff640 1 -- 172.21.15.40:0/503794857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44d409b760 msgr2=0x7f44d4137850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:17:26.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.194+0000 7f44cbfff640 1 --2- 172.21.15.40:0/503794857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44d409b760 0x7f44d4137850 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f44bc0097a0 tx=0x7f44bc02c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd2d388fd306d8dd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:26.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.194+0000 7f44d1d9b640 1 -- 172.21.15.40:0/503794857 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44bc032a70 con 0x7f44d409b760 2024-04-22T08:17:26.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.194+0000 7f44d1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:26.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.194+0000 7f44d1d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:17:26.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.194+0000 7f44d1d9b640 10 monclient: dump: 2024-04-22T08:17:26.200 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:26.200 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:26.200 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:26.200 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:26.200 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:26.200 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:26.200 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:26.200 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:26.200 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:26.200 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:26.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.194+0000 7f44d1d9b640 10 monclient: _finish_auth 0 2024-04-22T08:17:26.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.194+0000 7f44d1d9b640 10 monclient: _check_auth_tickets 2024-04-22T08:17:26.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.194+0000 7f44d1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:17:26.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.194+0000 7f44d8bfd640 5 monclient: authenticate success, global_id 34782 2024-04-22T08:17:26.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.194+0000 7f44d1d9b640 1 -- 172.21.15.40:0/503794857 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44bc032bd0 con 0x7f44d409b760 2024-04-22T08:17:26.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.194+0000 7f44d8bfd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:17:26.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.194+0000 7f44d1d9b640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:17:26.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.194+0000 7f44d8bfd640 10 monclient: get_monmap_and_config success 2024-04-22T08:17:26.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.194+0000 7f44d8bfd640 4 set_mon_vals no callback set 2024-04-22T08:17:26.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.194+0000 7f44d1d9b640 1 -- 172.21.15.40:0/503794857 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44bc033410 con 0x7f44d409b760 2024-04-22T08:17:26.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.194+0000 7f44d1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:26.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.194+0000 7f44d1d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:17:26.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.194+0000 7f44d1d9b640 10 monclient: dump: 2024-04-22T08:17:26.201 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:26.201 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:26.201 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:26.202 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:26.202 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:26.202 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:26.202 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:26.202 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:26.202 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:26.202 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:26.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.195+0000 7f44d8bfd640 10 monclient: shutdown 2024-04-22T08:17:26.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.195+0000 7f44d8bfd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:17:26.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.195+0000 7f44d8bfd640 1 -- 172.21.15.40:0/503794857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44d409b760 msgr2=0x7f44d4137850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:26.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.195+0000 7f44d8bfd640 1 --2- 172.21.15.40:0/503794857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44d409b760 0x7f44d4137850 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f44bc0097a0 tx=0x7f44bc02c820 comp rx=0 tx=0).stop 2024-04-22T08:17:26.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.195+0000 7f44d8bfd640 1 -- 172.21.15.40:0/503794857 shutdown_connections 2024-04-22T08:17:26.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.195+0000 7f44d8bfd640 1 --2- 172.21.15.40:0/503794857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44d409b760 0x7f44d4137850 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:26.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.195+0000 7f44d8bfd640 1 -- 172.21.15.40:0/503794857 >> 172.21.15.40:0/503794857 conn(0x7f44d40a0b50 msgr2=0x7f44d40a0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:17:26.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.195+0000 7f44d8bfd640 1 -- 172.21.15.40:0/503794857 shutdown_connections 2024-04-22T08:17:26.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.195+0000 7f44d8bfd640 1 -- 172.21.15.40:0/503794857 wait complete. 2024-04-22T08:17:26.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.196+0000 7f44d8bfd640 10 monclient: build_initial_monmap 2024-04-22T08:17:26.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.196+0000 7f44d8bfd640 10 monclient: monmap: 2024-04-22T08:17:26.203 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:17:26.203 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:17:26.203 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:17:26.203 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:17:26.203 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:17:26.203 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:26.204 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:17:26.204 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:17:26.204 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:17:26.204 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:26.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.196+0000 7f44d8bfd640 1 Processor -- start 2024-04-22T08:17:26.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.196+0000 7f44d8bfd640 1 -- start start 2024-04-22T08:17:26.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.196+0000 7f44d8bfd640 10 monclient: init 2024-04-22T08:17:26.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.196+0000 7f44d8bfd640 10 monclient: _reopen_session rank -1 2024-04-22T08:17:26.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.196+0000 7f44d8bfd640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-22T08:17:26.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.196+0000 7f44d8bfd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44d409f2e0 0x7f44d41ca900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:26.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.196+0000 7f44d8bfd640 10 monclient(hunting): picked mon.noname-a con 0x7f44d409f2e0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:17:26.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.196+0000 7f44d8bfd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f44d409fdd0 0x7f44d413ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:26.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44d8bfd640 10 monclient(hunting): picked mon.noname-c con 0x7f44d409fdd0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:17:26.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44d8bfd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44d413cfd0 0x7f44d4123510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:26.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44d8bfd640 10 monclient(hunting): picked mon.noname-b con 0x7f44d413cfd0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:17:26.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44d8bfd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:26.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44d8bfd640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d41cae40 con 0x7f44d409f2e0 2024-04-22T08:17:26.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44d8bfd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:26.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44d8bfd640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d4071270 con 0x7f44d413cfd0 2024-04-22T08:17:26.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44d8bfd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:26.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44d8bfd640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44d409bb50 con 0x7f44d409fdd0 2024-04-22T08:17:26.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44d8bfd640 10 monclient(hunting): _renew_subs 2024-04-22T08:17:26.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44d8bfd640 10 monclient(hunting): authenticate will time out at 1120.191799s 2024-04-22T08:17:26.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44d259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44d409f2e0 0x7f44d41ca900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:26.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44cbfff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f44d409fdd0 0x7f44d413ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:26.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44d259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44d409f2e0 0x7f44d41ca900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60290/0 (socket says 172.21.15.40:60290) 2024-04-22T08:17:26.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44d259c640 1 -- 172.21.15.40:0/1572978186 learned_addr learned my addr 172.21.15.40:0/1572978186 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:17:26.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44d259c640 10 monclient(hunting): get_auth_request con 0x7f44d409f2e0 auth_method 0 2024-04-22T08:17:26.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44d259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:26.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44d259c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:26.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44d259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:26.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44cbfff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f44d409fdd0 0x7f44d413ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:52576/0 (socket says 172.21.15.40:52576) 2024-04-22T08:17:26.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44cbfff640 10 monclient(hunting): get_auth_request con 0x7f44d409fdd0 auth_method 0 2024-04-22T08:17:26.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:26.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44cbfff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:26.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:26.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44d259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:26.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44d259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:26.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44d259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:26.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:26.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:26.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44d2d9d640 1 --2- 172.21.15.40:0/1572978186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44d413cfd0 0x7f44d4123510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:26.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:26.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44d2d9d640 10 monclient(hunting): get_auth_request con 0x7f44d413cfd0 auth_method 0 2024-04-22T08:17:26.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44d2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:26.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44d2d9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:26.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44d2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:26.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44d2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:26.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44d2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:26.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44d2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:26.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44d259c640 10 monclient(hunting): handle_auth_done global_id 34785 payload 824 2024-04-22T08:17:26.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44d259c640 1 -- 172.21.15.40:0/1572978186 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f44d409fdd0 msgr2=0x7f44d413ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:26.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44d259c640 1 --2- 172.21.15.40:0/1572978186 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f44d409fdd0 0x7f44d413ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:26.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44d259c640 1 -- 172.21.15.40:0/1572978186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44d413cfd0 msgr2=0x7f44d4123510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:26.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44d259c640 1 --2- 172.21.15.40:0/1572978186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44d413cfd0 0x7f44d4123510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:26.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44d259c640 10 monclient: _finish_hunting 0 2024-04-22T08:17:26.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44d259c640 1 monclient: found mon.noname-a 2024-04-22T08:17:26.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44d259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:17:26.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44d259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:17:26.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.197+0000 7f44d259c640 1 -- 172.21.15.40:0/1572978186 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44bc009400 con 0x7f44d409f2e0 2024-04-22T08:17:26.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.199+0000 7f44d259c640 1 -- 172.21.15.40:0/1572978186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44d409f2e0 msgr2=0x7f44d41ca900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:17:26.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.199+0000 7f44d259c640 1 --2- 172.21.15.40:0/1572978186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44d409f2e0 0x7f44d41ca900 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f44c400c460 tx=0x7f44c400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:26.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.199+0000 7f44caffd640 1 -- 172.21.15.40:0/1572978186 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c4007070 con 0x7f44d409f2e0 2024-04-22T08:17:26.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.199+0000 7f44caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:26.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.199+0000 7f44caffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:17:26.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.199+0000 7f44caffd640 10 monclient: dump: 2024-04-22T08:17:26.211 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:26.211 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:26.211 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:26.211 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:26.211 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:26.211 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:26.211 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:26.211 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:26.211 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:26.212 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:26.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.199+0000 7f44caffd640 10 monclient: _finish_auth 0 2024-04-22T08:17:26.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.199+0000 7f44caffd640 10 monclient: _check_auth_tickets 2024-04-22T08:17:26.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.199+0000 7f44caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:17:26.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.199+0000 7f44caffd640 1 -- 172.21.15.40:0/1572978186 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44c40071f0 con 0x7f44d409f2e0 2024-04-22T08:17:26.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.199+0000 7f44d8bfd640 5 monclient: authenticate success, global_id 34785 2024-04-22T08:17:26.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.200+0000 7f44d8bfd640 10 monclient: _renew_subs 2024-04-22T08:17:26.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.200+0000 7f44d8bfd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:17:26.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.200+0000 7f44d8bfd640 1 -- 172.21.15.40:0/1572978186 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44d41cb800 con 0x7f44d409f2e0 2024-04-22T08:17:26.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.200+0000 7f44caffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:17:26.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.200+0000 7f44caffd640 1 -- 172.21.15.40:0/1572978186 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c40075a0 con 0x7f44d409f2e0 2024-04-22T08:17:26.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.200+0000 7f44caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:26.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.200+0000 7f44caffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:17:26.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.200+0000 7f44caffd640 10 monclient: dump: 2024-04-22T08:17:26.212 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:26.213 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:26.213 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:26.213 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:26.213 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:26.213 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:26.213 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:26.213 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:26.213 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:26.213 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:26.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.200+0000 7f44d159a640 4 set_mon_vals no callback set 2024-04-22T08:17:26.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.200+0000 7f44d8bfd640 10 monclient: _renew_subs 2024-04-22T08:17:26.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.200+0000 7f44d8bfd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:17:26.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.200+0000 7f44d8bfd640 1 -- 172.21.15.40:0/1572978186 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44d4126e60 con 0x7f44d409f2e0 2024-04-22T08:17:26.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.201+0000 7f44caffd640 1 -- 172.21.15.40:0/1572978186 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f44c401b400 con 0x7f44d409f2e0 2024-04-22T08:17:26.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.201+0000 7f44d8bfd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:17:26.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.201+0000 7f44d8bfd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:17:26.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.201+0000 7f44d8bfd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:17:26.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.201+0000 7f44d8bfd640 1 -- 172.21.15.40:0/1572978186 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44d41250b0 con 0x7f44d409f2e0 2024-04-22T08:17:26.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.201+0000 7f44caffd640 1 --2- 172.21.15.40:0/1572978186 >> v2:172.21.15.189:6800/30168 conn(0x7f44b40377d0 0x7f44b4039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:26.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.201+0000 7f44caffd640 1 -- 172.21.15.40:0/1572978186 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f44c4049f70 con 0x7f44d409f2e0 2024-04-22T08:17:26.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.201+0000 7f44cbfff640 1 --2- 172.21.15.40:0/1572978186 >> v2:172.21.15.189:6800/30168 conn(0x7f44b40377d0 0x7f44b4039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:26.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.201+0000 7f44cbfff640 10 monclient: get_auth_request con 0x7f44b40377d0 auth_method 0 2024-04-22T08:17:26.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.204+0000 7f44cbfff640 1 --2- 172.21.15.40:0/1572978186 >> v2:172.21.15.189:6800/30168 conn(0x7f44b40377d0 0x7f44b4039c90 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f44bc034000 tx=0x7f44bc03e040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:26.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.206+0000 7f44caffd640 1 -- 172.21.15.40:0/1572978186 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f44c40106f0 con 0x7f44d409f2e0 2024-04-22T08:17:26.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.206+0000 7f44caffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:17:26.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.206+0000 7f44caffd640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:17:26.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.454+0000 7f44d8bfd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:17:26.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.454+0000 7f44d8bfd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:17:26.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.454+0000 7f44d8bfd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:17:26.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.454+0000 7f44d8bfd640 1 -- 172.21.15.40:0/1572978186 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f44d4072150 con 0x7f44d409f2e0 2024-04-22T08:17:26.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.455+0000 7f44caffd640 1 -- 172.21.15.40:0/1572978186 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f44c401a360 con 0x7f44d409f2e0 2024-04-22T08:17:26.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.455+0000 7f44caffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:17:26.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.455+0000 7f44caffd640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:17:26.456 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":46,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:17:26.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.457+0000 7f44d8bfd640 1 -- 172.21.15.40:0/1572978186 >> v2:172.21.15.189:6800/30168 conn(0x7f44b40377d0 msgr2=0x7f44b4039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:26.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.457+0000 7f44d8bfd640 1 --2- 172.21.15.40:0/1572978186 >> v2:172.21.15.189:6800/30168 conn(0x7f44b40377d0 0x7f44b4039c90 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f44bc034000 tx=0x7f44bc03e040 comp rx=0 tx=0).stop 2024-04-22T08:17:26.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.457+0000 7f44d8bfd640 10 monclient: shutdown 2024-04-22T08:17:26.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.457+0000 7f44d8bfd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:17:26.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.457+0000 7f44d8bfd640 1 -- 172.21.15.40:0/1572978186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44d409f2e0 msgr2=0x7f44d41ca900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:26.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.457+0000 7f44d8bfd640 1 --2- 172.21.15.40:0/1572978186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44d409f2e0 0x7f44d41ca900 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f44c400c460 tx=0x7f44c400c920 comp rx=0 tx=0).stop 2024-04-22T08:17:26.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.457+0000 7f44d8bfd640 1 -- 172.21.15.40:0/1572978186 shutdown_connections 2024-04-22T08:17:26.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.457+0000 7f44d8bfd640 1 --2- 172.21.15.40:0/1572978186 >> v2:172.21.15.189:6800/30168 conn(0x7f44b40377d0 0x7f44b4039c90 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:26.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.457+0000 7f44d8bfd640 1 --2- 172.21.15.40:0/1572978186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44d413cfd0 0x7f44d4123510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:26.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.457+0000 7f44d8bfd640 1 --2- 172.21.15.40:0/1572978186 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f44d409fdd0 0x7f44d413ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:26.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.457+0000 7f44d8bfd640 1 --2- 172.21.15.40:0/1572978186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44d409f2e0 0x7f44d41ca900 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:26.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.457+0000 7f44d8bfd640 1 -- 172.21.15.40:0/1572978186 >> 172.21.15.40:0/1572978186 conn(0x7f44d40a0b50 msgr2=0x7f44d4071a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:17:26.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.457+0000 7f44d8bfd640 1 -- 172.21.15.40:0/1572978186 shutdown_connections 2024-04-22T08:17:26.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.457+0000 7f44d8bfd640 1 -- 172.21.15.40:0/1572978186 wait complete. 2024-04-22T08:17:26.468 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-22T08:17:26.468 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-22T08:17:26.468 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-22T08:17:26.468 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-22T08:17:26.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.567+0000 7f8df14df640 10 monclient: get_monmap_and_config 2024-04-22T08:17:26.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.567+0000 7f8df14df640 10 monclient: build_initial_monmap 2024-04-22T08:17:26.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.567+0000 7f8df14df640 10 monclient: monmap: 2024-04-22T08:17:26.568 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:17:26.568 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:26.568 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:17:26.569343+0000 2024-04-22T08:17:26.568 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:17:26.569343+0000 2024-04-22T08:17:26.569 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:17:26.569 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:26.569 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:17:26.569 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:17:26.569 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:17:26.569 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:17:26.569 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:17:26.569 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:26.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.568+0000 7f8df14df640 1 Processor -- start 2024-04-22T08:17:26.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.568+0000 7f8df14df640 1 -- start start 2024-04-22T08:17:26.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.568+0000 7f8df14df640 10 monclient: init 2024-04-22T08:17:26.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.568+0000 7f8df14df640 10 monclient: _reopen_session rank -1 2024-04-22T08:17:26.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.568+0000 7f8df14df640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-04-22T08:17:26.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.568+0000 7f8df14df640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8dec09f270 addr v1:172.21.15.189:6789/0 2024-04-22T08:17:26.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.568+0000 7f8df14df640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8dec09f650 addr v1:172.21.15.40:6789/0 2024-04-22T08:17:26.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.568+0000 7f8df14df640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8dec1387f0 0x7f8dec13abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:26.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.568+0000 7f8df14df640 10 monclient(hunting): picked mon.noname-c con 0x7f8dec1387f0 addr v2:172.21.15.189:3300/0 2024-04-22T08:17:26.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.568+0000 7f8df14df640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dec09e510 con 0x7f8dec09f650 2024-04-22T08:17:26.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.568+0000 7f8df14df640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dec09ba30 con 0x7f8dec09f270 2024-04-22T08:17:26.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.568+0000 7f8df14df640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:26.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.568+0000 7f8df14df640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dec09b760 con 0x7f8dec1387f0 2024-04-22T08:17:26.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.568+0000 7f8df14df640 10 monclient(hunting): _renew_subs 2024-04-22T08:17:26.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.568+0000 7f8df14df640 10 monclient(hunting): authenticate will time out at 1120.563428s 2024-04-22T08:17:26.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.568+0000 7f8dea7fc640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f8dec09f650 0x7f8dec09fa80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:33336/0 (socket says 172.21.15.40:33336) 2024-04-22T08:17:26.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.568+0000 7f8dea7fc640 1 -- 172.21.15.40:0/3413221443 learned_addr learned my addr 172.21.15.40:0/3413221443 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:17:26.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.568+0000 7f8deb7fe640 1 --2- 172.21.15.40:0/3413221443 >> v2:172.21.15.189:3300/0 conn(0x7f8dec1387f0 0x7f8dec13abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:26.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.568+0000 7f8deb7fe640 10 monclient(hunting): get_auth_request con 0x7f8dec1387f0 auth_method 0 2024-04-22T08:17:26.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.568+0000 7f8deb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:26.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.568+0000 7f8deb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:26.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.568+0000 7f8deb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:26.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.569+0000 7f8deb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:26.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.569+0000 7f8deb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:26.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.569+0000 7f8deb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:26.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.569+0000 7f8deb7fe640 10 monclient(hunting): handle_auth_done global_id 64135 payload 274 2024-04-22T08:17:26.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.569+0000 7f8deb7fe640 1 -- 172.21.15.40:0/3413221443 >> v1:172.21.15.189:6789/0 conn(0x7f8dec09f270 legacy=0x7f8dec09d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:17:26.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.569+0000 7f8deb7fe640 1 -- 172.21.15.40:0/3413221443 >> v1:172.21.15.40:6789/0 conn(0x7f8dec09f650 legacy=0x7f8dec09fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:17:26.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.569+0000 7f8deb7fe640 10 monclient: _finish_hunting 0 2024-04-22T08:17:26.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.569+0000 7f8deb7fe640 1 monclient: found mon.noname-c 2024-04-22T08:17:26.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.569+0000 7f8deb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:17:26.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.569+0000 7f8deb7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.189:3300/0 2024-04-22T08:17:26.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.569+0000 7f8deb7fe640 1 -- 172.21.15.40:0/3413221443 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dec09c790 con 0x7f8dec1387f0 2024-04-22T08:17:26.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.572+0000 7f8deb7fe640 1 -- 172.21.15.40:0/3413221443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8dec1387f0 msgr2=0x7f8dec13abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:17:26.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.572+0000 7f8deb7fe640 1 --2- 172.21.15.40:0/3413221443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8dec1387f0 0x7f8dec13abe0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8dec09b970 tx=0x7f8de402df00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1a113e725c2af59 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:26.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.572+0000 7f8de9ffb640 1 -- 172.21.15.40:0/3413221443 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de4036440 con 0x7f8dec1387f0 2024-04-22T08:17:26.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.572+0000 7f8de9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:26.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.572+0000 7f8de9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:17:26.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.572+0000 7f8de9ffb640 10 monclient: dump: 2024-04-22T08:17:26.579 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:26.579 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:26.579 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:26.579 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:26.579 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:26.580 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:26.580 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:26.580 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:26.580 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:26.580 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:26.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.572+0000 7f8de9ffb640 10 monclient: _finish_auth 0 2024-04-22T08:17:26.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.572+0000 7f8de9ffb640 10 monclient: _check_auth_tickets 2024-04-22T08:17:26.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.572+0000 7f8de9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:17:26.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.572+0000 7f8df14df640 5 monclient: authenticate success, global_id 64135 2024-04-22T08:17:26.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.572+0000 7f8de9ffb640 1 -- 172.21.15.40:0/3413221443 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de40365a0 con 0x7f8dec1387f0 2024-04-22T08:17:26.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.572+0000 7f8de9ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:17:26.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.572+0000 7f8df14df640 10 monclient: get_monmap_and_config success 2024-04-22T08:17:26.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.572+0000 7f8df14df640 4 set_mon_vals no callback set 2024-04-22T08:17:26.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.572+0000 7f8de9ffb640 1 -- 172.21.15.40:0/3413221443 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de4036930 con 0x7f8dec1387f0 2024-04-22T08:17:26.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.572+0000 7f8de9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:26.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.572+0000 7f8de9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:17:26.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.572+0000 7f8de9ffb640 10 monclient: dump: 2024-04-22T08:17:26.581 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:26.581 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:26.581 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:26.581 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:26.581 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:26.581 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:26.581 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:26.581 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:26.581 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:26.581 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:26.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.572+0000 7f8df14df640 10 monclient: shutdown 2024-04-22T08:17:26.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.572+0000 7f8df14df640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:17:26.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.572+0000 7f8df14df640 1 -- 172.21.15.40:0/3413221443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8dec1387f0 msgr2=0x7f8dec13abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:26.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.572+0000 7f8df14df640 1 --2- 172.21.15.40:0/3413221443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8dec1387f0 0x7f8dec13abe0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8dec09b970 tx=0x7f8de402df00 comp rx=0 tx=0).stop 2024-04-22T08:17:26.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.572+0000 7f8df14df640 1 -- 172.21.15.40:0/3413221443 shutdown_connections 2024-04-22T08:17:26.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.572+0000 7f8df14df640 1 --2- 172.21.15.40:0/3413221443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8dec1387f0 0x7f8dec13abe0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:26.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.572+0000 7f8df14df640 1 -- 172.21.15.40:0/3413221443 >> 172.21.15.40:0/3413221443 conn(0x7f8dec0a0ae0 msgr2=0x7f8dec0a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:17:26.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.573+0000 7f8df14df640 1 -- 172.21.15.40:0/3413221443 shutdown_connections 2024-04-22T08:17:26.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.573+0000 7f8df14df640 1 -- 172.21.15.40:0/3413221443 wait complete. 2024-04-22T08:17:26.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.573+0000 7f8df14df640 10 monclient: build_initial_monmap 2024-04-22T08:17:26.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.573+0000 7f8df14df640 10 monclient: monmap: 2024-04-22T08:17:26.582 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:17:26.582 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:17:26.582 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:17:26.583 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:17:26.583 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:17:26.583 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:26.583 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:17:26.583 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:17:26.583 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:17:26.583 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:26.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.573+0000 7f8df14df640 1 Processor -- start 2024-04-22T08:17:26.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8df14df640 1 -- start start 2024-04-22T08:17:26.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8df14df640 10 monclient: init 2024-04-22T08:17:26.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8df14df640 10 monclient: _reopen_session rank -1 2024-04-22T08:17:26.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8df14df640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-22T08:17:26.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8df14df640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8dec09f270 0x7f8dec11a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:26.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8df14df640 10 monclient(hunting): picked mon.noname-b con 0x7f8dec09f270 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:17:26.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8df14df640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8dec09f650 0x7f8dec11ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:26.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8df14df640 10 monclient(hunting): picked mon.noname-c con 0x7f8dec09f650 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:17:26.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8df14df640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dec1387f0 0x7f8dec11b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:26.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8df14df640 10 monclient(hunting): picked mon.noname-a con 0x7f8dec1387f0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:17:26.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8df14df640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:26.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8df14df640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dec11d200 con 0x7f8dec1387f0 2024-04-22T08:17:26.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8df14df640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:26.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8df14df640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dec11bc00 con 0x7f8dec09f270 2024-04-22T08:17:26.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8df14df640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:26.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8df14df640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dec1349f0 con 0x7f8dec09f650 2024-04-22T08:17:26.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8df14df640 10 monclient(hunting): _renew_subs 2024-04-22T08:17:26.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8df14df640 10 monclient(hunting): authenticate will time out at 1120.569228s 2024-04-22T08:17:26.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8deb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dec1387f0 0x7f8dec11b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:26.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8dea7fc640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8dec09f650 0x7f8dec11ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:26.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8deb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dec1387f0 0x7f8dec11b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60302/0 (socket says 172.21.15.40:60302) 2024-04-22T08:17:26.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8dea7fc640 10 monclient(hunting): get_auth_request con 0x7f8dec09f650 auth_method 0 2024-04-22T08:17:26.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8deb7fe640 1 -- 172.21.15.40:0/1431264330 learned_addr learned my addr 172.21.15.40:0/1431264330 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:17:26.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8dea7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:26.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8dea7fc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:26.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8dea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:26.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8deb7fe640 10 monclient(hunting): get_auth_request con 0x7f8dec1387f0 auth_method 0 2024-04-22T08:17:26.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8deb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:26.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8deb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:26.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8deb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:26.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8dea7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:26.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8dea7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:26.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8dea7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:26.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8deb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:26.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8deb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:26.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8deaffd640 1 --2- 172.21.15.40:0/1431264330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8dec09f270 0x7f8dec11a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:26.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8deb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:26.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8deaffd640 10 monclient(hunting): get_auth_request con 0x7f8dec09f270 auth_method 0 2024-04-22T08:17:26.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8deaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:26.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8deaffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:26.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8deaffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:26.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8dea7fc640 10 monclient(hunting): handle_auth_done global_id 54335 payload 824 2024-04-22T08:17:26.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8dea7fc640 1 -- 172.21.15.40:0/1431264330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8dec09f270 msgr2=0x7f8dec11a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:26.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8dea7fc640 1 --2- 172.21.15.40:0/1431264330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8dec09f270 0x7f8dec11a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:26.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8dea7fc640 1 -- 172.21.15.40:0/1431264330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dec1387f0 msgr2=0x7f8dec11b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:26.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8dea7fc640 1 --2- 172.21.15.40:0/1431264330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dec1387f0 0x7f8dec11b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:26.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8dea7fc640 10 monclient: _finish_hunting 0 2024-04-22T08:17:26.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8dea7fc640 1 monclient: found mon.noname-c 2024-04-22T08:17:26.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8dea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:17:26.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8dea7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:17:26.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8dea7fc640 1 -- 172.21.15.40:0/1431264330 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de4009a70 con 0x7f8dec09f650 2024-04-22T08:17:26.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8deb7fe640 1 --2- 172.21.15.40:0/1431264330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dec1387f0 0x7f8dec11b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:17:26.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.574+0000 7f8deaffd640 1 --2- 172.21.15.40:0/1431264330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8dec09f270 0x7f8dec11a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T08:17:26.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.577+0000 7f8dea7fc640 1 -- 172.21.15.40:0/1431264330 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8dec09f650 msgr2=0x7f8dec11ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:17:26.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.577+0000 7f8dea7fc640 1 --2- 172.21.15.40:0/1431264330 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8dec09f650 0x7f8dec11ab70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8de000c410 tx=0x7f8de000c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:26.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.577+0000 7f8dd37fe640 1 -- 172.21.15.40:0/1431264330 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de00070d0 con 0x7f8dec09f650 2024-04-22T08:17:26.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.577+0000 7f8dd37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:26.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.577+0000 7f8dd37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:17:26.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.577+0000 7f8dd37fe640 10 monclient: dump: 2024-04-22T08:17:26.592 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:26.592 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:26.592 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:26.592 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:26.592 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:26.592 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:26.593 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:26.593 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:26.593 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:26.593 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:26.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.577+0000 7f8dd37fe640 10 monclient: _finish_auth 0 2024-04-22T08:17:26.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.577+0000 7f8dd37fe640 10 monclient: _check_auth_tickets 2024-04-22T08:17:26.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.577+0000 7f8dd37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:17:26.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.577+0000 7f8df14df640 5 monclient: authenticate success, global_id 54335 2024-04-22T08:17:26.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.577+0000 7f8dd37fe640 1 -- 172.21.15.40:0/1431264330 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de0007230 con 0x7f8dec09f650 2024-04-22T08:17:26.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.577+0000 7f8df14df640 10 monclient: _renew_subs 2024-04-22T08:17:26.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.577+0000 7f8df14df640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:17:26.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.577+0000 7f8df14df640 1 -- 172.21.15.40:0/1431264330 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dec09bc10 con 0x7f8dec09f650 2024-04-22T08:17:26.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.577+0000 7f8dd37fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:17:26.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.577+0000 7f8dd37fe640 1 -- 172.21.15.40:0/1431264330 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de00075e0 con 0x7f8dec09f650 2024-04-22T08:17:26.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.577+0000 7f8dd37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:26.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.577+0000 7f8dd37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:17:26.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.577+0000 7f8dd37fe640 10 monclient: dump: 2024-04-22T08:17:26.594 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:26.594 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:26.594 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:26.594 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:26.594 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:26.594 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:26.594 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:26.594 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:26.594 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:26.594 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:26.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.577+0000 7f8df14df640 10 monclient: _renew_subs 2024-04-22T08:17:26.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.577+0000 7f8de8ff9640 4 set_mon_vals no callback set 2024-04-22T08:17:26.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.577+0000 7f8df14df640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:17:26.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.577+0000 7f8df14df640 1 -- 172.21.15.40:0/1431264330 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dec11d7c0 con 0x7f8dec09f650 2024-04-22T08:17:26.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.578+0000 7f8dd37fe640 1 -- 172.21.15.40:0/1431264330 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f8de001b3b0 con 0x7f8dec09f650 2024-04-22T08:17:26.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.578+0000 7f8df14df640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:17:26.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.578+0000 7f8df14df640 1 --2- 172.21.15.40:0/1431264330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dec136c10 0x7f8dec136ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:26.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.578+0000 7f8df14df640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:26.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.578+0000 7f8df14df640 1 -- 172.21.15.40:0/1431264330 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dec0667a0 con 0x7f8dec136c10 2024-04-22T08:17:26.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.578+0000 7f8dd37fe640 1 --2- 172.21.15.40:0/1431264330 >> v2:172.21.15.189:6800/30168 conn(0x7f8dd80377d0 0x7f8dd8039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:26.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.578+0000 7f8dd37fe640 1 -- 172.21.15.40:0/1431264330 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f8de0049a00 con 0x7f8dec09f650 2024-04-22T08:17:26.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.578+0000 7f8deaffd640 1 --2- 172.21.15.40:0/1431264330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dec136c10 0x7f8dec136ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:26.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.578+0000 7f8deaffd640 10 monclient: get_auth_request con 0x7f8dec136c10 auth_method 0 2024-04-22T08:17:26.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.578+0000 7f8deaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:26.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.578+0000 7f8deaffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:26.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.578+0000 7f8deaffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:26.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.578+0000 7f8deaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:26.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.578+0000 7f8deaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:26.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.578+0000 7f8deaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:26.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.578+0000 7f8deb7fe640 1 --2- 172.21.15.40:0/1431264330 >> v2:172.21.15.189:6800/30168 conn(0x7f8dd80377d0 0x7f8dd8039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:26.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.578+0000 7f8deb7fe640 10 monclient: get_auth_request con 0x7f8dd80377d0 auth_method 0 2024-04-22T08:17:26.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.578+0000 7f8deaffd640 10 monclient(hunting): handle_auth_done global_id 34791 payload 824 2024-04-22T08:17:26.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.578+0000 7f8deaffd640 1 -- 172.21.15.40:0/1431264330 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8dec09e9e0 con 0x7f8dec136c10 2024-04-22T08:17:26.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.580+0000 7f8deaffd640 1 -- 172.21.15.40:0/1431264330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dec136c10 msgr2=0x7f8dec136ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:17:26.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.580+0000 7f8deaffd640 1 --2- 172.21.15.40:0/1431264330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dec136c10 0x7f8dec136ff0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc00e500 tx=0x7f8ddc00e9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:26.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.581+0000 7f8dd37fe640 1 -- 172.21.15.40:0/1431264330 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ddc013800 con 0x7f8dec136c10 2024-04-22T08:17:26.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.581+0000 7f8dd37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:26.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.581+0000 7f8dd37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:17:26.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.581+0000 7f8dd37fe640 10 monclient: dump: 2024-04-22T08:17:26.597 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:26.597 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:26.597 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:26.597 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:26.597 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:26.597 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:26.597 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:26.597 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:26.597 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:26.597 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:26.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.581+0000 7f8deb7fe640 1 --2- 172.21.15.40:0/1431264330 >> v2:172.21.15.189:6800/30168 conn(0x7f8dd80377d0 0x7f8dd8039c90 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8de402df40 tx=0x7f8de4032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:26.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.581+0000 7f8dd37fe640 1 -- 172.21.15.40:0/1431264330 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f8ddc013960 con 0x7f8dec136c10 2024-04-22T08:17:26.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.581+0000 7f8dd37fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:17:26.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.581+0000 7f8dd37fe640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:17:26.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.581+0000 7f8dd37fe640 1 -- 172.21.15.40:0/1431264330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dec136c10 msgr2=0x7f8dec136ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:26.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.581+0000 7f8dd37fe640 1 --2- 172.21.15.40:0/1431264330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dec136c10 0x7f8dec136ff0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc00e500 tx=0x7f8ddc00e9c0 comp rx=0 tx=0).stop 2024-04-22T08:17:26.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.581+0000 7f8dd37fe640 1 -- 172.21.15.40:0/1431264330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dec136c10 msgr2=0x7f8dec136ff0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:17:26.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.581+0000 7f8dd37fe640 1 --2- 172.21.15.40:0/1431264330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dec136c10 0x7f8dec136ff0 secure :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc00e500 tx=0x7f8ddc00e9c0 comp rx=0 tx=0).stop 2024-04-22T08:17:26.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.587+0000 7f8df14df640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:17:26.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.587+0000 7f8df14df640 1 --2- 172.21.15.40:0/1431264330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dec137530 0x7f8dec1d09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:26.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.587+0000 7f8df14df640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:26.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.587+0000 7f8df14df640 1 -- 172.21.15.40:0/1431264330 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dec04e9b0 con 0x7f8dec137530 2024-04-22T08:17:26.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.587+0000 7f8deaffd640 1 --2- 172.21.15.40:0/1431264330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dec137530 0x7f8dec1d09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:26.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.587+0000 7f8deaffd640 10 monclient: get_auth_request con 0x7f8dec137530 auth_method 0 2024-04-22T08:17:26.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.587+0000 7f8deaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:26.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.587+0000 7f8deaffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:26.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.587+0000 7f8deaffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:26.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.587+0000 7f8deaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:26.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.587+0000 7f8deaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:26.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.587+0000 7f8deaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:26.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.587+0000 7f8deaffd640 10 monclient(hunting): handle_auth_done global_id 34794 payload 824 2024-04-22T08:17:26.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.587+0000 7f8deaffd640 1 -- 172.21.15.40:0/1431264330 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8dec11d3b0 con 0x7f8dec137530 2024-04-22T08:17:26.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.590+0000 7f8deaffd640 1 -- 172.21.15.40:0/1431264330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dec137530 msgr2=0x7f8dec1d09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:17:26.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.590+0000 7f8deaffd640 1 --2- 172.21.15.40:0/1431264330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dec137530 0x7f8dec1d09c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc00e500 tx=0x7f8ddc003760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:26.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.590+0000 7f8dd37fe640 1 -- 172.21.15.40:0/1431264330 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ddc006df0 con 0x7f8dec137530 2024-04-22T08:17:26.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.590+0000 7f8dd37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:26.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.590+0000 7f8dd37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:17:26.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.590+0000 7f8dd37fe640 10 monclient: dump: 2024-04-22T08:17:26.600 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:26.600 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:26.600 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:26.600 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:26.600 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:26.600 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:26.600 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:26.601 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:26.601 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:26.601 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:26.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.590+0000 7f8dd37fe640 1 -- 172.21.15.40:0/1431264330 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4633 (secure 0 0 0) 0x7f8ddc007680 con 0x7f8dec137530 2024-04-22T08:17:26.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.590+0000 7f8dd37fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:17:26.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.590+0000 7f8dd37fe640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:17:26.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.590+0000 7f8dd37fe640 1 -- 172.21.15.40:0/1431264330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dec137530 msgr2=0x7f8dec1d09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:26.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.590+0000 7f8dd37fe640 1 --2- 172.21.15.40:0/1431264330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dec137530 0x7f8dec1d09c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc00e500 tx=0x7f8ddc003760 comp rx=0 tx=0).stop 2024-04-22T08:17:26.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.590+0000 7f8dd37fe640 1 -- 172.21.15.40:0/1431264330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dec137530 msgr2=0x7f8dec1d09c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:17:26.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.590+0000 7f8dd37fe640 1 --2- 172.21.15.40:0/1431264330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dec137530 0x7f8dec1d09c0 secure :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc00e500 tx=0x7f8ddc003760 comp rx=0 tx=0).stop 2024-04-22T08:17:26.602 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:17:26.602 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:17:26.602 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:17:26.602 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "leader", 2024-04-22T08:17:26.602 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 46, 2024-04-22T08:17:26.602 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:17:26.602 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:17:26.602 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:17:26.602 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:17:26.602 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:17:26.603 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 2, 2024-04-22T08:17:26.603 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:17:26.603 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:17:26.603 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:17:26.603 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:17:26.603 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:17:26.603 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:17:26.603 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:17:26.603 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:17:26.603 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:17:26.603 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:17:26.604 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:17:26.604 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:17:26.604 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:17:26.604 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:17:26.604 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:17:26.604 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:17:26.604 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:17:26.604 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:17:26.604 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:17:26.604 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:17:26.604 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:17:26.604 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:17:26.605 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:17:26.605 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:17:26.605 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:17:26.605 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:17:26.605 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:26.605 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:26.605 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:17:26.605 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:17:26.605 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:17:26.605 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:17:26.605 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:17:26.605 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:17:26.605 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:17:26.605 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:17:26.606 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:17:26.606 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:17:26.606 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:17:26.606 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:17:26.606 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:17:26.606 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:17:26.606 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:17:26.606 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:17:26.606 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:17:26.606 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:17:26.606 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:17:26.606 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:17:26.606 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:17:26.606 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:17:26.607 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:17:26.607 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:17:26.607 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:17:26.607 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:17:26.607 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:17:26.607 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:17:26.607 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:17:26.607 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:26.607 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:17:26.607 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:26.607 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:17:26.607 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:17:26.607 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:17:26.607 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:17:26.608 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:26.608 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:17:26.608 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:17:26.608 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:26.608 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:26.608 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:26.608 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:17:26.608 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:17:26.608 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:26.608 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:26.608 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:26.608 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:26.608 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:17:26.608 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:17:26.609 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:17:26.609 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:17:26.609 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:17:26.609 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:26.609 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:26.609 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:17:26.609 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:17:26.609 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:17:26.609 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:17:26.609 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:26.609 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:17:26.609 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:17:26.609 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:26.609 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:26.609 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:26.610 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:17:26.610 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:17:26.610 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:26.610 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:26.610 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:26.610 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:26.610 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:17:26.610 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:17:26.610 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:17:26.610 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:17:26.610 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:17:26.610 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:26.610 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:26.610 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:17:26.611 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:17:26.611 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:17:26.611 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:17:26.611 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:26.611 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:17:26.611 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:17:26.611 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:26.611 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:26.611 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:26.611 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:17:26.611 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:17:26.611 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:26.611 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:26.611 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:26.612 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:26.612 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:17:26.612 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:17:26.612 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:17:26.612 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:17:26.612 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:17:26.612 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:26.612 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:26.612 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:26.612 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:17:26.612 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:17:26.612 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:26.612 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:17:26.612 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:17:26.612 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:17:26.613 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:26.613 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:17:26.613 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:17:26.613 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:26.613 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:17:26.613 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:17:26.613 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 5 2024-04-22T08:17:26.613 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:26.613 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:17:26.613 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:17:26.613 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:26.613 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:17:26.613 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:17:26.613 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 4 2024-04-22T08:17:26.614 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:26.614 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:26.614 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:26.614 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:17:26.614 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:17:26.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.590+0000 7f8df14df640 1 -- 172.21.15.40:0/1431264330 >> v2:172.21.15.189:6800/30168 conn(0x7f8dd80377d0 msgr2=0x7f8dd8039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:26.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.590+0000 7f8df14df640 1 --2- 172.21.15.40:0/1431264330 >> v2:172.21.15.189:6800/30168 conn(0x7f8dd80377d0 0x7f8dd8039c90 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8de402df40 tx=0x7f8de4032000 comp rx=0 tx=0).stop 2024-04-22T08:17:26.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.590+0000 7f8df14df640 10 monclient: shutdown 2024-04-22T08:17:26.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.590+0000 7f8df14df640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:17:26.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.590+0000 7f8df14df640 1 -- 172.21.15.40:0/1431264330 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8dec09f650 msgr2=0x7f8dec11ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:26.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.590+0000 7f8df14df640 1 --2- 172.21.15.40:0/1431264330 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8dec09f650 0x7f8dec11ab70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8de000c410 tx=0x7f8de000c8d0 comp rx=0 tx=0).stop 2024-04-22T08:17:26.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.590+0000 7f8deb7fe640 1 -- 172.21.15.40:0/1431264330 reap_dead start 2024-04-22T08:17:26.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.590+0000 7f8deb7fe640 1 -- 172.21.15.40:0/1431264330 reap_dead start 2024-04-22T08:17:26.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.590+0000 7f8df14df640 1 -- 172.21.15.40:0/1431264330 shutdown_connections 2024-04-22T08:17:26.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.590+0000 7f8df14df640 1 -- 172.21.15.40:0/1431264330 >> 172.21.15.40:0/1431264330 conn(0x7f8dec0a0ae0 msgr2=0x7f8dec138040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:17:26.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.591+0000 7f8df14df640 1 -- 172.21.15.40:0/1431264330 shutdown_connections 2024-04-22T08:17:26.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.591+0000 7f8df14df640 1 -- 172.21.15.40:0/1431264330 wait complete. 2024-04-22T08:17:26.617 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:17:26.617 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-22T08:17:26.617 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-22T08:17:26.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.715+0000 7f101cb47640 10 monclient: get_monmap_and_config 2024-04-22T08:17:26.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.715+0000 7f101cb47640 10 monclient: build_initial_monmap 2024-04-22T08:17:26.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.715+0000 7f101cb47640 10 monclient: monmap: 2024-04-22T08:17:26.716 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:17:26.716 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:26.716 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:17:26.717731+0000 2024-04-22T08:17:26.717 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:17:26.717731+0000 2024-04-22T08:17:26.717 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:17:26.717 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:26.717 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:17:26.717 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:17:26.717 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:17:26.717 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:17:26.717 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:17:26.717 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:26.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.716+0000 7f101cb47640 1 Processor -- start 2024-04-22T08:17:26.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.716+0000 7f101cb47640 1 -- start start 2024-04-22T08:17:26.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.716+0000 7f101cb47640 10 monclient: init 2024-04-22T08:17:26.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.717+0000 7f101cb47640 10 monclient: _reopen_session rank -1 2024-04-22T08:17:26.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.717+0000 7f101cb47640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-04-22T08:17:26.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.717+0000 7f101cb47640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f10180a5a10 0x7f10180a32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:26.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.717+0000 7f101cb47640 10 monclient(hunting): picked mon.noname-a con 0x7f10180a5a10 addr v2:172.21.15.40:3300/0 2024-04-22T08:17:26.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.717+0000 7f101cb47640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f10180a1970 addr v1:172.21.15.189:6789/0 2024-04-22T08:17:26.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.717+0000 7f101cb47640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f10180a1dc0 addr v1:172.21.15.40:6789/0 2024-04-22T08:17:26.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.717+0000 7f101cb47640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f101809fe80 con 0x7f10180a1dc0 2024-04-22T08:17:26.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.717+0000 7f101cb47640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10180a37f0 con 0x7f10180a1970 2024-04-22T08:17:26.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.717+0000 7f101cb47640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:26.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.717+0000 7f101cb47640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10180a25b0 con 0x7f10180a5a10 2024-04-22T08:17:26.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.717+0000 7f101cb47640 10 monclient(hunting): _renew_subs 2024-04-22T08:17:26.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.717+0000 7f101cb47640 10 monclient(hunting): authenticate will time out at 1120.711999s 2024-04-22T08:17:26.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.717+0000 7f101659c640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f10180a5a10 0x7f10180a32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:26.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.717+0000 7f101659c640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f10180a5a10 0x7f10180a32b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60332/0 (socket says 172.21.15.40:60332) 2024-04-22T08:17:26.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.717+0000 7f1016d9d640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f10180a1dc0 0x7f1018136330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:33346/0 (socket says 172.21.15.40:33346) 2024-04-22T08:17:26.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.717+0000 7f101659c640 1 -- 172.21.15.40:0/641484157 learned_addr learned my addr 172.21.15.40:0/641484157 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:17:26.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.717+0000 7f101659c640 10 monclient(hunting): get_auth_request con 0x7f10180a5a10 auth_method 0 2024-04-22T08:17:26.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.717+0000 7f101659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:26.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.717+0000 7f101659c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:26.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.717+0000 7f101659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:26.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.717+0000 7f101659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:26.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.717+0000 7f101659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:26.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.717+0000 7f101659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:26.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.717+0000 7f101659c640 10 monclient(hunting): handle_auth_done global_id 34797 payload 274 2024-04-22T08:17:26.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.717+0000 7f101659c640 1 -- 172.21.15.40:0/641484157 >> v1:172.21.15.189:6789/0 conn(0x7f10180a1970 legacy=0x7f1018135c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:17:26.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.717+0000 7f101659c640 1 -- 172.21.15.40:0/641484157 >> v1:172.21.15.40:6789/0 conn(0x7f10180a1dc0 legacy=0x7f1018136330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:17:26.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.717+0000 7f101659c640 10 monclient: _finish_hunting 0 2024-04-22T08:17:26.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.717+0000 7f101659c640 1 monclient: found mon.noname-a 2024-04-22T08:17:26.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.717+0000 7f101659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:17:26.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.717+0000 7f101659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:17:26.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.717+0000 7f101659c640 1 -- 172.21.15.40:0/641484157 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1018136a40 con 0x7f10180a5a10 2024-04-22T08:17:26.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.720+0000 7f101659c640 1 -- 172.21.15.40:0/641484157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10180a5a10 msgr2=0x7f10180a32b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:17:26.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.720+0000 7f101659c640 1 --2- 172.21.15.40:0/641484157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10180a5a10 0x7f10180a32b0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f10180a3980 tx=0x7f100802b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=203bc57ae7556945 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:26.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.720+0000 7f101559a640 1 -- 172.21.15.40:0/641484157 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1008031a30 con 0x7f10180a5a10 2024-04-22T08:17:26.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.720+0000 7f101559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:26.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.721+0000 7f101559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:17:26.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.721+0000 7f101559a640 10 monclient: dump: 2024-04-22T08:17:26.729 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:26.729 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:26.729 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:26.729 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:26.729 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:26.729 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:26.729 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:26.729 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:26.729 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:26.729 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:26.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.721+0000 7f101559a640 10 monclient: _finish_auth 0 2024-04-22T08:17:26.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.721+0000 7f101559a640 10 monclient: _check_auth_tickets 2024-04-22T08:17:26.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.721+0000 7f101559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:17:26.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.721+0000 7f101559a640 1 -- 172.21.15.40:0/641484157 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1008031b90 con 0x7f10180a5a10 2024-04-22T08:17:26.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.721+0000 7f101cb47640 5 monclient: authenticate success, global_id 34797 2024-04-22T08:17:26.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.721+0000 7f101cb47640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:17:26.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.721+0000 7f101559a640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:17:26.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.721+0000 7f101cb47640 10 monclient: get_monmap_and_config success 2024-04-22T08:17:26.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.721+0000 7f101559a640 1 -- 172.21.15.40:0/641484157 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1008033430 con 0x7f10180a5a10 2024-04-22T08:17:26.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.721+0000 7f101cb47640 4 set_mon_vals no callback set 2024-04-22T08:17:26.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.721+0000 7f101559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:26.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.721+0000 7f101559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:17:26.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.721+0000 7f101559a640 10 monclient: dump: 2024-04-22T08:17:26.730 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:26.730 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:26.730 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:26.730 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:26.731 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:26.731 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:26.731 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:26.731 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:26.731 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:26.731 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:26.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.721+0000 7f101cb47640 10 monclient: shutdown 2024-04-22T08:17:26.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.721+0000 7f101cb47640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:17:26.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.721+0000 7f101cb47640 1 -- 172.21.15.40:0/641484157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10180a5a10 msgr2=0x7f10180a32b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:26.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.721+0000 7f101cb47640 1 --2- 172.21.15.40:0/641484157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10180a5a10 0x7f10180a32b0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f10180a3980 tx=0x7f100802b7e0 comp rx=0 tx=0).stop 2024-04-22T08:17:26.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.721+0000 7f101cb47640 1 -- 172.21.15.40:0/641484157 shutdown_connections 2024-04-22T08:17:26.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.721+0000 7f101cb47640 1 --2- 172.21.15.40:0/641484157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10180a5a10 0x7f10180a32b0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:26.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.721+0000 7f101cb47640 1 -- 172.21.15.40:0/641484157 >> 172.21.15.40:0/641484157 conn(0x7f10180a4ef0 msgr2=0x7f10180a5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:17:26.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.721+0000 7f101cb47640 1 -- 172.21.15.40:0/641484157 shutdown_connections 2024-04-22T08:17:26.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.721+0000 7f101cb47640 1 -- 172.21.15.40:0/641484157 wait complete. 2024-04-22T08:17:26.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.722+0000 7f101cb47640 10 monclient: build_initial_monmap 2024-04-22T08:17:26.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.722+0000 7f101cb47640 10 monclient: monmap: 2024-04-22T08:17:26.732 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:17:26.732 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:17:26.732 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:17:26.732 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:17:26.732 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:17:26.732 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:26.732 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:17:26.732 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:17:26.732 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:17:26.732 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:26.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.722+0000 7f101cb47640 1 Processor -- start 2024-04-22T08:17:26.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.722+0000 7f101cb47640 1 -- start start 2024-04-22T08:17:26.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.722+0000 7f101cb47640 10 monclient: init 2024-04-22T08:17:26.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f101cb47640 10 monclient: _reopen_session rank -1 2024-04-22T08:17:26.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f101cb47640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-22T08:17:26.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f101cb47640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10180a1970 0x7f10181344e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:26.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f101cb47640 10 monclient(hunting): picked mon.noname-a con 0x7f10180a1970 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:17:26.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f101cb47640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f10180a1dc0 0x7f1018134a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:26.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f101cb47640 10 monclient(hunting): picked mon.noname-c con 0x7f10180a1dc0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:17:26.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f101cb47640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10180a5a10 0x7f1018134f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:26.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f101cb47640 10 monclient(hunting): picked mon.noname-b con 0x7f10180a5a10 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:17:26.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f101cb47640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:26.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f101cb47640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10181377c0 con 0x7f10180a1970 2024-04-22T08:17:26.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f101cb47640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:26.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f101cb47640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1018138b60 con 0x7f10180a5a10 2024-04-22T08:17:26.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f101cb47640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:26.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f101cb47640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10180a27c0 con 0x7f10180a1dc0 2024-04-22T08:17:26.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f101cb47640 10 monclient(hunting): _renew_subs 2024-04-22T08:17:26.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f101cb47640 10 monclient(hunting): authenticate will time out at 1120.717981s 2024-04-22T08:17:26.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f101659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10180a1970 0x7f10181344e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:26.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f101659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10180a1970 0x7f10181344e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60338/0 (socket says 172.21.15.40:60338) 2024-04-22T08:17:26.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f101659c640 1 -- 172.21.15.40:0/3434604487 learned_addr learned my addr 172.21.15.40:0/3434604487 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:17:26.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f101659c640 10 monclient(hunting): get_auth_request con 0x7f10180a1970 auth_method 0 2024-04-22T08:17:26.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f101659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:26.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f101659c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:26.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f101659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:26.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f1015d9b640 1 --2- 172.21.15.40:0/3434604487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f10180a1dc0 0x7f1018134a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:26.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f1015d9b640 10 monclient(hunting): get_auth_request con 0x7f10180a1dc0 auth_method 0 2024-04-22T08:17:26.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f1015d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:26.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f1015d9b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:26.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f1015d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:26.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f101659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:26.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f101659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:26.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f101659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:26.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f1015d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:26.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f1015d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:26.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f1015d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:26.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f1016d9d640 1 --2- 172.21.15.40:0/3434604487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10180a5a10 0x7f1018134f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:26.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f1016d9d640 10 monclient(hunting): get_auth_request con 0x7f10180a5a10 auth_method 0 2024-04-22T08:17:26.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f1016d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:26.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f1016d9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:26.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f1016d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:26.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f101659c640 10 monclient(hunting): handle_auth_done global_id 34800 payload 824 2024-04-22T08:17:26.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f101659c640 1 -- 172.21.15.40:0/3434604487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f10180a1dc0 msgr2=0x7f1018134a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:26.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f101659c640 1 --2- 172.21.15.40:0/3434604487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f10180a1dc0 0x7f1018134a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:26.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f101659c640 1 -- 172.21.15.40:0/3434604487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10180a5a10 msgr2=0x7f1018134f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:26.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f101659c640 1 --2- 172.21.15.40:0/3434604487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10180a5a10 0x7f1018134f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:26.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f101659c640 10 monclient: _finish_hunting 0 2024-04-22T08:17:26.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f101659c640 1 monclient: found mon.noname-a 2024-04-22T08:17:26.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f101659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:17:26.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f101659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:17:26.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f101659c640 1 -- 172.21.15.40:0/3434604487 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10080094d0 con 0x7f10180a1970 2024-04-22T08:17:26.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.723+0000 7f1016d9d640 1 --2- 172.21.15.40:0/3434604487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10180a5a10 0x7f1018134f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T08:17:26.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.725+0000 7f101659c640 1 -- 172.21.15.40:0/3434604487 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10180a1970 msgr2=0x7f10181344e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:17:26.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.725+0000 7f101659c640 1 --2- 172.21.15.40:0/3434604487 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10180a1970 0x7f10181344e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f10080090b0 tx=0x7f10080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:26.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.726+0000 7f100effd640 1 -- 172.21.15.40:0/3434604487 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1008031f00 con 0x7f10180a1970 2024-04-22T08:17:26.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.726+0000 7f100effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:26.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.726+0000 7f100effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:17:26.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.726+0000 7f100effd640 10 monclient: dump: 2024-04-22T08:17:26.736 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:26.736 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:26.736 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:26.736 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:26.737 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:26.737 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:26.737 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:26.737 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:26.737 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:26.737 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:26.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.726+0000 7f100effd640 10 monclient: _finish_auth 0 2024-04-22T08:17:26.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.726+0000 7f100effd640 10 monclient: _check_auth_tickets 2024-04-22T08:17:26.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.726+0000 7f100effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:17:26.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.726+0000 7f100effd640 1 -- 172.21.15.40:0/3434604487 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f100803d470 con 0x7f10180a1970 2024-04-22T08:17:26.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.726+0000 7f100effd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:17:26.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.726+0000 7f100effd640 1 -- 172.21.15.40:0/3434604487 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f100803d820 con 0x7f10180a1970 2024-04-22T08:17:26.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.726+0000 7f100effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:26.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.726+0000 7f100effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:17:26.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.726+0000 7f100effd640 10 monclient: dump: 2024-04-22T08:17:26.738 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:26.738 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:26.738 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:26.738 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:26.738 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:26.738 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:26.738 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:26.738 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:26.738 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:26.738 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:26.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.726+0000 7f101cb47640 5 monclient: authenticate success, global_id 34800 2024-04-22T08:17:26.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.726+0000 7f101cb47640 10 monclient: _renew_subs 2024-04-22T08:17:26.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.726+0000 7f101cb47640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:17:26.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.726+0000 7f1014d99640 4 set_mon_vals no callback set 2024-04-22T08:17:26.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.726+0000 7f101cb47640 1 -- 172.21.15.40:0/3434604487 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1018138880 con 0x7f10180a1970 2024-04-22T08:17:26.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.726+0000 7f101cb47640 10 monclient: _renew_subs 2024-04-22T08:17:26.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.726+0000 7f101cb47640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:17:26.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.726+0000 7f101cb47640 1 -- 172.21.15.40:0/3434604487 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1018136de0 con 0x7f10180a1970 2024-04-22T08:17:26.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.726+0000 7f101cb47640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:17:26.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.727+0000 7f101cb47640 1 --2- 172.21.15.40:0/3434604487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f101813eb10 0x7f1018130d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:26.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.727+0000 7f101cb47640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:26.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.727+0000 7f101cb47640 1 -- 172.21.15.40:0/3434604487 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10180667a0 con 0x7f101813eb10 2024-04-22T08:17:26.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.727+0000 7f100effd640 1 -- 172.21.15.40:0/3434604487 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f100803a040 con 0x7f10180a1970 2024-04-22T08:17:26.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.727+0000 7f1015d9b640 1 --2- 172.21.15.40:0/3434604487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f101813eb10 0x7f1018130d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:26.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.727+0000 7f1015d9b640 10 monclient: get_auth_request con 0x7f101813eb10 auth_method 0 2024-04-22T08:17:26.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.727+0000 7f1015d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:26.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.727+0000 7f1015d9b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:26.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.727+0000 7f1015d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:26.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.727+0000 7f1015d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:26.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.727+0000 7f1015d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:26.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.727+0000 7f1015d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:26.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.727+0000 7f100effd640 1 --2- 172.21.15.40:0/3434604487 >> v2:172.21.15.189:6800/30168 conn(0x7f0ff80377d0 0x7f0ff8039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:26.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.727+0000 7f1015d9b640 10 monclient(hunting): handle_auth_done global_id 54341 payload 824 2024-04-22T08:17:26.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.727+0000 7f100effd640 1 -- 172.21.15.40:0/3434604487 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f100806c720 con 0x7f10180a1970 2024-04-22T08:17:26.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.727+0000 7f1015d9b640 1 -- 172.21.15.40:0/3434604487 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f101809ee10 con 0x7f101813eb10 2024-04-22T08:17:26.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.727+0000 7f1016d9d640 1 --2- 172.21.15.40:0/3434604487 >> v2:172.21.15.189:6800/30168 conn(0x7f0ff80377d0 0x7f0ff8039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:26.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.727+0000 7f1016d9d640 10 monclient: get_auth_request con 0x7f0ff80377d0 auth_method 0 2024-04-22T08:17:26.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.729+0000 7f1015d9b640 1 -- 172.21.15.40:0/3434604487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f101813eb10 msgr2=0x7f1018130d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:17:26.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.729+0000 7f1015d9b640 1 --2- 172.21.15.40:0/3434604487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f101813eb10 0x7f1018130d70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1018135ec0 tx=0x7f100000ec00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:26.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.729+0000 7f100effd640 1 -- 172.21.15.40:0/3434604487 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f100000c980 con 0x7f101813eb10 2024-04-22T08:17:26.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.729+0000 7f100effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:26.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.729+0000 7f100effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:17:26.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.729+0000 7f100effd640 10 monclient: dump: 2024-04-22T08:17:26.743 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:26.743 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:26.743 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:26.743 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:26.743 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:26.743 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:26.743 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:26.743 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:26.743 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:26.743 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:26.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.730+0000 7f100effd640 1 -- 172.21.15.40:0/3434604487 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f100000cae0 con 0x7f101813eb10 2024-04-22T08:17:26.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.730+0000 7f100effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:17:26.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.730+0000 7f100effd640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:17:26.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.730+0000 7f100effd640 1 -- 172.21.15.40:0/3434604487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f101813eb10 msgr2=0x7f1018130d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:26.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.730+0000 7f100effd640 1 --2- 172.21.15.40:0/3434604487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f101813eb10 0x7f1018130d70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1018135ec0 tx=0x7f100000ec00 comp rx=0 tx=0).stop 2024-04-22T08:17:26.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.730+0000 7f100effd640 1 -- 172.21.15.40:0/3434604487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f101813eb10 msgr2=0x7f1018130d70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:17:26.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.730+0000 7f100effd640 1 --2- 172.21.15.40:0/3434604487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f101813eb10 0x7f1018130d70 secure :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1018135ec0 tx=0x7f100000ec00 comp rx=0 tx=0).stop 2024-04-22T08:17:26.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.730+0000 7f1016d9d640 1 --2- 172.21.15.40:0/3434604487 >> v2:172.21.15.189:6800/30168 conn(0x7f0ff80377d0 0x7f0ff8039c90 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1010005710 tx=0x7f1010009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:26.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.736+0000 7f101cb47640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:17:26.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.736+0000 7f101cb47640 1 --2- 172.21.15.40:0/3434604487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f10181312b0 0x7f10181316c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:26.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.736+0000 7f101cb47640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:26.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.736+0000 7f101cb47640 1 -- 172.21.15.40:0/3434604487 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10180a0160 con 0x7f10181312b0 2024-04-22T08:17:26.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.736+0000 7f1015d9b640 1 --2- 172.21.15.40:0/3434604487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f10181312b0 0x7f10181316c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:26.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.736+0000 7f1015d9b640 10 monclient: get_auth_request con 0x7f10181312b0 auth_method 0 2024-04-22T08:17:26.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.736+0000 7f1015d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:26.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.736+0000 7f1015d9b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:26.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.736+0000 7f1015d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:26.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.736+0000 7f1015d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:26.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.736+0000 7f1015d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:26.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.736+0000 7f1015d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:26.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.736+0000 7f1015d9b640 10 monclient(hunting): handle_auth_done global_id 54344 payload 824 2024-04-22T08:17:26.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.736+0000 7f1015d9b640 1 -- 172.21.15.40:0/3434604487 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f101813ef70 con 0x7f10181312b0 2024-04-22T08:17:26.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.738+0000 7f1015d9b640 1 -- 172.21.15.40:0/3434604487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f10181312b0 msgr2=0x7f10181316c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:17:26.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.738+0000 7f1015d9b640 1 --2- 172.21.15.40:0/3434604487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f10181312b0 0x7f10181316c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1018135ec0 tx=0x7f10000046f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:26.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.738+0000 7f100effd640 1 -- 172.21.15.40:0/3434604487 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1000023070 con 0x7f10181312b0 2024-04-22T08:17:26.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.738+0000 7f100effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:26.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.738+0000 7f100effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:17:26.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.738+0000 7f100effd640 10 monclient: dump: 2024-04-22T08:17:26.745 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:26.745 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:26.745 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:26.745 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:26.745 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:26.745 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:26.746 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:26.746 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:26.746 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:26.746 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:26.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.739+0000 7f100effd640 1 -- 172.21.15.40:0/3434604487 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4801 (secure 0 0 0) 0x7f1000006600 con 0x7f10181312b0 2024-04-22T08:17:26.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.739+0000 7f100effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:17:26.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.739+0000 7f100effd640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:17:26.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.739+0000 7f100effd640 1 -- 172.21.15.40:0/3434604487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f10181312b0 msgr2=0x7f10181316c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:26.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.739+0000 7f100effd640 1 --2- 172.21.15.40:0/3434604487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f10181312b0 0x7f10181316c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1018135ec0 tx=0x7f10000046f0 comp rx=0 tx=0).stop 2024-04-22T08:17:26.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.739+0000 7f100effd640 1 -- 172.21.15.40:0/3434604487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f10181312b0 msgr2=0x7f10181316c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:17:26.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.739+0000 7f100effd640 1 --2- 172.21.15.40:0/3434604487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f10181312b0 0x7f10181316c0 secure :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1018135ec0 tx=0x7f10000046f0 comp rx=0 tx=0).stop 2024-04-22T08:17:26.747 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:17:26.747 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:17:26.747 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:17:26.747 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:17:26.747 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 46, 2024-04-22T08:17:26.747 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:17:26.747 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:17:26.747 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:17:26.747 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:17:26.747 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:17:26.747 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 2, 2024-04-22T08:17:26.747 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:17:26.747 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:17:26.748 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:17:26.748 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:17:26.748 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:17:26.748 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:17:26.748 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:17:26.748 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:17:26.748 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:17:26.748 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:17:26.748 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:17:26.748 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:17:26.748 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:17:26.748 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:17:26.748 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:17:26.748 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:17:26.749 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:17:26.749 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:17:26.749 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:17:26.749 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:17:26.749 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:17:26.749 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:17:26.749 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:17:26.749 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:17:26.749 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:17:26.749 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:17:26.749 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:26.749 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:26.749 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:17:26.749 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:17:26.750 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:17:26.750 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:17:26.750 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:17:26.750 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:17:26.750 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:17:26.750 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:17:26.750 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:17:26.750 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:17:26.750 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:17:26.750 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:17:26.750 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:17:26.750 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:17:26.750 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:17:26.750 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:17:26.751 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:17:26.751 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:17:26.751 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:17:26.751 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:17:26.751 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:17:26.751 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:17:26.751 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:17:26.751 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:17:26.751 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:17:26.751 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:17:26.751 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:17:26.751 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:17:26.751 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:17:26.752 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:26.752 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:17:26.752 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:26.752 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:17:26.752 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:17:26.752 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:17:26.752 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:17:26.752 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:26.752 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:17:26.752 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:17:26.752 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:26.752 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:26.752 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:26.752 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:17:26.752 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:17:26.753 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:26.753 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:26.753 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:26.753 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:26.753 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:17:26.753 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:17:26.753 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:17:26.753 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:17:26.753 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:17:26.753 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:26.753 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:26.753 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:17:26.753 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:17:26.753 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:17:26.754 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:17:26.754 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:26.754 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:17:26.754 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:17:26.754 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:26.754 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:26.754 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:26.754 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:17:26.754 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:17:26.754 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:26.754 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:26.754 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:26.754 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:26.754 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:17:26.754 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:17:26.755 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:17:26.755 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:17:26.755 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:17:26.755 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:26.755 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:26.755 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:17:26.755 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:17:26.755 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:17:26.755 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:17:26.755 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:26.755 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:17:26.755 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:17:26.755 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:26.755 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:26.755 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:26.756 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:17:26.756 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:17:26.756 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:26.756 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:26.756 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:26.756 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:26.756 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:17:26.756 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:17:26.756 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:17:26.756 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:17:26.756 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:17:26.756 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:26.756 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:26.756 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:26.757 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:17:26.757 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:17:26.757 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:26.757 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:17:26.757 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:17:26.757 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:17:26.757 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:26.757 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:17:26.757 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:17:26.757 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:26.757 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:17:26.757 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:17:26.757 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:17:26.757 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:26.757 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:17:26.758 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:17:26.758 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:26.758 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:17:26.758 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:17:26.758 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 3 2024-04-22T08:17:26.758 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:26.758 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:17:26.758 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": [ 2024-04-22T08:17:26.758 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:26.758 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:17:26.758 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:17:26.758 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:17:26.758 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:26.758 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:26.758 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:26.759 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:17:26.759 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:17:26.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.739+0000 7f101cb47640 1 -- 172.21.15.40:0/3434604487 >> v2:172.21.15.189:6800/30168 conn(0x7f0ff80377d0 msgr2=0x7f0ff8039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:26.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.739+0000 7f101cb47640 1 --2- 172.21.15.40:0/3434604487 >> v2:172.21.15.189:6800/30168 conn(0x7f0ff80377d0 0x7f0ff8039c90 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1010005710 tx=0x7f1010009040 comp rx=0 tx=0).stop 2024-04-22T08:17:26.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.739+0000 7f101cb47640 10 monclient: shutdown 2024-04-22T08:17:26.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.739+0000 7f101cb47640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:17:26.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.739+0000 7f101cb47640 1 -- 172.21.15.40:0/3434604487 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10180a1970 msgr2=0x7f10181344e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:26.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.739+0000 7f101cb47640 1 --2- 172.21.15.40:0/3434604487 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10180a1970 0x7f10181344e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f10080090b0 tx=0x7f10080046c0 comp rx=0 tx=0).stop 2024-04-22T08:17:26.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.739+0000 7f1016d9d640 1 -- 172.21.15.40:0/3434604487 reap_dead start 2024-04-22T08:17:26.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.739+0000 7f1016d9d640 1 -- 172.21.15.40:0/3434604487 reap_dead start 2024-04-22T08:17:26.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.739+0000 7f101cb47640 1 -- 172.21.15.40:0/3434604487 shutdown_connections 2024-04-22T08:17:26.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.739+0000 7f101cb47640 1 -- 172.21.15.40:0/3434604487 >> 172.21.15.40:0/3434604487 conn(0x7f10180a4ef0 msgr2=0x7f1018136260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:17:26.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.739+0000 7f101cb47640 1 -- 172.21.15.40:0/3434604487 shutdown_connections 2024-04-22T08:17:26.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.739+0000 7f101cb47640 1 -- 172.21.15.40:0/3434604487 wait complete. 2024-04-22T08:17:26.761 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:17:26.761 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-22T08:17:26.761 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-22T08:17:26.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.861+0000 7fdb2b21c640 10 monclient: get_monmap_and_config 2024-04-22T08:17:26.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.861+0000 7fdb2b21c640 10 monclient: build_initial_monmap 2024-04-22T08:17:26.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.861+0000 7fdb2b21c640 10 monclient: monmap: 2024-04-22T08:17:26.861 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:17:26.861 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:26.861 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:17:26.862801+0000 2024-04-22T08:17:26.862 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:17:26.862801+0000 2024-04-22T08:17:26.862 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:17:26.862 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:26.862 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:17:26.862 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:17:26.862 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:17:26.862 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:17:26.862 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:17:26.862 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:26.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.861+0000 7fdb2b21c640 1 Processor -- start 2024-04-22T08:17:26.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.862+0000 7fdb2b21c640 1 -- start start 2024-04-22T08:17:26.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.862+0000 7fdb2b21c640 10 monclient: init 2024-04-22T08:17:26.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.862+0000 7fdb2b21c640 10 monclient: _reopen_session rank -1 2024-04-22T08:17:26.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.862+0000 7fdb2b21c640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-04-22T08:17:26.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.862+0000 7fdb2b21c640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdb240a1a80 0x7fdb240a01b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:26.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.862+0000 7fdb2b21c640 10 monclient(hunting): picked mon.noname-a con 0x7fdb240a1a80 addr v2:172.21.15.40:3300/0 2024-04-22T08:17:26.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.862+0000 7fdb2b21c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdb240a1e60 addr v1:172.21.15.40:6789/0 2024-04-22T08:17:26.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.862+0000 7fdb2b21c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdb2413b0e0 0x7fdb2413d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:26.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.862+0000 7fdb2b21c640 10 monclient(hunting): picked mon.noname-b con 0x7fdb2413b0e0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:17:26.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.862+0000 7fdb2b21c640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb2409d7f0 con 0x7fdb240a1e60 2024-04-22T08:17:26.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.862+0000 7fdb2b21c640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:26.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.862+0000 7fdb2b21c640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb2409f1c0 con 0x7fdb240a1a80 2024-04-22T08:17:26.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.862+0000 7fdb2b21c640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:26.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.862+0000 7fdb2b21c640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2409dc30 con 0x7fdb2413b0e0 2024-04-22T08:17:26.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.862+0000 7fdb2b21c640 10 monclient(hunting): _renew_subs 2024-04-22T08:17:26.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.862+0000 7fdb2b21c640 10 monclient(hunting): authenticate will time out at 1120.857602s 2024-04-22T08:17:26.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.862+0000 7fdb297b9640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdb2413b0e0 0x7fdb2413d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:26.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.862+0000 7fdb297b9640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdb2413b0e0 0x7fdb2413d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:52638/0 (socket says 172.21.15.40:52638) 2024-04-22T08:17:26.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.862+0000 7fdb297b9640 1 -- 172.21.15.40:0/3403253399 learned_addr learned my addr 172.21.15.40:0/3403253399 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:17:26.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.862+0000 7fdb297b9640 10 monclient(hunting): get_auth_request con 0x7fdb2413b0e0 auth_method 0 2024-04-22T08:17:26.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.862+0000 7fdb297b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:26.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.862+0000 7fdb297b9640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:26.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.862+0000 7fdb297b9640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:26.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.863+0000 7fdb28fb8640 1 --2- 172.21.15.40:0/3403253399 >> v2:172.21.15.40:3300/0 conn(0x7fdb240a1a80 0x7fdb240a01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:26.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.863+0000 7fdb28fb8640 10 monclient(hunting): get_auth_request con 0x7fdb240a1a80 auth_method 0 2024-04-22T08:17:26.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.863+0000 7fdb28fb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:26.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.863+0000 7fdb28fb8640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:26.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.863+0000 7fdb28fb8640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:26.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.863+0000 7fdb297b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:26.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.863+0000 7fdb297b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:26.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.863+0000 7fdb297b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:26.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.863+0000 7fdb28fb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:26.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.863+0000 7fdb28fb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:26.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.863+0000 7fdb28fb8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:26.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.863+0000 7fdb297b9640 10 monclient(hunting): handle_auth_done global_id 54347 payload 274 2024-04-22T08:17:26.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.863+0000 7fdb297b9640 1 -- 172.21.15.40:0/3403253399 >> v2:172.21.15.40:3300/0 conn(0x7fdb240a1a80 msgr2=0x7fdb240a01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:17:26.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.863+0000 7fdb297b9640 1 --2- 172.21.15.40:0/3403253399 >> v2:172.21.15.40:3300/0 conn(0x7fdb240a1a80 0x7fdb240a01b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:26.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.863+0000 7fdb297b9640 1 -- 172.21.15.40:0/3403253399 >> v1:172.21.15.40:6789/0 conn(0x7fdb240a1e60 legacy=0x7fdb240a22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:17:26.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.863+0000 7fdb297b9640 10 monclient: _finish_hunting 0 2024-04-22T08:17:26.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.863+0000 7fdb297b9640 1 monclient: found mon.noname-b 2024-04-22T08:17:26.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.863+0000 7fdb297b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:17:26.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.863+0000 7fdb297b9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:17:26.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.863+0000 7fdb297b9640 1 -- 172.21.15.40:0/3403253399 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb2409eee0 con 0x7fdb2413b0e0 2024-04-22T08:17:26.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.866+0000 7fdb297b9640 1 -- 172.21.15.40:0/3403253399 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdb2413b0e0 msgr2=0x7fdb2413d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:17:26.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.866+0000 7fdb297b9640 1 --2- 172.21.15.40:0/3403253399 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdb2413b0e0 0x7fdb2413d4d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c009e50 tx=0x7fdb1c02bdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b71943acdb0d0ec server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:26.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.866+0000 7fdb237fe640 1 -- 172.21.15.40:0/3403253399 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb1c02cb60 con 0x7fdb2413b0e0 2024-04-22T08:17:26.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.866+0000 7fdb237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:26.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.866+0000 7fdb237fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:17:26.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.866+0000 7fdb237fe640 10 monclient: dump: 2024-04-22T08:17:26.870 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:26.870 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:26.870 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:26.870 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:26.870 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:26.870 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:26.870 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:26.870 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:26.870 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:26.870 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:26.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.866+0000 7fdb237fe640 10 monclient: _finish_auth 0 2024-04-22T08:17:26.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.866+0000 7fdb237fe640 10 monclient: _check_auth_tickets 2024-04-22T08:17:26.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.866+0000 7fdb237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:17:26.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.866+0000 7fdb237fe640 1 -- 172.21.15.40:0/3403253399 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb1c02ccc0 con 0x7fdb2413b0e0 2024-04-22T08:17:26.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.866+0000 7fdb237fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:17:26.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.866+0000 7fdb237fe640 1 -- 172.21.15.40:0/3403253399 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb1c034610 con 0x7fdb2413b0e0 2024-04-22T08:17:26.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.867+0000 7fdb237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:26.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.867+0000 7fdb237fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:17:26.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.867+0000 7fdb237fe640 10 monclient: dump: 2024-04-22T08:17:26.871 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:26.871 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:26.871 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:26.871 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:26.871 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:26.871 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:26.872 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:26.872 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:26.872 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:26.872 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:26.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.867+0000 7fdb2b21c640 5 monclient: authenticate success, global_id 54347 2024-04-22T08:17:26.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.867+0000 7fdb2b21c640 10 monclient: get_monmap_and_config success 2024-04-22T08:17:26.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.867+0000 7fdb2b21c640 4 set_mon_vals no callback set 2024-04-22T08:17:26.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.867+0000 7fdb2b21c640 10 monclient: shutdown 2024-04-22T08:17:26.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.867+0000 7fdb2b21c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:17:26.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.867+0000 7fdb2b21c640 1 -- 172.21.15.40:0/3403253399 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdb2413b0e0 msgr2=0x7fdb2413d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:26.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.867+0000 7fdb2b21c640 1 --2- 172.21.15.40:0/3403253399 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdb2413b0e0 0x7fdb2413d4d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c009e50 tx=0x7fdb1c02bdb0 comp rx=0 tx=0).stop 2024-04-22T08:17:26.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.867+0000 7fdb2b21c640 1 -- 172.21.15.40:0/3403253399 shutdown_connections 2024-04-22T08:17:26.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.867+0000 7fdb2b21c640 1 --2- 172.21.15.40:0/3403253399 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdb2413b0e0 0x7fdb2413d4d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:26.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.867+0000 7fdb2b21c640 1 --2- 172.21.15.40:0/3403253399 >> v2:172.21.15.40:3300/0 conn(0x7fdb240a1a80 0x7fdb240a01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:26.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.867+0000 7fdb2b21c640 1 -- 172.21.15.40:0/3403253399 >> 172.21.15.40:0/3403253399 conn(0x7fdb240a66a0 msgr2=0x7fdb240a2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:17:26.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.867+0000 7fdb2b21c640 1 -- 172.21.15.40:0/3403253399 shutdown_connections 2024-04-22T08:17:26.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.867+0000 7fdb2b21c640 1 -- 172.21.15.40:0/3403253399 wait complete. 2024-04-22T08:17:26.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.868+0000 7fdb2b21c640 10 monclient: build_initial_monmap 2024-04-22T08:17:26.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.868+0000 7fdb2b21c640 10 monclient: monmap: 2024-04-22T08:17:26.873 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:17:26.873 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:17:26.873 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:17:26.873 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:17:26.873 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:17:26.873 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:26.873 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:17:26.873 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:17:26.873 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:17:26.873 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:26.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.869+0000 7fdb2b21c640 1 Processor -- start 2024-04-22T08:17:26.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.869+0000 7fdb2b21c640 1 -- start start 2024-04-22T08:17:26.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.869+0000 7fdb2b21c640 10 monclient: init 2024-04-22T08:17:26.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.869+0000 7fdb2b21c640 10 monclient: _reopen_session rank -1 2024-04-22T08:17:26.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.869+0000 7fdb2b21c640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-22T08:17:26.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.869+0000 7fdb2b21c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdb240a1a80 0x7fdb241cac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:26.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.869+0000 7fdb2b21c640 10 monclient(hunting): picked mon.noname-c con 0x7fdb240a1a80 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:17:26.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.869+0000 7fdb2b21c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb240a1e60 0x7fdb241cb160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:26.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.870+0000 7fdb2b21c640 10 monclient(hunting): picked mon.noname-a con 0x7fdb240a1e60 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:17:26.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.870+0000 7fdb2b21c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb2413b0e0 0x7fdb24120ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:26.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.870+0000 7fdb2b21c640 10 monclient(hunting): picked mon.noname-b con 0x7fdb2413b0e0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:17:26.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.870+0000 7fdb2b21c640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:26.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.870+0000 7fdb2b21c640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2409cc20 con 0x7fdb240a1e60 2024-04-22T08:17:26.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.870+0000 7fdb2b21c640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:26.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.870+0000 7fdb2b21c640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2413d270 con 0x7fdb2413b0e0 2024-04-22T08:17:26.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.870+0000 7fdb2b21c640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:26.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.870+0000 7fdb2b21c640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb24122f70 con 0x7fdb240a1a80 2024-04-22T08:17:26.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.870+0000 7fdb2b21c640 10 monclient(hunting): _renew_subs 2024-04-22T08:17:26.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.870+0000 7fdb2b21c640 10 monclient(hunting): authenticate will time out at 1120.864809s 2024-04-22T08:17:26.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.870+0000 7fdb28fb8640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdb240a1a80 0x7fdb241cac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:26.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.870+0000 7fdb23fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb240a1e60 0x7fdb241cb160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:26.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.870+0000 7fdb28fb8640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdb240a1a80 0x7fdb241cac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:52648/0 (socket says 172.21.15.40:52648) 2024-04-22T08:17:26.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.870+0000 7fdb23fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb240a1e60 0x7fdb241cb160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60344/0 (socket says 172.21.15.40:60344) 2024-04-22T08:17:26.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.870+0000 7fdb28fb8640 1 -- 172.21.15.40:0/3235578724 learned_addr learned my addr 172.21.15.40:0/3235578724 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:17:26.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.870+0000 7fdb28fb8640 10 monclient(hunting): get_auth_request con 0x7fdb240a1a80 auth_method 0 2024-04-22T08:17:26.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.870+0000 7fdb28fb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:26.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.870+0000 7fdb28fb8640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:26.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.870+0000 7fdb28fb8640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:26.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.870+0000 7fdb23fff640 10 monclient(hunting): get_auth_request con 0x7fdb240a1e60 auth_method 0 2024-04-22T08:17:26.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.870+0000 7fdb23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:26.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.870+0000 7fdb23fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:26.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.870+0000 7fdb23fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:26.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.870+0000 7fdb28fb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:26.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.870+0000 7fdb28fb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:26.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.870+0000 7fdb28fb8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:26.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.870+0000 7fdb23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:26.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.870+0000 7fdb23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:26.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.870+0000 7fdb23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:26.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.870+0000 7fdb297b9640 1 --2- 172.21.15.40:0/3235578724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb2413b0e0 0x7fdb24120ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:26.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.870+0000 7fdb297b9640 10 monclient(hunting): get_auth_request con 0x7fdb2413b0e0 auth_method 0 2024-04-22T08:17:26.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.870+0000 7fdb297b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:26.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.870+0000 7fdb297b9640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:26.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.870+0000 7fdb297b9640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:26.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.870+0000 7fdb23fff640 10 monclient(hunting): handle_auth_done global_id 34806 payload 824 2024-04-22T08:17:26.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.870+0000 7fdb23fff640 1 -- 172.21.15.40:0/3235578724 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdb240a1a80 msgr2=0x7fdb241cac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:26.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.870+0000 7fdb23fff640 1 --2- 172.21.15.40:0/3235578724 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdb240a1a80 0x7fdb241cac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:26.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.870+0000 7fdb23fff640 1 -- 172.21.15.40:0/3235578724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb2413b0e0 msgr2=0x7fdb24120ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:26.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.870+0000 7fdb23fff640 1 --2- 172.21.15.40:0/3235578724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb2413b0e0 0x7fdb24120ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:26.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.870+0000 7fdb23fff640 10 monclient: _finish_hunting 0 2024-04-22T08:17:26.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.870+0000 7fdb23fff640 1 monclient: found mon.noname-a 2024-04-22T08:17:26.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.870+0000 7fdb23fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:17:26.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.870+0000 7fdb23fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:17:26.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.870+0000 7fdb23fff640 1 -- 172.21.15.40:0/3235578724 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb1c009b30 con 0x7fdb240a1e60 2024-04-22T08:17:26.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.870+0000 7fdb28fb8640 1 --2- 172.21.15.40:0/3235578724 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdb240a1a80 0x7fdb241cac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:17:26.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.872+0000 7fdb23fff640 1 -- 172.21.15.40:0/3235578724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb240a1e60 msgr2=0x7fdb241cb160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:17:26.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.872+0000 7fdb23fff640 1 --2- 172.21.15.40:0/3235578724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb240a1e60 0x7fdb241cb160 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fdb1000c410 tx=0x7fdb1000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:26.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.873+0000 7fdb217fa640 1 -- 172.21.15.40:0/3235578724 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb10005230 con 0x7fdb240a1e60 2024-04-22T08:17:26.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.873+0000 7fdb217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:26.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.873+0000 7fdb217fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:17:26.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.873+0000 7fdb217fa640 10 monclient: dump: 2024-04-22T08:17:26.882 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:26.882 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:26.882 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:26.882 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:26.882 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:26.882 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:26.882 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:26.882 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:26.882 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:26.883 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:26.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.873+0000 7fdb217fa640 10 monclient: _finish_auth 0 2024-04-22T08:17:26.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.873+0000 7fdb217fa640 10 monclient: _check_auth_tickets 2024-04-22T08:17:26.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.873+0000 7fdb217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:17:26.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.873+0000 7fdb2b21c640 5 monclient: authenticate success, global_id 34806 2024-04-22T08:17:26.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.873+0000 7fdb217fa640 1 -- 172.21.15.40:0/3235578724 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb10005390 con 0x7fdb240a1e60 2024-04-22T08:17:26.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.873+0000 7fdb217fa640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:17:26.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.873+0000 7fdb217fa640 1 -- 172.21.15.40:0/3235578724 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb100119e0 con 0x7fdb240a1e60 2024-04-22T08:17:26.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.873+0000 7fdb2b21c640 10 monclient: _renew_subs 2024-04-22T08:17:26.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.873+0000 7fdb2b21c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:17:26.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.873+0000 7fdb2b21c640 1 -- 172.21.15.40:0/3235578724 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb24123b90 con 0x7fdb240a1e60 2024-04-22T08:17:26.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.873+0000 7fdb217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:26.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.873+0000 7fdb22ffd640 4 set_mon_vals no callback set 2024-04-22T08:17:26.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.873+0000 7fdb217fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:17:26.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.873+0000 7fdb217fa640 10 monclient: dump: 2024-04-22T08:17:26.884 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:26.884 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:26.884 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:26.884 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:26.884 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:26.884 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:26.884 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:26.884 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:26.884 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:26.884 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:26.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.873+0000 7fdb2b21c640 10 monclient: _renew_subs 2024-04-22T08:17:26.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.873+0000 7fdb2b21c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:17:26.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.873+0000 7fdb2b21c640 1 -- 172.21.15.40:0/3235578724 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb24122c30 con 0x7fdb240a1e60 2024-04-22T08:17:26.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.874+0000 7fdb2b21c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:17:26.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.874+0000 7fdb2b21c640 1 --2- 172.21.15.40:0/3235578724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb24060ff0 0x7fdb240613d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:26.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.874+0000 7fdb2b21c640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:26.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.874+0000 7fdb2b21c640 1 -- 172.21.15.40:0/3235578724 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb24066770 con 0x7fdb24060ff0 2024-04-22T08:17:26.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.874+0000 7fdb217fa640 1 -- 172.21.15.40:0/3235578724 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fdb10011bc0 con 0x7fdb240a1e60 2024-04-22T08:17:26.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.874+0000 7fdb28fb8640 1 --2- 172.21.15.40:0/3235578724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb24060ff0 0x7fdb240613d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:26.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.874+0000 7fdb217fa640 1 --2- 172.21.15.40:0/3235578724 >> v2:172.21.15.189:6800/30168 conn(0x7fdb080377d0 0x7fdb08039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:26.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.874+0000 7fdb28fb8640 10 monclient: get_auth_request con 0x7fdb24060ff0 auth_method 0 2024-04-22T08:17:26.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.874+0000 7fdb28fb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:26.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.874+0000 7fdb28fb8640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:26.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.874+0000 7fdb28fb8640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:26.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.874+0000 7fdb217fa640 1 -- 172.21.15.40:0/3235578724 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 3898+0+0 (secure 0 0 0) 0x7fdb10049870 con 0x7fdb240a1e60 2024-04-22T08:17:26.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.874+0000 7fdb28fb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:26.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.874+0000 7fdb28fb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:26.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.874+0000 7fdb28fb8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:26.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.874+0000 7fdb297b9640 1 --2- 172.21.15.40:0/3235578724 >> v2:172.21.15.189:6800/30168 conn(0x7fdb080377d0 0x7fdb08039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:26.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.874+0000 7fdb297b9640 10 monclient: get_auth_request con 0x7fdb080377d0 auth_method 0 2024-04-22T08:17:26.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.875+0000 7fdb28fb8640 10 monclient(hunting): handle_auth_done global_id 64147 payload 824 2024-04-22T08:17:26.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.875+0000 7fdb28fb8640 1 -- 172.21.15.40:0/3235578724 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdb2413da10 con 0x7fdb24060ff0 2024-04-22T08:17:26.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.877+0000 7fdb28fb8640 1 -- 172.21.15.40:0/3235578724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb24060ff0 msgr2=0x7fdb240613d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:17:26.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.877+0000 7fdb28fb8640 1 --2- 172.21.15.40:0/3235578724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb24060ff0 0x7fdb240613d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fdb18011db0 tx=0x7fdb1800f610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:26.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.877+0000 7fdb217fa640 1 -- 172.21.15.40:0/3235578724 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb1800fe70 con 0x7fdb24060ff0 2024-04-22T08:17:26.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.877+0000 7fdb217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:26.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.877+0000 7fdb217fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:17:26.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.877+0000 7fdb217fa640 10 monclient: dump: 2024-04-22T08:17:26.886 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:26.886 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:26.887 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:26.887 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:26.887 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:26.887 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:26.887 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:26.887 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:26.887 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:26.887 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:26.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.877+0000 7fdb297b9640 1 --2- 172.21.15.40:0/3235578724 >> v2:172.21.15.189:6800/30168 conn(0x7fdb080377d0 0x7fdb08039c90 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fdb2406a130 tx=0x7fdb1c030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:26.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.877+0000 7fdb217fa640 1 -- 172.21.15.40:0/3235578724 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7fdb18004030 con 0x7fdb24060ff0 2024-04-22T08:17:26.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.877+0000 7fdb217fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:17:26.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.877+0000 7fdb217fa640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:17:26.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.877+0000 7fdb217fa640 1 -- 172.21.15.40:0/3235578724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb24060ff0 msgr2=0x7fdb240613d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:26.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.877+0000 7fdb217fa640 1 --2- 172.21.15.40:0/3235578724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb24060ff0 0x7fdb240613d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fdb18011db0 tx=0x7fdb1800f610 comp rx=0 tx=0).stop 2024-04-22T08:17:26.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.877+0000 7fdb217fa640 1 -- 172.21.15.40:0/3235578724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb24060ff0 msgr2=0x7fdb240613d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:17:26.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.877+0000 7fdb217fa640 1 --2- 172.21.15.40:0/3235578724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb24060ff0 0x7fdb240613d0 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fdb18011db0 tx=0x7fdb1800f610 comp rx=0 tx=0).stop 2024-04-22T08:17:26.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.884+0000 7fdb2b21c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:17:26.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.884+0000 7fdb2b21c640 1 --2- 172.21.15.40:0/3235578724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb2402b200 0x7fdb2402b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:26.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.884+0000 7fdb2b21c640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:26.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.884+0000 7fdb2b21c640 1 -- 172.21.15.40:0/3235578724 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb241241f0 con 0x7fdb2402b200 2024-04-22T08:17:26.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.884+0000 7fdb28fb8640 1 --2- 172.21.15.40:0/3235578724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb2402b200 0x7fdb2402b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:26.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.884+0000 7fdb28fb8640 10 monclient: get_auth_request con 0x7fdb2402b200 auth_method 0 2024-04-22T08:17:26.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.884+0000 7fdb28fb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:26.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.884+0000 7fdb28fb8640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:26.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.884+0000 7fdb28fb8640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:26.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.884+0000 7fdb28fb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:26.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.884+0000 7fdb28fb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:26.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.884+0000 7fdb28fb8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:26.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.884+0000 7fdb28fb8640 10 monclient(hunting): handle_auth_done global_id 64150 payload 824 2024-04-22T08:17:26.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.884+0000 7fdb28fb8640 1 -- 172.21.15.40:0/3235578724 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fdb2404e9b0 con 0x7fdb2402b200 2024-04-22T08:17:26.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.887+0000 7fdb28fb8640 1 -- 172.21.15.40:0/3235578724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb2402b200 msgr2=0x7fdb2402b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:17:26.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.887+0000 7fdb28fb8640 1 --2- 172.21.15.40:0/3235578724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb2402b200 0x7fdb2402b610 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fdb1801b000 tx=0x7fdb180046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:26.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.887+0000 7fdb217fa640 1 -- 172.21.15.40:0/3235578724 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb18003c30 con 0x7fdb2402b200 2024-04-22T08:17:26.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.887+0000 7fdb217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:26.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.887+0000 7fdb217fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:17:26.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.887+0000 7fdb217fa640 10 monclient: dump: 2024-04-22T08:17:26.891 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:26.891 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:26.891 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:26.892 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:26.892 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:26.892 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:26.892 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:26.892 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:26.892 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:26.892 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:26.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.887+0000 7fdb217fa640 1 -- 172.21.15.40:0/3235578724 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4461 (secure 0 0 0) 0x7fdb18003d90 con 0x7fdb2402b200 2024-04-22T08:17:26.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.887+0000 7fdb217fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:17:26.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.887+0000 7fdb217fa640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:17:26.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.887+0000 7fdb217fa640 1 -- 172.21.15.40:0/3235578724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb2402b200 msgr2=0x7fdb2402b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:26.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.887+0000 7fdb217fa640 1 --2- 172.21.15.40:0/3235578724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb2402b200 0x7fdb2402b610 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fdb1801b000 tx=0x7fdb180046c0 comp rx=0 tx=0).stop 2024-04-22T08:17:26.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.887+0000 7fdb217fa640 1 -- 172.21.15.40:0/3235578724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb2402b200 msgr2=0x7fdb2402b610 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:17:26.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.887+0000 7fdb217fa640 1 --2- 172.21.15.40:0/3235578724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb2402b200 0x7fdb2402b610 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fdb1801b000 tx=0x7fdb180046c0 comp rx=0 tx=0).stop 2024-04-22T08:17:26.893 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:17:26.893 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:17:26.893 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:17:26.893 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:17:26.893 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 46, 2024-04-22T08:17:26.893 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:17:26.893 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:17:26.893 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:17:26.893 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:17:26.893 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:17:26.893 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 3, 2024-04-22T08:17:26.894 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:17:26.894 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:17:26.894 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:17:26.894 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:17:26.894 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:17:26.894 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:17:26.894 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:17:26.894 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:17:26.894 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:17:26.894 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:17:26.894 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:17:26.894 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:17:26.894 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:17:26.894 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:17:26.895 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:17:26.895 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:17:26.895 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:17:26.895 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:17:26.895 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:17:26.895 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:17:26.895 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:17:26.895 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:17:26.895 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:17:26.895 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:17:26.895 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:17:26.895 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:17:26.895 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:26.895 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:26.895 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:17:26.896 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:17:26.896 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:17:26.896 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:17:26.896 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:17:26.896 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:17:26.896 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:17:26.896 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:17:26.896 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:17:26.896 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:17:26.896 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:17:26.896 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:17:26.896 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:17:26.896 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:17:26.896 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:17:26.897 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:17:26.897 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:17:26.897 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:17:26.897 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:17:26.897 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:17:26.897 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:17:26.897 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:17:26.897 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:17:26.897 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:17:26.897 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:17:26.897 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:17:26.897 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:17:26.897 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:17:26.897 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:17:26.897 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:26.898 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:17:26.898 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:26.898 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:17:26.898 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:17:26.898 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:17:26.898 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:17:26.898 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:26.898 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:17:26.898 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:17:26.898 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:26.898 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:26.898 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:26.898 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:17:26.898 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:17:26.899 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:26.899 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:26.899 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:26.899 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:26.899 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:17:26.899 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:17:26.899 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:17:26.899 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:17:26.899 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:17:26.899 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:26.899 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:26.899 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:17:26.899 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:17:26.899 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:17:26.899 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:17:26.900 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:26.900 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:17:26.900 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:17:26.900 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:26.900 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:26.900 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:26.900 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:17:26.900 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:17:26.900 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:26.900 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:26.900 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:26.900 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:26.900 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:17:26.900 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:17:26.900 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:17:26.901 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:17:26.901 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:17:26.901 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:26.901 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:26.901 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:17:26.901 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:17:26.901 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:17:26.901 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:17:26.901 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:26.901 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:17:26.901 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:17:26.901 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:26.901 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:26.901 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:26.902 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:17:26.902 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:17:26.902 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:26.902 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:26.902 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:26.902 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:26.902 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:17:26.902 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:17:26.902 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:17:26.902 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:17:26.902 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:17:26.902 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:26.902 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:26.902 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:26.902 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:17:26.903 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:17:26.903 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:26.903 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:17:26.903 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:17:26.903 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:17:26.903 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:26.903 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:17:26.903 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:17:26.903 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:26.903 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:17:26.903 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:17:26.904 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 2 2024-04-22T08:17:26.904 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:26.904 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:26.904 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:26.904 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:17:26.904 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:17:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.887+0000 7fdb2b21c640 1 -- 172.21.15.40:0/3235578724 >> v2:172.21.15.189:6800/30168 conn(0x7fdb080377d0 msgr2=0x7fdb08039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.887+0000 7fdb2b21c640 1 --2- 172.21.15.40:0/3235578724 >> v2:172.21.15.189:6800/30168 conn(0x7fdb080377d0 0x7fdb08039c90 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fdb2406a130 tx=0x7fdb1c030000 comp rx=0 tx=0).stop 2024-04-22T08:17:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.887+0000 7fdb2b21c640 10 monclient: shutdown 2024-04-22T08:17:26.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.887+0000 7fdb2b21c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:17:26.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.887+0000 7fdb2b21c640 1 -- 172.21.15.40:0/3235578724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb240a1e60 msgr2=0x7fdb241cb160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:26.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.887+0000 7fdb2b21c640 1 --2- 172.21.15.40:0/3235578724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb240a1e60 0x7fdb241cb160 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fdb1000c410 tx=0x7fdb1000c8d0 comp rx=0 tx=0).stop 2024-04-22T08:17:26.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.887+0000 7fdb297b9640 1 -- 172.21.15.40:0/3235578724 reap_dead start 2024-04-22T08:17:26.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.887+0000 7fdb2b21c640 1 -- 172.21.15.40:0/3235578724 shutdown_connections 2024-04-22T08:17:26.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.887+0000 7fdb2b21c640 1 --2- 172.21.15.40:0/3235578724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb240a1e60 0x7fdb241cb160 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:26.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.888+0000 7fdb2b21c640 1 -- 172.21.15.40:0/3235578724 >> 172.21.15.40:0/3235578724 conn(0x7fdb240a66a0 msgr2=0x7fdb2409ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:17:26.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.888+0000 7fdb2b21c640 1 -- 172.21.15.40:0/3235578724 shutdown_connections 2024-04-22T08:17:26.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:26.888+0000 7fdb2b21c640 1 -- 172.21.15.40:0/3235578724 wait complete. 2024-04-22T08:17:26.906 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:17:26.907 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-04-22T08:17:26.907 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2024-04-22T08:17:26.907 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-04-22T08:17:26.907 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-04-22T08:17:26.907 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-22T08:17:26.907 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T08:17:26.919 INFO:tasks.ceph.mon.a:Stopped 2024-04-22T08:17:26.919 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-04-22T08:17:26.919 INFO:tasks.ceph.mon.c:Sent signal 19 2024-04-22T08:17:26.919 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-04-22T08:17:41.920 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-04-22T08:17:41.921 INFO:tasks.ceph.mon.c:Sent signal 18 2024-04-22T08:17:41.921 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-22T08:17:41.921 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:17:42.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.031+0000 7fc1c6058640 10 monclient: get_monmap_and_config 2024-04-22T08:17:42.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.031+0000 7fc1c6058640 10 monclient: build_initial_monmap 2024-04-22T08:17:42.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.031+0000 7fc1c6058640 10 monclient: monmap: 2024-04-22T08:17:42.031 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:17:42.031 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:42.032 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:17:42.032604+0000 2024-04-22T08:17:42.032 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:17:42.032604+0000 2024-04-22T08:17:42.032 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:17:42.032 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:42.032 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:17:42.032 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:17:42.032 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:17:42.032 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:17:42.032 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:17:42.032 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:42.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.032+0000 7fc1c6058640 1 Processor -- start 2024-04-22T08:17:42.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.032+0000 7fc1c6058640 1 -- start start 2024-04-22T08:17:42.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.032+0000 7fc1c6058640 10 monclient: init 2024-04-22T08:17:42.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.033+0000 7fc1c6058640 10 monclient: _reopen_session rank -1 2024-04-22T08:17:42.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.033+0000 7fc1c6058640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-04-22T08:17:42.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.033+0000 7fc1c6058640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1c00a01d0 0x7fc1c012f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:42.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.033+0000 7fc1c6058640 10 monclient(hunting): picked mon.noname-b con 0x7fc1c00a01d0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:17:42.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.033+0000 7fc1c6058640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc1c0070f90 0x7fc1c00713f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:42.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.033+0000 7fc1c6058640 10 monclient(hunting): picked mon.noname-c con 0x7fc1c0070f90 addr v2:172.21.15.189:3300/0 2024-04-22T08:17:42.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.033+0000 7fc1c6058640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc1c006ce90 addr v1:172.21.15.40:6789/0 2024-04-22T08:17:42.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.033+0000 7fc1c6058640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1c009eee0 con 0x7fc1c006ce90 2024-04-22T08:17:42.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.033+0000 7fc1c6058640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:42.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.033+0000 7fc1c6058640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1c012f2f0 con 0x7fc1c0070f90 2024-04-22T08:17:42.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.033+0000 7fc1c6058640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:42.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.033+0000 7fc1c6058640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c00a05b0 con 0x7fc1c00a01d0 2024-04-22T08:17:42.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.033+0000 7fc1c6058640 10 monclient(hunting): _renew_subs 2024-04-22T08:17:42.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.033+0000 7fc1c6058640 10 monclient(hunting): authenticate will time out at 1136.027140s 2024-04-22T08:17:42.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.033+0000 7fc1bffff640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7fc1c006ce90 legacy=0x7fc1c006d300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-04-22T08:17:42.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.033+0000 7fc1bf7fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1c00a01d0 0x7fc1c012f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:42.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.033+0000 7fc1bf7fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1c00a01d0 0x7fc1c012f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:58752/0 (socket says 172.21.15.40:58752) 2024-04-22T08:17:42.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.033+0000 7fc1bf7fe640 1 -- 172.21.15.40:0/410493384 learned_addr learned my addr 172.21.15.40:0/410493384 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:17:42.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.033+0000 7fc1bf7fe640 10 monclient(hunting): get_auth_request con 0x7fc1c00a01d0 auth_method 0 2024-04-22T08:17:42.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.033+0000 7fc1bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:42.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.033+0000 7fc1bf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:42.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.033+0000 7fc1bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:42.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.033+0000 7fc1beffd640 1 --2- 172.21.15.40:0/410493384 >> v2:172.21.15.189:3300/0 conn(0x7fc1c0070f90 0x7fc1c00713f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:42.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.033+0000 7fc1beffd640 10 monclient(hunting): get_auth_request con 0x7fc1c0070f90 auth_method 0 2024-04-22T08:17:42.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.033+0000 7fc1beffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:42.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.033+0000 7fc1beffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:42.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.033+0000 7fc1beffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:42.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.033+0000 7fc1bf7fe640 1 -- 172.21.15.40:0/410493384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1c00a01d0 msgr2=0x7fc1c012f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:17:42.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.033+0000 7fc1bf7fe640 1 -- 172.21.15.40:0/410493384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1c00a01d0 msgr2=0x7fc1c012f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:17:42.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.033+0000 7fc1bf7fe640 1 --2- 172.21.15.40:0/410493384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1c00a01d0 0x7fc1c012f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:17:42.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.033+0000 7fc1bf7fe640 1 --2- 172.21.15.40:0/410493384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1c00a01d0 0x7fc1c012f6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:17:42.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.033+0000 7fc1beffd640 1 -- 172.21.15.40:0/410493384 >> v2:172.21.15.189:3300/0 conn(0x7fc1c0070f90 msgr2=0x7fc1c00713f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:17:42.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.033+0000 7fc1beffd640 1 -- 172.21.15.40:0/410493384 >> v2:172.21.15.189:3300/0 conn(0x7fc1c0070f90 msgr2=0x7fc1c00713f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:17:42.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.033+0000 7fc1beffd640 1 --2- 172.21.15.40:0/410493384 >> v2:172.21.15.189:3300/0 conn(0x7fc1c0070f90 0x7fc1c00713f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:17:42.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.033+0000 7fc1beffd640 1 --2- 172.21.15.40:0/410493384 >> v2:172.21.15.189:3300/0 conn(0x7fc1c0070f90 0x7fc1c00713f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:17:42.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.234+0000 7fc1bffff640 1 -- 172.21.15.40:0/410493384 >> v1:172.21.15.40:6789/0 conn(0x7fc1c006ce90 legacy=0x7fc1c006d300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-04-22T08:17:42.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.234+0000 7fc1bf7fe640 1 --2- 172.21.15.40:0/410493384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1c00a01d0 0x7fc1c012f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:42.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.234+0000 7fc1bf7fe640 10 monclient(hunting): get_auth_request con 0x7fc1c00a01d0 auth_method 0 2024-04-22T08:17:42.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.234+0000 7fc1bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:42.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.234+0000 7fc1bf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:42.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.234+0000 7fc1bf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:17:42.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.234+0000 7fc1bf7fe640 1 -- 172.21.15.40:0/410493384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1c00a01d0 msgr2=0x7fc1c012f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:17:42.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.234+0000 7fc1bf7fe640 1 -- 172.21.15.40:0/410493384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1c00a01d0 msgr2=0x7fc1c012f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:17:42.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.234+0000 7fc1bf7fe640 1 --2- 172.21.15.40:0/410493384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1c00a01d0 0x7fc1c012f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:17:42.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.234+0000 7fc1bf7fe640 1 --2- 172.21.15.40:0/410493384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1c00a01d0 0x7fc1c012f6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:17:42.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.234+0000 7fc1beffd640 1 --2- 172.21.15.40:0/410493384 >> v2:172.21.15.189:3300/0 conn(0x7fc1c0070f90 0x7fc1c00713f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:42.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.234+0000 7fc1beffd640 10 monclient(hunting): get_auth_request con 0x7fc1c0070f90 auth_method 0 2024-04-22T08:17:42.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.234+0000 7fc1beffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:42.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.234+0000 7fc1beffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:42.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.234+0000 7fc1beffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:17:42.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.234+0000 7fc1beffd640 1 -- 172.21.15.40:0/410493384 >> v2:172.21.15.189:3300/0 conn(0x7fc1c0070f90 msgr2=0x7fc1c00713f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:17:42.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.234+0000 7fc1beffd640 1 -- 172.21.15.40:0/410493384 >> v2:172.21.15.189:3300/0 conn(0x7fc1c0070f90 msgr2=0x7fc1c00713f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:17:42.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.234+0000 7fc1beffd640 1 --2- 172.21.15.40:0/410493384 >> v2:172.21.15.189:3300/0 conn(0x7fc1c0070f90 0x7fc1c00713f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:17:42.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.234+0000 7fc1beffd640 1 --2- 172.21.15.40:0/410493384 >> v2:172.21.15.189:3300/0 conn(0x7fc1c0070f90 0x7fc1c00713f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:17:42.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.635+0000 7fc1bffff640 1 -- 172.21.15.40:0/410493384 >> v1:172.21.15.40:6789/0 conn(0x7fc1c006ce90 legacy=0x7fc1c006d300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-04-22T08:17:42.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.635+0000 7fc1bf7fe640 1 --2- 172.21.15.40:0/410493384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1c00a01d0 0x7fc1c012f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:42.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.635+0000 7fc1bf7fe640 10 monclient(hunting): get_auth_request con 0x7fc1c00a01d0 auth_method 0 2024-04-22T08:17:42.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.635+0000 7fc1bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:42.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.635+0000 7fc1bf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:42.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.635+0000 7fc1bf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:17:42.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.635+0000 7fc1bf7fe640 1 -- 172.21.15.40:0/410493384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1c00a01d0 msgr2=0x7fc1c012f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:17:42.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.635+0000 7fc1bf7fe640 1 -- 172.21.15.40:0/410493384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1c00a01d0 msgr2=0x7fc1c012f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:17:42.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.635+0000 7fc1bf7fe640 1 --2- 172.21.15.40:0/410493384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1c00a01d0 0x7fc1c012f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:17:42.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.635+0000 7fc1bf7fe640 1 --2- 172.21.15.40:0/410493384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1c00a01d0 0x7fc1c012f6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:17:42.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.635+0000 7fc1beffd640 1 --2- 172.21.15.40:0/410493384 >> v2:172.21.15.189:3300/0 conn(0x7fc1c0070f90 0x7fc1c00713f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:42.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.635+0000 7fc1beffd640 10 monclient(hunting): get_auth_request con 0x7fc1c0070f90 auth_method 0 2024-04-22T08:17:42.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.635+0000 7fc1beffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:42.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.635+0000 7fc1beffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:42.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.635+0000 7fc1beffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:17:42.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.635+0000 7fc1beffd640 1 -- 172.21.15.40:0/410493384 >> v2:172.21.15.189:3300/0 conn(0x7fc1c0070f90 msgr2=0x7fc1c00713f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:17:42.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.635+0000 7fc1beffd640 1 -- 172.21.15.40:0/410493384 >> v2:172.21.15.189:3300/0 conn(0x7fc1c0070f90 msgr2=0x7fc1c00713f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:17:42.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.635+0000 7fc1beffd640 1 --2- 172.21.15.40:0/410493384 >> v2:172.21.15.189:3300/0 conn(0x7fc1c0070f90 0x7fc1c00713f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:17:42.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:42.635+0000 7fc1beffd640 1 --2- 172.21.15.40:0/410493384 >> v2:172.21.15.189:3300/0 conn(0x7fc1c0070f90 0x7fc1c00713f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:17:43.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:43.436+0000 7fc1bffff640 1 -- 172.21.15.40:0/410493384 >> v1:172.21.15.40:6789/0 conn(0x7fc1c006ce90 legacy=0x7fc1c006d300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-04-22T08:17:43.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:43.436+0000 7fc1bf7fe640 1 --2- 172.21.15.40:0/410493384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1c00a01d0 0x7fc1c012f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:43.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:43.436+0000 7fc1bf7fe640 10 monclient(hunting): get_auth_request con 0x7fc1c00a01d0 auth_method 0 2024-04-22T08:17:43.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:43.436+0000 7fc1bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:43.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:43.436+0000 7fc1bf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:43.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:43.436+0000 7fc1bf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:17:43.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:43.436+0000 7fc1bf7fe640 1 -- 172.21.15.40:0/410493384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1c00a01d0 msgr2=0x7fc1c012f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:17:43.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:43.436+0000 7fc1bf7fe640 1 -- 172.21.15.40:0/410493384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1c00a01d0 msgr2=0x7fc1c012f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:17:43.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:43.436+0000 7fc1bf7fe640 1 --2- 172.21.15.40:0/410493384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1c00a01d0 0x7fc1c012f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:17:43.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:43.436+0000 7fc1bf7fe640 1 --2- 172.21.15.40:0/410493384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1c00a01d0 0x7fc1c012f6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T08:17:43.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:43.436+0000 7fc1beffd640 1 --2- 172.21.15.40:0/410493384 >> v2:172.21.15.189:3300/0 conn(0x7fc1c0070f90 0x7fc1c00713f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:43.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:43.436+0000 7fc1beffd640 10 monclient(hunting): get_auth_request con 0x7fc1c0070f90 auth_method 0 2024-04-22T08:17:43.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:43.436+0000 7fc1beffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:43.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:43.436+0000 7fc1beffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:43.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:43.436+0000 7fc1beffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:17:43.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:43.437+0000 7fc1beffd640 1 -- 172.21.15.40:0/410493384 >> v2:172.21.15.189:3300/0 conn(0x7fc1c0070f90 msgr2=0x7fc1c00713f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:17:43.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:43.437+0000 7fc1beffd640 1 -- 172.21.15.40:0/410493384 >> v2:172.21.15.189:3300/0 conn(0x7fc1c0070f90 msgr2=0x7fc1c00713f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:17:43.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:43.437+0000 7fc1beffd640 1 --2- 172.21.15.40:0/410493384 >> v2:172.21.15.189:3300/0 conn(0x7fc1c0070f90 0x7fc1c00713f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:17:43.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:43.437+0000 7fc1beffd640 1 --2- 172.21.15.40:0/410493384 >> v2:172.21.15.189:3300/0 conn(0x7fc1c0070f90 0x7fc1c00713f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T08:17:45.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:45.033+0000 7fc1bd7fa640 10 monclient(hunting): tick 2024-04-22T08:17:45.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:45.033+0000 7fc1bd7fa640 10 monclient(hunting): _check_auth_tickets 2024-04-22T08:17:45.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:45.033+0000 7fc1bd7fa640 1 monclient(hunting): continuing hunt 2024-04-22T08:17:45.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:45.033+0000 7fc1bd7fa640 10 monclient(hunting): _reopen_session rank -1 2024-04-22T08:17:45.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:45.033+0000 7fc1bd7fa640 1 -- 172.21.15.40:0/410493384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1c00a01d0 msgr2=0x7fc1c012f6e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:17:45.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:45.033+0000 7fc1bd7fa640 1 --2- 172.21.15.40:0/410493384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1c00a01d0 0x7fc1c012f6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:45.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:45.033+0000 7fc1bd7fa640 1 -- 172.21.15.40:0/410493384 >> v2:172.21.15.189:3300/0 conn(0x7fc1c0070f90 msgr2=0x7fc1c00713f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:17:45.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:45.033+0000 7fc1bd7fa640 1 --2- 172.21.15.40:0/410493384 >> v2:172.21.15.189:3300/0 conn(0x7fc1c0070f90 0x7fc1c00713f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:45.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:45.033+0000 7fc1bd7fa640 1 -- 172.21.15.40:0/410493384 >> v1:172.21.15.40:6789/0 conn(0x7fc1c006ce90 legacy=0x7fc1c006d300 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:17:45.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:45.033+0000 7fc1bd7fa640 10 monclient: _add_conns ranks=[1,3] 2024-04-22T08:17:45.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:45.033+0000 7fc1bd7fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc1ac0036c0 addr v1:172.21.15.189:6789/0 2024-04-22T08:17:45.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:45.033+0000 7fc1bd7fa640 1 --2- 172.21.15.40:0/410493384 >> v2:172.21.15.40:3300/0 conn(0x7fc1ac0073f0 0x7fc1ac009890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:45.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:45.033+0000 7fc1bd7fa640 10 monclient(hunting): picked mon.noname-a con 0x7fc1ac0073f0 addr v2:172.21.15.40:3300/0 2024-04-22T08:17:45.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:45.033+0000 7fc1bd7fa640 1 -- 172.21.15.40:0/410493384 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1c006f4a0 con 0x7fc1ac0036c0 2024-04-22T08:17:45.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:45.033+0000 7fc1bd7fa640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:45.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:45.033+0000 7fc1bd7fa640 1 -- 172.21.15.40:0/410493384 --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1c012f2f0 con 0x7fc1ac0073f0 2024-04-22T08:17:45.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:45.033+0000 7fc1bd7fa640 10 monclient(hunting): _renew_subs 2024-04-22T08:17:45.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:45.033+0000 7fc1beffd640 1 -- 172.21.15.40:0/410493384 >> v2:172.21.15.40:3300/0 conn(0x7fc1ac0073f0 msgr2=0x7fc1ac009890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:17:45.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:45.033+0000 7fc1beffd640 1 --2- 172.21.15.40:0/410493384 >> v2:172.21.15.40:3300/0 conn(0x7fc1ac0073f0 0x7fc1ac009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:17:45.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:45.036+0000 7fc1bf7fe640 1 -- 172.21.15.40:0/410493384 >> v1:172.21.15.189:6789/0 conn(0x7fc1ac0036c0 legacy=0x7fc1ac005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:17:45.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:45.234+0000 7fc1beffd640 1 -- 172.21.15.40:0/410493384 >> v2:172.21.15.40:3300/0 conn(0x7fc1ac0073f0 msgr2=0x7fc1ac009890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:17:45.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:45.234+0000 7fc1beffd640 1 --2- 172.21.15.40:0/410493384 >> v2:172.21.15.40:3300/0 conn(0x7fc1ac0073f0 0x7fc1ac009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:17:45.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:45.635+0000 7fc1beffd640 1 -- 172.21.15.40:0/410493384 >> v2:172.21.15.40:3300/0 conn(0x7fc1ac0073f0 msgr2=0x7fc1ac009890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:17:45.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:45.635+0000 7fc1beffd640 1 --2- 172.21.15.40:0/410493384 >> v2:172.21.15.40:3300/0 conn(0x7fc1ac0073f0 0x7fc1ac009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:17:46.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.436+0000 7fc1beffd640 1 -- 172.21.15.40:0/410493384 >> v2:172.21.15.40:3300/0 conn(0x7fc1ac0073f0 msgr2=0x7fc1ac009890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:17:46.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.436+0000 7fc1beffd640 1 --2- 172.21.15.40:0/410493384 >> v2:172.21.15.40:3300/0 conn(0x7fc1ac0073f0 0x7fc1ac009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T08:17:46.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.933+0000 7fc1be7fc640 1 -- 172.21.15.40:0/410493384 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1737186478 0 0) 0x7fc1c006f4a0 con 0x7fc1ac0036c0 2024-04-22T08:17:46.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.933+0000 7fc1be7fc640 10 monclient(hunting): _negotiate 2024-04-22T08:17:46.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.933+0000 7fc1be7fc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:46.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.933+0000 7fc1be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:46.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.933+0000 7fc1be7fc640 10 monclient(hunting): my global_id is 64153 2024-04-22T08:17:46.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.933+0000 7fc1be7fc640 1 -- 172.21.15.40:0/410493384 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1a8003520 con 0x7fc1ac0036c0 2024-04-22T08:17:46.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.933+0000 7fc1be7fc640 1 -- 172.21.15.40:0/410493384 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 566450455 0 0) 0x7fc1a8003520 con 0x7fc1ac0036c0 2024-04-22T08:17:46.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.933+0000 7fc1be7fc640 1 -- 172.21.15.40:0/410493384 >> v2:172.21.15.40:3300/0 conn(0x7fc1ac0073f0 msgr2=0x7fc1ac009890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:17:46.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.933+0000 7fc1be7fc640 1 --2- 172.21.15.40:0/410493384 >> v2:172.21.15.40:3300/0 conn(0x7fc1ac0073f0 0x7fc1ac009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:46.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.933+0000 7fc1be7fc640 10 monclient: _finish_hunting 0 2024-04-22T08:17:46.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.933+0000 7fc1be7fc640 1 monclient: found mon.noname-c-legacy 2024-04-22T08:17:46.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.933+0000 7fc1be7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:17:46.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.933+0000 7fc1be7fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.189:6789/0 2024-04-22T08:17:46.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.933+0000 7fc1be7fc640 1 -- 172.21.15.40:0/410493384 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1ac00afc0 con 0x7fc1ac0036c0 2024-04-22T08:17:46.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.933+0000 7fc1be7fc640 10 monclient: _finish_auth 0 2024-04-22T08:17:46.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.933+0000 7fc1be7fc640 10 monclient: _check_auth_tickets 2024-04-22T08:17:46.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.933+0000 7fc1be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:17:46.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.933+0000 7fc1be7fc640 1 -- 172.21.15.40:0/410493384 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1087006497 0 0) 0x7fc1b000ea90 con 0x7fc1ac0036c0 2024-04-22T08:17:46.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.933+0000 7fc1be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:46.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.933+0000 7fc1be7fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-04-22T08:17:46.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.933+0000 7fc1be7fc640 10 monclient: dump: 2024-04-22T08:17:46.936 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:46.936 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:46.937 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:46.937 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:46.937 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:46.937 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:46.937 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:46.937 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:46.937 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:46.937 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:46.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.933+0000 7fc1be7fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] but i'm connected to v1:172.21.15.189:6789/0, reconnecting 2024-04-22T08:17:46.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.933+0000 7fc1be7fc640 10 monclient: _reopen_session rank -1 2024-04-22T08:17:46.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.933+0000 7fc1be7fc640 1 -- 172.21.15.40:0/410493384 >> v1:172.21.15.189:6789/0 conn(0x7fc1ac0036c0 legacy=0x7fc1ac005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:46.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.933+0000 7fc1be7fc640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-22T08:17:46.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.933+0000 7fc1bffff640 1 -- 172.21.15.40:0/410493384 reap_dead start 2024-04-22T08:17:46.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.933+0000 7fc1be7fc640 1 --2- 172.21.15.40:0/410493384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1a8003970 0x7fc1a8003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:46.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.933+0000 7fc1be7fc640 10 monclient(hunting): picked mon.b con 0x7fc1a8003970 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:17:46.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.933+0000 7fc1be7fc640 1 --2- 172.21.15.40:0/410493384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1a80047c0 0x7fc1a8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:46.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.933+0000 7fc1be7fc640 10 monclient(hunting): picked mon.c con 0x7fc1a80047c0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:17:46.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.933+0000 7fc1be7fc640 1 --2- 172.21.15.40:0/410493384 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1a80052a0 0x7fc1a800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:46.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.933+0000 7fc1be7fc640 10 monclient(hunting): picked mon.a con 0x7fc1a80052a0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:17:46.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.933+0000 7fc1be7fc640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:46.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.933+0000 7fc1be7fc640 1 -- 172.21.15.40:0/410493384 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c012f2f0 con 0x7fc1a80052a0 2024-04-22T08:17:46.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.933+0000 7fc1be7fc640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:46.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.933+0000 7fc1be7fc640 1 -- 172.21.15.40:0/410493384 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a800c1f0 con 0x7fc1a8003970 2024-04-22T08:17:46.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.933+0000 7fc1be7fc640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:46.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.933+0000 7fc1be7fc640 1 -- 172.21.15.40:0/410493384 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a800c3b0 con 0x7fc1a80047c0 2024-04-22T08:17:46.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.933+0000 7fc1be7fc640 10 monclient(hunting): _renew_subs 2024-04-22T08:17:46.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.933+0000 7fc1be7fc640 10 monclient(hunting): _finish_auth 0 2024-04-22T08:17:46.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.933+0000 7fc1bffff640 1 -- 172.21.15.40:0/410493384 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1a80052a0 msgr2=0x7fc1a800b980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:17:46.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.933+0000 7fc1bffff640 1 --2- 172.21.15.40:0/410493384 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1a80052a0 0x7fc1a800b980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:17:46.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.934+0000 7fc1beffd640 1 --2- 172.21.15.40:0/410493384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1a80047c0 0x7fc1a8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:46.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.934+0000 7fc1beffd640 10 monclient(hunting): get_auth_request con 0x7fc1a80047c0 auth_method 0 2024-04-22T08:17:46.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.934+0000 7fc1beffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:46.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.934+0000 7fc1beffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:46.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.934+0000 7fc1beffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:17:46.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.934+0000 7fc1bf7fe640 1 --2- 172.21.15.40:0/410493384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1a8003970 0x7fc1a8003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:46.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.934+0000 7fc1bf7fe640 10 monclient(hunting): get_auth_request con 0x7fc1a8003970 auth_method 0 2024-04-22T08:17:46.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.934+0000 7fc1bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:46.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.934+0000 7fc1bf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:46.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.934+0000 7fc1bf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:17:46.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.934+0000 7fc1beffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:46.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.934+0000 7fc1beffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:46.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.934+0000 7fc1beffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-22T08:17:46.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.934+0000 7fc1bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:46.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.934+0000 7fc1bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:46.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.934+0000 7fc1bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-22T08:17:46.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.934+0000 7fc1beffd640 10 monclient(hunting): handle_auth_done global_id 64153 payload 293 2024-04-22T08:17:46.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.934+0000 7fc1beffd640 1 -- 172.21.15.40:0/410493384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1a8003970 msgr2=0x7fc1a8003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:17:46.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.934+0000 7fc1beffd640 1 --2- 172.21.15.40:0/410493384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1a8003970 0x7fc1a8003d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:46.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.934+0000 7fc1beffd640 1 -- 172.21.15.40:0/410493384 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1a80052a0 msgr2=0x7fc1a800b980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:17:46.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.934+0000 7fc1beffd640 1 --2- 172.21.15.40:0/410493384 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1a80052a0 0x7fc1a800b980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:46.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.934+0000 7fc1beffd640 10 monclient: _finish_hunting 0 2024-04-22T08:17:46.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.934+0000 7fc1beffd640 1 monclient: found mon.c 2024-04-22T08:17:46.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.934+0000 7fc1beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:17:46.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.934+0000 7fc1beffd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:17:46.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.934+0000 7fc1beffd640 1 -- 172.21.15.40:0/410493384 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1a8003520 con 0x7fc1a80047c0 2024-04-22T08:17:46.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.934+0000 7fc1beffd640 10 monclient: _finish_auth 0 2024-04-22T08:17:46.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.934+0000 7fc1beffd640 10 monclient: _check_auth_tickets 2024-04-22T08:17:46.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.934+0000 7fc1beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:17:46.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.934+0000 7fc1c6058640 5 monclient: authenticate success, global_id 64153 2024-04-22T08:17:46.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.934+0000 7fc1c6058640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:17:46.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.937+0000 7fc1beffd640 1 -- 172.21.15.40:0/410493384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1a80047c0 msgr2=0x7fc1a8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:17:46.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.937+0000 7fc1beffd640 1 --2- 172.21.15.40:0/410493384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1a80047c0 0x7fc1a8004c40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc1b400f2a0 tx=0x7fc1a80057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c9768adde55fd56 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:46.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.938+0000 7fc1be7fc640 1 -- 172.21.15.40:0/410493384 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b4035410 con 0x7fc1a80047c0 2024-04-22T08:17:46.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.938+0000 7fc1be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:46.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.938+0000 7fc1be7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:17:46.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.938+0000 7fc1be7fc640 10 monclient: dump: 2024-04-22T08:17:46.947 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:46.947 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:46.947 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:46.947 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:46.947 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:46.947 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:46.947 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:46.947 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:46.947 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:46.947 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:46.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.938+0000 7fc1be7fc640 1 -- 172.21.15.40:0/410493384 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1b402f030 con 0x7fc1a80047c0 2024-04-22T08:17:46.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.938+0000 7fc1c6058640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:17:46.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.938+0000 7fc1be7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:17:46.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.938+0000 7fc1be7fc640 1 -- 172.21.15.40:0/410493384 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b40357b0 con 0x7fc1a80047c0 2024-04-22T08:17:46.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.938+0000 7fc1c6058640 10 monclient: get_monmap_and_config success 2024-04-22T08:17:46.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.938+0000 7fc1c6058640 4 set_mon_vals no callback set 2024-04-22T08:17:46.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.938+0000 7fc1be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:46.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.938+0000 7fc1be7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:17:46.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.938+0000 7fc1be7fc640 10 monclient: dump: 2024-04-22T08:17:46.948 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:46.948 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:46.948 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:46.948 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:46.948 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:46.948 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:46.948 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:46.949 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:46.949 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:46.949 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:46.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.939+0000 7fc1c6058640 10 monclient: shutdown 2024-04-22T08:17:46.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.939+0000 7fc1c6058640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:17:46.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.939+0000 7fc1c6058640 1 -- 172.21.15.40:0/410493384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1a80047c0 msgr2=0x7fc1a8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:46.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.939+0000 7fc1c6058640 1 --2- 172.21.15.40:0/410493384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1a80047c0 0x7fc1a8004c40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc1b400f2a0 tx=0x7fc1a80057f0 comp rx=0 tx=0).stop 2024-04-22T08:17:46.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.939+0000 7fc1c6058640 1 -- 172.21.15.40:0/410493384 shutdown_connections 2024-04-22T08:17:46.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.939+0000 7fc1c6058640 1 --2- 172.21.15.40:0/410493384 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1a80052a0 0x7fc1a800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:46.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.939+0000 7fc1c6058640 1 --2- 172.21.15.40:0/410493384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1a80047c0 0x7fc1a8004c40 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:46.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.939+0000 7fc1c6058640 1 --2- 172.21.15.40:0/410493384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1a8003970 0x7fc1a8003d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:46.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.939+0000 7fc1c6058640 1 -- 172.21.15.40:0/410493384 >> 172.21.15.40:0/410493384 conn(0x7fc1c00a5450 msgr2=0x7fc1c00a12c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:17:46.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.939+0000 7fc1c6058640 1 -- 172.21.15.40:0/410493384 shutdown_connections 2024-04-22T08:17:46.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.939+0000 7fc1c6058640 1 -- 172.21.15.40:0/410493384 wait complete. 2024-04-22T08:17:46.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.940+0000 7fc1c6058640 10 monclient: build_initial_monmap 2024-04-22T08:17:46.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.940+0000 7fc1c6058640 10 monclient: monmap: 2024-04-22T08:17:46.950 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:17:46.950 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:17:46.950 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:17:46.950 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:17:46.950 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:17:46.950 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:46.950 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:17:46.950 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:17:46.950 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:17:46.950 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:46.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.940+0000 7fc1c6058640 1 Processor -- start 2024-04-22T08:17:46.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.940+0000 7fc1c6058640 1 -- start start 2024-04-22T08:17:46.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.940+0000 7fc1c6058640 10 monclient: init 2024-04-22T08:17:46.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.941+0000 7fc1c6058640 10 monclient: _reopen_session rank -1 2024-04-22T08:17:46.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.941+0000 7fc1c6058640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-22T08:17:46.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.941+0000 7fc1c6058640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1a8003970 0x7fc1c01c5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:46.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.941+0000 7fc1c6058640 10 monclient(hunting): picked mon.noname-c con 0x7fc1a8003970 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:17:46.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.941+0000 7fc1c6058640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1a80047c0 0x7fc1c01c5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:46.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.941+0000 7fc1c6058640 10 monclient(hunting): picked mon.noname-a con 0x7fc1a80047c0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:17:46.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.941+0000 7fc1c6058640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1a80052a0 0x7fc1c01c64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:46.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.941+0000 7fc1c6058640 10 monclient(hunting): picked mon.noname-b con 0x7fc1a80052a0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:17:46.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.941+0000 7fc1c6058640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:46.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.941+0000 7fc1c6058640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c009da40 con 0x7fc1a80047c0 2024-04-22T08:17:46.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.941+0000 7fc1c6058640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:46.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.941+0000 7fc1c6058640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c006e870 con 0x7fc1a80052a0 2024-04-22T08:17:46.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.941+0000 7fc1c6058640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:46.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.941+0000 7fc1c6058640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c009b5e0 con 0x7fc1a8003970 2024-04-22T08:17:46.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.941+0000 7fc1c6058640 10 monclient(hunting): _renew_subs 2024-04-22T08:17:46.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.941+0000 7fc1c6058640 10 monclient(hunting): authenticate will time out at 1140.935084s 2024-04-22T08:17:46.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.941+0000 7fc1beffd640 1 -- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1a80047c0 msgr2=0x7fc1c01c5f90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:17:46.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.941+0000 7fc1beffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1a80047c0 0x7fc1c01c5f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:17:46.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.941+0000 7fc1bf7fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1a8003970 0x7fc1c01c5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:46.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.941+0000 7fc1bf7fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1a8003970 0x7fc1c01c5a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:58802/0 (socket says 172.21.15.40:58802) 2024-04-22T08:17:46.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.941+0000 7fc1bf7fe640 1 -- 172.21.15.40:0/4028253268 learned_addr learned my addr 172.21.15.40:0/4028253268 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:17:46.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.941+0000 7fc1bf7fe640 10 monclient(hunting): get_auth_request con 0x7fc1a8003970 auth_method 0 2024-04-22T08:17:46.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.941+0000 7fc1bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:46.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.941+0000 7fc1bf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:46.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.941+0000 7fc1bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:46.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.941+0000 7fc1bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:46.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.941+0000 7fc1bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:46.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.941+0000 7fc1bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:46.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.941+0000 7fc1bffff640 1 --2- 172.21.15.40:0/4028253268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1a80052a0 0x7fc1c01c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:46.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.941+0000 7fc1bffff640 10 monclient(hunting): get_auth_request con 0x7fc1a80052a0 auth_method 0 2024-04-22T08:17:46.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.941+0000 7fc1bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:46.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.941+0000 7fc1bffff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:46.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.941+0000 7fc1bffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:46.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.941+0000 7fc1bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:46.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.941+0000 7fc1bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:46.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.941+0000 7fc1bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:46.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.941+0000 7fc1bf7fe640 10 monclient(hunting): handle_auth_done global_id 54353 payload 824 2024-04-22T08:17:46.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.941+0000 7fc1bf7fe640 1 -- 172.21.15.40:0/4028253268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1a80052a0 msgr2=0x7fc1c01c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:46.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.941+0000 7fc1bf7fe640 1 --2- 172.21.15.40:0/4028253268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1a80052a0 0x7fc1c01c64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:46.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.941+0000 7fc1bf7fe640 1 -- 172.21.15.40:0/4028253268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1a80047c0 msgr2=0x7fc1c01c5f90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:17:46.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.941+0000 7fc1bf7fe640 1 --2- 172.21.15.40:0/4028253268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1a80047c0 0x7fc1c01c5f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:46.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.941+0000 7fc1bf7fe640 10 monclient: _finish_hunting 0 2024-04-22T08:17:46.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.941+0000 7fc1bf7fe640 1 monclient: found mon.noname-c 2024-04-22T08:17:46.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.941+0000 7fc1bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:17:46.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.941+0000 7fc1bf7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:17:46.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.941+0000 7fc1bf7fe640 1 -- 172.21.15.40:0/4028253268 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c009bf30 con 0x7fc1a8003970 2024-04-22T08:17:46.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.944+0000 7fc1bf7fe640 1 -- 172.21.15.40:0/4028253268 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1a8003970 msgr2=0x7fc1c01c5a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:17:46.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.944+0000 7fc1bf7fe640 1 --2- 172.21.15.40:0/4028253268 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1a8003970 0x7fc1c01c5a50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc1b0013a20 tx=0x7fc1b0013ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:46.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.944+0000 7fc1a7fff640 1 -- 172.21.15.40:0/4028253268 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b0018070 con 0x7fc1a8003970 2024-04-22T08:17:46.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.944+0000 7fc1a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:46.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.944+0000 7fc1a7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:17:46.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.944+0000 7fc1a7fff640 10 monclient: dump: 2024-04-22T08:17:46.954 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:46.954 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:46.954 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:46.955 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:46.955 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:46.955 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:46.955 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:46.955 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:46.955 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:46.955 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:46.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.944+0000 7fc1a7fff640 10 monclient: _finish_auth 0 2024-04-22T08:17:46.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.944+0000 7fc1a7fff640 10 monclient: _check_auth_tickets 2024-04-22T08:17:46.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.944+0000 7fc1a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:17:46.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.944+0000 7fc1a7fff640 1 -- 172.21.15.40:0/4028253268 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1b0019040 con 0x7fc1a8003970 2024-04-22T08:17:46.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.944+0000 7fc1c6058640 5 monclient: authenticate success, global_id 54353 2024-04-22T08:17:46.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.944+0000 7fc1c6058640 10 monclient: _renew_subs 2024-04-22T08:17:46.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.944+0000 7fc1c6058640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:17:46.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.944+0000 7fc1c6058640 1 -- 172.21.15.40:0/4028253268 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1c009d110 con 0x7fc1a8003970 2024-04-22T08:17:46.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.944+0000 7fc1a7fff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:17:46.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.944+0000 7fc1a7fff640 1 -- 172.21.15.40:0/4028253268 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b0004050 con 0x7fc1a8003970 2024-04-22T08:17:46.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.944+0000 7fc1a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:46.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.944+0000 7fc1a7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:17:46.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.944+0000 7fc1a7fff640 10 monclient: dump: 2024-04-22T08:17:46.956 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:46.956 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:46.956 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:46.956 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:46.956 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:46.956 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:46.956 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:46.956 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:46.957 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:46.957 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:46.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.944+0000 7fc1c6058640 10 monclient: _renew_subs 2024-04-22T08:17:46.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.944+0000 7fc1c6058640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:17:46.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.944+0000 7fc1c6058640 1 -- 172.21.15.40:0/4028253268 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1c0099c50 con 0x7fc1a8003970 2024-04-22T08:17:46.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.944+0000 7fc1bdffb640 4 set_mon_vals no callback set 2024-04-22T08:17:46.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.945+0000 7fc1a7fff640 1 -- 172.21.15.40:0/4028253268 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fc1b0004270 con 0x7fc1a8003970 2024-04-22T08:17:46.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.945+0000 7fc1c6058640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:17:46.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.945+0000 7fc1c6058640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:17:46.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.945+0000 7fc1a7fff640 1 --2- 172.21.15.40:0/4028253268 >> v2:172.21.15.189:6800/30168 conn(0x7fc1ac038090 0x7fc1ac03a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:46.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.945+0000 7fc1c6058640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:17:46.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.945+0000 7fc1c6058640 1 -- 172.21.15.40:0/4028253268 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1c009d350 con 0x7fc1a8003970 2024-04-22T08:17:46.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.945+0000 7fc1a7fff640 1 -- 172.21.15.40:0/4028253268 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 3898+0+0 (secure 0 0 0) 0x7fc1b0000e40 con 0x7fc1a8003970 2024-04-22T08:17:46.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.946+0000 7fc1beffd640 1 --2- 172.21.15.40:0/4028253268 >> v2:172.21.15.189:6800/30168 conn(0x7fc1ac038090 0x7fc1ac03a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:46.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.946+0000 7fc1beffd640 10 monclient: get_auth_request con 0x7fc1ac038090 auth_method 0 2024-04-22T08:17:46.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.948+0000 7fc1beffd640 1 --2- 172.21.15.40:0/4028253268 >> v2:172.21.15.189:6800/30168 conn(0x7fc1ac038090 0x7fc1ac03a550 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fc1b4034000 tx=0x7fc1b4030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:46.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.951+0000 7fc1a7fff640 1 -- 172.21.15.40:0/4028253268 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7fc1c009d350 con 0x7fc1a8003970 2024-04-22T08:17:46.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.951+0000 7fc1a7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:17:46.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:46.951+0000 7fc1a7fff640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:17:47.107 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36988]: starting. 2024-04-22T08:17:47.107 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36988]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-36987 2024-04-22T08:17:47.107 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36988]: created object 0... 2024-04-22T08:17:47.107 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36988]: created object 25... 2024-04-22T08:17:47.107 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36988]: created object 49... 2024-04-22T08:17:47.107 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36988]: finishing. 2024-04-22T08:17:47.107 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[36988]: shutting down. 2024-04-22T08:17:47.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.218+0000 7fc1c6058640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:17:47.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.218+0000 7fc1c6058640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:17:47.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.218+0000 7fc1c6058640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:17:47.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.218+0000 7fc1c6058640 1 -- 172.21.15.40:0/4028253268 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc1c009d510 con 0x7fc1a8003970 2024-04-22T08:17:47.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.218+0000 7fc1a7fff640 1 -- 172.21.15.40:0/4028253268 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1537 (secure 0 0 0) 0x7fc1b003f070 con 0x7fc1a8003970 2024-04-22T08:17:47.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.218+0000 7fc1a7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:17:47.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.218+0000 7fc1a7fff640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:17:47.219 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":48,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:17:47.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.220+0000 7fc1c6058640 1 -- 172.21.15.40:0/4028253268 >> v2:172.21.15.189:6800/30168 conn(0x7fc1ac038090 msgr2=0x7fc1ac03a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:47.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.220+0000 7fc1c6058640 1 --2- 172.21.15.40:0/4028253268 >> v2:172.21.15.189:6800/30168 conn(0x7fc1ac038090 0x7fc1ac03a550 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fc1b4034000 tx=0x7fc1b4030000 comp rx=0 tx=0).stop 2024-04-22T08:17:47.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.220+0000 7fc1c6058640 10 monclient: shutdown 2024-04-22T08:17:47.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.220+0000 7fc1c6058640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:17:47.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.220+0000 7fc1c6058640 1 -- 172.21.15.40:0/4028253268 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1a8003970 msgr2=0x7fc1c01c5a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:47.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.220+0000 7fc1c6058640 1 --2- 172.21.15.40:0/4028253268 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1a8003970 0x7fc1c01c5a50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc1b0013a20 tx=0x7fc1b0013ee0 comp rx=0 tx=0).stop 2024-04-22T08:17:47.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.220+0000 7fc1c6058640 1 -- 172.21.15.40:0/4028253268 shutdown_connections 2024-04-22T08:17:47.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.220+0000 7fc1c6058640 1 --2- 172.21.15.40:0/4028253268 >> v2:172.21.15.189:6800/30168 conn(0x7fc1ac038090 0x7fc1ac03a550 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:47.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.220+0000 7fc1c6058640 1 --2- 172.21.15.40:0/4028253268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1a80052a0 0x7fc1c01c64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:47.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.220+0000 7fc1c6058640 1 --2- 172.21.15.40:0/4028253268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1a80047c0 0x7fc1c01c5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:47.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.220+0000 7fc1c6058640 1 --2- 172.21.15.40:0/4028253268 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1a8003970 0x7fc1c01c5a50 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:47.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.220+0000 7fc1c6058640 1 -- 172.21.15.40:0/4028253268 >> 172.21.15.40:0/4028253268 conn(0x7fc1c00a5450 msgr2=0x7fc1c012c350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:17:47.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.221+0000 7fc1c6058640 1 -- 172.21.15.40:0/4028253268 shutdown_connections 2024-04-22T08:17:47.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.221+0000 7fc1c6058640 1 -- 172.21.15.40:0/4028253268 wait complete. 2024-04-22T08:17:47.231 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-22T08:17:47.231 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-04-22T08:17:47.231 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-22T08:17:47.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.331+0000 7fa94e729640 10 monclient: get_monmap_and_config 2024-04-22T08:17:47.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.331+0000 7fa94e729640 10 monclient: build_initial_monmap 2024-04-22T08:17:47.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.331+0000 7fa94e729640 10 monclient: monmap: 2024-04-22T08:17:47.331 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:17:47.331 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:47.331 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:17:47.332580+0000 2024-04-22T08:17:47.331 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:17:47.332580+0000 2024-04-22T08:17:47.332 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:17:47.332 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:47.332 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:17:47.332 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:17:47.332 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:17:47.332 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:17:47.332 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:17:47.332 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:47.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.332+0000 7fa94e729640 1 Processor -- start 2024-04-22T08:17:47.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.332+0000 7fa94e729640 1 -- start start 2024-04-22T08:17:47.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.332+0000 7fa94e729640 10 monclient: init 2024-04-22T08:17:47.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.332+0000 7fa94e729640 10 monclient: _reopen_session rank -1 2024-04-22T08:17:47.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.332+0000 7fa94e729640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-04-22T08:17:47.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.332+0000 7fa94e729640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa94809f1b0 0x7fa94809f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:47.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.332+0000 7fa94e729640 10 monclient(hunting): picked mon.noname-c con 0x7fa94809f1b0 addr v2:172.21.15.189:3300/0 2024-04-22T08:17:47.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.332+0000 7fa94e729640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa948133930 0x7fa948133dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:47.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.332+0000 7fa94e729640 10 monclient(hunting): picked mon.noname-a con 0x7fa948133930 addr v2:172.21.15.40:3300/0 2024-04-22T08:17:47.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.332+0000 7fa94e729640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa948134300 addr v1:172.21.15.189:6789/0 2024-04-22T08:17:47.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.332+0000 7fa94e729640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa94809de90 con 0x7fa948134300 2024-04-22T08:17:47.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.332+0000 7fa94e729640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:47.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.332+0000 7fa94e729640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa94809dae0 con 0x7fa948133930 2024-04-22T08:17:47.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.332+0000 7fa94e729640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:47.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.332+0000 7fa94e729640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa948134730 con 0x7fa94809f1b0 2024-04-22T08:17:47.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.333+0000 7fa94e729640 10 monclient(hunting): _renew_subs 2024-04-22T08:17:47.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.333+0000 7fa94e729640 10 monclient(hunting): authenticate will time out at 1141.326782s 2024-04-22T08:17:47.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.333+0000 7fa9477fe640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7fa948133930 msgr2=0x7fa948133dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:17:47.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.333+0000 7fa9477fe640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa948133930 0x7fa948133dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:17:47.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.333+0000 7fa947fff640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa94809f1b0 0x7fa94809f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:47.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.333+0000 7fa94ccc6640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa948134300 0x7fa94813ac60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.40:60760/0 (socket says 172.21.15.40:60760) 2024-04-22T08:17:47.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.333+0000 7fa94ccc6640 1 -- 172.21.15.40:0/3993504643 learned_addr learned my addr 172.21.15.40:0/3993504643 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:17:47.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.333+0000 7fa947fff640 10 monclient(hunting): get_auth_request con 0x7fa94809f1b0 auth_method 0 2024-04-22T08:17:47.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.333+0000 7fa947fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:47.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.333+0000 7fa947fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:47.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.333+0000 7fa947fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:47.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.333+0000 7fa947fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:47.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.333+0000 7fa947fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:47.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.333+0000 7fa947fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:47.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.333+0000 7fa947fff640 10 monclient(hunting): handle_auth_done global_id 64159 payload 274 2024-04-22T08:17:47.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.333+0000 7fa947fff640 1 -- 172.21.15.40:0/3993504643 >> v2:172.21.15.40:3300/0 conn(0x7fa948133930 msgr2=0x7fa948133dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:17:47.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.333+0000 7fa947fff640 1 --2- 172.21.15.40:0/3993504643 >> v2:172.21.15.40:3300/0 conn(0x7fa948133930 0x7fa948133dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:47.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.333+0000 7fa947fff640 1 -- 172.21.15.40:0/3993504643 >> v1:172.21.15.189:6789/0 conn(0x7fa948134300 legacy=0x7fa94813ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:17:47.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.333+0000 7fa947fff640 10 monclient: _finish_hunting 0 2024-04-22T08:17:47.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.333+0000 7fa947fff640 1 monclient: found mon.noname-c 2024-04-22T08:17:47.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.333+0000 7fa947fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:17:47.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.333+0000 7fa947fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.189:3300/0 2024-04-22T08:17:47.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.333+0000 7fa947fff640 1 -- 172.21.15.40:0/3993504643 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa94806c720 con 0x7fa94809f1b0 2024-04-22T08:17:47.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.336+0000 7fa947fff640 1 -- 172.21.15.40:0/3993504643 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa94809f1b0 msgr2=0x7fa94809f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:17:47.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.336+0000 7fa947fff640 1 --2- 172.21.15.40:0/3993504643 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa94809f1b0 0x7fa94809f590 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa94809b030 tx=0x7fa93802b740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45f3a564e6118ae6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:47.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.336+0000 7fa946ffd640 1 -- 172.21.15.40:0/3993504643 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa938030a00 con 0x7fa94809f1b0 2024-04-22T08:17:47.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.336+0000 7fa946ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:47.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.336+0000 7fa946ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:17:47.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.336+0000 7fa946ffd640 10 monclient: dump: 2024-04-22T08:17:47.343 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:47.343 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:47.343 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:47.343 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:47.343 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:47.343 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:47.343 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:47.343 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:47.343 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:47.343 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:47.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.336+0000 7fa946ffd640 10 monclient: _finish_auth 0 2024-04-22T08:17:47.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.336+0000 7fa946ffd640 10 monclient: _check_auth_tickets 2024-04-22T08:17:47.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.336+0000 7fa946ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:17:47.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.336+0000 7fa94e729640 5 monclient: authenticate success, global_id 64159 2024-04-22T08:17:47.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.336+0000 7fa94e729640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:17:47.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.336+0000 7fa946ffd640 1 -- 172.21.15.40:0/3993504643 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa93802e050 con 0x7fa94809f1b0 2024-04-22T08:17:47.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.336+0000 7fa946ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:17:47.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.336+0000 7fa946ffd640 1 -- 172.21.15.40:0/3993504643 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa938030d80 con 0x7fa94809f1b0 2024-04-22T08:17:47.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.336+0000 7fa94e729640 10 monclient: get_monmap_and_config success 2024-04-22T08:17:47.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.336+0000 7fa94e729640 4 set_mon_vals no callback set 2024-04-22T08:17:47.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.336+0000 7fa946ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:47.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.336+0000 7fa946ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:17:47.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.336+0000 7fa946ffd640 10 monclient: dump: 2024-04-22T08:17:47.344 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:47.345 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:47.345 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:47.345 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:47.345 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:47.345 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:47.345 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:47.345 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:47.345 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:47.345 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:47.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.337+0000 7fa94e729640 10 monclient: shutdown 2024-04-22T08:17:47.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.337+0000 7fa94e729640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:17:47.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.337+0000 7fa94e729640 1 -- 172.21.15.40:0/3993504643 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa94809f1b0 msgr2=0x7fa94809f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:47.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.337+0000 7fa94e729640 1 --2- 172.21.15.40:0/3993504643 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa94809f1b0 0x7fa94809f590 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa94809b030 tx=0x7fa93802b740 comp rx=0 tx=0).stop 2024-04-22T08:17:47.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.337+0000 7fa94e729640 1 -- 172.21.15.40:0/3993504643 shutdown_connections 2024-04-22T08:17:47.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.337+0000 7fa94e729640 1 --2- 172.21.15.40:0/3993504643 >> v2:172.21.15.40:3300/0 conn(0x7fa948133930 0x7fa948133dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:47.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.337+0000 7fa94e729640 1 --2- 172.21.15.40:0/3993504643 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa94809f1b0 0x7fa94809f590 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:47.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.337+0000 7fa94e729640 1 -- 172.21.15.40:0/3993504643 >> 172.21.15.40:0/3993504643 conn(0x7fa9480a0ae0 msgr2=0x7fa9480a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:17:47.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.337+0000 7fa94e729640 1 -- 172.21.15.40:0/3993504643 shutdown_connections 2024-04-22T08:17:47.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.337+0000 7fa94e729640 1 -- 172.21.15.40:0/3993504643 wait complete. 2024-04-22T08:17:47.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.337+0000 7fa94e729640 10 monclient: build_initial_monmap 2024-04-22T08:17:47.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.337+0000 7fa94e729640 10 monclient: monmap: 2024-04-22T08:17:47.346 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:17:47.346 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:17:47.346 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:17:47.346 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:17:47.346 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:17:47.346 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:47.347 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:17:47.347 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:17:47.347 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:17:47.347 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:47.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.338+0000 7fa94e729640 1 Processor -- start 2024-04-22T08:17:47.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.338+0000 7fa94e729640 1 -- start start 2024-04-22T08:17:47.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.338+0000 7fa94e729640 10 monclient: init 2024-04-22T08:17:47.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.338+0000 7fa94e729640 10 monclient: _reopen_session rank -1 2024-04-22T08:17:47.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.338+0000 7fa94e729640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-22T08:17:47.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.338+0000 7fa94e729640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa94809f1b0 0x7fa9481cab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:47.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.338+0000 7fa94e729640 10 monclient(hunting): picked mon.noname-b con 0x7fa94809f1b0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:17:47.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.338+0000 7fa94e729640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa948133930 0x7fa94812f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:47.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.338+0000 7fa94e729640 10 monclient(hunting): picked mon.noname-c con 0x7fa948133930 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:17:47.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.338+0000 7fa94e729640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa948134300 0x7fa94812fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:47.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.338+0000 7fa94e729640 10 monclient(hunting): picked mon.noname-a con 0x7fa948134300 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:17:47.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.338+0000 7fa94e729640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:47.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.338+0000 7fa94e729640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9480705b0 con 0x7fa948134300 2024-04-22T08:17:47.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.338+0000 7fa94e729640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:47.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.338+0000 7fa94e729640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9480718b0 con 0x7fa94809f1b0 2024-04-22T08:17:47.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.338+0000 7fa94e729640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:47.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.338+0000 7fa94e729640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa948123190 con 0x7fa948133930 2024-04-22T08:17:47.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.338+0000 7fa94e729640 10 monclient(hunting): _renew_subs 2024-04-22T08:17:47.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.338+0000 7fa94e729640 10 monclient(hunting): authenticate will time out at 1141.332694s 2024-04-22T08:17:47.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.338+0000 7fa94ccc6640 1 -- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa948134300 msgr2=0x7fa94812fec0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:17:47.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.338+0000 7fa94ccc6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa948134300 0x7fa94812fec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:17:47.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.339+0000 7fa9477fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa948133930 0x7fa94812f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:47.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.339+0000 7fa9477fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa948133930 0x7fa94812f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:58810/0 (socket says 172.21.15.40:58810) 2024-04-22T08:17:47.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.339+0000 7fa9477fe640 1 -- 172.21.15.40:0/798200336 learned_addr learned my addr 172.21.15.40:0/798200336 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:17:47.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.339+0000 7fa9477fe640 10 monclient(hunting): get_auth_request con 0x7fa948133930 auth_method 0 2024-04-22T08:17:47.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.339+0000 7fa9477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:47.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.339+0000 7fa9477fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:47.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.339+0000 7fa9477fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:47.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.339+0000 7fa9477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:47.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.339+0000 7fa9477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:47.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.339+0000 7fa947fff640 1 --2- 172.21.15.40:0/798200336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa94809f1b0 0x7fa9481cab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:47.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.339+0000 7fa9477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:47.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.339+0000 7fa947fff640 10 monclient(hunting): get_auth_request con 0x7fa94809f1b0 auth_method 0 2024-04-22T08:17:47.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.339+0000 7fa947fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:47.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.339+0000 7fa947fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:47.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.339+0000 7fa947fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:47.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.339+0000 7fa947fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:47.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.339+0000 7fa947fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:47.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.339+0000 7fa947fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:47.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.339+0000 7fa9477fe640 10 monclient(hunting): handle_auth_done global_id 54356 payload 824 2024-04-22T08:17:47.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.339+0000 7fa9477fe640 1 -- 172.21.15.40:0/798200336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa94809f1b0 msgr2=0x7fa9481cab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:47.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.339+0000 7fa9477fe640 1 --2- 172.21.15.40:0/798200336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa94809f1b0 0x7fa9481cab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:47.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.339+0000 7fa9477fe640 1 -- 172.21.15.40:0/798200336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa948134300 msgr2=0x7fa94812fec0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:17:47.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.339+0000 7fa9477fe640 1 --2- 172.21.15.40:0/798200336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa948134300 0x7fa94812fec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:47.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.339+0000 7fa9477fe640 10 monclient: _finish_hunting 0 2024-04-22T08:17:47.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.339+0000 7fa9477fe640 1 monclient: found mon.noname-c 2024-04-22T08:17:47.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.339+0000 7fa9477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:17:47.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.339+0000 7fa9477fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:17:47.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.339+0000 7fa9477fe640 1 -- 172.21.15.40:0/798200336 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa938009400 con 0x7fa948133930 2024-04-22T08:17:47.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.341+0000 7fa9477fe640 1 -- 172.21.15.40:0/798200336 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa948133930 msgr2=0x7fa94812f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:17:47.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.341+0000 7fa9477fe640 1 --2- 172.21.15.40:0/798200336 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa948133930 0x7fa94812f980 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa93c00b550 tx=0x7fa93c00ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:47.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.341+0000 7fa944ff9640 1 -- 172.21.15.40:0/798200336 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa93c005070 con 0x7fa948133930 2024-04-22T08:17:47.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.341+0000 7fa944ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:47.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.341+0000 7fa944ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:17:47.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.341+0000 7fa944ff9640 10 monclient: dump: 2024-04-22T08:17:47.351 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:47.351 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:47.351 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:47.351 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:47.351 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:47.351 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:47.351 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:47.351 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:47.351 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:47.351 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:47.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.341+0000 7fa944ff9640 10 monclient: _finish_auth 0 2024-04-22T08:17:47.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.341+0000 7fa944ff9640 10 monclient: _check_auth_tickets 2024-04-22T08:17:47.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.341+0000 7fa944ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:17:47.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.341+0000 7fa944ff9640 1 -- 172.21.15.40:0/798200336 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa93c0051f0 con 0x7fa948133930 2024-04-22T08:17:47.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.341+0000 7fa94e729640 5 monclient: authenticate success, global_id 54356 2024-04-22T08:17:47.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.341+0000 7fa94e729640 10 monclient: _renew_subs 2024-04-22T08:17:47.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.341+0000 7fa94e729640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:17:47.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.341+0000 7fa94e729640 1 -- 172.21.15.40:0/798200336 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9481228f0 con 0x7fa948133930 2024-04-22T08:17:47.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.341+0000 7fa944ff9640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:17:47.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.341+0000 7fa944ff9640 1 -- 172.21.15.40:0/798200336 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa93c0055a0 con 0x7fa948133930 2024-04-22T08:17:47.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.341+0000 7fa944ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:47.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.341+0000 7fa944ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:17:47.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.341+0000 7fa944ff9640 10 monclient: dump: 2024-04-22T08:17:47.352 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:47.352 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:47.352 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:47.352 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:47.352 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:47.353 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:47.353 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:47.353 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:47.353 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:47.353 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:47.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.341+0000 7fa94e729640 10 monclient: _renew_subs 2024-04-22T08:17:47.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.341+0000 7fa94e729640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:17:47.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.341+0000 7fa94e729640 1 -- 172.21.15.40:0/798200336 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9481cb0d0 con 0x7fa948133930 2024-04-22T08:17:47.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.341+0000 7fa945ffb640 4 set_mon_vals no callback set 2024-04-22T08:17:47.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.342+0000 7fa944ff9640 1 -- 172.21.15.40:0/798200336 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fa93c005700 con 0x7fa948133930 2024-04-22T08:17:47.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.342+0000 7fa94e729640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:17:47.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.342+0000 7fa94e729640 1 --2- 172.21.15.40:0/798200336 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa94809b4e0 0x7fa94809b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:47.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.342+0000 7fa94e729640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:47.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.342+0000 7fa94e729640 1 -- 172.21.15.40:0/798200336 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9480667a0 con 0x7fa94809b4e0 2024-04-22T08:17:47.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.342+0000 7fa947fff640 1 --2- 172.21.15.40:0/798200336 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa94809b4e0 0x7fa94809b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:47.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.342+0000 7fa947fff640 10 monclient: get_auth_request con 0x7fa94809b4e0 auth_method 0 2024-04-22T08:17:47.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.342+0000 7fa947fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:47.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.342+0000 7fa947fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:47.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.342+0000 7fa947fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:47.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.342+0000 7fa944ff9640 1 --2- 172.21.15.40:0/798200336 >> v2:172.21.15.189:6800/30168 conn(0x7fa934037780 0x7fa934039c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:47.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.342+0000 7fa947fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:47.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.342+0000 7fa947fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:47.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.342+0000 7fa944ff9640 1 -- 172.21.15.40:0/798200336 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 3898+0+0 (secure 0 0 0) 0x7fa93c04a300 con 0x7fa948133930 2024-04-22T08:17:47.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.342+0000 7fa947fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:47.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.343+0000 7fa947fff640 10 monclient(hunting): handle_auth_done global_id 54359 payload 824 2024-04-22T08:17:47.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.343+0000 7fa947fff640 1 -- 172.21.15.40:0/798200336 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa94809fad0 con 0x7fa94809b4e0 2024-04-22T08:17:47.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.343+0000 7fa94ccc6640 1 --2- 172.21.15.40:0/798200336 >> v2:172.21.15.189:6800/30168 conn(0x7fa934037780 0x7fa934039c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:47.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.343+0000 7fa94ccc6640 10 monclient: get_auth_request con 0x7fa934037780 auth_method 0 2024-04-22T08:17:47.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.345+0000 7fa947fff640 1 -- 172.21.15.40:0/798200336 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa94809b4e0 msgr2=0x7fa94809b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:17:47.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.345+0000 7fa947fff640 1 --2- 172.21.15.40:0/798200336 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa94809b4e0 0x7fa94809b8c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa938034000 tx=0x7fa9380046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:47.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.345+0000 7fa944ff9640 1 -- 172.21.15.40:0/798200336 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa93803cc90 con 0x7fa94809b4e0 2024-04-22T08:17:47.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.345+0000 7fa944ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:47.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.345+0000 7fa944ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:17:47.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.345+0000 7fa944ff9640 10 monclient: dump: 2024-04-22T08:17:47.358 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:47.358 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:47.358 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:47.358 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:47.358 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:47.358 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:47.358 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:47.358 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:47.358 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:47.358 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:47.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.345+0000 7fa94ccc6640 1 --2- 172.21.15.40:0/798200336 >> v2:172.21.15.189:6800/30168 conn(0x7fa934037780 0x7fa934039c40 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa9481223d0 tx=0x7fa940005080 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:47.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.345+0000 7fa944ff9640 1 -- 172.21.15.40:0/798200336 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7fa93803cdf0 con 0x7fa94809b4e0 2024-04-22T08:17:47.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.345+0000 7fa944ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:17:47.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.345+0000 7fa944ff9640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:17:47.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.345+0000 7fa944ff9640 1 -- 172.21.15.40:0/798200336 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa94809b4e0 msgr2=0x7fa94809b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:47.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.345+0000 7fa944ff9640 1 --2- 172.21.15.40:0/798200336 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa94809b4e0 0x7fa94809b8c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa938034000 tx=0x7fa9380046c0 comp rx=0 tx=0).stop 2024-04-22T08:17:47.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.345+0000 7fa944ff9640 1 -- 172.21.15.40:0/798200336 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa94809b4e0 msgr2=0x7fa94809b8c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:17:47.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.345+0000 7fa944ff9640 1 --2- 172.21.15.40:0/798200336 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa94809b4e0 0x7fa94809b8c0 secure :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa938034000 tx=0x7fa9380046c0 comp rx=0 tx=0).stop 2024-04-22T08:17:47.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.352+0000 7fa94e729640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:17:47.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.352+0000 7fa94e729640 1 --2- 172.21.15.40:0/798200336 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa94809be00 0x7fa94806e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:47.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.352+0000 7fa94e729640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:47.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.352+0000 7fa94e729640 1 -- 172.21.15.40:0/798200336 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9480a0490 con 0x7fa94809be00 2024-04-22T08:17:47.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.352+0000 7fa947fff640 1 --2- 172.21.15.40:0/798200336 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa94809be00 0x7fa94806e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:47.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.352+0000 7fa947fff640 10 monclient: get_auth_request con 0x7fa94809be00 auth_method 0 2024-04-22T08:17:47.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.352+0000 7fa947fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:47.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.352+0000 7fa947fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:47.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.352+0000 7fa947fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:47.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.352+0000 7fa947fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:47.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.352+0000 7fa947fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:47.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.352+0000 7fa947fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:47.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.352+0000 7fa947fff640 10 monclient(hunting): handle_auth_done global_id 54362 payload 824 2024-04-22T08:17:47.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.352+0000 7fa947fff640 1 -- 172.21.15.40:0/798200336 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa94809eec0 con 0x7fa94809be00 2024-04-22T08:17:47.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.354+0000 7fa947fff640 1 -- 172.21.15.40:0/798200336 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa94809be00 msgr2=0x7fa94806e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:17:47.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.354+0000 7fa947fff640 1 --2- 172.21.15.40:0/798200336 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa94809be00 0x7fa94806e8e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa948121650 tx=0x7fa938030000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:47.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.354+0000 7fa944ff9640 1 -- 172.21.15.40:0/798200336 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa938003e30 con 0x7fa94809be00 2024-04-22T08:17:47.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.354+0000 7fa944ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:47.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.354+0000 7fa944ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:17:47.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.354+0000 7fa944ff9640 10 monclient: dump: 2024-04-22T08:17:47.360 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:47.360 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:47.360 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:47.361 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:47.361 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:47.361 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:47.361 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:47.361 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:47.361 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:47.361 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:47.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.354+0000 7fa944ff9640 1 -- 172.21.15.40:0/798200336 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4790 (secure 0 0 0) 0x7fa93804f640 con 0x7fa94809be00 2024-04-22T08:17:47.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.354+0000 7fa944ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:17:47.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.354+0000 7fa944ff9640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:17:47.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.354+0000 7fa944ff9640 1 -- 172.21.15.40:0/798200336 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa94809be00 msgr2=0x7fa94806e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:47.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.354+0000 7fa944ff9640 1 --2- 172.21.15.40:0/798200336 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa94809be00 0x7fa94806e8e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa948121650 tx=0x7fa938030000 comp rx=0 tx=0).stop 2024-04-22T08:17:47.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.354+0000 7fa944ff9640 1 -- 172.21.15.40:0/798200336 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa94809be00 msgr2=0x7fa94806e8e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:17:47.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.354+0000 7fa944ff9640 1 --2- 172.21.15.40:0/798200336 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa94809be00 0x7fa94806e8e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:47.362 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:17:47.362 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:17:47.362 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:17:47.362 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:17:47.362 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 48, 2024-04-22T08:17:47.362 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:17:47.362 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:17:47.362 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:17:47.362 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:17:47.362 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 0, 2024-04-22T08:17:47.362 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:17:47.363 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:17:47.363 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:17:47.363 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:17:47.363 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:17:47.363 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:17:47.363 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:17:47.363 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:17:47.363 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:17:47.363 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:17:47.363 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:17:47.363 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:17:47.363 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:17:47.363 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:17:47.364 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:17:47.364 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:17:47.364 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:17:47.364 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:17:47.364 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:17:47.364 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:17:47.364 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:17:47.364 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:17:47.364 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:17:47.364 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:17:47.364 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:17:47.364 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:17:47.364 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:47.364 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:47.365 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:17:47.365 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:17:47.365 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:17:47.365 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:17:47.365 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:17:47.365 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:17:47.365 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:17:47.365 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:17:47.365 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:17:47.365 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:17:47.365 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:17:47.365 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:17:47.365 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:17:47.365 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:17:47.366 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:17:47.366 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:17:47.366 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:17:47.366 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:17:47.366 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:17:47.366 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:17:47.366 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:17:47.366 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:17:47.366 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:17:47.366 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:17:47.366 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:17:47.366 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:17:47.367 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:17:47.367 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:17:47.367 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:17:47.367 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:47.367 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:17:47.367 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:47.367 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:17:47.367 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:17:47.367 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:17:47.367 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:17:47.367 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:47.367 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:17:47.367 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:17:47.368 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:47.368 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:47.368 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:47.368 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:17:47.368 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:17:47.368 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:47.368 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:47.368 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:47.368 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:47.368 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:17:47.368 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:17:47.369 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:17:47.369 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:17:47.369 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:17:47.369 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:47.369 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:47.369 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:17:47.369 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:17:47.369 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:17:47.369 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:17:47.369 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:47.369 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:17:47.369 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:17:47.369 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:47.369 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:47.369 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:47.370 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:17:47.370 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:17:47.370 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:47.370 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:47.370 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:47.370 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:47.370 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:17:47.370 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:17:47.370 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:17:47.370 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:17:47.370 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:17:47.370 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:47.370 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:47.370 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:17:47.371 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:17:47.371 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:17:47.371 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:17:47.371 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:47.371 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:17:47.371 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:17:47.371 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:47.371 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:47.371 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:47.371 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:17:47.371 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:17:47.371 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:47.371 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:47.372 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:47.372 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:47.372 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:17:47.372 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:17:47.372 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:17:47.372 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:17:47.372 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:17:47.372 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:47.372 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:47.372 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:47.372 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:17:47.372 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:17:47.372 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:47.372 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:17:47.372 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:17:47.373 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:17:47.373 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:47.373 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:17:47.373 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:17:47.373 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:47.373 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:17:47.373 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:17:47.373 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 3 2024-04-22T08:17:47.373 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:47.373 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:17:47.373 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:17:47.373 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:47.373 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:17:47.373 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:17:47.373 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 5 2024-04-22T08:17:47.374 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:47.374 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:17:47.374 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": [ 2024-04-22T08:17:47.374 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:47.374 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:17:47.374 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:17:47.374 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:17:47.374 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:47.374 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:47.374 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:47.374 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:17:47.374 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:17:47.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.355+0000 7fa94e729640 1 -- 172.21.15.40:0/798200336 >> v2:172.21.15.189:6800/30168 conn(0x7fa934037780 msgr2=0x7fa934039c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:47.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.355+0000 7fa94e729640 1 --2- 172.21.15.40:0/798200336 >> v2:172.21.15.189:6800/30168 conn(0x7fa934037780 0x7fa934039c40 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa9481223d0 tx=0x7fa940005080 comp rx=0 tx=0).stop 2024-04-22T08:17:47.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.355+0000 7fa94e729640 10 monclient: shutdown 2024-04-22T08:17:47.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.355+0000 7fa94e729640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:17:47.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.355+0000 7fa94e729640 1 -- 172.21.15.40:0/798200336 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa948133930 msgr2=0x7fa94812f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:47.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.355+0000 7fa94e729640 1 --2- 172.21.15.40:0/798200336 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa948133930 0x7fa94812f980 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa93c00b550 tx=0x7fa93c00ba10 comp rx=0 tx=0).stop 2024-04-22T08:17:47.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.355+0000 7fa94ccc6640 1 -- 172.21.15.40:0/798200336 reap_dead start 2024-04-22T08:17:47.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.355+0000 7fa94ccc6640 1 -- 172.21.15.40:0/798200336 reap_dead start 2024-04-22T08:17:47.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.355+0000 7fa94e729640 1 -- 172.21.15.40:0/798200336 shutdown_connections 2024-04-22T08:17:47.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.355+0000 7fa94e729640 1 -- 172.21.15.40:0/798200336 >> 172.21.15.40:0/798200336 conn(0x7fa9480a0ae0 msgr2=0x7fa9480a2d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:17:47.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.355+0000 7fa94e729640 1 -- 172.21.15.40:0/798200336 shutdown_connections 2024-04-22T08:17:47.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.355+0000 7fa94e729640 1 -- 172.21.15.40:0/798200336 wait complete. 2024-04-22T08:17:47.377 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-22T08:17:47.377 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-04-22T08:17:47.377 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-22T08:17:47.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.477+0000 7faa085e4640 10 monclient: get_monmap_and_config 2024-04-22T08:17:47.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.477+0000 7faa085e4640 10 monclient: build_initial_monmap 2024-04-22T08:17:47.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.477+0000 7faa085e4640 10 monclient: monmap: 2024-04-22T08:17:47.477 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:17:47.477 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:47.478 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:17:47.478399+0000 2024-04-22T08:17:47.478 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:17:47.478399+0000 2024-04-22T08:17:47.478 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:17:47.478 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:47.478 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:17:47.478 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:17:47.478 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:17:47.478 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:17:47.478 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:17:47.478 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:47.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.478+0000 7faa085e4640 1 Processor -- start 2024-04-22T08:17:47.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.478+0000 7faa085e4640 1 -- start start 2024-04-22T08:17:47.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.478+0000 7faa085e4640 10 monclient: init 2024-04-22T08:17:47.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.478+0000 7faa085e4640 10 monclient: _reopen_session rank -1 2024-04-22T08:17:47.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.478+0000 7faa085e4640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-04-22T08:17:47.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.478+0000 7faa085e4640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7faa0012fad0 0x7faa0012feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:47.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.478+0000 7faa085e4640 10 monclient(hunting): picked mon.noname-c con 0x7faa0012fad0 addr v2:172.21.15.189:3300/0 2024-04-22T08:17:47.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.478+0000 7faa085e4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faa00132920 addr v1:172.21.15.189:6789/0 2024-04-22T08:17:47.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.478+0000 7faa085e4640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7faa00133460 0x7faa0013ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:47.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.478+0000 7faa085e4640 10 monclient(hunting): picked mon.noname-a con 0x7faa00133460 addr v2:172.21.15.40:3300/0 2024-04-22T08:17:47.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.478+0000 7faa085e4640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa0009ae50 con 0x7faa00132920 2024-04-22T08:17:47.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.478+0000 7faa085e4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:47.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.478+0000 7faa085e4640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa0009efc0 con 0x7faa00133460 2024-04-22T08:17:47.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.478+0000 7faa085e4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:47.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.478+0000 7faa085e4640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa0009d070 con 0x7faa0012fad0 2024-04-22T08:17:47.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.478+0000 7faa085e4640 10 monclient(hunting): _renew_subs 2024-04-22T08:17:47.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.478+0000 7faa085e4640 10 monclient(hunting): authenticate will time out at 1141.472610s 2024-04-22T08:17:47.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.478+0000 7faa06b81640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7faa00133460 msgr2=0x7faa0013ac60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:17:47.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.478+0000 7faa06b81640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7faa00133460 0x7faa0013ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:17:47.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.479+0000 7faa06380640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7faa0012fad0 0x7faa0012feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:47.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.479+0000 7faa05b7f640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7faa00132920 0x7faa00132d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.40:60764/0 (socket says 172.21.15.40:60764) 2024-04-22T08:17:47.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.479+0000 7faa05b7f640 1 -- 172.21.15.40:0/2217275113 learned_addr learned my addr 172.21.15.40:0/2217275113 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:17:47.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.479+0000 7faa06380640 10 monclient(hunting): get_auth_request con 0x7faa0012fad0 auth_method 0 2024-04-22T08:17:47.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.479+0000 7faa06380640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:47.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.479+0000 7faa06380640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:47.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.479+0000 7faa06380640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:47.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.479+0000 7faa06380640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:47.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.479+0000 7faa06380640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:47.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.479+0000 7faa06380640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:47.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.479+0000 7faa06380640 10 monclient(hunting): handle_auth_done global_id 64165 payload 274 2024-04-22T08:17:47.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.479+0000 7faa06380640 1 -- 172.21.15.40:0/2217275113 >> v2:172.21.15.40:3300/0 conn(0x7faa00133460 msgr2=0x7faa0013ac60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:17:47.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.479+0000 7faa06380640 1 --2- 172.21.15.40:0/2217275113 >> v2:172.21.15.40:3300/0 conn(0x7faa00133460 0x7faa0013ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:47.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.479+0000 7faa06380640 1 -- 172.21.15.40:0/2217275113 >> v1:172.21.15.189:6789/0 conn(0x7faa00132920 legacy=0x7faa00132d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:17:47.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.479+0000 7faa06380640 10 monclient: _finish_hunting 0 2024-04-22T08:17:47.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.479+0000 7faa06380640 1 monclient: found mon.noname-c 2024-04-22T08:17:47.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.479+0000 7faa06380640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:17:47.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.479+0000 7faa06380640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.189:3300/0 2024-04-22T08:17:47.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.479+0000 7faa06380640 1 -- 172.21.15.40:0/2217275113 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa0009d350 con 0x7faa0012fad0 2024-04-22T08:17:47.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.482+0000 7faa06380640 1 -- 172.21.15.40:0/2217275113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa0012fad0 msgr2=0x7faa0012feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:17:47.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.482+0000 7faa06380640 1 --2- 172.21.15.40:0/2217275113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa0012fad0 0x7faa0012feb0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7faa0006bb30 tx=0x7fa9fc02b740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a418aaa20e74ee57 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:47.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.482+0000 7faa0537e640 1 -- 172.21.15.40:0/2217275113 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9fc030a00 con 0x7faa0012fad0 2024-04-22T08:17:47.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.482+0000 7faa0537e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:47.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.482+0000 7faa0537e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:17:47.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.482+0000 7faa0537e640 10 monclient: dump: 2024-04-22T08:17:47.491 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:47.492 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:47.492 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:47.492 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:47.492 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:47.492 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:47.492 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:47.492 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:47.492 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:47.492 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:47.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.482+0000 7faa0537e640 10 monclient: _finish_auth 0 2024-04-22T08:17:47.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.482+0000 7faa0537e640 10 monclient: _check_auth_tickets 2024-04-22T08:17:47.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.482+0000 7faa0537e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:17:47.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.482+0000 7faa0537e640 1 -- 172.21.15.40:0/2217275113 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9fc02e050 con 0x7faa0012fad0 2024-04-22T08:17:47.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.482+0000 7faa0537e640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:17:47.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.482+0000 7faa0537e640 1 -- 172.21.15.40:0/2217275113 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9fc030d80 con 0x7faa0012fad0 2024-04-22T08:17:47.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.482+0000 7faa085e4640 5 monclient: authenticate success, global_id 64165 2024-04-22T08:17:47.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.482+0000 7faa085e4640 10 monclient: get_monmap_and_config success 2024-04-22T08:17:47.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.482+0000 7faa085e4640 4 set_mon_vals no callback set 2024-04-22T08:17:47.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.482+0000 7faa0537e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:47.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.482+0000 7faa0537e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:17:47.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.482+0000 7faa0537e640 10 monclient: dump: 2024-04-22T08:17:47.493 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:47.494 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:47.494 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:47.494 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:47.494 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:47.494 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:47.494 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:47.494 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:47.494 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:47.494 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:47.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.483+0000 7faa085e4640 10 monclient: shutdown 2024-04-22T08:17:47.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.483+0000 7faa085e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:17:47.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.483+0000 7faa085e4640 1 -- 172.21.15.40:0/2217275113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa0012fad0 msgr2=0x7faa0012feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:47.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.483+0000 7faa085e4640 1 --2- 172.21.15.40:0/2217275113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa0012fad0 0x7faa0012feb0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7faa0006bb30 tx=0x7fa9fc02b740 comp rx=0 tx=0).stop 2024-04-22T08:17:47.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.483+0000 7faa085e4640 1 -- 172.21.15.40:0/2217275113 shutdown_connections 2024-04-22T08:17:47.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.483+0000 7faa085e4640 1 --2- 172.21.15.40:0/2217275113 >> v2:172.21.15.40:3300/0 conn(0x7faa00133460 0x7faa0013ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:47.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.483+0000 7faa085e4640 1 --2- 172.21.15.40:0/2217275113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa0012fad0 0x7faa0012feb0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:47.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.483+0000 7faa085e4640 1 -- 172.21.15.40:0/2217275113 >> 172.21.15.40:0/2217275113 conn(0x7faa000a0ae0 msgr2=0x7faa000a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:17:47.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.483+0000 7faa085e4640 1 -- 172.21.15.40:0/2217275113 shutdown_connections 2024-04-22T08:17:47.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.483+0000 7faa085e4640 1 -- 172.21.15.40:0/2217275113 wait complete. 2024-04-22T08:17:47.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.483+0000 7faa085e4640 10 monclient: build_initial_monmap 2024-04-22T08:17:47.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.483+0000 7faa085e4640 10 monclient: monmap: 2024-04-22T08:17:47.495 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:17:47.495 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:17:47.495 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:17:47.495 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:17:47.496 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:17:47.496 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:47.496 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:17:47.496 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:17:47.496 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:17:47.496 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:47.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.484+0000 7faa085e4640 1 Processor -- start 2024-04-22T08:17:47.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.484+0000 7faa085e4640 1 -- start start 2024-04-22T08:17:47.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.484+0000 7faa085e4640 10 monclient: init 2024-04-22T08:17:47.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.484+0000 7faa085e4640 10 monclient: _reopen_session rank -1 2024-04-22T08:17:47.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.484+0000 7faa085e4640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-22T08:17:47.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.484+0000 7faa085e4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa0012fad0 0x7faa001c8990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:47.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.484+0000 7faa085e4640 10 monclient(hunting): picked mon.noname-b con 0x7faa0012fad0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:17:47.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.484+0000 7faa085e4640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faa00132920 0x7faa001c8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:47.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.484+0000 7faa085e4640 10 monclient(hunting): picked mon.noname-c con 0x7faa00132920 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:17:47.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.484+0000 7faa085e4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa00133460 0x7faa0011d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:47.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.484+0000 7faa085e4640 10 monclient(hunting): picked mon.noname-a con 0x7faa00133460 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:17:47.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.484+0000 7faa085e4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:47.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.484+0000 7faa085e4640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa000a3820 con 0x7faa00133460 2024-04-22T08:17:47.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.484+0000 7faa085e4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:47.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.484+0000 7faa085e4640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa0011f0a0 con 0x7faa0012fad0 2024-04-22T08:17:47.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.484+0000 7faa085e4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:47.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.484+0000 7faa085e4640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa00071870 con 0x7faa00132920 2024-04-22T08:17:47.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.484+0000 7faa085e4640 10 monclient(hunting): _renew_subs 2024-04-22T08:17:47.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.484+0000 7faa085e4640 10 monclient(hunting): authenticate will time out at 1141.478489s 2024-04-22T08:17:47.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.484+0000 7faa06b81640 1 -- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa00133460 msgr2=0x7faa0011d080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:17:47.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.484+0000 7faa06b81640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa00133460 0x7faa0011d080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:17:47.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.484+0000 7faa05b7f640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faa00132920 0x7faa001c8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:47.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.484+0000 7faa05b7f640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faa00132920 0x7faa001c8ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:58830/0 (socket says 172.21.15.40:58830) 2024-04-22T08:17:47.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.484+0000 7faa05b7f640 1 -- 172.21.15.40:0/1731719379 learned_addr learned my addr 172.21.15.40:0/1731719379 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:17:47.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.484+0000 7faa05b7f640 10 monclient(hunting): get_auth_request con 0x7faa00132920 auth_method 0 2024-04-22T08:17:47.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.484+0000 7faa05b7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:47.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.484+0000 7faa05b7f640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:47.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.484+0000 7faa05b7f640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:47.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.484+0000 7faa05b7f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:47.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.484+0000 7faa05b7f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:47.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.484+0000 7faa06380640 1 --2- 172.21.15.40:0/1731719379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa0012fad0 0x7faa001c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:47.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.484+0000 7faa05b7f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:47.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.484+0000 7faa06380640 10 monclient(hunting): get_auth_request con 0x7faa0012fad0 auth_method 0 2024-04-22T08:17:47.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.484+0000 7faa06380640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:47.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.484+0000 7faa06380640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:47.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.484+0000 7faa06380640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:47.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.485+0000 7faa06380640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:47.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.485+0000 7faa06380640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:47.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.485+0000 7faa06380640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:47.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.485+0000 7faa05b7f640 10 monclient(hunting): handle_auth_done global_id 54365 payload 824 2024-04-22T08:17:47.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.485+0000 7faa05b7f640 1 -- 172.21.15.40:0/1731719379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa0012fad0 msgr2=0x7faa001c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:47.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.485+0000 7faa05b7f640 1 --2- 172.21.15.40:0/1731719379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa0012fad0 0x7faa001c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:47.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.485+0000 7faa05b7f640 1 -- 172.21.15.40:0/1731719379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa00133460 msgr2=0x7faa0011d080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:17:47.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.485+0000 7faa05b7f640 1 --2- 172.21.15.40:0/1731719379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa00133460 0x7faa0011d080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:17:47.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.485+0000 7faa05b7f640 10 monclient: _finish_hunting 0 2024-04-22T08:17:47.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.485+0000 7faa05b7f640 1 monclient: found mon.noname-c 2024-04-22T08:17:47.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.485+0000 7faa05b7f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:17:47.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.485+0000 7faa05b7f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:17:47.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.485+0000 7faa05b7f640 1 -- 172.21.15.40:0/1731719379 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9fc009400 con 0x7faa00132920 2024-04-22T08:17:47.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.487+0000 7faa05b7f640 1 -- 172.21.15.40:0/1731719379 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faa00132920 msgr2=0x7faa001c8ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:17:47.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.487+0000 7faa05b7f640 1 --2- 172.21.15.40:0/1731719379 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faa00132920 0x7faa001c8ed0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa9f000c460 tx=0x7fa9f000c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:47.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.487+0000 7fa9f6ffd640 1 -- 172.21.15.40:0/1731719379 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f0007070 con 0x7faa00132920 2024-04-22T08:17:47.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.487+0000 7fa9f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:47.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.487+0000 7fa9f6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:17:47.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.487+0000 7fa9f6ffd640 10 monclient: dump: 2024-04-22T08:17:47.501 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:47.501 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:47.501 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:47.501 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:47.501 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:47.501 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:47.501 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:47.501 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:47.501 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:47.502 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:47.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.487+0000 7fa9f6ffd640 10 monclient: _finish_auth 0 2024-04-22T08:17:47.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.487+0000 7fa9f6ffd640 10 monclient: _check_auth_tickets 2024-04-22T08:17:47.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.487+0000 7fa9f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:17:47.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.487+0000 7fa9f6ffd640 1 -- 172.21.15.40:0/1731719379 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9f00071f0 con 0x7faa00132920 2024-04-22T08:17:47.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.487+0000 7faa085e4640 5 monclient: authenticate success, global_id 54365 2024-04-22T08:17:47.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.487+0000 7faa085e4640 10 monclient: _renew_subs 2024-04-22T08:17:47.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.487+0000 7faa085e4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:17:47.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.487+0000 7faa085e4640 1 -- 172.21.15.40:0/1731719379 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa00120c80 con 0x7faa00132920 2024-04-22T08:17:47.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.487+0000 7fa9f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:17:47.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.487+0000 7fa9f6ffd640 1 -- 172.21.15.40:0/1731719379 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f00075a0 con 0x7faa00132920 2024-04-22T08:17:47.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.487+0000 7fa9f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:47.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.487+0000 7fa9f6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:17:47.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.487+0000 7fa9f6ffd640 10 monclient: dump: 2024-04-22T08:17:47.503 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:47.503 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:47.503 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:47.503 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:47.503 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:47.503 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:47.503 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:47.503 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:47.503 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:47.503 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:47.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.487+0000 7fa9f7fff640 4 set_mon_vals no callback set 2024-04-22T08:17:47.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.487+0000 7faa085e4640 10 monclient: _renew_subs 2024-04-22T08:17:47.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.487+0000 7faa085e4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:17:47.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.487+0000 7faa085e4640 1 -- 172.21.15.40:0/1731719379 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa001c9410 con 0x7faa00132920 2024-04-22T08:17:47.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.488+0000 7faa085e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:17:47.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.488+0000 7faa085e4640 1 --2- 172.21.15.40:0/1731719379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa0011d5c0 0x7faa0011d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:47.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.488+0000 7faa085e4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:47.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.488+0000 7faa085e4640 1 -- 172.21.15.40:0/1731719379 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa000667a0 con 0x7faa0011d5c0 2024-04-22T08:17:47.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.488+0000 7fa9f6ffd640 1 -- 172.21.15.40:0/1731719379 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fa9f001b400 con 0x7faa00132920 2024-04-22T08:17:47.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.488+0000 7fa9f6ffd640 1 --2- 172.21.15.40:0/1731719379 >> v2:172.21.15.189:6800/30168 conn(0x7fa9e4037780 0x7fa9e4039c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:47.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.488+0000 7faa06380640 1 --2- 172.21.15.40:0/1731719379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa0011d5c0 0x7faa0011d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:47.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.488+0000 7fa9f6ffd640 1 -- 172.21.15.40:0/1731719379 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 3898+0+0 (secure 0 0 0) 0x7fa9f0049f40 con 0x7faa00132920 2024-04-22T08:17:47.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.488+0000 7faa06380640 10 monclient: get_auth_request con 0x7faa0011d5c0 auth_method 0 2024-04-22T08:17:47.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.488+0000 7faa06380640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:47.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.488+0000 7faa06380640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:47.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.488+0000 7faa06380640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:47.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.488+0000 7faa06380640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:47.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.488+0000 7faa06380640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:47.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.488+0000 7faa06380640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:47.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.488+0000 7faa06b81640 1 --2- 172.21.15.40:0/1731719379 >> v2:172.21.15.189:6800/30168 conn(0x7fa9e4037780 0x7fa9e4039c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:47.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.489+0000 7faa06b81640 10 monclient: get_auth_request con 0x7fa9e4037780 auth_method 0 2024-04-22T08:17:47.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.489+0000 7faa06380640 10 monclient(hunting): handle_auth_done global_id 64171 payload 824 2024-04-22T08:17:47.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.489+0000 7faa06380640 1 -- 172.21.15.40:0/1731719379 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7faa0009e2f0 con 0x7faa0011d5c0 2024-04-22T08:17:47.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.491+0000 7faa06380640 1 -- 172.21.15.40:0/1731719379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa0011d5c0 msgr2=0x7faa0011d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:17:47.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.491+0000 7faa06380640 1 --2- 172.21.15.40:0/1731719379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa0011d5c0 0x7faa0011d9a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7faa000725d0 tx=0x7fa9fc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:47.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.491+0000 7faa06b81640 1 --2- 172.21.15.40:0/1731719379 >> v2:172.21.15.189:6800/30168 conn(0x7fa9e4037780 0x7fa9e4039c40 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa9f8003cb0 tx=0x7fa9f80091f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:47.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.491+0000 7fa9f6ffd640 1 -- 172.21.15.40:0/1731719379 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9fc032a50 con 0x7faa0011d5c0 2024-04-22T08:17:47.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.491+0000 7fa9f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:47.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.491+0000 7fa9f6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:17:47.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.491+0000 7fa9f6ffd640 10 monclient: dump: 2024-04-22T08:17:47.506 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:47.506 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:47.506 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:47.506 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:47.506 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:47.506 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:47.507 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:47.507 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:47.507 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:47.507 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:47.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.491+0000 7fa9f6ffd640 1 -- 172.21.15.40:0/1731719379 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7fa9fc032bb0 con 0x7faa0011d5c0 2024-04-22T08:17:47.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.491+0000 7fa9f6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:17:47.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.491+0000 7fa9f6ffd640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:17:47.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.491+0000 7fa9f6ffd640 1 -- 172.21.15.40:0/1731719379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa0011d5c0 msgr2=0x7faa0011d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:47.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.491+0000 7fa9f6ffd640 1 --2- 172.21.15.40:0/1731719379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa0011d5c0 0x7faa0011d9a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7faa000725d0 tx=0x7fa9fc0046c0 comp rx=0 tx=0).stop 2024-04-22T08:17:47.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.491+0000 7fa9f6ffd640 1 -- 172.21.15.40:0/1731719379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa0011d5c0 msgr2=0x7faa0011d9a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:17:47.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.491+0000 7fa9f6ffd640 1 --2- 172.21.15.40:0/1731719379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa0011d5c0 0x7faa0011d9a0 secure :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7faa000725d0 tx=0x7fa9fc0046c0 comp rx=0 tx=0).stop 2024-04-22T08:17:47.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.498+0000 7faa085e4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:17:47.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.498+0000 7faa085e4640 1 --2- 172.21.15.40:0/1731719379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa0011dee0 0x7faa0006f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:17:47.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.498+0000 7faa085e4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:17:47.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.498+0000 7faa085e4640 1 -- 172.21.15.40:0/1731719379 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa00120900 con 0x7faa0011dee0 2024-04-22T08:17:47.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.498+0000 7faa06380640 1 --2- 172.21.15.40:0/1731719379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa0011dee0 0x7faa0006f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:17:47.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.498+0000 7faa06380640 10 monclient: get_auth_request con 0x7faa0011dee0 auth_method 0 2024-04-22T08:17:47.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.498+0000 7faa06380640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:17:47.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.498+0000 7faa06380640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:17:47.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.498+0000 7faa06380640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:17:47.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.498+0000 7faa06380640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:17:47.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.498+0000 7faa06380640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:17:47.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.498+0000 7faa06380640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:17:47.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.498+0000 7faa06380640 10 monclient(hunting): handle_auth_done global_id 64174 payload 824 2024-04-22T08:17:47.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.498+0000 7faa06380640 1 -- 172.21.15.40:0/1731719379 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7faa0004e9b0 con 0x7faa0011dee0 2024-04-22T08:17:47.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.501+0000 7faa06380640 1 -- 172.21.15.40:0/1731719379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa0011dee0 msgr2=0x7faa0006f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:17:47.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.501+0000 7faa06380640 1 --2- 172.21.15.40:0/1731719379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa0011dee0 0x7faa0006f740 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc000c00 tx=0x7fa9fc009190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:17:47.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.501+0000 7fa9f6ffd640 1 -- 172.21.15.40:0/1731719379 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9fc04f890 con 0x7faa0011dee0 2024-04-22T08:17:47.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.501+0000 7fa9f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:17:47.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.501+0000 7fa9f6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:17:47.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.501+0000 7fa9f6ffd640 10 monclient: dump: 2024-04-22T08:17:47.511 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:17:47.511 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:17:47.512 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:47.512 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:17:47.512 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:17:47.512 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:17:47.512 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:17:47.512 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:17:47.512 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:17:47.512 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:17:47.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.501+0000 7fa9f6ffd640 1 -- 172.21.15.40:0/1731719379 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4622 (secure 0 0 0) 0x7fa9fc03de90 con 0x7faa0011dee0 2024-04-22T08:17:47.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.501+0000 7fa9f6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:17:47.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.501+0000 7fa9f6ffd640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:17:47.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.501+0000 7fa9f6ffd640 1 -- 172.21.15.40:0/1731719379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa0011dee0 msgr2=0x7faa0006f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:47.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.501+0000 7fa9f6ffd640 1 --2- 172.21.15.40:0/1731719379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa0011dee0 0x7faa0006f740 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc000c00 tx=0x7fa9fc009190 comp rx=0 tx=0).stop 2024-04-22T08:17:47.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.501+0000 7fa9f6ffd640 1 -- 172.21.15.40:0/1731719379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa0011dee0 msgr2=0x7faa0006f740 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:17:47.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.501+0000 7fa9f6ffd640 1 --2- 172.21.15.40:0/1731719379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa0011dee0 0x7faa0006f740 secure :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc000c00 tx=0x7fa9fc009190 comp rx=0 tx=0).stop 2024-04-22T08:17:47.513 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:17:47.513 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:17:47.513 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:17:47.513 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "leader", 2024-04-22T08:17:47.514 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 48, 2024-04-22T08:17:47.514 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:17:47.514 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:17:47.514 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:17:47.514 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:17:47.514 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 0, 2024-04-22T08:17:47.514 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:17:47.514 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:17:47.514 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:17:47.514 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:17:47.514 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:17:47.515 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:17:47.515 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:17:47.515 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:17:47.515 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:17:47.515 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:17:47.515 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:17:47.515 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:17:47.515 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:17:47.515 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:17:47.515 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:17:47.515 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:17:47.515 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:17:47.516 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:17:47.516 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:17:47.516 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:17:47.516 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:17:47.516 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:17:47.516 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:17:47.516 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:17:47.516 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:17:47.516 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:17:47.516 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:47.516 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:47.516 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:17:47.517 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:17:47.517 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:17:47.517 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:17:47.517 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:17:47.517 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:17:47.517 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:17:47.517 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:17:47.517 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:17:47.517 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:17:47.517 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:17:47.517 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:17:47.517 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:17:47.518 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:17:47.518 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:17:47.518 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:17:47.518 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:17:47.518 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:17:47.518 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:17:47.518 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:17:47.518 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:17:47.518 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:17:47.518 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:17:47.518 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:17:47.518 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:17:47.519 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:17:47.519 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:17:47.519 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:17:47.519 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:17:47.519 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:47.519 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:17:47.519 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:47.519 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:17:47.519 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:17:47.519 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:17:47.519 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:17:47.520 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:47.520 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:17:47.520 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:17:47.520 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:47.520 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:47.520 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:47.520 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:17:47.520 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:17:47.520 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:47.520 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:47.520 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:47.520 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:47.521 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:17:47.521 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:17:47.521 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:17:47.521 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:17:47.521 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:17:47.521 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:47.521 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:47.521 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:17:47.521 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:17:47.521 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:17:47.521 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:17:47.521 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:47.522 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:17:47.522 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:17:47.522 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:47.522 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:47.522 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:47.522 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:17:47.522 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:17:47.522 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:47.522 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:47.522 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:47.522 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:47.522 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:17:47.523 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:17:47.523 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:17:47.523 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:17:47.523 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:17:47.523 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:47.523 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:47.523 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:17:47.523 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:17:47.523 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:17:47.523 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:17:47.523 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:47.523 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:17:47.524 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:17:47.524 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:47.524 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:47.524 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:47.524 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:17:47.524 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:17:47.524 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:17:47.524 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:47.524 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:47.524 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:47.524 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:17:47.524 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:17:47.524 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:17:47.525 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:17:47.525 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:17:47.525 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:47.525 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:47.525 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:47.525 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:17:47.525 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:17:47.525 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:47.525 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:17:47.525 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:17:47.525 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:17:47.526 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:47.526 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:17:47.526 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:17:47.526 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:47.526 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:17:47.526 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:17:47.526 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 3 2024-04-22T08:17:47.526 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:47.526 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:17:47.526 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:17:47.526 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:17:47.526 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:17:47.526 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:17:47.527 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 3 2024-04-22T08:17:47.527 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:17:47.527 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:17:47.527 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:17:47.527 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:17:47.527 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:17:47.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.502+0000 7faa085e4640 1 -- 172.21.15.40:0/1731719379 >> v2:172.21.15.189:6800/30168 conn(0x7fa9e4037780 msgr2=0x7fa9e4039c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:47.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.502+0000 7faa085e4640 1 --2- 172.21.15.40:0/1731719379 >> v2:172.21.15.189:6800/30168 conn(0x7fa9e4037780 0x7fa9e4039c40 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa9f8003cb0 tx=0x7fa9f80091f0 comp rx=0 tx=0).stop 2024-04-22T08:17:47.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.502+0000 7faa085e4640 10 monclient: shutdown 2024-04-22T08:17:47.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.502+0000 7faa085e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:17:47.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.502+0000 7faa085e4640 1 -- 172.21.15.40:0/1731719379 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faa00132920 msgr2=0x7faa001c8ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:17:47.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.502+0000 7faa085e4640 1 --2- 172.21.15.40:0/1731719379 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faa00132920 0x7faa001c8ed0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa9f000c460 tx=0x7fa9f000c920 comp rx=0 tx=0).stop 2024-04-22T08:17:47.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.502+0000 7faa06b81640 1 -- 172.21.15.40:0/1731719379 reap_dead start 2024-04-22T08:17:47.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.502+0000 7faa06b81640 1 -- 172.21.15.40:0/1731719379 reap_dead start 2024-04-22T08:17:47.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.502+0000 7faa085e4640 1 -- 172.21.15.40:0/1731719379 shutdown_connections 2024-04-22T08:17:47.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.502+0000 7faa085e4640 1 -- 172.21.15.40:0/1731719379 >> 172.21.15.40:0/1731719379 conn(0x7faa000a0ae0 msgr2=0x7faa0009f270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:17:47.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.502+0000 7faa085e4640 1 -- 172.21.15.40:0/1731719379 shutdown_connections 2024-04-22T08:17:47.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:17:47.502+0000 7faa085e4640 1 -- 172.21.15.40:0/1731719379 wait complete. 2024-04-22T08:17:47.530 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-22T08:17:47.530 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-04-22T08:17:47.998 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[36989]: starting. 2024-04-22T08:17:47.998 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[36989]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-36987 2024-04-22T08:17:47.998 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[36989]: shutting down. 2024-04-22T08:17:50.126 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:17:50.126 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37022]: starting. 2024-04-22T08:17:50.126 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37022]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-36987 2024-04-22T08:17:50.126 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37022]: created object 0... 2024-04-22T08:17:50.126 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37022]: created object 25... 2024-04-22T08:17:50.126 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37022]: created object 49... 2024-04-22T08:17:50.126 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37022]: finishing. 2024-04-22T08:17:50.127 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37022]: shutting down. 2024-04-22T08:17:50.368 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:17:50.368 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37024]: starting. 2024-04-22T08:17:50.368 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37024]: listing objects. 2024-04-22T08:17:50.368 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37024]: listed object 0... 2024-04-22T08:17:50.369 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37024]: listed object 25... 2024-04-22T08:17:50.369 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37024]: saw 50 objects 2024-04-22T08:17:50.369 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37024]: shutting down. 2024-04-22T08:17:51.015 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:17:51.015 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[37023]: starting. 2024-04-22T08:17:51.015 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[37023]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-36987 2024-04-22T08:17:51.015 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[37023]: shutting down. 2024-04-22T08:17:51.017 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:17:51.017 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:17:51.017 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:17:51.019 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:17:53.148 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37093]: starting. 2024-04-22T08:17:53.148 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37093]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-37092 2024-04-22T08:17:53.148 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37093]: created object 0... 2024-04-22T08:17:53.148 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37093]: created object 25... 2024-04-22T08:17:53.148 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37093]: created object 49... 2024-04-22T08:17:53.148 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37093]: finishing. 2024-04-22T08:17:53.148 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37093]: shutting down. 2024-04-22T08:17:53.946 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[37094]: starting. 2024-04-22T08:17:53.946 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[37094]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-37092 2024-04-22T08:17:53.946 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[37094]: shutting down. 2024-04-22T08:17:56.066 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:17:56.067 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37130]: starting. 2024-04-22T08:17:56.067 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37130]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-37092 2024-04-22T08:17:56.067 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37130]: created object 0... 2024-04-22T08:17:56.067 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37130]: created object 25... 2024-04-22T08:17:56.067 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37130]: created object 49... 2024-04-22T08:17:56.067 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37130]: finishing. 2024-04-22T08:17:56.067 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37130]: shutting down. 2024-04-22T08:17:56.106 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:17:56.106 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37132]: starting. 2024-04-22T08:17:56.106 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37132]: listing objects. 2024-04-22T08:17:56.106 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37132]: listed object 0... 2024-04-22T08:17:56.106 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37132]: listed object 25... 2024-04-22T08:17:56.106 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37132]: saw 50 objects 2024-04-22T08:17:56.106 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37132]: shutting down. 2024-04-22T08:17:56.962 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:17:56.962 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[37131]: starting. 2024-04-22T08:17:56.962 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[37131]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-37092 2024-04-22T08:17:56.962 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[37131]: shutting down. 2024-04-22T08:17:56.963 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:17:56.963 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:17:56.963 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:17:56.966 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:17:59.107 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37201]: starting. 2024-04-22T08:17:59.107 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37201]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-37200 2024-04-22T08:17:59.107 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37201]: created object 0... 2024-04-22T08:17:59.107 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37201]: created object 25... 2024-04-22T08:17:59.107 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37201]: created object 49... 2024-04-22T08:17:59.107 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37201]: finishing. 2024-04-22T08:17:59.108 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37201]: shutting down. 2024-04-22T08:17:59.983 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[37202]: starting. 2024-04-22T08:17:59.984 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[37202]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-37200 2024-04-22T08:17:59.984 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[37202]: shutting down. 2024-04-22T08:18:02.825 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:18:02.825 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37237]: starting. 2024-04-22T08:18:02.825 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37237]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-37200 2024-04-22T08:18:02.825 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37237]: created object 0... 2024-04-22T08:18:02.826 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37237]: created object 25... 2024-04-22T08:18:02.826 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37237]: created object 49... 2024-04-22T08:18:02.826 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37237]: finishing. 2024-04-22T08:18:02.826 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37237]: shutting down. 2024-04-22T08:18:02.865 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:18:02.865 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37239]: starting. 2024-04-22T08:18:02.865 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37239]: listing objects. 2024-04-22T08:18:02.865 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37239]: listed object 0... 2024-04-22T08:18:02.865 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37239]: listed object 25... 2024-04-22T08:18:02.865 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37239]: saw 50 objects 2024-04-22T08:18:02.865 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37239]: shutting down. 2024-04-22T08:18:03.897 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:18:03.897 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[37238]: starting. 2024-04-22T08:18:03.897 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[37238]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-37200 2024-04-22T08:18:03.897 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[37238]: shutting down. 2024-04-22T08:18:03.898 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:18:03.898 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:18:03.898 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:18:03.900 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:18:06.070 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37308]: starting. 2024-04-22T08:18:06.070 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37308]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-37307 2024-04-22T08:18:06.070 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37308]: created object 0... 2024-04-22T08:18:06.070 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37308]: created object 25... 2024-04-22T08:18:06.070 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37308]: created object 49... 2024-04-22T08:18:06.070 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37308]: finishing. 2024-04-22T08:18:06.070 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37308]: shutting down. 2024-04-22T08:18:06.967 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[37309]: starting. 2024-04-22T08:18:06.967 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[37309]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-37307 2024-04-22T08:18:06.967 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[37309]: shutting down. 2024-04-22T08:18:07.531 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-04-22T08:18:07.531 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-04-22T08:18:07.531 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-22T08:18:07.531 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-22T08:18:07.535 INFO:tasks.ceph.mon.a:Started 2024-04-22T08:18:07.535 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-04-22T08:18:07.535 INFO:tasks.ceph.mon.c:Sent signal 19 2024-04-22T08:18:07.535 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-04-22T08:18:22.537 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-04-22T08:18:22.537 INFO:tasks.ceph.mon.c:Sent signal 18 2024-04-22T08:18:22.537 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-22T08:18:22.538 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:18:22.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.651+0000 7f3128ba5640 10 monclient: get_monmap_and_config 2024-04-22T08:18:22.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.651+0000 7f3128ba5640 10 monclient: build_initial_monmap 2024-04-22T08:18:22.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.651+0000 7f3128ba5640 10 monclient: monmap: 2024-04-22T08:18:22.651 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:18:22.651 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:22.651 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:18:22.652045+0000 2024-04-22T08:18:22.651 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:18:22.652045+0000 2024-04-22T08:18:22.651 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:18:22.651 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:22.651 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:18:22.651 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:18:22.651 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:18:22.651 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:18:22.651 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:18:22.651 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:22.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.652+0000 7f3128ba5640 1 Processor -- start 2024-04-22T08:18:22.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.652+0000 7f3128ba5640 1 -- start start 2024-04-22T08:18:22.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.652+0000 7f3128ba5640 10 monclient: init 2024-04-22T08:18:22.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.652+0000 7f3128ba5640 10 monclient: _reopen_session rank -1 2024-04-22T08:18:22.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.652+0000 7f3128ba5640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-04-22T08:18:22.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.652+0000 7f3128ba5640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3124132b00 0x7f312409af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:22.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.652+0000 7f3128ba5640 10 monclient(hunting): picked mon.noname-c con 0x7f3124132b00 addr v2:172.21.15.189:3300/0 2024-04-22T08:18:22.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.652+0000 7f3128ba5640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3124132ee0 0x7f31240a0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:22.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.652+0000 7f3128ba5640 10 monclient(hunting): picked mon.noname-a con 0x7f3124132ee0 addr v2:172.21.15.40:3300/0 2024-04-22T08:18:22.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.652+0000 7f3128ba5640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3124133370 0x7f31240a0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:22.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.652+0000 7f3128ba5640 10 monclient(hunting): picked mon.noname-b con 0x7f3124133370 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:18:22.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.652+0000 7f3128ba5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:22.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.652+0000 7f3128ba5640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f312409bb40 con 0x7f3124132ee0 2024-04-22T08:18:22.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.652+0000 7f3128ba5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:22.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.652+0000 7f3128ba5640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f312406bca0 con 0x7f3124132b00 2024-04-22T08:18:22.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.652+0000 7f3128ba5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:22.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.652+0000 7f3128ba5640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f312409a220 con 0x7f3124133370 2024-04-22T08:18:22.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.652+0000 7f3128ba5640 10 monclient(hunting): _renew_subs 2024-04-22T08:18:22.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.652+0000 7f3128ba5640 10 monclient(hunting): authenticate will time out at 1176.646355s 2024-04-22T08:18:22.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.652+0000 7f3122d9d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3124133370 0x7f31240a0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:22.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.652+0000 7f311bfff640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3124132ee0 0x7f31240a0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:22.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.652+0000 7f3122d9d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3124133370 0x7f31240a0f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:51314/0 (socket says 172.21.15.40:51314) 2024-04-22T08:18:22.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.652+0000 7f311bfff640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3124132ee0 0x7f31240a0a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44526/0 (socket says 172.21.15.40:44526) 2024-04-22T08:18:22.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.652+0000 7f3122d9d640 1 -- 172.21.15.40:0/4056357385 learned_addr learned my addr 172.21.15.40:0/4056357385 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:18:22.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.652+0000 7f3122d9d640 10 monclient(hunting): get_auth_request con 0x7f3124133370 auth_method 0 2024-04-22T08:18:22.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.652+0000 7f3122d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:22.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.652+0000 7f3122d9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:22.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.652+0000 7f3122d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:22.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.652+0000 7f311bfff640 10 monclient(hunting): get_auth_request con 0x7f3124132ee0 auth_method 0 2024-04-22T08:18:22.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.652+0000 7f311bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:22.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.652+0000 7f311bfff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:22.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.652+0000 7f311bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:22.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.652+0000 7f3122d9d640 1 -- 172.21.15.40:0/4056357385 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3124133370 msgr2=0x7f31240a0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:18:22.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.652+0000 7f311bfff640 1 -- 172.21.15.40:0/4056357385 >> v2:172.21.15.40:3300/0 conn(0x7f3124132ee0 msgr2=0x7f31240a0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:18:22.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.652+0000 7f3122d9d640 1 -- 172.21.15.40:0/4056357385 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3124133370 msgr2=0x7f31240a0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:18:22.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.652+0000 7f311bfff640 1 -- 172.21.15.40:0/4056357385 >> v2:172.21.15.40:3300/0 conn(0x7f3124132ee0 msgr2=0x7f31240a0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:18:22.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.652+0000 7f3122d9d640 1 --2- 172.21.15.40:0/4056357385 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3124133370 0x7f31240a0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:18:22.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.652+0000 7f311bfff640 1 --2- 172.21.15.40:0/4056357385 >> v2:172.21.15.40:3300/0 conn(0x7f3124132ee0 0x7f31240a0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:18:22.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.652+0000 7f3122d9d640 1 --2- 172.21.15.40:0/4056357385 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3124133370 0x7f31240a0f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:18:22.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.652+0000 7f311bfff640 1 --2- 172.21.15.40:0/4056357385 >> v2:172.21.15.40:3300/0 conn(0x7f3124132ee0 0x7f31240a0a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:18:22.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.653+0000 7f312259c640 1 --2- 172.21.15.40:0/4056357385 >> v2:172.21.15.189:3300/0 conn(0x7f3124132b00 0x7f312409af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:22.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.653+0000 7f312259c640 10 monclient(hunting): get_auth_request con 0x7f3124132b00 auth_method 0 2024-04-22T08:18:22.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.653+0000 7f312259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:22.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.653+0000 7f312259c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:22.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.653+0000 7f312259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:22.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.653+0000 7f312259c640 1 -- 172.21.15.40:0/4056357385 >> v2:172.21.15.189:3300/0 conn(0x7f3124132b00 msgr2=0x7f312409af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-22T08:18:22.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.653+0000 7f312259c640 1 -- 172.21.15.40:0/4056357385 >> v2:172.21.15.189:3300/0 conn(0x7f3124132b00 msgr2=0x7f312409af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:18:22.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.653+0000 7f312259c640 1 --2- 172.21.15.40:0/4056357385 >> v2:172.21.15.189:3300/0 conn(0x7f3124132b00 0x7f312409af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:18:22.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.653+0000 7f312259c640 1 --2- 172.21.15.40:0/4056357385 >> v2:172.21.15.189:3300/0 conn(0x7f3124132b00 0x7f312409af70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:18:22.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.853+0000 7f3122d9d640 1 --2- 172.21.15.40:0/4056357385 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3124133370 0x7f31240a0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:22.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.853+0000 7f3122d9d640 10 monclient(hunting): get_auth_request con 0x7f3124133370 auth_method 0 2024-04-22T08:18:22.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.853+0000 7f3122d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:22.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.853+0000 7f3122d9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:22.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.853+0000 7f3122d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:18:22.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.853+0000 7f311bfff640 1 --2- 172.21.15.40:0/4056357385 >> v2:172.21.15.40:3300/0 conn(0x7f3124132ee0 0x7f31240a0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:22.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.853+0000 7f311bfff640 10 monclient(hunting): get_auth_request con 0x7f3124132ee0 auth_method 0 2024-04-22T08:18:22.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.853+0000 7f311bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:22.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.853+0000 7f311bfff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:22.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.853+0000 7f311bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:18:22.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.853+0000 7f3122d9d640 1 -- 172.21.15.40:0/4056357385 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3124133370 msgr2=0x7f31240a0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:18:22.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.853+0000 7f3122d9d640 1 -- 172.21.15.40:0/4056357385 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3124133370 msgr2=0x7f31240a0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:18:22.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.853+0000 7f3122d9d640 1 --2- 172.21.15.40:0/4056357385 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3124133370 0x7f31240a0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:18:22.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.853+0000 7f3122d9d640 1 --2- 172.21.15.40:0/4056357385 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3124133370 0x7f31240a0f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:18:22.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.853+0000 7f311bfff640 1 -- 172.21.15.40:0/4056357385 >> v2:172.21.15.40:3300/0 conn(0x7f3124132ee0 msgr2=0x7f31240a0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:18:22.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.853+0000 7f311bfff640 1 -- 172.21.15.40:0/4056357385 >> v2:172.21.15.40:3300/0 conn(0x7f3124132ee0 msgr2=0x7f31240a0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:18:22.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.853+0000 7f311bfff640 1 --2- 172.21.15.40:0/4056357385 >> v2:172.21.15.40:3300/0 conn(0x7f3124132ee0 0x7f31240a0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:18:22.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.853+0000 7f311bfff640 1 --2- 172.21.15.40:0/4056357385 >> v2:172.21.15.40:3300/0 conn(0x7f3124132ee0 0x7f31240a0a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:18:22.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.854+0000 7f312259c640 1 --2- 172.21.15.40:0/4056357385 >> v2:172.21.15.189:3300/0 conn(0x7f3124132b00 0x7f312409af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:22.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.854+0000 7f312259c640 10 monclient(hunting): get_auth_request con 0x7f3124132b00 auth_method 0 2024-04-22T08:18:22.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.854+0000 7f312259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:22.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.854+0000 7f312259c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:22.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.854+0000 7f312259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:18:22.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.854+0000 7f312259c640 1 -- 172.21.15.40:0/4056357385 >> v2:172.21.15.189:3300/0 conn(0x7f3124132b00 msgr2=0x7f312409af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:18:22.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.854+0000 7f312259c640 1 -- 172.21.15.40:0/4056357385 >> v2:172.21.15.189:3300/0 conn(0x7f3124132b00 msgr2=0x7f312409af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:18:22.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.854+0000 7f312259c640 1 --2- 172.21.15.40:0/4056357385 >> v2:172.21.15.189:3300/0 conn(0x7f3124132b00 0x7f312409af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:18:22.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:22.854+0000 7f312259c640 1 --2- 172.21.15.40:0/4056357385 >> v2:172.21.15.189:3300/0 conn(0x7f3124132b00 0x7f312409af70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:18:23.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:23.254+0000 7f3122d9d640 1 --2- 172.21.15.40:0/4056357385 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3124133370 0x7f31240a0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:23.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:23.254+0000 7f3122d9d640 10 monclient(hunting): get_auth_request con 0x7f3124133370 auth_method 0 2024-04-22T08:18:23.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:23.254+0000 7f3122d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:23.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:23.254+0000 7f3122d9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:23.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:23.254+0000 7f3122d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:18:23.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:23.254+0000 7f311bfff640 1 --2- 172.21.15.40:0/4056357385 >> v2:172.21.15.40:3300/0 conn(0x7f3124132ee0 0x7f31240a0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:23.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:23.254+0000 7f311bfff640 10 monclient(hunting): get_auth_request con 0x7f3124132ee0 auth_method 0 2024-04-22T08:18:23.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:23.254+0000 7f311bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:23.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:23.254+0000 7f311bfff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:23.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:23.254+0000 7f311bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:18:23.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:23.254+0000 7f3122d9d640 1 -- 172.21.15.40:0/4056357385 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3124133370 msgr2=0x7f31240a0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:18:23.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:23.254+0000 7f3122d9d640 1 -- 172.21.15.40:0/4056357385 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3124133370 msgr2=0x7f31240a0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:18:23.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:23.254+0000 7f3122d9d640 1 --2- 172.21.15.40:0/4056357385 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3124133370 0x7f31240a0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:18:23.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:23.254+0000 7f311bfff640 1 -- 172.21.15.40:0/4056357385 >> v2:172.21.15.40:3300/0 conn(0x7f3124132ee0 msgr2=0x7f31240a0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:18:23.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:23.254+0000 7f311bfff640 1 -- 172.21.15.40:0/4056357385 >> v2:172.21.15.40:3300/0 conn(0x7f3124132ee0 msgr2=0x7f31240a0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:18:23.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:23.254+0000 7f311bfff640 1 --2- 172.21.15.40:0/4056357385 >> v2:172.21.15.40:3300/0 conn(0x7f3124132ee0 0x7f31240a0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:18:23.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:23.254+0000 7f3122d9d640 1 --2- 172.21.15.40:0/4056357385 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3124133370 0x7f31240a0f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:18:23.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:23.254+0000 7f311bfff640 1 --2- 172.21.15.40:0/4056357385 >> v2:172.21.15.40:3300/0 conn(0x7f3124132ee0 0x7f31240a0a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:18:23.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:23.255+0000 7f312259c640 1 --2- 172.21.15.40:0/4056357385 >> v2:172.21.15.189:3300/0 conn(0x7f3124132b00 0x7f312409af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:23.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:23.255+0000 7f312259c640 10 monclient(hunting): get_auth_request con 0x7f3124132b00 auth_method 0 2024-04-22T08:18:23.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:23.255+0000 7f312259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:23.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:23.255+0000 7f312259c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:23.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:23.255+0000 7f312259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:18:23.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:23.255+0000 7f312259c640 1 -- 172.21.15.40:0/4056357385 >> v2:172.21.15.189:3300/0 conn(0x7f3124132b00 msgr2=0x7f312409af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:18:23.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:23.255+0000 7f312259c640 1 -- 172.21.15.40:0/4056357385 >> v2:172.21.15.189:3300/0 conn(0x7f3124132b00 msgr2=0x7f312409af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:18:23.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:23.255+0000 7f312259c640 1 --2- 172.21.15.40:0/4056357385 >> v2:172.21.15.189:3300/0 conn(0x7f3124132b00 0x7f312409af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:18:23.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:23.255+0000 7f312259c640 1 --2- 172.21.15.40:0/4056357385 >> v2:172.21.15.189:3300/0 conn(0x7f3124132b00 0x7f312409af70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:18:24.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.055+0000 7f3122d9d640 1 --2- 172.21.15.40:0/4056357385 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3124133370 0x7f31240a0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:24.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.055+0000 7f311bfff640 1 --2- 172.21.15.40:0/4056357385 >> v2:172.21.15.40:3300/0 conn(0x7f3124132ee0 0x7f31240a0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:24.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.055+0000 7f3122d9d640 10 monclient(hunting): get_auth_request con 0x7f3124133370 auth_method 0 2024-04-22T08:18:24.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.055+0000 7f3122d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:24.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.055+0000 7f3122d9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:24.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.055+0000 7f3122d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:18:24.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.055+0000 7f311bfff640 10 monclient(hunting): get_auth_request con 0x7f3124132ee0 auth_method 0 2024-04-22T08:18:24.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.055+0000 7f311bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:24.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.055+0000 7f311bfff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:24.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.055+0000 7f311bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:18:24.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.055+0000 7f3122d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:24.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.055+0000 7f3122d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:24.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.055+0000 7f3122d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:24.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.055+0000 7f311bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:24.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.055+0000 7f311bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:24.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.055+0000 7f311bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:24.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.056+0000 7f3122d9d640 10 monclient(hunting): handle_auth_done global_id 54443 payload 274 2024-04-22T08:18:24.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.056+0000 7f3122d9d640 1 -- 172.21.15.40:0/4056357385 >> v2:172.21.15.189:3300/0 conn(0x7f3124132b00 msgr2=0x7f312409af70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T08:18:24.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.056+0000 7f3122d9d640 1 --2- 172.21.15.40:0/4056357385 >> v2:172.21.15.189:3300/0 conn(0x7f3124132b00 0x7f312409af70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:24.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.056+0000 7f3122d9d640 1 -- 172.21.15.40:0/4056357385 >> v2:172.21.15.40:3300/0 conn(0x7f3124132ee0 msgr2=0x7f31240a0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:18:24.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.056+0000 7f3122d9d640 1 --2- 172.21.15.40:0/4056357385 >> v2:172.21.15.40:3300/0 conn(0x7f3124132ee0 0x7f31240a0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:24.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.056+0000 7f3122d9d640 10 monclient: _finish_hunting 0 2024-04-22T08:18:24.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.056+0000 7f3122d9d640 1 monclient: found mon.noname-b 2024-04-22T08:18:24.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.056+0000 7f3122d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:18:24.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.056+0000 7f3122d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:18:24.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.056+0000 7f3122d9d640 1 -- 172.21.15.40:0/4056357385 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f312409a500 con 0x7f3124133370 2024-04-22T08:18:24.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.056+0000 7f311bfff640 1 --2- 172.21.15.40:0/4056357385 >> v2:172.21.15.40:3300/0 conn(0x7f3124132ee0 0x7f31240a0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:18:24.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.059+0000 7f3122d9d640 1 -- 172.21.15.40:0/4056357385 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3124133370 msgr2=0x7f31240a0f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:18:24.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.059+0000 7f3122d9d640 1 --2- 172.21.15.40:0/4056357385 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3124133370 0x7f31240a0f40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f312406d750 tx=0x7f311c02c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff98318ec35e58a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:24.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.059+0000 7f3121d9b640 1 -- 172.21.15.40:0/4056357385 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311c003990 con 0x7f3124133370 2024-04-22T08:18:24.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.059+0000 7f3121d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:24.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.059+0000 7f3121d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:18:24.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.059+0000 7f3121d9b640 10 monclient: dump: 2024-04-22T08:18:24.059 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:24.059 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:24.059 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.059 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.059 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:24.060 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:24.060 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:24.060 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:24.060 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:24.060 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:24.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.059+0000 7f3121d9b640 10 monclient: _finish_auth 0 2024-04-22T08:18:24.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.059+0000 7f3121d9b640 10 monclient: _check_auth_tickets 2024-04-22T08:18:24.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.059+0000 7f3121d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:18:24.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.059+0000 7f3121d9b640 1 -- 172.21.15.40:0/4056357385 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f311c02f030 con 0x7f3124133370 2024-04-22T08:18:24.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.059+0000 7f3121d9b640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:18:24.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.059+0000 7f3121d9b640 1 -- 172.21.15.40:0/4056357385 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311c003b10 con 0x7f3124133370 2024-04-22T08:18:24.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.059+0000 7f3128ba5640 5 monclient: authenticate success, global_id 54443 2024-04-22T08:18:24.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.059+0000 7f3128ba5640 10 monclient: get_monmap_and_config success 2024-04-22T08:18:24.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.059+0000 7f3128ba5640 4 set_mon_vals no callback set 2024-04-22T08:18:24.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.059+0000 7f3121d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:24.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.059+0000 7f3121d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:18:24.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.059+0000 7f3121d9b640 10 monclient: dump: 2024-04-22T08:18:24.061 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:24.061 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:24.061 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.061 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.061 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:24.061 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:24.061 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:24.061 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:24.062 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:24.062 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:24.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.060+0000 7f3128ba5640 10 monclient: shutdown 2024-04-22T08:18:24.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.060+0000 7f3128ba5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:18:24.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.060+0000 7f3128ba5640 1 -- 172.21.15.40:0/4056357385 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3124133370 msgr2=0x7f31240a0f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:24.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.060+0000 7f3128ba5640 1 --2- 172.21.15.40:0/4056357385 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3124133370 0x7f31240a0f40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f312406d750 tx=0x7f311c02c490 comp rx=0 tx=0).stop 2024-04-22T08:18:24.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.060+0000 7f3128ba5640 1 -- 172.21.15.40:0/4056357385 shutdown_connections 2024-04-22T08:18:24.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.060+0000 7f3128ba5640 1 --2- 172.21.15.40:0/4056357385 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3124133370 0x7f31240a0f40 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:24.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.060+0000 7f3128ba5640 1 --2- 172.21.15.40:0/4056357385 >> v2:172.21.15.40:3300/0 conn(0x7f3124132ee0 0x7f31240a0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:24.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.060+0000 7f3128ba5640 1 --2- 172.21.15.40:0/4056357385 >> v2:172.21.15.189:3300/0 conn(0x7f3124132b00 0x7f312409af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:24.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.060+0000 7f3128ba5640 1 -- 172.21.15.40:0/4056357385 >> 172.21.15.40:0/4056357385 conn(0x7f31240a5960 msgr2=0x7f31240a1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:18:24.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.060+0000 7f3128ba5640 1 -- 172.21.15.40:0/4056357385 shutdown_connections 2024-04-22T08:18:24.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.060+0000 7f3128ba5640 1 -- 172.21.15.40:0/4056357385 wait complete. 2024-04-22T08:18:24.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.061+0000 7f3128ba5640 10 monclient: build_initial_monmap 2024-04-22T08:18:24.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.061+0000 7f3128ba5640 10 monclient: monmap: 2024-04-22T08:18:24.069 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:18:24.069 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:18:24.069 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:18:24.069 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:18:24.069 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:18:24.069 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:24.070 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:18:24.070 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:18:24.070 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:18:24.070 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:24.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.061+0000 7f3128ba5640 1 Processor -- start 2024-04-22T08:18:24.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.061+0000 7f3128ba5640 1 -- start start 2024-04-22T08:18:24.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.061+0000 7f3128ba5640 10 monclient: init 2024-04-22T08:18:24.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.062+0000 7f3128ba5640 10 monclient: _reopen_session rank -1 2024-04-22T08:18:24.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.062+0000 7f3128ba5640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-22T08:18:24.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.062+0000 7f3128ba5640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3124132b00 0x7f31241caae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:24.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.062+0000 7f3128ba5640 10 monclient(hunting): picked mon.noname-c con 0x7f3124132b00 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:18:24.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.062+0000 7f3128ba5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3124132ee0 0x7f31241cb020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:24.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.062+0000 7f3128ba5640 10 monclient(hunting): picked mon.noname-b con 0x7f3124132ee0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:18:24.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.062+0000 7f3128ba5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3124133370 0x7f3124121e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:24.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.062+0000 7f3128ba5640 10 monclient(hunting): picked mon.noname-a con 0x7f3124133370 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:18:24.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.062+0000 7f3128ba5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:24.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.062+0000 7f3128ba5640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3124124210 con 0x7f3124133370 2024-04-22T08:18:24.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.062+0000 7f3128ba5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:24.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.062+0000 7f3128ba5640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3124124400 con 0x7f3124132ee0 2024-04-22T08:18:24.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.062+0000 7f3128ba5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:24.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.062+0000 7f3128ba5640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f312409b940 con 0x7f3124132b00 2024-04-22T08:18:24.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.062+0000 7f3128ba5640 10 monclient(hunting): _renew_subs 2024-04-22T08:18:24.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.062+0000 7f3128ba5640 10 monclient(hunting): authenticate will time out at 1178.056188s 2024-04-22T08:18:24.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.062+0000 7f312259c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3124132b00 0x7f31241caae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:24.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.062+0000 7f3122d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3124133370 0x7f3124121e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:24.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.062+0000 7f312259c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3124132b00 0x7f31241caae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:51372/0 (socket says 172.21.15.40:51372) 2024-04-22T08:18:24.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.062+0000 7f312259c640 1 -- 172.21.15.40:0/3311183976 learned_addr learned my addr 172.21.15.40:0/3311183976 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:18:24.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.062+0000 7f312259c640 10 monclient(hunting): get_auth_request con 0x7f3124132b00 auth_method 0 2024-04-22T08:18:24.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.062+0000 7f312259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:24.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.062+0000 7f312259c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:24.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.062+0000 7f312259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:24.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.062+0000 7f3122d9d640 10 monclient(hunting): get_auth_request con 0x7f3124133370 auth_method 0 2024-04-22T08:18:24.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.062+0000 7f3122d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:24.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.062+0000 7f3122d9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:24.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.062+0000 7f3122d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:24.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.062+0000 7f311bfff640 1 --2- 172.21.15.40:0/3311183976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3124132ee0 0x7f31241cb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:24.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.062+0000 7f312259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:24.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.062+0000 7f312259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:24.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.062+0000 7f312259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:24.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.062+0000 7f3122d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:24.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.062+0000 7f3122d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:24.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.062+0000 7f3122d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:24.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.062+0000 7f311bfff640 10 monclient(hunting): get_auth_request con 0x7f3124132ee0 auth_method 0 2024-04-22T08:18:24.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.062+0000 7f311bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:24.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.062+0000 7f311bfff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:24.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.062+0000 7f311bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:24.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.062+0000 7f311bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:24.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.062+0000 7f311bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:24.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.062+0000 7f311bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:24.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.062+0000 7f312259c640 10 monclient(hunting): handle_auth_done global_id 54446 payload 824 2024-04-22T08:18:24.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.063+0000 7f312259c640 1 -- 172.21.15.40:0/3311183976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3124132ee0 msgr2=0x7f31241cb020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:24.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.063+0000 7f312259c640 1 --2- 172.21.15.40:0/3311183976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3124132ee0 0x7f31241cb020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:24.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.063+0000 7f312259c640 1 -- 172.21.15.40:0/3311183976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3124133370 msgr2=0x7f3124121e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:24.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.063+0000 7f312259c640 1 --2- 172.21.15.40:0/3311183976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3124133370 0x7f3124121e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:24.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.063+0000 7f312259c640 10 monclient: _finish_hunting 0 2024-04-22T08:18:24.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.063+0000 7f312259c640 1 monclient: found mon.noname-c 2024-04-22T08:18:24.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.063+0000 7f312259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:18:24.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.063+0000 7f312259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:18:24.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.063+0000 7f312259c640 1 -- 172.21.15.40:0/3311183976 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f310c00e040 con 0x7f3124132b00 2024-04-22T08:18:24.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.063+0000 7f3122d9d640 1 --2- 172.21.15.40:0/3311183976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3124133370 0x7f3124121e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:18:24.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.065+0000 7f312259c640 1 -- 172.21.15.40:0/3311183976 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3124132b00 msgr2=0x7f31241caae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:18:24.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.065+0000 7f312259c640 1 --2- 172.21.15.40:0/3311183976 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3124132b00 0x7f31241caae0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f31140116d0 tx=0x7f3114011b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:24.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.065+0000 7f311affd640 1 -- 172.21.15.40:0/3311183976 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3114004220 con 0x7f3124132b00 2024-04-22T08:18:24.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.065+0000 7f311affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:24.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.065+0000 7f311affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:18:24.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.065+0000 7f311affd640 10 monclient: dump: 2024-04-22T08:18:24.075 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:24.075 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:24.075 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.075 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.075 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:24.075 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:24.075 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:24.075 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:24.075 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:24.075 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:24.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.065+0000 7f311affd640 10 monclient: _finish_auth 0 2024-04-22T08:18:24.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.065+0000 7f311affd640 10 monclient: _check_auth_tickets 2024-04-22T08:18:24.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.065+0000 7f311affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:18:24.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.065+0000 7f3128ba5640 5 monclient: authenticate success, global_id 54446 2024-04-22T08:18:24.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.065+0000 7f311affd640 1 -- 172.21.15.40:0/3311183976 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31140043a0 con 0x7f3124132b00 2024-04-22T08:18:24.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.065+0000 7f311affd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:18:24.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.065+0000 7f311affd640 1 -- 172.21.15.40:0/3311183976 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311400dbf0 con 0x7f3124132b00 2024-04-22T08:18:24.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.065+0000 7f3128ba5640 10 monclient: _renew_subs 2024-04-22T08:18:24.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.065+0000 7f3128ba5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:18:24.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.065+0000 7f3128ba5640 1 -- 172.21.15.40:0/3311183976 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f311c0097e0 con 0x7f3124132b00 2024-04-22T08:18:24.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.065+0000 7f312159a640 4 set_mon_vals no callback set 2024-04-22T08:18:24.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.065+0000 7f311affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:24.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.065+0000 7f311affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:18:24.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.065+0000 7f311affd640 10 monclient: dump: 2024-04-22T08:18:24.077 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:24.077 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:24.077 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.077 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.077 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:24.077 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:24.077 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:24.077 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:24.077 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:24.077 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:24.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.065+0000 7f3128ba5640 10 monclient: _renew_subs 2024-04-22T08:18:24.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.065+0000 7f3128ba5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:18:24.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.065+0000 7f3128ba5640 1 -- 172.21.15.40:0/3311183976 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3124125500 con 0x7f3124132b00 2024-04-22T08:18:24.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.066+0000 7f311affd640 1 -- 172.21.15.40:0/3311183976 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f311400dd90 con 0x7f3124132b00 2024-04-22T08:18:24.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.066+0000 7f3128ba5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:18:24.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.066+0000 7f3128ba5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:18:24.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.066+0000 7f3128ba5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:18:24.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.066+0000 7f3128ba5640 1 -- 172.21.15.40:0/3311183976 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3124123a30 con 0x7f3124132b00 2024-04-22T08:18:24.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.067+0000 7f311affd640 1 --2- 172.21.15.40:0/3311183976 >> v2:172.21.15.189:6800/30168 conn(0x7f3104037850 0x7f3104039d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:24.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.067+0000 7f311affd640 1 -- 172.21.15.40:0/3311183976 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f311404a750 con 0x7f3124132b00 2024-04-22T08:18:24.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.067+0000 7f311bfff640 1 --2- 172.21.15.40:0/3311183976 >> v2:172.21.15.189:6800/30168 conn(0x7f3104037850 0x7f3104039d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:24.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.067+0000 7f311bfff640 10 monclient: get_auth_request con 0x7f3104037850 auth_method 0 2024-04-22T08:18:24.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.070+0000 7f311bfff640 1 --2- 172.21.15.40:0/3311183976 >> v2:172.21.15.189:6800/30168 conn(0x7f3104037850 0x7f3104039d10 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f312406c450 tx=0x7f310c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:24.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.072+0000 7f311affd640 1 -- 172.21.15.40:0/3311183976 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f3114025710 con 0x7f3124132b00 2024-04-22T08:18:24.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.072+0000 7f311affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:18:24.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.072+0000 7f311affd640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:18:24.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.323+0000 7f3128ba5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:18:24.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.323+0000 7f3128ba5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:18:24.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.323+0000 7f3128ba5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:18:24.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.323+0000 7f3128ba5640 1 -- 172.21.15.40:0/3311183976 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f312404e9b0 con 0x7f3124132b00 2024-04-22T08:18:24.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.324+0000 7f311affd640 1 -- 172.21.15.40:0/3311183976 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f3114026350 con 0x7f3124132b00 2024-04-22T08:18:24.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.324+0000 7f311affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:18:24.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.324+0000 7f311affd640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:18:24.324 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":56,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:18:24.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.326+0000 7f3128ba5640 1 -- 172.21.15.40:0/3311183976 >> v2:172.21.15.189:6800/30168 conn(0x7f3104037850 msgr2=0x7f3104039d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:24.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.326+0000 7f3128ba5640 1 --2- 172.21.15.40:0/3311183976 >> v2:172.21.15.189:6800/30168 conn(0x7f3104037850 0x7f3104039d10 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f312406c450 tx=0x7f310c008040 comp rx=0 tx=0).stop 2024-04-22T08:18:24.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.326+0000 7f3128ba5640 10 monclient: shutdown 2024-04-22T08:18:24.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.326+0000 7f3128ba5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:18:24.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.326+0000 7f3128ba5640 1 -- 172.21.15.40:0/3311183976 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3124132b00 msgr2=0x7f31241caae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:24.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.326+0000 7f3128ba5640 1 --2- 172.21.15.40:0/3311183976 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3124132b00 0x7f31241caae0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f31140116d0 tx=0x7f3114011b90 comp rx=0 tx=0).stop 2024-04-22T08:18:24.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.326+0000 7f3128ba5640 1 -- 172.21.15.40:0/3311183976 shutdown_connections 2024-04-22T08:18:24.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.326+0000 7f3128ba5640 1 --2- 172.21.15.40:0/3311183976 >> v2:172.21.15.189:6800/30168 conn(0x7f3104037850 0x7f3104039d10 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:24.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.326+0000 7f3128ba5640 1 --2- 172.21.15.40:0/3311183976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3124132ee0 0x7f31241cb020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:24.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.326+0000 7f3128ba5640 1 --2- 172.21.15.40:0/3311183976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3124133370 0x7f3124121e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:24.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.326+0000 7f3128ba5640 1 --2- 172.21.15.40:0/3311183976 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3124132b00 0x7f31241caae0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:24.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.326+0000 7f3128ba5640 1 -- 172.21.15.40:0/3311183976 >> 172.21.15.40:0/3311183976 conn(0x7f31240a5960 msgr2=0x7f312409eee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:18:24.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.326+0000 7f3128ba5640 1 -- 172.21.15.40:0/3311183976 shutdown_connections 2024-04-22T08:18:24.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.326+0000 7f3128ba5640 1 -- 172.21.15.40:0/3311183976 wait complete. 2024-04-22T08:18:24.336 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-22T08:18:24.336 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-22T08:18:24.336 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-22T08:18:24.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.436+0000 7f01a442d640 10 monclient: get_monmap_and_config 2024-04-22T08:18:24.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.436+0000 7f01a442d640 10 monclient: build_initial_monmap 2024-04-22T08:18:24.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.436+0000 7f01a442d640 10 monclient: monmap: 2024-04-22T08:18:24.436 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:18:24.436 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:24.436 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:18:24.437628+0000 2024-04-22T08:18:24.436 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:18:24.437628+0000 2024-04-22T08:18:24.437 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:18:24.437 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:24.437 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:18:24.437 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:18:24.437 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:18:24.437 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:18:24.437 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:18:24.437 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:24.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.437+0000 7f01a442d640 1 Processor -- start 2024-04-22T08:18:24.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.437+0000 7f01a442d640 1 -- start start 2024-04-22T08:18:24.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.437+0000 7f01a442d640 10 monclient: init 2024-04-22T08:18:24.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.437+0000 7f01a442d640 10 monclient: _reopen_session rank -1 2024-04-22T08:18:24.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.438+0000 7f01a442d640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-04-22T08:18:24.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.438+0000 7f01a442d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f019c0a1a80 addr v1:172.21.15.189:6789/0 2024-04-22T08:18:24.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.438+0000 7f01a442d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f019c0a2570 0x7f019c0a01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:24.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.438+0000 7f01a442d640 10 monclient(hunting): picked mon.noname-b con 0x7f019c0a2570 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:18:24.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.438+0000 7f01a442d640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f019c13b0e0 0x7f019c13d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:24.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.438+0000 7f01a442d640 10 monclient(hunting): picked mon.noname-c con 0x7f019c13b0e0 addr v2:172.21.15.189:3300/0 2024-04-22T08:18:24.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.438+0000 7f01a442d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f019c09d860 con 0x7f019c0a1a80 2024-04-22T08:18:24.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.438+0000 7f01a442d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:24.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.438+0000 7f01a442d640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f019c09f880 con 0x7f019c13b0e0 2024-04-22T08:18:24.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.438+0000 7f01a442d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:24.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.438+0000 7f01a442d640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f019c09ef50 con 0x7f019c0a2570 2024-04-22T08:18:24.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.438+0000 7f01a442d640 10 monclient(hunting): _renew_subs 2024-04-22T08:18:24.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.438+0000 7f01a442d640 10 monclient(hunting): authenticate will time out at 1178.431844s 2024-04-22T08:18:24.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.438+0000 7f01a19c8640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f019c0a2570 0x7f019c0a01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:24.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.438+0000 7f01a19c8640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f019c0a2570 0x7f019c0a01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:51382/0 (socket says 172.21.15.40:51382) 2024-04-22T08:18:24.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.438+0000 7f01a19c8640 1 -- 172.21.15.40:0/3319079135 learned_addr learned my addr 172.21.15.40:0/3319079135 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:18:24.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.438+0000 7f01a19c8640 10 monclient(hunting): get_auth_request con 0x7f019c0a2570 auth_method 0 2024-04-22T08:18:24.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.438+0000 7f01a19c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:24.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.438+0000 7f01a19c8640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:24.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.438+0000 7f01a19c8640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:24.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.438+0000 7f01a19c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:24.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.438+0000 7f01a19c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:24.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.438+0000 7f01a19c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:24.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.438+0000 7f01a29ca640 1 --2- 172.21.15.40:0/3319079135 >> v2:172.21.15.189:3300/0 conn(0x7f019c13b0e0 0x7f019c13d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:24.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.438+0000 7f01a29ca640 10 monclient(hunting): get_auth_request con 0x7f019c13b0e0 auth_method 0 2024-04-22T08:18:24.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.438+0000 7f01a29ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:24.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.438+0000 7f01a29ca640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:24.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.438+0000 7f01a29ca640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:24.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.438+0000 7f01a19c8640 10 monclient(hunting): handle_auth_done global_id 54449 payload 274 2024-04-22T08:18:24.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.438+0000 7f01a19c8640 1 -- 172.21.15.40:0/3319079135 >> v2:172.21.15.189:3300/0 conn(0x7f019c13b0e0 msgr2=0x7f019c13d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:18:24.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.438+0000 7f01a19c8640 1 --2- 172.21.15.40:0/3319079135 >> v2:172.21.15.189:3300/0 conn(0x7f019c13b0e0 0x7f019c13d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:24.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.438+0000 7f01a19c8640 1 -- 172.21.15.40:0/3319079135 >> v1:172.21.15.189:6789/0 conn(0x7f019c0a1a80 legacy=0x7f019c0a1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:18:24.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.438+0000 7f01a19c8640 10 monclient: _finish_hunting 0 2024-04-22T08:18:24.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.438+0000 7f01a19c8640 1 monclient: found mon.noname-b 2024-04-22T08:18:24.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.438+0000 7f01a19c8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:18:24.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.438+0000 7f01a19c8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:18:24.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.438+0000 7f01a19c8640 1 -- 172.21.15.40:0/3319079135 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f019c072160 con 0x7f019c0a2570 2024-04-22T08:18:24.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.441+0000 7f01a19c8640 1 -- 172.21.15.40:0/3319079135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f019c0a2570 msgr2=0x7f019c0a01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:18:24.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.441+0000 7f01a19c8640 1 --2- 172.21.15.40:0/3319079135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f019c0a2570 0x7f019c0a01d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0190009870 tx=0x7f019002b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=202fed9ad4716279 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:24.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.441+0000 7f01a11c7640 1 -- 172.21.15.40:0/3319079135 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0190031950 con 0x7f019c0a2570 2024-04-22T08:18:24.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.441+0000 7f01a11c7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:24.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.441+0000 7f01a11c7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:18:24.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.441+0000 7f01a11c7640 10 monclient: dump: 2024-04-22T08:18:24.452 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:24.452 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:24.452 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.452 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.452 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:24.452 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:24.452 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:24.452 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:24.452 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:24.452 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:24.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.441+0000 7f01a11c7640 10 monclient: _finish_auth 0 2024-04-22T08:18:24.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.441+0000 7f01a11c7640 10 monclient: _check_auth_tickets 2024-04-22T08:18:24.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.441+0000 7f01a11c7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:18:24.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.441+0000 7f01a442d640 5 monclient: authenticate success, global_id 54449 2024-04-22T08:18:24.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.441+0000 7f01a442d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:18:24.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.441+0000 7f01a11c7640 1 -- 172.21.15.40:0/3319079135 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0190031ab0 con 0x7f019c0a2570 2024-04-22T08:18:24.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.441+0000 7f01a11c7640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:18:24.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.441+0000 7f01a11c7640 1 -- 172.21.15.40:0/3319079135 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0190031e40 con 0x7f019c0a2570 2024-04-22T08:18:24.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.441+0000 7f01a11c7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:24.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.441+0000 7f01a11c7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:18:24.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.441+0000 7f01a11c7640 10 monclient: dump: 2024-04-22T08:18:24.454 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:24.454 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:24.454 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.454 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.454 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:24.454 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:24.454 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:24.454 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:24.454 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:24.454 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:24.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.441+0000 7f01a442d640 10 monclient: get_monmap_and_config success 2024-04-22T08:18:24.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.441+0000 7f01a442d640 4 set_mon_vals no callback set 2024-04-22T08:18:24.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.441+0000 7f01a442d640 10 monclient: shutdown 2024-04-22T08:18:24.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.441+0000 7f01a442d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:18:24.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.441+0000 7f01a442d640 1 -- 172.21.15.40:0/3319079135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f019c0a2570 msgr2=0x7f019c0a01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:24.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.442+0000 7f01a442d640 1 --2- 172.21.15.40:0/3319079135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f019c0a2570 0x7f019c0a01d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0190009870 tx=0x7f019002b7e0 comp rx=0 tx=0).stop 2024-04-22T08:18:24.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.442+0000 7f01a442d640 1 -- 172.21.15.40:0/3319079135 shutdown_connections 2024-04-22T08:18:24.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.442+0000 7f01a442d640 1 --2- 172.21.15.40:0/3319079135 >> v2:172.21.15.189:3300/0 conn(0x7f019c13b0e0 0x7f019c13d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:24.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.442+0000 7f01a442d640 1 --2- 172.21.15.40:0/3319079135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f019c0a2570 0x7f019c0a01d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:24.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.442+0000 7f01a442d640 1 -- 172.21.15.40:0/3319079135 >> 172.21.15.40:0/3319079135 conn(0x7f019c0a66a0 msgr2=0x7f019c0a2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:18:24.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.442+0000 7f01a442d640 1 -- 172.21.15.40:0/3319079135 shutdown_connections 2024-04-22T08:18:24.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.442+0000 7f01a442d640 1 -- 172.21.15.40:0/3319079135 wait complete. 2024-04-22T08:18:24.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.442+0000 7f01a442d640 10 monclient: build_initial_monmap 2024-04-22T08:18:24.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.442+0000 7f01a442d640 10 monclient: monmap: 2024-04-22T08:18:24.457 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:18:24.457 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:18:24.458 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:18:24.458 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:18:24.458 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:18:24.458 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:24.458 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:18:24.458 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:18:24.458 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:18:24.458 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:24.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.443+0000 7f01a442d640 1 Processor -- start 2024-04-22T08:18:24.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.443+0000 7f01a442d640 1 -- start start 2024-04-22T08:18:24.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.443+0000 7f01a442d640 10 monclient: init 2024-04-22T08:18:24.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.443+0000 7f01a442d640 10 monclient: _reopen_session rank -1 2024-04-22T08:18:24.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.443+0000 7f01a442d640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-22T08:18:24.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.443+0000 7f01a442d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f019c0a1a80 0x7f019c120ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:24.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.443+0000 7f01a442d640 10 monclient(hunting): picked mon.noname-a con 0x7f019c0a1a80 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:18:24.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.443+0000 7f01a442d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f019c0a2570 0x7f019c121010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:24.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.443+0000 7f01a442d640 10 monclient(hunting): picked mon.noname-c con 0x7f019c0a2570 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:18:24.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.443+0000 7f01a442d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f019c13b0e0 0x7f019c121550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:24.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.443+0000 7f01a442d640 10 monclient(hunting): picked mon.noname-b con 0x7f019c13b0e0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:18:24.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.443+0000 7f01a442d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:24.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.443+0000 7f01a442d640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f019c124730 con 0x7f019c0a1a80 2024-04-22T08:18:24.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.443+0000 7f01a442d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:24.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.443+0000 7f01a442d640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f019c123a70 con 0x7f019c13b0e0 2024-04-22T08:18:24.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.443+0000 7f01a442d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:24.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.443+0000 7f01a442d640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f019c13d490 con 0x7f019c0a2570 2024-04-22T08:18:24.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.443+0000 7f01a442d640 10 monclient(hunting): _renew_subs 2024-04-22T08:18:24.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.443+0000 7f01a442d640 10 monclient(hunting): authenticate will time out at 1178.437339s 2024-04-22T08:18:24.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.443+0000 7f01a21c9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f019c0a1a80 0x7f019c120ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:24.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.443+0000 7f01a19c8640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f019c0a2570 0x7f019c121010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:24.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.443+0000 7f01a21c9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f019c0a1a80 0x7f019c120ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44578/0 (socket says 172.21.15.40:44578) 2024-04-22T08:18:24.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.443+0000 7f01a21c9640 1 -- 172.21.15.40:0/3990899409 learned_addr learned my addr 172.21.15.40:0/3990899409 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:18:24.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.443+0000 7f01a21c9640 10 monclient(hunting): get_auth_request con 0x7f019c0a1a80 auth_method 0 2024-04-22T08:18:24.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.443+0000 7f01a21c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:24.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.443+0000 7f01a21c9640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:24.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.443+0000 7f01a21c9640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:24.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.443+0000 7f01a19c8640 10 monclient(hunting): get_auth_request con 0x7f019c0a2570 auth_method 0 2024-04-22T08:18:24.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.443+0000 7f01a19c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:24.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.443+0000 7f01a19c8640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:24.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.443+0000 7f01a19c8640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:24.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.443+0000 7f01a21c9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:24.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.443+0000 7f01a21c9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:24.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.443+0000 7f01a21c9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:24.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.443+0000 7f01a19c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:24.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.443+0000 7f01a19c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:24.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.443+0000 7f01a19c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:24.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.443+0000 7f01a19c8640 10 monclient(hunting): handle_auth_done global_id 54452 payload 824 2024-04-22T08:18:24.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.443+0000 7f01a29ca640 1 --2- 172.21.15.40:0/3990899409 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f019c13b0e0 0x7f019c121550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:24.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.443+0000 7f01a19c8640 1 -- 172.21.15.40:0/3990899409 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f019c13b0e0 msgr2=0x7f019c121550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:24.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.443+0000 7f01a19c8640 1 --2- 172.21.15.40:0/3990899409 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f019c13b0e0 0x7f019c121550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:24.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.443+0000 7f01a19c8640 1 -- 172.21.15.40:0/3990899409 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f019c0a1a80 msgr2=0x7f019c120ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:24.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.443+0000 7f01a19c8640 1 --2- 172.21.15.40:0/3990899409 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f019c0a1a80 0x7f019c120ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:24.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.444+0000 7f01a19c8640 10 monclient: _finish_hunting 0 2024-04-22T08:18:24.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.444+0000 7f01a19c8640 1 monclient: found mon.noname-c 2024-04-22T08:18:24.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.444+0000 7f01a19c8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:18:24.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.444+0000 7f01a19c8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:18:24.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.444+0000 7f01a19c8640 1 -- 172.21.15.40:0/3990899409 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01900094d0 con 0x7f019c0a2570 2024-04-22T08:18:24.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.444+0000 7f01a21c9640 1 --2- 172.21.15.40:0/3990899409 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f019c0a1a80 0x7f019c120ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:18:24.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.446+0000 7f01a19c8640 1 -- 172.21.15.40:0/3990899409 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f019c0a2570 msgr2=0x7f019c121010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:18:24.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.446+0000 7f01a19c8640 1 --2- 172.21.15.40:0/3990899409 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f019c0a2570 0x7f019c121010 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0190033000 tx=0x7f01900046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:24.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.446+0000 7f019affd640 1 -- 172.21.15.40:0/3990899409 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0190033d30 con 0x7f019c0a2570 2024-04-22T08:18:24.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.446+0000 7f019affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:24.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.446+0000 7f019affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:18:24.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.446+0000 7f019affd640 10 monclient: dump: 2024-04-22T08:18:24.463 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:24.463 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:24.463 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.463 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.464 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:24.464 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:24.464 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:24.464 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:24.464 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:24.464 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:24.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.446+0000 7f019affd640 10 monclient: _finish_auth 0 2024-04-22T08:18:24.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.446+0000 7f019affd640 10 monclient: _check_auth_tickets 2024-04-22T08:18:24.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.446+0000 7f019affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:18:24.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.446+0000 7f019affd640 1 -- 172.21.15.40:0/3990899409 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f019003c400 con 0x7f019c0a2570 2024-04-22T08:18:24.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.446+0000 7f01a442d640 5 monclient: authenticate success, global_id 54452 2024-04-22T08:18:24.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.446+0000 7f01a442d640 10 monclient: _renew_subs 2024-04-22T08:18:24.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.446+0000 7f01a442d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:18:24.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.446+0000 7f01a442d640 1 -- 172.21.15.40:0/3990899409 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f019c09cc20 con 0x7f019c0a2570 2024-04-22T08:18:24.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.446+0000 7f019affd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:18:24.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.446+0000 7f019affd640 1 -- 172.21.15.40:0/3990899409 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019003c7b0 con 0x7f019c0a2570 2024-04-22T08:18:24.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.446+0000 7f019affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:24.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.446+0000 7f019affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:18:24.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.446+0000 7f019affd640 10 monclient: dump: 2024-04-22T08:18:24.465 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:24.465 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:24.465 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.465 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.465 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:24.465 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:24.465 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:24.465 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:24.465 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:24.466 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:24.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.446+0000 7f019bfff640 4 set_mon_vals no callback set 2024-04-22T08:18:24.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.446+0000 7f01a442d640 10 monclient: _renew_subs 2024-04-22T08:18:24.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.446+0000 7f01a442d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:18:24.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.446+0000 7f01a442d640 1 -- 172.21.15.40:0/3990899409 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f019c1228d0 con 0x7f019c0a2570 2024-04-22T08:18:24.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.447+0000 7f01a442d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:18:24.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.447+0000 7f01a442d640 1 --2- 172.21.15.40:0/3990899409 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f019c06db80 0x7f019c06df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:24.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.447+0000 7f01a442d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:24.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.447+0000 7f01a442d640 1 -- 172.21.15.40:0/3990899409 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f019c066770 con 0x7f019c06db80 2024-04-22T08:18:24.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.447+0000 7f019affd640 1 -- 172.21.15.40:0/3990899409 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f019003a040 con 0x7f019c0a2570 2024-04-22T08:18:24.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.447+0000 7f01a21c9640 1 --2- 172.21.15.40:0/3990899409 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f019c06db80 0x7f019c06df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:24.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.447+0000 7f01a21c9640 10 monclient: get_auth_request con 0x7f019c06db80 auth_method 0 2024-04-22T08:18:24.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.447+0000 7f01a21c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:24.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.447+0000 7f01a21c9640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:24.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.447+0000 7f01a21c9640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:24.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.447+0000 7f01a21c9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:24.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.447+0000 7f01a21c9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:24.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.447+0000 7f01a21c9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:24.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.447+0000 7f019affd640 1 --2- 172.21.15.40:0/3990899409 >> v2:172.21.15.189:6800/30168 conn(0x7f01880377d0 0x7f0188039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:24.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.447+0000 7f01a21c9640 10 monclient(hunting): handle_auth_done global_id 74106 payload 824 2024-04-22T08:18:24.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.447+0000 7f01a21c9640 1 -- 172.21.15.40:0/3990899409 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f019c09dc10 con 0x7f019c06db80 2024-04-22T08:18:24.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.447+0000 7f019affd640 1 -- 172.21.15.40:0/3990899409 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f019006b6e0 con 0x7f019c0a2570 2024-04-22T08:18:24.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.448+0000 7f01a29ca640 1 --2- 172.21.15.40:0/3990899409 >> v2:172.21.15.189:6800/30168 conn(0x7f01880377d0 0x7f0188039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:24.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.448+0000 7f01a29ca640 10 monclient: get_auth_request con 0x7f01880377d0 auth_method 0 2024-04-22T08:18:24.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.449+0000 7f01a21c9640 1 -- 172.21.15.40:0/3990899409 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f019c06db80 msgr2=0x7f019c06df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:18:24.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.449+0000 7f01a21c9640 1 --2- 172.21.15.40:0/3990899409 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f019c06db80 0x7f019c06df60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f019400fb10 tx=0x7f0194004580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:24.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.450+0000 7f019affd640 1 -- 172.21.15.40:0/3990899409 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019400de50 con 0x7f019c06db80 2024-04-22T08:18:24.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.450+0000 7f019affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:24.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.450+0000 7f019affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:18:24.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.450+0000 7f019affd640 10 monclient: dump: 2024-04-22T08:18:24.468 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:24.469 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:24.469 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.469 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.469 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:24.469 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:24.469 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:24.469 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:24.469 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:24.469 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:24.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.450+0000 7f019affd640 1 -- 172.21.15.40:0/3990899409 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f0194014470 con 0x7f019c06db80 2024-04-22T08:18:24.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.450+0000 7f019affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:18:24.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.450+0000 7f019affd640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:18:24.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.450+0000 7f019affd640 1 -- 172.21.15.40:0/3990899409 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f019c06db80 msgr2=0x7f019c06df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:24.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.450+0000 7f019affd640 1 --2- 172.21.15.40:0/3990899409 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f019c06db80 0x7f019c06df60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f019400fb10 tx=0x7f0194004580 comp rx=0 tx=0).stop 2024-04-22T08:18:24.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.450+0000 7f019affd640 1 -- 172.21.15.40:0/3990899409 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f019c06db80 msgr2=0x7f019c06df60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:18:24.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.450+0000 7f019affd640 1 --2- 172.21.15.40:0/3990899409 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f019c06db80 0x7f019c06df60 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:24.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.450+0000 7f01a29ca640 1 --2- 172.21.15.40:0/3990899409 >> v2:172.21.15.189:6800/30168 conn(0x7f01880377d0 0x7f0188039c90 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f019c1239a0 tx=0x7f018c00b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:24.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.456+0000 7f01a442d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:18:24.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.456+0000 7f01a442d640 1 --2- 172.21.15.40:0/3990899409 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f019c06e4a0 0x7f019c1d4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:24.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.456+0000 7f01a442d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:24.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.456+0000 7f01a442d640 1 -- 172.21.15.40:0/3990899409 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f019c04e9b0 con 0x7f019c06e4a0 2024-04-22T08:18:24.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.456+0000 7f01a21c9640 1 --2- 172.21.15.40:0/3990899409 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f019c06e4a0 0x7f019c1d4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:24.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.456+0000 7f01a21c9640 10 monclient: get_auth_request con 0x7f019c06e4a0 auth_method 0 2024-04-22T08:18:24.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.456+0000 7f01a21c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:24.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.456+0000 7f01a21c9640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:24.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.456+0000 7f01a21c9640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:24.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.456+0000 7f01a21c9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:24.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.456+0000 7f01a21c9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:24.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.456+0000 7f01a21c9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:24.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.456+0000 7f01a21c9640 10 monclient(hunting): handle_auth_done global_id 74109 payload 824 2024-04-22T08:18:24.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.456+0000 7f01a21c9640 1 -- 172.21.15.40:0/3990899409 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f019c121fd0 con 0x7f019c06e4a0 2024-04-22T08:18:24.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.459+0000 7f01a21c9640 1 -- 172.21.15.40:0/3990899409 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f019c06e4a0 msgr2=0x7f019c1d4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:18:24.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.459+0000 7f01a21c9640 1 --2- 172.21.15.40:0/3990899409 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f019c06e4a0 0x7f019c1d4c10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0194019000 tx=0x7f0194003760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:24.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.459+0000 7f019affd640 1 -- 172.21.15.40:0/3990899409 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0194024070 con 0x7f019c06e4a0 2024-04-22T08:18:24.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.459+0000 7f019affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:24.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.459+0000 7f019affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:18:24.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.459+0000 7f019affd640 10 monclient: dump: 2024-04-22T08:18:24.474 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:24.474 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:24.474 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.474 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.474 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:24.474 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:24.474 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:24.474 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:24.474 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:24.474 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:24.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.459+0000 7f019affd640 1 -- 172.21.15.40:0/3990899409 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4463 (secure 0 0 0) 0x7f019401d420 con 0x7f019c06e4a0 2024-04-22T08:18:24.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.459+0000 7f019affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:18:24.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.459+0000 7f019affd640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:18:24.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.459+0000 7f019affd640 1 -- 172.21.15.40:0/3990899409 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f019c06e4a0 msgr2=0x7f019c1d4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:24.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.459+0000 7f019affd640 1 --2- 172.21.15.40:0/3990899409 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f019c06e4a0 0x7f019c1d4c10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0194019000 tx=0x7f0194003760 comp rx=0 tx=0).stop 2024-04-22T08:18:24.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.459+0000 7f019affd640 1 -- 172.21.15.40:0/3990899409 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f019c06e4a0 msgr2=0x7f019c1d4c10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:18:24.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.459+0000 7f019affd640 1 --2- 172.21.15.40:0/3990899409 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f019c06e4a0 0x7f019c1d4c10 secure :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0194019000 tx=0x7f0194003760 comp rx=0 tx=0).stop 2024-04-22T08:18:24.475 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:18:24.476 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:18:24.476 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:18:24.476 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "leader", 2024-04-22T08:18:24.476 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 56, 2024-04-22T08:18:24.476 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:18:24.476 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:18:24.476 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:18:24.476 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:18:24.476 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:18:24.476 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 0, 2024-04-22T08:18:24.476 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:18:24.476 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:18:24.477 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:18:24.477 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:18:24.477 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:18:24.477 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:18:24.477 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:18:24.477 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:18:24.477 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:18:24.477 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:18:24.477 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:18:24.477 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:18:24.477 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:18:24.477 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:18:24.477 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:18:24.478 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:18:24.478 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:18:24.478 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:18:24.478 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:18:24.478 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:18:24.478 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:18:24.478 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:18:24.478 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:18:24.478 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:18:24.478 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:18:24.478 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:18:24.478 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:24.478 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:24.479 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:18:24.479 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:18:24.479 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:18:24.479 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:18:24.479 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:18:24.479 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:18:24.479 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:18:24.479 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:18:24.479 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:18:24.479 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:18:24.479 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:18:24.479 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:18:24.479 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:18:24.479 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:18:24.480 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:18:24.480 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:18:24.480 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:18:24.480 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:18:24.480 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:18:24.480 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:18:24.480 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:18:24.480 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:18:24.480 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:18:24.480 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:18:24.480 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:18:24.480 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:18:24.481 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:18:24.481 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:18:24.481 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:18:24.481 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:24.481 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:18:24.481 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:24.481 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:18:24.481 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:18:24.481 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:18:24.481 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:18:24.481 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:24.481 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:18:24.481 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:18:24.482 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:24.482 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:24.482 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:24.482 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:18:24.482 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:18:24.482 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:24.482 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:24.482 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:24.482 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:24.482 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:18:24.482 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:18:24.482 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:18:24.482 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:18:24.482 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:18:24.483 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:24.483 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:24.483 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:18:24.483 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:18:24.483 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:18:24.483 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:18:24.483 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:24.483 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:18:24.483 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:18:24.483 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:24.483 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:24.483 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:24.484 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:18:24.484 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:18:24.484 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:24.484 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:24.484 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:24.484 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:24.484 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:18:24.484 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:18:24.484 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:18:24.484 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:18:24.484 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:18:24.484 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:24.484 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:24.484 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:18:24.485 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:18:24.485 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:18:24.485 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:18:24.485 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:24.485 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:18:24.485 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:18:24.485 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:24.485 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:24.485 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:24.485 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:18:24.485 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:18:24.485 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:24.485 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:24.485 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:24.486 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:24.486 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:18:24.486 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:18:24.486 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:18:24.486 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:18:24.486 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:18:24.486 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:24.486 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:24.486 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:24.486 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:18:24.486 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:18:24.486 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:24.486 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:18:24.486 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:18:24.487 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:18:24.487 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:24.487 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:18:24.487 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:18:24.487 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:24.487 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:18:24.487 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:18:24.487 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:18:24.487 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:24.487 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:24.487 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:24.487 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:18:24.487 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:18:24.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.459+0000 7f01a442d640 1 -- 172.21.15.40:0/3990899409 >> v2:172.21.15.189:6800/30168 conn(0x7f01880377d0 msgr2=0x7f0188039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:24.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.459+0000 7f01a442d640 1 --2- 172.21.15.40:0/3990899409 >> v2:172.21.15.189:6800/30168 conn(0x7f01880377d0 0x7f0188039c90 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f019c1239a0 tx=0x7f018c00b040 comp rx=0 tx=0).stop 2024-04-22T08:18:24.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.459+0000 7f01a442d640 10 monclient: shutdown 2024-04-22T08:18:24.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.459+0000 7f01a442d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:18:24.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.459+0000 7f01a442d640 1 -- 172.21.15.40:0/3990899409 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f019c0a2570 msgr2=0x7f019c121010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:24.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.459+0000 7f01a442d640 1 --2- 172.21.15.40:0/3990899409 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f019c0a2570 0x7f019c121010 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0190033000 tx=0x7f01900046c0 comp rx=0 tx=0).stop 2024-04-22T08:18:24.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.459+0000 7f01a29ca640 1 -- 172.21.15.40:0/3990899409 reap_dead start 2024-04-22T08:18:24.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.459+0000 7f01a29ca640 1 -- 172.21.15.40:0/3990899409 reap_dead start 2024-04-22T08:18:24.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.459+0000 7f01a442d640 1 -- 172.21.15.40:0/3990899409 shutdown_connections 2024-04-22T08:18:24.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.459+0000 7f01a442d640 1 -- 172.21.15.40:0/3990899409 >> 172.21.15.40:0/3990899409 conn(0x7f019c0a66a0 msgr2=0x7f019c0a4dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:18:24.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.459+0000 7f01a442d640 1 -- 172.21.15.40:0/3990899409 shutdown_connections 2024-04-22T08:18:24.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.459+0000 7f01a442d640 1 -- 172.21.15.40:0/3990899409 wait complete. 2024-04-22T08:18:24.490 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:18:24.490 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-22T08:18:24.491 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-22T08:18:24.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.590+0000 7fc8c44a1640 10 monclient: get_monmap_and_config 2024-04-22T08:18:24.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.590+0000 7fc8c44a1640 10 monclient: build_initial_monmap 2024-04-22T08:18:24.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.590+0000 7fc8c44a1640 10 monclient: monmap: 2024-04-22T08:18:24.590 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:18:24.590 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:24.590 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:18:24.591699+0000 2024-04-22T08:18:24.590 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:18:24.591699+0000 2024-04-22T08:18:24.591 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:18:24.591 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:24.591 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:18:24.591 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:18:24.591 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:18:24.591 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:18:24.591 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:18:24.591 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:24.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.591+0000 7fc8c44a1640 1 Processor -- start 2024-04-22T08:18:24.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.591+0000 7fc8c44a1640 1 -- start start 2024-04-22T08:18:24.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.591+0000 7fc8c44a1640 10 monclient: init 2024-04-22T08:18:24.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.592+0000 7fc8c44a1640 10 monclient: _reopen_session rank -1 2024-04-22T08:18:24.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.592+0000 7fc8c44a1640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-04-22T08:18:24.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.592+0000 7fc8c44a1640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc8bc0a1a80 0x7fc8bc0a01b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:24.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.592+0000 7fc8c44a1640 10 monclient(hunting): picked mon.noname-c con 0x7fc8bc0a1a80 addr v2:172.21.15.189:3300/0 2024-04-22T08:18:24.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.592+0000 7fc8c44a1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc8bc0a1e60 addr v1:172.21.15.189:6789/0 2024-04-22T08:18:24.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.592+0000 7fc8c44a1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc8bc13b0e0 addr v1:172.21.15.40:6789/0 2024-04-22T08:18:24.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.592+0000 7fc8c44a1640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8bc09d7f0 con 0x7fc8bc13b0e0 2024-04-22T08:18:24.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.592+0000 7fc8c44a1640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8bc09dc30 con 0x7fc8bc0a1e60 2024-04-22T08:18:24.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.592+0000 7fc8c44a1640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:24.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.592+0000 7fc8c44a1640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8bc09f1c0 con 0x7fc8bc0a1a80 2024-04-22T08:18:24.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.592+0000 7fc8c44a1640 10 monclient(hunting): _renew_subs 2024-04-22T08:18:24.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.592+0000 7fc8c44a1640 10 monclient(hunting): authenticate will time out at 1178.585941s 2024-04-22T08:18:24.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.592+0000 7fc8c2a3e640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fc8bc13b0e0 0x7fc8bc13d4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:36376/0 (socket says 172.21.15.40:36376) 2024-04-22T08:18:24.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.592+0000 7fc8c2a3e640 1 -- 172.21.15.40:0/4292571885 learned_addr learned my addr 172.21.15.40:0/4292571885 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:18:24.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.592+0000 7fc8c223d640 1 --2- 172.21.15.40:0/4292571885 >> v2:172.21.15.189:3300/0 conn(0x7fc8bc0a1a80 0x7fc8bc0a01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:24.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.592+0000 7fc8c223d640 10 monclient(hunting): get_auth_request con 0x7fc8bc0a1a80 auth_method 0 2024-04-22T08:18:24.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.592+0000 7fc8c223d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:24.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.592+0000 7fc8c223d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:24.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.592+0000 7fc8c223d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:24.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.592+0000 7fc8c223d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:24.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.592+0000 7fc8c223d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:24.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.592+0000 7fc8c223d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:24.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.592+0000 7fc8c223d640 10 monclient(hunting): handle_auth_done global_id 64267 payload 274 2024-04-22T08:18:24.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.592+0000 7fc8c223d640 1 -- 172.21.15.40:0/4292571885 >> v1:172.21.15.189:6789/0 conn(0x7fc8bc0a1e60 legacy=0x7fc8bc0a22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:18:24.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.592+0000 7fc8c223d640 1 -- 172.21.15.40:0/4292571885 >> v1:172.21.15.40:6789/0 conn(0x7fc8bc13b0e0 legacy=0x7fc8bc13d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:18:24.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.592+0000 7fc8c223d640 10 monclient: _finish_hunting 0 2024-04-22T08:18:24.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.592+0000 7fc8c223d640 1 monclient: found mon.noname-c 2024-04-22T08:18:24.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.592+0000 7fc8c223d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:18:24.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.592+0000 7fc8c223d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.189:3300/0 2024-04-22T08:18:24.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.592+0000 7fc8c223d640 1 -- 172.21.15.40:0/4292571885 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8bc09eee0 con 0x7fc8bc0a1a80 2024-04-22T08:18:24.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.595+0000 7fc8c223d640 1 -- 172.21.15.40:0/4292571885 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8bc0a1a80 msgr2=0x7fc8bc0a01b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:18:24.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.595+0000 7fc8c223d640 1 --2- 172.21.15.40:0/4292571885 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8bc0a1a80 0x7fc8bc0a01b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac009d90 tx=0x7fc8ac02bd00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c664fa020522a692 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:24.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.595+0000 7fc8c123b640 1 -- 172.21.15.40:0/4292571885 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ac02cda0 con 0x7fc8bc0a1a80 2024-04-22T08:18:24.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.595+0000 7fc8c123b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:24.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.596+0000 7fc8c123b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:18:24.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.596+0000 7fc8c123b640 10 monclient: dump: 2024-04-22T08:18:24.601 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:24.601 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:24.601 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.601 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.601 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:24.601 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:24.601 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:24.601 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:24.601 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:24.601 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:24.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.596+0000 7fc8c123b640 10 monclient: _finish_auth 0 2024-04-22T08:18:24.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.596+0000 7fc8c123b640 10 monclient: _check_auth_tickets 2024-04-22T08:18:24.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.596+0000 7fc8c123b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:18:24.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.596+0000 7fc8c44a1640 5 monclient: authenticate success, global_id 64267 2024-04-22T08:18:24.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.596+0000 7fc8c123b640 1 -- 172.21.15.40:0/4292571885 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8ac034440 con 0x7fc8bc0a1a80 2024-04-22T08:18:24.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.596+0000 7fc8c123b640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:18:24.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.596+0000 7fc8c44a1640 10 monclient: get_monmap_and_config success 2024-04-22T08:18:24.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.596+0000 7fc8c44a1640 4 set_mon_vals no callback set 2024-04-22T08:18:24.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.596+0000 7fc8c123b640 1 -- 172.21.15.40:0/4292571885 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ac0347d0 con 0x7fc8bc0a1a80 2024-04-22T08:18:24.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.596+0000 7fc8c123b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:24.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.596+0000 7fc8c123b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:18:24.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.596+0000 7fc8c123b640 10 monclient: dump: 2024-04-22T08:18:24.602 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:24.602 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:24.602 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.602 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.602 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:24.602 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:24.603 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:24.603 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:24.603 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:24.603 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:24.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.596+0000 7fc8c44a1640 10 monclient: shutdown 2024-04-22T08:18:24.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.596+0000 7fc8c44a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:18:24.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.596+0000 7fc8c44a1640 1 -- 172.21.15.40:0/4292571885 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8bc0a1a80 msgr2=0x7fc8bc0a01b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:24.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.596+0000 7fc8c44a1640 1 --2- 172.21.15.40:0/4292571885 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8bc0a1a80 0x7fc8bc0a01b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac009d90 tx=0x7fc8ac02bd00 comp rx=0 tx=0).stop 2024-04-22T08:18:24.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.596+0000 7fc8c44a1640 1 -- 172.21.15.40:0/4292571885 shutdown_connections 2024-04-22T08:18:24.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.596+0000 7fc8c44a1640 1 --2- 172.21.15.40:0/4292571885 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8bc0a1a80 0x7fc8bc0a01b0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:24.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.596+0000 7fc8c44a1640 1 -- 172.21.15.40:0/4292571885 >> 172.21.15.40:0/4292571885 conn(0x7fc8bc0a66a0 msgr2=0x7fc8bc0a2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:18:24.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.596+0000 7fc8c44a1640 1 -- 172.21.15.40:0/4292571885 shutdown_connections 2024-04-22T08:18:24.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.596+0000 7fc8c44a1640 1 -- 172.21.15.40:0/4292571885 wait complete. 2024-04-22T08:18:24.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.597+0000 7fc8c44a1640 10 monclient: build_initial_monmap 2024-04-22T08:18:24.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.597+0000 7fc8c44a1640 10 monclient: monmap: 2024-04-22T08:18:24.604 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:18:24.604 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:18:24.604 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:18:24.604 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:18:24.604 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:18:24.604 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:24.604 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:18:24.604 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:18:24.604 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:18:24.604 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:24.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.597+0000 7fc8c44a1640 1 Processor -- start 2024-04-22T08:18:24.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.597+0000 7fc8c44a1640 1 -- start start 2024-04-22T08:18:24.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.597+0000 7fc8c44a1640 10 monclient: init 2024-04-22T08:18:24.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.597+0000 7fc8c44a1640 10 monclient: _reopen_session rank -1 2024-04-22T08:18:24.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.597+0000 7fc8c44a1640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:18:24.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.597+0000 7fc8c44a1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8bc0a1a80 0x7fc8bc070ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:24.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.597+0000 7fc8c44a1640 10 monclient(hunting): picked mon.noname-a con 0x7fc8bc0a1a80 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:18:24.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.597+0000 7fc8c44a1640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8bc0a1e60 0x7fc8bc139e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:24.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.597+0000 7fc8c44a1640 10 monclient(hunting): picked mon.noname-b con 0x7fc8bc0a1e60 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:18:24.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.597+0000 7fc8c44a1640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc8bc13b0e0 0x7fc8bc120ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:24.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.597+0000 7fc8c44a1640 10 monclient(hunting): picked mon.noname-c con 0x7fc8bc13b0e0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:18:24.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.597+0000 7fc8c44a1640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:24.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.597+0000 7fc8c44a1640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8bc121e40 con 0x7fc8bc0a1a80 2024-04-22T08:18:24.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.597+0000 7fc8c44a1640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:24.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.597+0000 7fc8c44a1640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8bc099d40 con 0x7fc8bc0a1e60 2024-04-22T08:18:24.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.597+0000 7fc8c44a1640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:24.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.597+0000 7fc8c44a1640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8bc09cc20 con 0x7fc8bc13b0e0 2024-04-22T08:18:24.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.597+0000 7fc8c44a1640 10 monclient(hunting): _renew_subs 2024-04-22T08:18:24.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.597+0000 7fc8c44a1640 10 monclient(hunting): authenticate will time out at 1178.591706s 2024-04-22T08:18:24.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.598+0000 7fc8c223d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8bc0a1a80 0x7fc8bc070ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:24.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.598+0000 7fc8c223d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8bc0a1a80 0x7fc8bc070ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44616/0 (socket says 172.21.15.40:44616) 2024-04-22T08:18:24.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.598+0000 7fc8c223d640 1 -- 172.21.15.40:0/4030153123 learned_addr learned my addr 172.21.15.40:0/4030153123 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:18:24.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.598+0000 7fc8c223d640 10 monclient(hunting): get_auth_request con 0x7fc8bc0a1a80 auth_method 0 2024-04-22T08:18:24.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.598+0000 7fc8c223d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:24.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.598+0000 7fc8c223d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:24.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.598+0000 7fc8c223d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:24.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.598+0000 7fc8c2a3e640 1 --2- 172.21.15.40:0/4030153123 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc8bc13b0e0 0x7fc8bc120ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:24.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.598+0000 7fc8c2a3e640 10 monclient(hunting): get_auth_request con 0x7fc8bc13b0e0 auth_method 0 2024-04-22T08:18:24.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.598+0000 7fc8c2a3e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:24.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.598+0000 7fc8c2a3e640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:24.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.598+0000 7fc8c2a3e640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:24.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.598+0000 7fc8c223d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:24.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.598+0000 7fc8c223d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:24.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.598+0000 7fc8c223d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:24.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.598+0000 7fc8c2a3e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:24.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.598+0000 7fc8c2a3e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:24.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.598+0000 7fc8c2a3e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:24.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.598+0000 7fc8c1a3c640 1 --2- 172.21.15.40:0/4030153123 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8bc0a1e60 0x7fc8bc139e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:24.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.598+0000 7fc8c1a3c640 10 monclient(hunting): get_auth_request con 0x7fc8bc0a1e60 auth_method 0 2024-04-22T08:18:24.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.598+0000 7fc8c1a3c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:24.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.598+0000 7fc8c1a3c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:24.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.598+0000 7fc8c1a3c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:24.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.598+0000 7fc8c223d640 10 monclient(hunting): handle_auth_done global_id 74112 payload 824 2024-04-22T08:18:24.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.598+0000 7fc8c223d640 1 -- 172.21.15.40:0/4030153123 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc8bc13b0e0 msgr2=0x7fc8bc120ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:24.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.598+0000 7fc8c223d640 1 --2- 172.21.15.40:0/4030153123 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc8bc13b0e0 0x7fc8bc120ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:24.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.598+0000 7fc8c223d640 1 -- 172.21.15.40:0/4030153123 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8bc0a1e60 msgr2=0x7fc8bc139e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:24.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.598+0000 7fc8c223d640 1 --2- 172.21.15.40:0/4030153123 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8bc0a1e60 0x7fc8bc139e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:24.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.598+0000 7fc8c223d640 10 monclient: _finish_hunting 0 2024-04-22T08:18:24.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.598+0000 7fc8c223d640 1 monclient: found mon.noname-a 2024-04-22T08:18:24.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.598+0000 7fc8c223d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:18:24.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.598+0000 7fc8c223d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:18:24.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.598+0000 7fc8c223d640 1 -- 172.21.15.40:0/4030153123 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8ac009a70 con 0x7fc8bc0a1a80 2024-04-22T08:18:24.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.598+0000 7fc8c2a3e640 1 --2- 172.21.15.40:0/4030153123 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc8bc13b0e0 0x7fc8bc120ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:18:24.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.600+0000 7fc8c223d640 1 -- 172.21.15.40:0/4030153123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8bc0a1a80 msgr2=0x7fc8bc070ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:18:24.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.600+0000 7fc8c223d640 1 --2- 172.21.15.40:0/4030153123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8bc0a1a80 0x7fc8bc070ac0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac037040 tx=0x7fc8ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:24.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.600+0000 7fc8baffd640 1 -- 172.21.15.40:0/4030153123 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ac040650 con 0x7fc8bc0a1a80 2024-04-22T08:18:24.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.600+0000 7fc8baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:24.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.600+0000 7fc8baffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:18:24.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.600+0000 7fc8baffd640 10 monclient: dump: 2024-04-22T08:18:24.609 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:24.609 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:24.609 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.609 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.609 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:24.609 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:24.609 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:24.609 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:24.609 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:24.609 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:24.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.600+0000 7fc8baffd640 10 monclient: _finish_auth 0 2024-04-22T08:18:24.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.600+0000 7fc8baffd640 10 monclient: _check_auth_tickets 2024-04-22T08:18:24.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.600+0000 7fc8baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:18:24.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.600+0000 7fc8c44a1640 5 monclient: authenticate success, global_id 74112 2024-04-22T08:18:24.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.600+0000 7fc8baffd640 1 -- 172.21.15.40:0/4030153123 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8ac0407b0 con 0x7fc8bc0a1a80 2024-04-22T08:18:24.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.600+0000 7fc8c44a1640 10 monclient: _renew_subs 2024-04-22T08:18:24.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.600+0000 7fc8c44a1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:18:24.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.600+0000 7fc8c44a1640 1 -- 172.21.15.40:0/4030153123 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8bc123810 con 0x7fc8bc0a1a80 2024-04-22T08:18:24.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.600+0000 7fc8baffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:18:24.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.600+0000 7fc8baffd640 1 -- 172.21.15.40:0/4030153123 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ac040b60 con 0x7fc8bc0a1a80 2024-04-22T08:18:24.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.600+0000 7fc8baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:24.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.600+0000 7fc8baffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:18:24.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.600+0000 7fc8baffd640 10 monclient: dump: 2024-04-22T08:18:24.610 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:24.610 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:24.610 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.610 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.610 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:24.610 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:24.611 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:24.611 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:24.611 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:24.611 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:24.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.600+0000 7fc8c44a1640 10 monclient: _renew_subs 2024-04-22T08:18:24.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.600+0000 7fc8c0a3a640 4 set_mon_vals no callback set 2024-04-22T08:18:24.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.601+0000 7fc8c44a1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:18:24.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.601+0000 7fc8c44a1640 1 -- 172.21.15.40:0/4030153123 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8bc1cb9c0 con 0x7fc8bc0a1a80 2024-04-22T08:18:24.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.601+0000 7fc8c44a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:18:24.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.601+0000 7fc8c44a1640 1 --2- 172.21.15.40:0/4030153123 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc8bc121010 0x7fc8bc1213f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:24.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.601+0000 7fc8c44a1640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:24.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.601+0000 7fc8c44a1640 1 -- 172.21.15.40:0/4030153123 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8bc066770 con 0x7fc8bc121010 2024-04-22T08:18:24.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.601+0000 7fc8baffd640 1 -- 172.21.15.40:0/4030153123 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fc8ac040cc0 con 0x7fc8bc0a1a80 2024-04-22T08:18:24.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.601+0000 7fc8c1a3c640 1 --2- 172.21.15.40:0/4030153123 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc8bc121010 0x7fc8bc1213f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:24.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.601+0000 7fc8c1a3c640 10 monclient: get_auth_request con 0x7fc8bc121010 auth_method 0 2024-04-22T08:18:24.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.601+0000 7fc8c1a3c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:24.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.601+0000 7fc8c1a3c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:24.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.601+0000 7fc8c1a3c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:24.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.601+0000 7fc8c1a3c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:24.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.601+0000 7fc8c1a3c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:24.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.601+0000 7fc8c1a3c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:24.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.602+0000 7fc8baffd640 1 --2- 172.21.15.40:0/4030153123 >> v2:172.21.15.189:6800/30168 conn(0x7fc8a80377d0 0x7fc8a8039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:24.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.602+0000 7fc8baffd640 1 -- 172.21.15.40:0/4030153123 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 3898+0+0 (secure 0 0 0) 0x7fc8ac06d0c0 con 0x7fc8bc0a1a80 2024-04-22T08:18:24.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.602+0000 7fc8c1a3c640 10 monclient(hunting): handle_auth_done global_id 54458 payload 824 2024-04-22T08:18:24.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.602+0000 7fc8c1a3c640 1 -- 172.21.15.40:0/4030153123 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc8bc13ebf0 con 0x7fc8bc121010 2024-04-22T08:18:24.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.602+0000 7fc8c2a3e640 1 --2- 172.21.15.40:0/4030153123 >> v2:172.21.15.189:6800/30168 conn(0x7fc8a80377d0 0x7fc8a8039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:24.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.602+0000 7fc8c2a3e640 10 monclient: get_auth_request con 0x7fc8a80377d0 auth_method 0 2024-04-22T08:18:24.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.604+0000 7fc8c1a3c640 1 -- 172.21.15.40:0/4030153123 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc8bc121010 msgr2=0x7fc8bc1213f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:18:24.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.604+0000 7fc8c1a3c640 1 --2- 172.21.15.40:0/4030153123 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc8bc121010 0x7fc8bc1213f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc09eab0 tx=0x7fc8b000e960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:24.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.604+0000 7fc8baffd640 1 -- 172.21.15.40:0/4030153123 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b0007450 con 0x7fc8bc121010 2024-04-22T08:18:24.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.604+0000 7fc8baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:24.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.604+0000 7fc8baffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:18:24.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.604+0000 7fc8baffd640 10 monclient: dump: 2024-04-22T08:18:24.616 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:24.616 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:24.616 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.616 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.616 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:24.616 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:24.616 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:24.616 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:24.616 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:24.616 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:24.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.604+0000 7fc8baffd640 1 -- 172.21.15.40:0/4030153123 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7fc8b00075b0 con 0x7fc8bc121010 2024-04-22T08:18:24.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.604+0000 7fc8baffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:18:24.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.604+0000 7fc8baffd640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:18:24.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.604+0000 7fc8baffd640 1 -- 172.21.15.40:0/4030153123 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc8bc121010 msgr2=0x7fc8bc1213f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:24.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.604+0000 7fc8baffd640 1 --2- 172.21.15.40:0/4030153123 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc8bc121010 0x7fc8bc1213f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc09eab0 tx=0x7fc8b000e960 comp rx=0 tx=0).stop 2024-04-22T08:18:24.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.604+0000 7fc8baffd640 1 -- 172.21.15.40:0/4030153123 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc8bc121010 msgr2=0x7fc8bc1213f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:18:24.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.604+0000 7fc8baffd640 1 --2- 172.21.15.40:0/4030153123 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc8bc121010 0x7fc8bc1213f0 secure :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc09eab0 tx=0x7fc8b000e960 comp rx=0 tx=0).stop 2024-04-22T08:18:24.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.604+0000 7fc8c2a3e640 1 --2- 172.21.15.40:0/4030153123 >> v2:172.21.15.189:6800/30168 conn(0x7fc8a80377d0 0x7fc8a8039c90 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc8b4007c90 tx=0x7fc8b4007be0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:24.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.612+0000 7fc8c44a1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:18:24.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.612+0000 7fc8c44a1640 1 --2- 172.21.15.40:0/4030153123 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc8bc13d4d0 0x7fc8bc13d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:24.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.612+0000 7fc8c44a1640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:24.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.612+0000 7fc8c44a1640 1 -- 172.21.15.40:0/4030153123 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8bc123300 con 0x7fc8bc13d4d0 2024-04-22T08:18:24.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.612+0000 7fc8c1a3c640 1 --2- 172.21.15.40:0/4030153123 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc8bc13d4d0 0x7fc8bc13d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:24.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.612+0000 7fc8c1a3c640 10 monclient: get_auth_request con 0x7fc8bc13d4d0 auth_method 0 2024-04-22T08:18:24.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.612+0000 7fc8c1a3c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:24.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.612+0000 7fc8c1a3c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:24.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.612+0000 7fc8c1a3c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:24.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.612+0000 7fc8c1a3c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:24.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.612+0000 7fc8c1a3c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:24.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.612+0000 7fc8c1a3c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:24.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.612+0000 7fc8c1a3c640 10 monclient(hunting): handle_auth_done global_id 54461 payload 824 2024-04-22T08:18:24.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.612+0000 7fc8c1a3c640 1 -- 172.21.15.40:0/4030153123 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc8bc121930 con 0x7fc8bc13d4d0 2024-04-22T08:18:24.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.615+0000 7fc8c1a3c640 1 -- 172.21.15.40:0/4030153123 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc8bc13d4d0 msgr2=0x7fc8bc13d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:18:24.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.615+0000 7fc8c1a3c640 1 --2- 172.21.15.40:0/4030153123 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc8bc13d4d0 0x7fc8bc13d8e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc09eab0 tx=0x7fc8b0003760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:24.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.615+0000 7fc8baffd640 1 -- 172.21.15.40:0/4030153123 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b0013a40 con 0x7fc8bc13d4d0 2024-04-22T08:18:24.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.615+0000 7fc8baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:24.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.615+0000 7fc8baffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:18:24.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.615+0000 7fc8baffd640 10 monclient: dump: 2024-04-22T08:18:24.619 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:24.619 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:24.619 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.619 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.619 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:24.619 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:24.619 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:24.619 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:24.619 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:24.620 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:24.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.615+0000 7fc8baffd640 1 -- 172.21.15.40:0/4030153123 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4631 (secure 0 0 0) 0x7fc8b0013ba0 con 0x7fc8bc13d4d0 2024-04-22T08:18:24.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.615+0000 7fc8baffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:18:24.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.615+0000 7fc8baffd640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:18:24.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.615+0000 7fc8baffd640 1 -- 172.21.15.40:0/4030153123 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc8bc13d4d0 msgr2=0x7fc8bc13d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:24.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.615+0000 7fc8baffd640 1 --2- 172.21.15.40:0/4030153123 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc8bc13d4d0 0x7fc8bc13d8e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc09eab0 tx=0x7fc8b0003760 comp rx=0 tx=0).stop 2024-04-22T08:18:24.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.615+0000 7fc8baffd640 1 -- 172.21.15.40:0/4030153123 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc8bc13d4d0 msgr2=0x7fc8bc13d8e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:18:24.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.615+0000 7fc8baffd640 1 --2- 172.21.15.40:0/4030153123 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc8bc13d4d0 0x7fc8bc13d8e0 secure :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc09eab0 tx=0x7fc8b0003760 comp rx=0 tx=0).stop 2024-04-22T08:18:24.620 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:18:24.620 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:18:24.620 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:18:24.621 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:18:24.621 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 56, 2024-04-22T08:18:24.621 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:18:24.621 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:18:24.621 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:18:24.621 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:18:24.621 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:18:24.621 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 0, 2024-04-22T08:18:24.621 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:18:24.621 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:18:24.621 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:18:24.621 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:18:24.621 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:18:24.621 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:18:24.621 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:18:24.622 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:18:24.622 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:18:24.622 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:18:24.622 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:18:24.622 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:18:24.622 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:18:24.622 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:18:24.622 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:18:24.622 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:18:24.622 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:18:24.622 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:18:24.622 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:18:24.623 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:18:24.623 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:18:24.623 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:18:24.623 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:18:24.623 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:18:24.623 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:18:24.623 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:18:24.623 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:24.623 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:24.623 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:18:24.623 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:18:24.623 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:18:24.623 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:18:24.623 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:18:24.624 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:18:24.624 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:18:24.624 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:18:24.624 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:18:24.624 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:18:24.624 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:18:24.624 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:18:24.624 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:18:24.624 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:18:24.624 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:18:24.624 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:18:24.624 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:18:24.624 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:18:24.624 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:18:24.625 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:18:24.625 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:18:24.625 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:18:24.625 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:18:24.625 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:18:24.625 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:18:24.625 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:18:24.625 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:18:24.625 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:18:24.625 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:18:24.625 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:24.625 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:18:24.625 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:24.625 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:18:24.626 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:18:24.626 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:18:24.626 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:18:24.626 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:24.626 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:18:24.626 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:18:24.626 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:24.626 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:24.626 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:24.626 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:18:24.626 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:18:24.626 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:24.626 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:24.627 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:24.627 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:24.627 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:18:24.627 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:18:24.627 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:18:24.627 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:18:24.627 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:18:24.627 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:24.627 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:24.627 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:18:24.627 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:18:24.627 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:18:24.627 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:18:24.627 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:24.627 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:18:24.628 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:18:24.628 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:24.628 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:24.628 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:24.628 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:18:24.628 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:18:24.628 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:24.628 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:24.628 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:24.628 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:24.628 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:18:24.628 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:18:24.629 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:18:24.629 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:18:24.629 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:18:24.629 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:24.629 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:24.629 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:18:24.629 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:18:24.629 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:18:24.629 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:18:24.629 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:24.629 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:18:24.629 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:18:24.629 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:24.629 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:24.629 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:24.630 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:18:24.630 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:18:24.630 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:24.630 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:24.630 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:24.630 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:24.630 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:18:24.630 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:18:24.630 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:18:24.630 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:18:24.630 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:18:24.630 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:24.630 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:24.631 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:24.631 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:18:24.631 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:18:24.631 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:24.631 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:18:24.631 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:18:24.631 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:18:24.631 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:24.631 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:18:24.631 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:18:24.631 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:24.631 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:18:24.631 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:18:24.631 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 3 2024-04-22T08:18:24.632 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:24.632 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:18:24.632 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:18:24.632 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:24.632 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:18:24.632 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:18:24.632 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 5 2024-04-22T08:18:24.632 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:24.632 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:24.632 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:24.632 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:18:24.632 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:18:24.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.616+0000 7fc8c44a1640 1 -- 172.21.15.40:0/4030153123 >> v2:172.21.15.189:6800/30168 conn(0x7fc8a80377d0 msgr2=0x7fc8a8039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:24.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.616+0000 7fc8c44a1640 1 --2- 172.21.15.40:0/4030153123 >> v2:172.21.15.189:6800/30168 conn(0x7fc8a80377d0 0x7fc8a8039c90 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc8b4007c90 tx=0x7fc8b4007be0 comp rx=0 tx=0).stop 2024-04-22T08:18:24.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.616+0000 7fc8c44a1640 10 monclient: shutdown 2024-04-22T08:18:24.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.616+0000 7fc8c44a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:18:24.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.616+0000 7fc8c44a1640 1 -- 172.21.15.40:0/4030153123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8bc0a1a80 msgr2=0x7fc8bc070ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:24.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.616+0000 7fc8c44a1640 1 --2- 172.21.15.40:0/4030153123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8bc0a1a80 0x7fc8bc070ac0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac037040 tx=0x7fc8ac004a40 comp rx=0 tx=0).stop 2024-04-22T08:18:24.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.616+0000 7fc8c2a3e640 1 -- 172.21.15.40:0/4030153123 reap_dead start 2024-04-22T08:18:24.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.616+0000 7fc8c2a3e640 1 -- 172.21.15.40:0/4030153123 reap_dead start 2024-04-22T08:18:24.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.616+0000 7fc8c44a1640 1 -- 172.21.15.40:0/4030153123 shutdown_connections 2024-04-22T08:18:24.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.616+0000 7fc8c44a1640 1 -- 172.21.15.40:0/4030153123 >> 172.21.15.40:0/4030153123 conn(0x7fc8bc0a66a0 msgr2=0x7fc8bc09bcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:18:24.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.616+0000 7fc8c44a1640 1 -- 172.21.15.40:0/4030153123 shutdown_connections 2024-04-22T08:18:24.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.616+0000 7fc8c44a1640 1 -- 172.21.15.40:0/4030153123 wait complete. 2024-04-22T08:18:24.635 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:18:24.635 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-22T08:18:24.635 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-22T08:18:24.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.735+0000 7fc563692640 10 monclient: get_monmap_and_config 2024-04-22T08:18:24.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.735+0000 7fc563692640 10 monclient: build_initial_monmap 2024-04-22T08:18:24.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.735+0000 7fc563692640 10 monclient: monmap: 2024-04-22T08:18:24.735 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:18:24.735 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:24.735 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:18:24.736514+0000 2024-04-22T08:18:24.735 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:18:24.736514+0000 2024-04-22T08:18:24.736 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:18:24.736 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:24.736 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:18:24.736 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:18:24.736 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:18:24.736 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:18:24.736 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:18:24.736 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:24.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.736+0000 7fc563692640 1 Processor -- start 2024-04-22T08:18:24.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.736+0000 7fc563692640 1 -- start start 2024-04-22T08:18:24.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.736+0000 7fc563692640 10 monclient: init 2024-04-22T08:18:24.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.736+0000 7fc563692640 10 monclient: _reopen_session rank -1 2024-04-22T08:18:24.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.736+0000 7fc563692640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-04-22T08:18:24.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.736+0000 7fc563692640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc55c070e90 0x7fc55c071270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:24.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.737+0000 7fc563692640 10 monclient(hunting): picked mon.noname-a con 0x7fc55c070e90 addr v2:172.21.15.40:3300/0 2024-04-22T08:18:24.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.737+0000 7fc563692640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc55c135c10 0x7fc55c09de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:24.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.737+0000 7fc563692640 10 monclient(hunting): picked mon.noname-b con 0x7fc55c135c10 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:18:24.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.737+0000 7fc563692640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc55c136040 addr v1:172.21.15.40:6789/0 2024-04-22T08:18:24.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.737+0000 7fc563692640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc55c136940 con 0x7fc55c136040 2024-04-22T08:18:24.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.737+0000 7fc563692640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:24.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.737+0000 7fc563692640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc55c06d170 con 0x7fc55c070e90 2024-04-22T08:18:24.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.737+0000 7fc563692640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:24.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.737+0000 7fc563692640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc55c0717b0 con 0x7fc55c135c10 2024-04-22T08:18:24.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.737+0000 7fc563692640 10 monclient(hunting): _renew_subs 2024-04-22T08:18:24.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.737+0000 7fc563692640 10 monclient(hunting): authenticate will time out at 1178.730853s 2024-04-22T08:18:24.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.737+0000 7fc560c2d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc55c135c10 0x7fc55c09de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:24.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.737+0000 7fc56142e640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc55c070e90 0x7fc55c071270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:24.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.737+0000 7fc560c2d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc55c135c10 0x7fc55c09de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:51424/0 (socket says 172.21.15.40:51424) 2024-04-22T08:18:24.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.737+0000 7fc56142e640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc55c070e90 0x7fc55c071270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44632/0 (socket says 172.21.15.40:44632) 2024-04-22T08:18:24.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.737+0000 7fc560c2d640 1 -- 172.21.15.40:0/4058357856 learned_addr learned my addr 172.21.15.40:0/4058357856 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:18:24.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.737+0000 7fc560c2d640 10 monclient(hunting): get_auth_request con 0x7fc55c135c10 auth_method 0 2024-04-22T08:18:24.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.737+0000 7fc561c2f640 1 --1- 172.21.15.40:0/4058357856 >> v1:172.21.15.40:6789/0 conn(0x7fc55c136040 0x7fc55c09e340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:36388/0 (socket says 172.21.15.40:36388) 2024-04-22T08:18:24.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.737+0000 7fc560c2d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:24.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.737+0000 7fc560c2d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:24.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.737+0000 7fc560c2d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:24.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.737+0000 7fc56142e640 10 monclient(hunting): get_auth_request con 0x7fc55c070e90 auth_method 0 2024-04-22T08:18:24.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.737+0000 7fc56142e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:24.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.737+0000 7fc56142e640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:24.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.737+0000 7fc56142e640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:24.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.737+0000 7fc56142e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:24.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.737+0000 7fc56142e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:24.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.737+0000 7fc56142e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:24.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.737+0000 7fc560c2d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:24.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.737+0000 7fc560c2d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:24.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.737+0000 7fc560c2d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:24.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.737+0000 7fc56142e640 10 monclient(hunting): handle_auth_done global_id 74115 payload 274 2024-04-22T08:18:24.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.737+0000 7fc56142e640 1 -- 172.21.15.40:0/4058357856 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc55c135c10 msgr2=0x7fc55c09de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:18:24.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.737+0000 7fc56142e640 1 --2- 172.21.15.40:0/4058357856 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc55c135c10 0x7fc55c09de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:24.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.737+0000 7fc56142e640 1 -- 172.21.15.40:0/4058357856 >> v1:172.21.15.40:6789/0 conn(0x7fc55c136040 legacy=0x7fc55c09e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:18:24.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.737+0000 7fc56142e640 10 monclient: _finish_hunting 0 2024-04-22T08:18:24.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.737+0000 7fc56142e640 1 monclient: found mon.noname-a 2024-04-22T08:18:24.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.737+0000 7fc56142e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:18:24.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.737+0000 7fc56142e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:18:24.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.737+0000 7fc56142e640 1 -- 172.21.15.40:0/4058357856 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc55c06d450 con 0x7fc55c070e90 2024-04-22T08:18:24.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.737+0000 7fc560c2d640 1 --2- 172.21.15.40:0/4058357856 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc55c135c10 0x7fc55c09de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:18:24.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.740+0000 7fc56142e640 1 -- 172.21.15.40:0/4058357856 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc55c070e90 msgr2=0x7fc55c071270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:18:24.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.740+0000 7fc56142e640 1 --2- 172.21.15.40:0/4058357856 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc55c070e90 0x7fc55c071270 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc5580097a0 tx=0x7fc55802b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df2183dd262debe5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:24.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.740+0000 7fc553fff640 1 -- 172.21.15.40:0/4058357856 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc558030920 con 0x7fc55c070e90 2024-04-22T08:18:24.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.740+0000 7fc553fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:24.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.740+0000 7fc553fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:18:24.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.740+0000 7fc553fff640 10 monclient: dump: 2024-04-22T08:18:24.748 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:24.748 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:24.748 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.748 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.748 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:24.749 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:24.749 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:24.749 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:24.749 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:24.749 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:24.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.740+0000 7fc553fff640 10 monclient: _finish_auth 0 2024-04-22T08:18:24.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.740+0000 7fc553fff640 10 monclient: _check_auth_tickets 2024-04-22T08:18:24.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.740+0000 7fc553fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:18:24.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.740+0000 7fc563692640 5 monclient: authenticate success, global_id 74115 2024-04-22T08:18:24.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.740+0000 7fc553fff640 1 -- 172.21.15.40:0/4058357856 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc55802e050 con 0x7fc55c070e90 2024-04-22T08:18:24.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.740+0000 7fc563692640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:18:24.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.740+0000 7fc553fff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:18:24.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.740+0000 7fc563692640 10 monclient: get_monmap_and_config success 2024-04-22T08:18:24.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.740+0000 7fc563692640 4 set_mon_vals no callback set 2024-04-22T08:18:24.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.740+0000 7fc553fff640 1 -- 172.21.15.40:0/4058357856 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc558030ca0 con 0x7fc55c070e90 2024-04-22T08:18:24.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.740+0000 7fc553fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:24.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.740+0000 7fc553fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:18:24.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.740+0000 7fc553fff640 10 monclient: dump: 2024-04-22T08:18:24.750 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:24.750 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:24.750 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.750 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.750 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:24.750 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:24.750 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:24.750 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:24.750 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:24.750 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:24.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.740+0000 7fc563692640 10 monclient: shutdown 2024-04-22T08:18:24.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.740+0000 7fc563692640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:18:24.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.740+0000 7fc563692640 1 -- 172.21.15.40:0/4058357856 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc55c070e90 msgr2=0x7fc55c071270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:24.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.740+0000 7fc563692640 1 --2- 172.21.15.40:0/4058357856 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc55c070e90 0x7fc55c071270 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc5580097a0 tx=0x7fc55802b710 comp rx=0 tx=0).stop 2024-04-22T08:18:24.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.741+0000 7fc563692640 1 -- 172.21.15.40:0/4058357856 shutdown_connections 2024-04-22T08:18:24.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.741+0000 7fc563692640 1 --2- 172.21.15.40:0/4058357856 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc55c135c10 0x7fc55c09de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:24.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.741+0000 7fc563692640 1 --2- 172.21.15.40:0/4058357856 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc55c070e90 0x7fc55c071270 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:24.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.741+0000 7fc563692640 1 -- 172.21.15.40:0/4058357856 >> 172.21.15.40:0/4058357856 conn(0x7fc55c0a1780 msgr2=0x7fc55c0a1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:18:24.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.741+0000 7fc563692640 1 -- 172.21.15.40:0/4058357856 shutdown_connections 2024-04-22T08:18:24.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.741+0000 7fc563692640 1 -- 172.21.15.40:0/4058357856 wait complete. 2024-04-22T08:18:24.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.741+0000 7fc563692640 10 monclient: build_initial_monmap 2024-04-22T08:18:24.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.741+0000 7fc563692640 10 monclient: monmap: 2024-04-22T08:18:24.751 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:18:24.751 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:18:24.752 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:18:24.752 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:18:24.752 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:18:24.752 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:24.752 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:18:24.752 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:18:24.752 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:18:24.752 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:24.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc563692640 1 Processor -- start 2024-04-22T08:18:24.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc563692640 1 -- start start 2024-04-22T08:18:24.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc563692640 10 monclient: init 2024-04-22T08:18:24.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc563692640 10 monclient: _reopen_session rank -1 2024-04-22T08:18:24.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc563692640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-22T08:18:24.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc563692640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc55c070e90 0x7fc55c1cab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:24.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc563692640 10 monclient(hunting): picked mon.noname-c con 0x7fc55c070e90 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:18:24.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc563692640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc55c135c10 0x7fc55c1cb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:24.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc563692640 10 monclient(hunting): picked mon.noname-b con 0x7fc55c135c10 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:18:24.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc563692640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc55c136040 0x7fc55c06f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:24.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc563692640 10 monclient(hunting): picked mon.noname-a con 0x7fc55c136040 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:18:24.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc563692640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:24.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc563692640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc55c11d020 con 0x7fc55c136040 2024-04-22T08:18:24.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc563692640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:24.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc563692640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc55c12ccb0 con 0x7fc55c135c10 2024-04-22T08:18:24.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc563692640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:24.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc563692640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc55c11b280 con 0x7fc55c070e90 2024-04-22T08:18:24.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc563692640 10 monclient(hunting): _renew_subs 2024-04-22T08:18:24.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc563692640 10 monclient(hunting): authenticate will time out at 1178.736375s 2024-04-22T08:18:24.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc56142e640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc55c070e90 0x7fc55c1cab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:24.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc561c2f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc55c136040 0x7fc55c06f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:24.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc56142e640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc55c070e90 0x7fc55c1cab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:51434/0 (socket says 172.21.15.40:51434) 2024-04-22T08:18:24.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc56142e640 1 -- 172.21.15.40:0/640724072 learned_addr learned my addr 172.21.15.40:0/640724072 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:18:24.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc56142e640 10 monclient(hunting): get_auth_request con 0x7fc55c070e90 auth_method 0 2024-04-22T08:18:24.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc56142e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:24.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc56142e640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:24.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc56142e640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:24.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc561c2f640 10 monclient(hunting): get_auth_request con 0x7fc55c136040 auth_method 0 2024-04-22T08:18:24.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc561c2f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:24.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc561c2f640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:24.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc561c2f640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:24.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc561c2f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:24.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc561c2f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:24.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc561c2f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:24.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc56142e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:24.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc56142e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:24.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc56142e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:24.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc560c2d640 1 --2- 172.21.15.40:0/640724072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc55c135c10 0x7fc55c1cb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:24.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc560c2d640 10 monclient(hunting): get_auth_request con 0x7fc55c135c10 auth_method 0 2024-04-22T08:18:24.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc560c2d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:24.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc560c2d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:24.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc560c2d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:24.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc561c2f640 10 monclient(hunting): handle_auth_done global_id 74118 payload 824 2024-04-22T08:18:24.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc561c2f640 1 -- 172.21.15.40:0/640724072 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc55c070e90 msgr2=0x7fc55c1cab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:24.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc561c2f640 1 --2- 172.21.15.40:0/640724072 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc55c070e90 0x7fc55c1cab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:24.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc561c2f640 1 -- 172.21.15.40:0/640724072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc55c135c10 msgr2=0x7fc55c1cb0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:24.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc561c2f640 1 --2- 172.21.15.40:0/640724072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc55c135c10 0x7fc55c1cb0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:24.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc561c2f640 10 monclient: _finish_hunting 0 2024-04-22T08:18:24.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.742+0000 7fc561c2f640 1 monclient: found mon.noname-a 2024-04-22T08:18:24.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.743+0000 7fc561c2f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:18:24.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.743+0000 7fc561c2f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:18:24.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.743+0000 7fc561c2f640 1 -- 172.21.15.40:0/640724072 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc54c0094d0 con 0x7fc55c136040 2024-04-22T08:18:24.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.743+0000 7fc56142e640 1 --2- 172.21.15.40:0/640724072 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc55c070e90 0x7fc55c1cab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:18:24.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.745+0000 7fc561c2f640 1 -- 172.21.15.40:0/640724072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc55c136040 msgr2=0x7fc55c06f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:18:24.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.745+0000 7fc561c2f640 1 --2- 172.21.15.40:0/640724072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc55c136040 0x7fc55c06f670 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc55400c780 tx=0x7fc55400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:24.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.745+0000 7fc551ffb640 1 -- 172.21.15.40:0/640724072 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55400a840 con 0x7fc55c136040 2024-04-22T08:18:24.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.745+0000 7fc551ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:24.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.745+0000 7fc551ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:18:24.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.745+0000 7fc551ffb640 10 monclient: dump: 2024-04-22T08:18:24.756 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:24.756 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:24.756 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.756 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.756 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:24.756 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:24.756 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:24.757 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:24.757 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:24.757 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:24.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.745+0000 7fc551ffb640 10 monclient: _finish_auth 0 2024-04-22T08:18:24.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.745+0000 7fc551ffb640 10 monclient: _check_auth_tickets 2024-04-22T08:18:24.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.745+0000 7fc551ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:18:24.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.745+0000 7fc551ffb640 1 -- 172.21.15.40:0/640724072 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc55400a9a0 con 0x7fc55c136040 2024-04-22T08:18:24.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.745+0000 7fc563692640 5 monclient: authenticate success, global_id 74118 2024-04-22T08:18:24.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.745+0000 7fc563692640 10 monclient: _renew_subs 2024-04-22T08:18:24.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.745+0000 7fc563692640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:18:24.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.745+0000 7fc563692640 1 -- 172.21.15.40:0/640724072 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc558009400 con 0x7fc55c136040 2024-04-22T08:18:24.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.745+0000 7fc551ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:18:24.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.745+0000 7fc551ffb640 1 -- 172.21.15.40:0/640724072 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55400ad50 con 0x7fc55c136040 2024-04-22T08:18:24.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.745+0000 7fc551ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:24.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.745+0000 7fc551ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:18:24.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.745+0000 7fc551ffb640 10 monclient: dump: 2024-04-22T08:18:24.758 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:24.758 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:24.758 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.758 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.758 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:24.758 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:24.758 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:24.758 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:24.758 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:24.758 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:24.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.745+0000 7fc5537fe640 4 set_mon_vals no callback set 2024-04-22T08:18:24.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.745+0000 7fc563692640 10 monclient: _renew_subs 2024-04-22T08:18:24.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.745+0000 7fc563692640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:18:24.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.745+0000 7fc563692640 1 -- 172.21.15.40:0/640724072 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc55c11cd40 con 0x7fc55c136040 2024-04-22T08:18:24.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.746+0000 7fc563692640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:18:24.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.746+0000 7fc551ffb640 1 -- 172.21.15.40:0/640724072 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fc554007070 con 0x7fc55c136040 2024-04-22T08:18:24.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.746+0000 7fc563692640 1 --2- 172.21.15.40:0/640724072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc55c070370 0x7fc55c02b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:24.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.746+0000 7fc563692640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:24.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.746+0000 7fc563692640 1 -- 172.21.15.40:0/640724072 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc55c066900 con 0x7fc55c070370 2024-04-22T08:18:24.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.746+0000 7fc551ffb640 1 --2- 172.21.15.40:0/640724072 >> v2:172.21.15.189:6800/30168 conn(0x7fc5400377d0 0x7fc540039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:24.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.746+0000 7fc551ffb640 1 -- 172.21.15.40:0/640724072 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 3898+0+0 (secure 0 0 0) 0x7fc554049c10 con 0x7fc55c136040 2024-04-22T08:18:24.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.746+0000 7fc56142e640 1 --2- 172.21.15.40:0/640724072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc55c070370 0x7fc55c02b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:24.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.746+0000 7fc56142e640 10 monclient: get_auth_request con 0x7fc55c070370 auth_method 0 2024-04-22T08:18:24.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.746+0000 7fc56142e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:24.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.746+0000 7fc56142e640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:24.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.746+0000 7fc56142e640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:24.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.746+0000 7fc560c2d640 1 --2- 172.21.15.40:0/640724072 >> v2:172.21.15.189:6800/30168 conn(0x7fc5400377d0 0x7fc540039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:24.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.746+0000 7fc560c2d640 10 monclient: get_auth_request con 0x7fc5400377d0 auth_method 0 2024-04-22T08:18:24.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.746+0000 7fc56142e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:24.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.746+0000 7fc56142e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:24.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.746+0000 7fc56142e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:24.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.747+0000 7fc56142e640 10 monclient(hunting): handle_auth_done global_id 64276 payload 824 2024-04-22T08:18:24.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.747+0000 7fc56142e640 1 -- 172.21.15.40:0/640724072 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc55c0a3e20 con 0x7fc55c070370 2024-04-22T08:18:24.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.749+0000 7fc56142e640 1 -- 172.21.15.40:0/640724072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc55c070370 msgr2=0x7fc55c02b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:18:24.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.749+0000 7fc56142e640 1 --2- 172.21.15.40:0/640724072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc55c070370 0x7fc55c02b200 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc5580097a0 tx=0x7fc5580046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:24.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.749+0000 7fc560c2d640 1 --2- 172.21.15.40:0/640724072 >> v2:172.21.15.189:6800/30168 conn(0x7fc5400377d0 0x7fc540039c90 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fc55c1351e0 tx=0x7fc54c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:24.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.749+0000 7fc551ffb640 1 -- 172.21.15.40:0/640724072 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55803c440 con 0x7fc55c070370 2024-04-22T08:18:24.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.749+0000 7fc551ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:24.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.749+0000 7fc551ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:18:24.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.749+0000 7fc551ffb640 10 monclient: dump: 2024-04-22T08:18:24.763 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:24.763 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:24.763 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.763 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.763 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:24.763 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:24.763 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:24.763 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:24.763 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:24.763 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:24.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.750+0000 7fc551ffb640 1 -- 172.21.15.40:0/640724072 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7fc558040070 con 0x7fc55c070370 2024-04-22T08:18:24.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.750+0000 7fc551ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:18:24.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.750+0000 7fc551ffb640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:18:24.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.750+0000 7fc551ffb640 1 -- 172.21.15.40:0/640724072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc55c070370 msgr2=0x7fc55c02b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:24.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.750+0000 7fc551ffb640 1 --2- 172.21.15.40:0/640724072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc55c070370 0x7fc55c02b200 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc5580097a0 tx=0x7fc5580046c0 comp rx=0 tx=0).stop 2024-04-22T08:18:24.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.750+0000 7fc551ffb640 1 -- 172.21.15.40:0/640724072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc55c070370 msgr2=0x7fc55c02b200 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:18:24.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.750+0000 7fc551ffb640 1 --2- 172.21.15.40:0/640724072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc55c070370 0x7fc55c02b200 secure :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc5580097a0 tx=0x7fc5580046c0 comp rx=0 tx=0).stop 2024-04-22T08:18:24.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.756+0000 7fc563692640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:18:24.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.756+0000 7fc563692640 1 --2- 172.21.15.40:0/640724072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc55c070750 0x7fc55c02b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:24.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.756+0000 7fc563692640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:24.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.756+0000 7fc563692640 1 -- 172.21.15.40:0/640724072 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc55c11d260 con 0x7fc55c070750 2024-04-22T08:18:24.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.756+0000 7fc56142e640 1 --2- 172.21.15.40:0/640724072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc55c070750 0x7fc55c02b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:24.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.756+0000 7fc56142e640 10 monclient: get_auth_request con 0x7fc55c070750 auth_method 0 2024-04-22T08:18:24.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.756+0000 7fc56142e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:24.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.756+0000 7fc56142e640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:24.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.756+0000 7fc56142e640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:24.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.757+0000 7fc56142e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:24.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.757+0000 7fc56142e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:24.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.757+0000 7fc56142e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:24.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.757+0000 7fc56142e640 10 monclient(hunting): handle_auth_done global_id 64279 payload 824 2024-04-22T08:18:24.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.757+0000 7fc56142e640 1 -- 172.21.15.40:0/640724072 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc55c11be40 con 0x7fc55c070750 2024-04-22T08:18:24.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.759+0000 7fc56142e640 1 -- 172.21.15.40:0/640724072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc55c070750 msgr2=0x7fc55c02b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:18:24.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.759+0000 7fc56142e640 1 --2- 172.21.15.40:0/640724072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc55c070750 0x7fc55c02b740 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc55c070cb0 tx=0x7fc558008030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:24.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.760+0000 7fc551ffb640 1 -- 172.21.15.40:0/640724072 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55804fbf0 con 0x7fc55c070750 2024-04-22T08:18:24.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.760+0000 7fc551ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:24.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.760+0000 7fc551ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:18:24.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.760+0000 7fc551ffb640 10 monclient: dump: 2024-04-22T08:18:24.765 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:24.765 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:24.766 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.766 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.766 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:24.766 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:24.766 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:24.766 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:24.766 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:24.766 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:24.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.760+0000 7fc551ffb640 1 -- 172.21.15.40:0/640724072 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4801 (secure 0 0 0) 0x7fc55804fd50 con 0x7fc55c070750 2024-04-22T08:18:24.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.760+0000 7fc551ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:18:24.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.760+0000 7fc551ffb640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:18:24.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.760+0000 7fc551ffb640 1 -- 172.21.15.40:0/640724072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc55c070750 msgr2=0x7fc55c02b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:24.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.760+0000 7fc551ffb640 1 --2- 172.21.15.40:0/640724072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc55c070750 0x7fc55c02b740 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc55c070cb0 tx=0x7fc558008030 comp rx=0 tx=0).stop 2024-04-22T08:18:24.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.760+0000 7fc551ffb640 1 -- 172.21.15.40:0/640724072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc55c070750 msgr2=0x7fc55c02b740 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:18:24.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.760+0000 7fc551ffb640 1 --2- 172.21.15.40:0/640724072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc55c070750 0x7fc55c02b740 secure :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc55c070cb0 tx=0x7fc558008030 comp rx=0 tx=0).stop 2024-04-22T08:18:24.767 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:18:24.767 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:18:24.767 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:18:24.767 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:18:24.767 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 56, 2024-04-22T08:18:24.767 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:18:24.767 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:18:24.767 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:18:24.767 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:18:24.767 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:18:24.768 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 1, 2024-04-22T08:18:24.768 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:18:24.768 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:18:24.768 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:18:24.768 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:18:24.768 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:18:24.768 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:18:24.768 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:18:24.768 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:18:24.768 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:18:24.768 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:18:24.768 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:18:24.768 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:18:24.768 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:18:24.769 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:18:24.769 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:18:24.769 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:18:24.769 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:18:24.769 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:18:24.769 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:18:24.769 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:18:24.769 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:18:24.769 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:18:24.769 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:18:24.769 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:18:24.769 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:18:24.769 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:18:24.769 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:24.770 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:24.770 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:18:24.770 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:18:24.770 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:18:24.770 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:18:24.770 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:18:24.770 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:18:24.770 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:18:24.770 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:18:24.770 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:18:24.770 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:18:24.770 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:18:24.770 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:18:24.770 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:18:24.771 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:18:24.771 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:18:24.771 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:18:24.771 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:18:24.771 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:18:24.771 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:18:24.771 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:18:24.771 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:18:24.771 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:18:24.771 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:18:24.771 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:18:24.771 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:18:24.771 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:18:24.771 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:18:24.772 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:18:24.772 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:18:24.772 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:24.772 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:18:24.772 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:24.772 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:18:24.772 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:18:24.772 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:18:24.772 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:18:24.772 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:24.772 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:18:24.772 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:18:24.772 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:24.772 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:24.772 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:24.773 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:18:24.773 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:18:24.773 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:24.773 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:24.773 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:24.773 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:24.773 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:18:24.773 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:18:24.773 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:18:24.773 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:18:24.773 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:18:24.773 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:24.773 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:24.773 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:18:24.773 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:18:24.774 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:18:24.774 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:18:24.774 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:24.774 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:18:24.774 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:18:24.774 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:24.774 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:24.774 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:24.774 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:18:24.774 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:18:24.774 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:24.774 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:24.775 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:24.775 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:24.775 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:18:24.775 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:18:24.775 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:18:24.775 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:18:24.775 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:18:24.775 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:24.775 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:24.775 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:18:24.775 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:18:24.775 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:18:24.776 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:18:24.776 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:24.776 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:18:24.776 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:18:24.776 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:24.776 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:24.776 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:24.776 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:18:24.776 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:18:24.776 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:24.776 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:24.776 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:24.776 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:24.776 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:18:24.776 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:18:24.777 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:18:24.777 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:18:24.777 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:18:24.777 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:24.777 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:24.777 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:24.777 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:18:24.777 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:18:24.777 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:24.777 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:18:24.777 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:18:24.777 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:18:24.777 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:24.777 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:18:24.777 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:18:24.778 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:24.778 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:18:24.778 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:18:24.778 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 3 2024-04-22T08:18:24.778 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:24.778 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:18:24.778 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:18:24.778 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:24.778 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:18:24.778 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:18:24.778 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 4 2024-04-22T08:18:24.778 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:24.779 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:18:24.779 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": [ 2024-04-22T08:18:24.779 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:24.779 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:18:24.779 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:18:24.779 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:18:24.779 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:24.779 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:24.779 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:24.779 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:18:24.779 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:18:24.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.760+0000 7fc563692640 1 -- 172.21.15.40:0/640724072 >> v2:172.21.15.189:6800/30168 conn(0x7fc5400377d0 msgr2=0x7fc540039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:24.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.760+0000 7fc563692640 1 --2- 172.21.15.40:0/640724072 >> v2:172.21.15.189:6800/30168 conn(0x7fc5400377d0 0x7fc540039c90 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fc55c1351e0 tx=0x7fc54c008040 comp rx=0 tx=0).stop 2024-04-22T08:18:24.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.760+0000 7fc563692640 10 monclient: shutdown 2024-04-22T08:18:24.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.760+0000 7fc563692640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:18:24.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.760+0000 7fc563692640 1 -- 172.21.15.40:0/640724072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc55c136040 msgr2=0x7fc55c06f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:24.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.760+0000 7fc561c2f640 1 -- 172.21.15.40:0/640724072 reap_dead start 2024-04-22T08:18:24.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.760+0000 7fc563692640 1 --2- 172.21.15.40:0/640724072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc55c136040 0x7fc55c06f670 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc55400c780 tx=0x7fc55400cc40 comp rx=0 tx=0).stop 2024-04-22T08:18:24.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.760+0000 7fc563692640 1 -- 172.21.15.40:0/640724072 shutdown_connections 2024-04-22T08:18:24.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.760+0000 7fc563692640 1 --2- 172.21.15.40:0/640724072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc55c136040 0x7fc55c06f670 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:24.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.760+0000 7fc563692640 1 -- 172.21.15.40:0/640724072 >> 172.21.15.40:0/640724072 conn(0x7fc55c0a1780 msgr2=0x7fc55c1352a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:18:24.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.760+0000 7fc563692640 1 -- 172.21.15.40:0/640724072 shutdown_connections 2024-04-22T08:18:24.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.761+0000 7fc563692640 1 -- 172.21.15.40:0/640724072 wait complete. 2024-04-22T08:18:24.782 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:18:24.782 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-22T08:18:24.782 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-22T08:18:24.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.886+0000 7f51e8ae0640 10 monclient: get_monmap_and_config 2024-04-22T08:18:24.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.886+0000 7f51e8ae0640 10 monclient: build_initial_monmap 2024-04-22T08:18:24.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.886+0000 7f51e8ae0640 10 monclient: monmap: 2024-04-22T08:18:24.886 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:18:24.886 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:24.886 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:18:24.887148+0000 2024-04-22T08:18:24.886 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:18:24.887148+0000 2024-04-22T08:18:24.886 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:18:24.886 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:24.886 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:18:24.886 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:18:24.886 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:18:24.886 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:18:24.887 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:18:24.887 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:24.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.886+0000 7f51e8ae0640 1 Processor -- start 2024-04-22T08:18:24.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.886+0000 7f51e8ae0640 1 -- start start 2024-04-22T08:18:24.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.886+0000 7f51e8ae0640 10 monclient: init 2024-04-22T08:18:24.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.887+0000 7f51e8ae0640 10 monclient: _reopen_session rank -1 2024-04-22T08:18:24.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.887+0000 7f51e8ae0640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-04-22T08:18:24.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.887+0000 7f51e8ae0640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f51e4135e30 0x7f51e4136210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:24.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.887+0000 7f51e8ae0640 10 monclient(hunting): picked mon.noname-b con 0x7f51e4135e30 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:18:24.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.887+0000 7f51e8ae0640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f51e4071c90 0x7f51e4136750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:24.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.887+0000 7f51e8ae0640 10 monclient(hunting): picked mon.noname-c con 0x7f51e4071c90 addr v2:172.21.15.189:3300/0 2024-04-22T08:18:24.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.887+0000 7f51e8ae0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f51e40a19a0 addr v1:172.21.15.40:6789/0 2024-04-22T08:18:24.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.887+0000 7f51e8ae0640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51e409e310 con 0x7f51e40a19a0 2024-04-22T08:18:24.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.887+0000 7f51e8ae0640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:24.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.887+0000 7f51e8ae0640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51e409de80 con 0x7f51e4071c90 2024-04-22T08:18:24.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.887+0000 7f51e8ae0640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:24.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.887+0000 7f51e8ae0640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51e409f130 con 0x7f51e4135e30 2024-04-22T08:18:24.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.887+0000 7f51e8ae0640 10 monclient(hunting): _renew_subs 2024-04-22T08:18:24.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.887+0000 7f51e8ae0640 10 monclient(hunting): authenticate will time out at 1178.881406s 2024-04-22T08:18:24.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.887+0000 7f51e259c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f51e4135e30 0x7f51e4136210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:24.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.887+0000 7f51e2d9d640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f51e40a19a0 0x7f51e40a1df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:36392/0 (socket says 172.21.15.40:36392) 2024-04-22T08:18:24.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.887+0000 7f51e2d9d640 1 -- 172.21.15.40:0/390146617 learned_addr learned my addr 172.21.15.40:0/390146617 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:18:24.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.887+0000 7f51e259c640 10 monclient(hunting): get_auth_request con 0x7f51e4135e30 auth_method 0 2024-04-22T08:18:24.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.887+0000 7f51e259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:24.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.887+0000 7f51e259c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:24.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.887+0000 7f51e259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:24.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.887+0000 7f51e259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:24.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.887+0000 7f51e259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:24.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.887+0000 7f51e259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:24.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.887+0000 7f51e1d9b640 1 --2- 172.21.15.40:0/390146617 >> v2:172.21.15.189:3300/0 conn(0x7f51e4071c90 0x7f51e4136750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:24.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.887+0000 7f51e1d9b640 10 monclient(hunting): get_auth_request con 0x7f51e4071c90 auth_method 0 2024-04-22T08:18:24.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.887+0000 7f51e1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:24.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.887+0000 7f51e1d9b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:24.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.887+0000 7f51e1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:24.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.888+0000 7f51e259c640 10 monclient(hunting): handle_auth_done global_id 54470 payload 274 2024-04-22T08:18:24.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.888+0000 7f51e259c640 1 -- 172.21.15.40:0/390146617 >> v2:172.21.15.189:3300/0 conn(0x7f51e4071c90 msgr2=0x7f51e4136750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:18:24.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.888+0000 7f51e259c640 1 --2- 172.21.15.40:0/390146617 >> v2:172.21.15.189:3300/0 conn(0x7f51e4071c90 0x7f51e4136750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:24.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.888+0000 7f51e259c640 1 -- 172.21.15.40:0/390146617 >> v1:172.21.15.40:6789/0 conn(0x7f51e40a19a0 legacy=0x7f51e40a1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:18:24.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.888+0000 7f51e259c640 10 monclient: _finish_hunting 0 2024-04-22T08:18:24.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.888+0000 7f51e259c640 1 monclient: found mon.noname-b 2024-04-22T08:18:24.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.888+0000 7f51e259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:18:24.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.888+0000 7f51e259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:18:24.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.888+0000 7f51e259c640 1 -- 172.21.15.40:0/390146617 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51e409f410 con 0x7f51e4135e30 2024-04-22T08:18:24.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.890+0000 7f51e259c640 1 -- 172.21.15.40:0/390146617 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f51e4135e30 msgr2=0x7f51e4136210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:18:24.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.890+0000 7f51e259c640 1 --2- 172.21.15.40:0/390146617 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f51e4135e30 0x7f51e4136210 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f51d40097a0 tx=0x7f51d402b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b25f103c34cc297f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:24.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.891+0000 7f51e159a640 1 -- 172.21.15.40:0/390146617 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d4030920 con 0x7f51e4135e30 2024-04-22T08:18:24.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.891+0000 7f51e159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:24.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.891+0000 7f51e159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:18:24.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.891+0000 7f51e159a640 10 monclient: dump: 2024-04-22T08:18:24.896 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:24.896 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:24.896 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.896 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.896 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:24.896 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:24.896 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:24.896 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:24.896 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:24.896 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:24.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.891+0000 7f51e159a640 10 monclient: _finish_auth 0 2024-04-22T08:18:24.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.891+0000 7f51e159a640 10 monclient: _check_auth_tickets 2024-04-22T08:18:24.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.891+0000 7f51e159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:18:24.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.891+0000 7f51e159a640 1 -- 172.21.15.40:0/390146617 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51d402e050 con 0x7f51e4135e30 2024-04-22T08:18:24.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.891+0000 7f51e8ae0640 5 monclient: authenticate success, global_id 54470 2024-04-22T08:18:24.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.891+0000 7f51e8ae0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:18:24.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.891+0000 7f51e159a640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:18:24.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.891+0000 7f51e8ae0640 10 monclient: get_monmap_and_config success 2024-04-22T08:18:24.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.891+0000 7f51e159a640 1 -- 172.21.15.40:0/390146617 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d4030ca0 con 0x7f51e4135e30 2024-04-22T08:18:24.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.891+0000 7f51e159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:24.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.891+0000 7f51e8ae0640 4 set_mon_vals no callback set 2024-04-22T08:18:24.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.891+0000 7f51e159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:18:24.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.891+0000 7f51e159a640 10 monclient: dump: 2024-04-22T08:18:24.898 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:24.898 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:24.898 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.898 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.898 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:24.898 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:24.898 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:24.898 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:24.899 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:24.899 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:24.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.891+0000 7f51e8ae0640 10 monclient: shutdown 2024-04-22T08:18:24.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.891+0000 7f51e8ae0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:18:24.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.891+0000 7f51e8ae0640 1 -- 172.21.15.40:0/390146617 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f51e4135e30 msgr2=0x7f51e4136210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:24.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.891+0000 7f51e8ae0640 1 --2- 172.21.15.40:0/390146617 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f51e4135e30 0x7f51e4136210 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f51d40097a0 tx=0x7f51d402b710 comp rx=0 tx=0).stop 2024-04-22T08:18:24.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.891+0000 7f51e8ae0640 1 -- 172.21.15.40:0/390146617 shutdown_connections 2024-04-22T08:18:24.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.891+0000 7f51e8ae0640 1 --2- 172.21.15.40:0/390146617 >> v2:172.21.15.189:3300/0 conn(0x7f51e4071c90 0x7f51e4136750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:24.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.891+0000 7f51e8ae0640 1 --2- 172.21.15.40:0/390146617 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f51e4135e30 0x7f51e4136210 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:24.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.891+0000 7f51e8ae0640 1 -- 172.21.15.40:0/390146617 >> 172.21.15.40:0/390146617 conn(0x7f51e40a65c0 msgr2=0x7f51e40a2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:18:24.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.891+0000 7f51e8ae0640 1 -- 172.21.15.40:0/390146617 shutdown_connections 2024-04-22T08:18:24.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.891+0000 7f51e8ae0640 1 -- 172.21.15.40:0/390146617 wait complete. 2024-04-22T08:18:24.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.892+0000 7f51e8ae0640 10 monclient: build_initial_monmap 2024-04-22T08:18:24.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.892+0000 7f51e8ae0640 10 monclient: monmap: 2024-04-22T08:18:24.904 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:18:24.904 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:18:24.905 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:18:24.905 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:18:24.905 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:18:24.905 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:24.905 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:18:24.905 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:18:24.905 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:18:24.905 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:24.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.892+0000 7f51e8ae0640 1 Processor -- start 2024-04-22T08:18:24.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.892+0000 7f51e8ae0640 1 -- start start 2024-04-22T08:18:24.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.892+0000 7f51e8ae0640 10 monclient: init 2024-04-22T08:18:24.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e8ae0640 10 monclient: _reopen_session rank -1 2024-04-22T08:18:24.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e8ae0640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-22T08:18:24.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e8ae0640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f51e4071c90 0x7f51e41ca7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:24.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e8ae0640 10 monclient(hunting): picked mon.noname-c con 0x7f51e4071c90 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:18:24.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e8ae0640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51e40a19a0 0x7f51e4134c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:24.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e8ae0640 10 monclient(hunting): picked mon.noname-b con 0x7f51e40a19a0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:18:24.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e8ae0640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51e4135e30 0x7f51e41351a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:24.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e8ae0640 10 monclient(hunting): picked mon.noname-a con 0x7f51e4135e30 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:18:24.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e8ae0640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:24.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e8ae0640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51e41223d0 con 0x7f51e4135e30 2024-04-22T08:18:24.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e8ae0640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:24.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e8ae0640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51e4122620 con 0x7f51e40a19a0 2024-04-22T08:18:24.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e8ae0640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:24.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e8ae0640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51e409c9c0 con 0x7f51e4071c90 2024-04-22T08:18:24.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e8ae0640 10 monclient(hunting): _renew_subs 2024-04-22T08:18:24.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e8ae0640 10 monclient(hunting): authenticate will time out at 1178.886939s 2024-04-22T08:18:24.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e259c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f51e4071c90 0x7f51e41ca7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:24.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e259c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f51e4071c90 0x7f51e41ca7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:51450/0 (socket says 172.21.15.40:51450) 2024-04-22T08:18:24.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e259c640 1 -- 172.21.15.40:0/2588835531 learned_addr learned my addr 172.21.15.40:0/2588835531 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:18:24.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e259c640 10 monclient(hunting): get_auth_request con 0x7f51e4071c90 auth_method 0 2024-04-22T08:18:24.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:24.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e259c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:24.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:24.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e2d9d640 1 --2- 172.21.15.40:0/2588835531 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51e4135e30 0x7f51e41351a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:24.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e2d9d640 10 monclient(hunting): get_auth_request con 0x7f51e4135e30 auth_method 0 2024-04-22T08:18:24.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:24.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e2d9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:24.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:24.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:24.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:24.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:24.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:24.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:24.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:24.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e1d9b640 1 --2- 172.21.15.40:0/2588835531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51e40a19a0 0x7f51e4134c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:24.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e1d9b640 10 monclient(hunting): get_auth_request con 0x7f51e40a19a0 auth_method 0 2024-04-22T08:18:24.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:24.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e1d9b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:24.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:24.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e2d9d640 10 monclient(hunting): handle_auth_done global_id 74121 payload 824 2024-04-22T08:18:24.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e2d9d640 1 -- 172.21.15.40:0/2588835531 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f51e4071c90 msgr2=0x7f51e41ca7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:24.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e2d9d640 1 --2- 172.21.15.40:0/2588835531 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f51e4071c90 0x7f51e41ca7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:24.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e2d9d640 1 -- 172.21.15.40:0/2588835531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51e40a19a0 msgr2=0x7f51e4134c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:24.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e2d9d640 1 --2- 172.21.15.40:0/2588835531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51e40a19a0 0x7f51e4134c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:24.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e2d9d640 10 monclient: _finish_hunting 0 2024-04-22T08:18:24.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e2d9d640 1 monclient: found mon.noname-a 2024-04-22T08:18:24.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:18:24.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e2d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:18:24.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e2d9d640 1 -- 172.21.15.40:0/2588835531 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d4009400 con 0x7f51e4135e30 2024-04-22T08:18:24.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.893+0000 7f51e259c640 1 --2- 172.21.15.40:0/2588835531 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f51e4071c90 0x7f51e41ca7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:18:24.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.895+0000 7f51e2d9d640 1 -- 172.21.15.40:0/2588835531 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51e4135e30 msgr2=0x7f51e41351a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:18:24.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.895+0000 7f51e2d9d640 1 --2- 172.21.15.40:0/2588835531 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51e4135e30 0x7f51e41351a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f51dc00cb50 tx=0x7f51dc00cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:24.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.895+0000 7f51d2ffd640 1 -- 172.21.15.40:0/2588835531 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51dc00ade0 con 0x7f51e4135e30 2024-04-22T08:18:24.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.896+0000 7f51d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:24.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.896+0000 7f51d2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:18:24.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.896+0000 7f51d2ffd640 10 monclient: dump: 2024-04-22T08:18:24.909 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:24.909 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:24.909 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.910 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.910 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:24.910 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:24.910 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:24.910 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:24.910 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:24.910 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:24.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.896+0000 7f51d2ffd640 10 monclient: _finish_auth 0 2024-04-22T08:18:24.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.896+0000 7f51d2ffd640 10 monclient: _check_auth_tickets 2024-04-22T08:18:24.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.896+0000 7f51d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:18:24.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.896+0000 7f51e8ae0640 5 monclient: authenticate success, global_id 74121 2024-04-22T08:18:24.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.896+0000 7f51d2ffd640 1 -- 172.21.15.40:0/2588835531 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51dc012400 con 0x7f51e4135e30 2024-04-22T08:18:24.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.896+0000 7f51e8ae0640 10 monclient: _renew_subs 2024-04-22T08:18:24.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.896+0000 7f51e8ae0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:18:24.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.896+0000 7f51e8ae0640 1 -- 172.21.15.40:0/2588835531 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51e40720e0 con 0x7f51e4135e30 2024-04-22T08:18:24.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.896+0000 7f51d2ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:18:24.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.896+0000 7f51d2ffd640 1 -- 172.21.15.40:0/2588835531 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51dc0127b0 con 0x7f51e4135e30 2024-04-22T08:18:24.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.896+0000 7f51d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:24.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.896+0000 7f51d2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:18:24.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.896+0000 7f51e0d99640 4 set_mon_vals no callback set 2024-04-22T08:18:24.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.896+0000 7f51d2ffd640 10 monclient: dump: 2024-04-22T08:18:24.911 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:24.911 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:24.911 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.911 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:24.911 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:24.912 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:24.912 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:24.912 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:24.912 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:24.912 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:24.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.896+0000 7f51e8ae0640 10 monclient: _renew_subs 2024-04-22T08:18:24.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.896+0000 7f51e8ae0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:18:24.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.896+0000 7f51e8ae0640 1 -- 172.21.15.40:0/2588835531 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51e4071500 con 0x7f51e4135e30 2024-04-22T08:18:24.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.897+0000 7f51d2ffd640 1 -- 172.21.15.40:0/2588835531 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f51dc012910 con 0x7f51e4135e30 2024-04-22T08:18:24.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.897+0000 7f51e8ae0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:18:24.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.897+0000 7f51e8ae0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:18:24.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.897+0000 7f51e8ae0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:18:24.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.897+0000 7f51e8ae0640 1 -- 172.21.15.40:0/2588835531 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51e4135b90 con 0x7f51e4135e30 2024-04-22T08:18:24.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.897+0000 7f51d2ffd640 1 --2- 172.21.15.40:0/2588835531 >> v2:172.21.15.189:6800/30168 conn(0x7f51c40377d0 0x7f51c4039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:24.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.897+0000 7f51d2ffd640 1 -- 172.21.15.40:0/2588835531 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f51dc04a830 con 0x7f51e4135e30 2024-04-22T08:18:24.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.897+0000 7f51e259c640 1 --2- 172.21.15.40:0/2588835531 >> v2:172.21.15.189:6800/30168 conn(0x7f51c40377d0 0x7f51c4039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:24.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.897+0000 7f51e259c640 10 monclient: get_auth_request con 0x7f51c40377d0 auth_method 0 2024-04-22T08:18:24.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.900+0000 7f51e259c640 1 --2- 172.21.15.40:0/2588835531 >> v2:172.21.15.189:6800/30168 conn(0x7f51c40377d0 0x7f51c4039c90 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f51d402b750 tx=0x7f51d402f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:24.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.902+0000 7f51d2ffd640 1 -- 172.21.15.40:0/2588835531 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f51e4135b90 con 0x7f51e4135e30 2024-04-22T08:18:24.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.902+0000 7f51d2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:18:24.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:24.902+0000 7f51d2ffd640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:18:25.110 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:18:25.110 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37344]: starting. 2024-04-22T08:18:25.110 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37344]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-37307 2024-04-22T08:18:25.110 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37344]: created object 0... 2024-04-22T08:18:25.110 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37344]: created object 25... 2024-04-22T08:18:25.110 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37344]: created object 49... 2024-04-22T08:18:25.110 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37344]: finishing. 2024-04-22T08:18:25.110 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37344]: shutting down. 2024-04-22T08:18:25.149 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:18:25.150 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37346]: starting. 2024-04-22T08:18:25.150 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37346]: listing objects. 2024-04-22T08:18:25.150 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37346]: listed object 0... 2024-04-22T08:18:25.150 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37346]: listed object 25... 2024-04-22T08:18:25.150 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37346]: saw 50 objects 2024-04-22T08:18:25.150 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37346]: shutting down. 2024-04-22T08:18:25.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:25.161+0000 7f51e8ae0640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-22T08:18:25.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:25.161+0000 7f51e8ae0640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-22T08:18:25.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:25.161+0000 7f51e8ae0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:18:25.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:25.161+0000 7f51e8ae0640 1 -- 172.21.15.40:0/2588835531 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f51e4135b90 con 0x7f51e4135e30 2024-04-22T08:18:25.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:25.162+0000 7f51d2ffd640 1 -- 172.21.15.40:0/2588835531 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f51e4135b90 con 0x7f51e4135e30 2024-04-22T08:18:25.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:25.162+0000 7f51d2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-22T08:18:25.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:25.162+0000 7f51d2ffd640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:18:25.162 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.163+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:18:25.162 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.163+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=10,kv=9,logm=65} crc {auth=3305772193,config=1618278320,health=1401651998,kv=1350343662,logm=3926064696}) 2024-04-22T08:18:25.163 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.163+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=14,config=2,health=12,kv=9,logm=63} crc {auth=481556061,config=1618278320,health=1855666294,kv=1350343662,logm=1697863357}) 2024-04-22T08:18:25.163 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.163+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:18:25.163 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.163+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=10,kv=9,logm=65} crc {auth=3305772193,config=1618278320,health=1401651998,kv=1350343662,logm=3926064696}) 2024-04-22T08:18:25.163 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.163+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=14,config=2,health=11,kv=9,logm=64} crc {auth=481556061,config=1618278320,health=2610937323,kv=1350343662,logm=3245278524}) 2024-04-22T08:18:25.164 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.164+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:18:25.164 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.164+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3653096506}) 2024-04-22T08:18:25.164 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.164+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1612291487}) 2024-04-22T08:18:25.165 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.164+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:18:25.165 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.164+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3653096506}) 2024-04-22T08:18:25.165 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.164+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3930436397}) 2024-04-22T08:18:25.165 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.165+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:18:25.165 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.165+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=104214785}) 2024-04-22T08:18:25.165 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.165+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2871110029}) 2024-04-22T08:18:25.165 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.165+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:18:25.165 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.165+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=104214785}) 2024-04-22T08:18:25.165 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.165+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1983318146}) 2024-04-22T08:18:25.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:25.165+0000 7f51e8ae0640 1 -- 172.21.15.40:0/2588835531 >> v2:172.21.15.189:6800/30168 conn(0x7f51c40377d0 msgr2=0x7f51c4039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:25.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:25.165+0000 7f51e8ae0640 1 --2- 172.21.15.40:0/2588835531 >> v2:172.21.15.189:6800/30168 conn(0x7f51c40377d0 0x7f51c4039c90 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f51d402b750 tx=0x7f51d402f000 comp rx=0 tx=0).stop 2024-04-22T08:18:25.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:25.165+0000 7f51e8ae0640 10 monclient: shutdown 2024-04-22T08:18:25.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:25.165+0000 7f51e8ae0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:18:25.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:25.165+0000 7f51e8ae0640 1 -- 172.21.15.40:0/2588835531 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51e4135e30 msgr2=0x7f51e41351a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:25.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:25.165+0000 7f51e8ae0640 1 --2- 172.21.15.40:0/2588835531 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51e4135e30 0x7f51e41351a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f51dc00cb50 tx=0x7f51dc00cf50 comp rx=0 tx=0).stop 2024-04-22T08:18:25.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:25.165+0000 7f51e8ae0640 1 -- 172.21.15.40:0/2588835531 shutdown_connections 2024-04-22T08:18:25.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:25.165+0000 7f51e8ae0640 1 --2- 172.21.15.40:0/2588835531 >> v2:172.21.15.189:6800/30168 conn(0x7f51c40377d0 0x7f51c4039c90 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:25.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:25.165+0000 7f51e8ae0640 1 --2- 172.21.15.40:0/2588835531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51e40a19a0 0x7f51e4134c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:25.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:25.165+0000 7f51e8ae0640 1 --2- 172.21.15.40:0/2588835531 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51e4135e30 0x7f51e41351a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:25.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:25.165+0000 7f51e8ae0640 1 --2- 172.21.15.40:0/2588835531 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f51e4071c90 0x7f51e41ca7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:25.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:25.165+0000 7f51e8ae0640 1 -- 172.21.15.40:0/2588835531 >> 172.21.15.40:0/2588835531 conn(0x7f51e40a65c0 msgr2=0x7f51e409ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:18:25.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:25.166+0000 7f51e8ae0640 1 -- 172.21.15.40:0/2588835531 shutdown_connections 2024-04-22T08:18:25.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:25.166+0000 7f51e8ae0640 1 -- 172.21.15.40:0/2588835531 wait complete. 2024-04-22T08:18:25.172 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.172+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:18:25.172 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.172+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=382016397}) 2024-04-22T08:18:25.172 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.172+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2646890977}) 2024-04-22T08:18:25.172 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.172+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:18:25.172 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.172+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=382016397}) 2024-04-22T08:18:25.172 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.172+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2126085778}) 2024-04-22T08:18:25.173 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.173+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:18:25.173 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.173+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4189856440}) 2024-04-22T08:18:25.173 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.173+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2594735001}) 2024-04-22T08:18:25.173 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.173+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:18:25.173 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.173+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4189856440}) 2024-04-22T08:18:25.174 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.173+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=707266527}) 2024-04-22T08:18:25.176 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-22T08:18:25.607 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.607+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:18:25.607 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.607+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3334559119}) 2024-04-22T08:18:25.607 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.607+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1823727381}) 2024-04-22T08:18:25.607 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.607+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:18:25.607 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.607+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3334559119}) 2024-04-22T08:18:25.607 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.607+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1128538761}) 2024-04-22T08:18:25.610 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.610+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:18:25.610 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.610+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2137341452}) 2024-04-22T08:18:25.610 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.610+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3642149131}) 2024-04-22T08:18:25.610 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.610+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:18:25.610 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.610+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2137341452}) 2024-04-22T08:18:25.611 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.610+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=826377522}) 2024-04-22T08:18:25.862 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.863+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:18:25.862 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.863+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3404476378}) 2024-04-22T08:18:25.862 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.863+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3840326441}) 2024-04-22T08:18:25.862 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.863+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:18:25.862 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.863+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3404476378}) 2024-04-22T08:18:25.862 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.863+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3674111399}) 2024-04-22T08:18:25.863 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.864+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:18:25.863 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.864+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2640671398}) 2024-04-22T08:18:25.863 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.864+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1400263713}) 2024-04-22T08:18:25.863 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.864+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:18:25.863 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.864+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2640671398}) 2024-04-22T08:18:25.864 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.864+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2818151983}) 2024-04-22T08:18:25.869 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.870+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:18:25.869 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.870+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3841080788}) 2024-04-22T08:18:25.869 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.870+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3284233846}) 2024-04-22T08:18:25.869 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.870+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:18:25.869 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.870+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3841080788}) 2024-04-22T08:18:25.869 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.870+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=767311792}) 2024-04-22T08:18:25.876 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:18:25.876 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[37345]: starting. 2024-04-22T08:18:25.876 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[37345]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-37307 2024-04-22T08:18:25.876 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[37345]: shutting down. 2024-04-22T08:18:25.877 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.878+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:18:25.877 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.878+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2949634322}) 2024-04-22T08:18:25.878 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.878+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3921401354}) 2024-04-22T08:18:25.878 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.878+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:18:25.878 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.878+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2949634322}) 2024-04-22T08:18:25.878 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.878+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1750876765}) 2024-04-22T08:18:25.878 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:18:25.878 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:18:25.878 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:18:25.880 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:18:25.938 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.938+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:18:25.938 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.938+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1913333902}) 2024-04-22T08:18:25.938 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.938+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=125422703}) 2024-04-22T08:18:25.938 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.938+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:18:25.939 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.938+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1913333902}) 2024-04-22T08:18:25.939 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.938+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=881798762}) 2024-04-22T08:18:25.971 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.971+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:18:25.971 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.971+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3535917534}) 2024-04-22T08:18:25.971 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.971+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=219139228}) 2024-04-22T08:18:25.971 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.971+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:18:25.971 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.971+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3535917534}) 2024-04-22T08:18:25.971 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.971+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1569039000}) 2024-04-22T08:18:25.972 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.972+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:18:25.972 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.972+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=45,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=7,monmap=3,osd_metadata=6,osdmap=14} crc {logm=1741136371,mdsmap=1531151339,mgr=878880162,mgr_command_descs=470045235,mgr_metadata=4222303120,mgrstat=2044650957,mon_config_key=646836014,monmap=3469086636,osd_metadata=4151711904,osdmap=4246900291}) 2024-04-22T08:18:25.972 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.972+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=44,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=7,monmap=3,osd_metadata=4,osd_pg_creating=1,osdmap=17} crc {logm=3326951559,mdsmap=1531151339,mgr=36904295,mgr_command_descs=470045235,mgr_metadata=4222303120,mgrstat=2724211570,mon_config_key=2517516856,monmap=3469086636,osd_metadata=2137561785,osd_pg_creating=1582871894,osdmap=128924284}) 2024-04-22T08:18:25.972 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.972+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:18:25.972 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.972+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=45,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=7,monmap=3,osd_metadata=6,osdmap=14} crc {logm=1741136371,mdsmap=1531151339,mgr=878880162,mgr_command_descs=470045235,mgr_metadata=4222303120,mgrstat=2044650957,mon_config_key=646836014,monmap=3469086636,osd_metadata=4151711904,osdmap=4246900291}) 2024-04-22T08:18:25.972 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.972+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=47,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=8,monmap=1,osd_metadata=5,osd_pg_creating=1,osdmap=13} crc {logm=3832860877,mdsmap=2147943554,mgr=3080842177,mgr_command_descs=470045235,mgr_metadata=4222303120,mgrstat=1652918959,mon_config_key=4255067121,monmap=2998930919,osd_metadata=999924489,osd_pg_creating=1582871894,osdmap=2086075734}) 2024-04-22T08:18:25.974 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.973+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:18:25.974 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.973+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4070512006}) 2024-04-22T08:18:25.974 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.973+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=592356044}) 2024-04-22T08:18:25.974 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.973+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:18:25.974 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.973+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4070512006}) 2024-04-22T08:18:25.974 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.973+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2422682539}) 2024-04-22T08:18:25.974 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.974+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:18:25.974 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.974+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3566144355}) 2024-04-22T08:18:25.975 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.974+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3643223266}) 2024-04-22T08:18:25.975 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.974+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:18:25.975 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.974+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3566144355}) 2024-04-22T08:18:25.975 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.974+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=176776044}) 2024-04-22T08:18:25.977 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.978+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:18:25.978 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.978+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=72} crc {osdmap=1513908400}) 2024-04-22T08:18:25.978 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.978+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=70} crc {osdmap=1335047767}) 2024-04-22T08:18:25.978 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.978+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:18:25.978 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.978+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=72} crc {osdmap=1513908400}) 2024-04-22T08:18:25.978 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:18:25.978+0000 7f9a64012640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=71} crc {osdmap=3933279741}) 2024-04-22T08:18:26.177 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-22T08:18:26.177 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:18:26.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.288+0000 7f47a5522640 10 monclient: get_monmap_and_config 2024-04-22T08:18:26.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.288+0000 7f47a5522640 10 monclient: build_initial_monmap 2024-04-22T08:18:26.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.288+0000 7f47a5522640 10 monclient: monmap: 2024-04-22T08:18:26.288 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:18:26.288 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:26.288 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:18:26.289638+0000 2024-04-22T08:18:26.288 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:18:26.289638+0000 2024-04-22T08:18:26.288 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:18:26.289 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:26.289 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:18:26.289 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:18:26.289 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:18:26.289 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:18:26.289 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:18:26.289 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:26.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.289+0000 7f47a5522640 1 Processor -- start 2024-04-22T08:18:26.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.289+0000 7f47a5522640 1 -- start start 2024-04-22T08:18:26.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.289+0000 7f47a5522640 10 monclient: init 2024-04-22T08:18:26.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.289+0000 7f47a5522640 10 monclient: _reopen_session rank -1 2024-04-22T08:18:26.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.289+0000 7f47a5522640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-04-22T08:18:26.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.290+0000 7f47a5522640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f47a00a2560 addr v1:172.21.15.40:6789/0 2024-04-22T08:18:26.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.290+0000 7f47a5522640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f47a00a1110 0x7f47a0138d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:26.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.290+0000 7f47a5522640 10 monclient(hunting): picked mon.noname-c con 0x7f47a00a1110 addr v2:172.21.15.189:3300/0 2024-04-22T08:18:26.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.290+0000 7f47a5522640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f47a00a15c0 0x7f47a013b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:26.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.290+0000 7f47a5522640 10 monclient(hunting): picked mon.noname-b con 0x7f47a00a15c0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:18:26.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.290+0000 7f47a5522640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47a00a2940 con 0x7f47a00a2560 2024-04-22T08:18:26.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.290+0000 7f47a5522640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:26.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.290+0000 7f47a5522640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47a00719f0 con 0x7f47a00a1110 2024-04-22T08:18:26.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.290+0000 7f47a5522640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:26.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.290+0000 7f47a5522640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47a009ce90 con 0x7f47a00a15c0 2024-04-22T08:18:26.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.290+0000 7f47a5522640 10 monclient(hunting): _renew_subs 2024-04-22T08:18:26.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.290+0000 7f47a5522640 10 monclient(hunting): authenticate will time out at 1180.283877s 2024-04-22T08:18:26.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.290+0000 7f479effd640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f47a00a2560 0x7f47a00a0a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:36406/0 (socket says 172.21.15.40:36406) 2024-04-22T08:18:26.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.290+0000 7f479effd640 1 -- 172.21.15.40:0/4207396214 learned_addr learned my addr 172.21.15.40:0/4207396214 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:18:26.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.290+0000 7f479f7fe640 1 --2- 172.21.15.40:0/4207396214 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f47a00a15c0 0x7f47a013b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:26.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.290+0000 7f479f7fe640 10 monclient(hunting): get_auth_request con 0x7f47a00a15c0 auth_method 0 2024-04-22T08:18:26.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.290+0000 7f479f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:26.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.290+0000 7f479f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:26.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.290+0000 7f479f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:26.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.290+0000 7f479f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:26.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.290+0000 7f479f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:26.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.290+0000 7f479f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:26.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.290+0000 7f479e7fc640 1 --2- 172.21.15.40:0/4207396214 >> v2:172.21.15.189:3300/0 conn(0x7f47a00a1110 0x7f47a0138d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:26.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.290+0000 7f479e7fc640 10 monclient(hunting): get_auth_request con 0x7f47a00a1110 auth_method 0 2024-04-22T08:18:26.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.290+0000 7f479e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:26.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.290+0000 7f479e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:26.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.290+0000 7f479e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:26.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.290+0000 7f479f7fe640 10 monclient(hunting): handle_auth_done global_id 54485 payload 274 2024-04-22T08:18:26.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.290+0000 7f479f7fe640 1 -- 172.21.15.40:0/4207396214 >> v2:172.21.15.189:3300/0 conn(0x7f47a00a1110 msgr2=0x7f47a0138d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:18:26.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.290+0000 7f479f7fe640 1 --2- 172.21.15.40:0/4207396214 >> v2:172.21.15.189:3300/0 conn(0x7f47a00a1110 0x7f47a0138d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:26.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.290+0000 7f479f7fe640 1 -- 172.21.15.40:0/4207396214 >> v1:172.21.15.40:6789/0 conn(0x7f47a00a2560 legacy=0x7f47a00a0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:18:26.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.290+0000 7f479f7fe640 10 monclient: _finish_hunting 0 2024-04-22T08:18:26.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.290+0000 7f479f7fe640 1 monclient: found mon.noname-b 2024-04-22T08:18:26.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.290+0000 7f479f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:18:26.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.290+0000 7f479f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:18:26.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.290+0000 7f479f7fe640 1 -- 172.21.15.40:0/4207396214 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a009d170 con 0x7f47a00a15c0 2024-04-22T08:18:26.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.293+0000 7f479f7fe640 1 -- 172.21.15.40:0/4207396214 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f47a00a15c0 msgr2=0x7f47a013b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:18:26.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.293+0000 7f479f7fe640 1 --2- 172.21.15.40:0/4207396214 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f47a00a15c0 0x7f47a013b2b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4798009d90 tx=0x7f479802bcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e248f3593da09cf9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:26.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.293+0000 7f479dffb640 1 -- 172.21.15.40:0/4207396214 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f479802ccb0 con 0x7f47a00a15c0 2024-04-22T08:18:26.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.293+0000 7f479dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:26.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.293+0000 7f479dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:18:26.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.293+0000 7f479dffb640 10 monclient: dump: 2024-04-22T08:18:26.301 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:26.301 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:26.301 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:26.301 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:26.301 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:26.301 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:26.301 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:26.301 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:26.301 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:26.301 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:26.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.293+0000 7f479dffb640 10 monclient: _finish_auth 0 2024-04-22T08:18:26.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.293+0000 7f479dffb640 10 monclient: _check_auth_tickets 2024-04-22T08:18:26.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.293+0000 7f479dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:18:26.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.293+0000 7f479dffb640 1 -- 172.21.15.40:0/4207396214 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f479802ce10 con 0x7f47a00a15c0 2024-04-22T08:18:26.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.293+0000 7f479dffb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:18:26.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.293+0000 7f479dffb640 1 -- 172.21.15.40:0/4207396214 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4798034660 con 0x7f47a00a15c0 2024-04-22T08:18:26.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.293+0000 7f47a5522640 5 monclient: authenticate success, global_id 54485 2024-04-22T08:18:26.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.293+0000 7f47a5522640 10 monclient: get_monmap_and_config success 2024-04-22T08:18:26.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.293+0000 7f47a5522640 4 set_mon_vals no callback set 2024-04-22T08:18:26.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.293+0000 7f479dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:26.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.293+0000 7f479dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:18:26.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.293+0000 7f479dffb640 10 monclient: dump: 2024-04-22T08:18:26.302 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:26.302 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:26.302 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:26.302 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:26.302 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:26.303 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:26.303 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:26.303 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:26.303 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:26.303 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:26.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.294+0000 7f47a5522640 10 monclient: shutdown 2024-04-22T08:18:26.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.294+0000 7f47a5522640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:18:26.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.294+0000 7f47a5522640 1 -- 172.21.15.40:0/4207396214 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f47a00a15c0 msgr2=0x7f47a013b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:26.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.294+0000 7f47a5522640 1 --2- 172.21.15.40:0/4207396214 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f47a00a15c0 0x7f47a013b2b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4798009d90 tx=0x7f479802bcf0 comp rx=0 tx=0).stop 2024-04-22T08:18:26.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.294+0000 7f47a5522640 1 -- 172.21.15.40:0/4207396214 shutdown_connections 2024-04-22T08:18:26.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.294+0000 7f47a5522640 1 --2- 172.21.15.40:0/4207396214 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f47a00a15c0 0x7f47a013b2b0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:26.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.294+0000 7f47a5522640 1 --2- 172.21.15.40:0/4207396214 >> v2:172.21.15.189:3300/0 conn(0x7f47a00a1110 0x7f47a0138d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:26.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.294+0000 7f47a5522640 1 -- 172.21.15.40:0/4207396214 >> 172.21.15.40:0/4207396214 conn(0x7f47a00a5960 msgr2=0x7f47a00a1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:18:26.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.294+0000 7f47a5522640 1 -- 172.21.15.40:0/4207396214 shutdown_connections 2024-04-22T08:18:26.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.294+0000 7f47a5522640 1 -- 172.21.15.40:0/4207396214 wait complete. 2024-04-22T08:18:26.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.294+0000 7f47a5522640 10 monclient: build_initial_monmap 2024-04-22T08:18:26.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.294+0000 7f47a5522640 10 monclient: monmap: 2024-04-22T08:18:26.304 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:18:26.304 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:18:26.304 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:18:26.304 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:18:26.304 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:18:26.304 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:26.304 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:18:26.304 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:18:26.304 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:18:26.304 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:26.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.295+0000 7f47a5522640 1 Processor -- start 2024-04-22T08:18:26.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.295+0000 7f47a5522640 1 -- start start 2024-04-22T08:18:26.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.295+0000 7f47a5522640 10 monclient: init 2024-04-22T08:18:26.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.295+0000 7f47a5522640 10 monclient: _reopen_session rank -1 2024-04-22T08:18:26.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.295+0000 7f47a5522640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-22T08:18:26.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.295+0000 7f47a5522640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47a00a1110 0x7f47a01c8a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:26.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.295+0000 7f47a5522640 10 monclient(hunting): picked mon.noname-b con 0x7f47a00a1110 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:18:26.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.295+0000 7f47a5522640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f47a00a15c0 0x7f47a01c8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:26.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.295+0000 7f47a5522640 10 monclient(hunting): picked mon.noname-c con 0x7f47a00a15c0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:18:26.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.295+0000 7f47a5522640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47a00a2560 0x7f47a012f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:26.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.295+0000 7f47a5522640 10 monclient(hunting): picked mon.noname-a con 0x7f47a00a2560 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:18:26.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.295+0000 7f47a5522640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:26.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.295+0000 7f47a5522640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a013acd0 con 0x7f47a00a2560 2024-04-22T08:18:26.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.295+0000 7f47a5522640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:26.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.295+0000 7f47a5522640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a006e7e0 con 0x7f47a00a1110 2024-04-22T08:18:26.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.295+0000 7f47a5522640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:26.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.295+0000 7f47a5522640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47a006eac0 con 0x7f47a00a15c0 2024-04-22T08:18:26.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.295+0000 7f47a5522640 10 monclient(hunting): _renew_subs 2024-04-22T08:18:26.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.295+0000 7f47a5522640 10 monclient(hunting): authenticate will time out at 1180.289622s 2024-04-22T08:18:26.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.295+0000 7f479e7fc640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f47a00a15c0 0x7f47a01c8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:26.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.295+0000 7f479f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47a00a2560 0x7f47a012f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:26.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.295+0000 7f479e7fc640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f47a00a15c0 0x7f47a01c8fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:51454/0 (socket says 172.21.15.40:51454) 2024-04-22T08:18:26.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.295+0000 7f479e7fc640 1 -- 172.21.15.40:0/3992626697 learned_addr learned my addr 172.21.15.40:0/3992626697 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:18:26.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.295+0000 7f479e7fc640 10 monclient(hunting): get_auth_request con 0x7f47a00a15c0 auth_method 0 2024-04-22T08:18:26.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.295+0000 7f479e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:26.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.295+0000 7f479e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:26.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.295+0000 7f479e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:26.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.296+0000 7f479f7fe640 10 monclient(hunting): get_auth_request con 0x7f47a00a2560 auth_method 0 2024-04-22T08:18:26.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.296+0000 7f479f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:26.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.296+0000 7f479f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:26.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.296+0000 7f479f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:26.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.296+0000 7f479e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:26.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.296+0000 7f479e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:26.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.296+0000 7f479e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:26.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.296+0000 7f479f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:26.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.296+0000 7f479f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:26.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.296+0000 7f479f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:26.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.296+0000 7f479effd640 1 --2- 172.21.15.40:0/3992626697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47a00a1110 0x7f47a01c8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:26.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.296+0000 7f479effd640 10 monclient(hunting): get_auth_request con 0x7f47a00a1110 auth_method 0 2024-04-22T08:18:26.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.296+0000 7f479effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:26.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.296+0000 7f479effd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:26.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.296+0000 7f479effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:26.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.296+0000 7f479e7fc640 10 monclient(hunting): handle_auth_done global_id 54488 payload 824 2024-04-22T08:18:26.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.296+0000 7f479e7fc640 1 -- 172.21.15.40:0/3992626697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47a00a1110 msgr2=0x7f47a01c8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:26.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.296+0000 7f479e7fc640 1 --2- 172.21.15.40:0/3992626697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47a00a1110 0x7f47a01c8a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:26.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.296+0000 7f479e7fc640 1 -- 172.21.15.40:0/3992626697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47a00a2560 msgr2=0x7f47a012f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:26.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.296+0000 7f479e7fc640 1 --2- 172.21.15.40:0/3992626697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47a00a2560 0x7f47a012f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:26.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.296+0000 7f479e7fc640 10 monclient: _finish_hunting 0 2024-04-22T08:18:26.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.296+0000 7f479e7fc640 1 monclient: found mon.noname-c 2024-04-22T08:18:26.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.296+0000 7f479e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:18:26.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.296+0000 7f479e7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:18:26.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.296+0000 7f479e7fc640 1 -- 172.21.15.40:0/3992626697 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4798009a70 con 0x7f47a00a15c0 2024-04-22T08:18:26.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.298+0000 7f479e7fc640 1 -- 172.21.15.40:0/3992626697 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f47a00a15c0 msgr2=0x7f47a01c8fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:18:26.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.298+0000 7f479e7fc640 1 --2- 172.21.15.40:0/3992626697 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f47a00a15c0 0x7f47a01c8fd0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f479400d5f0 tx=0x7f479400dab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:26.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.298+0000 7f47877fe640 1 -- 172.21.15.40:0/3992626697 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4794012800 con 0x7f47a00a15c0 2024-04-22T08:18:26.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.298+0000 7f47877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:26.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.298+0000 7f47877fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:18:26.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.298+0000 7f47877fe640 10 monclient: dump: 2024-04-22T08:18:26.310 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:26.310 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:26.310 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:26.311 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:26.311 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:26.311 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:26.311 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:26.311 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:26.311 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:26.311 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:26.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.298+0000 7f47877fe640 10 monclient: _finish_auth 0 2024-04-22T08:18:26.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.298+0000 7f47877fe640 10 monclient: _check_auth_tickets 2024-04-22T08:18:26.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.298+0000 7f47877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:18:26.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.298+0000 7f47877fe640 1 -- 172.21.15.40:0/3992626697 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4794012960 con 0x7f47a00a15c0 2024-04-22T08:18:26.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.298+0000 7f47a5522640 5 monclient: authenticate success, global_id 54488 2024-04-22T08:18:26.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.298+0000 7f47a5522640 10 monclient: _renew_subs 2024-04-22T08:18:26.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.298+0000 7f47a5522640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:18:26.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.298+0000 7f47a5522640 1 -- 172.21.15.40:0/3992626697 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47a006e500 con 0x7f47a00a15c0 2024-04-22T08:18:26.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.298+0000 7f47877fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:18:26.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.298+0000 7f47877fe640 1 -- 172.21.15.40:0/3992626697 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4794012d10 con 0x7f47a00a15c0 2024-04-22T08:18:26.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.298+0000 7f47877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:26.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.298+0000 7f47877fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:18:26.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.298+0000 7f47877fe640 10 monclient: dump: 2024-04-22T08:18:26.312 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:26.312 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:26.312 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:26.312 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:26.312 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:26.312 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:26.312 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:26.312 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:26.312 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:26.313 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:26.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.298+0000 7f47a5522640 10 monclient: _renew_subs 2024-04-22T08:18:26.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.298+0000 7f47a5522640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:18:26.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.298+0000 7f47a5522640 1 -- 172.21.15.40:0/3992626697 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47a006f1c0 con 0x7f47a00a15c0 2024-04-22T08:18:26.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.298+0000 7f479d7fa640 4 set_mon_vals no callback set 2024-04-22T08:18:26.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.299+0000 7f47877fe640 1 -- 172.21.15.40:0/3992626697 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f479401e460 con 0x7f47a00a15c0 2024-04-22T08:18:26.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.299+0000 7f47a5522640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:18:26.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.299+0000 7f47a5522640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:18:26.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.299+0000 7f47a5522640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:18:26.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.299+0000 7f47a5522640 1 -- 172.21.15.40:0/3992626697 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47a004e9b0 con 0x7f47a00a15c0 2024-04-22T08:18:26.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.299+0000 7f47877fe640 1 --2- 172.21.15.40:0/3992626697 >> v2:172.21.15.189:6800/30168 conn(0x7f478c0377d0 0x7f478c039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:26.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.300+0000 7f47877fe640 1 -- 172.21.15.40:0/3992626697 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f479404a770 con 0x7f47a00a15c0 2024-04-22T08:18:26.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.300+0000 7f479effd640 1 --2- 172.21.15.40:0/3992626697 >> v2:172.21.15.189:6800/30168 conn(0x7f478c0377d0 0x7f478c039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:26.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.300+0000 7f479effd640 10 monclient: get_auth_request con 0x7f478c0377d0 auth_method 0 2024-04-22T08:18:26.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.302+0000 7f479effd640 1 --2- 172.21.15.40:0/3992626697 >> v2:172.21.15.189:6800/30168 conn(0x7f478c0377d0 0x7f478c039c90 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f479000c7f0 tx=0x7f4790009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:26.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.304+0000 7f47877fe640 1 -- 172.21.15.40:0/3992626697 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f479401c720 con 0x7f47a00a15c0 2024-04-22T08:18:26.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.304+0000 7f47877fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:18:26.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.304+0000 7f47877fe640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:18:26.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.552+0000 7f47a5522640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:18:26.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.552+0000 7f47a5522640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:18:26.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.552+0000 7f47a5522640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:18:26.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.552+0000 7f47a5522640 1 -- 172.21.15.40:0/3992626697 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f47a006a1a0 con 0x7f47a00a15c0 2024-04-22T08:18:26.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.553+0000 7f47877fe640 1 -- 172.21.15.40:0/3992626697 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f479401c360 con 0x7f47a00a15c0 2024-04-22T08:18:26.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.553+0000 7f47877fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:18:26.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.553+0000 7f47877fe640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:18:26.553 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":56,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:18:26.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.555+0000 7f47a5522640 1 -- 172.21.15.40:0/3992626697 >> v2:172.21.15.189:6800/30168 conn(0x7f478c0377d0 msgr2=0x7f478c039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:26.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.555+0000 7f47a5522640 1 --2- 172.21.15.40:0/3992626697 >> v2:172.21.15.189:6800/30168 conn(0x7f478c0377d0 0x7f478c039c90 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f479000c7f0 tx=0x7f4790009040 comp rx=0 tx=0).stop 2024-04-22T08:18:26.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.555+0000 7f47a5522640 10 monclient: shutdown 2024-04-22T08:18:26.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.555+0000 7f47a5522640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:18:26.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.555+0000 7f47a5522640 1 -- 172.21.15.40:0/3992626697 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f47a00a15c0 msgr2=0x7f47a01c8fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:26.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.555+0000 7f47a5522640 1 --2- 172.21.15.40:0/3992626697 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f47a00a15c0 0x7f47a01c8fd0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f479400d5f0 tx=0x7f479400dab0 comp rx=0 tx=0).stop 2024-04-22T08:18:26.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.555+0000 7f47a5522640 1 -- 172.21.15.40:0/3992626697 shutdown_connections 2024-04-22T08:18:26.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.555+0000 7f47a5522640 1 --2- 172.21.15.40:0/3992626697 >> v2:172.21.15.189:6800/30168 conn(0x7f478c0377d0 0x7f478c039c90 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:26.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.555+0000 7f47a5522640 1 --2- 172.21.15.40:0/3992626697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47a00a2560 0x7f47a012f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:26.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.555+0000 7f47a5522640 1 --2- 172.21.15.40:0/3992626697 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f47a00a15c0 0x7f47a01c8fd0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:26.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.555+0000 7f47a5522640 1 --2- 172.21.15.40:0/3992626697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47a00a1110 0x7f47a01c8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:26.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.555+0000 7f47a5522640 1 -- 172.21.15.40:0/3992626697 >> 172.21.15.40:0/3992626697 conn(0x7f47a00a5960 msgr2=0x7f47a00a0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:18:26.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.555+0000 7f47a5522640 1 -- 172.21.15.40:0/3992626697 shutdown_connections 2024-04-22T08:18:26.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.555+0000 7f47a5522640 1 -- 172.21.15.40:0/3992626697 wait complete. 2024-04-22T08:18:26.565 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-22T08:18:26.565 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-22T08:18:26.565 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-22T08:18:26.565 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-22T08:18:26.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.666+0000 7ff0e9155640 10 monclient: get_monmap_and_config 2024-04-22T08:18:26.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.666+0000 7ff0e9155640 10 monclient: build_initial_monmap 2024-04-22T08:18:26.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.666+0000 7ff0e9155640 10 monclient: monmap: 2024-04-22T08:18:26.667 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:18:26.667 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:26.667 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:18:26.667638+0000 2024-04-22T08:18:26.667 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:18:26.667638+0000 2024-04-22T08:18:26.667 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:18:26.667 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:26.667 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:18:26.667 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:18:26.667 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:18:26.667 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:18:26.667 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:18:26.667 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:26.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.667+0000 7ff0e9155640 1 Processor -- start 2024-04-22T08:18:26.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.667+0000 7ff0e9155640 1 -- start start 2024-04-22T08:18:26.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.667+0000 7ff0e9155640 10 monclient: init 2024-04-22T08:18:26.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.667+0000 7ff0e9155640 10 monclient: _reopen_session rank -1 2024-04-22T08:18:26.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.667+0000 7ff0e9155640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-04-22T08:18:26.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.667+0000 7ff0e9155640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff0e4136070 0x7ff0e409aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:26.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.667+0000 7ff0e9155640 10 monclient(hunting): picked mon.noname-c con 0x7ff0e4136070 addr v2:172.21.15.189:3300/0 2024-04-22T08:18:26.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.667+0000 7ff0e9155640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff0e4136450 0x7ff0e40a0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:26.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.667+0000 7ff0e9155640 10 monclient(hunting): picked mon.noname-a con 0x7ff0e4136450 addr v2:172.21.15.40:3300/0 2024-04-22T08:18:26.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.667+0000 7ff0e9155640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff0e41368b0 addr v1:172.21.15.40:6789/0 2024-04-22T08:18:26.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.667+0000 7ff0e9155640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0e406bb50 con 0x7ff0e41368b0 2024-04-22T08:18:26.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.667+0000 7ff0e9155640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:26.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.667+0000 7ff0e9155640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0e409a390 con 0x7ff0e4136450 2024-04-22T08:18:26.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.667+0000 7ff0e9155640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:26.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.667+0000 7ff0e9155640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0e4099dd0 con 0x7ff0e4136070 2024-04-22T08:18:26.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.667+0000 7ff0e9155640 10 monclient(hunting): _renew_subs 2024-04-22T08:18:26.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.667+0000 7ff0e9155640 10 monclient(hunting): authenticate will time out at 1180.661730s 2024-04-22T08:18:26.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.668+0000 7ff0e259c640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff0e4136450 0x7ff0e40a0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:26.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.668+0000 7ff0e259c640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff0e4136450 0x7ff0e40a0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44666/0 (socket says 172.21.15.40:44666) 2024-04-22T08:18:26.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.668+0000 7ff0e259c640 1 -- 172.21.15.40:0/50143435 learned_addr learned my addr 172.21.15.40:0/50143435 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:18:26.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.668+0000 7ff0e259c640 10 monclient(hunting): get_auth_request con 0x7ff0e4136450 auth_method 0 2024-04-22T08:18:26.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.668+0000 7ff0e259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:26.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.668+0000 7ff0e259c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:26.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.668+0000 7ff0e259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:26.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.668+0000 7ff0e259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:26.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.668+0000 7ff0e259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:26.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.668+0000 7ff0e259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:26.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.668+0000 7ff0e2d9d640 1 --2- 172.21.15.40:0/50143435 >> v2:172.21.15.189:3300/0 conn(0x7ff0e4136070 0x7ff0e409aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:26.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.668+0000 7ff0e2d9d640 10 monclient(hunting): get_auth_request con 0x7ff0e4136070 auth_method 0 2024-04-22T08:18:26.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.668+0000 7ff0e2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:26.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.668+0000 7ff0e2d9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:26.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.668+0000 7ff0e2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:26.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.668+0000 7ff0e259c640 10 monclient(hunting): handle_auth_done global_id 74133 payload 274 2024-04-22T08:18:26.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.668+0000 7ff0e259c640 1 -- 172.21.15.40:0/50143435 >> v2:172.21.15.189:3300/0 conn(0x7ff0e4136070 msgr2=0x7ff0e409aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:18:26.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.668+0000 7ff0e259c640 1 --2- 172.21.15.40:0/50143435 >> v2:172.21.15.189:3300/0 conn(0x7ff0e4136070 0x7ff0e409aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:26.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.668+0000 7ff0e259c640 1 -- 172.21.15.40:0/50143435 >> v1:172.21.15.40:6789/0 conn(0x7ff0e41368b0 legacy=0x7ff0e40a0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:18:26.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.668+0000 7ff0e259c640 10 monclient: _finish_hunting 0 2024-04-22T08:18:26.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.668+0000 7ff0e259c640 1 monclient: found mon.noname-a 2024-04-22T08:18:26.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.668+0000 7ff0e259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:18:26.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.668+0000 7ff0e259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:18:26.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.668+0000 7ff0e259c640 1 -- 172.21.15.40:0/50143435 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e409a0b0 con 0x7ff0e4136450 2024-04-22T08:18:26.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.671+0000 7ff0e259c640 1 -- 172.21.15.40:0/50143435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0e4136450 msgr2=0x7ff0e40a0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:18:26.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.671+0000 7ff0e259c640 1 --2- 172.21.15.40:0/50143435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0e4136450 0x7ff0e40a0a70 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff0e406c170 tx=0x7ff0cc02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d026f8b39c8b2463 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:26.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.671+0000 7ff0e1d9b640 1 -- 172.21.15.40:0/50143435 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc031960 con 0x7ff0e4136450 2024-04-22T08:18:26.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.671+0000 7ff0e1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:26.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.671+0000 7ff0e1d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:18:26.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.671+0000 7ff0e1d9b640 10 monclient: dump: 2024-04-22T08:18:26.675 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:26.675 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:26.676 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:26.676 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:26.676 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:26.676 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:26.676 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:26.676 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:26.676 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:26.676 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:26.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.671+0000 7ff0e1d9b640 10 monclient: _finish_auth 0 2024-04-22T08:18:26.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.671+0000 7ff0e1d9b640 10 monclient: _check_auth_tickets 2024-04-22T08:18:26.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.671+0000 7ff0e1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:18:26.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.671+0000 7ff0e9155640 5 monclient: authenticate success, global_id 74133 2024-04-22T08:18:26.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.671+0000 7ff0e1d9b640 1 -- 172.21.15.40:0/50143435 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0cc031ac0 con 0x7ff0e4136450 2024-04-22T08:18:26.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.671+0000 7ff0e9155640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:18:26.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.671+0000 7ff0e1d9b640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:18:26.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.671+0000 7ff0e9155640 10 monclient: get_monmap_and_config success 2024-04-22T08:18:26.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.671+0000 7ff0e1d9b640 1 -- 172.21.15.40:0/50143435 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc031e50 con 0x7ff0e4136450 2024-04-22T08:18:26.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.671+0000 7ff0e1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:26.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.671+0000 7ff0e9155640 4 set_mon_vals no callback set 2024-04-22T08:18:26.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.671+0000 7ff0e1d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:18:26.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.671+0000 7ff0e1d9b640 10 monclient: dump: 2024-04-22T08:18:26.677 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:26.677 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:26.677 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:26.677 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:26.677 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:26.677 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:26.677 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:26.677 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:26.677 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:26.678 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:26.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.671+0000 7ff0e9155640 10 monclient: shutdown 2024-04-22T08:18:26.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.671+0000 7ff0e9155640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:18:26.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.671+0000 7ff0e9155640 1 -- 172.21.15.40:0/50143435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0e4136450 msgr2=0x7ff0e40a0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:26.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.671+0000 7ff0e9155640 1 --2- 172.21.15.40:0/50143435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0e4136450 0x7ff0e40a0a70 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff0e406c170 tx=0x7ff0cc02b7f0 comp rx=0 tx=0).stop 2024-04-22T08:18:26.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.671+0000 7ff0e9155640 1 -- 172.21.15.40:0/50143435 shutdown_connections 2024-04-22T08:18:26.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.671+0000 7ff0e9155640 1 --2- 172.21.15.40:0/50143435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0e4136450 0x7ff0e40a0a70 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:26.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.671+0000 7ff0e9155640 1 --2- 172.21.15.40:0/50143435 >> v2:172.21.15.189:3300/0 conn(0x7ff0e4136070 0x7ff0e409aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:26.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.671+0000 7ff0e9155640 1 -- 172.21.15.40:0/50143435 >> 172.21.15.40:0/50143435 conn(0x7ff0e40a1b60 msgr2=0x7ff0e40a1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:18:26.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.672+0000 7ff0e9155640 1 -- 172.21.15.40:0/50143435 shutdown_connections 2024-04-22T08:18:26.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.672+0000 7ff0e9155640 1 -- 172.21.15.40:0/50143435 wait complete. 2024-04-22T08:18:26.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.672+0000 7ff0e9155640 10 monclient: build_initial_monmap 2024-04-22T08:18:26.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.672+0000 7ff0e9155640 10 monclient: monmap: 2024-04-22T08:18:26.679 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:18:26.679 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:18:26.679 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:18:26.679 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:18:26.679 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:18:26.679 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:26.679 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:18:26.679 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:18:26.679 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:18:26.679 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:26.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.673+0000 7ff0e9155640 1 Processor -- start 2024-04-22T08:18:26.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.673+0000 7ff0e9155640 1 -- start start 2024-04-22T08:18:26.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.673+0000 7ff0e9155640 10 monclient: init 2024-04-22T08:18:26.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.673+0000 7ff0e9155640 10 monclient: _reopen_session rank -1 2024-04-22T08:18:26.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.673+0000 7ff0e9155640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-22T08:18:26.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.673+0000 7ff0e9155640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0e4136070 0x7ff0e41cad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:26.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.673+0000 7ff0e9155640 10 monclient(hunting): picked mon.noname-b con 0x7ff0e4136070 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:18:26.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.673+0000 7ff0e9155640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff0e4136450 0x7ff0e41cb250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:26.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.673+0000 7ff0e9155640 10 monclient(hunting): picked mon.noname-c con 0x7ff0e4136450 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:18:26.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.673+0000 7ff0e9155640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0e41222d0 0x7ff0e4122760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:26.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.673+0000 7ff0e9155640 10 monclient(hunting): picked mon.noname-a con 0x7ff0e41222d0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:18:26.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.673+0000 7ff0e9155640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:26.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.673+0000 7ff0e9155640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e409ed30 con 0x7ff0e41222d0 2024-04-22T08:18:26.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.673+0000 7ff0e9155640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:26.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.673+0000 7ff0e9155640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e4124400 con 0x7ff0e4136070 2024-04-22T08:18:26.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.673+0000 7ff0e9155640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:26.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.673+0000 7ff0e9155640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e4126230 con 0x7ff0e4136450 2024-04-22T08:18:26.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.673+0000 7ff0e9155640 10 monclient(hunting): _renew_subs 2024-04-22T08:18:26.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.673+0000 7ff0e9155640 10 monclient(hunting): authenticate will time out at 1180.667500s 2024-04-22T08:18:26.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.673+0000 7ff0e359e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0e41222d0 0x7ff0e4122760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:26.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.673+0000 7ff0e259c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff0e4136450 0x7ff0e41cb250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:26.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.673+0000 7ff0e359e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0e41222d0 0x7ff0e4122760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44680/0 (socket says 172.21.15.40:44680) 2024-04-22T08:18:26.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.673+0000 7ff0e359e640 1 -- 172.21.15.40:0/1820915948 learned_addr learned my addr 172.21.15.40:0/1820915948 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:18:26.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.673+0000 7ff0e259c640 10 monclient(hunting): get_auth_request con 0x7ff0e4136450 auth_method 0 2024-04-22T08:18:26.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.673+0000 7ff0e259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:26.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.673+0000 7ff0e259c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:26.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.673+0000 7ff0e259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:26.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.673+0000 7ff0e359e640 10 monclient(hunting): get_auth_request con 0x7ff0e41222d0 auth_method 0 2024-04-22T08:18:26.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.673+0000 7ff0e359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:26.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.673+0000 7ff0e359e640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:26.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.673+0000 7ff0e359e640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:26.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.673+0000 7ff0e259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:26.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.673+0000 7ff0e259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:26.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.673+0000 7ff0e259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:26.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.673+0000 7ff0e359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:26.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.673+0000 7ff0e359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:26.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.673+0000 7ff0e359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:26.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.674+0000 7ff0e2d9d640 1 --2- 172.21.15.40:0/1820915948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0e4136070 0x7ff0e41cad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:26.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.674+0000 7ff0e2d9d640 10 monclient(hunting): get_auth_request con 0x7ff0e4136070 auth_method 0 2024-04-22T08:18:26.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.674+0000 7ff0e2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:26.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.674+0000 7ff0e2d9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:26.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.674+0000 7ff0e2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:26.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.674+0000 7ff0e2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:26.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.674+0000 7ff0e2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:26.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.674+0000 7ff0e2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:26.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.674+0000 7ff0e259c640 10 monclient(hunting): handle_auth_done global_id 54491 payload 824 2024-04-22T08:18:26.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.674+0000 7ff0e259c640 1 -- 172.21.15.40:0/1820915948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0e4136070 msgr2=0x7ff0e41cad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:26.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.674+0000 7ff0e259c640 1 --2- 172.21.15.40:0/1820915948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0e4136070 0x7ff0e41cad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:26.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.674+0000 7ff0e259c640 1 -- 172.21.15.40:0/1820915948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0e41222d0 msgr2=0x7ff0e4122760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:26.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.674+0000 7ff0e259c640 1 --2- 172.21.15.40:0/1820915948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0e41222d0 0x7ff0e4122760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:26.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.674+0000 7ff0e259c640 10 monclient: _finish_hunting 0 2024-04-22T08:18:26.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.674+0000 7ff0e259c640 1 monclient: found mon.noname-c 2024-04-22T08:18:26.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.674+0000 7ff0e259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:18:26.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.674+0000 7ff0e259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:18:26.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.674+0000 7ff0e259c640 1 -- 172.21.15.40:0/1820915948 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0cc0094d0 con 0x7ff0e4136450 2024-04-22T08:18:26.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.674+0000 7ff0e359e640 1 --2- 172.21.15.40:0/1820915948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0e41222d0 0x7ff0e4122760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:18:26.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.676+0000 7ff0e259c640 1 -- 172.21.15.40:0/1820915948 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff0e4136450 msgr2=0x7ff0e41cb250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:18:26.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.676+0000 7ff0e259c640 1 --2- 172.21.15.40:0/1820915948 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff0e4136450 0x7ff0e41cb250 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc033000 tx=0x7ff0cc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:26.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.676+0000 7ff0db7fe640 1 -- 172.21.15.40:0/1820915948 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc031e30 con 0x7ff0e4136450 2024-04-22T08:18:26.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.676+0000 7ff0db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:26.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.676+0000 7ff0db7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:18:26.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.676+0000 7ff0db7fe640 10 monclient: dump: 2024-04-22T08:18:26.689 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:26.689 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:26.689 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:26.690 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:26.690 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:26.690 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:26.690 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:26.690 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:26.690 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:26.690 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:26.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.676+0000 7ff0db7fe640 10 monclient: _finish_auth 0 2024-04-22T08:18:26.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.676+0000 7ff0db7fe640 10 monclient: _check_auth_tickets 2024-04-22T08:18:26.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.676+0000 7ff0db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:18:26.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.676+0000 7ff0e9155640 5 monclient: authenticate success, global_id 54491 2024-04-22T08:18:26.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.676+0000 7ff0db7fe640 1 -- 172.21.15.40:0/1820915948 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0cc03c400 con 0x7ff0e4136450 2024-04-22T08:18:26.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.676+0000 7ff0db7fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:18:26.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.676+0000 7ff0db7fe640 1 -- 172.21.15.40:0/1820915948 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc03c770 con 0x7ff0e4136450 2024-04-22T08:18:26.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.676+0000 7ff0e9155640 10 monclient: _renew_subs 2024-04-22T08:18:26.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.676+0000 7ff0e9155640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:18:26.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.676+0000 7ff0e9155640 1 -- 172.21.15.40:0/1820915948 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0e4124aa0 con 0x7ff0e4136450 2024-04-22T08:18:26.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.676+0000 7ff0e159a640 4 set_mon_vals no callback set 2024-04-22T08:18:26.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.676+0000 7ff0db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:26.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.676+0000 7ff0db7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:18:26.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.676+0000 7ff0db7fe640 10 monclient: dump: 2024-04-22T08:18:26.691 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:26.691 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:26.691 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:26.691 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:26.691 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:26.691 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:26.692 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:26.692 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:26.692 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:26.692 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:26.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.676+0000 7ff0e9155640 10 monclient: _renew_subs 2024-04-22T08:18:26.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.676+0000 7ff0e9155640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:18:26.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.676+0000 7ff0e9155640 1 -- 172.21.15.40:0/1820915948 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0e4125f50 con 0x7ff0e4136450 2024-04-22T08:18:26.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.677+0000 7ff0e9155640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:18:26.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.677+0000 7ff0db7fe640 1 -- 172.21.15.40:0/1820915948 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7ff0cc03a040 con 0x7ff0e4136450 2024-04-22T08:18:26.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.677+0000 7ff0e9155640 1 --2- 172.21.15.40:0/1820915948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0e41368b0 0x7ff0e402b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:26.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.677+0000 7ff0e9155640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:26.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.677+0000 7ff0e9155640 1 -- 172.21.15.40:0/1820915948 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e40667a0 con 0x7ff0e41368b0 2024-04-22T08:18:26.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.677+0000 7ff0e2d9d640 1 --2- 172.21.15.40:0/1820915948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0e41368b0 0x7ff0e402b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:26.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.677+0000 7ff0e2d9d640 10 monclient: get_auth_request con 0x7ff0e41368b0 auth_method 0 2024-04-22T08:18:26.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.677+0000 7ff0e2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:26.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.677+0000 7ff0e2d9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:26.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.677+0000 7ff0e2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:26.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.677+0000 7ff0db7fe640 1 --2- 172.21.15.40:0/1820915948 >> v2:172.21.15.189:6800/30168 conn(0x7ff0c40377d0 0x7ff0c4039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:26.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.677+0000 7ff0e2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:26.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.677+0000 7ff0e2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:26.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.677+0000 7ff0e2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:26.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.677+0000 7ff0db7fe640 1 -- 172.21.15.40:0/1820915948 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 3506+0+0 (secure 0 0 0) 0x7ff0cc06bdb0 con 0x7ff0e4136450 2024-04-22T08:18:26.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.677+0000 7ff0e2d9d640 10 monclient(hunting): handle_auth_done global_id 74139 payload 824 2024-04-22T08:18:26.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.677+0000 7ff0e2d9d640 1 -- 172.21.15.40:0/1820915948 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff0e406c810 con 0x7ff0e41368b0 2024-04-22T08:18:26.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.678+0000 7ff0e359e640 1 --2- 172.21.15.40:0/1820915948 >> v2:172.21.15.189:6800/30168 conn(0x7ff0c40377d0 0x7ff0c4039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:26.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.678+0000 7ff0e359e640 10 monclient: get_auth_request con 0x7ff0c40377d0 auth_method 0 2024-04-22T08:18:26.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.680+0000 7ff0e2d9d640 1 -- 172.21.15.40:0/1820915948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0e41368b0 msgr2=0x7ff0e402b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:18:26.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.680+0000 7ff0e2d9d640 1 --2- 172.21.15.40:0/1820915948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0e41368b0 0x7ff0e402b200 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc010780 tx=0x7ff0dc010c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:26.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.680+0000 7ff0db7fe640 1 -- 172.21.15.40:0/1820915948 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0dc004450 con 0x7ff0e41368b0 2024-04-22T08:18:26.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.680+0000 7ff0db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:26.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.680+0000 7ff0db7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:18:26.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.680+0000 7ff0db7fe640 10 monclient: dump: 2024-04-22T08:18:26.694 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:26.694 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:26.694 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:26.694 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:26.694 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:26.694 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:26.694 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:26.694 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:26.694 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:26.695 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:26.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.680+0000 7ff0db7fe640 1 -- 172.21.15.40:0/1820915948 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7ff0dc00e820 con 0x7ff0e41368b0 2024-04-22T08:18:26.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.680+0000 7ff0db7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:18:26.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.680+0000 7ff0db7fe640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:18:26.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.680+0000 7ff0db7fe640 1 -- 172.21.15.40:0/1820915948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0e41368b0 msgr2=0x7ff0e402b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:26.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.680+0000 7ff0db7fe640 1 --2- 172.21.15.40:0/1820915948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0e41368b0 0x7ff0e402b200 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc010780 tx=0x7ff0dc010c40 comp rx=0 tx=0).stop 2024-04-22T08:18:26.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.680+0000 7ff0db7fe640 1 -- 172.21.15.40:0/1820915948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0e41368b0 msgr2=0x7ff0e402b200 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:18:26.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.680+0000 7ff0db7fe640 1 --2- 172.21.15.40:0/1820915948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0e41368b0 0x7ff0e402b200 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc010780 tx=0x7ff0dc010c40 comp rx=0 tx=0).stop 2024-04-22T08:18:26.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.680+0000 7ff0e359e640 1 --2- 172.21.15.40:0/1820915948 >> v2:172.21.15.189:6800/30168 conn(0x7ff0c40377d0 0x7ff0c4039c90 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff0d4007fd0 tx=0x7ff0d4003de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:26.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.686+0000 7ff0e9155640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:18:26.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.686+0000 7ff0e9155640 1 --2- 172.21.15.40:0/1820915948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0e4123460 0x7ff0e402b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:26.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.686+0000 7ff0e9155640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:26.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.686+0000 7ff0e9155640 1 -- 172.21.15.40:0/1820915948 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e404e9b0 con 0x7ff0e4123460 2024-04-22T08:18:26.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.686+0000 7ff0e2d9d640 1 --2- 172.21.15.40:0/1820915948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0e4123460 0x7ff0e402b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:26.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.686+0000 7ff0e2d9d640 10 monclient: get_auth_request con 0x7ff0e4123460 auth_method 0 2024-04-22T08:18:26.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.686+0000 7ff0e2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:26.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.686+0000 7ff0e2d9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:26.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.686+0000 7ff0e2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:26.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.687+0000 7ff0e2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:26.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.687+0000 7ff0e2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:26.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.687+0000 7ff0e2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:26.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.687+0000 7ff0e2d9d640 10 monclient(hunting): handle_auth_done global_id 74142 payload 824 2024-04-22T08:18:26.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.687+0000 7ff0e2d9d640 1 -- 172.21.15.40:0/1820915948 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff0e4125a50 con 0x7ff0e4123460 2024-04-22T08:18:26.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.689+0000 7ff0e2d9d640 1 -- 172.21.15.40:0/1820915948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0e4123460 msgr2=0x7ff0e402b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:18:26.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.689+0000 7ff0e2d9d640 1 --2- 172.21.15.40:0/1820915948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0e4123460 0x7ff0e402b740 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc019000 tx=0x7ff0dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:26.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.689+0000 7ff0db7fe640 1 -- 172.21.15.40:0/1820915948 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0dc025070 con 0x7ff0e4123460 2024-04-22T08:18:26.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.689+0000 7ff0db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:26.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.689+0000 7ff0db7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:18:26.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.689+0000 7ff0db7fe640 10 monclient: dump: 2024-04-22T08:18:26.699 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:26.699 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:26.699 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:26.699 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:26.699 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:26.699 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:26.699 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:26.699 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:26.699 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:26.699 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:26.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.689+0000 7ff0db7fe640 1 -- 172.21.15.40:0/1820915948 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4463 (secure 0 0 0) 0x7ff0dc003cb0 con 0x7ff0e4123460 2024-04-22T08:18:26.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.689+0000 7ff0db7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:18:26.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.689+0000 7ff0db7fe640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:18:26.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.689+0000 7ff0db7fe640 1 -- 172.21.15.40:0/1820915948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0e4123460 msgr2=0x7ff0e402b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:26.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.689+0000 7ff0db7fe640 1 --2- 172.21.15.40:0/1820915948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0e4123460 0x7ff0e402b740 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc019000 tx=0x7ff0dc0046c0 comp rx=0 tx=0).stop 2024-04-22T08:18:26.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.689+0000 7ff0db7fe640 1 -- 172.21.15.40:0/1820915948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0e4123460 msgr2=0x7ff0e402b740 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:18:26.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.689+0000 7ff0db7fe640 1 --2- 172.21.15.40:0/1820915948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0e4123460 0x7ff0e402b740 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc019000 tx=0x7ff0dc0046c0 comp rx=0 tx=0).stop 2024-04-22T08:18:26.700 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:18:26.700 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:18:26.700 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:18:26.700 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "leader", 2024-04-22T08:18:26.700 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 56, 2024-04-22T08:18:26.700 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:18:26.701 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:18:26.701 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:18:26.701 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:18:26.701 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:18:26.701 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 2, 2024-04-22T08:18:26.701 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:18:26.701 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:18:26.701 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:18:26.701 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:18:26.701 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:18:26.701 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:18:26.701 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:18:26.701 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:18:26.702 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:18:26.702 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:18:26.702 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:18:26.702 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:18:26.702 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:18:26.702 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:18:26.702 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:18:26.702 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:18:26.702 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:18:26.702 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:18:26.702 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:18:26.702 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:18:26.703 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:18:26.703 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:18:26.705 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:18:26.705 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:18:26.705 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:18:26.705 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:18:26.706 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:26.706 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:26.706 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:18:26.706 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:18:26.706 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:18:26.706 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:18:26.706 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:18:26.706 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:18:26.706 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:18:26.706 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:18:26.706 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:18:26.706 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:18:26.706 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:18:26.706 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:18:26.707 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:18:26.707 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:18:26.707 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:18:26.707 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:18:26.707 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:18:26.707 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:18:26.707 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:18:26.707 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:18:26.707 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:18:26.707 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:18:26.707 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:18:26.707 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:18:26.708 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:18:26.708 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:18:26.708 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:18:26.708 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:18:26.708 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:18:26.708 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:26.708 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:18:26.708 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:26.708 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:18:26.708 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:18:26.708 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:18:26.708 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:18:26.708 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:26.708 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:18:26.709 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:18:26.709 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:26.709 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:26.709 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:26.709 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:18:26.709 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:18:26.709 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:26.709 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:26.709 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:26.709 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:26.709 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:18:26.709 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:18:26.709 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:18:26.709 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:18:26.710 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:18:26.710 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:26.710 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:26.710 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:18:26.710 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:18:26.710 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:18:26.710 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:18:26.710 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:26.710 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:18:26.710 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:18:26.710 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:26.710 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:26.710 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:26.711 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:18:26.711 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:18:26.711 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:26.711 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:26.711 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:26.711 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:26.711 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:18:26.711 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:18:26.711 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:18:26.711 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:18:26.711 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:18:26.711 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:26.711 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:26.711 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:18:26.711 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:18:26.712 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:18:26.712 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:18:26.712 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:26.712 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:18:26.712 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:18:26.712 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:26.712 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:26.712 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:26.712 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:18:26.712 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:18:26.712 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:26.713 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:26.713 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:26.713 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:26.713 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:18:26.713 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:18:26.713 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:18:26.713 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:18:26.713 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:18:26.713 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:26.713 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:26.713 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:26.713 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:18:26.713 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:18:26.713 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:26.714 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:18:26.714 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:18:26.714 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:18:26.714 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:26.714 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:18:26.714 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:18:26.714 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:26.714 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:18:26.714 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:18:26.714 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:18:26.714 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:26.714 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:26.714 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:26.714 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:18:26.715 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:18:26.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.690+0000 7ff0e9155640 1 -- 172.21.15.40:0/1820915948 >> v2:172.21.15.189:6800/30168 conn(0x7ff0c40377d0 msgr2=0x7ff0c4039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:26.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.690+0000 7ff0e9155640 1 --2- 172.21.15.40:0/1820915948 >> v2:172.21.15.189:6800/30168 conn(0x7ff0c40377d0 0x7ff0c4039c90 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff0d4007fd0 tx=0x7ff0d4003de0 comp rx=0 tx=0).stop 2024-04-22T08:18:26.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.690+0000 7ff0e9155640 10 monclient: shutdown 2024-04-22T08:18:26.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.690+0000 7ff0e9155640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:18:26.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.690+0000 7ff0e9155640 1 -- 172.21.15.40:0/1820915948 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff0e4136450 msgr2=0x7ff0e41cb250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:26.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.690+0000 7ff0e9155640 1 --2- 172.21.15.40:0/1820915948 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff0e4136450 0x7ff0e41cb250 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc033000 tx=0x7ff0cc0046c0 comp rx=0 tx=0).stop 2024-04-22T08:18:26.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.690+0000 7ff0e359e640 1 -- 172.21.15.40:0/1820915948 reap_dead start 2024-04-22T08:18:26.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.690+0000 7ff0e359e640 1 -- 172.21.15.40:0/1820915948 reap_dead start 2024-04-22T08:18:26.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.690+0000 7ff0e9155640 1 -- 172.21.15.40:0/1820915948 shutdown_connections 2024-04-22T08:18:26.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.690+0000 7ff0e9155640 1 -- 172.21.15.40:0/1820915948 >> 172.21.15.40:0/1820915948 conn(0x7ff0e40a1b60 msgr2=0x7ff0e40a37f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:18:26.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.690+0000 7ff0e9155640 1 -- 172.21.15.40:0/1820915948 shutdown_connections 2024-04-22T08:18:26.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.690+0000 7ff0e9155640 1 -- 172.21.15.40:0/1820915948 wait complete. 2024-04-22T08:18:26.717 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:18:26.717 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-22T08:18:26.717 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-22T08:18:26.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.817+0000 7f0e626a1640 10 monclient: get_monmap_and_config 2024-04-22T08:18:26.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.817+0000 7f0e626a1640 10 monclient: build_initial_monmap 2024-04-22T08:18:26.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.817+0000 7f0e626a1640 10 monclient: monmap: 2024-04-22T08:18:26.817 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:18:26.817 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:26.817 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:18:26.818345+0000 2024-04-22T08:18:26.817 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:18:26.818345+0000 2024-04-22T08:18:26.817 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:18:26.818 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:26.818 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:18:26.818 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:18:26.818 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:18:26.818 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:18:26.818 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:18:26.818 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:26.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.818+0000 7f0e626a1640 1 Processor -- start 2024-04-22T08:18:26.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.818+0000 7f0e626a1640 1 -- start start 2024-04-22T08:18:26.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.818+0000 7f0e626a1640 10 monclient: init 2024-04-22T08:18:26.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.818+0000 7f0e626a1640 10 monclient: _reopen_session rank -1 2024-04-22T08:18:26.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.818+0000 7f0e626a1640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-04-22T08:18:26.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.818+0000 7f0e626a1640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0e5c0736d0 0x7f0e5c135f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:26.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.818+0000 7f0e626a1640 10 monclient(hunting): picked mon.noname-c con 0x7f0e5c0736d0 addr v2:172.21.15.189:3300/0 2024-04-22T08:18:26.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.818+0000 7f0e626a1640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0e5c070eb0 0x7f0e5c071320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:26.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.818+0000 7f0e626a1640 10 monclient(hunting): picked mon.noname-a con 0x7f0e5c070eb0 addr v2:172.21.15.40:3300/0 2024-04-22T08:18:26.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.818+0000 7f0e626a1640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e5c09de00 0x7f0e5c09e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:26.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.818+0000 7f0e626a1640 10 monclient(hunting): picked mon.noname-b con 0x7f0e5c09de00 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:18:26.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.818+0000 7f0e626a1640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:26.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.818+0000 7f0e626a1640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e5c071860 con 0x7f0e5c070eb0 2024-04-22T08:18:26.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.818+0000 7f0e626a1640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:26.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.818+0000 7f0e626a1640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e5c073ab0 con 0x7f0e5c0736d0 2024-04-22T08:18:26.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.818+0000 7f0e626a1640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:26.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.818+0000 7f0e626a1640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5c136510 con 0x7f0e5c09de00 2024-04-22T08:18:26.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.818+0000 7f0e626a1640 10 monclient(hunting): _renew_subs 2024-04-22T08:18:26.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.818+0000 7f0e626a1640 10 monclient(hunting): authenticate will time out at 1180.812546s 2024-04-22T08:18:26.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.818+0000 7f0e60c3e640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e5c09de00 0x7f0e5c09e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:26.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.818+0000 7f0e5b7fe640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0e5c070eb0 0x7f0e5c071320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:26.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.818+0000 7f0e60c3e640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e5c09de00 0x7f0e5c09e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:51472/0 (socket says 172.21.15.40:51472) 2024-04-22T08:18:26.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.818+0000 7f0e5b7fe640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0e5c070eb0 0x7f0e5c071320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44704/0 (socket says 172.21.15.40:44704) 2024-04-22T08:18:26.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.818+0000 7f0e60c3e640 1 -- 172.21.15.40:0/265753151 learned_addr learned my addr 172.21.15.40:0/265753151 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:18:26.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.818+0000 7f0e60c3e640 10 monclient(hunting): get_auth_request con 0x7f0e5c09de00 auth_method 0 2024-04-22T08:18:26.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.818+0000 7f0e60c3e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:26.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.818+0000 7f0e60c3e640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:26.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.818+0000 7f0e60c3e640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:26.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.818+0000 7f0e5b7fe640 10 monclient(hunting): get_auth_request con 0x7f0e5c070eb0 auth_method 0 2024-04-22T08:18:26.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.819+0000 7f0e5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:26.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.819+0000 7f0e5b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:26.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.819+0000 7f0e5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:26.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.819+0000 7f0e60c3e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:26.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.819+0000 7f0e60c3e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:26.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.819+0000 7f0e60c3e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:26.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.819+0000 7f0e5b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:26.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.819+0000 7f0e5b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:26.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.819+0000 7f0e5b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:26.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.819+0000 7f0e5bfff640 1 --2- 172.21.15.40:0/265753151 >> v2:172.21.15.189:3300/0 conn(0x7f0e5c0736d0 0x7f0e5c135f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:26.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.819+0000 7f0e5bfff640 10 monclient(hunting): get_auth_request con 0x7f0e5c0736d0 auth_method 0 2024-04-22T08:18:26.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.819+0000 7f0e5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:26.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.819+0000 7f0e5bfff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:26.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.819+0000 7f0e5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:26.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.819+0000 7f0e60c3e640 10 monclient(hunting): handle_auth_done global_id 54494 payload 274 2024-04-22T08:18:26.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.819+0000 7f0e60c3e640 1 -- 172.21.15.40:0/265753151 >> v2:172.21.15.189:3300/0 conn(0x7f0e5c0736d0 msgr2=0x7f0e5c135f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:18:26.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.819+0000 7f0e60c3e640 1 --2- 172.21.15.40:0/265753151 >> v2:172.21.15.189:3300/0 conn(0x7f0e5c0736d0 0x7f0e5c135f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:26.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.819+0000 7f0e60c3e640 1 -- 172.21.15.40:0/265753151 >> v2:172.21.15.40:3300/0 conn(0x7f0e5c070eb0 msgr2=0x7f0e5c071320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:18:26.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.819+0000 7f0e60c3e640 1 --2- 172.21.15.40:0/265753151 >> v2:172.21.15.40:3300/0 conn(0x7f0e5c070eb0 0x7f0e5c071320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:26.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.819+0000 7f0e60c3e640 10 monclient: _finish_hunting 0 2024-04-22T08:18:26.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.819+0000 7f0e60c3e640 1 monclient: found mon.noname-b 2024-04-22T08:18:26.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.819+0000 7f0e60c3e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:18:26.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.819+0000 7f0e60c3e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:18:26.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.819+0000 7f0e60c3e640 1 -- 172.21.15.40:0/265753151 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e5c06ce90 con 0x7f0e5c09de00 2024-04-22T08:18:26.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.819+0000 7f0e5b7fe640 1 --2- 172.21.15.40:0/265753151 >> v2:172.21.15.40:3300/0 conn(0x7f0e5c070eb0 0x7f0e5c071320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:18:26.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.822+0000 7f0e60c3e640 1 -- 172.21.15.40:0/265753151 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e5c09de00 msgr2=0x7f0e5c09e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:18:26.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.822+0000 7f0e60c3e640 1 --2- 172.21.15.40:0/265753151 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e5c09de00 0x7f0e5c09e290 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c09eea0 tx=0x7f0e5402bdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20ddac566f4477de server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:26.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.822+0000 7f0e5affd640 1 -- 172.21.15.40:0/265753151 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5402cb60 con 0x7f0e5c09de00 2024-04-22T08:18:26.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.822+0000 7f0e5affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:26.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.822+0000 7f0e5affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:18:26.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.822+0000 7f0e5affd640 10 monclient: dump: 2024-04-22T08:18:26.831 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:26.831 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:26.831 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:26.831 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:26.831 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:26.831 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:26.831 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:26.831 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:26.832 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:26.832 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:26.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.822+0000 7f0e5affd640 10 monclient: _finish_auth 0 2024-04-22T08:18:26.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.822+0000 7f0e5affd640 10 monclient: _check_auth_tickets 2024-04-22T08:18:26.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.822+0000 7f0e5affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:18:26.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.822+0000 7f0e5affd640 1 -- 172.21.15.40:0/265753151 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e5402ccc0 con 0x7f0e5c09de00 2024-04-22T08:18:26.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.822+0000 7f0e626a1640 5 monclient: authenticate success, global_id 54494 2024-04-22T08:18:26.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.822+0000 7f0e626a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:18:26.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.822+0000 7f0e5affd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:18:26.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.822+0000 7f0e626a1640 10 monclient: get_monmap_and_config success 2024-04-22T08:18:26.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.822+0000 7f0e626a1640 4 set_mon_vals no callback set 2024-04-22T08:18:26.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.822+0000 7f0e5affd640 1 -- 172.21.15.40:0/265753151 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e54034610 con 0x7f0e5c09de00 2024-04-22T08:18:26.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.822+0000 7f0e5affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:26.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.822+0000 7f0e5affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:18:26.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.822+0000 7f0e5affd640 10 monclient: dump: 2024-04-22T08:18:26.833 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:26.833 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:26.833 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:26.833 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:26.833 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:26.833 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:26.833 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:26.833 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:26.834 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:26.834 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:26.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.822+0000 7f0e626a1640 10 monclient: shutdown 2024-04-22T08:18:26.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.822+0000 7f0e626a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:18:26.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.822+0000 7f0e626a1640 1 -- 172.21.15.40:0/265753151 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e5c09de00 msgr2=0x7f0e5c09e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:26.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.822+0000 7f0e626a1640 1 --2- 172.21.15.40:0/265753151 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e5c09de00 0x7f0e5c09e290 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c09eea0 tx=0x7f0e5402bdb0 comp rx=0 tx=0).stop 2024-04-22T08:18:26.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.822+0000 7f0e626a1640 1 -- 172.21.15.40:0/265753151 shutdown_connections 2024-04-22T08:18:26.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.822+0000 7f0e626a1640 1 --2- 172.21.15.40:0/265753151 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e5c09de00 0x7f0e5c09e290 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:26.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.822+0000 7f0e626a1640 1 --2- 172.21.15.40:0/265753151 >> v2:172.21.15.40:3300/0 conn(0x7f0e5c070eb0 0x7f0e5c071320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:26.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.822+0000 7f0e626a1640 1 --2- 172.21.15.40:0/265753151 >> v2:172.21.15.189:3300/0 conn(0x7f0e5c0736d0 0x7f0e5c135f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:26.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.822+0000 7f0e626a1640 1 -- 172.21.15.40:0/265753151 >> 172.21.15.40:0/265753151 conn(0x7f0e5c0a1780 msgr2=0x7f0e5c0a1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:18:26.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.822+0000 7f0e626a1640 1 -- 172.21.15.40:0/265753151 shutdown_connections 2024-04-22T08:18:26.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.822+0000 7f0e626a1640 1 -- 172.21.15.40:0/265753151 wait complete. 2024-04-22T08:18:26.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.823+0000 7f0e626a1640 10 monclient: build_initial_monmap 2024-04-22T08:18:26.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.823+0000 7f0e626a1640 10 monclient: monmap: 2024-04-22T08:18:26.835 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:18:26.835 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:18:26.835 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:18:26.835 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:18:26.835 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:18:26.835 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:26.835 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:18:26.836 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:18:26.836 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:18:26.836 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:26.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.823+0000 7f0e626a1640 1 Processor -- start 2024-04-22T08:18:26.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.823+0000 7f0e626a1640 1 -- start start 2024-04-22T08:18:26.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.823+0000 7f0e626a1640 10 monclient: init 2024-04-22T08:18:26.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e626a1640 10 monclient: _reopen_session rank -1 2024-04-22T08:18:26.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e626a1640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-22T08:18:26.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e626a1640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e5c070eb0 0x7f0e5c1cab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:26.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e626a1640 10 monclient(hunting): picked mon.noname-c con 0x7f0e5c070eb0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:18:26.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e626a1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e5c0736d0 0x7f0e5c1cb070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:26.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e626a1640 10 monclient(hunting): picked mon.noname-a con 0x7f0e5c0736d0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:18:26.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e626a1640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e5c09de00 0x7f0e5c11c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:26.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e626a1640 10 monclient(hunting): picked mon.noname-b con 0x7f0e5c09de00 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:18:26.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e626a1640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:26.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e626a1640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5c11edc0 con 0x7f0e5c0736d0 2024-04-22T08:18:26.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e626a1640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:26.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e626a1640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5c0a4090 con 0x7f0e5c09de00 2024-04-22T08:18:26.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e626a1640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:26.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e626a1640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5c1350b0 con 0x7f0e5c070eb0 2024-04-22T08:18:26.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e626a1640 10 monclient(hunting): _renew_subs 2024-04-22T08:18:26.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e626a1640 10 monclient(hunting): authenticate will time out at 1180.818067s 2024-04-22T08:18:26.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e5b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e5c0736d0 0x7f0e5c1cb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:26.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e5bfff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e5c070eb0 0x7f0e5c1cab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:26.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e5bfff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e5c070eb0 0x7f0e5c1cab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:51480/0 (socket says 172.21.15.40:51480) 2024-04-22T08:18:26.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e5b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e5c0736d0 0x7f0e5c1cb070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44712/0 (socket says 172.21.15.40:44712) 2024-04-22T08:18:26.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e5bfff640 1 -- 172.21.15.40:0/3762538217 learned_addr learned my addr 172.21.15.40:0/3762538217 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:18:26.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e5bfff640 10 monclient(hunting): get_auth_request con 0x7f0e5c070eb0 auth_method 0 2024-04-22T08:18:26.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:26.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e5bfff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:26.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:26.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e5b7fe640 10 monclient(hunting): get_auth_request con 0x7f0e5c0736d0 auth_method 0 2024-04-22T08:18:26.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:26.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e5b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:26.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:26.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e5b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:26.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e5b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:26.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e5b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:26.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e60c3e640 1 --2- 172.21.15.40:0/3762538217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e5c09de00 0x7f0e5c11c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:26.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e5bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:26.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e5bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:26.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e5bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:26.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e60c3e640 10 monclient(hunting): get_auth_request con 0x7f0e5c09de00 auth_method 0 2024-04-22T08:18:26.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e60c3e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:26.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e60c3e640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:26.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e60c3e640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:26.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e60c3e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:26.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e60c3e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:26.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e60c3e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:26.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e5b7fe640 10 monclient(hunting): handle_auth_done global_id 74148 payload 824 2024-04-22T08:18:26.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e5b7fe640 1 -- 172.21.15.40:0/3762538217 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e5c070eb0 msgr2=0x7f0e5c1cab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:26.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e5b7fe640 1 --2- 172.21.15.40:0/3762538217 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e5c070eb0 0x7f0e5c1cab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:26.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e5b7fe640 1 -- 172.21.15.40:0/3762538217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e5c09de00 msgr2=0x7f0e5c11c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:26.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e5b7fe640 1 --2- 172.21.15.40:0/3762538217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e5c09de00 0x7f0e5c11c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:26.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e5b7fe640 10 monclient: _finish_hunting 0 2024-04-22T08:18:26.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e5b7fe640 1 monclient: found mon.noname-a 2024-04-22T08:18:26.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e5b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:18:26.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e5b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:18:26.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e5b7fe640 1 -- 172.21.15.40:0/3762538217 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e50009400 con 0x7f0e5c0736d0 2024-04-22T08:18:26.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.824+0000 7f0e5bfff640 1 --2- 172.21.15.40:0/3762538217 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e5c070eb0 0x7f0e5c1cab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:18:26.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.826+0000 7f0e5b7fe640 1 -- 172.21.15.40:0/3762538217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e5c0736d0 msgr2=0x7f0e5c1cb070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:18:26.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.827+0000 7f0e5b7fe640 1 --2- 172.21.15.40:0/3762538217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e5c0736d0 0x7f0e5c1cb070 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0e5000e870 tx=0x7f0e5000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:26.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.827+0000 7f0e58ff9640 1 -- 172.21.15.40:0/3762538217 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5000cb50 con 0x7f0e5c0736d0 2024-04-22T08:18:26.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.827+0000 7f0e58ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:26.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.827+0000 7f0e58ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:18:26.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.827+0000 7f0e58ff9640 10 monclient: dump: 2024-04-22T08:18:26.846 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:26.846 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:26.846 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:26.846 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:26.846 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:26.847 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:26.847 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:26.847 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:26.847 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:26.847 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:26.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.827+0000 7f0e58ff9640 10 monclient: _finish_auth 0 2024-04-22T08:18:26.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.827+0000 7f0e58ff9640 10 monclient: _check_auth_tickets 2024-04-22T08:18:26.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.827+0000 7f0e58ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:18:26.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.827+0000 7f0e58ff9640 1 -- 172.21.15.40:0/3762538217 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e5000ccb0 con 0x7f0e5c0736d0 2024-04-22T08:18:26.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.827+0000 7f0e626a1640 5 monclient: authenticate success, global_id 74148 2024-04-22T08:18:26.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.827+0000 7f0e58ff9640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:18:26.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.827+0000 7f0e58ff9640 1 -- 172.21.15.40:0/3762538217 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e50013690 con 0x7f0e5c0736d0 2024-04-22T08:18:26.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.827+0000 7f0e58ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:26.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.827+0000 7f0e58ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:18:26.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.827+0000 7f0e58ff9640 10 monclient: dump: 2024-04-22T08:18:26.848 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:26.848 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:26.848 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:26.848 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:26.848 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:26.848 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:26.848 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:26.849 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:26.849 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:26.849 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:26.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.827+0000 7f0e626a1640 10 monclient: _renew_subs 2024-04-22T08:18:26.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.827+0000 7f0e5a7fc640 4 set_mon_vals no callback set 2024-04-22T08:18:26.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.827+0000 7f0e626a1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:18:26.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.827+0000 7f0e626a1640 1 -- 172.21.15.40:0/3762538217 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e54009b30 con 0x7f0e5c0736d0 2024-04-22T08:18:26.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.827+0000 7f0e626a1640 10 monclient: _renew_subs 2024-04-22T08:18:26.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.827+0000 7f0e626a1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:18:26.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.827+0000 7f0e626a1640 1 -- 172.21.15.40:0/3762538217 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e5c11fcb0 con 0x7f0e5c0736d0 2024-04-22T08:18:26.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.828+0000 7f0e626a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:18:26.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.828+0000 7f0e626a1640 1 --2- 172.21.15.40:0/3762538217 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e5c120970 0x7f0e5c11d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:26.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.828+0000 7f0e626a1640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:26.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.828+0000 7f0e626a1640 1 -- 172.21.15.40:0/3762538217 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5c066900 con 0x7f0e5c120970 2024-04-22T08:18:26.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.828+0000 7f0e5b7fe640 1 --2- 172.21.15.40:0/3762538217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e5c0736d0 0x7f0e5c1cb070 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0e5000e870 tx=0x7f0e5000ed30 comp rx=0 tx=0).queue_received will delay after 1.71377e+09 on 0x7f0e50013830 mgrmap(e 5) v1 2024-04-22T08:18:26.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.828+0000 7f0e58ff9640 1 -- 172.21.15.40:0/3762538217 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f0e50013830 con 0x7f0e5c0736d0 2024-04-22T08:18:26.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.828+0000 7f0e5bfff640 1 --2- 172.21.15.40:0/3762538217 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e5c120970 0x7f0e5c11d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:26.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.828+0000 7f0e5bfff640 10 monclient: get_auth_request con 0x7f0e5c120970 auth_method 0 2024-04-22T08:18:26.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.828+0000 7f0e5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:26.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.828+0000 7f0e5bfff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:26.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.828+0000 7f0e5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:26.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.828+0000 7f0e5bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:26.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.828+0000 7f0e5bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:26.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.828+0000 7f0e5bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:26.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.828+0000 7f0e58ff9640 1 --2- 172.21.15.40:0/3762538217 >> v2:172.21.15.189:6800/30168 conn(0x7f0e480377d0 0x7f0e48039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:26.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.828+0000 7f0e5bfff640 10 monclient(hunting): handle_auth_done global_id 54500 payload 824 2024-04-22T08:18:26.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.828+0000 7f0e5bfff640 1 -- 172.21.15.40:0/3762538217 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0e5c0a3980 con 0x7f0e5c120970 2024-04-22T08:18:26.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.828+0000 7f0e60c3e640 1 --2- 172.21.15.40:0/3762538217 >> v2:172.21.15.189:6800/30168 conn(0x7f0e480377d0 0x7f0e48039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:26.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.828+0000 7f0e60c3e640 10 monclient: get_auth_request con 0x7f0e480377d0 auth_method 0 2024-04-22T08:18:26.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.830+0000 7f0e5bfff640 1 -- 172.21.15.40:0/3762538217 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e5c120970 msgr2=0x7f0e5c11d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:18:26.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.830+0000 7f0e5bfff640 1 --2- 172.21.15.40:0/3762538217 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e5c120970 0x7f0e5c11d320 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c010c60 tx=0x7f0e4c00e5a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:26.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.830+0000 7f0e58ff9640 1 -- 172.21.15.40:0/3762538217 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c010ea0 con 0x7f0e5c120970 2024-04-22T08:18:26.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.830+0000 7f0e58ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:26.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.831+0000 7f0e58ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:18:26.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.831+0000 7f0e58ff9640 10 monclient: dump: 2024-04-22T08:18:26.852 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:26.852 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:26.852 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:26.852 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:26.852 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:26.852 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:26.852 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:26.852 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:26.852 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:26.852 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:26.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.831+0000 7f0e60c3e640 1 --2- 172.21.15.40:0/3762538217 >> v2:172.21.15.189:6800/30168 conn(0x7f0e480377d0 0x7f0e48039c90 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c073f30 tx=0x7f0e54030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:26.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.831+0000 7f0e58ff9640 1 -- 172.21.15.40:0/3762538217 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f0e4c004020 con 0x7f0e5c120970 2024-04-22T08:18:26.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.831+0000 7f0e58ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:18:26.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.831+0000 7f0e58ff9640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:18:26.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.831+0000 7f0e58ff9640 1 -- 172.21.15.40:0/3762538217 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e5c120970 msgr2=0x7f0e5c11d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:26.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.831+0000 7f0e58ff9640 1 --2- 172.21.15.40:0/3762538217 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e5c120970 0x7f0e5c11d320 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c010c60 tx=0x7f0e4c00e5a0 comp rx=0 tx=0).stop 2024-04-22T08:18:26.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.831+0000 7f0e58ff9640 1 -- 172.21.15.40:0/3762538217 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e5c120970 msgr2=0x7f0e5c11d320 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:18:26.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.831+0000 7f0e58ff9640 1 --2- 172.21.15.40:0/3762538217 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e5c120970 0x7f0e5c11d320 secure :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c010c60 tx=0x7f0e4c00e5a0 comp rx=0 tx=0).stop 2024-04-22T08:18:26.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.837+0000 7f0e626a1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:18:26.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.837+0000 7f0e626a1640 1 --2- 172.21.15.40:0/3762538217 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e5c02b200 0x7f0e5c02b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:26.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.837+0000 7f0e626a1640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:26.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.837+0000 7f0e626a1640 1 -- 172.21.15.40:0/3762538217 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5c06d450 con 0x7f0e5c02b200 2024-04-22T08:18:26.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.837+0000 7f0e5bfff640 1 --2- 172.21.15.40:0/3762538217 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e5c02b200 0x7f0e5c02b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:26.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.837+0000 7f0e5bfff640 10 monclient: get_auth_request con 0x7f0e5c02b200 auth_method 0 2024-04-22T08:18:26.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.837+0000 7f0e5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:26.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.837+0000 7f0e5bfff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:26.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.837+0000 7f0e5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:26.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.837+0000 7f0e5bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:26.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.837+0000 7f0e5bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:26.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.837+0000 7f0e5bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:26.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.837+0000 7f0e5bfff640 10 monclient(hunting): handle_auth_done global_id 54503 payload 824 2024-04-22T08:18:26.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.837+0000 7f0e5bfff640 1 -- 172.21.15.40:0/3762538217 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0e5c09e7d0 con 0x7f0e5c02b200 2024-04-22T08:18:26.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.840+0000 7f0e5bfff640 1 -- 172.21.15.40:0/3762538217 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e5c02b200 msgr2=0x7f0e5c02b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:18:26.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.840+0000 7f0e5bfff640 1 --2- 172.21.15.40:0/3762538217 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e5c02b200 0x7f0e5c02b610 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c019000 tx=0x7f0e4c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:26.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.840+0000 7f0e58ff9640 1 -- 172.21.15.40:0/3762538217 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c003810 con 0x7f0e5c02b200 2024-04-22T08:18:26.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.840+0000 7f0e58ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:26.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.840+0000 7f0e58ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:18:26.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.840+0000 7f0e58ff9640 10 monclient: dump: 2024-04-22T08:18:26.855 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:26.855 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:26.855 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:26.855 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:26.855 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:26.855 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:26.855 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:26.855 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:26.855 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:26.856 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:26.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.840+0000 7f0e58ff9640 1 -- 172.21.15.40:0/3762538217 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4631 (secure 0 0 0) 0x7f0e4c003970 con 0x7f0e5c02b200 2024-04-22T08:18:26.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.840+0000 7f0e58ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:18:26.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.840+0000 7f0e58ff9640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:18:26.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.840+0000 7f0e58ff9640 1 -- 172.21.15.40:0/3762538217 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e5c02b200 msgr2=0x7f0e5c02b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:26.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.840+0000 7f0e58ff9640 1 --2- 172.21.15.40:0/3762538217 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e5c02b200 0x7f0e5c02b610 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c019000 tx=0x7f0e4c0046c0 comp rx=0 tx=0).stop 2024-04-22T08:18:26.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.840+0000 7f0e58ff9640 1 -- 172.21.15.40:0/3762538217 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e5c02b200 msgr2=0x7f0e5c02b610 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:18:26.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.840+0000 7f0e58ff9640 1 --2- 172.21.15.40:0/3762538217 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e5c02b200 0x7f0e5c02b610 secure :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c019000 tx=0x7f0e4c0046c0 comp rx=0 tx=0).stop 2024-04-22T08:18:26.856 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:18:26.857 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:18:26.857 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:18:26.857 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:18:26.857 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 56, 2024-04-22T08:18:26.857 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:18:26.857 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:18:26.857 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:18:26.857 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:18:26.857 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:18:26.857 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 3, 2024-04-22T08:18:26.857 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:18:26.858 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:18:26.858 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:18:26.858 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:18:26.858 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:18:26.858 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:18:26.858 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:18:26.858 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:18:26.858 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:18:26.858 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:18:26.858 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:18:26.858 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:18:26.859 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:18:26.859 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:18:26.859 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:18:26.859 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:18:26.859 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:18:26.859 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:18:26.859 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:18:26.859 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:18:26.859 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:18:26.859 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:18:26.859 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:18:26.859 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:18:26.859 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:18:26.860 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:18:26.860 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:26.860 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:26.860 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:18:26.860 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:18:26.860 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:18:26.860 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:18:26.860 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:18:26.860 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:18:26.860 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:18:26.860 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:18:26.861 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:18:26.861 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:18:26.861 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:18:26.861 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:18:26.861 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:18:26.861 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:18:26.861 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:18:26.861 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:18:26.861 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:18:26.861 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:18:26.861 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:18:26.861 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:18:26.862 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:18:26.862 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:18:26.862 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:18:26.862 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:18:26.862 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:18:26.862 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:18:26.862 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:18:26.862 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:18:26.862 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:18:26.862 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:26.862 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:18:26.862 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:26.863 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:18:26.863 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:18:26.863 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:18:26.863 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:18:26.863 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:26.863 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:18:26.863 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:18:26.863 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:26.863 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:26.863 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:26.863 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:18:26.863 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:18:26.863 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:26.864 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:26.864 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:26.864 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:26.864 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:18:26.864 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:18:26.864 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:18:26.864 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:18:26.864 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:18:26.864 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:26.864 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:26.864 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:18:26.864 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:18:26.865 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:18:26.865 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:18:26.865 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:26.865 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:18:26.865 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:18:26.865 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:26.865 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:26.865 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:26.865 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:18:26.865 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:18:26.865 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:26.865 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:26.865 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:26.866 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:26.866 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:18:26.866 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:18:26.866 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:18:26.866 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:18:26.866 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:18:26.866 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:26.866 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:26.866 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:18:26.866 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:18:26.866 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:18:26.866 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:18:26.867 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:26.867 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:18:26.867 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:18:26.867 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:26.867 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:26.867 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:26.867 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:18:26.867 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:18:26.867 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:26.867 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:26.867 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:26.867 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:26.867 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:18:26.868 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:18:26.868 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:18:26.868 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:18:26.868 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:18:26.868 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:26.868 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:26.868 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:26.868 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:18:26.868 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:18:26.868 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:26.868 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:18:26.869 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:18:26.869 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:18:26.869 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:26.869 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:18:26.869 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:18:26.869 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:26.869 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:18:26.869 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:18:26.869 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 3 2024-04-22T08:18:26.869 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:26.869 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:18:26.869 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:18:26.870 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:26.870 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:18:26.870 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:18:26.870 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 4 2024-04-22T08:18:26.870 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:26.870 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:26.870 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:26.870 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:18:26.870 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:18:26.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.840+0000 7f0e626a1640 1 -- 172.21.15.40:0/3762538217 >> v2:172.21.15.189:6800/30168 conn(0x7f0e480377d0 msgr2=0x7f0e48039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:26.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.840+0000 7f0e626a1640 1 --2- 172.21.15.40:0/3762538217 >> v2:172.21.15.189:6800/30168 conn(0x7f0e480377d0 0x7f0e48039c90 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c073f30 tx=0x7f0e54030000 comp rx=0 tx=0).stop 2024-04-22T08:18:26.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.840+0000 7f0e626a1640 10 monclient: shutdown 2024-04-22T08:18:26.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.840+0000 7f0e626a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:18:26.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.840+0000 7f0e626a1640 1 -- 172.21.15.40:0/3762538217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e5c0736d0 msgr2=0x7f0e5c1cb070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:26.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.840+0000 7f0e60c3e640 1 -- 172.21.15.40:0/3762538217 reap_dead start 2024-04-22T08:18:26.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.840+0000 7f0e626a1640 1 --2- 172.21.15.40:0/3762538217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e5c0736d0 0x7f0e5c1cb070 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0e5000e870 tx=0x7f0e5000ed30 comp rx=0 tx=0).stop 2024-04-22T08:18:26.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.840+0000 7f0e58ff9640 1 -- 172.21.15.40:0/3762538217 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 3506+0+0 (unknown 0 0 0) 0x7f0e5004a410 con 0x7f0e5c0736d0 2024-04-22T08:18:26.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.840+0000 7f0e626a1640 1 -- 172.21.15.40:0/3762538217 shutdown_connections 2024-04-22T08:18:26.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.840+0000 7f0e626a1640 1 --2- 172.21.15.40:0/3762538217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e5c0736d0 0x7f0e5c1cb070 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:26.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.840+0000 7f0e626a1640 1 -- 172.21.15.40:0/3762538217 >> 172.21.15.40:0/3762538217 conn(0x7f0e5c0a1780 msgr2=0x7f0e5c132f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:18:26.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.840+0000 7f0e626a1640 1 -- 172.21.15.40:0/3762538217 shutdown_connections 2024-04-22T08:18:26.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.840+0000 7f0e626a1640 1 -- 172.21.15.40:0/3762538217 wait complete. 2024-04-22T08:18:26.873 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:18:26.873 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-22T08:18:26.873 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-22T08:18:26.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.978+0000 7f173b147640 10 monclient: get_monmap_and_config 2024-04-22T08:18:26.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.978+0000 7f173b147640 10 monclient: build_initial_monmap 2024-04-22T08:18:26.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.978+0000 7f173b147640 10 monclient: monmap: 2024-04-22T08:18:26.978 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:18:26.978 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:26.978 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:18:26.978933+0000 2024-04-22T08:18:26.978 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:18:26.978933+0000 2024-04-22T08:18:26.978 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:18:26.978 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:26.978 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:18:26.978 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:18:26.978 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:18:26.978 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:18:26.978 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:18:26.978 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:26.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.978+0000 7f173b147640 1 Processor -- start 2024-04-22T08:18:26.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.978+0000 7f173b147640 1 -- start start 2024-04-22T08:18:26.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.978+0000 7f173b147640 10 monclient: init 2024-04-22T08:18:26.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.979+0000 7f173b147640 10 monclient: _reopen_session rank -1 2024-04-22T08:18:26.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.979+0000 7f173b147640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-04-22T08:18:26.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.979+0000 7f173b147640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f17340a1c60 0x7f17340a31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:26.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.979+0000 7f173b147640 10 monclient(hunting): picked mon.noname-a con 0x7f17340a1c60 addr v2:172.21.15.40:3300/0 2024-04-22T08:18:26.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.979+0000 7f173b147640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f17340a2040 addr v1:172.21.15.40:6789/0 2024-04-22T08:18:26.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.979+0000 7f173b147640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f17340a24d0 addr v1:172.21.15.189:6789/0 2024-04-22T08:18:26.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.979+0000 7f173b147640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17340a41f0 con 0x7f17340a2040 2024-04-22T08:18:26.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.979+0000 7f173b147640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f173409fe90 con 0x7f17340a24d0 2024-04-22T08:18:26.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.979+0000 7f173b147640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:26.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.979+0000 7f173b147640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17341369d0 con 0x7f17340a1c60 2024-04-22T08:18:26.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.979+0000 7f173b147640 10 monclient(hunting): _renew_subs 2024-04-22T08:18:26.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.979+0000 7f173b147640 10 monclient(hunting): authenticate will time out at 1180.973132s 2024-04-22T08:18:26.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.979+0000 7f1738ee3640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f17340a1c60 0x7f17340a31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:26.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.979+0000 7f1738ee3640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f17340a1c60 0x7f17340a31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44716/0 (socket says 172.21.15.40:44716) 2024-04-22T08:18:26.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.979+0000 7f1738ee3640 1 -- 172.21.15.40:0/2004677605 learned_addr learned my addr 172.21.15.40:0/2004677605 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:18:26.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.979+0000 7f1733fff640 1 --1- 172.21.15.40:0/2004677605 >> v1:172.21.15.40:6789/0 conn(0x7f17340a2040 0x7f1734135bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:36414/0 (socket says 172.21.15.40:36414) 2024-04-22T08:18:26.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.979+0000 7f1738ee3640 10 monclient(hunting): get_auth_request con 0x7f17340a1c60 auth_method 0 2024-04-22T08:18:26.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.979+0000 7f1738ee3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:26.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.979+0000 7f1738ee3640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:26.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.979+0000 7f1738ee3640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:26.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.979+0000 7f1738ee3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:26.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.979+0000 7f1738ee3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:26.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.979+0000 7f1738ee3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:26.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.979+0000 7f1738ee3640 10 monclient(hunting): handle_auth_done global_id 74151 payload 274 2024-04-22T08:18:26.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.979+0000 7f1738ee3640 1 -- 172.21.15.40:0/2004677605 >> v1:172.21.15.189:6789/0 conn(0x7f17340a24d0 legacy=0x7f17341362c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:18:26.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.979+0000 7f1738ee3640 1 -- 172.21.15.40:0/2004677605 >> v1:172.21.15.40:6789/0 conn(0x7f17340a2040 legacy=0x7f1734135bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:18:26.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.979+0000 7f1738ee3640 10 monclient: _finish_hunting 0 2024-04-22T08:18:26.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.979+0000 7f1738ee3640 1 monclient: found mon.noname-a 2024-04-22T08:18:26.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.979+0000 7f1738ee3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:18:26.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.979+0000 7f1738ee3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:18:26.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.979+0000 7f1738ee3640 1 -- 172.21.15.40:0/2004677605 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17340a0170 con 0x7f17340a1c60 2024-04-22T08:18:26.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.982+0000 7f1738ee3640 1 -- 172.21.15.40:0/2004677605 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17340a1c60 msgr2=0x7f17340a31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:18:26.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.982+0000 7f1738ee3640 1 --2- 172.21.15.40:0/2004677605 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17340a1c60 0x7f17340a31d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1724009870 tx=0x7f172402b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a68a197d4ef90d26 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:26.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.982+0000 7f17337fe640 1 -- 172.21.15.40:0/2004677605 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1724031a40 con 0x7f17340a1c60 2024-04-22T08:18:26.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.982+0000 7f17337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:26.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.982+0000 7f17337fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:18:26.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.982+0000 7f17337fe640 10 monclient: dump: 2024-04-22T08:18:26.986 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:26.987 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:26.987 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:26.987 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:26.987 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:26.987 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:26.987 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:26.987 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:26.987 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:26.987 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:26.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.982+0000 7f17337fe640 10 monclient: _finish_auth 0 2024-04-22T08:18:26.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.982+0000 7f17337fe640 10 monclient: _check_auth_tickets 2024-04-22T08:18:26.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.982+0000 7f17337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:18:26.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.982+0000 7f17337fe640 1 -- 172.21.15.40:0/2004677605 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1724031ba0 con 0x7f17340a1c60 2024-04-22T08:18:26.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.982+0000 7f17337fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:18:26.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.982+0000 7f17337fe640 1 -- 172.21.15.40:0/2004677605 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1724033440 con 0x7f17340a1c60 2024-04-22T08:18:26.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.982+0000 7f173b147640 5 monclient: authenticate success, global_id 74151 2024-04-22T08:18:26.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.982+0000 7f173b147640 10 monclient: get_monmap_and_config success 2024-04-22T08:18:26.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.982+0000 7f173b147640 4 set_mon_vals no callback set 2024-04-22T08:18:26.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.982+0000 7f17337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:26.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.983+0000 7f17337fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:18:26.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.983+0000 7f17337fe640 10 monclient: dump: 2024-04-22T08:18:26.988 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:26.988 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:26.988 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:26.988 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:26.988 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:26.988 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:26.988 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:26.989 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:26.989 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:26.989 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:26.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.983+0000 7f173b147640 10 monclient: shutdown 2024-04-22T08:18:26.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.983+0000 7f173b147640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:18:26.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.983+0000 7f173b147640 1 -- 172.21.15.40:0/2004677605 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17340a1c60 msgr2=0x7f17340a31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:26.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.983+0000 7f173b147640 1 --2- 172.21.15.40:0/2004677605 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17340a1c60 0x7f17340a31d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1724009870 tx=0x7f172402b7f0 comp rx=0 tx=0).stop 2024-04-22T08:18:26.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.983+0000 7f173b147640 1 -- 172.21.15.40:0/2004677605 shutdown_connections 2024-04-22T08:18:26.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.983+0000 7f173b147640 1 --2- 172.21.15.40:0/2004677605 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17340a1c60 0x7f17340a31d0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:26.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.983+0000 7f173b147640 1 -- 172.21.15.40:0/2004677605 >> 172.21.15.40:0/2004677605 conn(0x7f17340a89a0 msgr2=0x7f17340a4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:18:26.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.983+0000 7f173b147640 1 -- 172.21.15.40:0/2004677605 shutdown_connections 2024-04-22T08:18:26.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.983+0000 7f173b147640 1 -- 172.21.15.40:0/2004677605 wait complete. 2024-04-22T08:18:26.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.984+0000 7f173b147640 10 monclient: build_initial_monmap 2024-04-22T08:18:26.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.984+0000 7f173b147640 10 monclient: monmap: 2024-04-22T08:18:26.989 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:18:26.990 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:18:26.990 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:18:26.990 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:18:26.990 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:18:26.990 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:26.990 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:18:26.990 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:18:26.990 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:18:26.990 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:26.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.984+0000 7f173b147640 1 Processor -- start 2024-04-22T08:18:26.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.984+0000 7f173b147640 1 -- start start 2024-04-22T08:18:26.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.984+0000 7f173b147640 10 monclient: init 2024-04-22T08:18:26.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.984+0000 7f173b147640 10 monclient: _reopen_session rank -1 2024-04-22T08:18:26.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.984+0000 7f173b147640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-22T08:18:26.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.984+0000 7f173b147640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17340a1c60 0x7f17341cace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:26.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.984+0000 7f173b147640 10 monclient(hunting): picked mon.noname-a con 0x7f17340a1c60 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:18:26.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.984+0000 7f173b147640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f17340a2040 0x7f17341cb220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:26.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.984+0000 7f173b147640 10 monclient(hunting): picked mon.noname-c con 0x7f17340a2040 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:18:26.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.984+0000 7f173b147640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17340a24d0 0x7f173411f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:26.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.984+0000 7f173b147640 10 monclient(hunting): picked mon.noname-b con 0x7f17340a24d0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:18:26.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.984+0000 7f173b147640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:26.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.984+0000 7f173b147640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1734122d40 con 0x7f17340a1c60 2024-04-22T08:18:26.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.985+0000 7f173b147640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:26.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.985+0000 7f173b147640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1734122f90 con 0x7f17340a24d0 2024-04-22T08:18:26.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.985+0000 7f173b147640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:26.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.985+0000 7f173b147640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1734121420 con 0x7f17340a2040 2024-04-22T08:18:26.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.985+0000 7f173b147640 10 monclient(hunting): _renew_subs 2024-04-22T08:18:26.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.985+0000 7f173b147640 10 monclient(hunting): authenticate will time out at 1180.978790s 2024-04-22T08:18:26.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.985+0000 7f1738ee3640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17340a1c60 0x7f17341cace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:26.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.985+0000 7f1733fff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f17340a2040 0x7f17341cb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:26.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.985+0000 7f1738ee3640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17340a1c60 0x7f17341cace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44728/0 (socket says 172.21.15.40:44728) 2024-04-22T08:18:26.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.985+0000 7f1738ee3640 1 -- 172.21.15.40:0/240479322 learned_addr learned my addr 172.21.15.40:0/240479322 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:18:26.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.985+0000 7f1738ee3640 10 monclient(hunting): get_auth_request con 0x7f17340a1c60 auth_method 0 2024-04-22T08:18:26.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.985+0000 7f1738ee3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:26.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.985+0000 7f1738ee3640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:26.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.985+0000 7f1738ee3640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:26.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.985+0000 7f1733fff640 10 monclient(hunting): get_auth_request con 0x7f17340a2040 auth_method 0 2024-04-22T08:18:26.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.985+0000 7f1733fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:26.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.985+0000 7f1733fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:26.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.985+0000 7f1733fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:26.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.985+0000 7f1738ee3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:26.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.985+0000 7f1738ee3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:26.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.985+0000 7f1738ee3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:26.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.985+0000 7f1733fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:26.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.985+0000 7f1733fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:26.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.985+0000 7f1733fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:26.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.985+0000 7f17396e4640 1 --2- 172.21.15.40:0/240479322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17340a24d0 0x7f173411f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:26.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.985+0000 7f17396e4640 10 monclient(hunting): get_auth_request con 0x7f17340a24d0 auth_method 0 2024-04-22T08:18:26.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.985+0000 7f17396e4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:26.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.985+0000 7f17396e4640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:26.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.985+0000 7f17396e4640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:26.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.985+0000 7f1738ee3640 10 monclient(hunting): handle_auth_done global_id 74154 payload 824 2024-04-22T08:18:26.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.985+0000 7f1738ee3640 1 -- 172.21.15.40:0/240479322 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f17340a2040 msgr2=0x7f17341cb220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:26.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.985+0000 7f1738ee3640 1 --2- 172.21.15.40:0/240479322 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f17340a2040 0x7f17341cb220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:26.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.985+0000 7f1738ee3640 1 -- 172.21.15.40:0/240479322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17340a24d0 msgr2=0x7f173411f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:26.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.985+0000 7f1738ee3640 1 --2- 172.21.15.40:0/240479322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17340a24d0 0x7f173411f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:26.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.985+0000 7f1738ee3640 10 monclient: _finish_hunting 0 2024-04-22T08:18:26.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.985+0000 7f1738ee3640 1 monclient: found mon.noname-a 2024-04-22T08:18:26.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.985+0000 7f1738ee3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:18:26.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.985+0000 7f1738ee3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:18:26.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.985+0000 7f1738ee3640 1 -- 172.21.15.40:0/240479322 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17240094d0 con 0x7f17340a1c60 2024-04-22T08:18:26.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.985+0000 7f17396e4640 1 --2- 172.21.15.40:0/240479322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17340a24d0 0x7f173411f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T08:18:26.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.987+0000 7f1738ee3640 1 -- 172.21.15.40:0/240479322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17340a1c60 msgr2=0x7f17341cace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:18:26.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.987+0000 7f1738ee3640 1 --2- 172.21.15.40:0/240479322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17340a1c60 0x7f17341cace0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f17240090b0 tx=0x7f17240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:26.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.987+0000 7f17317fa640 1 -- 172.21.15.40:0/240479322 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f172403c400 con 0x7f17340a1c60 2024-04-22T08:18:26.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.987+0000 7f17317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:26.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.987+0000 7f17317fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:18:26.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.987+0000 7f17317fa640 10 monclient: dump: 2024-04-22T08:18:26.998 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:26.998 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:26.999 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:26.999 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:26.999 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:26.999 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:26.999 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:26.999 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:26.999 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:26.999 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:26.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.987+0000 7f17317fa640 10 monclient: _finish_auth 0 2024-04-22T08:18:26.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.987+0000 7f17317fa640 10 monclient: _check_auth_tickets 2024-04-22T08:18:26.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.987+0000 7f17317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:18:26.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.987+0000 7f17317fa640 1 -- 172.21.15.40:0/240479322 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f172403c560 con 0x7f17340a1c60 2024-04-22T08:18:26.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.987+0000 7f173b147640 5 monclient: authenticate success, global_id 74154 2024-04-22T08:18:27.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.987+0000 7f17317fa640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:18:27.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.987+0000 7f17317fa640 1 -- 172.21.15.40:0/240479322 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f172403c8f0 con 0x7f17340a1c60 2024-04-22T08:18:27.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.987+0000 7f173b147640 10 monclient: _renew_subs 2024-04-22T08:18:27.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.987+0000 7f173b147640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:18:27.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.987+0000 7f173b147640 1 -- 172.21.15.40:0/240479322 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1734121ae0 con 0x7f17340a1c60 2024-04-22T08:18:27.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.988+0000 7f17317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:27.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.988+0000 7f17317fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:18:27.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.988+0000 7f17317fa640 10 monclient: dump: 2024-04-22T08:18:27.000 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:27.000 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:27.000 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:27.000 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:27.000 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:27.000 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:27.001 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:27.001 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:27.001 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:27.001 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:27.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.987+0000 7f1732ffd640 4 set_mon_vals no callback set 2024-04-22T08:18:27.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.988+0000 7f173b147640 10 monclient: _renew_subs 2024-04-22T08:18:27.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.988+0000 7f173b147640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:18:27.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.988+0000 7f173b147640 1 -- 172.21.15.40:0/240479322 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1734120f10 con 0x7f17340a1c60 2024-04-22T08:18:27.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.988+0000 7f173b147640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:18:27.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.988+0000 7f173b147640 1 --2- 172.21.15.40:0/240479322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1734120050 0x7f1734120430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:27.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.988+0000 7f173b147640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:27.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.988+0000 7f173b147640 1 -- 172.21.15.40:0/240479322 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1734123120 con 0x7f1734120050 2024-04-22T08:18:27.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.988+0000 7f17317fa640 1 -- 172.21.15.40:0/240479322 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f172403a040 con 0x7f17340a1c60 2024-04-22T08:18:27.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.988+0000 7f17317fa640 1 --2- 172.21.15.40:0/240479322 >> v2:172.21.15.189:6800/30168 conn(0x7f17200377d0 0x7f1720039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:27.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.989+0000 7f17317fa640 1 -- 172.21.15.40:0/240479322 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f172406c2c0 con 0x7f17340a1c60 2024-04-22T08:18:27.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.989+0000 7f1733fff640 1 --2- 172.21.15.40:0/240479322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1734120050 0x7f1734120430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:27.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.989+0000 7f1733fff640 10 monclient: get_auth_request con 0x7f1734120050 auth_method 0 2024-04-22T08:18:27.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.989+0000 7f1733fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:27.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.989+0000 7f1733fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:27.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.989+0000 7f1733fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:27.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.989+0000 7f17396e4640 1 --2- 172.21.15.40:0/240479322 >> v2:172.21.15.189:6800/30168 conn(0x7f17200377d0 0x7f1720039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:27.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.989+0000 7f17396e4640 10 monclient: get_auth_request con 0x7f17200377d0 auth_method 0 2024-04-22T08:18:27.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.989+0000 7f1733fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:27.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.989+0000 7f1733fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:27.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.989+0000 7f1733fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:27.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.989+0000 7f1733fff640 10 monclient(hunting): handle_auth_done global_id 64321 payload 824 2024-04-22T08:18:27.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.989+0000 7f1733fff640 1 -- 172.21.15.40:0/240479322 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f173409fac0 con 0x7f1734120050 2024-04-22T08:18:27.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.991+0000 7f17396e4640 1 --2- 172.21.15.40:0/240479322 >> v2:172.21.15.189:6800/30168 conn(0x7f17200377d0 0x7f1720039c90 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f172c004530 tx=0x7f172c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:27.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.991+0000 7f1733fff640 1 -- 172.21.15.40:0/240479322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1734120050 msgr2=0x7f1734120430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:18:27.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.991+0000 7f1733fff640 1 --2- 172.21.15.40:0/240479322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1734120050 0x7f1734120430 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1728007fd0 tx=0x7f1728005200 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:27.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.992+0000 7f17317fa640 1 -- 172.21.15.40:0/240479322 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17280138f0 con 0x7f1734120050 2024-04-22T08:18:27.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.992+0000 7f17317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:27.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.992+0000 7f17317fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:18:27.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.992+0000 7f17317fa640 10 monclient: dump: 2024-04-22T08:18:27.003 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:27.003 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:27.003 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:27.003 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:27.003 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:27.003 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:27.003 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:27.003 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:27.004 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:27.004 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:27.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.992+0000 7f17317fa640 1 -- 172.21.15.40:0/240479322 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f1728013a50 con 0x7f1734120050 2024-04-22T08:18:27.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.992+0000 7f17317fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:18:27.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.992+0000 7f17317fa640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:18:27.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.992+0000 7f17317fa640 1 -- 172.21.15.40:0/240479322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1734120050 msgr2=0x7f1734120430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:27.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.992+0000 7f17317fa640 1 --2- 172.21.15.40:0/240479322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1734120050 0x7f1734120430 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1728007fd0 tx=0x7f1728005200 comp rx=0 tx=0).stop 2024-04-22T08:18:27.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.992+0000 7f17317fa640 1 -- 172.21.15.40:0/240479322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1734120050 msgr2=0x7f1734120430 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:18:27.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.992+0000 7f17317fa640 1 --2- 172.21.15.40:0/240479322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1734120050 0x7f1734120430 secure :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1728007fd0 tx=0x7f1728005200 comp rx=0 tx=0).stop 2024-04-22T08:18:27.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.998+0000 7f173b147640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:18:27.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.998+0000 7f173b147640 1 --2- 172.21.15.40:0/240479322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f173402b200 0x7f173402b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:27.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.998+0000 7f173b147640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:27.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.998+0000 7f173b147640 1 -- 172.21.15.40:0/240479322 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1734066650 con 0x7f173402b200 2024-04-22T08:18:27.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.999+0000 7f1733fff640 1 --2- 172.21.15.40:0/240479322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f173402b200 0x7f173402b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:27.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.999+0000 7f1733fff640 10 monclient: get_auth_request con 0x7f173402b200 auth_method 0 2024-04-22T08:18:27.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.999+0000 7f1733fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:27.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.999+0000 7f1733fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:27.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.999+0000 7f1733fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:27.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.999+0000 7f1733fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:27.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.999+0000 7f1733fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:27.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.999+0000 7f1733fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:27.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.999+0000 7f1733fff640 10 monclient(hunting): handle_auth_done global_id 64324 payload 824 2024-04-22T08:18:27.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:26.999+0000 7f1733fff640 1 -- 172.21.15.40:0/240479322 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1734122af0 con 0x7f173402b200 2024-04-22T08:18:27.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:27.002+0000 7f1733fff640 1 -- 172.21.15.40:0/240479322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f173402b200 msgr2=0x7f173402b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:18:27.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:27.002+0000 7f1733fff640 1 --2- 172.21.15.40:0/240479322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f173402b200 0x7f173402b610 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f1728017000 tx=0x7f1728004930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:27.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:27.002+0000 7f17317fa640 1 -- 172.21.15.40:0/240479322 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1728024070 con 0x7f173402b200 2024-04-22T08:18:27.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:27.002+0000 7f17317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:27.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:27.002+0000 7f17317fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:18:27.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:27.002+0000 7f17317fa640 10 monclient: dump: 2024-04-22T08:18:27.007 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:27.007 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:27.007 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:27.007 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:27.007 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:27.007 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:27.007 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:27.007 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:27.007 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:27.008 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:27.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:27.002+0000 7f17317fa640 1 -- 172.21.15.40:0/240479322 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4801 (secure 0 0 0) 0x7f1728006680 con 0x7f173402b200 2024-04-22T08:18:27.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:27.002+0000 7f17317fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:18:27.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:27.002+0000 7f17317fa640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:18:27.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:27.002+0000 7f17317fa640 1 -- 172.21.15.40:0/240479322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f173402b200 msgr2=0x7f173402b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:27.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:27.002+0000 7f17317fa640 1 --2- 172.21.15.40:0/240479322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f173402b200 0x7f173402b610 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f1728017000 tx=0x7f1728004930 comp rx=0 tx=0).stop 2024-04-22T08:18:27.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:27.002+0000 7f17317fa640 1 -- 172.21.15.40:0/240479322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f173402b200 msgr2=0x7f173402b610 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:18:27.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:27.002+0000 7f17317fa640 1 --2- 172.21.15.40:0/240479322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f173402b200 0x7f173402b610 secure :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f1728017000 tx=0x7f1728004930 comp rx=0 tx=0).stop 2024-04-22T08:18:27.008 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:18:27.009 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:18:27.009 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:18:27.009 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:18:27.009 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 56, 2024-04-22T08:18:27.009 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:18:27.009 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:18:27.009 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:18:27.009 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:18:27.009 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:18:27.009 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 3, 2024-04-22T08:18:27.009 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:18:27.009 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:18:27.010 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:18:27.010 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:18:27.010 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:18:27.010 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:18:27.010 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:18:27.010 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:18:27.010 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:18:27.010 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:18:27.010 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:18:27.010 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:18:27.010 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:18:27.010 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:18:27.010 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:18:27.010 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:18:27.011 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:18:27.011 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:18:27.011 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:18:27.011 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:18:27.011 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:18:27.011 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:18:27.011 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:18:27.011 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:18:27.011 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:18:27.011 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:18:27.011 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:27.011 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:27.011 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:18:27.011 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:18:27.011 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:18:27.012 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:18:27.012 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:18:27.012 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:18:27.012 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:18:27.012 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:18:27.012 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:18:27.012 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:18:27.012 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:18:27.012 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:18:27.012 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:18:27.012 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:18:27.012 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:18:27.012 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:18:27.012 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:18:27.012 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:18:27.013 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:18:27.013 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:18:27.013 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:18:27.013 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:18:27.013 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:18:27.013 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:18:27.013 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:18:27.013 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:18:27.013 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:18:27.013 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:18:27.013 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:18:27.013 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:27.013 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:18:27.013 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:27.013 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:18:27.014 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:18:27.014 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:18:27.014 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:18:27.014 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:27.014 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:18:27.014 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:18:27.014 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:27.014 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:27.014 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:27.014 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:18:27.014 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:18:27.014 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:27.014 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:27.014 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:27.015 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:27.015 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:18:27.015 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:18:27.015 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:18:27.015 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:18:27.015 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:18:27.015 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:27.015 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:27.015 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:18:27.015 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:18:27.015 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:18:27.015 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:18:27.015 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:27.015 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:18:27.015 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:18:27.016 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:27.016 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:27.016 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:27.016 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:18:27.016 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:18:27.016 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:27.016 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:27.016 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:27.016 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:27.016 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:18:27.016 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:18:27.016 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:18:27.016 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:18:27.016 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:18:27.016 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:27.017 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:27.017 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:18:27.017 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:18:27.017 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:18:27.017 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:18:27.017 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:27.017 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:18:27.017 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:18:27.017 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:27.017 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:27.017 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:27.017 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:18:27.017 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:18:27.017 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:27.017 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:27.017 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:27.018 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:27.018 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:18:27.018 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:18:27.018 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:18:27.018 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:18:27.018 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:18:27.018 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:27.018 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:27.018 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:27.018 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:18:27.018 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:18:27.018 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:27.018 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:18:27.018 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:18:27.019 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:18:27.019 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:27.019 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:18:27.019 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:18:27.019 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:27.019 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:18:27.019 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:18:27.019 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 3 2024-04-22T08:18:27.019 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:27.019 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:18:27.019 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:18:27.019 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:27.019 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:18:27.019 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:18:27.020 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 4 2024-04-22T08:18:27.020 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:27.020 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:18:27.020 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": [ 2024-04-22T08:18:27.020 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:27.020 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:18:27.020 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:18:27.020 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:18:27.020 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:27.020 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:27.020 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:27.020 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:18:27.020 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:18:27.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:27.003+0000 7f173b147640 1 -- 172.21.15.40:0/240479322 >> v2:172.21.15.189:6800/30168 conn(0x7f17200377d0 msgr2=0x7f1720039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:27.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:27.003+0000 7f173b147640 1 --2- 172.21.15.40:0/240479322 >> v2:172.21.15.189:6800/30168 conn(0x7f17200377d0 0x7f1720039c90 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f172c004530 tx=0x7f172c009040 comp rx=0 tx=0).stop 2024-04-22T08:18:27.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:27.003+0000 7f173b147640 10 monclient: shutdown 2024-04-22T08:18:27.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:27.003+0000 7f173b147640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:18:27.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:27.003+0000 7f173b147640 1 -- 172.21.15.40:0/240479322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17340a1c60 msgr2=0x7f17341cace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:27.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:27.003+0000 7f173b147640 1 --2- 172.21.15.40:0/240479322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17340a1c60 0x7f17341cace0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f17240090b0 tx=0x7f17240046c0 comp rx=0 tx=0).stop 2024-04-22T08:18:27.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:27.003+0000 7f17396e4640 1 -- 172.21.15.40:0/240479322 reap_dead start 2024-04-22T08:18:27.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:27.003+0000 7f17396e4640 1 -- 172.21.15.40:0/240479322 reap_dead start 2024-04-22T08:18:27.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:27.003+0000 7f173b147640 1 -- 172.21.15.40:0/240479322 shutdown_connections 2024-04-22T08:18:27.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:27.003+0000 7f173b147640 1 -- 172.21.15.40:0/240479322 >> 172.21.15.40:0/240479322 conn(0x7f17340a89a0 msgr2=0x7f173409de90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:18:27.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:27.003+0000 7f173b147640 1 -- 172.21.15.40:0/240479322 shutdown_connections 2024-04-22T08:18:27.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:27.003+0000 7f173b147640 1 -- 172.21.15.40:0/240479322 wait complete. 2024-04-22T08:18:27.023 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:18:27.023 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-04-22T08:18:27.023 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2024-04-22T08:18:27.023 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-04-22T08:18:27.023 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-04-22T08:18:27.023 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-22T08:18:27.023 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T08:18:27.033 INFO:tasks.ceph.mon.a:Stopped 2024-04-22T08:18:27.033 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-04-22T08:18:27.033 INFO:tasks.ceph.mon.c:Sent signal 19 2024-04-22T08:18:27.033 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-04-22T08:18:42.034 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-04-22T08:18:42.035 INFO:tasks.ceph.mon.c:Sent signal 18 2024-04-22T08:18:42.035 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-22T08:18:42.035 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:18:42.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.145+0000 7fd6aa677640 10 monclient: get_monmap_and_config 2024-04-22T08:18:42.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.145+0000 7fd6aa677640 10 monclient: build_initial_monmap 2024-04-22T08:18:42.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.145+0000 7fd6aa677640 10 monclient: monmap: 2024-04-22T08:18:42.145 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:18:42.145 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:42.146 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:18:42.146847+0000 2024-04-22T08:18:42.146 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:18:42.146847+0000 2024-04-22T08:18:42.146 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:18:42.146 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:42.146 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:18:42.146 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:18:42.146 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:18:42.146 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:18:42.146 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:18:42.146 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:42.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.145+0000 7fd6aa677640 1 Processor -- start 2024-04-22T08:18:42.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.145+0000 7fd6aa677640 1 -- start start 2024-04-22T08:18:42.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.145+0000 7fd6aa677640 10 monclient: init 2024-04-22T08:18:42.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.146+0000 7fd6aa677640 10 monclient: _reopen_session rank -1 2024-04-22T08:18:42.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.146+0000 7fd6aa677640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-04-22T08:18:42.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.146+0000 7fd6aa677640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6a409ee10 0x7fd6a40a32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:42.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.146+0000 7fd6aa677640 10 monclient(hunting): picked mon.noname-b con 0x7fd6a409ee10 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:18:42.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.146+0000 7fd6aa677640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd6a40a1d30 addr v1:172.21.15.189:6789/0 2024-04-22T08:18:42.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.146+0000 7fd6aa677640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd6a40a2150 addr v1:172.21.15.40:6789/0 2024-04-22T08:18:42.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.146+0000 7fd6aa677640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6a40a2580 con 0x7fd6a40a2150 2024-04-22T08:18:42.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.146+0000 7fd6aa677640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6a409f1f0 con 0x7fd6a40a1d30 2024-04-22T08:18:42.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.146+0000 7fd6aa677640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:42.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.146+0000 7fd6aa677640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a4136bb0 con 0x7fd6a409ee10 2024-04-22T08:18:42.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.146+0000 7fd6aa677640 10 monclient(hunting): _renew_subs 2024-04-22T08:18:42.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.146+0000 7fd6aa677640 10 monclient(hunting): authenticate will time out at 1196.141153s 2024-04-22T08:18:42.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.146+0000 7fd6a8c14640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7fd6a40a2150 legacy=0x7fd6a41364a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-04-22T08:18:42.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.146+0000 7fd6a3fff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6a409ee10 0x7fd6a40a32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:42.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.146+0000 7fd6a3fff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6a409ee10 0x7fd6a40a32b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:51188/0 (socket says 172.21.15.40:51188) 2024-04-22T08:18:42.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.146+0000 7fd6a3fff640 1 -- 172.21.15.40:0/2217647052 learned_addr learned my addr 172.21.15.40:0/2217647052 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:18:42.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.146+0000 7fd6a3fff640 10 monclient(hunting): get_auth_request con 0x7fd6a409ee10 auth_method 0 2024-04-22T08:18:42.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.146+0000 7fd6a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:42.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.146+0000 7fd6a3fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:42.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.146+0000 7fd6a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:42.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.146+0000 7fd6a3fff640 1 -- 172.21.15.40:0/2217647052 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6a409ee10 msgr2=0x7fd6a40a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:18:42.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.146+0000 7fd6a3fff640 1 -- 172.21.15.40:0/2217647052 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6a409ee10 msgr2=0x7fd6a40a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:18:42.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.146+0000 7fd6a3fff640 1 --2- 172.21.15.40:0/2217647052 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6a409ee10 0x7fd6a40a32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:18:42.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.146+0000 7fd6a3fff640 1 --2- 172.21.15.40:0/2217647052 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6a409ee10 0x7fd6a40a32b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:18:42.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.148+0000 7fd6a37fe640 1 -- 172.21.15.40:0/2217647052 >> v1:172.21.15.189:6789/0 conn(0x7fd6a40a1d30 legacy=0x7fd6a4135d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:18:42.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.347+0000 7fd6a8c14640 1 -- 172.21.15.40:0/2217647052 >> v1:172.21.15.40:6789/0 conn(0x7fd6a40a2150 legacy=0x7fd6a41364a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-04-22T08:18:42.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.347+0000 7fd6a3fff640 1 --2- 172.21.15.40:0/2217647052 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6a409ee10 0x7fd6a40a32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:42.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.347+0000 7fd6a3fff640 10 monclient(hunting): get_auth_request con 0x7fd6a409ee10 auth_method 0 2024-04-22T08:18:42.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.347+0000 7fd6a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:42.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.347+0000 7fd6a3fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:42.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.347+0000 7fd6a3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:18:42.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.347+0000 7fd6a3fff640 1 -- 172.21.15.40:0/2217647052 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6a409ee10 msgr2=0x7fd6a40a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:18:42.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.347+0000 7fd6a3fff640 1 -- 172.21.15.40:0/2217647052 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6a409ee10 msgr2=0x7fd6a40a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:18:42.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.347+0000 7fd6a3fff640 1 --2- 172.21.15.40:0/2217647052 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6a409ee10 0x7fd6a40a32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:18:42.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.347+0000 7fd6a3fff640 1 --2- 172.21.15.40:0/2217647052 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6a409ee10 0x7fd6a40a32b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:18:42.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.748+0000 7fd6a8c14640 1 -- 172.21.15.40:0/2217647052 >> v1:172.21.15.40:6789/0 conn(0x7fd6a40a2150 legacy=0x7fd6a41364a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-04-22T08:18:42.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.748+0000 7fd6a3fff640 1 --2- 172.21.15.40:0/2217647052 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6a409ee10 0x7fd6a40a32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:42.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.748+0000 7fd6a3fff640 10 monclient(hunting): get_auth_request con 0x7fd6a409ee10 auth_method 0 2024-04-22T08:18:42.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.748+0000 7fd6a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:42.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.748+0000 7fd6a3fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:42.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.748+0000 7fd6a3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:18:42.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.748+0000 7fd6a3fff640 1 -- 172.21.15.40:0/2217647052 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6a409ee10 msgr2=0x7fd6a40a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:18:42.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.748+0000 7fd6a3fff640 1 -- 172.21.15.40:0/2217647052 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6a409ee10 msgr2=0x7fd6a40a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:18:42.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.748+0000 7fd6a3fff640 1 --2- 172.21.15.40:0/2217647052 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6a409ee10 0x7fd6a40a32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:18:42.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:42.748+0000 7fd6a3fff640 1 --2- 172.21.15.40:0/2217647052 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6a409ee10 0x7fd6a40a32b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:18:43.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:43.549+0000 7fd6a8c14640 1 -- 172.21.15.40:0/2217647052 >> v1:172.21.15.40:6789/0 conn(0x7fd6a40a2150 legacy=0x7fd6a41364a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-04-22T08:18:43.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:43.549+0000 7fd6a3fff640 1 --2- 172.21.15.40:0/2217647052 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6a409ee10 0x7fd6a40a32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:43.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:43.549+0000 7fd6a3fff640 10 monclient(hunting): get_auth_request con 0x7fd6a409ee10 auth_method 0 2024-04-22T08:18:43.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:43.549+0000 7fd6a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:43.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:43.549+0000 7fd6a3fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:43.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:43.549+0000 7fd6a3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:18:43.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:43.549+0000 7fd6a3fff640 1 -- 172.21.15.40:0/2217647052 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6a409ee10 msgr2=0x7fd6a40a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:18:43.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:43.549+0000 7fd6a3fff640 1 -- 172.21.15.40:0/2217647052 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6a409ee10 msgr2=0x7fd6a40a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:18:43.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:43.549+0000 7fd6a3fff640 1 --2- 172.21.15.40:0/2217647052 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6a409ee10 0x7fd6a40a32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:18:43.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:43.549+0000 7fd6a3fff640 1 --2- 172.21.15.40:0/2217647052 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6a409ee10 0x7fd6a40a32b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T08:18:45.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.146+0000 7fd6a1ffb640 10 monclient(hunting): tick 2024-04-22T08:18:45.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.146+0000 7fd6a1ffb640 10 monclient(hunting): _check_auth_tickets 2024-04-22T08:18:45.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.146+0000 7fd6a1ffb640 1 monclient(hunting): continuing hunt 2024-04-22T08:18:45.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.146+0000 7fd6a1ffb640 10 monclient(hunting): _reopen_session rank -1 2024-04-22T08:18:45.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.146+0000 7fd6a1ffb640 1 -- 172.21.15.40:0/2217647052 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6a409ee10 msgr2=0x7fd6a40a32b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:18:45.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.146+0000 7fd6a1ffb640 1 --2- 172.21.15.40:0/2217647052 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6a409ee10 0x7fd6a40a32b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:45.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.146+0000 7fd6a1ffb640 1 -- 172.21.15.40:0/2217647052 >> v1:172.21.15.189:6789/0 conn(0x7fd6a40a1d30 legacy=0x7fd6a4135d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:45.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.146+0000 7fd6a1ffb640 1 -- 172.21.15.40:0/2217647052 >> v1:172.21.15.40:6789/0 conn(0x7fd6a40a2150 legacy=0x7fd6a41364a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:18:45.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.146+0000 7fd6a1ffb640 10 monclient: _add_conns ranks=[4,3] 2024-04-22T08:18:45.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.146+0000 7fd6a1ffb640 1 --2- 172.21.15.40:0/2217647052 >> v2:172.21.15.189:3300/0 conn(0x7fd690003770 0x7fd690005bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:45.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.146+0000 7fd6a1ffb640 10 monclient(hunting): picked mon.noname-c con 0x7fd690003770 addr v2:172.21.15.189:3300/0 2024-04-22T08:18:45.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.146+0000 7fd6a1ffb640 1 --2- 172.21.15.40:0/2217647052 >> v2:172.21.15.40:3300/0 conn(0x7fd6900062e0 0x7fd690008780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:45.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.146+0000 7fd6a1ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fd6900062e0 addr v2:172.21.15.40:3300/0 2024-04-22T08:18:45.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.146+0000 7fd6a1ffb640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:45.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.146+0000 7fd6a1ffb640 1 -- 172.21.15.40:0/2217647052 --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6a4136bb0 con 0x7fd6900062e0 2024-04-22T08:18:45.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.146+0000 7fd6a1ffb640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:45.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.146+0000 7fd6a1ffb640 1 -- 172.21.15.40:0/2217647052 --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd690008e10 con 0x7fd690003770 2024-04-22T08:18:45.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.146+0000 7fd6a1ffb640 10 monclient(hunting): _renew_subs 2024-04-22T08:18:45.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.146+0000 7fd6a37fe640 1 -- 172.21.15.40:0/2217647052 >> v2:172.21.15.40:3300/0 conn(0x7fd6900062e0 msgr2=0x7fd690008780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:18:45.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.146+0000 7fd6a37fe640 1 --2- 172.21.15.40:0/2217647052 >> v2:172.21.15.40:3300/0 conn(0x7fd6900062e0 0x7fd690008780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:18:45.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.146+0000 7fd6a3fff640 1 --2- 172.21.15.40:0/2217647052 >> v2:172.21.15.189:3300/0 conn(0x7fd690003770 0x7fd690005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:45.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.146+0000 7fd6a3fff640 10 monclient(hunting): get_auth_request con 0x7fd690003770 auth_method 0 2024-04-22T08:18:45.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.146+0000 7fd6a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:45.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.146+0000 7fd6a3fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:45.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.146+0000 7fd6a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:45.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.146+0000 7fd6a3fff640 1 -- 172.21.15.40:0/2217647052 >> v2:172.21.15.189:3300/0 conn(0x7fd690003770 msgr2=0x7fd690005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:18:45.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.146+0000 7fd6a3fff640 1 -- 172.21.15.40:0/2217647052 >> v2:172.21.15.189:3300/0 conn(0x7fd690003770 msgr2=0x7fd690005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:18:45.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.146+0000 7fd6a3fff640 1 --2- 172.21.15.40:0/2217647052 >> v2:172.21.15.189:3300/0 conn(0x7fd690003770 0x7fd690005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:18:45.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.146+0000 7fd6a3fff640 1 --2- 172.21.15.40:0/2217647052 >> v2:172.21.15.189:3300/0 conn(0x7fd690003770 0x7fd690005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:18:45.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.347+0000 7fd6a37fe640 1 -- 172.21.15.40:0/2217647052 >> v2:172.21.15.40:3300/0 conn(0x7fd6900062e0 msgr2=0x7fd690008780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:18:45.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.347+0000 7fd6a37fe640 1 --2- 172.21.15.40:0/2217647052 >> v2:172.21.15.40:3300/0 conn(0x7fd6900062e0 0x7fd690008780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:18:45.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.347+0000 7fd6a3fff640 1 --2- 172.21.15.40:0/2217647052 >> v2:172.21.15.189:3300/0 conn(0x7fd690003770 0x7fd690005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:45.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.347+0000 7fd6a3fff640 10 monclient(hunting): get_auth_request con 0x7fd690003770 auth_method 0 2024-04-22T08:18:45.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.347+0000 7fd6a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:45.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.347+0000 7fd6a3fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:45.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.347+0000 7fd6a3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:18:45.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.347+0000 7fd6a3fff640 1 -- 172.21.15.40:0/2217647052 >> v2:172.21.15.189:3300/0 conn(0x7fd690003770 msgr2=0x7fd690005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:18:45.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.347+0000 7fd6a3fff640 1 -- 172.21.15.40:0/2217647052 >> v2:172.21.15.189:3300/0 conn(0x7fd690003770 msgr2=0x7fd690005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:18:45.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.347+0000 7fd6a3fff640 1 --2- 172.21.15.40:0/2217647052 >> v2:172.21.15.189:3300/0 conn(0x7fd690003770 0x7fd690005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:18:45.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.347+0000 7fd6a3fff640 1 --2- 172.21.15.40:0/2217647052 >> v2:172.21.15.189:3300/0 conn(0x7fd690003770 0x7fd690005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:18:45.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.748+0000 7fd6a37fe640 1 -- 172.21.15.40:0/2217647052 >> v2:172.21.15.40:3300/0 conn(0x7fd6900062e0 msgr2=0x7fd690008780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:18:45.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.748+0000 7fd6a37fe640 1 --2- 172.21.15.40:0/2217647052 >> v2:172.21.15.40:3300/0 conn(0x7fd6900062e0 0x7fd690008780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:18:45.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.748+0000 7fd6a3fff640 1 --2- 172.21.15.40:0/2217647052 >> v2:172.21.15.189:3300/0 conn(0x7fd690003770 0x7fd690005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:45.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.748+0000 7fd6a3fff640 10 monclient(hunting): get_auth_request con 0x7fd690003770 auth_method 0 2024-04-22T08:18:45.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.748+0000 7fd6a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:45.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.748+0000 7fd6a3fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:45.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.748+0000 7fd6a3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:18:45.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.748+0000 7fd6a3fff640 1 -- 172.21.15.40:0/2217647052 >> v2:172.21.15.189:3300/0 conn(0x7fd690003770 msgr2=0x7fd690005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:18:45.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.748+0000 7fd6a3fff640 1 -- 172.21.15.40:0/2217647052 >> v2:172.21.15.189:3300/0 conn(0x7fd690003770 msgr2=0x7fd690005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:18:45.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.748+0000 7fd6a3fff640 1 --2- 172.21.15.40:0/2217647052 >> v2:172.21.15.189:3300/0 conn(0x7fd690003770 0x7fd690005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:18:45.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:45.748+0000 7fd6a3fff640 1 --2- 172.21.15.40:0/2217647052 >> v2:172.21.15.189:3300/0 conn(0x7fd690003770 0x7fd690005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:18:46.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:46.549+0000 7fd6a37fe640 1 -- 172.21.15.40:0/2217647052 >> v2:172.21.15.40:3300/0 conn(0x7fd6900062e0 msgr2=0x7fd690008780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:18:46.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:46.549+0000 7fd6a37fe640 1 --2- 172.21.15.40:0/2217647052 >> v2:172.21.15.40:3300/0 conn(0x7fd6900062e0 0x7fd690008780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T08:18:46.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:46.549+0000 7fd6a3fff640 1 --2- 172.21.15.40:0/2217647052 >> v2:172.21.15.189:3300/0 conn(0x7fd690003770 0x7fd690005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:46.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:46.549+0000 7fd6a3fff640 10 monclient(hunting): get_auth_request con 0x7fd690003770 auth_method 0 2024-04-22T08:18:46.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:46.549+0000 7fd6a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:46.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:46.549+0000 7fd6a3fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:46.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:46.549+0000 7fd6a3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:18:46.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:46.549+0000 7fd6a3fff640 1 -- 172.21.15.40:0/2217647052 >> v2:172.21.15.189:3300/0 conn(0x7fd690003770 msgr2=0x7fd690005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:18:46.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:46.549+0000 7fd6a3fff640 1 -- 172.21.15.40:0/2217647052 >> v2:172.21.15.189:3300/0 conn(0x7fd690003770 msgr2=0x7fd690005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:18:46.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:46.549+0000 7fd6a3fff640 1 --2- 172.21.15.40:0/2217647052 >> v2:172.21.15.189:3300/0 conn(0x7fd690003770 0x7fd690005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:18:46.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:46.549+0000 7fd6a3fff640 1 --2- 172.21.15.40:0/2217647052 >> v2:172.21.15.189:3300/0 conn(0x7fd690003770 0x7fd690005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T08:18:47.948 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37415]: starting. 2024-04-22T08:18:47.948 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37415]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-37414 2024-04-22T08:18:47.949 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37415]: created object 0... 2024-04-22T08:18:47.949 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37415]: created object 25... 2024-04-22T08:18:47.949 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37415]: created object 49... 2024-04-22T08:18:47.949 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37415]: finishing. 2024-04-22T08:18:47.949 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37415]: shutting down. 2024-04-22T08:18:48.106 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[37416]: starting. 2024-04-22T08:18:48.106 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[37416]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-37414 2024-04-22T08:18:48.106 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[37416]: shutting down. 2024-04-22T08:18:48.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.146+0000 7fd6a1ffb640 10 monclient(hunting): tick 2024-04-22T08:18:48.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.146+0000 7fd6a1ffb640 10 monclient(hunting): _check_auth_tickets 2024-04-22T08:18:48.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.146+0000 7fd6a1ffb640 1 monclient(hunting): continuing hunt 2024-04-22T08:18:48.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.146+0000 7fd6a1ffb640 10 monclient(hunting): _reopen_session rank -1 2024-04-22T08:18:48.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.146+0000 7fd6a1ffb640 1 -- 172.21.15.40:0/2217647052 >> v2:172.21.15.189:3300/0 conn(0x7fd690003770 msgr2=0x7fd690005bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:18:48.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.146+0000 7fd6a1ffb640 1 --2- 172.21.15.40:0/2217647052 >> v2:172.21.15.189:3300/0 conn(0x7fd690003770 0x7fd690005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:48.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.146+0000 7fd6a1ffb640 1 -- 172.21.15.40:0/2217647052 >> v2:172.21.15.40:3300/0 conn(0x7fd6900062e0 msgr2=0x7fd690008780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:18:48.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.146+0000 7fd6a1ffb640 1 --2- 172.21.15.40:0/2217647052 >> v2:172.21.15.40:3300/0 conn(0x7fd6900062e0 0x7fd690008780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:48.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.146+0000 7fd6a1ffb640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-04-22T08:18:48.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.146+0000 7fd6a1ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd6a40a1d30 addr v1:172.21.15.189:6789/0 2024-04-22T08:18:48.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.146+0000 7fd6a8c14640 1 -- 172.21.15.40:0/2217647052 reap_dead start 2024-04-22T08:18:48.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.146+0000 7fd6a1ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd6a40a2150 addr v1:172.21.15.40:6789/0 2024-04-22T08:18:48.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.146+0000 7fd6a1ffb640 1 --2- 172.21.15.40:0/2217647052 >> v2:172.21.15.189:3300/0 conn(0x7fd690003770 0x7fd690005bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:48.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.146+0000 7fd6a1ffb640 10 monclient(hunting): picked mon.noname-c con 0x7fd690003770 addr v2:172.21.15.189:3300/0 2024-04-22T08:18:48.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.146+0000 7fd6a1ffb640 1 -- 172.21.15.40:0/2217647052 --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6a40a39c0 con 0x7fd6a40a2150 2024-04-22T08:18:48.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.146+0000 7fd6a1ffb640 1 -- 172.21.15.40:0/2217647052 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6a40a2580 con 0x7fd6a40a1d30 2024-04-22T08:18:48.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.146+0000 7fd6a1ffb640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:48.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.146+0000 7fd6a1ffb640 1 -- 172.21.15.40:0/2217647052 --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6a4136bb0 con 0x7fd690003770 2024-04-22T08:18:48.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.146+0000 7fd6a1ffb640 10 monclient(hunting): _renew_subs 2024-04-22T08:18:48.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.146+0000 7fd6a37fe640 1 -- 172.21.15.40:0/2217647052 >> v1:172.21.15.40:6789/0 conn(0x7fd6a40a2150 legacy=0x7fd69000e9a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-04-22T08:18:48.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.147+0000 7fd6a8c14640 1 --2- 172.21.15.40:0/2217647052 >> v2:172.21.15.189:3300/0 conn(0x7fd690003770 0x7fd690005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:48.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.147+0000 7fd6a8c14640 10 monclient(hunting): get_auth_request con 0x7fd690003770 auth_method 0 2024-04-22T08:18:48.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.147+0000 7fd6a8c14640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:48.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.147+0000 7fd6a8c14640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:48.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.147+0000 7fd6a8c14640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:48.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.147+0000 7fd6a8c14640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:48.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.147+0000 7fd6a8c14640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:48.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.147+0000 7fd6a8c14640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:48.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.149+0000 7fd6a3fff640 1 -- 172.21.15.40:0/2217647052 >> v1:172.21.15.189:6789/0 conn(0x7fd6a40a1d30 legacy=0x7fd69000b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:18:48.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.149+0000 7fd6a8c14640 10 monclient(hunting): handle_auth_done global_id 64336 payload 274 2024-04-22T08:18:48.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.149+0000 7fd6a8c14640 1 -- 172.21.15.40:0/2217647052 >> v1:172.21.15.189:6789/0 conn(0x7fd6a40a1d30 legacy=0x7fd69000b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:48.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.149+0000 7fd6a8c14640 1 -- 172.21.15.40:0/2217647052 >> v1:172.21.15.40:6789/0 conn(0x7fd6a40a2150 legacy=0x7fd69000e9a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:18:48.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.149+0000 7fd6a8c14640 10 monclient: _finish_hunting 0 2024-04-22T08:18:48.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.149+0000 7fd6a8c14640 1 monclient: found mon.noname-c 2024-04-22T08:18:48.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.149+0000 7fd6a8c14640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:18:48.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.149+0000 7fd6a8c14640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.189:3300/0 2024-04-22T08:18:48.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.149+0000 7fd6a8c14640 1 -- 172.21.15.40:0/2217647052 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd690007780 con 0x7fd690003770 2024-04-22T08:18:48.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.152+0000 7fd6a8c14640 1 -- 172.21.15.40:0/2217647052 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd690003770 msgr2=0x7fd690005bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:18:48.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.152+0000 7fd6a8c14640 1 --2- 172.21.15.40:0/2217647052 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd690003770 0x7fd690005bf0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd6900066c0 tx=0x7fd69c02c3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae022027e314833e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:48.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.152+0000 7fd6a2ffd640 1 -- 172.21.15.40:0/2217647052 DispatchQueue::entry inject delay of 0.002000 2024-04-22T08:18:48.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.154+0000 7fd6a2ffd640 1 -- 172.21.15.40:0/2217647052 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69c030770 con 0x7fd690003770 2024-04-22T08:18:48.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.154+0000 7fd6a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:48.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.155+0000 7fd6a2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:18:48.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.155+0000 7fd6a2ffd640 10 monclient: dump: 2024-04-22T08:18:48.156 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:48.156 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:48.156 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:48.156 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:48.156 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:48.156 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:48.156 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:48.156 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:48.156 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:48.156 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:48.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.155+0000 7fd6a2ffd640 10 monclient: _finish_auth 0 2024-04-22T08:18:48.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.155+0000 7fd6a2ffd640 10 monclient: _check_auth_tickets 2024-04-22T08:18:48.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.155+0000 7fd6a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:18:48.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.155+0000 7fd6a2ffd640 1 -- 172.21.15.40:0/2217647052 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd69c02f070 con 0x7fd690003770 2024-04-22T08:18:48.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.155+0000 7fd6aa677640 5 monclient: authenticate success, global_id 64336 2024-04-22T08:18:48.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.155+0000 7fd6aa677640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:18:48.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.155+0000 7fd6a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:18:48.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.155+0000 7fd6aa677640 10 monclient: get_monmap_and_config success 2024-04-22T08:18:48.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.155+0000 7fd6aa677640 4 set_mon_vals no callback set 2024-04-22T08:18:48.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.155+0000 7fd6a2ffd640 1 -- 172.21.15.40:0/2217647052 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69c030b10 con 0x7fd690003770 2024-04-22T08:18:48.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.155+0000 7fd6a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:48.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.155+0000 7fd6a2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:18:48.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.155+0000 7fd6a2ffd640 10 monclient: dump: 2024-04-22T08:18:48.158 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:48.158 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:48.158 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:48.158 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:48.158 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:48.158 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:48.158 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:48.158 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:48.158 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:48.158 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:48.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.155+0000 7fd6aa677640 10 monclient: shutdown 2024-04-22T08:18:48.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.155+0000 7fd6aa677640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:18:48.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.155+0000 7fd6aa677640 1 -- 172.21.15.40:0/2217647052 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd690003770 msgr2=0x7fd690005bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:48.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.155+0000 7fd6aa677640 1 --2- 172.21.15.40:0/2217647052 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd690003770 0x7fd690005bf0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd6900066c0 tx=0x7fd69c02c3b0 comp rx=0 tx=0).stop 2024-04-22T08:18:48.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.156+0000 7fd6aa677640 1 -- 172.21.15.40:0/2217647052 shutdown_connections 2024-04-22T08:18:48.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.156+0000 7fd6aa677640 1 --2- 172.21.15.40:0/2217647052 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd690003770 0x7fd690005bf0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:48.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.156+0000 7fd6aa677640 1 -- 172.21.15.40:0/2217647052 >> 172.21.15.40:0/2217647052 conn(0x7fd6a40a4ef0 msgr2=0x7fd6a40a5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:18:48.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.156+0000 7fd6aa677640 1 -- 172.21.15.40:0/2217647052 shutdown_connections 2024-04-22T08:18:48.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.156+0000 7fd6aa677640 1 -- 172.21.15.40:0/2217647052 wait complete. 2024-04-22T08:18:48.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.156+0000 7fd6aa677640 10 monclient: build_initial_monmap 2024-04-22T08:18:48.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.156+0000 7fd6aa677640 10 monclient: monmap: 2024-04-22T08:18:48.167 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:18:48.167 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:18:48.167 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:18:48.167 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:18:48.167 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:18:48.168 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:48.168 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:18:48.168 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:18:48.168 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:18:48.168 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:48.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.157+0000 7fd6aa677640 1 Processor -- start 2024-04-22T08:18:48.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.157+0000 7fd6aa677640 1 -- start start 2024-04-22T08:18:48.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.157+0000 7fd6aa677640 10 monclient: init 2024-04-22T08:18:48.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.157+0000 7fd6aa677640 10 monclient: _reopen_session rank -1 2024-04-22T08:18:48.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.157+0000 7fd6aa677640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-22T08:18:48.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.158+0000 7fd6aa677640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd690003770 0x7fd6a41caef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:48.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.158+0000 7fd6aa677640 10 monclient(hunting): picked mon.noname-a con 0x7fd690003770 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:18:48.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.158+0000 7fd6aa677640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6a40a1d30 0x7fd6a41cb430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:48.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.158+0000 7fd6aa677640 10 monclient(hunting): picked mon.noname-c con 0x7fd6a40a1d30 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:18:48.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.158+0000 7fd6aa677640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6a40a2150 0x7fd6a41211d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:48.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.158+0000 7fd6aa677640 10 monclient(hunting): picked mon.noname-b con 0x7fd6a40a2150 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:18:48.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.158+0000 7fd6aa677640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:48.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.158+0000 7fd6aa677640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a4124f70 con 0x7fd690003770 2024-04-22T08:18:48.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.158+0000 7fd6aa677640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:48.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.158+0000 7fd6aa677640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a41248b0 con 0x7fd6a40a2150 2024-04-22T08:18:48.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.158+0000 7fd6aa677640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:48.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.158+0000 7fd6aa677640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a41239c0 con 0x7fd6a40a1d30 2024-04-22T08:18:48.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.158+0000 7fd6aa677640 10 monclient(hunting): _renew_subs 2024-04-22T08:18:48.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.158+0000 7fd6a3fff640 1 -- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd690003770 msgr2=0x7fd6a41caef0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:18:48.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.158+0000 7fd6aa677640 10 monclient(hunting): authenticate will time out at 1202.152875s 2024-04-22T08:18:48.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.158+0000 7fd6a3fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd690003770 0x7fd6a41caef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:18:48.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.158+0000 7fd6a37fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6a40a1d30 0x7fd6a41cb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:48.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.158+0000 7fd6a37fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6a40a1d30 0x7fd6a41cb430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:51228/0 (socket says 172.21.15.40:51228) 2024-04-22T08:18:48.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.158+0000 7fd6a37fe640 1 -- 172.21.15.40:0/1423461901 learned_addr learned my addr 172.21.15.40:0/1423461901 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:18:48.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.158+0000 7fd6a37fe640 10 monclient(hunting): get_auth_request con 0x7fd6a40a1d30 auth_method 0 2024-04-22T08:18:48.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.158+0000 7fd6a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:48.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.158+0000 7fd6a37fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:48.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.158+0000 7fd6a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:48.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.158+0000 7fd6a8c14640 1 --2- 172.21.15.40:0/1423461901 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6a40a2150 0x7fd6a41211d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:48.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.158+0000 7fd6a8c14640 10 monclient(hunting): get_auth_request con 0x7fd6a40a2150 auth_method 0 2024-04-22T08:18:48.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.158+0000 7fd6a8c14640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:48.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.158+0000 7fd6a8c14640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:48.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.158+0000 7fd6a8c14640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:48.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.158+0000 7fd6a37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:48.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.158+0000 7fd6a37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:48.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.158+0000 7fd6a37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:48.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.158+0000 7fd6a8c14640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:48.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.158+0000 7fd6a8c14640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:48.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.158+0000 7fd6a8c14640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:48.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.158+0000 7fd6a37fe640 10 monclient(hunting): handle_auth_done global_id 54527 payload 824 2024-04-22T08:18:48.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.158+0000 7fd6a37fe640 1 -- 172.21.15.40:0/1423461901 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6a40a2150 msgr2=0x7fd6a41211d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:48.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.158+0000 7fd6a37fe640 1 --2- 172.21.15.40:0/1423461901 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6a40a2150 0x7fd6a41211d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:48.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.158+0000 7fd6a37fe640 1 -- 172.21.15.40:0/1423461901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd690003770 msgr2=0x7fd6a41caef0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:18:48.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.158+0000 7fd6a37fe640 1 --2- 172.21.15.40:0/1423461901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd690003770 0x7fd6a41caef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:48.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.158+0000 7fd6a37fe640 10 monclient: _finish_hunting 0 2024-04-22T08:18:48.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.158+0000 7fd6a37fe640 1 monclient: found mon.noname-c 2024-04-22T08:18:48.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.158+0000 7fd6a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:18:48.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.158+0000 7fd6a37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:18:48.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.158+0000 7fd6a37fe640 1 -- 172.21.15.40:0/1423461901 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd69c007ab0 con 0x7fd6a40a1d30 2024-04-22T08:18:48.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.158+0000 7fd6a8c14640 1 --2- 172.21.15.40:0/1423461901 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6a40a2150 0x7fd6a41211d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:18:48.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.161+0000 7fd6a37fe640 1 -- 172.21.15.40:0/1423461901 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6a40a1d30 msgr2=0x7fd6a41cb430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:18:48.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.161+0000 7fd6a37fe640 1 --2- 172.21.15.40:0/1423461901 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6a40a1d30 0x7fd6a41cb430 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd69800c4e0 tx=0x7fd69800c9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:48.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.161+0000 7fd6a0ff9640 1 -- 172.21.15.40:0/1423461901 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6980057c0 con 0x7fd6a40a1d30 2024-04-22T08:18:48.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.161+0000 7fd6a0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:48.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.161+0000 7fd6a0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:18:48.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.161+0000 7fd6a0ff9640 10 monclient: dump: 2024-04-22T08:18:48.172 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:48.172 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:48.172 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:48.172 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:48.173 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:48.173 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:48.173 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:48.173 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:48.173 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:48.173 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:48.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.161+0000 7fd6a0ff9640 10 monclient: _finish_auth 0 2024-04-22T08:18:48.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.161+0000 7fd6a0ff9640 10 monclient: _check_auth_tickets 2024-04-22T08:18:48.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.161+0000 7fd6a0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:18:48.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.161+0000 7fd6a0ff9640 1 -- 172.21.15.40:0/1423461901 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd698005940 con 0x7fd6a40a1d30 2024-04-22T08:18:48.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.161+0000 7fd6aa677640 5 monclient: authenticate success, global_id 54527 2024-04-22T08:18:48.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.161+0000 7fd6aa677640 10 monclient: _renew_subs 2024-04-22T08:18:48.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.161+0000 7fd6aa677640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:18:48.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.161+0000 7fd6aa677640 1 -- 172.21.15.40:0/1423461901 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6a40a2730 con 0x7fd6a40a1d30 2024-04-22T08:18:48.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.161+0000 7fd6a0ff9640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:18:48.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.161+0000 7fd6a0ff9640 1 -- 172.21.15.40:0/1423461901 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd698005cf0 con 0x7fd6a40a1d30 2024-04-22T08:18:48.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.161+0000 7fd6a0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:48.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.161+0000 7fd6a0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:18:48.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.161+0000 7fd6a0ff9640 10 monclient: dump: 2024-04-22T08:18:48.174 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:48.174 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:48.174 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:48.174 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:48.174 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:48.174 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:48.174 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:48.174 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:48.174 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:48.174 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:48.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.161+0000 7fd6a27fc640 4 set_mon_vals no callback set 2024-04-22T08:18:48.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.161+0000 7fd6aa677640 10 monclient: _renew_subs 2024-04-22T08:18:48.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.161+0000 7fd6aa677640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:18:48.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.161+0000 7fd6aa677640 1 -- 172.21.15.40:0/1423461901 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6a41360d0 con 0x7fd6a40a1d30 2024-04-22T08:18:48.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.162+0000 7fd6a0ff9640 1 -- 172.21.15.40:0/1423461901 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fd69801b3b0 con 0x7fd6a40a1d30 2024-04-22T08:18:48.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.162+0000 7fd6a0ff9640 1 --2- 172.21.15.40:0/1423461901 >> v2:172.21.15.189:6800/30168 conn(0x7fd690038500 0x7fd69003a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:48.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.162+0000 7fd6a0ff9640 1 -- 172.21.15.40:0/1423461901 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 3506+0+0 (secure 0 0 0) 0x7fd698049820 con 0x7fd6a40a1d30 2024-04-22T08:18:48.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.162+0000 7fd6aa677640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:18:48.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.162+0000 7fd6a3fff640 1 --2- 172.21.15.40:0/1423461901 >> v2:172.21.15.189:6800/30168 conn(0x7fd690038500 0x7fd69003a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:48.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.162+0000 7fd6aa677640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:18:48.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.162+0000 7fd6aa677640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:18:48.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.162+0000 7fd6aa677640 1 -- 172.21.15.40:0/1423461901 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6a4124dd0 con 0x7fd6a40a1d30 2024-04-22T08:18:48.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.162+0000 7fd6a3fff640 10 monclient: get_auth_request con 0x7fd690038500 auth_method 0 2024-04-22T08:18:48.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.165+0000 7fd6a3fff640 1 --2- 172.21.15.40:0/1423461901 >> v2:172.21.15.189:6800/30168 conn(0x7fd690038500 0x7fd69003a9c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fd694000c00 tx=0x7fd694008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:48.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.167+0000 7fd6a0ff9640 1 -- 172.21.15.40:0/1423461901 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7fd6a4124dd0 con 0x7fd6a40a1d30 2024-04-22T08:18:48.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.167+0000 7fd6a0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:18:48.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.167+0000 7fd6a0ff9640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:18:48.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.416+0000 7fd6aa677640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:18:48.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.416+0000 7fd6aa677640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:18:48.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.416+0000 7fd6aa677640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:18:48.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.416+0000 7fd6aa677640 1 -- 172.21.15.40:0/1423461901 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd6a4124dd0 con 0x7fd6a40a1d30 2024-04-22T08:18:48.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.417+0000 7fd6a0ff9640 1 -- 172.21.15.40:0/1423461901 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1537 (secure 0 0 0) 0x7fd69803b620 con 0x7fd6a40a1d30 2024-04-22T08:18:48.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.417+0000 7fd6a0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:18:48.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.417+0000 7fd6a0ff9640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:18:48.418 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":58,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:18:48.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.419+0000 7fd6aa677640 1 -- 172.21.15.40:0/1423461901 >> v2:172.21.15.189:6800/30168 conn(0x7fd690038500 msgr2=0x7fd69003a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:48.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.419+0000 7fd6aa677640 1 --2- 172.21.15.40:0/1423461901 >> v2:172.21.15.189:6800/30168 conn(0x7fd690038500 0x7fd69003a9c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fd694000c00 tx=0x7fd694008040 comp rx=0 tx=0).stop 2024-04-22T08:18:48.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.419+0000 7fd6aa677640 10 monclient: shutdown 2024-04-22T08:18:48.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.419+0000 7fd6aa677640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:18:48.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.419+0000 7fd6aa677640 1 -- 172.21.15.40:0/1423461901 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6a40a1d30 msgr2=0x7fd6a41cb430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:48.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.419+0000 7fd6aa677640 1 --2- 172.21.15.40:0/1423461901 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6a40a1d30 0x7fd6a41cb430 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd69800c4e0 tx=0x7fd69800c9a0 comp rx=0 tx=0).stop 2024-04-22T08:18:48.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.419+0000 7fd6aa677640 1 -- 172.21.15.40:0/1423461901 shutdown_connections 2024-04-22T08:18:48.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.419+0000 7fd6aa677640 1 --2- 172.21.15.40:0/1423461901 >> v2:172.21.15.189:6800/30168 conn(0x7fd690038500 0x7fd69003a9c0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:48.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.419+0000 7fd6aa677640 1 --2- 172.21.15.40:0/1423461901 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6a40a2150 0x7fd6a41211d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:48.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.419+0000 7fd6aa677640 1 --2- 172.21.15.40:0/1423461901 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6a40a1d30 0x7fd6a41cb430 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:48.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.419+0000 7fd6aa677640 1 --2- 172.21.15.40:0/1423461901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd690003770 0x7fd6a41caef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:48.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.419+0000 7fd6aa677640 1 -- 172.21.15.40:0/1423461901 >> 172.21.15.40:0/1423461901 conn(0x7fd6a40a4ef0 msgr2=0x7fd6a40a61f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:18:48.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.419+0000 7fd6aa677640 1 -- 172.21.15.40:0/1423461901 shutdown_connections 2024-04-22T08:18:48.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.419+0000 7fd6aa677640 1 -- 172.21.15.40:0/1423461901 wait complete. 2024-04-22T08:18:48.430 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-22T08:18:48.430 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-04-22T08:18:48.430 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-22T08:18:48.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.532+0000 7f8245dd0640 10 monclient: get_monmap_and_config 2024-04-22T08:18:48.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.532+0000 7f8245dd0640 10 monclient: build_initial_monmap 2024-04-22T08:18:48.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.532+0000 7f8245dd0640 10 monclient: monmap: 2024-04-22T08:18:48.534 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:18:48.534 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:48.534 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:18:48.534636+0000 2024-04-22T08:18:48.534 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:18:48.534636+0000 2024-04-22T08:18:48.534 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:18:48.534 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:48.534 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:18:48.534 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:18:48.534 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:18:48.534 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:18:48.534 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:18:48.535 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:48.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.533+0000 7f8245dd0640 1 Processor -- start 2024-04-22T08:18:48.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.533+0000 7f8245dd0640 1 -- start start 2024-04-22T08:18:48.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.533+0000 7f8245dd0640 10 monclient: init 2024-04-22T08:18:48.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.534+0000 7f8245dd0640 10 monclient: _reopen_session rank -1 2024-04-22T08:18:48.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.534+0000 7f8245dd0640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-04-22T08:18:48.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.534+0000 7f8245dd0640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f824009f270 0x7f824009d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:48.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.534+0000 7f8245dd0640 10 monclient(hunting): picked mon.noname-b con 0x7f824009f270 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:18:48.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.534+0000 7f8245dd0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f824009f650 addr v1:172.21.15.40:6789/0 2024-04-22T08:18:48.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.534+0000 7f8245dd0640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f82401377e0 0x7f8240139bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:48.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.534+0000 7f8245dd0640 10 monclient(hunting): picked mon.noname-a con 0x7f82401377e0 addr v2:172.21.15.40:3300/0 2024-04-22T08:18:48.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.534+0000 7f8245dd0640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f824009e510 con 0x7f824009f650 2024-04-22T08:18:48.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.534+0000 7f8245dd0640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:48.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.534+0000 7f8245dd0640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f824009ba30 con 0x7f82401377e0 2024-04-22T08:18:48.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.534+0000 7f8245dd0640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:48.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.534+0000 7f8245dd0640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f824009b760 con 0x7f824009f270 2024-04-22T08:18:48.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.534+0000 7f8245dd0640 10 monclient(hunting): _renew_subs 2024-04-22T08:18:48.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.534+0000 7f8245dd0640 10 monclient(hunting): authenticate will time out at 1202.528876s 2024-04-22T08:18:48.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.534+0000 7f823effd640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f824009f650 legacy=0x7f824009fa50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-04-22T08:18:48.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.534+0000 7f823ffff640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f82401377e0 msgr2=0x7f8240139bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:18:48.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.534+0000 7f823ffff640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f82401377e0 0x7f8240139bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:18:48.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.534+0000 7f823f7fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f824009f270 0x7f824009d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:48.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.534+0000 7f823f7fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f824009f270 0x7f824009d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:51234/0 (socket says 172.21.15.40:51234) 2024-04-22T08:18:48.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.534+0000 7f823f7fe640 1 -- 172.21.15.40:0/197741267 learned_addr learned my addr 172.21.15.40:0/197741267 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:18:48.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.534+0000 7f823f7fe640 10 monclient(hunting): get_auth_request con 0x7f824009f270 auth_method 0 2024-04-22T08:18:48.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.534+0000 7f823f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:48.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.534+0000 7f823f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:48.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.534+0000 7f823f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:48.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.534+0000 7f823f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:48.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.534+0000 7f823f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:48.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.534+0000 7f823f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:48.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.534+0000 7f823f7fe640 10 monclient(hunting): handle_auth_done global_id 54530 payload 274 2024-04-22T08:18:48.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.534+0000 7f823f7fe640 1 -- 172.21.15.40:0/197741267 >> v2:172.21.15.40:3300/0 conn(0x7f82401377e0 msgr2=0x7f8240139bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:18:48.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.534+0000 7f823f7fe640 1 --2- 172.21.15.40:0/197741267 >> v2:172.21.15.40:3300/0 conn(0x7f82401377e0 0x7f8240139bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:48.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.534+0000 7f823f7fe640 1 -- 172.21.15.40:0/197741267 >> v1:172.21.15.40:6789/0 conn(0x7f824009f650 legacy=0x7f824009fa50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:18:48.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.534+0000 7f823f7fe640 10 monclient: _finish_hunting 0 2024-04-22T08:18:48.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.534+0000 7f823f7fe640 1 monclient: found mon.noname-b 2024-04-22T08:18:48.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.534+0000 7f823f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:18:48.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.534+0000 7f823f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:18:48.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.534+0000 7f823f7fe640 1 -- 172.21.15.40:0/197741267 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f824009c790 con 0x7f824009f270 2024-04-22T08:18:48.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.537+0000 7f823f7fe640 1 -- 172.21.15.40:0/197741267 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f824009f270 msgr2=0x7f824009d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:18:48.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.537+0000 7f823f7fe640 1 --2- 172.21.15.40:0/197741267 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f824009f270 0x7f824009d450 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f824006bb10 tx=0x7f823002b7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91ff52dd5b37add4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:48.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.537+0000 7f823e7fc640 1 -- 172.21.15.40:0/197741267 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8230031a40 con 0x7f824009f270 2024-04-22T08:18:48.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.537+0000 7f823e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:48.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.537+0000 7f823e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:18:48.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.537+0000 7f823e7fc640 10 monclient: dump: 2024-04-22T08:18:48.547 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:48.547 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:48.548 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:48.548 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:48.548 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:48.548 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:48.548 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:48.548 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:48.548 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:48.548 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:48.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.537+0000 7f823e7fc640 10 monclient: _finish_auth 0 2024-04-22T08:18:48.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.537+0000 7f823e7fc640 10 monclient: _check_auth_tickets 2024-04-22T08:18:48.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.537+0000 7f823e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:18:48.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.537+0000 7f823e7fc640 1 -- 172.21.15.40:0/197741267 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8230031ba0 con 0x7f824009f270 2024-04-22T08:18:48.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.537+0000 7f823e7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:18:48.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.537+0000 7f823e7fc640 1 -- 172.21.15.40:0/197741267 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8230033440 con 0x7f824009f270 2024-04-22T08:18:48.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.537+0000 7f8245dd0640 5 monclient: authenticate success, global_id 54530 2024-04-22T08:18:48.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.537+0000 7f8245dd0640 10 monclient: get_monmap_and_config success 2024-04-22T08:18:48.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.537+0000 7f8245dd0640 4 set_mon_vals no callback set 2024-04-22T08:18:48.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.537+0000 7f823e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:48.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.537+0000 7f823e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:18:48.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.537+0000 7f823e7fc640 10 monclient: dump: 2024-04-22T08:18:48.549 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:48.549 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:48.549 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:48.549 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:48.549 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:48.549 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:48.549 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:48.549 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:48.549 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:48.549 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:48.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.538+0000 7f8245dd0640 10 monclient: shutdown 2024-04-22T08:18:48.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.538+0000 7f8245dd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:18:48.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.538+0000 7f8245dd0640 1 -- 172.21.15.40:0/197741267 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f824009f270 msgr2=0x7f824009d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:48.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.538+0000 7f8245dd0640 1 --2- 172.21.15.40:0/197741267 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f824009f270 0x7f824009d450 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f824006bb10 tx=0x7f823002b7f0 comp rx=0 tx=0).stop 2024-04-22T08:18:48.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.538+0000 7f8245dd0640 1 -- 172.21.15.40:0/197741267 shutdown_connections 2024-04-22T08:18:48.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.538+0000 7f8245dd0640 1 --2- 172.21.15.40:0/197741267 >> v2:172.21.15.40:3300/0 conn(0x7f82401377e0 0x7f8240139bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:48.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.538+0000 7f8245dd0640 1 --2- 172.21.15.40:0/197741267 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f824009f270 0x7f824009d450 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:48.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.538+0000 7f8245dd0640 1 -- 172.21.15.40:0/197741267 >> 172.21.15.40:0/197741267 conn(0x7f82400a0ae0 msgr2=0x7f82400a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:18:48.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.538+0000 7f8245dd0640 1 -- 172.21.15.40:0/197741267 shutdown_connections 2024-04-22T08:18:48.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.538+0000 7f8245dd0640 1 -- 172.21.15.40:0/197741267 wait complete. 2024-04-22T08:18:48.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.538+0000 7f8245dd0640 10 monclient: build_initial_monmap 2024-04-22T08:18:48.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.538+0000 7f8245dd0640 10 monclient: monmap: 2024-04-22T08:18:48.550 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:18:48.551 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:18:48.551 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:18:48.551 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:18:48.551 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:18:48.551 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:48.551 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:18:48.551 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:18:48.551 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:18:48.551 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:48.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.539+0000 7f8245dd0640 1 Processor -- start 2024-04-22T08:18:48.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.539+0000 7f8245dd0640 1 -- start start 2024-04-22T08:18:48.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.539+0000 7f8245dd0640 10 monclient: init 2024-04-22T08:18:48.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.539+0000 7f8245dd0640 10 monclient: _reopen_session rank -1 2024-04-22T08:18:48.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.539+0000 7f8245dd0640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-22T08:18:48.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.539+0000 7f8245dd0640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f824009f270 0x7f82401c8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:48.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.539+0000 7f8245dd0640 10 monclient(hunting): picked mon.noname-b con 0x7f824009f270 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:18:48.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.539+0000 7f8245dd0640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f824009f650 0x7f8240136c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:48.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.539+0000 7f8245dd0640 10 monclient(hunting): picked mon.noname-a con 0x7f824009f650 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:18:48.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.539+0000 7f8245dd0640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f82401377e0 0x7f8240137150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:48.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.539+0000 7f8245dd0640 10 monclient(hunting): picked mon.noname-c con 0x7f82401377e0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:18:48.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.539+0000 7f8245dd0640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:48.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.539+0000 7f8245dd0640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f824009bc10 con 0x7f824009f650 2024-04-22T08:18:48.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.539+0000 7f8245dd0640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:48.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.539+0000 7f8245dd0640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82401354b0 con 0x7f824009f270 2024-04-22T08:18:48.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.539+0000 7f8245dd0640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:48.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.539+0000 7f8245dd0640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82400a33f0 con 0x7f82401377e0 2024-04-22T08:18:48.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.539+0000 7f8245dd0640 10 monclient(hunting): _renew_subs 2024-04-22T08:18:48.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.539+0000 7f8245dd0640 10 monclient(hunting): authenticate will time out at 1202.534702s 2024-04-22T08:18:48.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.539+0000 7f823effd640 1 -- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f824009f650 msgr2=0x7f8240136c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:18:48.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.539+0000 7f823effd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f824009f650 0x7f8240136c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:18:48.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.540+0000 7f823ffff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f82401377e0 0x7f8240137150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:48.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.540+0000 7f823ffff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f82401377e0 0x7f8240137150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:51248/0 (socket says 172.21.15.40:51248) 2024-04-22T08:18:48.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.540+0000 7f823ffff640 1 -- 172.21.15.40:0/15427029 learned_addr learned my addr 172.21.15.40:0/15427029 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:18:48.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.540+0000 7f823ffff640 10 monclient(hunting): get_auth_request con 0x7f82401377e0 auth_method 0 2024-04-22T08:18:48.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.540+0000 7f823ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:48.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.540+0000 7f823ffff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:48.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.540+0000 7f823ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:48.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.540+0000 7f823f7fe640 1 --2- 172.21.15.40:0/15427029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f824009f270 0x7f82401c8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:48.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.540+0000 7f823ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:48.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.540+0000 7f823ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:48.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.540+0000 7f823ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:48.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.540+0000 7f823f7fe640 10 monclient(hunting): get_auth_request con 0x7f824009f270 auth_method 0 2024-04-22T08:18:48.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.540+0000 7f823f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:48.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.540+0000 7f823f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:48.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.540+0000 7f823f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:48.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.540+0000 7f823f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:48.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.540+0000 7f823f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:48.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.540+0000 7f823f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:48.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.540+0000 7f823ffff640 10 monclient(hunting): handle_auth_done global_id 54533 payload 824 2024-04-22T08:18:48.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.540+0000 7f823ffff640 1 -- 172.21.15.40:0/15427029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f824009f270 msgr2=0x7f82401c8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:48.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.540+0000 7f823ffff640 1 --2- 172.21.15.40:0/15427029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f824009f270 0x7f82401c8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:48.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.540+0000 7f823ffff640 1 -- 172.21.15.40:0/15427029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f824009f650 msgr2=0x7f8240136c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:18:48.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.540+0000 7f823ffff640 1 --2- 172.21.15.40:0/15427029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f824009f650 0x7f8240136c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:48.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.540+0000 7f823ffff640 10 monclient: _finish_hunting 0 2024-04-22T08:18:48.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.540+0000 7f823ffff640 1 monclient: found mon.noname-c 2024-04-22T08:18:48.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.540+0000 7f823ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:18:48.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.540+0000 7f823ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:18:48.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.540+0000 7f823ffff640 1 -- 172.21.15.40:0/15427029 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82300094d0 con 0x7f82401377e0 2024-04-22T08:18:48.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.542+0000 7f823ffff640 1 -- 172.21.15.40:0/15427029 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f82401377e0 msgr2=0x7f8240137150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:18:48.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.542+0000 7f823ffff640 1 --2- 172.21.15.40:0/15427029 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f82401377e0 0x7f8240137150 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f823800be60 tx=0x7f823800bf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:48.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.542+0000 7f8227fff640 1 -- 172.21.15.40:0/15427029 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8238015670 con 0x7f82401377e0 2024-04-22T08:18:48.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.542+0000 7f8227fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:48.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.542+0000 7f8227fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:18:48.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.542+0000 7f8227fff640 10 monclient: dump: 2024-04-22T08:18:48.555 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:48.555 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:48.555 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:48.555 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:48.555 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:48.555 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:48.556 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:48.556 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:48.556 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:48.556 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:48.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.542+0000 7f8227fff640 10 monclient: _finish_auth 0 2024-04-22T08:18:48.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.542+0000 7f8227fff640 10 monclient: _check_auth_tickets 2024-04-22T08:18:48.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.542+0000 7f8227fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:18:48.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.542+0000 7f8227fff640 1 -- 172.21.15.40:0/15427029 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82380157d0 con 0x7f82401377e0 2024-04-22T08:18:48.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.542+0000 7f8227fff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:18:48.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.542+0000 7f8227fff640 1 -- 172.21.15.40:0/15427029 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8238015b80 con 0x7f82401377e0 2024-04-22T08:18:48.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.542+0000 7f8227fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:48.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.542+0000 7f8227fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:18:48.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.542+0000 7f8227fff640 10 monclient: dump: 2024-04-22T08:18:48.557 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:48.557 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:48.557 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:48.557 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:48.557 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:48.558 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:48.558 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:48.558 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:48.558 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:48.558 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:48.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.542+0000 7f8245dd0640 5 monclient: authenticate success, global_id 54533 2024-04-22T08:18:48.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.542+0000 7f8245dd0640 10 monclient: _renew_subs 2024-04-22T08:18:48.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.542+0000 7f8245dd0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:18:48.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.542+0000 7f8245dd0640 1 -- 172.21.15.40:0/15427029 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8240133150 con 0x7f82401377e0 2024-04-22T08:18:48.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.542+0000 7f823dffb640 4 set_mon_vals no callback set 2024-04-22T08:18:48.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.542+0000 7f8245dd0640 10 monclient: _renew_subs 2024-04-22T08:18:48.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.542+0000 7f8245dd0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:18:48.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.542+0000 7f8245dd0640 1 -- 172.21.15.40:0/15427029 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8240134c10 con 0x7f82401377e0 2024-04-22T08:18:48.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.543+0000 7f8245dd0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:18:48.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.543+0000 7f8245dd0640 1 --2- 172.21.15.40:0/15427029 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8240132170 0x7f824002b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:48.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.543+0000 7f8245dd0640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:48.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.543+0000 7f8245dd0640 1 -- 172.21.15.40:0/15427029 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82400667a0 con 0x7f8240132170 2024-04-22T08:18:48.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.543+0000 7f8227fff640 1 -- 172.21.15.40:0/15427029 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f8238012070 con 0x7f82401377e0 2024-04-22T08:18:48.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.543+0000 7f823f7fe640 1 --2- 172.21.15.40:0/15427029 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8240132170 0x7f824002b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:48.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.543+0000 7f823f7fe640 10 monclient: get_auth_request con 0x7f8240132170 auth_method 0 2024-04-22T08:18:48.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.543+0000 7f823f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:48.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.543+0000 7f823f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:48.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.543+0000 7f823f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:48.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.544+0000 7f823f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:48.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.544+0000 7f823f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:48.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.544+0000 7f823f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:48.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.544+0000 7f8227fff640 1 --2- 172.21.15.40:0/15427029 >> v2:172.21.15.189:6800/30168 conn(0x7f822c037800 0x7f822c039cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:48.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.544+0000 7f8227fff640 1 -- 172.21.15.40:0/15427029 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f823804a9c0 con 0x7f82401377e0 2024-04-22T08:18:48.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.544+0000 7f823f7fe640 10 monclient(hunting): handle_auth_done global_id 54536 payload 824 2024-04-22T08:18:48.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.544+0000 7f823f7fe640 1 -- 172.21.15.40:0/15427029 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f824009d990 con 0x7f8240132170 2024-04-22T08:18:48.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.544+0000 7f823effd640 1 --2- 172.21.15.40:0/15427029 >> v2:172.21.15.189:6800/30168 conn(0x7f822c037800 0x7f822c039cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:48.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.544+0000 7f823effd640 10 monclient: get_auth_request con 0x7f822c037800 auth_method 0 2024-04-22T08:18:48.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.546+0000 7f823f7fe640 1 -- 172.21.15.40:0/15427029 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8240132170 msgr2=0x7f824002b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:18:48.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.546+0000 7f823f7fe640 1 --2- 172.21.15.40:0/15427029 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8240132170 0x7f824002b200 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8230033000 tx=0x7f82300046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:48.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.546+0000 7f8227fff640 1 -- 172.21.15.40:0/15427029 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8230033640 con 0x7f8240132170 2024-04-22T08:18:48.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.546+0000 7f8227fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:48.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.546+0000 7f8227fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:18:48.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.546+0000 7f8227fff640 10 monclient: dump: 2024-04-22T08:18:48.560 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:48.560 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:48.560 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:48.560 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:48.561 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:48.561 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:48.561 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:48.561 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:48.561 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:48.561 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:48.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.546+0000 7f8227fff640 1 -- 172.21.15.40:0/15427029 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f82300337a0 con 0x7f8240132170 2024-04-22T08:18:48.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.546+0000 7f8227fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:18:48.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.546+0000 7f8227fff640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:18:48.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.546+0000 7f8227fff640 1 -- 172.21.15.40:0/15427029 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8240132170 msgr2=0x7f824002b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:48.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.546+0000 7f8227fff640 1 --2- 172.21.15.40:0/15427029 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8240132170 0x7f824002b200 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8230033000 tx=0x7f82300046c0 comp rx=0 tx=0).stop 2024-04-22T08:18:48.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.546+0000 7f8227fff640 1 -- 172.21.15.40:0/15427029 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8240132170 msgr2=0x7f824002b200 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:18:48.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.546+0000 7f8227fff640 1 --2- 172.21.15.40:0/15427029 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8240132170 0x7f824002b200 secure :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8230033000 tx=0x7f82300046c0 comp rx=0 tx=0).stop 2024-04-22T08:18:48.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.547+0000 7f823effd640 1 --2- 172.21.15.40:0/15427029 >> v2:172.21.15.189:6800/30168 conn(0x7f822c037800 0x7f822c039cc0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f82400716f0 tx=0x7f8234006d30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:48.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.553+0000 7f8245dd0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:18:48.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.553+0000 7f8245dd0640 1 --2- 172.21.15.40:0/15427029 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8240132550 0x7f824002b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:48.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.553+0000 7f8245dd0640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:48.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.553+0000 7f8245dd0640 1 -- 172.21.15.40:0/15427029 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f824013b080 con 0x7f8240132550 2024-04-22T08:18:48.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.553+0000 7f823f7fe640 1 --2- 172.21.15.40:0/15427029 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8240132550 0x7f824002b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:48.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.553+0000 7f823f7fe640 10 monclient: get_auth_request con 0x7f8240132550 auth_method 0 2024-04-22T08:18:48.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.553+0000 7f823f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:48.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.553+0000 7f823f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:48.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.553+0000 7f823f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:48.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.553+0000 7f823f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:48.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.553+0000 7f823f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:48.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.553+0000 7f823f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:48.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.553+0000 7f823f7fe640 10 monclient(hunting): handle_auth_done global_id 54539 payload 824 2024-04-22T08:18:48.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.553+0000 7f823f7fe640 1 -- 172.21.15.40:0/15427029 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f824004e9b0 con 0x7f8240132550 2024-04-22T08:18:48.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.555+0000 7f823f7fe640 1 -- 172.21.15.40:0/15427029 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8240132550 msgr2=0x7f824002b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:18:48.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.555+0000 7f823f7fe640 1 --2- 172.21.15.40:0/15427029 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8240132550 0x7f824002b740 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f823002b830 tx=0x7f8230031000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:48.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.555+0000 7f8227fff640 1 -- 172.21.15.40:0/15427029 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8230003820 con 0x7f8240132550 2024-04-22T08:18:48.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.555+0000 7f8227fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:48.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.555+0000 7f8227fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:18:48.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.555+0000 7f8227fff640 10 monclient: dump: 2024-04-22T08:18:48.565 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:48.565 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:48.565 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:48.565 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:48.565 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:48.565 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:48.565 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:48.565 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:48.566 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:48.566 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:48.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.556+0000 7f8227fff640 1 -- 172.21.15.40:0/15427029 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4620 (secure 0 0 0) 0x7f8230003980 con 0x7f8240132550 2024-04-22T08:18:48.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.556+0000 7f8227fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:18:48.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.556+0000 7f8227fff640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:18:48.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.556+0000 7f8227fff640 1 -- 172.21.15.40:0/15427029 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8240132550 msgr2=0x7f824002b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:48.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.556+0000 7f8227fff640 1 --2- 172.21.15.40:0/15427029 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8240132550 0x7f824002b740 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f823002b830 tx=0x7f8230031000 comp rx=0 tx=0).stop 2024-04-22T08:18:48.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.556+0000 7f8227fff640 1 -- 172.21.15.40:0/15427029 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8240132550 msgr2=0x7f824002b740 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:18:48.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.556+0000 7f8227fff640 1 --2- 172.21.15.40:0/15427029 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8240132550 0x7f824002b740 secure :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f823002b830 tx=0x7f8230031000 comp rx=0 tx=0).stop 2024-04-22T08:18:48.566 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:18:48.567 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:18:48.567 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:18:48.567 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:18:48.567 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 58, 2024-04-22T08:18:48.567 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:18:48.567 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:18:48.567 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:18:48.567 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:18:48.568 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 1, 2024-04-22T08:18:48.568 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:18:48.568 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:18:48.568 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:18:48.568 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:18:48.568 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:18:48.568 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:18:48.568 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:18:48.568 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:18:48.568 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:18:48.568 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:18:48.568 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:18:48.568 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:18:48.569 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:18:48.569 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:18:48.569 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:18:48.569 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:18:48.569 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:18:48.569 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:18:48.569 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:18:48.569 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:18:48.569 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:18:48.569 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:18:48.569 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:18:48.569 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:18:48.570 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:18:48.570 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:18:48.570 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:48.570 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:48.570 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:18:48.570 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:18:48.570 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:18:48.570 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:18:48.570 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:18:48.570 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:18:48.570 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:18:48.570 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:18:48.570 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:18:48.570 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:18:48.571 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:18:48.571 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:18:48.571 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:18:48.571 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:18:48.571 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:18:48.571 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:18:48.571 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:18:48.571 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:18:48.571 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:18:48.571 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:18:48.571 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:18:48.572 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:18:48.572 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:18:48.572 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:18:48.572 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:18:48.572 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:18:48.572 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:18:48.572 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:18:48.572 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:18:48.572 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:48.572 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:18:48.572 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:48.573 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:18:48.573 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:18:48.573 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:18:48.573 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:18:48.573 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:48.573 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:18:48.573 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:18:48.573 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:48.573 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:48.573 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:48.573 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:18:48.573 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:18:48.573 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:48.573 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:48.574 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:48.574 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:48.574 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:18:48.574 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:18:48.574 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:18:48.574 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:18:48.574 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:18:48.574 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:48.574 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:48.574 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:18:48.574 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:18:48.574 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:18:48.574 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:18:48.575 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:48.575 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:18:48.575 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:18:48.575 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:48.575 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:48.575 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:48.575 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:18:48.575 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:18:48.575 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:48.575 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:48.575 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:48.575 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:48.575 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:18:48.576 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:18:48.576 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:18:48.576 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:18:48.576 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:18:48.576 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:48.576 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:48.576 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:18:48.576 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:18:48.576 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:18:48.576 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:18:48.576 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:48.576 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:18:48.576 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:18:48.577 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:48.577 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:48.577 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:48.577 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:18:48.577 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:18:48.577 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:48.577 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:48.577 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:48.577 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:48.577 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:18:48.577 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:18:48.577 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:18:48.577 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:18:48.577 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:18:48.578 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:48.578 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:48.578 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:48.578 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:18:48.578 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:18:48.578 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:48.578 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:18:48.578 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:18:48.578 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:18:48.578 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:48.578 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:18:48.578 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:18:48.578 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:48.578 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:18:48.579 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:18:48.579 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 3 2024-04-22T08:18:48.579 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:48.579 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:18:48.579 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:18:48.579 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:48.579 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:18:48.579 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:18:48.579 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 7 2024-04-22T08:18:48.579 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:48.579 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:48.579 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:48.579 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:18:48.579 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:18:48.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.556+0000 7f8245dd0640 1 -- 172.21.15.40:0/15427029 >> v2:172.21.15.189:6800/30168 conn(0x7f822c037800 msgr2=0x7f822c039cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:48.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.556+0000 7f8245dd0640 1 --2- 172.21.15.40:0/15427029 >> v2:172.21.15.189:6800/30168 conn(0x7f822c037800 0x7f822c039cc0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f82400716f0 tx=0x7f8234006d30 comp rx=0 tx=0).stop 2024-04-22T08:18:48.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.556+0000 7f8245dd0640 10 monclient: shutdown 2024-04-22T08:18:48.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.556+0000 7f8245dd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:18:48.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.556+0000 7f8245dd0640 1 -- 172.21.15.40:0/15427029 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f82401377e0 msgr2=0x7f8240137150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:48.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.556+0000 7f8245dd0640 1 --2- 172.21.15.40:0/15427029 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f82401377e0 0x7f8240137150 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f823800be60 tx=0x7f823800bf50 comp rx=0 tx=0).stop 2024-04-22T08:18:48.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.556+0000 7f823ffff640 1 -- 172.21.15.40:0/15427029 reap_dead start 2024-04-22T08:18:48.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.556+0000 7f8245dd0640 1 -- 172.21.15.40:0/15427029 shutdown_connections 2024-04-22T08:18:48.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.556+0000 7f8245dd0640 1 --2- 172.21.15.40:0/15427029 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f82401377e0 0x7f8240137150 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:48.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.556+0000 7f8245dd0640 1 -- 172.21.15.40:0/15427029 >> 172.21.15.40:0/15427029 conn(0x7f82400a0ae0 msgr2=0x7f824013a970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:18:48.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.556+0000 7f8245dd0640 1 -- 172.21.15.40:0/15427029 shutdown_connections 2024-04-22T08:18:48.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.556+0000 7f8245dd0640 1 -- 172.21.15.40:0/15427029 wait complete. 2024-04-22T08:18:48.582 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-22T08:18:48.582 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-04-22T08:18:48.582 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-22T08:18:48.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.682+0000 7f548d25f640 10 monclient: get_monmap_and_config 2024-04-22T08:18:48.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.682+0000 7f548d25f640 10 monclient: build_initial_monmap 2024-04-22T08:18:48.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.682+0000 7f548d25f640 10 monclient: monmap: 2024-04-22T08:18:48.684 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:18:48.684 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:48.684 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:18:48.684727+0000 2024-04-22T08:18:48.684 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:18:48.684727+0000 2024-04-22T08:18:48.684 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:18:48.684 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:48.684 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:18:48.684 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:18:48.684 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:18:48.685 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:18:48.685 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:18:48.685 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:48.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.683+0000 7f548d25f640 1 Processor -- start 2024-04-22T08:18:48.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.683+0000 7f548d25f640 1 -- start start 2024-04-22T08:18:48.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.683+0000 7f548d25f640 10 monclient: init 2024-04-22T08:18:48.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.684+0000 7f548d25f640 10 monclient: _reopen_session rank -1 2024-04-22T08:18:48.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.684+0000 7f548d25f640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-04-22T08:18:48.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.684+0000 7f548d25f640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f548812fad0 0x7f548812feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:48.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.684+0000 7f548d25f640 10 monclient(hunting): picked mon.noname-c con 0x7f548812fad0 addr v2:172.21.15.189:3300/0 2024-04-22T08:18:48.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.684+0000 7f548d25f640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5488132920 0x7f5488132d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:48.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.684+0000 7f548d25f640 10 monclient(hunting): picked mon.noname-a con 0x7f5488132920 addr v2:172.21.15.40:3300/0 2024-04-22T08:18:48.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.684+0000 7f548d25f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5488133290 addr v1:172.21.15.189:6789/0 2024-04-22T08:18:48.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.684+0000 7f548d25f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f548809ae50 con 0x7f5488133290 2024-04-22T08:18:48.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.684+0000 7f548d25f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:48.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.684+0000 7f548d25f640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f548809efc0 con 0x7f5488132920 2024-04-22T08:18:48.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.684+0000 7f548d25f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:48.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.684+0000 7f548d25f640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f548809d070 con 0x7f548812fad0 2024-04-22T08:18:48.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.684+0000 7f548d25f640 10 monclient(hunting): _renew_subs 2024-04-22T08:18:48.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.684+0000 7f548d25f640 10 monclient(hunting): authenticate will time out at 1202.678931s 2024-04-22T08:18:48.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.684+0000 7f548659c640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f5488132920 msgr2=0x7f5488132d50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:18:48.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.684+0000 7f548659c640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5488132920 0x7f5488132d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:18:48.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.684+0000 7f5486d9d640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f548812fad0 0x7f548812feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:48.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.684+0000 7f5486d9d640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f548812fad0 0x7f548812feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.40:59588/0 (socket says 172.21.15.40:59588) 2024-04-22T08:18:48.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.684+0000 7f5486d9d640 1 -- 172.21.15.40:0/2073060200 learned_addr learned my addr 172.21.15.40:0/2073060200 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:18:48.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.684+0000 7f548759e640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f5488133290 0x7f5488139c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.40:59192/0 (socket says 172.21.15.40:59192) 2024-04-22T08:18:48.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.684+0000 7f5486d9d640 10 monclient(hunting): get_auth_request con 0x7f548812fad0 auth_method 0 2024-04-22T08:18:48.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.684+0000 7f5486d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:48.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.684+0000 7f5486d9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:48.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.684+0000 7f5486d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:48.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.686+0000 7f548759e640 1 -- 172.21.15.40:0/2073060200 >> v1:172.21.15.189:6789/0 conn(0x7f5488133290 legacy=0x7f5488139c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:18:48.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.686+0000 7f5486d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:48.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.686+0000 7f5486d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:48.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.686+0000 7f5486d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:48.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.687+0000 7f5486d9d640 10 monclient(hunting): handle_auth_done global_id 64348 payload 274 2024-04-22T08:18:48.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.687+0000 7f5486d9d640 1 -- 172.21.15.40:0/2073060200 >> v2:172.21.15.40:3300/0 conn(0x7f5488132920 msgr2=0x7f5488132d50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:18:48.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.687+0000 7f5486d9d640 1 --2- 172.21.15.40:0/2073060200 >> v2:172.21.15.40:3300/0 conn(0x7f5488132920 0x7f5488132d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:48.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.687+0000 7f5486d9d640 1 -- 172.21.15.40:0/2073060200 >> v1:172.21.15.189:6789/0 conn(0x7f5488133290 legacy=0x7f5488139c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:48.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.687+0000 7f5486d9d640 10 monclient: _finish_hunting 0 2024-04-22T08:18:48.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.687+0000 7f5486d9d640 1 monclient: found mon.noname-c 2024-04-22T08:18:48.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.687+0000 7f5486d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:18:48.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.687+0000 7f5486d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.189:3300/0 2024-04-22T08:18:48.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.687+0000 7f5486d9d640 1 -- 172.21.15.40:0/2073060200 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f548809d350 con 0x7f548812fad0 2024-04-22T08:18:48.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.687+0000 7f5485d9b640 1 -- 172.21.15.40:0/2073060200 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2175270819 0 0) 0x7f548809ae50 con 0x7f5488133290 2024-04-22T08:18:48.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.687+0000 7f5485d9b640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-04-22T08:18:48.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.690+0000 7f5486d9d640 1 -- 172.21.15.40:0/2073060200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f548812fad0 msgr2=0x7f548812feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:18:48.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.690+0000 7f5486d9d640 1 --2- 172.21.15.40:0/2073060200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f548812fad0 0x7f548812feb0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f548806bb30 tx=0x7f547802e0f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d54cbc28ec7aca22 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:48.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.690+0000 7f5485d9b640 1 -- 172.21.15.40:0/2073060200 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5478036640 con 0x7f548812fad0 2024-04-22T08:18:48.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.690+0000 7f5485d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:48.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.690+0000 7f5485d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:18:48.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.690+0000 7f5485d9b640 10 monclient: dump: 2024-04-22T08:18:48.695 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:48.695 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:48.695 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:48.696 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:48.696 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:48.696 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:48.696 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:48.696 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:48.696 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:48.696 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:48.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.690+0000 7f5485d9b640 10 monclient: _finish_auth 0 2024-04-22T08:18:48.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.690+0000 7f5485d9b640 10 monclient: _check_auth_tickets 2024-04-22T08:18:48.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.690+0000 7f5485d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:18:48.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.690+0000 7f5485d9b640 1 -- 172.21.15.40:0/2073060200 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54780367a0 con 0x7f548812fad0 2024-04-22T08:18:48.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.690+0000 7f548d25f640 5 monclient: authenticate success, global_id 64348 2024-04-22T08:18:48.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.690+0000 7f548d25f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:18:48.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.690+0000 7f5485d9b640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:18:48.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.690+0000 7f548d25f640 10 monclient: get_monmap_and_config success 2024-04-22T08:18:48.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.690+0000 7f548d25f640 4 set_mon_vals no callback set 2024-04-22T08:18:48.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.690+0000 7f5485d9b640 1 -- 172.21.15.40:0/2073060200 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5478036b30 con 0x7f548812fad0 2024-04-22T08:18:48.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.690+0000 7f5485d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:48.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.690+0000 7f5485d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:18:48.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.690+0000 7f5485d9b640 10 monclient: dump: 2024-04-22T08:18:48.697 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:48.697 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:48.697 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:48.697 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:48.697 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:48.697 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:48.697 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:48.697 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:48.697 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:48.697 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:48.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.690+0000 7f548d25f640 10 monclient: shutdown 2024-04-22T08:18:48.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.690+0000 7f548d25f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:18:48.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.690+0000 7f548d25f640 1 -- 172.21.15.40:0/2073060200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f548812fad0 msgr2=0x7f548812feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:48.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.690+0000 7f548d25f640 1 --2- 172.21.15.40:0/2073060200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f548812fad0 0x7f548812feb0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f548806bb30 tx=0x7f547802e0f0 comp rx=0 tx=0).stop 2024-04-22T08:18:48.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.690+0000 7f548d25f640 1 -- 172.21.15.40:0/2073060200 shutdown_connections 2024-04-22T08:18:48.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.690+0000 7f548d25f640 1 --2- 172.21.15.40:0/2073060200 >> v2:172.21.15.40:3300/0 conn(0x7f5488132920 0x7f5488132d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:48.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.690+0000 7f548d25f640 1 --2- 172.21.15.40:0/2073060200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f548812fad0 0x7f548812feb0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:48.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.690+0000 7f548d25f640 1 -- 172.21.15.40:0/2073060200 >> 172.21.15.40:0/2073060200 conn(0x7f54880a0ae0 msgr2=0x7f54880a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:18:48.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.690+0000 7f548d25f640 1 -- 172.21.15.40:0/2073060200 shutdown_connections 2024-04-22T08:18:48.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.690+0000 7f548d25f640 1 -- 172.21.15.40:0/2073060200 wait complete. 2024-04-22T08:18:48.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.691+0000 7f548d25f640 10 monclient: build_initial_monmap 2024-04-22T08:18:48.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.691+0000 7f548d25f640 10 monclient: monmap: 2024-04-22T08:18:48.698 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:18:48.698 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:18:48.699 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:18:48.699 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:18:48.699 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:18:48.699 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:48.699 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:18:48.699 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:18:48.699 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:18:48.699 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:48.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f548d25f640 1 Processor -- start 2024-04-22T08:18:48.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f548d25f640 1 -- start start 2024-04-22T08:18:48.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f548d25f640 10 monclient: init 2024-04-22T08:18:48.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f548d25f640 10 monclient: _reopen_session rank -1 2024-04-22T08:18:48.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f548d25f640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-22T08:18:48.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f548d25f640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f548812fad0 0x7f548806ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:48.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f548d25f640 10 monclient(hunting): picked mon.noname-c con 0x7f548812fad0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:18:48.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f548d25f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5488132920 0x7f5488070450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:48.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f548d25f640 10 monclient(hunting): picked mon.noname-b con 0x7f5488132920 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:18:48.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f548d25f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5488133290 0x7f5488070990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:48.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f548d25f640 10 monclient(hunting): picked mon.noname-a con 0x7f5488133290 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:18:48.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f548d25f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:48.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f548d25f640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548811ca50 con 0x7f5488133290 2024-04-22T08:18:48.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f548d25f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:48.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f548d25f640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5488071870 con 0x7f5488132920 2024-04-22T08:18:48.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f548d25f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:48.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f548d25f640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f548809a700 con 0x7f548812fad0 2024-04-22T08:18:48.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f548d25f640 10 monclient(hunting): _renew_subs 2024-04-22T08:18:48.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f548d25f640 10 monclient(hunting): authenticate will time out at 1202.687308s 2024-04-22T08:18:48.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f548759e640 1 -- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5488133290 msgr2=0x7f5488070990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:18:48.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f548759e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5488133290 0x7f5488070990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:18:48.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f5486d9d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f548812fad0 0x7f548806ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:48.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f5486d9d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f548812fad0 0x7f548806ff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:51268/0 (socket says 172.21.15.40:51268) 2024-04-22T08:18:48.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f5486d9d640 1 -- 172.21.15.40:0/2279473934 learned_addr learned my addr 172.21.15.40:0/2279473934 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:18:48.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f5486d9d640 10 monclient(hunting): get_auth_request con 0x7f548812fad0 auth_method 0 2024-04-22T08:18:48.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f5486d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:48.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f5486d9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:48.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f5486d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:48.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f5486d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:48.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f5486d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:48.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f5486d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:48.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f548659c640 1 --2- 172.21.15.40:0/2279473934 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5488132920 0x7f5488070450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:48.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f548659c640 10 monclient(hunting): get_auth_request con 0x7f5488132920 auth_method 0 2024-04-22T08:18:48.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f548659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:48.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f548659c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:48.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f548659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:48.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f548659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:48.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f548659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:48.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f548659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:48.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f5486d9d640 10 monclient(hunting): handle_auth_done global_id 54542 payload 824 2024-04-22T08:18:48.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f5486d9d640 1 -- 172.21.15.40:0/2279473934 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5488132920 msgr2=0x7f5488070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:48.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f5486d9d640 1 --2- 172.21.15.40:0/2279473934 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5488132920 0x7f5488070450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:48.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f5486d9d640 1 -- 172.21.15.40:0/2279473934 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5488133290 msgr2=0x7f5488070990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:18:48.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f5486d9d640 1 --2- 172.21.15.40:0/2279473934 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5488133290 0x7f5488070990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:48.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f5486d9d640 10 monclient: _finish_hunting 0 2024-04-22T08:18:48.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f5486d9d640 1 monclient: found mon.noname-c 2024-04-22T08:18:48.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f5486d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:18:48.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f5486d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:18:48.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.692+0000 7f5486d9d640 1 -- 172.21.15.40:0/2279473934 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f547800b770 con 0x7f548812fad0 2024-04-22T08:18:48.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.695+0000 7f5486d9d640 1 -- 172.21.15.40:0/2279473934 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f548812fad0 msgr2=0x7f548806ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:18:48.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.695+0000 7f5486d9d640 1 --2- 172.21.15.40:0/2279473934 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f548812fad0 0x7f548806ff10 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f547800b2c0 tx=0x7f5478004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:48.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.695+0000 7f546f7fe640 1 -- 172.21.15.40:0/2279473934 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f547803fa20 con 0x7f548812fad0 2024-04-22T08:18:48.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.695+0000 7f546f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:48.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.695+0000 7f546f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:18:48.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.695+0000 7f546f7fe640 10 monclient: dump: 2024-04-22T08:18:48.706 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:48.706 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:48.706 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:48.706 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:48.706 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:48.706 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:48.706 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:48.706 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:48.707 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:48.707 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:48.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.695+0000 7f546f7fe640 10 monclient: _finish_auth 0 2024-04-22T08:18:48.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.695+0000 7f546f7fe640 10 monclient: _check_auth_tickets 2024-04-22T08:18:48.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.695+0000 7f546f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:18:48.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.695+0000 7f548d25f640 5 monclient: authenticate success, global_id 54542 2024-04-22T08:18:48.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.695+0000 7f546f7fe640 1 -- 172.21.15.40:0/2279473934 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f547803fba0 con 0x7f548812fad0 2024-04-22T08:18:48.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.695+0000 7f548d25f640 10 monclient: _renew_subs 2024-04-22T08:18:48.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.695+0000 7f548d25f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:18:48.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.695+0000 7f548d25f640 1 -- 172.21.15.40:0/2279473934 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f548811c140 con 0x7f548812fad0 2024-04-22T08:18:48.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.695+0000 7f546f7fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:18:48.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.695+0000 7f546f7fe640 1 -- 172.21.15.40:0/2279473934 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f547804a3b0 con 0x7f548812fad0 2024-04-22T08:18:48.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.695+0000 7f546f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:48.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.695+0000 7f546f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:18:48.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.695+0000 7f546f7fe640 10 monclient: dump: 2024-04-22T08:18:48.708 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:48.708 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:48.708 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:48.708 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:48.708 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:48.708 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:48.708 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:48.708 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:48.708 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:48.708 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:48.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.695+0000 7f548d25f640 10 monclient: _renew_subs 2024-04-22T08:18:48.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.695+0000 7f548559a640 4 set_mon_vals no callback set 2024-04-22T08:18:48.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.695+0000 7f548d25f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:18:48.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.695+0000 7f548d25f640 1 -- 172.21.15.40:0/2279473934 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f548811c4d0 con 0x7f548812fad0 2024-04-22T08:18:48.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.696+0000 7f548d25f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:18:48.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.696+0000 7f546f7fe640 1 -- 172.21.15.40:0/2279473934 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f547803d040 con 0x7f548812fad0 2024-04-22T08:18:48.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.696+0000 7f548d25f640 1 --2- 172.21.15.40:0/2279473934 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f548811a6c0 0x7f548806c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:48.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.696+0000 7f548d25f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:48.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.696+0000 7f548d25f640 1 -- 172.21.15.40:0/2279473934 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54880667a0 con 0x7f548811a6c0 2024-04-22T08:18:48.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.696+0000 7f546f7fe640 1 --2- 172.21.15.40:0/2279473934 >> v2:172.21.15.189:6800/30168 conn(0x7f5474037780 0x7f5474039c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:48.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.696+0000 7f546f7fe640 1 -- 172.21.15.40:0/2279473934 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f547806e9d0 con 0x7f548812fad0 2024-04-22T08:18:48.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.696+0000 7f548659c640 1 --2- 172.21.15.40:0/2279473934 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f548811a6c0 0x7f548806c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:48.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.696+0000 7f548659c640 10 monclient: get_auth_request con 0x7f548811a6c0 auth_method 0 2024-04-22T08:18:48.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.696+0000 7f548659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:48.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.696+0000 7f548659c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:48.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.696+0000 7f548659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:48.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.696+0000 7f548759e640 1 --2- 172.21.15.40:0/2279473934 >> v2:172.21.15.189:6800/30168 conn(0x7f5474037780 0x7f5474039c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:48.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.696+0000 7f548759e640 10 monclient: get_auth_request con 0x7f5474037780 auth_method 0 2024-04-22T08:18:48.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.696+0000 7f548659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:48.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.696+0000 7f548659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:48.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.696+0000 7f548659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:48.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.697+0000 7f548659c640 10 monclient(hunting): handle_auth_done global_id 64357 payload 824 2024-04-22T08:18:48.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.697+0000 7f548659c640 1 -- 172.21.15.40:0/2279473934 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f548809c9b0 con 0x7f548811a6c0 2024-04-22T08:18:48.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.699+0000 7f548659c640 1 -- 172.21.15.40:0/2279473934 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f548811a6c0 msgr2=0x7f548806c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:18:48.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.699+0000 7f548659c640 1 --2- 172.21.15.40:0/2279473934 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f548811a6c0 0x7f548806c720 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f547c003980 tx=0x7f547c009aa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:48.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.699+0000 7f548759e640 1 --2- 172.21.15.40:0/2279473934 >> v2:172.21.15.189:6800/30168 conn(0x7f5474037780 0x7f5474039c40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f54800017a0 tx=0x7f5480007cc0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:48.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.699+0000 7f546f7fe640 1 -- 172.21.15.40:0/2279473934 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f547c005a40 con 0x7f548811a6c0 2024-04-22T08:18:48.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.699+0000 7f546f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:48.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.699+0000 7f546f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:18:48.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.699+0000 7f546f7fe640 10 monclient: dump: 2024-04-22T08:18:48.711 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:48.711 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:48.711 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:48.711 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:48.711 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:48.711 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:48.711 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:48.711 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:48.711 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:48.711 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:48.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.699+0000 7f546f7fe640 1 -- 172.21.15.40:0/2279473934 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f547c005ba0 con 0x7f548811a6c0 2024-04-22T08:18:48.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.699+0000 7f546f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:18:48.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.699+0000 7f546f7fe640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:18:48.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.699+0000 7f546f7fe640 1 -- 172.21.15.40:0/2279473934 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f548811a6c0 msgr2=0x7f548806c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:48.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.699+0000 7f546f7fe640 1 --2- 172.21.15.40:0/2279473934 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f548811a6c0 0x7f548806c720 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f547c003980 tx=0x7f547c009aa0 comp rx=0 tx=0).stop 2024-04-22T08:18:48.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.699+0000 7f546f7fe640 1 -- 172.21.15.40:0/2279473934 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f548811a6c0 msgr2=0x7f548806c720 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:18:48.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.699+0000 7f546f7fe640 1 --2- 172.21.15.40:0/2279473934 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f548811a6c0 0x7f548806c720 secure :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f547c003980 tx=0x7f547c009aa0 comp rx=0 tx=0).stop 2024-04-22T08:18:48.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.706+0000 7f548d25f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:18:48.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.706+0000 7f548d25f640 1 --2- 172.21.15.40:0/2279473934 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f548806cc60 0x7f548806d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:18:48.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.706+0000 7f548d25f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:18:48.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.706+0000 7f548d25f640 1 -- 172.21.15.40:0/2279473934 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548811c750 con 0x7f548806cc60 2024-04-22T08:18:48.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.706+0000 7f548659c640 1 --2- 172.21.15.40:0/2279473934 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f548806cc60 0x7f548806d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:18:48.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.706+0000 7f548659c640 10 monclient: get_auth_request con 0x7f548806cc60 auth_method 0 2024-04-22T08:18:48.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.706+0000 7f548659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:18:48.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.706+0000 7f548659c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:18:48.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.706+0000 7f548659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:18:48.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.706+0000 7f548659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:18:48.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.706+0000 7f548659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:18:48.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.706+0000 7f548659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:18:48.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.706+0000 7f548659c640 10 monclient(hunting): handle_auth_done global_id 64360 payload 824 2024-04-22T08:18:48.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.706+0000 7f548659c640 1 -- 172.21.15.40:0/2279473934 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f548804e9b0 con 0x7f548806cc60 2024-04-22T08:18:48.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.709+0000 7f548659c640 1 -- 172.21.15.40:0/2279473934 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f548806cc60 msgr2=0x7f548806d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:18:48.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.709+0000 7f548659c640 1 --2- 172.21.15.40:0/2279473934 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f548806cc60 0x7f548806d070 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f547c003980 tx=0x7f547c0059d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:18:48.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.709+0000 7f546f7fe640 1 -- 172.21.15.40:0/2279473934 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f547c006940 con 0x7f548806cc60 2024-04-22T08:18:48.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.709+0000 7f546f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:18:48.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.709+0000 7f546f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:18:48.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.709+0000 7f546f7fe640 10 monclient: dump: 2024-04-22T08:18:48.715 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:18:48.715 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:18:48.715 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:48.715 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:18:48.715 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:18:48.715 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:18:48.715 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:18:48.715 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:18:48.716 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:18:48.716 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:18:48.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.709+0000 7f546f7fe640 1 -- 172.21.15.40:0/2279473934 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4792 (secure 0 0 0) 0x7f547c01a400 con 0x7f548806cc60 2024-04-22T08:18:48.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.709+0000 7f546f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:18:48.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.709+0000 7f546f7fe640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:18:48.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.709+0000 7f546f7fe640 1 -- 172.21.15.40:0/2279473934 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f548806cc60 msgr2=0x7f548806d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:48.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.709+0000 7f546f7fe640 1 --2- 172.21.15.40:0/2279473934 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f548806cc60 0x7f548806d070 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f547c003980 tx=0x7f547c0059d0 comp rx=0 tx=0).stop 2024-04-22T08:18:48.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.709+0000 7f546f7fe640 1 -- 172.21.15.40:0/2279473934 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f548806cc60 msgr2=0x7f548806d070 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:18:48.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.709+0000 7f546f7fe640 1 --2- 172.21.15.40:0/2279473934 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f548806cc60 0x7f548806d070 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:18:48.716 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:18:48.716 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:18:48.717 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:18:48.717 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "leader", 2024-04-22T08:18:48.717 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 58, 2024-04-22T08:18:48.717 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:18:48.717 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:18:48.717 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:18:48.717 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:18:48.717 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 1, 2024-04-22T08:18:48.717 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:18:48.717 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:18:48.717 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:18:48.717 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:18:48.717 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:18:48.718 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:18:48.718 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:18:48.718 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:18:48.718 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:18:48.718 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:18:48.718 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:18:48.718 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:18:48.718 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:18:48.718 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:18:48.718 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:18:48.718 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:18:48.718 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:18:48.718 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:18:48.718 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:18:48.719 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:18:48.719 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:18:48.719 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:18:48.719 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:18:48.719 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:18:48.719 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:18:48.719 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:18:48.719 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:48.719 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:48.719 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:18:48.719 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:18:48.719 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:18:48.719 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:18:48.719 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:18:48.720 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:18:48.720 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:18:48.720 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:18:48.720 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:18:48.720 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:18:48.720 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:18:48.720 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:18:48.720 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:18:48.720 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:18:48.720 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:18:48.720 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:18:48.720 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:18:48.720 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:18:48.720 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:18:48.721 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:18:48.721 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:18:48.721 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:18:48.721 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:18:48.721 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:18:48.721 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:18:48.721 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:18:48.721 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:18:48.721 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:18:48.721 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:18:48.721 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:48.721 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:18:48.721 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:48.722 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:18:48.722 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:18:48.722 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:18:48.722 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:18:48.722 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:48.722 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:18:48.722 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:18:48.722 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:48.722 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:48.722 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:48.722 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:18:48.722 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:18:48.722 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:48.723 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:48.723 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:48.723 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:48.723 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:18:48.723 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:18:48.723 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:18:48.723 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:18:48.723 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:18:48.723 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:48.723 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:48.723 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:18:48.723 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:18:48.723 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:18:48.723 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:18:48.724 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:48.724 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:18:48.724 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:18:48.724 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:48.724 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:48.724 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:48.724 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:18:48.724 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:18:48.724 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:48.724 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:48.724 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:48.724 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:48.724 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:18:48.724 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:18:48.725 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:18:48.725 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:18:48.725 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:18:48.725 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:48.725 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:48.725 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:18:48.725 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:18:48.725 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:18:48.725 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:18:48.725 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:48.725 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:18:48.725 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:18:48.725 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:48.726 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:48.726 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:48.726 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:18:48.726 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:18:48.726 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:18:48.726 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:48.726 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:48.726 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:48.726 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:18:48.726 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:18:48.726 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:18:48.726 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:18:48.727 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:18:48.727 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:48.727 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:48.727 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:48.727 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:18:48.727 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:18:48.727 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:48.727 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:18:48.727 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:18:48.727 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:18:48.727 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:48.727 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:18:48.727 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:18:48.727 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:48.727 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:18:48.728 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:18:48.728 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 3 2024-04-22T08:18:48.728 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:48.728 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:18:48.728 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:18:48.728 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:48.728 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:18:48.728 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:18:48.728 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 3 2024-04-22T08:18:48.728 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:48.728 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:18:48.728 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": [ 2024-04-22T08:18:48.728 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:18:48.728 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:18:48.729 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:18:48.729 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:18:48.729 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:18:48.729 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:18:48.729 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:18:48.729 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:18:48.729 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:18:48.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.709+0000 7f548d25f640 1 -- 172.21.15.40:0/2279473934 >> v2:172.21.15.189:6800/30168 conn(0x7f5474037780 msgr2=0x7f5474039c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:48.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.709+0000 7f548d25f640 1 --2- 172.21.15.40:0/2279473934 >> v2:172.21.15.189:6800/30168 conn(0x7f5474037780 0x7f5474039c40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f54800017a0 tx=0x7f5480007cc0 comp rx=0 tx=0).stop 2024-04-22T08:18:48.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.709+0000 7f548d25f640 10 monclient: shutdown 2024-04-22T08:18:48.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.709+0000 7f548d25f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:18:48.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.709+0000 7f548d25f640 1 -- 172.21.15.40:0/2279473934 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f548812fad0 msgr2=0x7f548806ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:18:48.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.709+0000 7f548d25f640 1 --2- 172.21.15.40:0/2279473934 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f548812fad0 0x7f548806ff10 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f547800b2c0 tx=0x7f5478004730 comp rx=0 tx=0).stop 2024-04-22T08:18:48.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.709+0000 7f548759e640 1 -- 172.21.15.40:0/2279473934 reap_dead start 2024-04-22T08:18:48.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.710+0000 7f548759e640 1 -- 172.21.15.40:0/2279473934 reap_dead start 2024-04-22T08:18:48.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.710+0000 7f548d25f640 1 -- 172.21.15.40:0/2279473934 shutdown_connections 2024-04-22T08:18:48.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.710+0000 7f548d25f640 1 -- 172.21.15.40:0/2279473934 >> 172.21.15.40:0/2279473934 conn(0x7f54880a0ae0 msgr2=0x7f548812f040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:18:48.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.710+0000 7f548d25f640 1 -- 172.21.15.40:0/2279473934 shutdown_connections 2024-04-22T08:18:48.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:18:48.710+0000 7f548d25f640 1 -- 172.21.15.40:0/2279473934 wait complete. 2024-04-22T08:18:48.732 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-22T08:18:48.732 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-04-22T08:18:50.073 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:18:50.073 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37449]: starting. 2024-04-22T08:18:50.073 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37449]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-37414 2024-04-22T08:18:50.073 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37449]: created object 0... 2024-04-22T08:18:50.073 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37449]: created object 25... 2024-04-22T08:18:50.073 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37449]: created object 49... 2024-04-22T08:18:50.073 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37449]: finishing. 2024-04-22T08:18:50.074 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37449]: shutting down. 2024-04-22T08:18:50.316 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:18:50.316 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37451]: starting. 2024-04-22T08:18:50.316 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37451]: listing objects. 2024-04-22T08:18:50.316 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37451]: listed object 0... 2024-04-22T08:18:50.316 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37451]: listed object 25... 2024-04-22T08:18:50.316 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37451]: saw 50 objects 2024-04-22T08:18:50.316 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37451]: shutting down. 2024-04-22T08:18:50.953 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:18:50.953 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[37450]: starting. 2024-04-22T08:18:50.953 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[37450]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-37414 2024-04-22T08:18:50.953 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[37450]: shutting down. 2024-04-22T08:18:50.954 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:18:50.954 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:18:50.955 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:18:50.957 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:18:53.081 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37520]: starting. 2024-04-22T08:18:53.081 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37520]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-37519 2024-04-22T08:18:53.082 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37520]: created object 0... 2024-04-22T08:18:53.082 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37520]: created object 25... 2024-04-22T08:18:53.082 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37520]: created object 49... 2024-04-22T08:18:53.082 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37520]: finishing. 2024-04-22T08:18:53.082 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37520]: shutting down. 2024-04-22T08:18:53.967 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[37521]: starting. 2024-04-22T08:18:53.967 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[37521]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-37519 2024-04-22T08:18:53.967 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[37521]: shutting down. 2024-04-22T08:18:56.108 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:18:56.109 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37556]: starting. 2024-04-22T08:18:56.109 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37556]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-37519 2024-04-22T08:18:56.109 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37556]: created object 0... 2024-04-22T08:18:56.109 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37556]: created object 25... 2024-04-22T08:18:56.109 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37556]: created object 49... 2024-04-22T08:18:56.109 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37556]: finishing. 2024-04-22T08:18:56.109 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37556]: shutting down. 2024-04-22T08:18:56.149 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:18:56.149 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37558]: starting. 2024-04-22T08:18:56.149 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37558]: listing objects. 2024-04-22T08:18:56.150 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37558]: listed object 0... 2024-04-22T08:18:56.150 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37558]: listed object 25... 2024-04-22T08:18:56.150 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37558]: saw 50 objects 2024-04-22T08:18:56.150 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37558]: shutting down. 2024-04-22T08:18:56.983 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:18:56.983 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[37557]: starting. 2024-04-22T08:18:56.983 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[37557]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-37519 2024-04-22T08:18:56.983 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[37557]: shutting down. 2024-04-22T08:18:56.984 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:18:56.984 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:18:56.985 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:18:56.987 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:18:59.074 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37627]: starting. 2024-04-22T08:18:59.074 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37627]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-37626 2024-04-22T08:18:59.075 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37627]: created object 0... 2024-04-22T08:18:59.075 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37627]: created object 25... 2024-04-22T08:18:59.075 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37627]: created object 49... 2024-04-22T08:18:59.075 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37627]: finishing. 2024-04-22T08:18:59.075 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37627]: shutting down. 2024-04-22T08:18:59.959 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[37628]: starting. 2024-04-22T08:18:59.959 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[37628]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-37626 2024-04-22T08:18:59.959 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[37628]: shutting down. 2024-04-22T08:19:02.337 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:19:02.337 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37663]: starting. 2024-04-22T08:19:02.337 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37663]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-37626 2024-04-22T08:19:02.337 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37663]: created object 0... 2024-04-22T08:19:02.337 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37663]: created object 25... 2024-04-22T08:19:02.337 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37663]: created object 49... 2024-04-22T08:19:02.337 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37663]: finishing. 2024-04-22T08:19:02.337 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37663]: shutting down. 2024-04-22T08:19:02.378 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:19:02.378 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37665]: starting. 2024-04-22T08:19:02.378 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37665]: listing objects. 2024-04-22T08:19:02.378 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37665]: listed object 0... 2024-04-22T08:19:02.378 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37665]: listed object 25... 2024-04-22T08:19:02.378 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37665]: saw 50 objects 2024-04-22T08:19:02.379 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37665]: shutting down. 2024-04-22T08:19:03.016 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:19:03.017 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[37664]: starting. 2024-04-22T08:19:03.017 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[37664]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-37626 2024-04-22T08:19:03.017 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[37664]: shutting down. 2024-04-22T08:19:03.018 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:19:03.018 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:19:03.018 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:19:03.021 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:19:05.180 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37734]: starting. 2024-04-22T08:19:05.180 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37734]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-37733 2024-04-22T08:19:05.180 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37734]: created object 0... 2024-04-22T08:19:05.180 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37734]: created object 25... 2024-04-22T08:19:05.180 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37734]: created object 49... 2024-04-22T08:19:05.180 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37734]: finishing. 2024-04-22T08:19:05.180 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37734]: shutting down. 2024-04-22T08:19:06.066 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[37735]: starting. 2024-04-22T08:19:06.066 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[37735]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-37733 2024-04-22T08:19:06.066 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[37735]: shutting down. 2024-04-22T08:19:08.404 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:19:08.404 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37770]: starting. 2024-04-22T08:19:08.404 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37770]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-37733 2024-04-22T08:19:08.405 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37770]: created object 0... 2024-04-22T08:19:08.405 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37770]: created object 25... 2024-04-22T08:19:08.405 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37770]: created object 49... 2024-04-22T08:19:08.405 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37770]: finishing. 2024-04-22T08:19:08.405 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37770]: shutting down. 2024-04-22T08:19:08.448 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:19:08.448 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37772]: starting. 2024-04-22T08:19:08.449 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37772]: listing objects. 2024-04-22T08:19:08.449 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37772]: listed object 0... 2024-04-22T08:19:08.449 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37772]: listed object 25... 2024-04-22T08:19:08.449 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37772]: saw 50 objects 2024-04-22T08:19:08.449 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37772]: shutting down. 2024-04-22T08:19:08.733 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-04-22T08:19:08.733 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-04-22T08:19:08.733 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-22T08:19:08.733 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-22T08:19:08.736 INFO:tasks.ceph.mon.a:Started 2024-04-22T08:19:08.736 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-04-22T08:19:08.736 INFO:tasks.ceph.mon.c:Sent signal 19 2024-04-22T08:19:08.736 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-04-22T08:19:23.738 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-04-22T08:19:23.738 INFO:tasks.ceph.mon.c:Sent signal 18 2024-04-22T08:19:23.738 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-22T08:19:23.738 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:19:23.773 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:19:23.773 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[37771]: starting. 2024-04-22T08:19:23.773 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[37771]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-37733 2024-04-22T08:19:23.773 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[37771]: shutting down. 2024-04-22T08:19:23.774 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:19:23.774 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:19:23.774 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:19:23.777 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:19:23.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.851+0000 7f632262d640 10 monclient: get_monmap_and_config 2024-04-22T08:19:23.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.851+0000 7f632262d640 10 monclient: build_initial_monmap 2024-04-22T08:19:23.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.852+0000 7f632262d640 10 monclient: monmap: 2024-04-22T08:19:23.851 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:19:23.851 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:23.851 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:19:23.852761+0000 2024-04-22T08:19:23.851 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:19:23.852761+0000 2024-04-22T08:19:23.851 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:19:23.851 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:23.851 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:19:23.852 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:19:23.852 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:19:23.852 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:19:23.852 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:19:23.852 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:23.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.852+0000 7f632262d640 1 Processor -- start 2024-04-22T08:19:23.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.852+0000 7f632262d640 1 -- start start 2024-04-22T08:19:23.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.852+0000 7f632262d640 10 monclient: init 2024-04-22T08:19:23.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.853+0000 7f632262d640 10 monclient: _reopen_session rank -1 2024-04-22T08:19:23.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.853+0000 7f632262d640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-04-22T08:19:23.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.853+0000 7f632262d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f631c132b00 addr v1:172.21.15.40:6789/0 2024-04-22T08:19:23.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.853+0000 7f632262d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f631c132ee0 addr v1:172.21.15.189:6789/0 2024-04-22T08:19:23.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.853+0000 7f632262d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f631c133350 0x7f631c13e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:23.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.853+0000 7f632262d640 10 monclient(hunting): picked mon.noname-b con 0x7f631c133350 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:19:23.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.853+0000 7f632262d640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f631c06c840 con 0x7f631c132b00 2024-04-22T08:19:23.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.853+0000 7f632262d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f631c09a220 con 0x7f631c132ee0 2024-04-22T08:19:23.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.853+0000 7f632262d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:23.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.853+0000 7f632262d640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f631c09a500 con 0x7f631c133350 2024-04-22T08:19:23.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.853+0000 7f632262d640 10 monclient(hunting): _renew_subs 2024-04-22T08:19:23.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.853+0000 7f632262d640 10 monclient(hunting): authenticate will time out at 1237.847138s 2024-04-22T08:19:23.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.853+0000 7f631bfff640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f631c132b00 0x7f631c09af70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:58050/0 (socket says 172.21.15.40:58050) 2024-04-22T08:19:23.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.853+0000 7f6320bca640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f631c133350 0x7f631c13e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:23.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.853+0000 7f631bfff640 1 -- 172.21.15.40:0/2052151978 learned_addr learned my addr 172.21.15.40:0/2052151978 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:19:23.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.853+0000 7f6320bca640 10 monclient(hunting): get_auth_request con 0x7f631c133350 auth_method 0 2024-04-22T08:19:23.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.853+0000 7f6320bca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:23.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.853+0000 7f6320bca640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:23.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.853+0000 7f6320bca640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:23.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.853+0000 7f6320bca640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:23.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.853+0000 7f6320bca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:23.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.853+0000 7f6320bca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:23.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.853+0000 7f6320bca640 10 monclient(hunting): handle_auth_done global_id 54623 payload 274 2024-04-22T08:19:23.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.853+0000 7f6320bca640 1 -- 172.21.15.40:0/2052151978 >> v1:172.21.15.189:6789/0 conn(0x7f631c132ee0 legacy=0x7f631c13af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:19:23.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.853+0000 7f6320bca640 1 -- 172.21.15.40:0/2052151978 >> v1:172.21.15.40:6789/0 conn(0x7f631c132b00 legacy=0x7f631c09af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:19:23.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.853+0000 7f6320bca640 10 monclient: _finish_hunting 0 2024-04-22T08:19:23.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.853+0000 7f6320bca640 1 monclient: found mon.noname-b 2024-04-22T08:19:23.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.853+0000 7f6320bca640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:19:23.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.853+0000 7f6320bca640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:19:23.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.853+0000 7f6320bca640 1 -- 172.21.15.40:0/2052151978 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f631c09a7e0 con 0x7f631c133350 2024-04-22T08:19:23.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.856+0000 7f6320bca640 1 -- 172.21.15.40:0/2052151978 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f631c133350 msgr2=0x7f631c13e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:23.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.856+0000 7f6320bca640 1 --2- 172.21.15.40:0/2052151978 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f631c133350 0x7f631c13e690 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6314009dc0 tx=0x7f631402bce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2bc7ffdbffbe39cd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:23.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.856+0000 7f631affd640 1 -- 172.21.15.40:0/2052151978 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f631402cd80 con 0x7f631c133350 2024-04-22T08:19:23.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.857+0000 7f631affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:23.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.857+0000 7f631affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:19:23.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.857+0000 7f631affd640 10 monclient: dump: 2024-04-22T08:19:23.858 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:23.858 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:23.858 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:23.858 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:23.858 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:23.858 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:23.859 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:23.859 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:23.859 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:23.859 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:23.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.857+0000 7f631affd640 10 monclient: _finish_auth 0 2024-04-22T08:19:23.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.857+0000 7f631affd640 10 monclient: _check_auth_tickets 2024-04-22T08:19:23.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.857+0000 7f631affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:19:23.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.857+0000 7f631affd640 1 -- 172.21.15.40:0/2052151978 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6314034440 con 0x7f631c133350 2024-04-22T08:19:23.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.857+0000 7f632262d640 5 monclient: authenticate success, global_id 54623 2024-04-22T08:19:23.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.857+0000 7f632262d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:19:23.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.857+0000 7f631affd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:19:23.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.857+0000 7f632262d640 10 monclient: get_monmap_and_config success 2024-04-22T08:19:23.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.857+0000 7f631affd640 1 -- 172.21.15.40:0/2052151978 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63140347d0 con 0x7f631c133350 2024-04-22T08:19:23.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.857+0000 7f631affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:23.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.857+0000 7f632262d640 4 set_mon_vals no callback set 2024-04-22T08:19:23.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.857+0000 7f631affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:19:23.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.857+0000 7f631affd640 10 monclient: dump: 2024-04-22T08:19:23.860 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:23.860 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:23.860 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:23.860 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:23.860 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:23.860 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:23.860 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:23.860 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:23.860 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:23.860 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:23.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.857+0000 7f632262d640 10 monclient: shutdown 2024-04-22T08:19:23.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.857+0000 7f632262d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:19:23.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.857+0000 7f632262d640 1 -- 172.21.15.40:0/2052151978 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f631c133350 msgr2=0x7f631c13e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:23.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.857+0000 7f632262d640 1 --2- 172.21.15.40:0/2052151978 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f631c133350 0x7f631c13e690 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6314009dc0 tx=0x7f631402bce0 comp rx=0 tx=0).stop 2024-04-22T08:19:23.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.857+0000 7f632262d640 1 -- 172.21.15.40:0/2052151978 shutdown_connections 2024-04-22T08:19:23.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.857+0000 7f632262d640 1 --2- 172.21.15.40:0/2052151978 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f631c133350 0x7f631c13e690 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:23.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.857+0000 7f632262d640 1 -- 172.21.15.40:0/2052151978 >> 172.21.15.40:0/2052151978 conn(0x7f631c0a5960 msgr2=0x7f631c0a1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:19:23.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.857+0000 7f632262d640 1 -- 172.21.15.40:0/2052151978 shutdown_connections 2024-04-22T08:19:23.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.857+0000 7f632262d640 1 -- 172.21.15.40:0/2052151978 wait complete. 2024-04-22T08:19:23.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.858+0000 7f632262d640 10 monclient: build_initial_monmap 2024-04-22T08:19:23.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.858+0000 7f632262d640 10 monclient: monmap: 2024-04-22T08:19:23.861 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:19:23.861 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:19:23.861 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:19:23.861 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:19:23.861 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:19:23.861 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:23.862 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:19:23.862 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:19:23.862 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:19:23.862 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:23.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.858+0000 7f632262d640 1 Processor -- start 2024-04-22T08:19:23.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.858+0000 7f632262d640 1 -- start start 2024-04-22T08:19:23.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.858+0000 7f632262d640 10 monclient: init 2024-04-22T08:19:23.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f632262d640 10 monclient: _reopen_session rank -1 2024-04-22T08:19:23.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f632262d640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-22T08:19:23.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f632262d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f631c132b00 0x7f631c12b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:23.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f632262d640 10 monclient(hunting): picked mon.noname-a con 0x7f631c132b00 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:19:23.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f632262d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f631c132ee0 0x7f631c12bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:23.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f632262d640 10 monclient(hunting): picked mon.noname-c con 0x7f631c132ee0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:19:23.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f632262d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f631c133350 0x7f631c134880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:23.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f632262d640 10 monclient(hunting): picked mon.noname-b con 0x7f631c133350 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:19:23.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f632262d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:23.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f632262d640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f631c138240 con 0x7f631c132b00 2024-04-22T08:19:23.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f632262d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:23.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f632262d640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f631c138490 con 0x7f631c133350 2024-04-22T08:19:23.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f632262d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:23.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f632262d640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f631c09b8c0 con 0x7f631c132ee0 2024-04-22T08:19:23.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f632262d640 10 monclient(hunting): _renew_subs 2024-04-22T08:19:23.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f632262d640 10 monclient(hunting): authenticate will time out at 1237.852976s 2024-04-22T08:19:23.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f631b7fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f631c132ee0 0x7f631c12bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:23.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f631bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f631c132b00 0x7f631c12b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:23.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f631b7fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f631c132ee0 0x7f631c12bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:36880/0 (socket says 172.21.15.40:36880) 2024-04-22T08:19:23.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f631bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f631c132b00 0x7f631c12b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52834/0 (socket says 172.21.15.40:52834) 2024-04-22T08:19:23.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f631b7fe640 1 -- 172.21.15.40:0/2664227235 learned_addr learned my addr 172.21.15.40:0/2664227235 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:19:23.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f631b7fe640 10 monclient(hunting): get_auth_request con 0x7f631c132ee0 auth_method 0 2024-04-22T08:19:23.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f631b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:23.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f631b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:23.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f631b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:23.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f631bfff640 10 monclient(hunting): get_auth_request con 0x7f631c132b00 auth_method 0 2024-04-22T08:19:23.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f631bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:23.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f631bfff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:23.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f631bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:23.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f6320bca640 1 --2- 172.21.15.40:0/2664227235 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f631c133350 0x7f631c134880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:23.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f6320bca640 10 monclient(hunting): get_auth_request con 0x7f631c133350 auth_method 0 2024-04-22T08:19:23.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f6320bca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:23.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f6320bca640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:23.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f6320bca640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:23.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f631b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:23.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f631b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:23.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f631b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:23.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f631bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:23.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f631bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:23.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f631bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:23.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f6320bca640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:23.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f6320bca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:23.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f6320bca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:23.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f631b7fe640 10 monclient(hunting): handle_auth_done global_id 54626 payload 824 2024-04-22T08:19:23.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f631b7fe640 1 -- 172.21.15.40:0/2664227235 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f631c133350 msgr2=0x7f631c134880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:23.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f631b7fe640 1 --2- 172.21.15.40:0/2664227235 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f631c133350 0x7f631c134880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:23.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f631b7fe640 1 -- 172.21.15.40:0/2664227235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f631c132b00 msgr2=0x7f631c12b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:23.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f631b7fe640 1 --2- 172.21.15.40:0/2664227235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f631c132b00 0x7f631c12b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:23.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f631b7fe640 10 monclient: _finish_hunting 0 2024-04-22T08:19:23.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f631b7fe640 1 monclient: found mon.noname-c 2024-04-22T08:19:23.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f631b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:19:23.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f631b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:19:23.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f631b7fe640 1 -- 172.21.15.40:0/2664227235 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6314009a70 con 0x7f631c132ee0 2024-04-22T08:19:23.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.859+0000 7f631bfff640 1 --2- 172.21.15.40:0/2664227235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f631c132b00 0x7f631c12b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:19:23.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.861+0000 7f631b7fe640 1 -- 172.21.15.40:0/2664227235 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f631c132ee0 msgr2=0x7f631c12bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:23.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.861+0000 7f631b7fe640 1 --2- 172.21.15.40:0/2664227235 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f631c132ee0 0x7f631c12bef0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f630400c460 tx=0x7f630400c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:23.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.862+0000 7f6318ff9640 1 -- 172.21.15.40:0/2664227235 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6304012a40 con 0x7f631c132ee0 2024-04-22T08:19:23.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.862+0000 7f6318ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:23.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.862+0000 7f6318ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:19:23.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.862+0000 7f6318ff9640 10 monclient: dump: 2024-04-22T08:19:23.870 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:23.870 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:23.870 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:23.870 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:23.870 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:23.870 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:23.870 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:23.870 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:23.870 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:23.870 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:23.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.862+0000 7f6318ff9640 10 monclient: _finish_auth 0 2024-04-22T08:19:23.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.862+0000 7f6318ff9640 10 monclient: _check_auth_tickets 2024-04-22T08:19:23.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.862+0000 7f6318ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:19:23.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.862+0000 7f6318ff9640 1 -- 172.21.15.40:0/2664227235 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6304012be0 con 0x7f631c132ee0 2024-04-22T08:19:23.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.862+0000 7f632262d640 5 monclient: authenticate success, global_id 54626 2024-04-22T08:19:23.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.862+0000 7f632262d640 10 monclient: _renew_subs 2024-04-22T08:19:23.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.862+0000 7f632262d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:19:23.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.862+0000 7f632262d640 1 -- 172.21.15.40:0/2664227235 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f631c12c8e0 con 0x7f631c132ee0 2024-04-22T08:19:23.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.862+0000 7f6318ff9640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:19:23.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.862+0000 7f6318ff9640 1 -- 172.21.15.40:0/2664227235 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6304011380 con 0x7f631c132ee0 2024-04-22T08:19:23.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.862+0000 7f6318ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:23.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.862+0000 7f6318ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:19:23.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.862+0000 7f6318ff9640 10 monclient: dump: 2024-04-22T08:19:23.871 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:23.871 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:23.871 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:23.872 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:23.872 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:23.872 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:23.872 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:23.872 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:23.872 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:23.872 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:23.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.862+0000 7f631a7fc640 4 set_mon_vals no callback set 2024-04-22T08:19:23.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.862+0000 7f632262d640 10 monclient: _renew_subs 2024-04-22T08:19:23.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.862+0000 7f632262d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:19:23.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.862+0000 7f632262d640 1 -- 172.21.15.40:0/2664227235 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f631c1360c0 con 0x7f631c132ee0 2024-04-22T08:19:23.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.862+0000 7f6318ff9640 1 -- 172.21.15.40:0/2664227235 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f6304011520 con 0x7f631c132ee0 2024-04-22T08:19:23.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.863+0000 7f632262d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:19:23.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.863+0000 7f632262d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:19:23.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.863+0000 7f632262d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:19:23.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.863+0000 7f632262d640 1 -- 172.21.15.40:0/2664227235 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f631c137d10 con 0x7f631c132ee0 2024-04-22T08:19:23.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.863+0000 7f6318ff9640 1 --2- 172.21.15.40:0/2664227235 >> v2:172.21.15.189:6800/30168 conn(0x7f62fc037800 0x7f62fc039cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:23.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.863+0000 7f6318ff9640 1 -- 172.21.15.40:0/2664227235 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f630404a180 con 0x7f631c132ee0 2024-04-22T08:19:23.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.863+0000 7f631bfff640 1 --2- 172.21.15.40:0/2664227235 >> v2:172.21.15.189:6800/30168 conn(0x7f62fc037800 0x7f62fc039cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:23.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.863+0000 7f631bfff640 10 monclient: get_auth_request con 0x7f62fc037800 auth_method 0 2024-04-22T08:19:23.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.865+0000 7f631bfff640 1 --2- 172.21.15.40:0/2664227235 >> v2:172.21.15.189:6800/30168 conn(0x7f62fc037800 0x7f62fc039cc0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f630c007c90 tx=0x7f630c007be0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:23.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.867+0000 7f6318ff9640 1 -- 172.21.15.40:0/2664227235 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f630401a3b0 con 0x7f631c132ee0 2024-04-22T08:19:23.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.868+0000 7f6318ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:19:23.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:23.868+0000 7f6318ff9640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:19:24.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.118+0000 7f632262d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:19:24.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.118+0000 7f632262d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:19:24.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.118+0000 7f632262d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:19:24.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.118+0000 7f632262d640 1 -- 172.21.15.40:0/2664227235 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f631c1385d0 con 0x7f631c132ee0 2024-04-22T08:19:24.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.118+0000 7f6318ff9640 1 -- 172.21.15.40:0/2664227235 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f63040102e0 con 0x7f631c132ee0 2024-04-22T08:19:24.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.118+0000 7f6318ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:19:24.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.118+0000 7f6318ff9640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:19:24.118 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":66,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:19:24.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.120+0000 7f632262d640 1 -- 172.21.15.40:0/2664227235 >> v2:172.21.15.189:6800/30168 conn(0x7f62fc037800 msgr2=0x7f62fc039cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:24.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.120+0000 7f632262d640 1 --2- 172.21.15.40:0/2664227235 >> v2:172.21.15.189:6800/30168 conn(0x7f62fc037800 0x7f62fc039cc0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f630c007c90 tx=0x7f630c007be0 comp rx=0 tx=0).stop 2024-04-22T08:19:24.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.120+0000 7f632262d640 10 monclient: shutdown 2024-04-22T08:19:24.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.120+0000 7f632262d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:19:24.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.120+0000 7f632262d640 1 -- 172.21.15.40:0/2664227235 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f631c132ee0 msgr2=0x7f631c12bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:24.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.120+0000 7f632262d640 1 --2- 172.21.15.40:0/2664227235 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f631c132ee0 0x7f631c12bef0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f630400c460 tx=0x7f630400c920 comp rx=0 tx=0).stop 2024-04-22T08:19:24.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.120+0000 7f632262d640 1 -- 172.21.15.40:0/2664227235 shutdown_connections 2024-04-22T08:19:24.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.120+0000 7f632262d640 1 --2- 172.21.15.40:0/2664227235 >> v2:172.21.15.189:6800/30168 conn(0x7f62fc037800 0x7f62fc039cc0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:24.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.120+0000 7f632262d640 1 --2- 172.21.15.40:0/2664227235 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f631c133350 0x7f631c134880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:24.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.120+0000 7f632262d640 1 --2- 172.21.15.40:0/2664227235 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f631c132ee0 0x7f631c12bef0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:24.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.120+0000 7f632262d640 1 --2- 172.21.15.40:0/2664227235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f631c132b00 0x7f631c12b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:24.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.120+0000 7f632262d640 1 -- 172.21.15.40:0/2664227235 >> 172.21.15.40:0/2664227235 conn(0x7f631c0a5960 msgr2=0x7f631c0a3880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:19:24.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.120+0000 7f632262d640 1 -- 172.21.15.40:0/2664227235 shutdown_connections 2024-04-22T08:19:24.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.120+0000 7f632262d640 1 -- 172.21.15.40:0/2664227235 wait complete. 2024-04-22T08:19:24.130 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-22T08:19:24.130 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-22T08:19:24.130 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-22T08:19:24.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.231+0000 7fc966f34640 10 monclient: get_monmap_and_config 2024-04-22T08:19:24.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.231+0000 7fc966f34640 10 monclient: build_initial_monmap 2024-04-22T08:19:24.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.231+0000 7fc966f34640 10 monclient: monmap: 2024-04-22T08:19:24.231 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:19:24.231 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:24.231 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:19:24.232014+0000 2024-04-22T08:19:24.231 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:19:24.232014+0000 2024-04-22T08:19:24.231 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:19:24.231 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:24.231 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:19:24.231 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:19:24.231 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:19:24.231 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:19:24.232 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:19:24.232 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:24.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.231+0000 7fc966f34640 1 Processor -- start 2024-04-22T08:19:24.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.231+0000 7fc966f34640 1 -- start start 2024-04-22T08:19:24.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.231+0000 7fc966f34640 10 monclient: init 2024-04-22T08:19:24.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.232+0000 7fc966f34640 10 monclient: _reopen_session rank -1 2024-04-22T08:19:24.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.232+0000 7fc966f34640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-04-22T08:19:24.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.232+0000 7fc966f34640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc9600a1c60 0x7fc9600a31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:24.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.232+0000 7fc966f34640 10 monclient(hunting): picked mon.noname-b con 0x7fc9600a1c60 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:19:24.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.232+0000 7fc966f34640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc9600a2040 0x7fc960135bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:24.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.232+0000 7fc966f34640 10 monclient(hunting): picked mon.noname-c con 0x7fc9600a2040 addr v2:172.21.15.189:3300/0 2024-04-22T08:19:24.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.232+0000 7fc966f34640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc9600a24a0 addr v1:172.21.15.40:6789/0 2024-04-22T08:19:24.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.232+0000 7fc966f34640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9600a41f0 con 0x7fc9600a24a0 2024-04-22T08:19:24.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.232+0000 7fc966f34640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:24.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.232+0000 7fc966f34640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc96009fe90 con 0x7fc9600a2040 2024-04-22T08:19:24.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.232+0000 7fc966f34640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:24.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.232+0000 7fc966f34640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9600a0170 con 0x7fc9600a1c60 2024-04-22T08:19:24.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.232+0000 7fc966f34640 10 monclient(hunting): _renew_subs 2024-04-22T08:19:24.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.232+0000 7fc966f34640 10 monclient(hunting): authenticate will time out at 1238.226271s 2024-04-22T08:19:24.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.232+0000 7fc964cd0640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc9600a1c60 0x7fc9600a31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:24.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.232+0000 7fc9654d1640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fc9600a24a0 0x7fc9601360f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:58052/0 (socket says 172.21.15.40:58052) 2024-04-22T08:19:24.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.232+0000 7fc9654d1640 1 -- 172.21.15.40:0/3187525231 learned_addr learned my addr 172.21.15.40:0/3187525231 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:19:24.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.232+0000 7fc964cd0640 10 monclient(hunting): get_auth_request con 0x7fc9600a1c60 auth_method 0 2024-04-22T08:19:24.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.232+0000 7fc964cd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:24.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.232+0000 7fc964cd0640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:24.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.232+0000 7fc964cd0640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:24.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.232+0000 7fc964cd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:24.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.232+0000 7fc964cd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:24.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.232+0000 7fc964cd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:24.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.232+0000 7fc95ffff640 1 --2- 172.21.15.40:0/3187525231 >> v2:172.21.15.189:3300/0 conn(0x7fc9600a2040 0x7fc960135bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:24.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.232+0000 7fc95ffff640 10 monclient(hunting): get_auth_request con 0x7fc9600a2040 auth_method 0 2024-04-22T08:19:24.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.232+0000 7fc95ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:24.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.232+0000 7fc95ffff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:24.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.232+0000 7fc95ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:24.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.232+0000 7fc964cd0640 10 monclient(hunting): handle_auth_done global_id 54629 payload 274 2024-04-22T08:19:24.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.232+0000 7fc964cd0640 1 -- 172.21.15.40:0/3187525231 >> v2:172.21.15.189:3300/0 conn(0x7fc9600a2040 msgr2=0x7fc960135bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:19:24.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.232+0000 7fc964cd0640 1 --2- 172.21.15.40:0/3187525231 >> v2:172.21.15.189:3300/0 conn(0x7fc9600a2040 0x7fc960135bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:24.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.232+0000 7fc964cd0640 1 -- 172.21.15.40:0/3187525231 >> v1:172.21.15.40:6789/0 conn(0x7fc9600a24a0 legacy=0x7fc9601360f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:19:24.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.232+0000 7fc964cd0640 10 monclient: _finish_hunting 0 2024-04-22T08:19:24.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.232+0000 7fc964cd0640 1 monclient: found mon.noname-b 2024-04-22T08:19:24.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.232+0000 7fc964cd0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:19:24.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.233+0000 7fc964cd0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:19:24.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.233+0000 7fc964cd0640 1 -- 172.21.15.40:0/3187525231 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9600a0450 con 0x7fc9600a1c60 2024-04-22T08:19:24.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.233+0000 7fc95ffff640 1 --2- 172.21.15.40:0/3187525231 >> v2:172.21.15.189:3300/0 conn(0x7fc9600a2040 0x7fc960135bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T08:19:24.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.235+0000 7fc964cd0640 1 -- 172.21.15.40:0/3187525231 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc9600a1c60 msgr2=0x7fc9600a31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:24.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.235+0000 7fc964cd0640 1 --2- 172.21.15.40:0/3187525231 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc9600a1c60 0x7fc9600a31d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc958009d90 tx=0x7fc95802bcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62239653143799e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:24.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.236+0000 7fc95f7fe640 1 -- 172.21.15.40:0/3187525231 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95802ccb0 con 0x7fc9600a1c60 2024-04-22T08:19:24.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.236+0000 7fc95f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:24.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.236+0000 7fc95f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:19:24.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.236+0000 7fc95f7fe640 10 monclient: dump: 2024-04-22T08:19:24.239 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:24.239 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:24.239 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:24.239 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:24.240 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:24.240 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:24.240 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:24.240 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:24.240 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:24.240 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:24.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.236+0000 7fc95f7fe640 10 monclient: _finish_auth 0 2024-04-22T08:19:24.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.236+0000 7fc95f7fe640 10 monclient: _check_auth_tickets 2024-04-22T08:19:24.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.236+0000 7fc95f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:19:24.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.236+0000 7fc95f7fe640 1 -- 172.21.15.40:0/3187525231 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc95802ce10 con 0x7fc9600a1c60 2024-04-22T08:19:24.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.236+0000 7fc966f34640 5 monclient: authenticate success, global_id 54629 2024-04-22T08:19:24.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.236+0000 7fc966f34640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:19:24.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.236+0000 7fc95f7fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:19:24.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.236+0000 7fc966f34640 10 monclient: get_monmap_and_config success 2024-04-22T08:19:24.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.236+0000 7fc95f7fe640 1 -- 172.21.15.40:0/3187525231 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc958034660 con 0x7fc9600a1c60 2024-04-22T08:19:24.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.236+0000 7fc966f34640 4 set_mon_vals no callback set 2024-04-22T08:19:24.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.236+0000 7fc95f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:24.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.236+0000 7fc95f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:19:24.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.236+0000 7fc95f7fe640 10 monclient: dump: 2024-04-22T08:19:24.241 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:24.241 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:24.241 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:24.241 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:24.241 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:24.241 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:24.241 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:24.241 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:24.241 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:24.242 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:24.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.236+0000 7fc966f34640 10 monclient: shutdown 2024-04-22T08:19:24.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.236+0000 7fc966f34640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:19:24.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.236+0000 7fc966f34640 1 -- 172.21.15.40:0/3187525231 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc9600a1c60 msgr2=0x7fc9600a31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:24.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.236+0000 7fc966f34640 1 --2- 172.21.15.40:0/3187525231 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc9600a1c60 0x7fc9600a31d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc958009d90 tx=0x7fc95802bcf0 comp rx=0 tx=0).stop 2024-04-22T08:19:24.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.236+0000 7fc966f34640 1 -- 172.21.15.40:0/3187525231 shutdown_connections 2024-04-22T08:19:24.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.236+0000 7fc966f34640 1 --2- 172.21.15.40:0/3187525231 >> v2:172.21.15.189:3300/0 conn(0x7fc9600a2040 0x7fc960135bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:24.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.236+0000 7fc966f34640 1 --2- 172.21.15.40:0/3187525231 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc9600a1c60 0x7fc9600a31d0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:24.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.236+0000 7fc966f34640 1 -- 172.21.15.40:0/3187525231 >> 172.21.15.40:0/3187525231 conn(0x7fc9600a89a0 msgr2=0x7fc9600a4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:19:24.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.236+0000 7fc966f34640 1 -- 172.21.15.40:0/3187525231 shutdown_connections 2024-04-22T08:19:24.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.236+0000 7fc966f34640 1 -- 172.21.15.40:0/3187525231 wait complete. 2024-04-22T08:19:24.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.237+0000 7fc966f34640 10 monclient: build_initial_monmap 2024-04-22T08:19:24.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.237+0000 7fc966f34640 10 monclient: monmap: 2024-04-22T08:19:24.242 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:19:24.243 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:19:24.243 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:19:24.243 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:19:24.243 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:19:24.243 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:24.243 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:19:24.243 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:19:24.243 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:19:24.243 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:24.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.237+0000 7fc966f34640 1 Processor -- start 2024-04-22T08:19:24.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.237+0000 7fc966f34640 1 -- start start 2024-04-22T08:19:24.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.237+0000 7fc966f34640 10 monclient: init 2024-04-22T08:19:24.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc966f34640 10 monclient: _reopen_session rank -1 2024-04-22T08:19:24.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc966f34640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-22T08:19:24.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc966f34640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9600a1c60 0x7fc9601cac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:24.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc966f34640 10 monclient(hunting): picked mon.noname-b con 0x7fc9600a1c60 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:19:24.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc966f34640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc9600a2040 0x7fc9601cb150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:24.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc966f34640 10 monclient(hunting): picked mon.noname-c con 0x7fc9600a2040 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:19:24.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc966f34640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9600a24a0 0x7fc9601222d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:24.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc966f34640 10 monclient(hunting): picked mon.noname-a con 0x7fc9600a24a0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:19:24.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc966f34640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:24.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc966f34640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc960125ac0 con 0x7fc9600a24a0 2024-04-22T08:19:24.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc966f34640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:24.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc966f34640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9600a4790 con 0x7fc9600a1c60 2024-04-22T08:19:24.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc966f34640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:24.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc966f34640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc960070da0 con 0x7fc9600a2040 2024-04-22T08:19:24.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc966f34640 10 monclient(hunting): _renew_subs 2024-04-22T08:19:24.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc966f34640 10 monclient(hunting): authenticate will time out at 1238.231929s 2024-04-22T08:19:24.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc9654d1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9600a24a0 0x7fc9601222d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:24.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc95ffff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc9600a2040 0x7fc9601cb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:24.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc9654d1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9600a24a0 0x7fc9601222d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52838/0 (socket says 172.21.15.40:52838) 2024-04-22T08:19:24.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc95ffff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc9600a2040 0x7fc9601cb150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:36896/0 (socket says 172.21.15.40:36896) 2024-04-22T08:19:24.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc9654d1640 1 -- 172.21.15.40:0/1853051130 learned_addr learned my addr 172.21.15.40:0/1853051130 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:19:24.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc9654d1640 10 monclient(hunting): get_auth_request con 0x7fc9600a24a0 auth_method 0 2024-04-22T08:19:24.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc9654d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:24.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc9654d1640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:24.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc9654d1640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:24.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc95ffff640 10 monclient(hunting): get_auth_request con 0x7fc9600a2040 auth_method 0 2024-04-22T08:19:24.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc95ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:24.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc95ffff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:24.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc95ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:24.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc9654d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:24.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc9654d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:24.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc9654d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:24.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc95ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:24.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc95ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:24.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc95ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:24.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc964cd0640 1 --2- 172.21.15.40:0/1853051130 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9600a1c60 0x7fc9601cac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:24.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc964cd0640 10 monclient(hunting): get_auth_request con 0x7fc9600a1c60 auth_method 0 2024-04-22T08:19:24.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc964cd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:24.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc964cd0640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:24.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc964cd0640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:24.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc964cd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:24.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc964cd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:24.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc964cd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:24.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc95ffff640 10 monclient(hunting): handle_auth_done global_id 54632 payload 824 2024-04-22T08:19:24.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc95ffff640 1 -- 172.21.15.40:0/1853051130 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9600a1c60 msgr2=0x7fc9601cac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:24.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc95ffff640 1 --2- 172.21.15.40:0/1853051130 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9600a1c60 0x7fc9601cac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:24.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc95ffff640 1 -- 172.21.15.40:0/1853051130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9600a24a0 msgr2=0x7fc9601222d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:24.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc95ffff640 1 --2- 172.21.15.40:0/1853051130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9600a24a0 0x7fc9601222d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:24.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc95ffff640 10 monclient: _finish_hunting 0 2024-04-22T08:19:24.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc95ffff640 1 monclient: found mon.noname-c 2024-04-22T08:19:24.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc95ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:19:24.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc95ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:19:24.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc95ffff640 1 -- 172.21.15.40:0/1853051130 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc958009a70 con 0x7fc9600a2040 2024-04-22T08:19:24.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.238+0000 7fc9654d1640 1 --2- 172.21.15.40:0/1853051130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9600a24a0 0x7fc9601222d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:19:24.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.240+0000 7fc95ffff640 1 -- 172.21.15.40:0/1853051130 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc9600a2040 msgr2=0x7fc9601cb150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:24.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.240+0000 7fc95ffff640 1 --2- 172.21.15.40:0/1853051130 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc9600a2040 0x7fc9601cb150 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc954006fd0 tx=0x7fc95400dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:24.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.241+0000 7fc95d7fa640 1 -- 172.21.15.40:0/1853051130 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc954004030 con 0x7fc9600a2040 2024-04-22T08:19:24.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.241+0000 7fc95d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:24.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.241+0000 7fc95d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:19:24.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.241+0000 7fc95d7fa640 10 monclient: dump: 2024-04-22T08:19:24.253 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:24.253 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:24.253 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:24.253 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:24.253 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:24.253 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:24.254 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:24.254 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:24.254 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:24.254 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:24.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.241+0000 7fc95d7fa640 10 monclient: _finish_auth 0 2024-04-22T08:19:24.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.241+0000 7fc95d7fa640 10 monclient: _check_auth_tickets 2024-04-22T08:19:24.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.241+0000 7fc95d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:19:24.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.241+0000 7fc966f34640 5 monclient: authenticate success, global_id 54632 2024-04-22T08:19:24.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.241+0000 7fc95d7fa640 1 -- 172.21.15.40:0/1853051130 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc954004190 con 0x7fc9600a2040 2024-04-22T08:19:24.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.241+0000 7fc966f34640 10 monclient: _renew_subs 2024-04-22T08:19:24.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.241+0000 7fc966f34640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:19:24.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.241+0000 7fc966f34640 1 -- 172.21.15.40:0/1853051130 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc960126080 con 0x7fc9600a2040 2024-04-22T08:19:24.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.241+0000 7fc95d7fa640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:19:24.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.241+0000 7fc95d7fa640 1 -- 172.21.15.40:0/1853051130 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc954004500 con 0x7fc9600a2040 2024-04-22T08:19:24.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.241+0000 7fc95d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:24.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.241+0000 7fc95d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:19:24.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.241+0000 7fc95d7fa640 10 monclient: dump: 2024-04-22T08:19:24.255 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:24.255 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:24.255 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:24.255 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:24.255 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:24.255 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:24.255 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:24.255 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:24.255 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:24.255 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:24.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.241+0000 7fc95effd640 4 set_mon_vals no callback set 2024-04-22T08:19:24.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.241+0000 7fc966f34640 10 monclient: _renew_subs 2024-04-22T08:19:24.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.241+0000 7fc966f34640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:19:24.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.241+0000 7fc966f34640 1 -- 172.21.15.40:0/1853051130 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc960125890 con 0x7fc9600a2040 2024-04-22T08:19:24.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.241+0000 7fc95d7fa640 1 -- 172.21.15.40:0/1853051130 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fc95400b8d0 con 0x7fc9600a2040 2024-04-22T08:19:24.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.242+0000 7fc966f34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:19:24.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.242+0000 7fc966f34640 1 --2- 172.21.15.40:0/1853051130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9601264a0 0x7fc960122810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:24.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.242+0000 7fc966f34640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:24.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.242+0000 7fc966f34640 1 -- 172.21.15.40:0/1853051130 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc960066650 con 0x7fc9601264a0 2024-04-22T08:19:24.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.242+0000 7fc95d7fa640 1 --2- 172.21.15.40:0/1853051130 >> v2:172.21.15.189:6800/30168 conn(0x7fc94c037850 0x7fc94c039d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:24.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.242+0000 7fc95d7fa640 1 -- 172.21.15.40:0/1853051130 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 3506+0+0 (secure 0 0 0) 0x7fc95404a320 con 0x7fc9600a2040 2024-04-22T08:19:24.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.242+0000 7fc964cd0640 1 --2- 172.21.15.40:0/1853051130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9601264a0 0x7fc960122810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:24.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.242+0000 7fc964cd0640 10 monclient: get_auth_request con 0x7fc9601264a0 auth_method 0 2024-04-22T08:19:24.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.242+0000 7fc964cd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:24.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.242+0000 7fc964cd0640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:24.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.242+0000 7fc964cd0640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:24.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.242+0000 7fc964cd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:24.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.242+0000 7fc964cd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:24.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.242+0000 7fc964cd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:24.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.242+0000 7fc9654d1640 1 --2- 172.21.15.40:0/1853051130 >> v2:172.21.15.189:6800/30168 conn(0x7fc94c037850 0x7fc94c039d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:24.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.242+0000 7fc9654d1640 10 monclient: get_auth_request con 0x7fc94c037850 auth_method 0 2024-04-22T08:19:24.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.242+0000 7fc964cd0640 10 monclient(hunting): handle_auth_done global_id 84117 payload 824 2024-04-22T08:19:24.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.242+0000 7fc964cd0640 1 -- 172.21.15.40:0/1853051130 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc960136800 con 0x7fc9601264a0 2024-04-22T08:19:24.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.244+0000 7fc964cd0640 1 -- 172.21.15.40:0/1853051130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9601264a0 msgr2=0x7fc960122810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:24.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.244+0000 7fc964cd0640 1 --2- 172.21.15.40:0/1853051130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9601264a0 0x7fc960122810 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc960071b30 tx=0x7fc958004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:24.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.245+0000 7fc95d7fa640 1 -- 172.21.15.40:0/1853051130 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95803f800 con 0x7fc9601264a0 2024-04-22T08:19:24.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.245+0000 7fc95d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:24.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.245+0000 7fc95d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:19:24.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.245+0000 7fc95d7fa640 10 monclient: dump: 2024-04-22T08:19:24.258 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:24.258 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:24.258 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:24.258 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:24.258 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:24.258 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:24.259 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:24.259 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:24.259 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:24.259 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:24.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.245+0000 7fc9654d1640 1 --2- 172.21.15.40:0/1853051130 >> v2:172.21.15.189:6800/30168 conn(0x7fc94c037850 0x7fc94c039d10 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fc950007d50 tx=0x7fc950007ca0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:24.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.245+0000 7fc95d7fa640 1 -- 172.21.15.40:0/1853051130 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7fc95803f960 con 0x7fc9601264a0 2024-04-22T08:19:24.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.245+0000 7fc95d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:19:24.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.245+0000 7fc95d7fa640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:19:24.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.245+0000 7fc95d7fa640 1 -- 172.21.15.40:0/1853051130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9601264a0 msgr2=0x7fc960122810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:24.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.245+0000 7fc95d7fa640 1 --2- 172.21.15.40:0/1853051130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9601264a0 0x7fc960122810 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc960071b30 tx=0x7fc958004a40 comp rx=0 tx=0).stop 2024-04-22T08:19:24.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.245+0000 7fc95d7fa640 1 -- 172.21.15.40:0/1853051130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9601264a0 msgr2=0x7fc960122810 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:19:24.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.245+0000 7fc95d7fa640 1 --2- 172.21.15.40:0/1853051130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9601264a0 0x7fc960122810 secure :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc960071b30 tx=0x7fc958004a40 comp rx=0 tx=0).stop 2024-04-22T08:19:24.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.251+0000 7fc966f34640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:19:24.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.251+0000 7fc966f34640 1 --2- 172.21.15.40:0/1853051130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc960122d50 0x7fc960123160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:24.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.251+0000 7fc966f34640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:24.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.251+0000 7fc966f34640 1 -- 172.21.15.40:0/1853051130 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc960125cc0 con 0x7fc960122d50 2024-04-22T08:19:24.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.251+0000 7fc964cd0640 1 --2- 172.21.15.40:0/1853051130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc960122d50 0x7fc960123160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:24.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.251+0000 7fc964cd0640 10 monclient: get_auth_request con 0x7fc960122d50 auth_method 0 2024-04-22T08:19:24.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.251+0000 7fc964cd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:24.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.251+0000 7fc964cd0640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:24.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.251+0000 7fc964cd0640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:24.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.251+0000 7fc964cd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:24.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.251+0000 7fc964cd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:24.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.251+0000 7fc964cd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:24.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.251+0000 7fc964cd0640 10 monclient(hunting): handle_auth_done global_id 84120 payload 824 2024-04-22T08:19:24.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.251+0000 7fc964cd0640 1 -- 172.21.15.40:0/1853051130 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc9601236a0 con 0x7fc960122d50 2024-04-22T08:19:24.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.254+0000 7fc964cd0640 1 -- 172.21.15.40:0/1853051130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc960122d50 msgr2=0x7fc960123160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:24.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.254+0000 7fc964cd0640 1 --2- 172.21.15.40:0/1853051130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc960122d50 0x7fc960123160 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc958000f80 tx=0x7fc958031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:24.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.254+0000 7fc95d7fa640 1 -- 172.21.15.40:0/1853051130 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95804f070 con 0x7fc960122d50 2024-04-22T08:19:24.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.254+0000 7fc95d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:24.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.254+0000 7fc95d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:19:24.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.254+0000 7fc95d7fa640 10 monclient: dump: 2024-04-22T08:19:24.262 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:24.262 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:24.262 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:24.263 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:24.263 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:24.263 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:24.263 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:24.263 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:24.263 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:24.263 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:24.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.254+0000 7fc95d7fa640 1 -- 172.21.15.40:0/1853051130 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4463 (secure 0 0 0) 0x7fc958003f50 con 0x7fc960122d50 2024-04-22T08:19:24.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.254+0000 7fc95d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:19:24.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.254+0000 7fc95d7fa640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:19:24.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.254+0000 7fc95d7fa640 1 -- 172.21.15.40:0/1853051130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc960122d50 msgr2=0x7fc960123160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:24.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.254+0000 7fc95d7fa640 1 --2- 172.21.15.40:0/1853051130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc960122d50 0x7fc960123160 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc958000f80 tx=0x7fc958031000 comp rx=0 tx=0).stop 2024-04-22T08:19:24.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.254+0000 7fc95d7fa640 1 -- 172.21.15.40:0/1853051130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc960122d50 msgr2=0x7fc960123160 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:19:24.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.254+0000 7fc95d7fa640 1 --2- 172.21.15.40:0/1853051130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc960122d50 0x7fc960123160 secure :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc958000f80 tx=0x7fc958031000 comp rx=0 tx=0).stop 2024-04-22T08:19:24.264 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:19:24.264 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:19:24.264 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:19:24.264 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "leader", 2024-04-22T08:19:24.264 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 66, 2024-04-22T08:19:24.264 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:19:24.264 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:19:24.264 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:19:24.264 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:19:24.265 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:19:24.265 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 0, 2024-04-22T08:19:24.265 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:19:24.265 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:19:24.265 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:19:24.265 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:19:24.265 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:19:24.265 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:19:24.265 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:19:24.265 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:19:24.265 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:19:24.265 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:19:24.265 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:19:24.266 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:19:24.266 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:19:24.266 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:19:24.266 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:19:24.266 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:19:24.266 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:19:24.266 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:19:24.266 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:19:24.266 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:19:24.266 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:19:24.266 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:19:24.266 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:19:24.266 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:19:24.267 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:19:24.267 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:19:24.267 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:24.267 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:24.267 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:19:24.267 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:19:24.267 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:19:24.267 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:19:24.267 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:19:24.267 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:19:24.267 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:19:24.267 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:19:24.267 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:19:24.267 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:19:24.268 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:19:24.268 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:19:24.268 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:19:24.268 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:19:24.268 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:19:24.268 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:19:24.268 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:19:24.268 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:19:24.268 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:19:24.268 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:19:24.268 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:19:24.268 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:19:24.268 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:19:24.268 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:19:24.268 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:19:24.269 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:19:24.269 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:19:24.269 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:19:24.269 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:19:24.269 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:24.269 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:19:24.269 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:24.269 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:19:24.269 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:19:24.269 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:19:24.269 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:19:24.269 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:24.269 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:19:24.269 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:19:24.269 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:24.269 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:24.270 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:24.270 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:19:24.270 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:19:24.270 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:24.270 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:24.270 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:24.270 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:24.270 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:19:24.270 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:19:24.270 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:19:24.270 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:19:24.270 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:19:24.270 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:24.270 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:24.271 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:19:24.271 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:19:24.271 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:19:24.271 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:19:24.271 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:24.271 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:19:24.271 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:19:24.271 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:24.271 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:24.271 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:24.271 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:19:24.271 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:19:24.271 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:24.271 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:24.271 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:24.272 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:24.272 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:19:24.272 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:19:24.272 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:19:24.272 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:19:24.272 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:19:24.272 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:24.272 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:24.272 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:19:24.272 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:19:24.272 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:19:24.272 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:19:24.272 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:24.273 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:19:24.273 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:19:24.273 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:24.273 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:24.273 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:24.273 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:19:24.273 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:19:24.273 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:24.273 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:24.273 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:24.273 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:24.273 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:19:24.273 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:19:24.274 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:19:24.274 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:19:24.274 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:19:24.274 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:24.274 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:24.274 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:24.274 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:19:24.274 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:19:24.274 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:24.274 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:19:24.274 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:19:24.274 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:19:24.274 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:24.274 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:19:24.274 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:19:24.275 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:24.275 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:19:24.275 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:19:24.275 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:19:24.275 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:24.275 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:24.275 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:24.275 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:19:24.275 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:19:24.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.254+0000 7fc966f34640 1 -- 172.21.15.40:0/1853051130 >> v2:172.21.15.189:6800/30168 conn(0x7fc94c037850 msgr2=0x7fc94c039d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:24.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.254+0000 7fc966f34640 1 --2- 172.21.15.40:0/1853051130 >> v2:172.21.15.189:6800/30168 conn(0x7fc94c037850 0x7fc94c039d10 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fc950007d50 tx=0x7fc950007ca0 comp rx=0 tx=0).stop 2024-04-22T08:19:24.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.254+0000 7fc966f34640 10 monclient: shutdown 2024-04-22T08:19:24.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.254+0000 7fc966f34640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:19:24.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.254+0000 7fc966f34640 1 -- 172.21.15.40:0/1853051130 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc9600a2040 msgr2=0x7fc9601cb150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:24.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.254+0000 7fc966f34640 1 --2- 172.21.15.40:0/1853051130 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc9600a2040 0x7fc9601cb150 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc954006fd0 tx=0x7fc95400dc60 comp rx=0 tx=0).stop 2024-04-22T08:19:24.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.254+0000 7fc9654d1640 1 -- 172.21.15.40:0/1853051130 reap_dead start 2024-04-22T08:19:24.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.254+0000 7fc9654d1640 1 -- 172.21.15.40:0/1853051130 reap_dead start 2024-04-22T08:19:24.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.254+0000 7fc966f34640 1 -- 172.21.15.40:0/1853051130 shutdown_connections 2024-04-22T08:19:24.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.254+0000 7fc966f34640 1 -- 172.21.15.40:0/1853051130 >> 172.21.15.40:0/1853051130 conn(0x7fc9600a89a0 msgr2=0x7fc9600a0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:19:24.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.255+0000 7fc966f34640 1 -- 172.21.15.40:0/1853051130 shutdown_connections 2024-04-22T08:19:24.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.255+0000 7fc966f34640 1 -- 172.21.15.40:0/1853051130 wait complete. 2024-04-22T08:19:24.278 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:19:24.278 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-22T08:19:24.278 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-22T08:19:24.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.378+0000 7f5a51ad5640 10 monclient: get_monmap_and_config 2024-04-22T08:19:24.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.378+0000 7f5a51ad5640 10 monclient: build_initial_monmap 2024-04-22T08:19:24.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.378+0000 7f5a51ad5640 10 monclient: monmap: 2024-04-22T08:19:24.378 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:19:24.378 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:24.378 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:19:24.379247+0000 2024-04-22T08:19:24.378 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:19:24.379247+0000 2024-04-22T08:19:24.378 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:19:24.378 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:24.378 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:19:24.378 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:19:24.378 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:19:24.378 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:19:24.379 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:19:24.379 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:24.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.379+0000 7f5a51ad5640 1 Processor -- start 2024-04-22T08:19:24.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.379+0000 7f5a51ad5640 1 -- start start 2024-04-22T08:19:24.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.379+0000 7f5a51ad5640 10 monclient: init 2024-04-22T08:19:24.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.379+0000 7f5a51ad5640 10 monclient: _reopen_session rank -1 2024-04-22T08:19:24.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.379+0000 7f5a51ad5640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-04-22T08:19:24.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.379+0000 7f5a51ad5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5a4c135fe0 addr v1:172.21.15.40:6789/0 2024-04-22T08:19:24.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.379+0000 7f5a51ad5640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5a4c0a01c0 0x7f5a4c13ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:24.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.379+0000 7f5a51ad5640 10 monclient(hunting): picked mon.noname-a con 0x7f5a4c0a01c0 addr v2:172.21.15.40:3300/0 2024-04-22T08:19:24.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.379+0000 7f5a51ad5640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5a4c13b3a0 0x7f5a4c13d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:24.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.379+0000 7f5a51ad5640 10 monclient(hunting): picked mon.noname-c con 0x7f5a4c13b3a0 addr v2:172.21.15.189:3300/0 2024-04-22T08:19:24.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.379+0000 7f5a51ad5640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a4c0a05d0 con 0x7f5a4c135fe0 2024-04-22T08:19:24.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.379+0000 7f5a51ad5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:24.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.379+0000 7f5a51ad5640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a4c09f330 con 0x7f5a4c0a01c0 2024-04-22T08:19:24.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.379+0000 7f5a51ad5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:24.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.379+0000 7f5a51ad5640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a4c136ad0 con 0x7f5a4c13b3a0 2024-04-22T08:19:24.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.379+0000 7f5a51ad5640 10 monclient(hunting): _renew_subs 2024-04-22T08:19:24.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.379+0000 7f5a51ad5640 10 monclient(hunting): authenticate will time out at 1238.373439s 2024-04-22T08:19:24.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.379+0000 7f5a4affd640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5a4c0a01c0 0x7f5a4c13ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:24.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.379+0000 7f5a4affd640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5a4c0a01c0 0x7f5a4c13ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52870/0 (socket says 172.21.15.40:52870) 2024-04-22T08:19:24.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.379+0000 7f5a4b7fe640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f5a4c135fe0 0x7f5a4c1363c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:58062/0 (socket says 172.21.15.40:58062) 2024-04-22T08:19:24.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.379+0000 7f5a4affd640 1 -- 172.21.15.40:0/808960053 learned_addr learned my addr 172.21.15.40:0/808960053 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:19:24.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.379+0000 7f5a4affd640 10 monclient(hunting): get_auth_request con 0x7f5a4c0a01c0 auth_method 0 2024-04-22T08:19:24.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.379+0000 7f5a4affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:24.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.379+0000 7f5a4affd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:24.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.379+0000 7f5a4affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:24.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.379+0000 7f5a4affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:24.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.379+0000 7f5a4affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:24.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.379+0000 7f5a4affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:24.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.380+0000 7f5a4bfff640 1 --2- 172.21.15.40:0/808960053 >> v2:172.21.15.189:3300/0 conn(0x7f5a4c13b3a0 0x7f5a4c13d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:24.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.380+0000 7f5a4bfff640 10 monclient(hunting): get_auth_request con 0x7f5a4c13b3a0 auth_method 0 2024-04-22T08:19:24.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.380+0000 7f5a4bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:24.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.380+0000 7f5a4bfff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:24.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.380+0000 7f5a4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:24.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.380+0000 7f5a4affd640 10 monclient(hunting): handle_auth_done global_id 84123 payload 274 2024-04-22T08:19:24.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.380+0000 7f5a4affd640 1 -- 172.21.15.40:0/808960053 >> v2:172.21.15.189:3300/0 conn(0x7f5a4c13b3a0 msgr2=0x7f5a4c13d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:19:24.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.380+0000 7f5a4affd640 1 --2- 172.21.15.40:0/808960053 >> v2:172.21.15.189:3300/0 conn(0x7f5a4c13b3a0 0x7f5a4c13d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:24.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.380+0000 7f5a4affd640 1 -- 172.21.15.40:0/808960053 >> v1:172.21.15.40:6789/0 conn(0x7f5a4c135fe0 legacy=0x7f5a4c1363c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:19:24.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.380+0000 7f5a4affd640 10 monclient: _finish_hunting 0 2024-04-22T08:19:24.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.380+0000 7f5a4affd640 1 monclient: found mon.noname-a 2024-04-22T08:19:24.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.380+0000 7f5a4affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:19:24.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.380+0000 7f5a4affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:19:24.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.380+0000 7f5a4affd640 1 -- 172.21.15.40:0/808960053 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a4c09d7f0 con 0x7f5a4c0a01c0 2024-04-22T08:19:24.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.382+0000 7f5a4affd640 1 -- 172.21.15.40:0/808960053 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a4c0a01c0 msgr2=0x7f5a4c13ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:24.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.382+0000 7f5a4affd640 1 --2- 172.21.15.40:0/808960053 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a4c0a01c0 0x7f5a4c13ae60 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c0a3c10 tx=0x7f5a4002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fb286db849ef85b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:24.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.383+0000 7f5a4a7fc640 1 -- 172.21.15.40:0/808960053 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a40031960 con 0x7f5a4c0a01c0 2024-04-22T08:19:24.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.383+0000 7f5a4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:24.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.383+0000 7f5a4a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:19:24.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.383+0000 7f5a4a7fc640 10 monclient: dump: 2024-04-22T08:19:24.390 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:24.391 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:24.391 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:24.391 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:24.391 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:24.391 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:24.391 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:24.391 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:24.391 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:24.391 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:24.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.383+0000 7f5a4a7fc640 10 monclient: _finish_auth 0 2024-04-22T08:19:24.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.383+0000 7f5a4a7fc640 10 monclient: _check_auth_tickets 2024-04-22T08:19:24.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.383+0000 7f5a4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:19:24.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.383+0000 7f5a51ad5640 5 monclient: authenticate success, global_id 84123 2024-04-22T08:19:24.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.383+0000 7f5a4a7fc640 1 -- 172.21.15.40:0/808960053 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a40031ac0 con 0x7f5a4c0a01c0 2024-04-22T08:19:24.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.383+0000 7f5a51ad5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:19:24.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.383+0000 7f5a4a7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:19:24.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.383+0000 7f5a51ad5640 10 monclient: get_monmap_and_config success 2024-04-22T08:19:24.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.383+0000 7f5a4a7fc640 1 -- 172.21.15.40:0/808960053 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a40031e50 con 0x7f5a4c0a01c0 2024-04-22T08:19:24.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.383+0000 7f5a4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:24.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.383+0000 7f5a4a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:19:24.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.383+0000 7f5a51ad5640 4 set_mon_vals no callback set 2024-04-22T08:19:24.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.383+0000 7f5a4a7fc640 10 monclient: dump: 2024-04-22T08:19:24.392 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:24.392 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:24.392 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:24.392 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:24.392 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:24.392 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:24.393 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:24.393 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:24.393 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:24.393 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:24.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.383+0000 7f5a51ad5640 10 monclient: shutdown 2024-04-22T08:19:24.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.383+0000 7f5a51ad5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:19:24.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.383+0000 7f5a51ad5640 1 -- 172.21.15.40:0/808960053 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a4c0a01c0 msgr2=0x7f5a4c13ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:24.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.383+0000 7f5a51ad5640 1 --2- 172.21.15.40:0/808960053 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a4c0a01c0 0x7f5a4c13ae60 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c0a3c10 tx=0x7f5a4002b7f0 comp rx=0 tx=0).stop 2024-04-22T08:19:24.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.383+0000 7f5a51ad5640 1 -- 172.21.15.40:0/808960053 shutdown_connections 2024-04-22T08:19:24.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.383+0000 7f5a51ad5640 1 --2- 172.21.15.40:0/808960053 >> v2:172.21.15.189:3300/0 conn(0x7f5a4c13b3a0 0x7f5a4c13d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:24.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.383+0000 7f5a51ad5640 1 --2- 172.21.15.40:0/808960053 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a4c0a01c0 0x7f5a4c13ae60 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:24.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.383+0000 7f5a51ad5640 1 -- 172.21.15.40:0/808960053 >> 172.21.15.40:0/808960053 conn(0x7f5a4c0a66a0 msgr2=0x7f5a4c0a2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:19:24.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.383+0000 7f5a51ad5640 1 -- 172.21.15.40:0/808960053 shutdown_connections 2024-04-22T08:19:24.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.384+0000 7f5a51ad5640 1 -- 172.21.15.40:0/808960053 wait complete. 2024-04-22T08:19:24.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.384+0000 7f5a51ad5640 10 monclient: build_initial_monmap 2024-04-22T08:19:24.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.384+0000 7f5a51ad5640 10 monclient: monmap: 2024-04-22T08:19:24.394 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:19:24.394 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:19:24.394 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:19:24.394 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:19:24.394 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:19:24.394 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:24.394 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:19:24.394 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:19:24.394 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:19:24.394 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:24.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.384+0000 7f5a51ad5640 1 Processor -- start 2024-04-22T08:19:24.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a51ad5640 1 -- start start 2024-04-22T08:19:24.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a51ad5640 10 monclient: init 2024-04-22T08:19:24.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a51ad5640 10 monclient: _reopen_session rank -1 2024-04-22T08:19:24.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a51ad5640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-22T08:19:24.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a51ad5640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a4c0a01c0 0x7f5a4c070ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:24.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a51ad5640 10 monclient(hunting): picked mon.noname-c con 0x7f5a4c0a01c0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:19:24.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a51ad5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a4c135fe0 0x7f5a4c1cab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:24.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a51ad5640 10 monclient(hunting): picked mon.noname-b con 0x7f5a4c135fe0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:19:24.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a51ad5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a4c13b3a0 0x7f5a4c1cb060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:24.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a51ad5640 10 monclient(hunting): picked mon.noname-a con 0x7f5a4c13b3a0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:19:24.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a51ad5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:24.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a51ad5640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4c09cce0 con 0x7f5a4c13b3a0 2024-04-22T08:19:24.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a51ad5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:24.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a51ad5640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4c13ad70 con 0x7f5a4c135fe0 2024-04-22T08:19:24.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a51ad5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:24.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a51ad5640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4c121130 con 0x7f5a4c0a01c0 2024-04-22T08:19:24.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a51ad5640 10 monclient(hunting): _renew_subs 2024-04-22T08:19:24.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a51ad5640 10 monclient(hunting): authenticate will time out at 1238.379222s 2024-04-22T08:19:24.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a4bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a4c13b3a0 0x7f5a4c1cb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:24.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a4b7fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a4c0a01c0 0x7f5a4c070ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:24.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a4bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a4c13b3a0 0x7f5a4c1cb060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52878/0 (socket says 172.21.15.40:52878) 2024-04-22T08:19:24.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a4b7fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a4c0a01c0 0x7f5a4c070ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:36912/0 (socket says 172.21.15.40:36912) 2024-04-22T08:19:24.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a4bfff640 1 -- 172.21.15.40:0/2005584234 learned_addr learned my addr 172.21.15.40:0/2005584234 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:19:24.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a4bfff640 10 monclient(hunting): get_auth_request con 0x7f5a4c13b3a0 auth_method 0 2024-04-22T08:19:24.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a4bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:24.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a4bfff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:24.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:24.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a4b7fe640 10 monclient(hunting): get_auth_request con 0x7f5a4c0a01c0 auth_method 0 2024-04-22T08:19:24.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:24.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a4b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:24.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:24.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a4affd640 1 --2- 172.21.15.40:0/2005584234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a4c135fe0 0x7f5a4c1cab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:24.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a4affd640 10 monclient(hunting): get_auth_request con 0x7f5a4c135fe0 auth_method 0 2024-04-22T08:19:24.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a4affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:24.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a4affd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:24.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a4affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:24.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a4bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:24.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a4bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:24.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a4bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:24.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a4b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:24.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a4b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:24.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a4b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:24.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a4affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:24.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a4affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:24.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a4affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:24.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a4bfff640 10 monclient(hunting): handle_auth_done global_id 84126 payload 824 2024-04-22T08:19:24.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a4bfff640 1 -- 172.21.15.40:0/2005584234 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a4c0a01c0 msgr2=0x7f5a4c070ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:24.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a4bfff640 1 --2- 172.21.15.40:0/2005584234 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a4c0a01c0 0x7f5a4c070ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:24.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a4bfff640 1 -- 172.21.15.40:0/2005584234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a4c135fe0 msgr2=0x7f5a4c1cab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:24.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a4bfff640 1 --2- 172.21.15.40:0/2005584234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a4c135fe0 0x7f5a4c1cab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:24.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a4bfff640 10 monclient: _finish_hunting 0 2024-04-22T08:19:24.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a4bfff640 1 monclient: found mon.noname-a 2024-04-22T08:19:24.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a4bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:19:24.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a4bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:19:24.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a4bfff640 1 -- 172.21.15.40:0/2005584234 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a400094d0 con 0x7f5a4c13b3a0 2024-04-22T08:19:24.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a4b7fe640 1 --2- 172.21.15.40:0/2005584234 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a4c0a01c0 0x7f5a4c070ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:19:24.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.385+0000 7f5a4affd640 1 --2- 172.21.15.40:0/2005584234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a4c135fe0 0x7f5a4c1cab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:19:24.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.388+0000 7f5a4bfff640 1 -- 172.21.15.40:0/2005584234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a4c13b3a0 msgr2=0x7f5a4c1cb060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:24.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.388+0000 7f5a4bfff640 1 --2- 172.21.15.40:0/2005584234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a4c13b3a0 0x7f5a4c1cb060 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5a44007e40 tx=0x7f5a4400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:24.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.388+0000 7f5a33fff640 1 -- 172.21.15.40:0/2005584234 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a44013790 con 0x7f5a4c13b3a0 2024-04-22T08:19:24.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.388+0000 7f5a33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:24.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.388+0000 7f5a33fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:19:24.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.388+0000 7f5a33fff640 10 monclient: dump: 2024-04-22T08:19:24.399 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:24.399 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:24.399 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:24.399 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:24.399 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:24.399 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:24.399 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:24.399 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:24.400 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:24.400 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:24.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.388+0000 7f5a33fff640 10 monclient: _finish_auth 0 2024-04-22T08:19:24.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.388+0000 7f5a33fff640 10 monclient: _check_auth_tickets 2024-04-22T08:19:24.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.388+0000 7f5a33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:19:24.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.388+0000 7f5a33fff640 1 -- 172.21.15.40:0/2005584234 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a44013910 con 0x7f5a4c13b3a0 2024-04-22T08:19:24.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.388+0000 7f5a51ad5640 5 monclient: authenticate success, global_id 84126 2024-04-22T08:19:24.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.388+0000 7f5a51ad5640 10 monclient: _renew_subs 2024-04-22T08:19:24.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.388+0000 7f5a51ad5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:19:24.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.388+0000 7f5a51ad5640 1 -- 172.21.15.40:0/2005584234 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a4c070110 con 0x7f5a4c13b3a0 2024-04-22T08:19:24.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.388+0000 7f5a33fff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:19:24.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.388+0000 7f5a33fff640 1 -- 172.21.15.40:0/2005584234 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a44013cc0 con 0x7f5a4c13b3a0 2024-04-22T08:19:24.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.388+0000 7f5a33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:24.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.388+0000 7f5a33fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:19:24.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.388+0000 7f5a33fff640 10 monclient: dump: 2024-04-22T08:19:24.401 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:24.401 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:24.401 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:24.401 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:24.401 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:24.401 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:24.401 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:24.401 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:24.401 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:24.401 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:24.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.388+0000 7f5a51ad5640 10 monclient: _renew_subs 2024-04-22T08:19:24.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.388+0000 7f5a49ffb640 4 set_mon_vals no callback set 2024-04-22T08:19:24.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.388+0000 7f5a51ad5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:19:24.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.388+0000 7f5a51ad5640 1 -- 172.21.15.40:0/2005584234 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a4c0709b0 con 0x7f5a4c13b3a0 2024-04-22T08:19:24.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.389+0000 7f5a33fff640 1 -- 172.21.15.40:0/2005584234 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f5a4401e460 con 0x7f5a4c13b3a0 2024-04-22T08:19:24.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.389+0000 7f5a51ad5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:19:24.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.389+0000 7f5a51ad5640 1 --2- 172.21.15.40:0/2005584234 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a4c06ec80 0x7f5a4c02b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:24.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.389+0000 7f5a51ad5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:24.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.389+0000 7f5a51ad5640 1 -- 172.21.15.40:0/2005584234 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4c066770 con 0x7f5a4c06ec80 2024-04-22T08:19:24.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.389+0000 7f5a33fff640 1 --2- 172.21.15.40:0/2005584234 >> v2:172.21.15.189:6800/30168 conn(0x7f5a380377d0 0x7f5a38039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:24.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.389+0000 7f5a33fff640 1 -- 172.21.15.40:0/2005584234 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f5a4404abf0 con 0x7f5a4c13b3a0 2024-04-22T08:19:24.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.389+0000 7f5a4b7fe640 1 --2- 172.21.15.40:0/2005584234 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a4c06ec80 0x7f5a4c02b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:24.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.389+0000 7f5a4b7fe640 10 monclient: get_auth_request con 0x7f5a4c06ec80 auth_method 0 2024-04-22T08:19:24.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.389+0000 7f5a4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:24.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.389+0000 7f5a4b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:24.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.389+0000 7f5a4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:24.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.389+0000 7f5a4b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:24.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.389+0000 7f5a4b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:24.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.389+0000 7f5a4b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:24.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.389+0000 7f5a4affd640 1 --2- 172.21.15.40:0/2005584234 >> v2:172.21.15.189:6800/30168 conn(0x7f5a380377d0 0x7f5a38039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:24.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.389+0000 7f5a4affd640 10 monclient: get_auth_request con 0x7f5a380377d0 auth_method 0 2024-04-22T08:19:24.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.389+0000 7f5a4b7fe640 10 monclient(hunting): handle_auth_done global_id 54638 payload 824 2024-04-22T08:19:24.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.389+0000 7f5a4b7fe640 1 -- 172.21.15.40:0/2005584234 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5a4c09dba0 con 0x7f5a4c06ec80 2024-04-22T08:19:24.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.391+0000 7f5a4b7fe640 1 -- 172.21.15.40:0/2005584234 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a4c06ec80 msgr2=0x7f5a4c02b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:24.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.391+0000 7f5a4b7fe640 1 --2- 172.21.15.40:0/2005584234 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a4c06ec80 0x7f5a4c02b200 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c005120 tx=0x7f5a3c0055e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:24.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.392+0000 7f5a33fff640 1 -- 172.21.15.40:0/2005584234 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c00fd20 con 0x7f5a4c06ec80 2024-04-22T08:19:24.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.392+0000 7f5a33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:24.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.392+0000 7f5a33fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:19:24.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.392+0000 7f5a33fff640 10 monclient: dump: 2024-04-22T08:19:24.406 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:24.406 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:24.406 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:24.406 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:24.406 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:24.406 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:24.406 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:24.406 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:24.406 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:24.406 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:24.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.392+0000 7f5a4affd640 1 --2- 172.21.15.40:0/2005584234 >> v2:172.21.15.189:6800/30168 conn(0x7f5a380377d0 0x7f5a38039c90 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f5a40033000 tx=0x7f5a40008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:24.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.392+0000 7f5a33fff640 1 -- 172.21.15.40:0/2005584234 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f5a3c00fe80 con 0x7f5a4c06ec80 2024-04-22T08:19:24.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.392+0000 7f5a33fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:19:24.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.392+0000 7f5a33fff640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:19:24.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.392+0000 7f5a33fff640 1 -- 172.21.15.40:0/2005584234 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a4c06ec80 msgr2=0x7f5a4c02b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:24.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.392+0000 7f5a33fff640 1 --2- 172.21.15.40:0/2005584234 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a4c06ec80 0x7f5a4c02b200 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c005120 tx=0x7f5a3c0055e0 comp rx=0 tx=0).stop 2024-04-22T08:19:24.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.392+0000 7f5a33fff640 1 -- 172.21.15.40:0/2005584234 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a4c06ec80 msgr2=0x7f5a4c02b200 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:19:24.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.392+0000 7f5a33fff640 1 --2- 172.21.15.40:0/2005584234 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a4c06ec80 0x7f5a4c02b200 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:24.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.398+0000 7f5a51ad5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:19:24.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.398+0000 7f5a51ad5640 1 --2- 172.21.15.40:0/2005584234 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a4c06f060 0x7f5a4c02b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:24.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.398+0000 7f5a51ad5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:24.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.398+0000 7f5a51ad5640 1 -- 172.21.15.40:0/2005584234 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4c09ef50 con 0x7f5a4c06f060 2024-04-22T08:19:24.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.398+0000 7f5a4b7fe640 1 --2- 172.21.15.40:0/2005584234 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a4c06f060 0x7f5a4c02b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:24.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.398+0000 7f5a4b7fe640 10 monclient: get_auth_request con 0x7f5a4c06f060 auth_method 0 2024-04-22T08:19:24.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.398+0000 7f5a4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:24.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.398+0000 7f5a4b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:24.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.398+0000 7f5a4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:24.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.398+0000 7f5a4b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:24.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.398+0000 7f5a4b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:24.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.398+0000 7f5a4b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:24.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.398+0000 7f5a4b7fe640 10 monclient(hunting): handle_auth_done global_id 54641 payload 824 2024-04-22T08:19:24.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.398+0000 7f5a4b7fe640 1 -- 172.21.15.40:0/2005584234 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5a4c0a34f0 con 0x7f5a4c06f060 2024-04-22T08:19:24.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.401+0000 7f5a4b7fe640 1 -- 172.21.15.40:0/2005584234 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a4c06f060 msgr2=0x7f5a4c02b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:24.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.401+0000 7f5a4b7fe640 1 --2- 172.21.15.40:0/2005584234 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a4c06f060 0x7f5a4c02b740 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c017000 tx=0x7f5a3c004930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:24.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.401+0000 7f5a33fff640 1 -- 172.21.15.40:0/2005584234 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c024070 con 0x7f5a4c06f060 2024-04-22T08:19:24.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.401+0000 7f5a33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:24.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.401+0000 7f5a33fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:19:24.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.401+0000 7f5a33fff640 10 monclient: dump: 2024-04-22T08:19:24.408 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:24.408 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:24.409 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:24.409 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:24.409 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:24.409 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:24.409 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:24.409 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:24.409 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:24.409 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:24.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.401+0000 7f5a33fff640 1 -- 172.21.15.40:0/2005584234 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4631 (secure 0 0 0) 0x7f5a3c01d6c0 con 0x7f5a4c06f060 2024-04-22T08:19:24.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.401+0000 7f5a33fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:19:24.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.401+0000 7f5a33fff640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:19:24.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.401+0000 7f5a33fff640 1 -- 172.21.15.40:0/2005584234 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a4c06f060 msgr2=0x7f5a4c02b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:24.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.401+0000 7f5a33fff640 1 --2- 172.21.15.40:0/2005584234 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a4c06f060 0x7f5a4c02b740 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c017000 tx=0x7f5a3c004930 comp rx=0 tx=0).stop 2024-04-22T08:19:24.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.401+0000 7f5a33fff640 1 -- 172.21.15.40:0/2005584234 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a4c06f060 msgr2=0x7f5a4c02b740 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:19:24.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.401+0000 7f5a33fff640 1 --2- 172.21.15.40:0/2005584234 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a4c06f060 0x7f5a4c02b740 secure :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c017000 tx=0x7f5a3c004930 comp rx=0 tx=0).stop 2024-04-22T08:19:24.410 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:19:24.410 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:19:24.410 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:19:24.410 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:19:24.410 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 66, 2024-04-22T08:19:24.410 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:19:24.410 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:19:24.410 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:19:24.410 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:19:24.411 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:19:24.411 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 0, 2024-04-22T08:19:24.411 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:19:24.411 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:19:24.411 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:19:24.411 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:19:24.411 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:19:24.411 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:19:24.411 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:19:24.411 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:19:24.411 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:19:24.411 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:19:24.411 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:19:24.411 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:19:24.412 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:19:24.412 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:19:24.412 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:19:24.412 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:19:24.412 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:19:24.412 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:19:24.412 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:19:24.412 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:19:24.412 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:19:24.412 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:19:24.412 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:19:24.412 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:19:24.412 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:19:24.412 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:19:24.412 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:24.413 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:24.413 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:19:24.413 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:19:24.413 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:19:24.413 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:19:24.413 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:19:24.413 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:19:24.413 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:19:24.413 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:19:24.413 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:19:24.413 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:19:24.413 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:19:24.413 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:19:24.413 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:19:24.414 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:19:24.414 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:19:24.414 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:19:24.414 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:19:24.414 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:19:24.414 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:19:24.414 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:19:24.414 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:19:24.414 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:19:24.414 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:19:24.414 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:19:24.414 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:19:24.414 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:19:24.414 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:19:24.415 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:19:24.415 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:19:24.415 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:24.415 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:19:24.415 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:24.415 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:19:24.415 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:19:24.415 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:19:24.415 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:19:24.415 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:24.415 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:19:24.415 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:19:24.415 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:24.415 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:24.416 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:24.416 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:19:24.416 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:19:24.416 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:24.416 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:24.416 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:24.416 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:24.416 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:19:24.416 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:19:24.416 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:19:24.416 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:19:24.416 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:19:24.416 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:24.416 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:24.416 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:19:24.417 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:19:24.417 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:19:24.417 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:19:24.417 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:24.417 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:19:24.417 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:19:24.417 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:24.417 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:24.417 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:24.417 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:19:24.417 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:19:24.417 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:24.417 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:24.417 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:24.418 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:24.418 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:19:24.418 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:19:24.418 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:19:24.418 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:19:24.418 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:19:24.418 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:24.418 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:24.418 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:19:24.418 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:19:24.418 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:19:24.418 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:19:24.418 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:24.418 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:19:24.419 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:19:24.419 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:24.419 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:24.419 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:24.419 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:19:24.419 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:19:24.419 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:24.419 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:24.419 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:24.419 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:24.419 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:19:24.419 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:19:24.419 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:19:24.419 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:19:24.419 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:19:24.420 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:24.420 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:24.420 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:24.420 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:19:24.420 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:19:24.420 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:24.420 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:19:24.420 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:19:24.420 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:19:24.420 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:24.420 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:19:24.420 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:19:24.420 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:24.420 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:19:24.421 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:19:24.421 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 3 2024-04-22T08:19:24.421 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:24.421 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:19:24.421 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:19:24.421 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:24.421 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:19:24.421 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:19:24.421 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 4 2024-04-22T08:19:24.421 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:24.421 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:24.421 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:24.421 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:19:24.421 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:19:24.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.402+0000 7f5a51ad5640 1 -- 172.21.15.40:0/2005584234 >> v2:172.21.15.189:6800/30168 conn(0x7f5a380377d0 msgr2=0x7f5a38039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:24.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.402+0000 7f5a51ad5640 1 --2- 172.21.15.40:0/2005584234 >> v2:172.21.15.189:6800/30168 conn(0x7f5a380377d0 0x7f5a38039c90 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f5a40033000 tx=0x7f5a40008030 comp rx=0 tx=0).stop 2024-04-22T08:19:24.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.402+0000 7f5a51ad5640 10 monclient: shutdown 2024-04-22T08:19:24.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.402+0000 7f5a51ad5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:19:24.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.402+0000 7f5a51ad5640 1 -- 172.21.15.40:0/2005584234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a4c13b3a0 msgr2=0x7f5a4c1cb060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:24.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.402+0000 7f5a51ad5640 1 --2- 172.21.15.40:0/2005584234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a4c13b3a0 0x7f5a4c1cb060 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5a44007e40 tx=0x7f5a4400e4b0 comp rx=0 tx=0).stop 2024-04-22T08:19:24.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.402+0000 7f5a4bfff640 1 -- 172.21.15.40:0/2005584234 reap_dead start 2024-04-22T08:19:24.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.402+0000 7f5a4bfff640 1 -- 172.21.15.40:0/2005584234 reap_dead start 2024-04-22T08:19:24.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.402+0000 7f5a51ad5640 1 -- 172.21.15.40:0/2005584234 shutdown_connections 2024-04-22T08:19:24.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.402+0000 7f5a51ad5640 1 -- 172.21.15.40:0/2005584234 >> 172.21.15.40:0/2005584234 conn(0x7f5a4c0a66a0 msgr2=0x7f5a4c136c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:19:24.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.402+0000 7f5a51ad5640 1 -- 172.21.15.40:0/2005584234 shutdown_connections 2024-04-22T08:19:24.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.402+0000 7f5a51ad5640 1 -- 172.21.15.40:0/2005584234 wait complete. 2024-04-22T08:19:24.424 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:19:24.424 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-22T08:19:24.424 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-22T08:19:24.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.524+0000 7f5269fa9640 10 monclient: get_monmap_and_config 2024-04-22T08:19:24.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.524+0000 7f5269fa9640 10 monclient: build_initial_monmap 2024-04-22T08:19:24.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.524+0000 7f5269fa9640 10 monclient: monmap: 2024-04-22T08:19:24.524 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:19:24.524 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:24.524 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:19:24.525449+0000 2024-04-22T08:19:24.525 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:19:24.525449+0000 2024-04-22T08:19:24.525 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:19:24.525 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:24.526 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:19:24.526 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:19:24.526 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:19:24.526 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:19:24.526 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:19:24.526 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:24.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.525+0000 7f5269fa9640 1 Processor -- start 2024-04-22T08:19:24.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.525+0000 7f5269fa9640 1 -- start start 2024-04-22T08:19:24.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.525+0000 7f5269fa9640 10 monclient: init 2024-04-22T08:19:24.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.525+0000 7f5269fa9640 10 monclient: _reopen_session rank -1 2024-04-22T08:19:24.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.525+0000 7f5269fa9640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-04-22T08:19:24.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.525+0000 7f5269fa9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5264136070 addr v1:172.21.15.40:6789/0 2024-04-22T08:19:24.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.525+0000 7f5269fa9640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5264136450 0x7f526413aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:24.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.525+0000 7f5269fa9640 10 monclient(hunting): picked mon.noname-a con 0x7f5264136450 addr v2:172.21.15.40:3300/0 2024-04-22T08:19:24.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.525+0000 7f5269fa9640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f52641368b0 0x7f526413d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:24.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.525+0000 7f5269fa9640 10 monclient(hunting): picked mon.noname-c con 0x7f52641368b0 addr v2:172.21.15.189:3300/0 2024-04-22T08:19:24.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.525+0000 7f5269fa9640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f526406bb50 con 0x7f5264136070 2024-04-22T08:19:24.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.525+0000 7f5269fa9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:24.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.525+0000 7f5269fa9640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f526409a390 con 0x7f5264136450 2024-04-22T08:19:24.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.525+0000 7f5269fa9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:24.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.525+0000 7f5269fa9640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5264099dd0 con 0x7f52641368b0 2024-04-22T08:19:24.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.525+0000 7f5269fa9640 10 monclient(hunting): _renew_subs 2024-04-22T08:19:24.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.525+0000 7f5269fa9640 10 monclient(hunting): authenticate will time out at 1238.519767s 2024-04-22T08:19:24.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.526+0000 7f52637fe640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f5264136070 0x7f526409aa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:58072/0 (socket says 172.21.15.40:58072) 2024-04-22T08:19:24.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.526+0000 7f52637fe640 1 -- 172.21.15.40:0/316442825 learned_addr learned my addr 172.21.15.40:0/316442825 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:19:24.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.526+0000 7f5262ffd640 1 --2- 172.21.15.40:0/316442825 >> v2:172.21.15.40:3300/0 conn(0x7f5264136450 0x7f526413aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:24.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.526+0000 7f5262ffd640 10 monclient(hunting): get_auth_request con 0x7f5264136450 auth_method 0 2024-04-22T08:19:24.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.526+0000 7f5262ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:24.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.526+0000 7f5262ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:24.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.526+0000 7f5262ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:24.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.526+0000 7f5262ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:24.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.526+0000 7f5262ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:24.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.526+0000 7f5262ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:24.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.526+0000 7f5263fff640 1 --2- 172.21.15.40:0/316442825 >> v2:172.21.15.189:3300/0 conn(0x7f52641368b0 0x7f526413d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:24.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.526+0000 7f5263fff640 10 monclient(hunting): get_auth_request con 0x7f52641368b0 auth_method 0 2024-04-22T08:19:24.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.526+0000 7f5263fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:24.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.526+0000 7f5263fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:24.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.526+0000 7f5263fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:24.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.526+0000 7f5262ffd640 10 monclient(hunting): handle_auth_done global_id 84129 payload 274 2024-04-22T08:19:24.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.526+0000 7f5262ffd640 1 -- 172.21.15.40:0/316442825 >> v2:172.21.15.189:3300/0 conn(0x7f52641368b0 msgr2=0x7f526413d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:19:24.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.526+0000 7f5262ffd640 1 --2- 172.21.15.40:0/316442825 >> v2:172.21.15.189:3300/0 conn(0x7f52641368b0 0x7f526413d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:24.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.526+0000 7f5262ffd640 1 -- 172.21.15.40:0/316442825 >> v1:172.21.15.40:6789/0 conn(0x7f5264136070 legacy=0x7f526409aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:19:24.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.526+0000 7f5262ffd640 10 monclient: _finish_hunting 0 2024-04-22T08:19:24.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.526+0000 7f5262ffd640 1 monclient: found mon.noname-a 2024-04-22T08:19:24.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.526+0000 7f5262ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:19:24.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.526+0000 7f5262ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:19:24.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.526+0000 7f5262ffd640 1 -- 172.21.15.40:0/316442825 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f526409a0b0 con 0x7f5264136450 2024-04-22T08:19:24.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.526+0000 7f5263fff640 1 --2- 172.21.15.40:0/316442825 >> v2:172.21.15.189:3300/0 conn(0x7f52641368b0 0x7f526413d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T08:19:24.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.529+0000 7f5262ffd640 1 -- 172.21.15.40:0/316442825 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5264136450 msgr2=0x7f526413aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:24.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.529+0000 7f5262ffd640 1 --2- 172.21.15.40:0/316442825 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5264136450 0x7f526413aed0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f526409bd30 tx=0x7f525802b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90817b32877ad3de server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:24.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.529+0000 7f52627fc640 1 -- 172.21.15.40:0/316442825 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5258031920 con 0x7f5264136450 2024-04-22T08:19:24.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.529+0000 7f52627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:24.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.529+0000 7f52627fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:19:24.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.529+0000 7f52627fc640 10 monclient: dump: 2024-04-22T08:19:24.537 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:24.537 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:24.537 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:24.537 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:24.537 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:24.537 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:24.537 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:24.537 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:24.537 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:24.537 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:24.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.529+0000 7f52627fc640 10 monclient: _finish_auth 0 2024-04-22T08:19:24.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.529+0000 7f52627fc640 10 monclient: _check_auth_tickets 2024-04-22T08:19:24.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.529+0000 7f52627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:19:24.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.529+0000 7f52627fc640 1 -- 172.21.15.40:0/316442825 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f525802f050 con 0x7f5264136450 2024-04-22T08:19:24.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.529+0000 7f52627fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:19:24.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.529+0000 7f52627fc640 1 -- 172.21.15.40:0/316442825 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5258031ca0 con 0x7f5264136450 2024-04-22T08:19:24.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.529+0000 7f5269fa9640 5 monclient: authenticate success, global_id 84129 2024-04-22T08:19:24.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.529+0000 7f5269fa9640 10 monclient: get_monmap_and_config success 2024-04-22T08:19:24.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.529+0000 7f5269fa9640 4 set_mon_vals no callback set 2024-04-22T08:19:24.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.529+0000 7f52627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:24.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.529+0000 7f52627fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:19:24.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.529+0000 7f52627fc640 10 monclient: dump: 2024-04-22T08:19:24.538 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:24.539 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:24.539 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:24.539 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:24.539 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:24.539 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:24.539 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:24.539 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:24.539 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:24.539 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:24.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.529+0000 7f5269fa9640 10 monclient: shutdown 2024-04-22T08:19:24.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.529+0000 7f5269fa9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:19:24.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.529+0000 7f5269fa9640 1 -- 172.21.15.40:0/316442825 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5264136450 msgr2=0x7f526413aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:24.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.529+0000 7f5269fa9640 1 --2- 172.21.15.40:0/316442825 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5264136450 0x7f526413aed0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f526409bd30 tx=0x7f525802b7b0 comp rx=0 tx=0).stop 2024-04-22T08:19:24.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.529+0000 7f5269fa9640 1 -- 172.21.15.40:0/316442825 shutdown_connections 2024-04-22T08:19:24.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.529+0000 7f5269fa9640 1 --2- 172.21.15.40:0/316442825 >> v2:172.21.15.189:3300/0 conn(0x7f52641368b0 0x7f526413d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:24.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.529+0000 7f5269fa9640 1 --2- 172.21.15.40:0/316442825 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5264136450 0x7f526413aed0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:24.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.530+0000 7f5269fa9640 1 -- 172.21.15.40:0/316442825 >> 172.21.15.40:0/316442825 conn(0x7f52640a1b60 msgr2=0x7f52640a1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:19:24.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.530+0000 7f5269fa9640 1 -- 172.21.15.40:0/316442825 shutdown_connections 2024-04-22T08:19:24.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.530+0000 7f5269fa9640 1 -- 172.21.15.40:0/316442825 wait complete. 2024-04-22T08:19:24.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.530+0000 7f5269fa9640 10 monclient: build_initial_monmap 2024-04-22T08:19:24.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.530+0000 7f5269fa9640 10 monclient: monmap: 2024-04-22T08:19:24.540 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:19:24.540 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:19:24.540 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:19:24.540 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:19:24.540 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:19:24.540 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:24.540 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:19:24.541 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:19:24.541 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:19:24.541 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:24.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.530+0000 7f5269fa9640 1 Processor -- start 2024-04-22T08:19:24.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.531+0000 7f5269fa9640 1 -- start start 2024-04-22T08:19:24.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.531+0000 7f5269fa9640 10 monclient: init 2024-04-22T08:19:24.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.531+0000 7f5269fa9640 10 monclient: _reopen_session rank -1 2024-04-22T08:19:24.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.531+0000 7f5269fa9640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-22T08:19:24.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.531+0000 7f5269fa9640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5264136070 0x7f5264134780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:24.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.531+0000 7f5269fa9640 10 monclient(hunting): picked mon.noname-c con 0x7f5264136070 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:19:24.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.531+0000 7f5269fa9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5264136450 0x7f5264134cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:24.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.531+0000 7f5269fa9640 10 monclient(hunting): picked mon.noname-b con 0x7f5264136450 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:19:24.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.531+0000 7f5269fa9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5264138610 0x7f5264138a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:24.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.531+0000 7f5269fa9640 10 monclient(hunting): picked mon.noname-a con 0x7f5264138610 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:19:24.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.531+0000 7f5269fa9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:24.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.531+0000 7f5269fa9640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f526413c210 con 0x7f5264138610 2024-04-22T08:19:24.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.531+0000 7f5269fa9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:24.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.531+0000 7f5269fa9640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f526413a120 con 0x7f5264136450 2024-04-22T08:19:24.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.531+0000 7f5269fa9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:24.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.531+0000 7f5269fa9640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f526413d130 con 0x7f5264136070 2024-04-22T08:19:24.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.531+0000 7f5269fa9640 10 monclient(hunting): _renew_subs 2024-04-22T08:19:24.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.531+0000 7f5269fa9640 10 monclient(hunting): authenticate will time out at 1238.525388s 2024-04-22T08:19:24.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.531+0000 7f52637fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5264136070 0x7f5264134780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:24.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.531+0000 7f5263fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5264138610 0x7f5264138a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:24.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.531+0000 7f52637fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5264136070 0x7f5264134780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:36930/0 (socket says 172.21.15.40:36930) 2024-04-22T08:19:24.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.531+0000 7f52637fe640 1 -- 172.21.15.40:0/395949381 learned_addr learned my addr 172.21.15.40:0/395949381 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:19:24.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.531+0000 7f52637fe640 10 monclient(hunting): get_auth_request con 0x7f5264136070 auth_method 0 2024-04-22T08:19:24.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.531+0000 7f52637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:24.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.531+0000 7f52637fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:24.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.531+0000 7f52637fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:24.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.531+0000 7f5263fff640 10 monclient(hunting): get_auth_request con 0x7f5264138610 auth_method 0 2024-04-22T08:19:24.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.531+0000 7f5263fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:24.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.531+0000 7f5263fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:24.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.531+0000 7f5263fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:24.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.531+0000 7f52637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:24.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.531+0000 7f52637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:24.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.531+0000 7f52637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:24.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.531+0000 7f5263fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:24.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.531+0000 7f5263fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:24.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.531+0000 7f5263fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:24.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.531+0000 7f5262ffd640 1 --2- 172.21.15.40:0/395949381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5264136450 0x7f5264134cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:24.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.531+0000 7f5262ffd640 10 monclient(hunting): get_auth_request con 0x7f5264136450 auth_method 0 2024-04-22T08:19:24.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.531+0000 7f5262ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:24.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.531+0000 7f5262ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:24.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.531+0000 7f5262ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:24.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.532+0000 7f52637fe640 10 monclient(hunting): handle_auth_done global_id 54644 payload 824 2024-04-22T08:19:24.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.532+0000 7f52637fe640 1 -- 172.21.15.40:0/395949381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5264136450 msgr2=0x7f5264134cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:24.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.532+0000 7f52637fe640 1 --2- 172.21.15.40:0/395949381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5264136450 0x7f5264134cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:24.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.532+0000 7f52637fe640 1 -- 172.21.15.40:0/395949381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5264138610 msgr2=0x7f5264138a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:24.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.532+0000 7f52637fe640 1 --2- 172.21.15.40:0/395949381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5264138610 0x7f5264138a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:24.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.532+0000 7f52637fe640 10 monclient: _finish_hunting 0 2024-04-22T08:19:24.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.532+0000 7f52637fe640 1 monclient: found mon.noname-c 2024-04-22T08:19:24.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.532+0000 7f52637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:19:24.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.532+0000 7f52637fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:19:24.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.532+0000 7f52637fe640 1 -- 172.21.15.40:0/395949381 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5258009400 con 0x7f5264136070 2024-04-22T08:19:24.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.532+0000 7f5262ffd640 1 --2- 172.21.15.40:0/395949381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5264136450 0x7f5264134cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T08:19:24.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.532+0000 7f5263fff640 1 --2- 172.21.15.40:0/395949381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5264138610 0x7f5264138a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:19:24.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.534+0000 7f52637fe640 1 -- 172.21.15.40:0/395949381 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5264136070 msgr2=0x7f5264134780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:24.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.534+0000 7f52637fe640 1 --2- 172.21.15.40:0/395949381 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5264136070 0x7f5264134780 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f525400cc10 tx=0x7f525400a430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:24.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.534+0000 7f524bfff640 1 -- 172.21.15.40:0/395949381 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5254012400 con 0x7f5264136070 2024-04-22T08:19:24.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.534+0000 7f524bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:24.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.534+0000 7f524bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:19:24.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.534+0000 7f524bfff640 10 monclient: dump: 2024-04-22T08:19:24.551 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:24.551 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:24.551 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:24.551 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:24.551 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:24.551 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:24.552 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:24.552 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:24.552 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:24.552 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:24.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.534+0000 7f524bfff640 10 monclient: _finish_auth 0 2024-04-22T08:19:24.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.534+0000 7f524bfff640 10 monclient: _check_auth_tickets 2024-04-22T08:19:24.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.534+0000 7f524bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:19:24.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.534+0000 7f524bfff640 1 -- 172.21.15.40:0/395949381 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52540125a0 con 0x7f5264136070 2024-04-22T08:19:24.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.534+0000 7f524bfff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:19:24.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.534+0000 7f524bfff640 1 -- 172.21.15.40:0/395949381 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5254012950 con 0x7f5264136070 2024-04-22T08:19:24.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.534+0000 7f5269fa9640 5 monclient: authenticate success, global_id 54644 2024-04-22T08:19:24.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.534+0000 7f5269fa9640 10 monclient: _renew_subs 2024-04-22T08:19:24.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.534+0000 7f5269fa9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:19:24.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.534+0000 7f5269fa9640 1 -- 172.21.15.40:0/395949381 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f526409b470 con 0x7f5264136070 2024-04-22T08:19:24.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.534+0000 7f5261ffb640 4 set_mon_vals no callback set 2024-04-22T08:19:24.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.534+0000 7f524bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:24.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.534+0000 7f524bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:19:24.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.534+0000 7f524bfff640 10 monclient: dump: 2024-04-22T08:19:24.553 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:24.553 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:24.553 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:24.553 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:24.553 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:24.553 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:24.553 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:24.553 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:24.553 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:24.554 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:24.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.534+0000 7f5269fa9640 10 monclient: _renew_subs 2024-04-22T08:19:24.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.534+0000 7f5269fa9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:19:24.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.534+0000 7f5269fa9640 1 -- 172.21.15.40:0/395949381 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f526413bc50 con 0x7f5264136070 2024-04-22T08:19:24.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.535+0000 7f524bfff640 1 -- 172.21.15.40:0/395949381 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f5254012b30 con 0x7f5264136070 2024-04-22T08:19:24.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.535+0000 7f5269fa9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:19:24.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.535+0000 7f5269fa9640 1 --2- 172.21.15.40:0/395949381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52641368b0 0x7f526402b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:24.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.535+0000 7f5269fa9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:24.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.535+0000 7f5269fa9640 1 -- 172.21.15.40:0/395949381 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52640667a0 con 0x7f52641368b0 2024-04-22T08:19:24.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.535+0000 7f524bfff640 1 --2- 172.21.15.40:0/395949381 >> v2:172.21.15.189:6800/30168 conn(0x7f5250037800 0x7f5250039cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:24.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.535+0000 7f524bfff640 1 -- 172.21.15.40:0/395949381 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f525404a8e0 con 0x7f5264136070 2024-04-22T08:19:24.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.535+0000 7f5263fff640 1 --2- 172.21.15.40:0/395949381 >> v2:172.21.15.189:6800/30168 conn(0x7f5250037800 0x7f5250039cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:24.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.535+0000 7f5262ffd640 1 --2- 172.21.15.40:0/395949381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52641368b0 0x7f526402b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:24.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.535+0000 7f5262ffd640 10 monclient: get_auth_request con 0x7f52641368b0 auth_method 0 2024-04-22T08:19:24.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.535+0000 7f5262ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:24.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.535+0000 7f5262ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:24.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.535+0000 7f5262ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:24.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.536+0000 7f5263fff640 10 monclient: get_auth_request con 0x7f5250037800 auth_method 0 2024-04-22T08:19:24.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.536+0000 7f5262ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:24.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.536+0000 7f5262ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:24.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.536+0000 7f5262ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:24.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.536+0000 7f5262ffd640 10 monclient(hunting): handle_auth_done global_id 64471 payload 824 2024-04-22T08:19:24.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.536+0000 7f5262ffd640 1 -- 172.21.15.40:0/395949381 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f526413d960 con 0x7f52641368b0 2024-04-22T08:19:24.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.538+0000 7f5262ffd640 1 -- 172.21.15.40:0/395949381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52641368b0 msgr2=0x7f526402b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:24.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.538+0000 7f5262ffd640 1 --2- 172.21.15.40:0/395949381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52641368b0 0x7f526402b200 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5258035000 tx=0x7f52580046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:24.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.538+0000 7f5263fff640 1 --2- 172.21.15.40:0/395949381 >> v2:172.21.15.189:6800/30168 conn(0x7f5250037800 0x7f5250039cc0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f525c006fd0 tx=0x7f525c004730 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:24.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.539+0000 7f524bfff640 1 -- 172.21.15.40:0/395949381 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f525803db40 con 0x7f52641368b0 2024-04-22T08:19:24.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.539+0000 7f524bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:24.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.539+0000 7f524bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:19:24.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.539+0000 7f524bfff640 10 monclient: dump: 2024-04-22T08:19:24.556 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:24.556 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:24.556 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:24.556 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:24.556 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:24.556 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:24.556 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:24.556 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:24.556 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:24.556 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:24.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.539+0000 7f524bfff640 1 -- 172.21.15.40:0/395949381 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f525803dca0 con 0x7f52641368b0 2024-04-22T08:19:24.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.539+0000 7f524bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:19:24.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.539+0000 7f524bfff640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:19:24.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.539+0000 7f524bfff640 1 -- 172.21.15.40:0/395949381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52641368b0 msgr2=0x7f526402b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:24.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.539+0000 7f524bfff640 1 --2- 172.21.15.40:0/395949381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52641368b0 0x7f526402b200 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5258035000 tx=0x7f52580046c0 comp rx=0 tx=0).stop 2024-04-22T08:19:24.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.539+0000 7f524bfff640 1 -- 172.21.15.40:0/395949381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52641368b0 msgr2=0x7f526402b200 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:19:24.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.539+0000 7f524bfff640 1 --2- 172.21.15.40:0/395949381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52641368b0 0x7f526402b200 secure :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5258035000 tx=0x7f52580046c0 comp rx=0 tx=0).stop 2024-04-22T08:19:24.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.545+0000 7f5269fa9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:19:24.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.545+0000 7f5269fa9640 1 --2- 172.21.15.40:0/395949381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5264139780 0x7f526402b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:24.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.545+0000 7f5269fa9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:24.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.545+0000 7f5269fa9640 1 -- 172.21.15.40:0/395949381 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f526413be40 con 0x7f5264139780 2024-04-22T08:19:24.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.546+0000 7f5262ffd640 1 --2- 172.21.15.40:0/395949381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5264139780 0x7f526402b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:24.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.546+0000 7f5262ffd640 10 monclient: get_auth_request con 0x7f5264139780 auth_method 0 2024-04-22T08:19:24.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.546+0000 7f5262ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:24.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.546+0000 7f5262ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:24.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.546+0000 7f5262ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:24.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.546+0000 7f5262ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:24.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.546+0000 7f5262ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:24.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.546+0000 7f5262ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:24.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.546+0000 7f5262ffd640 10 monclient(hunting): handle_auth_done global_id 64474 payload 824 2024-04-22T08:19:24.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.546+0000 7f5262ffd640 1 -- 172.21.15.40:0/395949381 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f526404e9b0 con 0x7f5264139780 2024-04-22T08:19:24.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.549+0000 7f5262ffd640 1 -- 172.21.15.40:0/395949381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5264139780 msgr2=0x7f526402b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:24.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.549+0000 7f5262ffd640 1 --2- 172.21.15.40:0/395949381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5264139780 0x7f526402b740 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5258000c00 tx=0x7f5258031000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:24.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.549+0000 7f524bfff640 1 -- 172.21.15.40:0/395949381 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5258050480 con 0x7f5264139780 2024-04-22T08:19:24.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.549+0000 7f524bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:24.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.549+0000 7f524bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:19:24.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.549+0000 7f524bfff640 10 monclient: dump: 2024-04-22T08:19:24.558 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:24.558 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:24.559 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:24.559 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:24.559 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:24.559 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:24.559 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:24.559 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:24.559 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:24.559 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:24.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.549+0000 7f524bfff640 1 -- 172.21.15.40:0/395949381 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4801 (secure 0 0 0) 0x7f52580505e0 con 0x7f5264139780 2024-04-22T08:19:24.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.549+0000 7f524bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:19:24.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.549+0000 7f524bfff640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:19:24.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.549+0000 7f524bfff640 1 -- 172.21.15.40:0/395949381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5264139780 msgr2=0x7f526402b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:24.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.549+0000 7f524bfff640 1 --2- 172.21.15.40:0/395949381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5264139780 0x7f526402b740 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5258000c00 tx=0x7f5258031000 comp rx=0 tx=0).stop 2024-04-22T08:19:24.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.549+0000 7f524bfff640 1 -- 172.21.15.40:0/395949381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5264139780 msgr2=0x7f526402b740 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:19:24.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.549+0000 7f524bfff640 1 --2- 172.21.15.40:0/395949381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5264139780 0x7f526402b740 secure :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5258000c00 tx=0x7f5258031000 comp rx=0 tx=0).stop 2024-04-22T08:19:24.560 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:19:24.560 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:19:24.560 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:19:24.561 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:19:24.561 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 66, 2024-04-22T08:19:24.561 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:19:24.561 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:19:24.561 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:19:24.561 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:19:24.561 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:19:24.561 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 0, 2024-04-22T08:19:24.561 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:19:24.561 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:19:24.562 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:19:24.562 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:19:24.562 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:19:24.562 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:19:24.562 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:19:24.562 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:19:24.562 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:19:24.562 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:19:24.562 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:19:24.562 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:19:24.563 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:19:24.563 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:19:24.563 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:19:24.563 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:19:24.563 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:19:24.563 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:19:24.563 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:19:24.563 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:19:24.563 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:19:24.563 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:19:24.564 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:19:24.564 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:19:24.564 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:19:24.564 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:19:24.564 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:24.564 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:24.564 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:19:24.564 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:19:24.564 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:19:24.564 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:19:24.565 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:19:24.565 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:19:24.565 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:19:24.565 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:19:24.565 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:19:24.565 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:19:24.565 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:19:24.565 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:19:24.565 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:19:24.565 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:19:24.566 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:19:24.566 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:19:24.566 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:19:24.566 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:19:24.566 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:19:24.566 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:19:24.566 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:19:24.566 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:19:24.566 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:19:24.566 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:19:24.567 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:19:24.567 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:19:24.567 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:19:24.567 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:19:24.567 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:19:24.567 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:24.567 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:19:24.567 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:24.567 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:19:24.567 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:19:24.568 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:19:24.568 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:19:24.568 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:24.568 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:19:24.568 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:19:24.568 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:24.568 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:24.568 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:24.568 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:19:24.568 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:19:24.568 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:24.569 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:24.569 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:24.569 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:24.569 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:19:24.569 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:19:24.569 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:19:24.569 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:19:24.569 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:19:24.569 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:24.569 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:24.570 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:19:24.570 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:19:24.570 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:19:24.570 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:19:24.570 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:24.570 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:19:24.570 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:19:24.570 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:24.570 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:24.570 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:24.570 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:19:24.570 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:19:24.570 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:24.570 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:24.571 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:24.571 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:24.571 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:19:24.571 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:19:24.571 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:19:24.571 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:19:24.571 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:19:24.571 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:24.571 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:24.571 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:19:24.571 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:19:24.571 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:19:24.571 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:19:24.571 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:24.572 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:19:24.572 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:19:24.572 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:24.572 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:24.572 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:24.572 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:19:24.572 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:19:24.572 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:24.572 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:24.572 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:24.572 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:24.572 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:19:24.572 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:19:24.572 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:19:24.572 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:19:24.573 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:19:24.573 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:24.573 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:24.573 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:24.573 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:19:24.573 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:19:24.573 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:24.573 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:19:24.573 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:19:24.573 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:19:24.573 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:24.573 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:19:24.573 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:19:24.573 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:24.574 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:19:24.574 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:19:24.574 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 3 2024-04-22T08:19:24.574 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:24.574 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:19:24.574 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:19:24.574 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:24.574 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:19:24.574 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:19:24.574 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 4 2024-04-22T08:19:24.574 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:24.574 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:19:24.574 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": [ 2024-04-22T08:19:24.574 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:24.575 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:19:24.575 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:19:24.575 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:19:24.575 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:24.575 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:24.575 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:24.575 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:19:24.575 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:19:24.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.550+0000 7f5269fa9640 1 -- 172.21.15.40:0/395949381 >> v2:172.21.15.189:6800/30168 conn(0x7f5250037800 msgr2=0x7f5250039cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:24.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.550+0000 7f5269fa9640 1 --2- 172.21.15.40:0/395949381 >> v2:172.21.15.189:6800/30168 conn(0x7f5250037800 0x7f5250039cc0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f525c006fd0 tx=0x7f525c004730 comp rx=0 tx=0).stop 2024-04-22T08:19:24.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.550+0000 7f5269fa9640 10 monclient: shutdown 2024-04-22T08:19:24.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.550+0000 7f5269fa9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:19:24.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.550+0000 7f5269fa9640 1 -- 172.21.15.40:0/395949381 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5264136070 msgr2=0x7f5264134780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:24.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.550+0000 7f5269fa9640 1 --2- 172.21.15.40:0/395949381 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5264136070 0x7f5264134780 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f525400cc10 tx=0x7f525400a430 comp rx=0 tx=0).stop 2024-04-22T08:19:24.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.550+0000 7f5263fff640 1 -- 172.21.15.40:0/395949381 reap_dead start 2024-04-22T08:19:24.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.550+0000 7f5269fa9640 1 -- 172.21.15.40:0/395949381 shutdown_connections 2024-04-22T08:19:24.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.550+0000 7f5269fa9640 1 --2- 172.21.15.40:0/395949381 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5264136070 0x7f5264134780 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:24.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.550+0000 7f5269fa9640 1 -- 172.21.15.40:0/395949381 >> 172.21.15.40:0/395949381 conn(0x7f52640a1b60 msgr2=0x7f526406e030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:19:24.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.550+0000 7f5269fa9640 1 -- 172.21.15.40:0/395949381 shutdown_connections 2024-04-22T08:19:24.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.550+0000 7f5269fa9640 1 -- 172.21.15.40:0/395949381 wait complete. 2024-04-22T08:19:24.577 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:19:24.578 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-22T08:19:24.578 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-22T08:19:24.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.677+0000 7fb0d39eb640 10 monclient: get_monmap_and_config 2024-04-22T08:19:24.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.677+0000 7fb0d39eb640 10 monclient: build_initial_monmap 2024-04-22T08:19:24.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.677+0000 7fb0d39eb640 10 monclient: monmap: 2024-04-22T08:19:24.677 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:19:24.677 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:24.677 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:19:24.677840+0000 2024-04-22T08:19:24.677 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:19:24.677840+0000 2024-04-22T08:19:24.678 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:19:24.678 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:24.678 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:19:24.678 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:19:24.678 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:19:24.678 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:19:24.678 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:19:24.678 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:24.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.677+0000 7fb0d39eb640 1 Processor -- start 2024-04-22T08:19:24.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.677+0000 7fb0d39eb640 1 -- start start 2024-04-22T08:19:24.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.677+0000 7fb0d39eb640 10 monclient: init 2024-04-22T08:19:24.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d39eb640 10 monclient: _reopen_session rank -1 2024-04-22T08:19:24.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d39eb640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-04-22T08:19:24.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d39eb640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb0cc09f2e0 0x7fb0cc09f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:24.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d39eb640 10 monclient(hunting): picked mon.noname-a con 0x7fb0cc09f2e0 addr v2:172.21.15.40:3300/0 2024-04-22T08:19:24.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d39eb640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0cc09b760 0x7fb0cc09fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:24.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d39eb640 10 monclient(hunting): picked mon.noname-b con 0x7fb0cc09b760 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:19:24.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d39eb640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb0cc136840 0x7fb0cc138c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:24.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d39eb640 10 monclient(hunting): picked mon.noname-c con 0x7fb0cc136840 addr v2:172.21.15.189:3300/0 2024-04-22T08:19:24.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d39eb640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:24.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d39eb640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0cc09cf70 con 0x7fb0cc09f2e0 2024-04-22T08:19:24.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d39eb640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:24.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d39eb640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0cc0a0140 con 0x7fb0cc136840 2024-04-22T08:19:24.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d39eb640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:24.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d39eb640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0cc09c6d0 con 0x7fb0cc09b760 2024-04-22T08:19:24.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d39eb640 10 monclient(hunting): _renew_subs 2024-04-22T08:19:24.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d39eb640 10 monclient(hunting): authenticate will time out at 1238.672158s 2024-04-22T08:19:24.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d1787640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb0cc09f2e0 0x7fb0cc09f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:24.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d0f86640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0cc09b760 0x7fb0cc09fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:24.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d0f86640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0cc09b760 0x7fb0cc09fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:36940/0 (socket says 172.21.15.40:36940) 2024-04-22T08:19:24.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d0f86640 1 -- 172.21.15.40:0/2269560532 learned_addr learned my addr 172.21.15.40:0/2269560532 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:19:24.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d1787640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb0cc09f2e0 0x7fb0cc09f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52916/0 (socket says 172.21.15.40:52916) 2024-04-22T08:19:24.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d0f86640 10 monclient(hunting): get_auth_request con 0x7fb0cc09b760 auth_method 0 2024-04-22T08:19:24.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d0f86640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:24.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d0f86640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:24.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d0f86640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:24.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d1787640 10 monclient(hunting): get_auth_request con 0x7fb0cc09f2e0 auth_method 0 2024-04-22T08:19:24.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d1787640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:24.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d1787640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:24.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d1787640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:24.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d1787640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:24.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d1787640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:24.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d1787640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:24.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d0f86640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:24.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d0f86640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:24.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d0f86640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:24.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d1f88640 1 --2- 172.21.15.40:0/2269560532 >> v2:172.21.15.189:3300/0 conn(0x7fb0cc136840 0x7fb0cc138c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:24.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d1f88640 10 monclient(hunting): get_auth_request con 0x7fb0cc136840 auth_method 0 2024-04-22T08:19:24.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d1f88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:24.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d1f88640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:24.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d1f88640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:24.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d1787640 10 monclient(hunting): handle_auth_done global_id 84135 payload 274 2024-04-22T08:19:24.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d1787640 1 -- 172.21.15.40:0/2269560532 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0cc09b760 msgr2=0x7fb0cc09fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:19:24.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d1787640 1 --2- 172.21.15.40:0/2269560532 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0cc09b760 0x7fb0cc09fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:24.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d1787640 1 -- 172.21.15.40:0/2269560532 >> v2:172.21.15.189:3300/0 conn(0x7fb0cc136840 msgr2=0x7fb0cc138c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:19:24.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d1787640 1 --2- 172.21.15.40:0/2269560532 >> v2:172.21.15.189:3300/0 conn(0x7fb0cc136840 0x7fb0cc138c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:24.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d1787640 10 monclient: _finish_hunting 0 2024-04-22T08:19:24.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d1787640 1 monclient: found mon.noname-a 2024-04-22T08:19:24.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d1787640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:19:24.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d1787640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:19:24.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d1787640 1 -- 172.21.15.40:0/2269560532 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0cc09c9b0 con 0x7fb0cc09f2e0 2024-04-22T08:19:24.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.678+0000 7fb0d0f86640 1 --2- 172.21.15.40:0/2269560532 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0cc09b760 0x7fb0cc09fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:19:24.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.681+0000 7fb0d1787640 1 -- 172.21.15.40:0/2269560532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0cc09f2e0 msgr2=0x7fb0cc09f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:24.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.681+0000 7fb0d1787640 1 --2- 172.21.15.40:0/2269560532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0cc09f2e0 0x7fb0cc09f6c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc09d460 tx=0x7fb0c802b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c73c256b5e8beec8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:24.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.681+0000 7fb0c3fff640 1 -- 172.21.15.40:0/2269560532 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c8030920 con 0x7fb0cc09f2e0 2024-04-22T08:19:24.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.681+0000 7fb0c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:24.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.681+0000 7fb0c3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:19:24.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.681+0000 7fb0c3fff640 10 monclient: dump: 2024-04-22T08:19:24.686 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:24.686 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:24.686 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:24.686 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:24.687 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:24.687 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:24.687 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:24.687 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:24.687 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:24.687 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:24.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.681+0000 7fb0c3fff640 10 monclient: _finish_auth 0 2024-04-22T08:19:24.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.681+0000 7fb0c3fff640 10 monclient: _check_auth_tickets 2024-04-22T08:19:24.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.681+0000 7fb0c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:19:24.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.681+0000 7fb0c3fff640 1 -- 172.21.15.40:0/2269560532 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0c802e050 con 0x7fb0cc09f2e0 2024-04-22T08:19:24.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.681+0000 7fb0c3fff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:19:24.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.681+0000 7fb0c3fff640 1 -- 172.21.15.40:0/2269560532 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c8030ca0 con 0x7fb0cc09f2e0 2024-04-22T08:19:24.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.681+0000 7fb0d39eb640 5 monclient: authenticate success, global_id 84135 2024-04-22T08:19:24.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.681+0000 7fb0d39eb640 10 monclient: get_monmap_and_config success 2024-04-22T08:19:24.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.681+0000 7fb0d39eb640 4 set_mon_vals no callback set 2024-04-22T08:19:24.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.681+0000 7fb0c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:24.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.681+0000 7fb0c3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:19:24.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.681+0000 7fb0c3fff640 10 monclient: dump: 2024-04-22T08:19:24.688 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:24.688 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:24.688 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:24.688 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:24.688 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:24.688 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:24.688 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:24.688 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:24.688 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:24.688 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:24.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.682+0000 7fb0d39eb640 10 monclient: shutdown 2024-04-22T08:19:24.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.682+0000 7fb0d39eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:19:24.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.682+0000 7fb0d39eb640 1 -- 172.21.15.40:0/2269560532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0cc09f2e0 msgr2=0x7fb0cc09f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:24.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.682+0000 7fb0d39eb640 1 --2- 172.21.15.40:0/2269560532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0cc09f2e0 0x7fb0cc09f6c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc09d460 tx=0x7fb0c802b710 comp rx=0 tx=0).stop 2024-04-22T08:19:24.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.682+0000 7fb0d39eb640 1 -- 172.21.15.40:0/2269560532 shutdown_connections 2024-04-22T08:19:24.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.682+0000 7fb0d39eb640 1 --2- 172.21.15.40:0/2269560532 >> v2:172.21.15.189:3300/0 conn(0x7fb0cc136840 0x7fb0cc138c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:24.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.682+0000 7fb0d39eb640 1 --2- 172.21.15.40:0/2269560532 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0cc09b760 0x7fb0cc09fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:24.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.682+0000 7fb0d39eb640 1 --2- 172.21.15.40:0/2269560532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0cc09f2e0 0x7fb0cc09f6c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:24.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.682+0000 7fb0d39eb640 1 -- 172.21.15.40:0/2269560532 >> 172.21.15.40:0/2269560532 conn(0x7fb0cc0a0b50 msgr2=0x7fb0cc0a0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:19:24.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.682+0000 7fb0d39eb640 1 -- 172.21.15.40:0/2269560532 shutdown_connections 2024-04-22T08:19:24.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.682+0000 7fb0d39eb640 1 -- 172.21.15.40:0/2269560532 wait complete. 2024-04-22T08:19:24.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.683+0000 7fb0d39eb640 10 monclient: build_initial_monmap 2024-04-22T08:19:24.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.683+0000 7fb0d39eb640 10 monclient: monmap: 2024-04-22T08:19:24.689 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:19:24.690 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:19:24.690 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:19:24.690 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:19:24.690 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:19:24.690 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:24.690 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:19:24.690 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:19:24.690 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:19:24.690 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:24.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.683+0000 7fb0d39eb640 1 Processor -- start 2024-04-22T08:19:24.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.683+0000 7fb0d39eb640 1 -- start start 2024-04-22T08:19:24.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.683+0000 7fb0d39eb640 10 monclient: init 2024-04-22T08:19:24.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.683+0000 7fb0d39eb640 10 monclient: _reopen_session rank -1 2024-04-22T08:19:24.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.683+0000 7fb0d39eb640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-22T08:19:24.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.683+0000 7fb0d39eb640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0cc09f2e0 0x7fb0cc1c86b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:24.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.683+0000 7fb0d39eb640 10 monclient(hunting): picked mon.noname-c con 0x7fb0cc09f2e0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:19:24.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.683+0000 7fb0d39eb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0cc136840 0x7fb0cc135ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:24.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.683+0000 7fb0d39eb640 10 monclient(hunting): picked mon.noname-a con 0x7fb0cc136840 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:19:24.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.683+0000 7fb0d39eb640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0cc11b600 0x7fb0cc11ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:24.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.683+0000 7fb0d39eb640 10 monclient(hunting): picked mon.noname-b con 0x7fb0cc11b600 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:19:24.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.683+0000 7fb0d39eb640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:24.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.683+0000 7fb0d39eb640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0cc09bbc0 con 0x7fb0cc136840 2024-04-22T08:19:24.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.683+0000 7fb0d39eb640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:24.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.683+0000 7fb0d39eb640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0cc0a3640 con 0x7fb0cc11b600 2024-04-22T08:19:24.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.683+0000 7fb0d39eb640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:24.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.683+0000 7fb0d39eb640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0cc11d1c0 con 0x7fb0cc09f2e0 2024-04-22T08:19:24.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.683+0000 7fb0d39eb640 10 monclient(hunting): _renew_subs 2024-04-22T08:19:24.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.683+0000 7fb0d39eb640 10 monclient(hunting): authenticate will time out at 1238.677745s 2024-04-22T08:19:24.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.684+0000 7fb0d1787640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0cc09f2e0 0x7fb0cc1c86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:24.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.684+0000 7fb0d1787640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0cc09f2e0 0x7fb0cc1c86b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:36948/0 (socket says 172.21.15.40:36948) 2024-04-22T08:19:24.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.684+0000 7fb0d1787640 1 -- 172.21.15.40:0/3723207859 learned_addr learned my addr 172.21.15.40:0/3723207859 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:19:24.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.684+0000 7fb0d1787640 10 monclient(hunting): get_auth_request con 0x7fb0cc09f2e0 auth_method 0 2024-04-22T08:19:24.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.684+0000 7fb0d1787640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:24.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.684+0000 7fb0d1787640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:24.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.684+0000 7fb0d1787640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:24.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.684+0000 7fb0d0f86640 1 --2- 172.21.15.40:0/3723207859 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0cc136840 0x7fb0cc135ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:24.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.684+0000 7fb0d0f86640 10 monclient(hunting): get_auth_request con 0x7fb0cc136840 auth_method 0 2024-04-22T08:19:24.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.684+0000 7fb0d0f86640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:24.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.684+0000 7fb0d0f86640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:24.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.684+0000 7fb0d0f86640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:24.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.684+0000 7fb0d1787640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:24.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.684+0000 7fb0d1787640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:24.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.684+0000 7fb0d1787640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:24.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.684+0000 7fb0d0f86640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:24.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.684+0000 7fb0d0f86640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:24.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.684+0000 7fb0d0f86640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:24.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.684+0000 7fb0d1787640 10 monclient(hunting): handle_auth_done global_id 54650 payload 824 2024-04-22T08:19:24.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.684+0000 7fb0d1787640 1 -- 172.21.15.40:0/3723207859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0cc11b600 msgr2=0x7fb0cc11ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:24.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.684+0000 7fb0d1787640 1 --2- 172.21.15.40:0/3723207859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0cc11b600 0x7fb0cc11ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:24.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.684+0000 7fb0d1787640 1 -- 172.21.15.40:0/3723207859 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0cc136840 msgr2=0x7fb0cc135ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:24.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.684+0000 7fb0d1787640 1 --2- 172.21.15.40:0/3723207859 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0cc136840 0x7fb0cc135ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:24.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.684+0000 7fb0d1787640 10 monclient: _finish_hunting 0 2024-04-22T08:19:24.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.684+0000 7fb0d1787640 1 monclient: found mon.noname-c 2024-04-22T08:19:24.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.684+0000 7fb0d1787640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:19:24.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.684+0000 7fb0d1787640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:19:24.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.684+0000 7fb0d1787640 1 -- 172.21.15.40:0/3723207859 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0bc0094d0 con 0x7fb0cc09f2e0 2024-04-22T08:19:24.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.686+0000 7fb0d1787640 1 -- 172.21.15.40:0/3723207859 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0cc09f2e0 msgr2=0x7fb0cc1c86b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:24.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.686+0000 7fb0d1787640 1 --2- 172.21.15.40:0/3723207859 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0cc09f2e0 0x7fb0cc1c86b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb0c8009890 tx=0x7fb0c80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:24.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.686+0000 7fb0c1ffb640 1 -- 172.21.15.40:0/3723207859 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c803caf0 con 0x7fb0cc09f2e0 2024-04-22T08:19:24.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.686+0000 7fb0c1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:24.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.686+0000 7fb0c1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:19:24.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.686+0000 7fb0c1ffb640 10 monclient: dump: 2024-04-22T08:19:24.698 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:24.698 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:24.698 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:24.698 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:24.698 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:24.699 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:24.699 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:24.699 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:24.699 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:24.699 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:24.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.686+0000 7fb0c1ffb640 10 monclient: _finish_auth 0 2024-04-22T08:19:24.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.686+0000 7fb0c1ffb640 10 monclient: _check_auth_tickets 2024-04-22T08:19:24.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.686+0000 7fb0c1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:19:24.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.686+0000 7fb0c1ffb640 1 -- 172.21.15.40:0/3723207859 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0c803cc50 con 0x7fb0cc09f2e0 2024-04-22T08:19:24.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.686+0000 7fb0d39eb640 5 monclient: authenticate success, global_id 54650 2024-04-22T08:19:24.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.686+0000 7fb0d39eb640 10 monclient: _renew_subs 2024-04-22T08:19:24.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.686+0000 7fb0d39eb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:19:24.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.686+0000 7fb0d39eb640 1 -- 172.21.15.40:0/3723207859 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0c8009400 con 0x7fb0cc09f2e0 2024-04-22T08:19:24.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.686+0000 7fb0c1ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:19:24.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.686+0000 7fb0c1ffb640 1 -- 172.21.15.40:0/3723207859 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c8032400 con 0x7fb0cc09f2e0 2024-04-22T08:19:24.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.686+0000 7fb0c1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:24.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.686+0000 7fb0c1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:19:24.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.686+0000 7fb0c1ffb640 10 monclient: dump: 2024-04-22T08:19:24.700 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:24.700 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:24.700 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:24.700 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:24.700 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:24.700 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:24.700 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:24.700 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:24.700 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:24.700 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:24.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.686+0000 7fb0d39eb640 10 monclient: _renew_subs 2024-04-22T08:19:24.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.687+0000 7fb0d39eb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:19:24.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.687+0000 7fb0d39eb640 1 -- 172.21.15.40:0/3723207859 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0cc11ec80 con 0x7fb0cc09f2e0 2024-04-22T08:19:24.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.686+0000 7fb0c37fe640 4 set_mon_vals no callback set 2024-04-22T08:19:24.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.687+0000 7fb0c1ffb640 1 -- 172.21.15.40:0/3723207859 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fb0c8039070 con 0x7fb0cc09f2e0 2024-04-22T08:19:24.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.687+0000 7fb0d39eb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:19:24.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.687+0000 7fb0d39eb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:19:24.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.687+0000 7fb0d39eb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:19:24.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.687+0000 7fb0d39eb640 1 -- 172.21.15.40:0/3723207859 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0cc04e9b0 con 0x7fb0cc09f2e0 2024-04-22T08:19:24.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.688+0000 7fb0c1ffb640 1 --2- 172.21.15.40:0/3723207859 >> v2:172.21.15.189:6800/30168 conn(0x7fb0b00377d0 0x7fb0b0039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:24.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.688+0000 7fb0c1ffb640 1 -- 172.21.15.40:0/3723207859 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 3506+0+0 (secure 0 0 0) 0x7fb0c806bd90 con 0x7fb0cc09f2e0 2024-04-22T08:19:24.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.688+0000 7fb0d0f86640 1 --2- 172.21.15.40:0/3723207859 >> v2:172.21.15.189:6800/30168 conn(0x7fb0b00377d0 0x7fb0b0039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:24.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.688+0000 7fb0d0f86640 10 monclient: get_auth_request con 0x7fb0b00377d0 auth_method 0 2024-04-22T08:19:24.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.690+0000 7fb0d0f86640 1 --2- 172.21.15.40:0/3723207859 >> v2:172.21.15.189:6800/30168 conn(0x7fb0b00377d0 0x7fb0b0039c90 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc11cdc0 tx=0x7fb0bc009240 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:24.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.692+0000 7fb0c1ffb640 1 -- 172.21.15.40:0/3723207859 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7fb0c80443b0 con 0x7fb0cc09f2e0 2024-04-22T08:19:24.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.692+0000 7fb0c1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:19:24.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.692+0000 7fb0c1ffb640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:19:24.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.942+0000 7fb0d39eb640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-22T08:19:24.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.942+0000 7fb0d39eb640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-22T08:19:24.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.942+0000 7fb0d39eb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:19:24.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.942+0000 7fb0d39eb640 1 -- 172.21.15.40:0/3723207859 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fb0cc09b450 con 0x7fb0cc09f2e0 2024-04-22T08:19:24.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.944+0000 7fb0c1ffb640 1 -- 172.21.15.40:0/3723207859 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fb0c8037070 con 0x7fb0cc09f2e0 2024-04-22T08:19:24.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.944+0000 7fb0c1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-22T08:19:24.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.944+0000 7fb0c1ffb640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:19:24.944 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.944+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:19:24.944 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.944+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=13,config=2,health=11,kv=9,logm=65} crc {auth=1179773443,config=1618278320,health=2072305242,kv=1350343662,logm=2049353523}) 2024-04-22T08:19:24.944 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.944+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=14,config=2,health=12,kv=9,logm=63} crc {auth=778267362,config=1618278320,health=3886673345,kv=1350343662,logm=2269327808}) 2024-04-22T08:19:24.944 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.944+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:19:24.944 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.944+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=13,config=2,health=11,kv=9,logm=65} crc {auth=1179773443,config=1618278320,health=2072305242,kv=1350343662,logm=2049353523}) 2024-04-22T08:19:24.944 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.944+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=13,config=2,health=12,kv=9,logm=64} crc {auth=3703001955,config=1618278320,health=2240505080,kv=1350343662,logm=1644722992}) 2024-04-22T08:19:24.945 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.945+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:19:24.945 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.945+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3484197856}) 2024-04-22T08:19:24.946 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.945+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=341232515}) 2024-04-22T08:19:24.946 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.945+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:19:24.946 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.945+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3484197856}) 2024-04-22T08:19:24.946 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.945+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=441368266}) 2024-04-22T08:19:24.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.946+0000 7fb0d39eb640 1 -- 172.21.15.40:0/3723207859 >> v2:172.21.15.189:6800/30168 conn(0x7fb0b00377d0 msgr2=0x7fb0b0039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:24.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.946+0000 7fb0d39eb640 1 --2- 172.21.15.40:0/3723207859 >> v2:172.21.15.189:6800/30168 conn(0x7fb0b00377d0 0x7fb0b0039c90 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc11cdc0 tx=0x7fb0bc009240 comp rx=0 tx=0).stop 2024-04-22T08:19:24.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.946+0000 7fb0d39eb640 10 monclient: shutdown 2024-04-22T08:19:24.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.946+0000 7fb0d39eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:19:24.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.946+0000 7fb0d39eb640 1 -- 172.21.15.40:0/3723207859 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0cc09f2e0 msgr2=0x7fb0cc1c86b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:24.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.946+0000 7fb0d39eb640 1 --2- 172.21.15.40:0/3723207859 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0cc09f2e0 0x7fb0cc1c86b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb0c8009890 tx=0x7fb0c80046c0 comp rx=0 tx=0).stop 2024-04-22T08:19:24.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.946+0000 7fb0d39eb640 1 -- 172.21.15.40:0/3723207859 shutdown_connections 2024-04-22T08:19:24.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.946+0000 7fb0d39eb640 1 --2- 172.21.15.40:0/3723207859 >> v2:172.21.15.189:6800/30168 conn(0x7fb0b00377d0 0x7fb0b0039c90 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:24.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.946+0000 7fb0d39eb640 1 --2- 172.21.15.40:0/3723207859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0cc11b600 0x7fb0cc11ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:24.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.946+0000 7fb0d39eb640 1 --2- 172.21.15.40:0/3723207859 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0cc136840 0x7fb0cc135ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:24.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.946+0000 7fb0d39eb640 1 --2- 172.21.15.40:0/3723207859 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0cc09f2e0 0x7fb0cc1c86b0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:24.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.946+0000 7fb0d39eb640 1 -- 172.21.15.40:0/3723207859 >> 172.21.15.40:0/3723207859 conn(0x7fb0cc0a0b50 msgr2=0x7fb0cc0a27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:19:24.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.946+0000 7fb0d39eb640 1 -- 172.21.15.40:0/3723207859 shutdown_connections 2024-04-22T08:19:24.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:24.946+0000 7fb0d39eb640 1 -- 172.21.15.40:0/3723207859 wait complete. 2024-04-22T08:19:24.950 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.946+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:19:24.950 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.946+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2948556182}) 2024-04-22T08:19:24.950 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.946+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4266194337}) 2024-04-22T08:19:24.950 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.946+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:19:24.950 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.946+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2948556182}) 2024-04-22T08:19:24.950 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.946+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1718067818}) 2024-04-22T08:19:24.950 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.947+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:19:24.950 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.947+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4141711160}) 2024-04-22T08:19:24.951 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.947+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=873751329}) 2024-04-22T08:19:24.951 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.947+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:19:24.951 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.947+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4141711160}) 2024-04-22T08:19:24.951 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.947+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3742654122}) 2024-04-22T08:19:24.951 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.948+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:19:24.951 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.948+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3119564472}) 2024-04-22T08:19:24.951 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.948+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1041529085}) 2024-04-22T08:19:24.951 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.948+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:19:24.951 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.948+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3119564472}) 2024-04-22T08:19:24.951 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.948+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3048247368}) 2024-04-22T08:19:24.951 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.949+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:19:24.951 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.949+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4254995856}) 2024-04-22T08:19:24.951 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.949+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3764804431}) 2024-04-22T08:19:24.951 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.949+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:19:24.951 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.949+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4254995856}) 2024-04-22T08:19:24.952 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.949+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1974897063}) 2024-04-22T08:19:24.952 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.950+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:19:24.952 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.950+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3858230107}) 2024-04-22T08:19:24.952 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.950+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=672955708}) 2024-04-22T08:19:24.952 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.950+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:19:24.952 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.950+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3858230107}) 2024-04-22T08:19:24.952 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.950+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1890360517}) 2024-04-22T08:19:24.953 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.951+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:19:24.953 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.951+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=645107979}) 2024-04-22T08:19:24.953 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.951+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1536795352}) 2024-04-22T08:19:24.953 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.951+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:19:24.953 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.951+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=645107979}) 2024-04-22T08:19:24.953 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.951+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1761656082}) 2024-04-22T08:19:24.953 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.952+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:19:24.954 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.952+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=598868605}) 2024-04-22T08:19:24.954 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.952+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1409933416}) 2024-04-22T08:19:24.954 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.952+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:19:24.954 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.952+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=598868605}) 2024-04-22T08:19:24.954 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.952+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2175153076}) 2024-04-22T08:19:24.954 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.953+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:19:24.954 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.953+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2086063627}) 2024-04-22T08:19:24.954 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.953+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1307580552}) 2024-04-22T08:19:24.954 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.953+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:19:24.954 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.953+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2086063627}) 2024-04-22T08:19:24.954 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.953+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1196785205}) 2024-04-22T08:19:24.955 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.954+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:19:24.955 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.954+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2337387453}) 2024-04-22T08:19:24.955 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.954+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2222759761}) 2024-04-22T08:19:24.955 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.954+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:19:24.955 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.954+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2337387453}) 2024-04-22T08:19:24.956 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.954+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3197546714}) 2024-04-22T08:19:24.956 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.955+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:19:24.956 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.955+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3749538800}) 2024-04-22T08:19:24.956 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.955+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3777298754}) 2024-04-22T08:19:24.956 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.955+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:19:24.956 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.955+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3749538800}) 2024-04-22T08:19:24.956 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.955+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=921028252}) 2024-04-22T08:19:24.956 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.956+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:19:24.957 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.956+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=297846320}) 2024-04-22T08:19:24.957 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.956+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4244938398}) 2024-04-22T08:19:24.957 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.956+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:19:24.957 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.956+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=297846320}) 2024-04-22T08:19:24.957 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.956+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=470005558}) 2024-04-22T08:19:24.958 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-22T08:19:24.963 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.964+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:19:24.963 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.964+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2712102096}) 2024-04-22T08:19:24.963 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.964+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1643929357}) 2024-04-22T08:19:24.964 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.964+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:19:24.964 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.964+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2712102096}) 2024-04-22T08:19:24.964 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.964+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3288436428}) 2024-04-22T08:19:24.965 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.965+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:19:24.965 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.965+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4293940210}) 2024-04-22T08:19:24.965 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.965+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1945939961}) 2024-04-22T08:19:24.965 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.965+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:19:24.965 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.965+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4293940210}) 2024-04-22T08:19:24.965 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.965+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4245010382}) 2024-04-22T08:19:24.966 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.966+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:19:24.966 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.966+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=32,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=8,monmap=3,osd_metadata=5,osd_pg_creating=1,osdmap=31} crc {logm=1233577380,mdsmap=1531151339,mgr=878880162,mgr_command_descs=470045235,mgr_metadata=4222303120,mgrstat=385240936,mon_config_key=4255067121,monmap=3469086636,osd_metadata=999924489,osd_pg_creating=2980157349,osdmap=3973429748}) 2024-04-22T08:19:24.966 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.966+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=33,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=8,monmap=2,osd_metadata=6,osd_pg_creating=1,osdmap=30} crc {logm=791816595,mdsmap=1531151339,mgr=878880162,mgr_command_descs=470045235,mgr_metadata=4222303120,mgrstat=1847099615,mon_config_key=4255067121,monmap=2343331355,osd_metadata=4151711904,osd_pg_creating=2980157349,osdmap=888336455}) 2024-04-22T08:19:24.966 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.966+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:19:24.966 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.966+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=32,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=8,monmap=3,osd_metadata=5,osd_pg_creating=1,osdmap=31} crc {logm=1233577380,mdsmap=1531151339,mgr=878880162,mgr_command_descs=470045235,mgr_metadata=4222303120,mgrstat=385240936,mon_config_key=4255067121,monmap=3469086636,osd_metadata=999924489,osd_pg_creating=2980157349,osdmap=3973429748}) 2024-04-22T08:19:24.966 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.966+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=31,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=8,mon_config_key=8,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=31} crc {logm=1359348639,mdsmap=1531151339,mgr=3329140828,mgr_metadata=4222303120,mgrstat=1846826165,mon_config_key=4255067121,monmap=3469086636,osd_metadata=4151711904,osd_pg_creating=2980157349,osdmap=555995549}) 2024-04-22T08:19:24.974 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.974+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:19:24.974 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.974+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=461433930}) 2024-04-22T08:19:24.974 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.974+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1928356134}) 2024-04-22T08:19:24.974 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.974+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:19:24.974 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.974+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=461433930}) 2024-04-22T08:19:24.974 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.974+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4028257876}) 2024-04-22T08:19:24.975 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.975+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:19:24.975 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.975+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2195039266}) 2024-04-22T08:19:24.975 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.975+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3977751293}) 2024-04-22T08:19:24.975 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.975+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:19:24.975 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.975+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2195039266}) 2024-04-22T08:19:24.975 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.975+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=946771313}) 2024-04-22T08:19:24.980 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.981+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:19:24.981 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.981+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=89} crc {osdmap=2632229809}) 2024-04-22T08:19:24.981 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.981+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=92} crc {osdmap=3975370079}) 2024-04-22T08:19:24.981 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.981+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:19:24.981 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.981+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=89} crc {osdmap=2632229809}) 2024-04-22T08:19:24.981 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:19:24.981+0000 7f0557cd8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=90} crc {osdmap=1309518963}) 2024-04-22T08:19:25.959 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-22T08:19:25.959 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:19:26.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.072+0000 7f09b34c7640 10 monclient: get_monmap_and_config 2024-04-22T08:19:26.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.072+0000 7f09b34c7640 10 monclient: build_initial_monmap 2024-04-22T08:19:26.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.072+0000 7f09b34c7640 10 monclient: monmap: 2024-04-22T08:19:26.073 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:19:26.073 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:26.073 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:19:26.073346+0000 2024-04-22T08:19:26.073 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:19:26.073346+0000 2024-04-22T08:19:26.073 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:19:26.073 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:26.073 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:19:26.073 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:19:26.073 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:19:26.074 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:19:26.074 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:19:26.074 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:26.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.073+0000 7f09b34c7640 1 Processor -- start 2024-04-22T08:19:26.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.073+0000 7f09b34c7640 1 -- start start 2024-04-22T08:19:26.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.073+0000 7f09b34c7640 10 monclient: init 2024-04-22T08:19:26.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.074+0000 7f09b34c7640 10 monclient: _reopen_session rank -1 2024-04-22T08:19:26.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.074+0000 7f09b34c7640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-04-22T08:19:26.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.074+0000 7f09b34c7640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f09ac09db70 0x7f09ac09df50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:26.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.074+0000 7f09b34c7640 10 monclient(hunting): picked mon.noname-b con 0x7f09ac09db70 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:19:26.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.074+0000 7f09b34c7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f09ac12edf0 addr v1:172.21.15.189:6789/0 2024-04-22T08:19:26.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.074+0000 7f09b34c7640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f09ac12f1d0 0x7f09ac1324e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:26.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.074+0000 7f09b34c7640 10 monclient(hunting): picked mon.noname-a con 0x7f09ac12f1d0 addr v2:172.21.15.40:3300/0 2024-04-22T08:19:26.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.074+0000 7f09b34c7640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09ac09bb90 con 0x7f09ac12edf0 2024-04-22T08:19:26.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.074+0000 7f09b34c7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:26.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.074+0000 7f09b34c7640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09ac12ec10 con 0x7f09ac12f1d0 2024-04-22T08:19:26.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.074+0000 7f09b34c7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:26.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.074+0000 7f09b34c7640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09ac09c150 con 0x7f09ac09db70 2024-04-22T08:19:26.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.074+0000 7f09b34c7640 10 monclient(hunting): _renew_subs 2024-04-22T08:19:26.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.074+0000 7f09b34c7640 10 monclient(hunting): authenticate will time out at 1240.068512s 2024-04-22T08:19:26.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.075+0000 7f09b0a62640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f09ac12edf0 0x7f09ac131dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.40:54256/0 (socket says 172.21.15.40:54256) 2024-04-22T08:19:26.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.075+0000 7f09b0a62640 1 -- 172.21.15.40:0/1671034149 learned_addr learned my addr 172.21.15.40:0/1671034149 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:19:26.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.075+0000 7f09b1a64640 1 --2- 172.21.15.40:0/1671034149 >> v2:172.21.15.40:3300/0 conn(0x7f09ac12f1d0 0x7f09ac1324e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:26.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.075+0000 7f09b1263640 1 --2- 172.21.15.40:0/1671034149 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f09ac09db70 0x7f09ac09df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:26.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.075+0000 7f09b1a64640 10 monclient(hunting): get_auth_request con 0x7f09ac12f1d0 auth_method 0 2024-04-22T08:19:26.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.075+0000 7f09b1a64640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:26.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.075+0000 7f09b1a64640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:26.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.075+0000 7f09b1a64640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:26.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.075+0000 7f09b1263640 10 monclient(hunting): get_auth_request con 0x7f09ac09db70 auth_method 0 2024-04-22T08:19:26.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.075+0000 7f09b1263640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:26.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.075+0000 7f09b1263640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:26.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.075+0000 7f09b1263640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:26.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.075+0000 7f09b1a64640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:26.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.075+0000 7f09b1a64640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:26.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.075+0000 7f09b1a64640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:26.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.075+0000 7f09b1a64640 10 monclient(hunting): handle_auth_done global_id 84141 payload 274 2024-04-22T08:19:26.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.075+0000 7f09b1a64640 1 -- 172.21.15.40:0/1671034149 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f09ac09db70 msgr2=0x7f09ac09df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:19:26.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.075+0000 7f09b1a64640 1 --2- 172.21.15.40:0/1671034149 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f09ac09db70 0x7f09ac09df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:26.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.075+0000 7f09b1a64640 1 -- 172.21.15.40:0/1671034149 >> v1:172.21.15.189:6789/0 conn(0x7f09ac12edf0 legacy=0x7f09ac131dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:19:26.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.075+0000 7f09b1a64640 10 monclient: _finish_hunting 0 2024-04-22T08:19:26.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.075+0000 7f09b1a64640 1 monclient: found mon.noname-a 2024-04-22T08:19:26.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.075+0000 7f09b1a64640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:19:26.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.075+0000 7f09b1a64640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:19:26.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.075+0000 7f09b1a64640 1 -- 172.21.15.40:0/1671034149 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09ac09be70 con 0x7f09ac12f1d0 2024-04-22T08:19:26.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.079+0000 7f09b1a64640 1 -- 172.21.15.40:0/1671034149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09ac12f1d0 msgr2=0x7f09ac1324e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:26.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.079+0000 7f09b1a64640 1 --2- 172.21.15.40:0/1671034149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09ac12f1d0 0x7f09ac1324e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f09a4009b10 tx=0x7f09a402e210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4058c68e0771733 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:26.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.082+0000 7f09a3fff640 1 -- 172.21.15.40:0/1671034149 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a4036640 con 0x7f09ac12f1d0 2024-04-22T08:19:26.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.082+0000 7f09a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:26.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.082+0000 7f09a3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:19:26.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.082+0000 7f09a3fff640 10 monclient: dump: 2024-04-22T08:19:26.082 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:26.082 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:26.082 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.082 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.082 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:26.083 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:26.083 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:26.083 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:26.083 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:26.083 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:26.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.082+0000 7f09a3fff640 10 monclient: _finish_auth 0 2024-04-22T08:19:26.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.082+0000 7f09a3fff640 10 monclient: _check_auth_tickets 2024-04-22T08:19:26.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.082+0000 7f09a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:19:26.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.082+0000 7f09a3fff640 1 -- 172.21.15.40:0/1671034149 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09a40367a0 con 0x7f09ac12f1d0 2024-04-22T08:19:26.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.082+0000 7f09a3fff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:19:26.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.082+0000 7f09a3fff640 1 -- 172.21.15.40:0/1671034149 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a4036b30 con 0x7f09ac12f1d0 2024-04-22T08:19:26.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.082+0000 7f09a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:26.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.082+0000 7f09a3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:19:26.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.082+0000 7f09a3fff640 10 monclient: dump: 2024-04-22T08:19:26.088 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:26.088 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:26.088 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.088 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.088 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:26.088 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:26.088 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:26.088 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:26.089 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:26.089 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:26.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.082+0000 7f09b34c7640 5 monclient: authenticate success, global_id 84141 2024-04-22T08:19:26.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.082+0000 7f09b34c7640 10 monclient: get_monmap_and_config success 2024-04-22T08:19:26.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.082+0000 7f09b34c7640 4 set_mon_vals no callback set 2024-04-22T08:19:26.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.083+0000 7f09b34c7640 10 monclient: shutdown 2024-04-22T08:19:26.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.083+0000 7f09b34c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:19:26.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.083+0000 7f09b34c7640 1 -- 172.21.15.40:0/1671034149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09ac12f1d0 msgr2=0x7f09ac1324e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:26.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.083+0000 7f09b34c7640 1 --2- 172.21.15.40:0/1671034149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09ac12f1d0 0x7f09ac1324e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f09a4009b10 tx=0x7f09a402e210 comp rx=0 tx=0).stop 2024-04-22T08:19:26.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.083+0000 7f09b34c7640 1 -- 172.21.15.40:0/1671034149 shutdown_connections 2024-04-22T08:19:26.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.083+0000 7f09b34c7640 1 --2- 172.21.15.40:0/1671034149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09ac12f1d0 0x7f09ac1324e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:26.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.083+0000 7f09b34c7640 1 --2- 172.21.15.40:0/1671034149 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f09ac09db70 0x7f09ac09df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:26.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.083+0000 7f09b34c7640 1 -- 172.21.15.40:0/1671034149 >> 172.21.15.40:0/1671034149 conn(0x7f09ac0a0b50 msgr2=0x7f09ac0a0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:19:26.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.084+0000 7f09b34c7640 1 -- 172.21.15.40:0/1671034149 shutdown_connections 2024-04-22T08:19:26.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.084+0000 7f09b34c7640 1 -- 172.21.15.40:0/1671034149 wait complete. 2024-04-22T08:19:26.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.084+0000 7f09b34c7640 10 monclient: build_initial_monmap 2024-04-22T08:19:26.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.084+0000 7f09b34c7640 10 monclient: monmap: 2024-04-22T08:19:26.090 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:19:26.090 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:19:26.090 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:19:26.090 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:19:26.090 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:19:26.090 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:26.090 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:19:26.091 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:19:26.091 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:19:26.091 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:26.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.085+0000 7f09b34c7640 1 Processor -- start 2024-04-22T08:19:26.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.085+0000 7f09b34c7640 1 -- start start 2024-04-22T08:19:26.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.085+0000 7f09b34c7640 10 monclient: init 2024-04-22T08:19:26.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.085+0000 7f09b34c7640 10 monclient: _reopen_session rank -1 2024-04-22T08:19:26.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.085+0000 7f09b34c7640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-22T08:19:26.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.085+0000 7f09b34c7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09ac09db70 0x7f09ac1caab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:26.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.085+0000 7f09b34c7640 10 monclient(hunting): picked mon.noname-a con 0x7f09ac09db70 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:19:26.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.085+0000 7f09b34c7640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f09ac12edf0 0x7f09ac12ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:26.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.085+0000 7f09b34c7640 10 monclient(hunting): picked mon.noname-c con 0x7f09ac12edf0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:19:26.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.085+0000 7f09b34c7640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09ac12f1d0 0x7f09ac1b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:26.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.085+0000 7f09b34c7640 10 monclient(hunting): picked mon.noname-b con 0x7f09ac12f1d0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:19:26.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.085+0000 7f09b34c7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:26.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.085+0000 7f09b34c7640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ac1bb150 con 0x7f09ac09db70 2024-04-22T08:19:26.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.085+0000 7f09b34c7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:26.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.085+0000 7f09b34c7640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ac11b610 con 0x7f09ac12f1d0 2024-04-22T08:19:26.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.085+0000 7f09b34c7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:26.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.085+0000 7f09b34c7640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09ac1ba050 con 0x7f09ac12edf0 2024-04-22T08:19:26.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.085+0000 7f09b34c7640 10 monclient(hunting): _renew_subs 2024-04-22T08:19:26.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.085+0000 7f09b34c7640 10 monclient(hunting): authenticate will time out at 1240.079659s 2024-04-22T08:19:26.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.086+0000 7f09b1263640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09ac09db70 0x7f09ac1caab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:26.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.086+0000 7f09b1263640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09ac09db70 0x7f09ac1caab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52958/0 (socket says 172.21.15.40:52958) 2024-04-22T08:19:26.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.086+0000 7f09b1263640 1 -- 172.21.15.40:0/1570194888 learned_addr learned my addr 172.21.15.40:0/1570194888 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:19:26.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.086+0000 7f09b1263640 10 monclient(hunting): get_auth_request con 0x7f09ac09db70 auth_method 0 2024-04-22T08:19:26.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.086+0000 7f09b1263640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:26.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.086+0000 7f09b1263640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:26.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.086+0000 7f09b1263640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:26.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.086+0000 7f09b0a62640 1 --2- 172.21.15.40:0/1570194888 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f09ac12edf0 0x7f09ac12ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:26.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.086+0000 7f09b0a62640 10 monclient(hunting): get_auth_request con 0x7f09ac12edf0 auth_method 0 2024-04-22T08:19:26.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.086+0000 7f09b0a62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:26.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.086+0000 7f09b0a62640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:26.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.086+0000 7f09b0a62640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:26.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.086+0000 7f09b1263640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:26.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.086+0000 7f09b1263640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:26.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.086+0000 7f09b1263640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:26.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.086+0000 7f09b1a64640 1 --2- 172.21.15.40:0/1570194888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09ac12f1d0 0x7f09ac1b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:26.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.086+0000 7f09b1a64640 10 monclient(hunting): get_auth_request con 0x7f09ac12f1d0 auth_method 0 2024-04-22T08:19:26.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.086+0000 7f09b1a64640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:26.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.086+0000 7f09b1a64640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:26.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.086+0000 7f09b1a64640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:26.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.086+0000 7f09b0a62640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:26.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.086+0000 7f09b0a62640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:26.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.086+0000 7f09b0a62640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:26.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.086+0000 7f09b1a64640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:26.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.086+0000 7f09b1a64640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:26.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.086+0000 7f09b1a64640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:26.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.086+0000 7f09b1263640 10 monclient(hunting): handle_auth_done global_id 84144 payload 824 2024-04-22T08:19:26.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.086+0000 7f09b1263640 1 -- 172.21.15.40:0/1570194888 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f09ac12edf0 msgr2=0x7f09ac12ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:26.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.086+0000 7f09b1263640 1 --2- 172.21.15.40:0/1570194888 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f09ac12edf0 0x7f09ac12ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:26.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.086+0000 7f09b1263640 1 -- 172.21.15.40:0/1570194888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09ac12f1d0 msgr2=0x7f09ac1b8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:26.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.086+0000 7f09b1263640 1 --2- 172.21.15.40:0/1570194888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09ac12f1d0 0x7f09ac1b8b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:26.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.086+0000 7f09b1263640 10 monclient: _finish_hunting 0 2024-04-22T08:19:26.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.086+0000 7f09b1263640 1 monclient: found mon.noname-a 2024-04-22T08:19:26.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.086+0000 7f09b1263640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:19:26.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.086+0000 7f09b1263640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:19:26.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.086+0000 7f09b1263640 1 -- 172.21.15.40:0/1570194888 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a4009770 con 0x7f09ac09db70 2024-04-22T08:19:26.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.088+0000 7f09b1263640 1 -- 172.21.15.40:0/1570194888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09ac09db70 msgr2=0x7f09ac1caab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:26.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.088+0000 7f09b1263640 1 --2- 172.21.15.40:0/1570194888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09ac09db70 0x7f09ac1caab0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f09a800d800 tx=0x7f09a800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:26.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.088+0000 7f09a1ffb640 1 -- 172.21.15.40:0/1570194888 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a80098b0 con 0x7f09ac09db70 2024-04-22T08:19:26.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.088+0000 7f09a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:26.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.088+0000 7f09a1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:19:26.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.088+0000 7f09a1ffb640 10 monclient: dump: 2024-04-22T08:19:26.098 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:26.098 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:26.098 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.098 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.098 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:26.099 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:26.099 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:26.099 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:26.099 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:26.099 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:26.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.088+0000 7f09a1ffb640 10 monclient: _finish_auth 0 2024-04-22T08:19:26.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.088+0000 7f09a1ffb640 10 monclient: _check_auth_tickets 2024-04-22T08:19:26.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.088+0000 7f09a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:19:26.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.088+0000 7f09a1ffb640 1 -- 172.21.15.40:0/1570194888 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09a8009a10 con 0x7f09ac09db70 2024-04-22T08:19:26.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.088+0000 7f09b34c7640 5 monclient: authenticate success, global_id 84144 2024-04-22T08:19:26.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.088+0000 7f09a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:19:26.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.088+0000 7f09b34c7640 10 monclient: _renew_subs 2024-04-22T08:19:26.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.088+0000 7f09a1ffb640 1 -- 172.21.15.40:0/1570194888 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a8009ba0 con 0x7f09ac09db70 2024-04-22T08:19:26.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.088+0000 7f09b34c7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:19:26.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.088+0000 7f09b34c7640 1 -- 172.21.15.40:0/1570194888 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09ac131dd0 con 0x7f09ac09db70 2024-04-22T08:19:26.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.088+0000 7f09a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:26.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.089+0000 7f09a1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:19:26.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.088+0000 7f09a37fe640 4 set_mon_vals no callback set 2024-04-22T08:19:26.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.089+0000 7f09a1ffb640 10 monclient: dump: 2024-04-22T08:19:26.100 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:26.100 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:26.100 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.100 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.101 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:26.101 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:26.101 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:26.101 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:26.101 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:26.101 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:26.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.089+0000 7f09b34c7640 10 monclient: _renew_subs 2024-04-22T08:19:26.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.089+0000 7f09b34c7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:19:26.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.089+0000 7f09b34c7640 1 -- 172.21.15.40:0/1570194888 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09ac11ace0 con 0x7f09ac09db70 2024-04-22T08:19:26.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.089+0000 7f09a1ffb640 1 -- 172.21.15.40:0/1570194888 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f09a8009d00 con 0x7f09ac09db70 2024-04-22T08:19:26.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.090+0000 7f09a1ffb640 1 --2- 172.21.15.40:0/1570194888 >> v2:172.21.15.189:6800/30168 conn(0x7f0990037800 0x7f0990039cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:26.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.090+0000 7f09a1ffb640 1 -- 172.21.15.40:0/1570194888 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f09a804a830 con 0x7f09ac09db70 2024-04-22T08:19:26.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.090+0000 7f09b34c7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:19:26.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.090+0000 7f09b34c7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:19:26.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.090+0000 7f09b34c7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:19:26.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.090+0000 7f09b34c7640 1 -- 172.21.15.40:0/1570194888 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09ac04e9b0 con 0x7f09ac09db70 2024-04-22T08:19:26.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.090+0000 7f09b0a62640 1 --2- 172.21.15.40:0/1570194888 >> v2:172.21.15.189:6800/30168 conn(0x7f0990037800 0x7f0990039cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:26.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.090+0000 7f09b0a62640 10 monclient: get_auth_request con 0x7f0990037800 auth_method 0 2024-04-22T08:19:26.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.092+0000 7f09b0a62640 1 --2- 172.21.15.40:0/1570194888 >> v2:172.21.15.189:6800/30168 conn(0x7f0990037800 0x7f0990039cc0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f099c007c90 tx=0x7f099c007be0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:26.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.095+0000 7f09a1ffb640 1 -- 172.21.15.40:0/1570194888 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f09a8018020 con 0x7f09ac09db70 2024-04-22T08:19:26.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.095+0000 7f09a1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:19:26.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.095+0000 7f09a1ffb640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:19:26.200 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37843]: starting. 2024-04-22T08:19:26.200 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37843]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-37842 2024-04-22T08:19:26.200 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37843]: created object 0... 2024-04-22T08:19:26.200 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37843]: created object 25... 2024-04-22T08:19:26.200 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37843]: created object 49... 2024-04-22T08:19:26.200 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37843]: finishing. 2024-04-22T08:19:26.200 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37843]: shutting down. 2024-04-22T08:19:26.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.355+0000 7f09b34c7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:19:26.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.355+0000 7f09b34c7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:19:26.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.355+0000 7f09b34c7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:19:26.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.355+0000 7f09b34c7640 1 -- 172.21.15.40:0/1570194888 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f09ac11b020 con 0x7f09ac09db70 2024-04-22T08:19:26.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.355+0000 7f09a1ffb640 1 -- 172.21.15.40:0/1570194888 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f09a803c080 con 0x7f09ac09db70 2024-04-22T08:19:26.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.355+0000 7f09a1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:19:26.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.355+0000 7f09a1ffb640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:19:26.355 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":66,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:19:26.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.357+0000 7f09b34c7640 1 -- 172.21.15.40:0/1570194888 >> v2:172.21.15.189:6800/30168 conn(0x7f0990037800 msgr2=0x7f0990039cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:26.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.357+0000 7f09b34c7640 1 --2- 172.21.15.40:0/1570194888 >> v2:172.21.15.189:6800/30168 conn(0x7f0990037800 0x7f0990039cc0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f099c007c90 tx=0x7f099c007be0 comp rx=0 tx=0).stop 2024-04-22T08:19:26.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.357+0000 7f09b34c7640 10 monclient: shutdown 2024-04-22T08:19:26.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.357+0000 7f09b34c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:19:26.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.357+0000 7f09b34c7640 1 -- 172.21.15.40:0/1570194888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09ac09db70 msgr2=0x7f09ac1caab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:26.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.357+0000 7f09b34c7640 1 --2- 172.21.15.40:0/1570194888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09ac09db70 0x7f09ac1caab0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f09a800d800 tx=0x7f09a800dcc0 comp rx=0 tx=0).stop 2024-04-22T08:19:26.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.357+0000 7f09b34c7640 1 -- 172.21.15.40:0/1570194888 shutdown_connections 2024-04-22T08:19:26.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.357+0000 7f09b34c7640 1 --2- 172.21.15.40:0/1570194888 >> v2:172.21.15.189:6800/30168 conn(0x7f0990037800 0x7f0990039cc0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:26.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.357+0000 7f09b34c7640 1 --2- 172.21.15.40:0/1570194888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09ac12f1d0 0x7f09ac1b8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:26.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.357+0000 7f09b34c7640 1 --2- 172.21.15.40:0/1570194888 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f09ac12edf0 0x7f09ac12ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:26.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.357+0000 7f09b34c7640 1 --2- 172.21.15.40:0/1570194888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09ac09db70 0x7f09ac1caab0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:26.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.357+0000 7f09b34c7640 1 -- 172.21.15.40:0/1570194888 >> 172.21.15.40:0/1570194888 conn(0x7f09ac0a0b50 msgr2=0x7f09ac09fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:19:26.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.357+0000 7f09b34c7640 1 -- 172.21.15.40:0/1570194888 shutdown_connections 2024-04-22T08:19:26.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.357+0000 7f09b34c7640 1 -- 172.21.15.40:0/1570194888 wait complete. 2024-04-22T08:19:26.367 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-22T08:19:26.367 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-22T08:19:26.367 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-22T08:19:26.367 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-22T08:19:26.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.467+0000 7fe3da088640 10 monclient: get_monmap_and_config 2024-04-22T08:19:26.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.467+0000 7fe3da088640 10 monclient: build_initial_monmap 2024-04-22T08:19:26.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.467+0000 7fe3da088640 10 monclient: monmap: 2024-04-22T08:19:26.467 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:19:26.467 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:26.467 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:19:26.468018+0000 2024-04-22T08:19:26.467 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:19:26.468018+0000 2024-04-22T08:19:26.467 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:19:26.467 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:26.467 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:19:26.467 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:19:26.467 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:19:26.467 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:19:26.467 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:19:26.467 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:26.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.467+0000 7fe3da088640 1 Processor -- start 2024-04-22T08:19:26.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.467+0000 7fe3da088640 1 -- start start 2024-04-22T08:19:26.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.467+0000 7fe3da088640 10 monclient: init 2024-04-22T08:19:26.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.468+0000 7fe3da088640 10 monclient: _reopen_session rank -1 2024-04-22T08:19:26.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.468+0000 7fe3da088640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-04-22T08:19:26.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.468+0000 7fe3da088640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe3d4070e90 addr v1:172.21.15.40:6789/0 2024-04-22T08:19:26.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.468+0000 7fe3da088640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe3d4071270 0x7fe3d413ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:26.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.468+0000 7fe3da088640 10 monclient(hunting): picked mon.noname-a con 0x7fe3d4071270 addr v2:172.21.15.40:3300/0 2024-04-22T08:19:26.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.468+0000 7fe3da088640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3d413b2d0 0x7fe3d413d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:26.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.468+0000 7fe3da088640 10 monclient(hunting): picked mon.noname-b con 0x7fe3d413b2d0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:19:26.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.468+0000 7fe3da088640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3d4136940 con 0x7fe3d4070e90 2024-04-22T08:19:26.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.468+0000 7fe3da088640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:26.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.468+0000 7fe3da088640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3d406d170 con 0x7fe3d4071270 2024-04-22T08:19:26.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.468+0000 7fe3da088640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:26.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.468+0000 7fe3da088640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d4136320 con 0x7fe3d413b2d0 2024-04-22T08:19:26.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.468+0000 7fe3da088640 10 monclient(hunting): _renew_subs 2024-04-22T08:19:26.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.468+0000 7fe3da088640 10 monclient(hunting): authenticate will time out at 1240.462234s 2024-04-22T08:19:26.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.468+0000 7fe3d2ffd640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe3d4071270 0x7fe3d413ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:26.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.468+0000 7fe3d3fff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3d413b2d0 0x7fe3d413d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:26.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.468+0000 7fe3d2ffd640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe3d4071270 0x7fe3d413ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52960/0 (socket says 172.21.15.40:52960) 2024-04-22T08:19:26.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.468+0000 7fe3d2ffd640 1 -- 172.21.15.40:0/879096464 learned_addr learned my addr 172.21.15.40:0/879096464 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:19:26.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.468+0000 7fe3d2ffd640 10 monclient(hunting): get_auth_request con 0x7fe3d4071270 auth_method 0 2024-04-22T08:19:26.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.468+0000 7fe3d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:26.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.468+0000 7fe3d2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:26.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.468+0000 7fe3d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:26.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.468+0000 7fe3d3fff640 10 monclient(hunting): get_auth_request con 0x7fe3d413b2d0 auth_method 0 2024-04-22T08:19:26.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.468+0000 7fe3d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:26.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.468+0000 7fe3d3fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:26.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.468+0000 7fe3d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:26.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.468+0000 7fe3d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:26.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.468+0000 7fe3d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:26.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.468+0000 7fe3d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:26.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.468+0000 7fe3d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:26.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.468+0000 7fe3d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:26.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.468+0000 7fe3d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:26.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.468+0000 7fe3d3fff640 10 monclient(hunting): handle_auth_done global_id 54659 payload 274 2024-04-22T08:19:26.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.468+0000 7fe3d3fff640 1 -- 172.21.15.40:0/879096464 >> v2:172.21.15.40:3300/0 conn(0x7fe3d4071270 msgr2=0x7fe3d413ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:19:26.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.468+0000 7fe3d3fff640 1 --2- 172.21.15.40:0/879096464 >> v2:172.21.15.40:3300/0 conn(0x7fe3d4071270 0x7fe3d413ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:26.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.468+0000 7fe3d3fff640 1 -- 172.21.15.40:0/879096464 >> v1:172.21.15.40:6789/0 conn(0x7fe3d4070e90 legacy=0x7fe3d4135c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:19:26.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.468+0000 7fe3d3fff640 10 monclient: _finish_hunting 0 2024-04-22T08:19:26.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.468+0000 7fe3d3fff640 1 monclient: found mon.noname-b 2024-04-22T08:19:26.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.468+0000 7fe3d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:19:26.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.468+0000 7fe3d3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:19:26.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.468+0000 7fe3d3fff640 1 -- 172.21.15.40:0/879096464 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d406d450 con 0x7fe3d413b2d0 2024-04-22T08:19:26.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.471+0000 7fe3d3fff640 1 -- 172.21.15.40:0/879096464 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3d413b2d0 msgr2=0x7fe3d413d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:26.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.471+0000 7fe3d3fff640 1 --2- 172.21.15.40:0/879096464 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3d413b2d0 0x7fe3d413d6c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc009d90 tx=0x7fe3cc02bcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3642254488318e41 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:26.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.471+0000 7fe3d27fc640 1 -- 172.21.15.40:0/879096464 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3cc02ccb0 con 0x7fe3d413b2d0 2024-04-22T08:19:26.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.471+0000 7fe3d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:26.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.471+0000 7fe3d27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:19:26.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.471+0000 7fe3d27fc640 10 monclient: dump: 2024-04-22T08:19:26.481 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:26.482 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:26.482 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.482 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.482 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:26.482 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:26.482 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:26.482 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:26.482 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:26.482 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:26.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.471+0000 7fe3d27fc640 10 monclient: _finish_auth 0 2024-04-22T08:19:26.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.471+0000 7fe3d27fc640 10 monclient: _check_auth_tickets 2024-04-22T08:19:26.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.471+0000 7fe3d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:19:26.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.471+0000 7fe3d27fc640 1 -- 172.21.15.40:0/879096464 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3cc02ce10 con 0x7fe3d413b2d0 2024-04-22T08:19:26.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.471+0000 7fe3d27fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:19:26.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.471+0000 7fe3d27fc640 1 -- 172.21.15.40:0/879096464 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3cc034660 con 0x7fe3d413b2d0 2024-04-22T08:19:26.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.471+0000 7fe3da088640 5 monclient: authenticate success, global_id 54659 2024-04-22T08:19:26.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.471+0000 7fe3da088640 10 monclient: get_monmap_and_config success 2024-04-22T08:19:26.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.471+0000 7fe3da088640 4 set_mon_vals no callback set 2024-04-22T08:19:26.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.471+0000 7fe3d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:26.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.471+0000 7fe3d27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:19:26.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.471+0000 7fe3d27fc640 10 monclient: dump: 2024-04-22T08:19:26.483 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:26.483 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:26.483 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.484 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.484 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:26.484 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:26.484 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:26.484 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:26.484 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:26.484 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:26.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.472+0000 7fe3da088640 10 monclient: shutdown 2024-04-22T08:19:26.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.472+0000 7fe3da088640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:19:26.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.472+0000 7fe3da088640 1 -- 172.21.15.40:0/879096464 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3d413b2d0 msgr2=0x7fe3d413d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:26.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.472+0000 7fe3da088640 1 --2- 172.21.15.40:0/879096464 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3d413b2d0 0x7fe3d413d6c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc009d90 tx=0x7fe3cc02bcf0 comp rx=0 tx=0).stop 2024-04-22T08:19:26.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.472+0000 7fe3da088640 1 -- 172.21.15.40:0/879096464 shutdown_connections 2024-04-22T08:19:26.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.472+0000 7fe3da088640 1 --2- 172.21.15.40:0/879096464 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3d413b2d0 0x7fe3d413d6c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:26.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.472+0000 7fe3da088640 1 --2- 172.21.15.40:0/879096464 >> v2:172.21.15.40:3300/0 conn(0x7fe3d4071270 0x7fe3d413ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:26.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.472+0000 7fe3da088640 1 -- 172.21.15.40:0/879096464 >> 172.21.15.40:0/879096464 conn(0x7fe3d40a1780 msgr2=0x7fe3d40a1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:19:26.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.472+0000 7fe3da088640 1 -- 172.21.15.40:0/879096464 shutdown_connections 2024-04-22T08:19:26.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.472+0000 7fe3da088640 1 -- 172.21.15.40:0/879096464 wait complete. 2024-04-22T08:19:26.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.473+0000 7fe3da088640 10 monclient: build_initial_monmap 2024-04-22T08:19:26.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.473+0000 7fe3da088640 10 monclient: monmap: 2024-04-22T08:19:26.485 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:19:26.485 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:19:26.485 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:19:26.485 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:19:26.485 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:19:26.485 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:26.485 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:19:26.485 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:19:26.486 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:19:26.486 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:26.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.473+0000 7fe3da088640 1 Processor -- start 2024-04-22T08:19:26.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.473+0000 7fe3da088640 1 -- start start 2024-04-22T08:19:26.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.473+0000 7fe3da088640 10 monclient: init 2024-04-22T08:19:26.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.474+0000 7fe3da088640 10 monclient: _reopen_session rank -1 2024-04-22T08:19:26.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.474+0000 7fe3da088640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-22T08:19:26.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.474+0000 7fe3da088640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3d4070e90 0x7fe3d41ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:26.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.474+0000 7fe3da088640 10 monclient(hunting): picked mon.noname-b con 0x7fe3d4070e90 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:19:26.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.474+0000 7fe3da088640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3d4071270 0x7fe3d412f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:26.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.474+0000 7fe3da088640 10 monclient(hunting): picked mon.noname-a con 0x7fe3d4071270 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:19:26.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.474+0000 7fe3da088640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3d413b2d0 0x7fe3d412f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:26.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.474+0000 7fe3da088640 10 monclient(hunting): picked mon.noname-c con 0x7fe3d413b2d0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:19:26.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.474+0000 7fe3da088640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:26.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.474+0000 7fe3da088640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d40a07a0 con 0x7fe3d4071270 2024-04-22T08:19:26.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.474+0000 7fe3da088640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:26.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.474+0000 7fe3da088640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d413a380 con 0x7fe3d4070e90 2024-04-22T08:19:26.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.474+0000 7fe3da088640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:26.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.474+0000 7fe3da088640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d40a40d0 con 0x7fe3d413b2d0 2024-04-22T08:19:26.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.474+0000 7fe3da088640 10 monclient(hunting): _renew_subs 2024-04-22T08:19:26.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.474+0000 7fe3da088640 10 monclient(hunting): authenticate will time out at 1240.467891s 2024-04-22T08:19:26.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.474+0000 7fe3d2ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3d4071270 0x7fe3d412f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:26.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.474+0000 7fe3d3fff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3d413b2d0 0x7fe3d412f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:26.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.474+0000 7fe3d2ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3d4071270 0x7fe3d412f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52968/0 (socket says 172.21.15.40:52968) 2024-04-22T08:19:26.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.474+0000 7fe3d3fff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3d413b2d0 0x7fe3d412f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:36980/0 (socket says 172.21.15.40:36980) 2024-04-22T08:19:26.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.474+0000 7fe3d2ffd640 1 -- 172.21.15.40:0/24307873 learned_addr learned my addr 172.21.15.40:0/24307873 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:19:26.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.474+0000 7fe3d2ffd640 10 monclient(hunting): get_auth_request con 0x7fe3d4071270 auth_method 0 2024-04-22T08:19:26.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.474+0000 7fe3d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:26.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.474+0000 7fe3d2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:26.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.474+0000 7fe3d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:26.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.474+0000 7fe3d3fff640 10 monclient(hunting): get_auth_request con 0x7fe3d413b2d0 auth_method 0 2024-04-22T08:19:26.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.474+0000 7fe3d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:26.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.474+0000 7fe3d3fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:26.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.474+0000 7fe3d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:26.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.474+0000 7fe3d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:26.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.474+0000 7fe3d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:26.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.474+0000 7fe3d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:26.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.474+0000 7fe3d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:26.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.474+0000 7fe3d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:26.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.474+0000 7fe3d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:26.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.474+0000 7fe3d3fff640 10 monclient(hunting): handle_auth_done global_id 54662 payload 824 2024-04-22T08:19:26.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.474+0000 7fe3d37fe640 1 --2- 172.21.15.40:0/24307873 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3d4070e90 0x7fe3d41ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:26.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.474+0000 7fe3d3fff640 1 -- 172.21.15.40:0/24307873 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3d4070e90 msgr2=0x7fe3d41ca6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:26.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.474+0000 7fe3d3fff640 1 --2- 172.21.15.40:0/24307873 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3d4070e90 0x7fe3d41ca6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:26.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.474+0000 7fe3d3fff640 1 -- 172.21.15.40:0/24307873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3d4071270 msgr2=0x7fe3d412f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:26.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.474+0000 7fe3d3fff640 1 --2- 172.21.15.40:0/24307873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3d4071270 0x7fe3d412f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:26.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.474+0000 7fe3d3fff640 10 monclient: _finish_hunting 0 2024-04-22T08:19:26.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.474+0000 7fe3d3fff640 1 monclient: found mon.noname-c 2024-04-22T08:19:26.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.474+0000 7fe3d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:19:26.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.474+0000 7fe3d3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:19:26.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.474+0000 7fe3d3fff640 1 -- 172.21.15.40:0/24307873 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3cc009a70 con 0x7fe3d413b2d0 2024-04-22T08:19:26.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.476+0000 7fe3d3fff640 1 -- 172.21.15.40:0/24307873 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3d413b2d0 msgr2=0x7fe3d412f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:26.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.476+0000 7fe3d3fff640 1 --2- 172.21.15.40:0/24307873 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3d413b2d0 0x7fe3d412f5c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe3d4136460 tx=0x7fe3cc004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:26.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.476+0000 7fe3bbfff640 1 -- 172.21.15.40:0/24307873 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3cc040650 con 0x7fe3d413b2d0 2024-04-22T08:19:26.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.476+0000 7fe3bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:26.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.476+0000 7fe3bbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:19:26.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.476+0000 7fe3bbfff640 10 monclient: dump: 2024-04-22T08:19:26.490 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:26.490 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:26.490 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.491 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.491 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:26.492 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:26.492 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:26.492 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:26.492 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:26.492 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:26.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.476+0000 7fe3bbfff640 10 monclient: _finish_auth 0 2024-04-22T08:19:26.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.476+0000 7fe3bbfff640 10 monclient: _check_auth_tickets 2024-04-22T08:19:26.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.476+0000 7fe3bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:19:26.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.476+0000 7fe3da088640 5 monclient: authenticate success, global_id 54662 2024-04-22T08:19:26.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.476+0000 7fe3bbfff640 1 -- 172.21.15.40:0/24307873 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3cc0407b0 con 0x7fe3d413b2d0 2024-04-22T08:19:26.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.476+0000 7fe3bbfff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:19:26.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.476+0000 7fe3bbfff640 1 -- 172.21.15.40:0/24307873 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3cc040b60 con 0x7fe3d413b2d0 2024-04-22T08:19:26.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.476+0000 7fe3bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:26.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.477+0000 7fe3bbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:19:26.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.477+0000 7fe3bbfff640 10 monclient: dump: 2024-04-22T08:19:26.494 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:26.494 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:26.494 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.494 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.494 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:26.494 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:26.494 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:26.494 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:26.494 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:26.494 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:26.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.476+0000 7fe3d1ffb640 4 set_mon_vals no callback set 2024-04-22T08:19:26.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.477+0000 7fe3da088640 10 monclient: _renew_subs 2024-04-22T08:19:26.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.477+0000 7fe3da088640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:19:26.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.477+0000 7fe3da088640 1 -- 172.21.15.40:0/24307873 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3d411b750 con 0x7fe3d413b2d0 2024-04-22T08:19:26.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.477+0000 7fe3da088640 10 monclient: _renew_subs 2024-04-22T08:19:26.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.477+0000 7fe3da088640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:19:26.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.477+0000 7fe3da088640 1 -- 172.21.15.40:0/24307873 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3d4139480 con 0x7fe3d413b2d0 2024-04-22T08:19:26.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.477+0000 7fe3da088640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:19:26.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.477+0000 7fe3da088640 1 --2- 172.21.15.40:0/24307873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3d41378d0 0x7fe3d4137cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:26.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.477+0000 7fe3da088640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:26.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.477+0000 7fe3da088640 1 -- 172.21.15.40:0/24307873 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d4066900 con 0x7fe3d41378d0 2024-04-22T08:19:26.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.477+0000 7fe3bbfff640 1 -- 172.21.15.40:0/24307873 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fe3cc040d40 con 0x7fe3d413b2d0 2024-04-22T08:19:26.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.477+0000 7fe3d37fe640 1 --2- 172.21.15.40:0/24307873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3d41378d0 0x7fe3d4137cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:26.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.478+0000 7fe3d37fe640 10 monclient: get_auth_request con 0x7fe3d41378d0 auth_method 0 2024-04-22T08:19:26.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.478+0000 7fe3d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:26.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.478+0000 7fe3d37fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:26.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.478+0000 7fe3d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:26.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.478+0000 7fe3d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:26.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.478+0000 7fe3d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:26.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.478+0000 7fe3d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:26.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.478+0000 7fe3bbfff640 1 --2- 172.21.15.40:0/24307873 >> v2:172.21.15.189:6800/30168 conn(0x7fe3c00377d0 0x7fe3c0039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:26.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.478+0000 7fe3bbfff640 1 -- 172.21.15.40:0/24307873 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 3898+0+0 (secure 0 0 0) 0x7fe3cc06cfb0 con 0x7fe3d413b2d0 2024-04-22T08:19:26.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.478+0000 7fe3d37fe640 10 monclient(hunting): handle_auth_done global_id 84153 payload 824 2024-04-22T08:19:26.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.478+0000 7fe3d37fe640 1 -- 172.21.15.40:0/24307873 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe3d4071680 con 0x7fe3d41378d0 2024-04-22T08:19:26.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.478+0000 7fe3d2ffd640 1 --2- 172.21.15.40:0/24307873 >> v2:172.21.15.189:6800/30168 conn(0x7fe3c00377d0 0x7fe3c0039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:26.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.478+0000 7fe3d2ffd640 10 monclient: get_auth_request con 0x7fe3c00377d0 auth_method 0 2024-04-22T08:19:26.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.480+0000 7fe3d37fe640 1 -- 172.21.15.40:0/24307873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3d41378d0 msgr2=0x7fe3d4137cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:26.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.480+0000 7fe3d37fe640 1 --2- 172.21.15.40:0/24307873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3d41378d0 0x7fe3d4137cb0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe3c400d3b0 tx=0x7fe3c400d870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:26.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.480+0000 7fe3bbfff640 1 -- 172.21.15.40:0/24307873 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c4012a10 con 0x7fe3d41378d0 2024-04-22T08:19:26.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.480+0000 7fe3bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:26.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.480+0000 7fe3bbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:19:26.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.480+0000 7fe3bbfff640 10 monclient: dump: 2024-04-22T08:19:26.497 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:26.497 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:26.497 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.497 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.497 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:26.497 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:26.497 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:26.497 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:26.497 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:26.498 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:26.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.480+0000 7fe3bbfff640 1 -- 172.21.15.40:0/24307873 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7fe3c4012b70 con 0x7fe3d41378d0 2024-04-22T08:19:26.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.480+0000 7fe3bbfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:19:26.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.480+0000 7fe3bbfff640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:19:26.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.480+0000 7fe3bbfff640 1 -- 172.21.15.40:0/24307873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3d41378d0 msgr2=0x7fe3d4137cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:26.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.480+0000 7fe3bbfff640 1 --2- 172.21.15.40:0/24307873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3d41378d0 0x7fe3d4137cb0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe3c400d3b0 tx=0x7fe3c400d870 comp rx=0 tx=0).stop 2024-04-22T08:19:26.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.480+0000 7fe3bbfff640 1 -- 172.21.15.40:0/24307873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3d41378d0 msgr2=0x7fe3d4137cb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:19:26.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.480+0000 7fe3d2ffd640 1 --2- 172.21.15.40:0/24307873 >> v2:172.21.15.189:6800/30168 conn(0x7fe3c00377d0 0x7fe3c0039c90 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fe3d4133a00 tx=0x7fe3c8009240 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:26.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.480+0000 7fe3bbfff640 1 --2- 172.21.15.40:0/24307873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3d41378d0 0x7fe3d4137cb0 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe3c400d3b0 tx=0x7fe3c400d870 comp rx=0 tx=0).stop 2024-04-22T08:19:26.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.487+0000 7fe3da088640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:19:26.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.487+0000 7fe3da088640 1 --2- 172.21.15.40:0/24307873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3d41381f0 0x7fe3d406dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:26.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.487+0000 7fe3da088640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:26.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.487+0000 7fe3da088640 1 -- 172.21.15.40:0/24307873 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d411b8e0 con 0x7fe3d41381f0 2024-04-22T08:19:26.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.487+0000 7fe3d37fe640 1 --2- 172.21.15.40:0/24307873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3d41381f0 0x7fe3d406dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:26.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.487+0000 7fe3d37fe640 10 monclient: get_auth_request con 0x7fe3d41381f0 auth_method 0 2024-04-22T08:19:26.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.487+0000 7fe3d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:26.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.487+0000 7fe3d37fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:26.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.487+0000 7fe3d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:26.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.487+0000 7fe3d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:26.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.487+0000 7fe3d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:26.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.487+0000 7fe3d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:26.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.487+0000 7fe3d37fe640 10 monclient(hunting): handle_auth_done global_id 84156 payload 824 2024-04-22T08:19:26.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.487+0000 7fe3d37fe640 1 -- 172.21.15.40:0/24307873 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe3d40a3e60 con 0x7fe3d41381f0 2024-04-22T08:19:26.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.489+0000 7fe3d37fe640 1 -- 172.21.15.40:0/24307873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3d41381f0 msgr2=0x7fe3d406dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:26.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.489+0000 7fe3d37fe640 1 --2- 172.21.15.40:0/24307873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3d41381f0 0x7fe3d406dc30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe3c4017000 tx=0x7fe3c4003760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:26.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.490+0000 7fe3bbfff640 1 -- 172.21.15.40:0/24307873 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c401ba70 con 0x7fe3d41381f0 2024-04-22T08:19:26.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.490+0000 7fe3bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:26.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.490+0000 7fe3bbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:19:26.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.490+0000 7fe3bbfff640 10 monclient: dump: 2024-04-22T08:19:26.502 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:26.502 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:26.502 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.502 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.502 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:26.502 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:26.502 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:26.502 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:26.502 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:26.502 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:26.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.490+0000 7fe3bbfff640 1 -- 172.21.15.40:0/24307873 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4463 (secure 0 0 0) 0x7fe3c4012e50 con 0x7fe3d41381f0 2024-04-22T08:19:26.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.490+0000 7fe3bbfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:19:26.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.490+0000 7fe3bbfff640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:19:26.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.490+0000 7fe3bbfff640 1 -- 172.21.15.40:0/24307873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3d41381f0 msgr2=0x7fe3d406dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:26.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.490+0000 7fe3bbfff640 1 --2- 172.21.15.40:0/24307873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3d41381f0 0x7fe3d406dc30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe3c4017000 tx=0x7fe3c4003760 comp rx=0 tx=0).stop 2024-04-22T08:19:26.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.490+0000 7fe3bbfff640 1 -- 172.21.15.40:0/24307873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3d41381f0 msgr2=0x7fe3d406dc30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:19:26.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.490+0000 7fe3bbfff640 1 --2- 172.21.15.40:0/24307873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3d41381f0 0x7fe3d406dc30 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe3c4017000 tx=0x7fe3c4003760 comp rx=0 tx=0).stop 2024-04-22T08:19:26.503 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:19:26.503 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:19:26.503 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:19:26.503 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "leader", 2024-04-22T08:19:26.504 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 66, 2024-04-22T08:19:26.504 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:19:26.504 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:19:26.504 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:19:26.504 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:19:26.504 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:19:26.504 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 2, 2024-04-22T08:19:26.504 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:19:26.504 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:19:26.504 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:19:26.504 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:19:26.504 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:19:26.504 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:19:26.504 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:19:26.505 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:19:26.505 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:19:26.505 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:19:26.505 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:19:26.505 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:19:26.505 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:19:26.505 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:19:26.505 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:19:26.505 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:19:26.505 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:19:26.506 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:19:26.506 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:19:26.506 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:19:26.506 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:19:26.506 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:19:26.506 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:19:26.506 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:19:26.506 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:19:26.506 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:19:26.506 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:26.506 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:26.506 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:19:26.506 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:19:26.506 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:19:26.507 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:19:26.507 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:19:26.507 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:19:26.507 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:19:26.507 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:19:26.507 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:19:26.507 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:19:26.507 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:19:26.507 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:19:26.507 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:19:26.507 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:19:26.507 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:19:26.507 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:19:26.507 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:19:26.507 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:19:26.508 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:19:26.508 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:19:26.508 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:19:26.508 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:19:26.508 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:19:26.508 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:19:26.508 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:19:26.508 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:19:26.508 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:19:26.508 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:19:26.508 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:19:26.508 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:26.508 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:19:26.508 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:26.508 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:19:26.508 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:19:26.509 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:19:26.509 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:19:26.509 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:26.509 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:19:26.509 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:19:26.509 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:26.509 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:26.509 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:26.509 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:19:26.509 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:19:26.509 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:26.509 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:26.509 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:26.509 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:26.510 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:19:26.510 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:19:26.510 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:19:26.510 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:19:26.510 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:19:26.510 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:26.510 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:26.510 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:19:26.510 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:19:26.510 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:19:26.510 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:19:26.510 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:26.510 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:19:26.511 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:19:26.511 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:26.511 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:26.511 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:26.511 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:19:26.511 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:19:26.511 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:26.511 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:26.511 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:26.511 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:26.511 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:19:26.512 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:19:26.512 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:19:26.512 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:19:26.512 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:19:26.512 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:26.512 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:26.512 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:19:26.512 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:19:26.512 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:19:26.512 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:19:26.512 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:26.512 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:19:26.512 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:19:26.513 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:26.513 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:26.513 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:26.513 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:19:26.513 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:19:26.513 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:26.513 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:26.513 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:26.513 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:26.513 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:19:26.513 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:19:26.513 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:19:26.513 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:19:26.513 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:19:26.514 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:26.514 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:26.514 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:26.514 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:19:26.514 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:19:26.514 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:26.514 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:19:26.514 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:19:26.514 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:19:26.514 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:26.514 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:19:26.514 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:19:26.514 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:26.514 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:19:26.515 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:19:26.515 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:19:26.515 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:26.515 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:26.515 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:26.515 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:19:26.515 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:19:26.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.490+0000 7fe3da088640 1 -- 172.21.15.40:0/24307873 >> v2:172.21.15.189:6800/30168 conn(0x7fe3c00377d0 msgr2=0x7fe3c0039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:26.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.490+0000 7fe3da088640 1 --2- 172.21.15.40:0/24307873 >> v2:172.21.15.189:6800/30168 conn(0x7fe3c00377d0 0x7fe3c0039c90 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fe3d4133a00 tx=0x7fe3c8009240 comp rx=0 tx=0).stop 2024-04-22T08:19:26.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.490+0000 7fe3da088640 10 monclient: shutdown 2024-04-22T08:19:26.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.490+0000 7fe3da088640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:19:26.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.490+0000 7fe3da088640 1 -- 172.21.15.40:0/24307873 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3d413b2d0 msgr2=0x7fe3d412f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:26.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.490+0000 7fe3da088640 1 --2- 172.21.15.40:0/24307873 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3d413b2d0 0x7fe3d412f5c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe3d4136460 tx=0x7fe3cc004a40 comp rx=0 tx=0).stop 2024-04-22T08:19:26.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.490+0000 7fe3d3fff640 1 -- 172.21.15.40:0/24307873 reap_dead start 2024-04-22T08:19:26.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.490+0000 7fe3d3fff640 1 -- 172.21.15.40:0/24307873 reap_dead start 2024-04-22T08:19:26.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.490+0000 7fe3da088640 1 -- 172.21.15.40:0/24307873 shutdown_connections 2024-04-22T08:19:26.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.490+0000 7fe3da088640 1 -- 172.21.15.40:0/24307873 >> 172.21.15.40:0/24307873 conn(0x7fe3d40a1780 msgr2=0x7fe3d41348f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:19:26.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.490+0000 7fe3da088640 1 -- 172.21.15.40:0/24307873 shutdown_connections 2024-04-22T08:19:26.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.490+0000 7fe3da088640 1 -- 172.21.15.40:0/24307873 wait complete. 2024-04-22T08:19:26.517 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:19:26.518 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-22T08:19:26.518 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-22T08:19:26.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.617+0000 7f2aa1a10640 10 monclient: get_monmap_and_config 2024-04-22T08:19:26.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.617+0000 7f2aa1a10640 10 monclient: build_initial_monmap 2024-04-22T08:19:26.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.617+0000 7f2aa1a10640 10 monclient: monmap: 2024-04-22T08:19:26.618 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:19:26.618 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:26.618 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:19:26.618667+0000 2024-04-22T08:19:26.618 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:19:26.618667+0000 2024-04-22T08:19:26.618 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:19:26.618 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:26.618 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:19:26.619 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:19:26.619 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:19:26.619 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:19:26.619 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:19:26.619 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:26.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.618+0000 7f2aa1a10640 1 Processor -- start 2024-04-22T08:19:26.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.618+0000 7f2aa1a10640 1 -- start start 2024-04-22T08:19:26.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.618+0000 7f2aa1a10640 10 monclient: init 2024-04-22T08:19:26.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.618+0000 7f2aa1a10640 10 monclient: _reopen_session rank -1 2024-04-22T08:19:26.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.618+0000 7f2aa1a10640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-04-22T08:19:26.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.618+0000 7f2aa1a10640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2a9c09f1b0 0x7f2a9c09f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:26.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.618+0000 7f2aa1a10640 10 monclient(hunting): picked mon.noname-a con 0x7f2a9c09f1b0 addr v2:172.21.15.40:3300/0 2024-04-22T08:19:26.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.618+0000 7f2aa1a10640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2a9c133930 addr v1:172.21.15.40:6789/0 2024-04-22T08:19:26.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.618+0000 7f2aa1a10640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2a9c1344d0 0x7f2a9c13bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:26.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.618+0000 7f2aa1a10640 10 monclient(hunting): picked mon.noname-c con 0x7f2a9c1344d0 addr v2:172.21.15.189:3300/0 2024-04-22T08:19:26.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.618+0000 7f2aa1a10640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a9c09de90 con 0x7f2a9c133930 2024-04-22T08:19:26.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.618+0000 7f2aa1a10640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:26.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.618+0000 7f2aa1a10640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a9c09dae0 con 0x7f2a9c09f1b0 2024-04-22T08:19:26.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.618+0000 7f2aa1a10640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:26.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.618+0000 7f2aa1a10640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a9c06c720 con 0x7f2a9c1344d0 2024-04-22T08:19:26.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.618+0000 7f2aa1a10640 10 monclient(hunting): _renew_subs 2024-04-22T08:19:26.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.618+0000 7f2aa1a10640 10 monclient(hunting): authenticate will time out at 1240.612745s 2024-04-22T08:19:26.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.619+0000 7f2a9affd640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2a9c09f1b0 0x7f2a9c09f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:26.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.619+0000 7f2a9a7fc640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f2a9c133930 0x7f2a9c133dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:58090/0 (socket says 172.21.15.40:58090) 2024-04-22T08:19:26.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.619+0000 7f2a9a7fc640 1 -- 172.21.15.40:0/2969096226 learned_addr learned my addr 172.21.15.40:0/2969096226 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:19:26.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.619+0000 7f2a9affd640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2a9c09f1b0 0x7f2a9c09f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52988/0 (socket says 172.21.15.40:52988) 2024-04-22T08:19:26.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.619+0000 7f2a9affd640 10 monclient(hunting): get_auth_request con 0x7f2a9c09f1b0 auth_method 0 2024-04-22T08:19:26.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.619+0000 7f2a9affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:26.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.619+0000 7f2a9affd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:26.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.619+0000 7f2a9affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:26.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.619+0000 7f2a9b7fe640 1 --2- 172.21.15.40:0/2969096226 >> v2:172.21.15.189:3300/0 conn(0x7f2a9c1344d0 0x7f2a9c13bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:26.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.619+0000 7f2a9b7fe640 10 monclient(hunting): get_auth_request con 0x7f2a9c1344d0 auth_method 0 2024-04-22T08:19:26.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.619+0000 7f2a9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:26.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.619+0000 7f2a9b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:26.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.619+0000 7f2a9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:26.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.619+0000 7f2a9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:26.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.619+0000 7f2a9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:26.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.619+0000 7f2a9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:26.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.619+0000 7f2a9b7fe640 10 monclient(hunting): handle_auth_done global_id 64483 payload 274 2024-04-22T08:19:26.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.619+0000 7f2a9b7fe640 1 -- 172.21.15.40:0/2969096226 >> v2:172.21.15.40:3300/0 conn(0x7f2a9c09f1b0 msgr2=0x7f2a9c09f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:19:26.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.619+0000 7f2a9b7fe640 1 --2- 172.21.15.40:0/2969096226 >> v2:172.21.15.40:3300/0 conn(0x7f2a9c09f1b0 0x7f2a9c09f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:26.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.619+0000 7f2a9b7fe640 1 -- 172.21.15.40:0/2969096226 >> v1:172.21.15.40:6789/0 conn(0x7f2a9c133930 legacy=0x7f2a9c133dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:19:26.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.619+0000 7f2a9b7fe640 10 monclient: _finish_hunting 0 2024-04-22T08:19:26.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.619+0000 7f2a9b7fe640 1 monclient: found mon.noname-c 2024-04-22T08:19:26.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.619+0000 7f2a9b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:19:26.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.619+0000 7f2a9b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.189:3300/0 2024-04-22T08:19:26.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.619+0000 7f2a9b7fe640 1 -- 172.21.15.40:0/2969096226 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a9c09e420 con 0x7f2a9c1344d0 2024-04-22T08:19:26.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.622+0000 7f2a9b7fe640 1 -- 172.21.15.40:0/2969096226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a9c1344d0 msgr2=0x7f2a9c13bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:26.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.622+0000 7f2a9b7fe640 1 --2- 172.21.15.40:0/2969096226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a9c1344d0 0x7f2a9c13bc70 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c0725d0 tx=0x7f2a8c02bd20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=afae805e94243836 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:26.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.623+0000 7f2a99ffb640 1 -- 172.21.15.40:0/2969096226 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c02cce0 con 0x7f2a9c1344d0 2024-04-22T08:19:26.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.623+0000 7f2a99ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:26.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.623+0000 7f2a99ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:19:26.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.623+0000 7f2a99ffb640 10 monclient: dump: 2024-04-22T08:19:26.635 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:26.635 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:26.635 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.635 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.635 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:26.635 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:26.635 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:26.635 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:26.635 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:26.635 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:26.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.623+0000 7f2a99ffb640 10 monclient: _finish_auth 0 2024-04-22T08:19:26.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.623+0000 7f2a99ffb640 10 monclient: _check_auth_tickets 2024-04-22T08:19:26.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.623+0000 7f2a99ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:19:26.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.623+0000 7f2a99ffb640 1 -- 172.21.15.40:0/2969096226 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a8c02ce40 con 0x7f2a9c1344d0 2024-04-22T08:19:26.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.623+0000 7f2a99ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:19:26.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.623+0000 7f2a99ffb640 1 -- 172.21.15.40:0/2969096226 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c034660 con 0x7f2a9c1344d0 2024-04-22T08:19:26.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.623+0000 7f2a99ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:26.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.623+0000 7f2a99ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:19:26.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.623+0000 7f2a99ffb640 10 monclient: dump: 2024-04-22T08:19:26.636 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:26.636 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:26.636 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.636 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.636 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:26.636 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:26.636 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:26.636 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:26.636 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:26.637 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:26.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.623+0000 7f2aa1a10640 5 monclient: authenticate success, global_id 64483 2024-04-22T08:19:26.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.623+0000 7f2aa1a10640 10 monclient: get_monmap_and_config success 2024-04-22T08:19:26.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.623+0000 7f2aa1a10640 4 set_mon_vals no callback set 2024-04-22T08:19:26.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.624+0000 7f2aa1a10640 10 monclient: shutdown 2024-04-22T08:19:26.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.624+0000 7f2aa1a10640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:19:26.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.624+0000 7f2aa1a10640 1 -- 172.21.15.40:0/2969096226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a9c1344d0 msgr2=0x7f2a9c13bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:26.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.624+0000 7f2aa1a10640 1 --2- 172.21.15.40:0/2969096226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a9c1344d0 0x7f2a9c13bc70 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c0725d0 tx=0x7f2a8c02bd20 comp rx=0 tx=0).stop 2024-04-22T08:19:26.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.624+0000 7f2aa1a10640 1 -- 172.21.15.40:0/2969096226 shutdown_connections 2024-04-22T08:19:26.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.624+0000 7f2aa1a10640 1 --2- 172.21.15.40:0/2969096226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a9c1344d0 0x7f2a9c13bc70 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:26.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.624+0000 7f2aa1a10640 1 --2- 172.21.15.40:0/2969096226 >> v2:172.21.15.40:3300/0 conn(0x7f2a9c09f1b0 0x7f2a9c09f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:26.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.624+0000 7f2aa1a10640 1 -- 172.21.15.40:0/2969096226 >> 172.21.15.40:0/2969096226 conn(0x7f2a9c0a0ae0 msgr2=0x7f2a9c0a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:19:26.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.624+0000 7f2aa1a10640 1 -- 172.21.15.40:0/2969096226 shutdown_connections 2024-04-22T08:19:26.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.624+0000 7f2aa1a10640 1 -- 172.21.15.40:0/2969096226 wait complete. 2024-04-22T08:19:26.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.624+0000 7f2aa1a10640 10 monclient: build_initial_monmap 2024-04-22T08:19:26.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.624+0000 7f2aa1a10640 10 monclient: monmap: 2024-04-22T08:19:26.638 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:19:26.638 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:19:26.638 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:19:26.638 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:19:26.638 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:19:26.638 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:26.638 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:19:26.638 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:19:26.638 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:19:26.638 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:26.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.624+0000 7f2aa1a10640 1 Processor -- start 2024-04-22T08:19:26.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.625+0000 7f2aa1a10640 1 -- start start 2024-04-22T08:19:26.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.625+0000 7f2aa1a10640 10 monclient: init 2024-04-22T08:19:26.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.625+0000 7f2aa1a10640 10 monclient: _reopen_session rank -1 2024-04-22T08:19:26.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.625+0000 7f2aa1a10640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-22T08:19:26.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.625+0000 7f2aa1a10640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a9c09f1b0 0x7f2a9c1cab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:26.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.625+0000 7f2aa1a10640 10 monclient(hunting): picked mon.noname-a con 0x7f2a9c09f1b0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:19:26.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.625+0000 7f2aa1a10640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a9c133930 0x7f2a9c1cb060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:26.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.625+0000 7f2aa1a10640 10 monclient(hunting): picked mon.noname-c con 0x7f2a9c133930 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:19:26.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.625+0000 7f2aa1a10640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a9c1344d0 0x7f2a9c09b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:26.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.625+0000 7f2aa1a10640 10 monclient(hunting): picked mon.noname-b con 0x7f2a9c1344d0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:19:26.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.625+0000 7f2aa1a10640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:26.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.625+0000 7f2aa1a10640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a9c124b70 con 0x7f2a9c09f1b0 2024-04-22T08:19:26.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.625+0000 7f2aa1a10640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:26.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.625+0000 7f2aa1a10640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a9c0718b0 con 0x7f2a9c1344d0 2024-04-22T08:19:26.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.625+0000 7f2aa1a10640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:26.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.625+0000 7f2aa1a10640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a9c070510 con 0x7f2a9c133930 2024-04-22T08:19:26.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.625+0000 7f2aa1a10640 10 monclient(hunting): _renew_subs 2024-04-22T08:19:26.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.625+0000 7f2aa1a10640 10 monclient(hunting): authenticate will time out at 1240.619391s 2024-04-22T08:19:26.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.625+0000 7f2a9affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a9c09f1b0 0x7f2a9c1cab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:26.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.625+0000 7f2a9a7fc640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a9c133930 0x7f2a9c1cb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:26.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.625+0000 7f2a9affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a9c09f1b0 0x7f2a9c1cab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52994/0 (socket says 172.21.15.40:52994) 2024-04-22T08:19:26.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.625+0000 7f2a9affd640 1 -- 172.21.15.40:0/1972966188 learned_addr learned my addr 172.21.15.40:0/1972966188 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:19:26.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.625+0000 7f2a9affd640 10 monclient(hunting): get_auth_request con 0x7f2a9c09f1b0 auth_method 0 2024-04-22T08:19:26.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.625+0000 7f2a9a7fc640 1 --2- 172.21.15.40:0/1972966188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a9c133930 0x7f2a9c1cb060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:36986/0 (socket says 172.21.15.40:36986) 2024-04-22T08:19:26.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.625+0000 7f2a9affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:26.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.625+0000 7f2a9affd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:26.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.625+0000 7f2a9affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:26.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.625+0000 7f2a9a7fc640 10 monclient(hunting): get_auth_request con 0x7f2a9c133930 auth_method 0 2024-04-22T08:19:26.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.625+0000 7f2a9a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:26.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.625+0000 7f2a9a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:26.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.625+0000 7f2a9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:26.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.625+0000 7f2a9affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:26.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.625+0000 7f2a9affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:26.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.625+0000 7f2a9affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:26.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.625+0000 7f2a9a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:26.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.625+0000 7f2a9a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:26.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.625+0000 7f2a9a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:26.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.625+0000 7f2a9b7fe640 1 --2- 172.21.15.40:0/1972966188 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a9c1344d0 0x7f2a9c09b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:26.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.625+0000 7f2a9b7fe640 10 monclient(hunting): get_auth_request con 0x7f2a9c1344d0 auth_method 0 2024-04-22T08:19:26.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.625+0000 7f2a9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:26.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.625+0000 7f2a9b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:26.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.625+0000 7f2a9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:26.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.626+0000 7f2a9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:26.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.626+0000 7f2a9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:26.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.626+0000 7f2a9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:26.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.626+0000 7f2a9affd640 10 monclient(hunting): handle_auth_done global_id 84162 payload 824 2024-04-22T08:19:26.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.626+0000 7f2a9affd640 1 -- 172.21.15.40:0/1972966188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a9c133930 msgr2=0x7f2a9c1cb060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:26.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.626+0000 7f2a9affd640 1 --2- 172.21.15.40:0/1972966188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a9c133930 0x7f2a9c1cb060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:26.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.626+0000 7f2a9affd640 1 -- 172.21.15.40:0/1972966188 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a9c1344d0 msgr2=0x7f2a9c09b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:26.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.626+0000 7f2a9affd640 1 --2- 172.21.15.40:0/1972966188 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a9c1344d0 0x7f2a9c09b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:26.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.626+0000 7f2a9affd640 10 monclient: _finish_hunting 0 2024-04-22T08:19:26.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.626+0000 7f2a9affd640 1 monclient: found mon.noname-a 2024-04-22T08:19:26.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.626+0000 7f2a9affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:19:26.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.626+0000 7f2a9affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:19:26.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.626+0000 7f2a9affd640 1 -- 172.21.15.40:0/1972966188 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a8c009a70 con 0x7f2a9c09f1b0 2024-04-22T08:19:26.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.626+0000 7f2a9a7fc640 1 --2- 172.21.15.40:0/1972966188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a9c133930 0x7f2a9c1cb060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:19:26.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.628+0000 7f2a9affd640 1 -- 172.21.15.40:0/1972966188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a9c09f1b0 msgr2=0x7f2a9c1cab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:26.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.628+0000 7f2a9affd640 1 --2- 172.21.15.40:0/1972966188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a9c09f1b0 0x7f2a9c1cab20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2a9400d800 tx=0x7f2a9400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:26.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.628+0000 7f2a93fff640 1 -- 172.21.15.40:0/1972966188 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9400b8d0 con 0x7f2a9c09f1b0 2024-04-22T08:19:26.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.628+0000 7f2a93fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:26.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.628+0000 7f2a93fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:19:26.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.628+0000 7f2a93fff640 10 monclient: dump: 2024-04-22T08:19:26.643 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:26.643 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:26.643 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.643 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.643 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:26.643 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:26.643 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:26.643 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:26.643 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:26.644 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:26.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.628+0000 7f2a93fff640 10 monclient: _finish_auth 0 2024-04-22T08:19:26.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.628+0000 7f2a93fff640 10 monclient: _check_auth_tickets 2024-04-22T08:19:26.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.628+0000 7f2a93fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:19:26.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.628+0000 7f2a93fff640 1 -- 172.21.15.40:0/1972966188 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a9400ba30 con 0x7f2a9c09f1b0 2024-04-22T08:19:26.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.628+0000 7f2aa1a10640 5 monclient: authenticate success, global_id 84162 2024-04-22T08:19:26.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.628+0000 7f2aa1a10640 10 monclient: _renew_subs 2024-04-22T08:19:26.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.628+0000 7f2aa1a10640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:19:26.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.628+0000 7f2aa1a10640 1 -- 172.21.15.40:0/1972966188 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a9c09d160 con 0x7f2a9c09f1b0 2024-04-22T08:19:26.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.628+0000 7f2a93fff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:19:26.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.628+0000 7f2a93fff640 1 -- 172.21.15.40:0/1972966188 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9400bdc0 con 0x7f2a9c09f1b0 2024-04-22T08:19:26.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.628+0000 7f2a93fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:26.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.628+0000 7f2a93fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:19:26.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.628+0000 7f2a93fff640 10 monclient: dump: 2024-04-22T08:19:26.644 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:26.645 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:26.645 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.645 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.645 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:26.645 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:26.645 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:26.645 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:26.645 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:26.645 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:26.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.628+0000 7f2a997fa640 4 set_mon_vals no callback set 2024-04-22T08:19:26.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.628+0000 7f2aa1a10640 10 monclient: _renew_subs 2024-04-22T08:19:26.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.628+0000 7f2aa1a10640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:19:26.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.628+0000 7f2aa1a10640 1 -- 172.21.15.40:0/1972966188 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a9c1248b0 con 0x7f2a9c09f1b0 2024-04-22T08:19:26.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.629+0000 7f2a93fff640 1 -- 172.21.15.40:0/1972966188 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f2a9401e460 con 0x7f2a9c09f1b0 2024-04-22T08:19:26.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.629+0000 7f2a93fff640 1 --2- 172.21.15.40:0/1972966188 >> v2:172.21.15.189:6800/30168 conn(0x7f2a7c0377d0 0x7f2a7c039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:26.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.629+0000 7f2aa1a10640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:19:26.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.629+0000 7f2aa1a10640 1 --2- 172.21.15.40:0/1972966188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a9c09c210 0x7f2a9c09c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:26.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.629+0000 7f2aa1a10640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:26.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.629+0000 7f2aa1a10640 1 -- 172.21.15.40:0/1972966188 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a9c0667a0 con 0x7f2a9c09c210 2024-04-22T08:19:26.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.629+0000 7f2a93fff640 1 -- 172.21.15.40:0/1972966188 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f2a9404a950 con 0x7f2a9c09f1b0 2024-04-22T08:19:26.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.629+0000 7f2a9b7fe640 1 --2- 172.21.15.40:0/1972966188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a9c09c210 0x7f2a9c09c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:26.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.629+0000 7f2a9b7fe640 10 monclient: get_auth_request con 0x7f2a9c09c210 auth_method 0 2024-04-22T08:19:26.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.629+0000 7f2a9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:26.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.629+0000 7f2a9b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:26.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.629+0000 7f2a9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:26.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.629+0000 7f2a9a7fc640 1 --2- 172.21.15.40:0/1972966188 >> v2:172.21.15.189:6800/30168 conn(0x7f2a7c0377d0 0x7f2a7c039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:26.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.629+0000 7f2a9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:26.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.629+0000 7f2a9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:26.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.629+0000 7f2a9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:26.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.629+0000 7f2a9a7fc640 10 monclient: get_auth_request con 0x7f2a7c0377d0 auth_method 0 2024-04-22T08:19:26.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.629+0000 7f2a9b7fe640 10 monclient(hunting): handle_auth_done global_id 54668 payload 824 2024-04-22T08:19:26.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.629+0000 7f2a9b7fe640 1 -- 172.21.15.40:0/1972966188 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2a9c09eec0 con 0x7f2a9c09c210 2024-04-22T08:19:26.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.632+0000 7f2a9b7fe640 1 -- 172.21.15.40:0/1972966188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a9c09c210 msgr2=0x7f2a9c09c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:26.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.632+0000 7f2a9b7fe640 1 --2- 172.21.15.40:0/1972966188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a9c09c210 0x7f2a9c09c5f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c036000 tx=0x7f2a8c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:26.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.632+0000 7f2a9a7fc640 1 --2- 172.21.15.40:0/1972966188 >> v2:172.21.15.189:6800/30168 conn(0x7f2a7c0377d0 0x7f2a7c039c90 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f2a84007c90 tx=0x7f2a84007c00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:26.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.632+0000 7f2a93fff640 1 -- 172.21.15.40:0/1972966188 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c034850 con 0x7f2a9c09c210 2024-04-22T08:19:26.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.632+0000 7f2a93fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:26.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.632+0000 7f2a93fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:19:26.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.632+0000 7f2a93fff640 10 monclient: dump: 2024-04-22T08:19:26.647 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:26.647 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:26.647 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.648 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.648 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:26.648 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:26.648 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:26.648 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:26.648 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:26.648 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:26.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.632+0000 7f2a93fff640 1 -- 172.21.15.40:0/1972966188 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f2a8c0349b0 con 0x7f2a9c09c210 2024-04-22T08:19:26.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.632+0000 7f2a93fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:19:26.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.632+0000 7f2a93fff640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:19:26.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.632+0000 7f2a93fff640 1 -- 172.21.15.40:0/1972966188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a9c09c210 msgr2=0x7f2a9c09c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:26.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.632+0000 7f2a93fff640 1 --2- 172.21.15.40:0/1972966188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a9c09c210 0x7f2a9c09c5f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c036000 tx=0x7f2a8c004a40 comp rx=0 tx=0).stop 2024-04-22T08:19:26.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.632+0000 7f2a93fff640 1 -- 172.21.15.40:0/1972966188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a9c09c210 msgr2=0x7f2a9c09c5f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:19:26.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.632+0000 7f2a93fff640 1 --2- 172.21.15.40:0/1972966188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a9c09c210 0x7f2a9c09c5f0 secure :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c036000 tx=0x7f2a8c004a40 comp rx=0 tx=0).stop 2024-04-22T08:19:26.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.638+0000 7f2aa1a10640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:19:26.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.638+0000 7f2aa1a10640 1 --2- 172.21.15.40:0/1972966188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a9c02b200 0x7f2a9c02b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:26.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.638+0000 7f2aa1a10640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:26.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.638+0000 7f2aa1a10640 1 -- 172.21.15.40:0/1972966188 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a9c124ce0 con 0x7f2a9c02b200 2024-04-22T08:19:26.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.639+0000 7f2a9b7fe640 1 --2- 172.21.15.40:0/1972966188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a9c02b200 0x7f2a9c02b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:26.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.639+0000 7f2a9b7fe640 10 monclient: get_auth_request con 0x7f2a9c02b200 auth_method 0 2024-04-22T08:19:26.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.639+0000 7f2a9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:26.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.639+0000 7f2a9b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:26.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.639+0000 7f2a9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:26.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.639+0000 7f2a9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:26.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.639+0000 7f2a9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:26.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.639+0000 7f2a9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:26.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.639+0000 7f2a9b7fe640 10 monclient(hunting): handle_auth_done global_id 54671 payload 824 2024-04-22T08:19:26.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.639+0000 7f2a9b7fe640 1 -- 172.21.15.40:0/1972966188 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2a9c09fad0 con 0x7f2a9c02b200 2024-04-22T08:19:26.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.641+0000 7f2a9b7fe640 1 -- 172.21.15.40:0/1972966188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a9c02b200 msgr2=0x7f2a9c02b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:26.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.641+0000 7f2a9b7fe640 1 --2- 172.21.15.40:0/1972966188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a9c02b200 0x7f2a9c02b610 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c02bd60 tx=0x7f2a8c031000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:26.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.641+0000 7f2a93fff640 1 -- 172.21.15.40:0/1972966188 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c0505f0 con 0x7f2a9c02b200 2024-04-22T08:19:26.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.641+0000 7f2a93fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:26.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.641+0000 7f2a93fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:19:26.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.641+0000 7f2a93fff640 10 monclient: dump: 2024-04-22T08:19:26.652 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:26.652 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:26.652 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.652 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.652 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:26.652 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:26.652 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:26.652 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:26.652 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:26.652 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:26.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.641+0000 7f2a93fff640 1 -- 172.21.15.40:0/1972966188 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4631 (secure 0 0 0) 0x7f2a8c050750 con 0x7f2a9c02b200 2024-04-22T08:19:26.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.641+0000 7f2a93fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:19:26.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.641+0000 7f2a93fff640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:19:26.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.641+0000 7f2a93fff640 1 -- 172.21.15.40:0/1972966188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a9c02b200 msgr2=0x7f2a9c02b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:26.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.641+0000 7f2a93fff640 1 --2- 172.21.15.40:0/1972966188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a9c02b200 0x7f2a9c02b610 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c02bd60 tx=0x7f2a8c031000 comp rx=0 tx=0).stop 2024-04-22T08:19:26.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.641+0000 7f2a93fff640 1 -- 172.21.15.40:0/1972966188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a9c02b200 msgr2=0x7f2a9c02b610 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:19:26.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.641+0000 7f2a93fff640 1 --2- 172.21.15.40:0/1972966188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a9c02b200 0x7f2a9c02b610 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:26.653 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:19:26.653 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:19:26.653 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:19:26.653 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:19:26.653 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 66, 2024-04-22T08:19:26.654 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:19:26.654 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:19:26.654 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:19:26.654 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:19:26.654 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:19:26.654 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 2, 2024-04-22T08:19:26.654 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:19:26.654 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:19:26.654 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:19:26.654 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:19:26.654 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:19:26.654 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:19:26.654 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:19:26.655 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:19:26.655 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:19:26.655 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:19:26.655 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:19:26.655 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:19:26.655 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:19:26.655 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:19:26.655 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:19:26.655 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:19:26.655 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:19:26.655 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:19:26.655 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:19:26.655 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:19:26.655 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:19:26.655 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:19:26.656 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:19:26.656 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:19:26.656 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:19:26.656 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:19:26.656 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:26.656 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:26.656 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:19:26.656 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:19:26.656 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:19:26.656 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:19:26.656 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:19:26.656 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:19:26.656 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:19:26.656 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:19:26.657 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:19:26.657 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:19:26.657 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:19:26.657 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:19:26.657 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:19:26.657 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:19:26.657 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:19:26.657 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:19:26.657 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:19:26.657 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:19:26.657 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:19:26.657 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:19:26.657 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:19:26.657 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:19:26.657 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:19:26.658 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:19:26.658 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:19:26.658 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:19:26.658 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:19:26.658 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:19:26.658 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:19:26.658 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:26.658 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:19:26.658 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:26.658 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:19:26.658 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:19:26.658 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:19:26.658 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:19:26.658 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:26.659 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:19:26.659 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:19:26.659 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:26.659 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:26.659 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:26.659 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:19:26.659 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:19:26.659 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:26.659 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:26.659 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:26.659 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:26.659 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:19:26.659 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:19:26.659 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:19:26.659 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:19:26.660 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:19:26.660 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:26.660 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:26.660 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:19:26.660 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:19:26.660 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:19:26.660 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:19:26.660 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:26.660 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:19:26.660 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:19:26.660 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:26.660 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:26.660 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:26.660 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:19:26.660 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:19:26.661 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:26.661 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:26.661 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:26.661 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:26.661 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:19:26.661 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:19:26.661 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:19:26.661 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:19:26.661 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:19:26.661 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:26.661 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:26.661 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:19:26.661 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:19:26.661 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:19:26.661 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:19:26.662 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:26.662 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:19:26.662 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:19:26.662 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:26.662 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:26.662 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:26.662 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:19:26.662 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:19:26.662 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:26.662 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:26.662 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:26.662 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:26.662 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:19:26.662 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:19:26.663 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:19:26.663 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:19:26.663 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:19:26.663 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:26.663 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:26.663 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:26.663 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:19:26.663 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:19:26.663 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:26.663 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:19:26.663 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:19:26.663 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:19:26.663 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:26.663 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:19:26.663 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:19:26.664 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:26.664 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:19:26.664 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:19:26.664 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 3 2024-04-22T08:19:26.664 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:26.664 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:19:26.664 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:19:26.664 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:26.664 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:19:26.664 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:19:26.664 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 4 2024-04-22T08:19:26.664 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:26.664 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:26.664 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:26.664 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:19:26.665 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:19:26.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.642+0000 7f2aa1a10640 1 -- 172.21.15.40:0/1972966188 >> v2:172.21.15.189:6800/30168 conn(0x7f2a7c0377d0 msgr2=0x7f2a7c039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:26.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.642+0000 7f2aa1a10640 1 --2- 172.21.15.40:0/1972966188 >> v2:172.21.15.189:6800/30168 conn(0x7f2a7c0377d0 0x7f2a7c039c90 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f2a84007c90 tx=0x7f2a84007c00 comp rx=0 tx=0).stop 2024-04-22T08:19:26.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.642+0000 7f2aa1a10640 10 monclient: shutdown 2024-04-22T08:19:26.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.642+0000 7f2aa1a10640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:19:26.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.642+0000 7f2aa1a10640 1 -- 172.21.15.40:0/1972966188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a9c09f1b0 msgr2=0x7f2a9c1cab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:26.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.642+0000 7f2a9b7fe640 1 -- 172.21.15.40:0/1972966188 reap_dead start 2024-04-22T08:19:26.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.642+0000 7f2aa1a10640 1 --2- 172.21.15.40:0/1972966188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a9c09f1b0 0x7f2a9c1cab20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2a9400d800 tx=0x7f2a9400dcc0 comp rx=0 tx=0).stop 2024-04-22T08:19:26.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.642+0000 7f2aa1a10640 1 -- 172.21.15.40:0/1972966188 shutdown_connections 2024-04-22T08:19:26.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.642+0000 7f2aa1a10640 1 --2- 172.21.15.40:0/1972966188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a9c09f1b0 0x7f2a9c1cab20 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:26.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.642+0000 7f2aa1a10640 1 -- 172.21.15.40:0/1972966188 >> 172.21.15.40:0/1972966188 conn(0x7f2a9c0a0ae0 msgr2=0x7f2a9c133010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:19:26.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.642+0000 7f2aa1a10640 1 -- 172.21.15.40:0/1972966188 shutdown_connections 2024-04-22T08:19:26.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.642+0000 7f2aa1a10640 1 -- 172.21.15.40:0/1972966188 wait complete. 2024-04-22T08:19:26.667 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:19:26.667 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-22T08:19:26.667 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-22T08:19:26.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.767+0000 7f2e1e44b640 10 monclient: get_monmap_and_config 2024-04-22T08:19:26.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.767+0000 7f2e1e44b640 10 monclient: build_initial_monmap 2024-04-22T08:19:26.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.767+0000 7f2e1e44b640 10 monclient: monmap: 2024-04-22T08:19:26.767 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:19:26.767 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:26.767 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:19:26.768501+0000 2024-04-22T08:19:26.767 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:19:26.768501+0000 2024-04-22T08:19:26.767 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:19:26.767 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:26.767 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:19:26.768 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:19:26.768 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:19:26.768 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:19:26.768 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:19:26.768 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:26.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.768+0000 7f2e1e44b640 1 Processor -- start 2024-04-22T08:19:26.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.768+0000 7f2e1e44b640 1 -- start start 2024-04-22T08:19:26.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.768+0000 7f2e1e44b640 10 monclient: init 2024-04-22T08:19:26.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.768+0000 7f2e1e44b640 10 monclient: _reopen_session rank -1 2024-04-22T08:19:26.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.768+0000 7f2e1e44b640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-04-22T08:19:26.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.769+0000 7f2e1e44b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2e18136070 addr v1:172.21.15.40:6789/0 2024-04-22T08:19:26.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.769+0000 7f2e1e44b640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2e1809abd0 0x7f2e1813aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:26.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.769+0000 7f2e1e44b640 10 monclient(hunting): picked mon.noname-a con 0x7f2e1809abd0 addr v2:172.21.15.40:3300/0 2024-04-22T08:19:26.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.769+0000 7f2e1e44b640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e1813b410 0x7f2e1813d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:26.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.769+0000 7f2e1e44b640 10 monclient(hunting): picked mon.noname-b con 0x7f2e1813b410 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:19:26.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.769+0000 7f2e1e44b640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e1806c6a0 con 0x7f2e18136070 2024-04-22T08:19:26.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.769+0000 7f2e1e44b640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:26.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.769+0000 7f2e1e44b640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e1809a1f0 con 0x7f2e1809abd0 2024-04-22T08:19:26.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.769+0000 7f2e1e44b640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:26.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.769+0000 7f2e1e44b640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1809a820 con 0x7f2e1813b410 2024-04-22T08:19:26.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.769+0000 7f2e1e44b640 10 monclient(hunting): _renew_subs 2024-04-22T08:19:26.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.769+0000 7f2e1e44b640 10 monclient(hunting): authenticate will time out at 1240.762862s 2024-04-22T08:19:26.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.769+0000 7f2e1c9e8640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e1813b410 0x7f2e1813d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:26.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.769+0000 7f2e177fe640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2e1809abd0 0x7f2e1813aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:26.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.769+0000 7f2e1c9e8640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e1813b410 0x7f2e1813d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:37016/0 (socket says 172.21.15.40:37016) 2024-04-22T08:19:26.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.769+0000 7f2e1c9e8640 1 -- 172.21.15.40:0/2225479297 learned_addr learned my addr 172.21.15.40:0/2225479297 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:19:26.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.769+0000 7f2e1c9e8640 10 monclient(hunting): get_auth_request con 0x7f2e1813b410 auth_method 0 2024-04-22T08:19:26.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.769+0000 7f2e1c9e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:26.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.769+0000 7f2e1c9e8640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:26.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.769+0000 7f2e1c9e8640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:26.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.769+0000 7f2e177fe640 10 monclient(hunting): get_auth_request con 0x7f2e1809abd0 auth_method 0 2024-04-22T08:19:26.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.769+0000 7f2e177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:26.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.769+0000 7f2e177fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:26.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.769+0000 7f2e177fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:26.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.769+0000 7f2e1c9e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:26.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.769+0000 7f2e1c9e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:26.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.769+0000 7f2e1c9e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:26.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.769+0000 7f2e1c9e8640 10 monclient(hunting): handle_auth_done global_id 54674 payload 274 2024-04-22T08:19:26.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.769+0000 7f2e1c9e8640 1 -- 172.21.15.40:0/2225479297 >> v2:172.21.15.40:3300/0 conn(0x7f2e1809abd0 msgr2=0x7f2e1813aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:19:26.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.769+0000 7f2e1c9e8640 1 --2- 172.21.15.40:0/2225479297 >> v2:172.21.15.40:3300/0 conn(0x7f2e1809abd0 0x7f2e1813aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:26.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.769+0000 7f2e1c9e8640 1 -- 172.21.15.40:0/2225479297 >> v1:172.21.15.40:6789/0 conn(0x7f2e18136070 legacy=0x7f2e18136450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:19:26.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.769+0000 7f2e1c9e8640 10 monclient: _finish_hunting 0 2024-04-22T08:19:26.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.769+0000 7f2e1c9e8640 1 monclient: found mon.noname-b 2024-04-22T08:19:26.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.769+0000 7f2e1c9e8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:19:26.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.769+0000 7f2e1c9e8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:19:26.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.769+0000 7f2e1c9e8640 1 -- 172.21.15.40:0/2225479297 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e1806bb50 con 0x7f2e1813b410 2024-04-22T08:19:26.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.772+0000 7f2e1c9e8640 1 -- 172.21.15.40:0/2225479297 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e1813b410 msgr2=0x7f2e1813d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:26.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.772+0000 7f2e1c9e8640 1 --2- 172.21.15.40:0/2225479297 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e1813b410 0x7f2e1813d800 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f2e10009e50 tx=0x7f2e1002bdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a46e5323fc8293d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:26.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.772+0000 7f2e16ffd640 1 -- 172.21.15.40:0/2225479297 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e1002cb60 con 0x7f2e1813b410 2024-04-22T08:19:26.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.772+0000 7f2e16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:26.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.772+0000 7f2e16ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:19:26.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.772+0000 7f2e16ffd640 10 monclient: dump: 2024-04-22T08:19:26.779 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:26.779 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:26.779 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.779 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.779 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:26.779 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:26.780 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:26.780 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:26.780 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:26.780 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:26.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.772+0000 7f2e16ffd640 10 monclient: _finish_auth 0 2024-04-22T08:19:26.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.772+0000 7f2e16ffd640 10 monclient: _check_auth_tickets 2024-04-22T08:19:26.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.772+0000 7f2e16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:19:26.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.772+0000 7f2e16ffd640 1 -- 172.21.15.40:0/2225479297 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e1002ccc0 con 0x7f2e1813b410 2024-04-22T08:19:26.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.772+0000 7f2e1e44b640 5 monclient: authenticate success, global_id 54674 2024-04-22T08:19:26.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.772+0000 7f2e1e44b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:19:26.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.772+0000 7f2e16ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:19:26.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.772+0000 7f2e1e44b640 10 monclient: get_monmap_and_config success 2024-04-22T08:19:26.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.772+0000 7f2e16ffd640 1 -- 172.21.15.40:0/2225479297 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e10034610 con 0x7f2e1813b410 2024-04-22T08:19:26.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.772+0000 7f2e16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:26.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.772+0000 7f2e1e44b640 4 set_mon_vals no callback set 2024-04-22T08:19:26.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.772+0000 7f2e16ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:19:26.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.772+0000 7f2e16ffd640 10 monclient: dump: 2024-04-22T08:19:26.781 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:26.781 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:26.781 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.781 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.781 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:26.781 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:26.781 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:26.781 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:26.781 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:26.781 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:26.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.773+0000 7f2e1e44b640 10 monclient: shutdown 2024-04-22T08:19:26.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.773+0000 7f2e1e44b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:19:26.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.773+0000 7f2e1e44b640 1 -- 172.21.15.40:0/2225479297 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e1813b410 msgr2=0x7f2e1813d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:26.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.773+0000 7f2e1e44b640 1 --2- 172.21.15.40:0/2225479297 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e1813b410 0x7f2e1813d800 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f2e10009e50 tx=0x7f2e1002bdb0 comp rx=0 tx=0).stop 2024-04-22T08:19:26.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.773+0000 7f2e1e44b640 1 -- 172.21.15.40:0/2225479297 shutdown_connections 2024-04-22T08:19:26.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.773+0000 7f2e1e44b640 1 --2- 172.21.15.40:0/2225479297 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e1813b410 0x7f2e1813d800 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:26.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.773+0000 7f2e1e44b640 1 --2- 172.21.15.40:0/2225479297 >> v2:172.21.15.40:3300/0 conn(0x7f2e1809abd0 0x7f2e1813aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:26.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.773+0000 7f2e1e44b640 1 -- 172.21.15.40:0/2225479297 >> 172.21.15.40:0/2225479297 conn(0x7f2e180a1b60 msgr2=0x7f2e180a1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:19:26.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.773+0000 7f2e1e44b640 1 -- 172.21.15.40:0/2225479297 shutdown_connections 2024-04-22T08:19:26.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.773+0000 7f2e1e44b640 1 -- 172.21.15.40:0/2225479297 wait complete. 2024-04-22T08:19:26.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.773+0000 7f2e1e44b640 10 monclient: build_initial_monmap 2024-04-22T08:19:26.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.773+0000 7f2e1e44b640 10 monclient: monmap: 2024-04-22T08:19:26.782 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:19:26.782 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:19:26.782 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:19:26.782 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:19:26.783 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:19:26.783 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:26.783 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:19:26.783 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:19:26.783 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:19:26.783 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:26.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.774+0000 7f2e1e44b640 1 Processor -- start 2024-04-22T08:19:26.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.774+0000 7f2e1e44b640 1 -- start start 2024-04-22T08:19:26.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.774+0000 7f2e1e44b640 10 monclient: init 2024-04-22T08:19:26.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.774+0000 7f2e1e44b640 10 monclient: _reopen_session rank -1 2024-04-22T08:19:26.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.774+0000 7f2e1e44b640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-22T08:19:26.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.774+0000 7f2e1e44b640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e1809abd0 0x7f2e1811fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:26.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.774+0000 7f2e1e44b640 10 monclient(hunting): picked mon.noname-c con 0x7f2e1809abd0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:19:26.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.774+0000 7f2e1e44b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e18136070 0x7f2e18134aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:26.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.774+0000 7f2e1e44b640 10 monclient(hunting): picked mon.noname-b con 0x7f2e18136070 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:19:26.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.774+0000 7f2e1e44b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e1813b410 0x7f2e1806fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:26.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.774+0000 7f2e1e44b640 10 monclient(hunting): picked mon.noname-a con 0x7f2e1813b410 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:19:26.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.774+0000 7f2e1e44b640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:26.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.774+0000 7f2e1e44b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1813ace0 con 0x7f2e1813b410 2024-04-22T08:19:26.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.774+0000 7f2e1e44b640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:26.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.774+0000 7f2e1e44b640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e18071ba0 con 0x7f2e18136070 2024-04-22T08:19:26.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.774+0000 7f2e1e44b640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:26.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.774+0000 7f2e1e44b640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e181cb3e0 con 0x7f2e1809abd0 2024-04-22T08:19:26.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.774+0000 7f2e1e44b640 10 monclient(hunting): _renew_subs 2024-04-22T08:19:26.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.774+0000 7f2e1e44b640 10 monclient(hunting): authenticate will time out at 1240.768615s 2024-04-22T08:19:26.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.774+0000 7f2e17fff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e1809abd0 0x7f2e1811fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:26.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.774+0000 7f2e1c9e8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e1813b410 0x7f2e1806fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:26.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.774+0000 7f2e17fff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e1809abd0 0x7f2e1811fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:37024/0 (socket says 172.21.15.40:37024) 2024-04-22T08:19:26.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.774+0000 7f2e1c9e8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e1813b410 0x7f2e1806fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53008/0 (socket says 172.21.15.40:53008) 2024-04-22T08:19:26.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.774+0000 7f2e17fff640 1 -- 172.21.15.40:0/4206542068 learned_addr learned my addr 172.21.15.40:0/4206542068 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:19:26.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.774+0000 7f2e17fff640 10 monclient(hunting): get_auth_request con 0x7f2e1809abd0 auth_method 0 2024-04-22T08:19:26.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.774+0000 7f2e17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:26.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.774+0000 7f2e17fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:26.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.774+0000 7f2e17fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:26.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.774+0000 7f2e1c9e8640 10 monclient(hunting): get_auth_request con 0x7f2e1813b410 auth_method 0 2024-04-22T08:19:26.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.775+0000 7f2e1c9e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:26.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.775+0000 7f2e1c9e8640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:26.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.775+0000 7f2e1c9e8640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:26.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.775+0000 7f2e17fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:26.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.775+0000 7f2e17fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:26.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.775+0000 7f2e17fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:26.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.775+0000 7f2e1c9e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:26.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.775+0000 7f2e1c9e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:26.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.775+0000 7f2e1c9e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:26.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.775+0000 7f2e177fe640 1 --2- 172.21.15.40:0/4206542068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e18136070 0x7f2e18134aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:26.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.775+0000 7f2e177fe640 10 monclient(hunting): get_auth_request con 0x7f2e18136070 auth_method 0 2024-04-22T08:19:26.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.775+0000 7f2e177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:26.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.775+0000 7f2e177fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:26.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.775+0000 7f2e177fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:26.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.775+0000 7f2e17fff640 10 monclient(hunting): handle_auth_done global_id 54677 payload 824 2024-04-22T08:19:26.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.775+0000 7f2e17fff640 1 -- 172.21.15.40:0/4206542068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e18136070 msgr2=0x7f2e18134aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:26.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.775+0000 7f2e17fff640 1 --2- 172.21.15.40:0/4206542068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e18136070 0x7f2e18134aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:26.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.775+0000 7f2e17fff640 1 -- 172.21.15.40:0/4206542068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e1813b410 msgr2=0x7f2e1806fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:26.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.775+0000 7f2e17fff640 1 --2- 172.21.15.40:0/4206542068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e1813b410 0x7f2e1806fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:26.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.775+0000 7f2e17fff640 10 monclient: _finish_hunting 0 2024-04-22T08:19:26.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.775+0000 7f2e17fff640 1 monclient: found mon.noname-c 2024-04-22T08:19:26.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.775+0000 7f2e17fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:19:26.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.775+0000 7f2e17fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:19:26.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.775+0000 7f2e17fff640 1 -- 172.21.15.40:0/4206542068 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e10009b30 con 0x7f2e1809abd0 2024-04-22T08:19:26.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.775+0000 7f2e1c9e8640 1 --2- 172.21.15.40:0/4206542068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e1813b410 0x7f2e1806fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:19:26.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.775+0000 7f2e177fe640 1 --2- 172.21.15.40:0/4206542068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e18136070 0x7f2e18134aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T08:19:26.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.777+0000 7f2e17fff640 1 -- 172.21.15.40:0/4206542068 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e1809abd0 msgr2=0x7f2e1811fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:26.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.777+0000 7f2e17fff640 1 --2- 172.21.15.40:0/4206542068 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e1809abd0 0x7f2e1811fc50 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c00c430 tx=0x7f2e0c00c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:26.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.777+0000 7f2e14ff9640 1 -- 172.21.15.40:0/4206542068 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e0c0119a0 con 0x7f2e1809abd0 2024-04-22T08:19:26.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.777+0000 7f2e14ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:26.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.777+0000 7f2e14ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:19:26.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.777+0000 7f2e14ff9640 10 monclient: dump: 2024-04-22T08:19:26.787 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:26.787 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:26.787 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.788 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.788 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:26.788 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:26.788 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:26.788 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:26.788 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:26.788 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:26.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.777+0000 7f2e14ff9640 10 monclient: _finish_auth 0 2024-04-22T08:19:26.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.777+0000 7f2e14ff9640 10 monclient: _check_auth_tickets 2024-04-22T08:19:26.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.777+0000 7f2e14ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:19:26.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.777+0000 7f2e14ff9640 1 -- 172.21.15.40:0/4206542068 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e0c011b00 con 0x7f2e1809abd0 2024-04-22T08:19:26.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.777+0000 7f2e1e44b640 5 monclient: authenticate success, global_id 54677 2024-04-22T08:19:26.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.777+0000 7f2e1e44b640 10 monclient: _renew_subs 2024-04-22T08:19:26.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.777+0000 7f2e1e44b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:19:26.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.777+0000 7f2e1e44b640 1 -- 172.21.15.40:0/4206542068 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e1806c890 con 0x7f2e1809abd0 2024-04-22T08:19:26.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.777+0000 7f2e14ff9640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:19:26.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.777+0000 7f2e14ff9640 1 -- 172.21.15.40:0/4206542068 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e0c012420 con 0x7f2e1809abd0 2024-04-22T08:19:26.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.777+0000 7f2e14ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:26.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.777+0000 7f2e14ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:19:26.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.777+0000 7f2e14ff9640 10 monclient: dump: 2024-04-22T08:19:26.789 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:26.789 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:26.789 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.789 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.789 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:26.789 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:26.789 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:26.789 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:26.789 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:26.790 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:26.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.777+0000 7f2e167fc640 4 set_mon_vals no callback set 2024-04-22T08:19:26.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.777+0000 7f2e1e44b640 10 monclient: _renew_subs 2024-04-22T08:19:26.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.777+0000 7f2e1e44b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:19:26.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.777+0000 7f2e1e44b640 1 -- 172.21.15.40:0/4206542068 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e18120bb0 con 0x7f2e1809abd0 2024-04-22T08:19:26.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.778+0000 7f2e1e44b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:19:26.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.778+0000 7f2e1e44b640 1 --2- 172.21.15.40:0/4206542068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e1802b200 0x7f2e1802b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:26.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.778+0000 7f2e1e44b640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:26.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.778+0000 7f2e1e44b640 1 -- 172.21.15.40:0/4206542068 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e180667a0 con 0x7f2e1802b200 2024-04-22T08:19:26.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.778+0000 7f2e14ff9640 1 -- 172.21.15.40:0/4206542068 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f2e0c012640 con 0x7f2e1809abd0 2024-04-22T08:19:26.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.778+0000 7f2e177fe640 1 --2- 172.21.15.40:0/4206542068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e1802b200 0x7f2e1802b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:26.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.778+0000 7f2e14ff9640 1 --2- 172.21.15.40:0/4206542068 >> v2:172.21.15.189:6800/30168 conn(0x7f2e04037820 0x7f2e04039ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:26.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.778+0000 7f2e177fe640 10 monclient: get_auth_request con 0x7f2e1802b200 auth_method 0 2024-04-22T08:19:26.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.778+0000 7f2e177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:26.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.778+0000 7f2e177fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:26.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.778+0000 7f2e177fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:26.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.778+0000 7f2e14ff9640 1 -- 172.21.15.40:0/4206542068 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f2e0c04a310 con 0x7f2e1809abd0 2024-04-22T08:19:26.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.778+0000 7f2e177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:26.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.778+0000 7f2e177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:26.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.778+0000 7f2e177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:26.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.779+0000 7f2e1c9e8640 1 --2- 172.21.15.40:0/4206542068 >> v2:172.21.15.189:6800/30168 conn(0x7f2e04037820 0x7f2e04039ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:26.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.779+0000 7f2e1c9e8640 10 monclient: get_auth_request con 0x7f2e04037820 auth_method 0 2024-04-22T08:19:26.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.779+0000 7f2e177fe640 10 monclient(hunting): handle_auth_done global_id 64492 payload 824 2024-04-22T08:19:26.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.779+0000 7f2e177fe640 1 -- 172.21.15.40:0/4206542068 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2e1809aff0 con 0x7f2e1802b200 2024-04-22T08:19:26.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.781+0000 7f2e177fe640 1 -- 172.21.15.40:0/4206542068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e1802b200 msgr2=0x7f2e1802b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:26.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.781+0000 7f2e177fe640 1 --2- 172.21.15.40:0/4206542068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e1802b200 0x7f2e1802b5e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2e18120ae0 tx=0x7f2e0800f9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:26.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.781+0000 7f2e1c9e8640 1 --2- 172.21.15.40:0/4206542068 >> v2:172.21.15.189:6800/30168 conn(0x7f2e04037820 0x7f2e04039ce0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f2e10035000 tx=0x7f2e10030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:26.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.781+0000 7f2e14ff9640 1 -- 172.21.15.40:0/4206542068 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e08004230 con 0x7f2e1802b200 2024-04-22T08:19:26.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.781+0000 7f2e14ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:26.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.781+0000 7f2e14ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:19:26.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.781+0000 7f2e14ff9640 10 monclient: dump: 2024-04-22T08:19:26.794 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:26.794 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:26.794 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.794 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.795 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:26.795 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:26.795 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:26.795 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:26.795 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:26.795 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:26.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.781+0000 7f2e14ff9640 1 -- 172.21.15.40:0/4206542068 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f2e08004390 con 0x7f2e1802b200 2024-04-22T08:19:26.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.781+0000 7f2e14ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:19:26.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.781+0000 7f2e14ff9640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:19:26.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.781+0000 7f2e14ff9640 1 -- 172.21.15.40:0/4206542068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e1802b200 msgr2=0x7f2e1802b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:26.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.781+0000 7f2e14ff9640 1 --2- 172.21.15.40:0/4206542068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e1802b200 0x7f2e1802b5e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2e18120ae0 tx=0x7f2e0800f9e0 comp rx=0 tx=0).stop 2024-04-22T08:19:26.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.782+0000 7f2e14ff9640 1 -- 172.21.15.40:0/4206542068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e1802b200 msgr2=0x7f2e1802b5e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:19:26.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.782+0000 7f2e14ff9640 1 --2- 172.21.15.40:0/4206542068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e1802b200 0x7f2e1802b5e0 secure :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2e18120ae0 tx=0x7f2e0800f9e0 comp rx=0 tx=0).stop 2024-04-22T08:19:26.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.788+0000 7f2e1e44b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:19:26.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.788+0000 7f2e1e44b640 1 --2- 172.21.15.40:0/4206542068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e1802bb20 0x7f2e1802bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:26.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.788+0000 7f2e1e44b640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:26.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.788+0000 7f2e1e44b640 1 -- 172.21.15.40:0/4206542068 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e180999f0 con 0x7f2e1802bb20 2024-04-22T08:19:26.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.788+0000 7f2e177fe640 1 --2- 172.21.15.40:0/4206542068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e1802bb20 0x7f2e1802bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:26.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.788+0000 7f2e177fe640 10 monclient: get_auth_request con 0x7f2e1802bb20 auth_method 0 2024-04-22T08:19:26.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.788+0000 7f2e177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:26.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.788+0000 7f2e177fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:26.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.788+0000 7f2e177fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:26.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.788+0000 7f2e177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:26.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.788+0000 7f2e177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:26.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.788+0000 7f2e177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:26.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.788+0000 7f2e177fe640 10 monclient(hunting): handle_auth_done global_id 64495 payload 824 2024-04-22T08:19:26.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.788+0000 7f2e177fe640 1 -- 172.21.15.40:0/4206542068 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2e180a28f0 con 0x7f2e1802bb20 2024-04-22T08:19:26.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.791+0000 7f2e177fe640 1 -- 172.21.15.40:0/4206542068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e1802bb20 msgr2=0x7f2e1802bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:26.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.791+0000 7f2e177fe640 1 --2- 172.21.15.40:0/4206542068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e1802bb20 0x7f2e1802bf30 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2e08019000 tx=0x7f2e080046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:26.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.791+0000 7f2e14ff9640 1 -- 172.21.15.40:0/4206542068 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e08024070 con 0x7f2e1802bb20 2024-04-22T08:19:26.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.791+0000 7f2e14ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:26.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.791+0000 7f2e14ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:19:26.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.791+0000 7f2e14ff9640 10 monclient: dump: 2024-04-22T08:19:26.797 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:26.797 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:26.797 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.797 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.797 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:26.797 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:26.797 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:26.797 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:26.797 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:26.797 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:26.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.791+0000 7f2e14ff9640 1 -- 172.21.15.40:0/4206542068 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4801 (secure 0 0 0) 0x7f2e08003d90 con 0x7f2e1802bb20 2024-04-22T08:19:26.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.791+0000 7f2e14ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:19:26.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.791+0000 7f2e14ff9640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:19:26.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.791+0000 7f2e14ff9640 1 -- 172.21.15.40:0/4206542068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e1802bb20 msgr2=0x7f2e1802bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:26.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.791+0000 7f2e14ff9640 1 --2- 172.21.15.40:0/4206542068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e1802bb20 0x7f2e1802bf30 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2e08019000 tx=0x7f2e080046c0 comp rx=0 tx=0).stop 2024-04-22T08:19:26.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.791+0000 7f2e14ff9640 1 -- 172.21.15.40:0/4206542068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e1802bb20 msgr2=0x7f2e1802bf30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:19:26.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.791+0000 7f2e14ff9640 1 --2- 172.21.15.40:0/4206542068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e1802bb20 0x7f2e1802bf30 secure :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2e08019000 tx=0x7f2e080046c0 comp rx=0 tx=0).stop 2024-04-22T08:19:26.798 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:19:26.798 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:19:26.799 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:19:26.799 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:19:26.799 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 66, 2024-04-22T08:19:26.799 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:19:26.799 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:19:26.799 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:19:26.799 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:19:26.799 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:19:26.799 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 3, 2024-04-22T08:19:26.799 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:19:26.799 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:19:26.799 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:19:26.799 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:19:26.799 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:19:26.800 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:19:26.800 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:19:26.800 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:19:26.800 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:19:26.800 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:19:26.800 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:19:26.800 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:19:26.800 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:19:26.800 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:19:26.800 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:19:26.800 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:19:26.800 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:19:26.800 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:19:26.800 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:19:26.801 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:19:26.801 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:19:26.801 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:19:26.801 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:19:26.801 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:19:26.801 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:19:26.801 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:19:26.801 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:26.801 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:26.801 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:19:26.801 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:19:26.801 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:19:26.801 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:19:26.801 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:19:26.802 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:19:26.802 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:19:26.802 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:19:26.802 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:19:26.802 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:19:26.802 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:19:26.802 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:19:26.802 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:19:26.802 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:19:26.802 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:19:26.802 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:19:26.802 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:19:26.802 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:19:26.802 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:19:26.802 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:19:26.803 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:19:26.803 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:19:26.803 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:19:26.803 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:19:26.803 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:19:26.803 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:19:26.803 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:19:26.803 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:19:26.803 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:19:26.803 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:26.803 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:19:26.803 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:26.803 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:19:26.803 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:19:26.804 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:19:26.804 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:19:26.804 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:26.804 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:19:26.804 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:19:26.804 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:26.804 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:26.804 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:26.804 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:19:26.804 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:19:26.804 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:26.804 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:26.804 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:26.804 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:26.804 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:19:26.805 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:19:26.805 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:19:26.805 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:19:26.805 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:19:26.805 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:26.805 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:26.805 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:19:26.805 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:19:26.805 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:19:26.805 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:19:26.805 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:26.805 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:19:26.805 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:19:26.805 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:26.805 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:26.806 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:26.806 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:19:26.806 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:19:26.806 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:26.806 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:26.806 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:26.806 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:26.806 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:19:26.806 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:19:26.806 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:19:26.806 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:19:26.806 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:19:26.806 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:26.806 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:26.807 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:19:26.807 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:19:26.807 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:19:26.807 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:19:26.807 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:26.807 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:19:26.807 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:19:26.807 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:26.807 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:26.807 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:26.807 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:19:26.807 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:19:26.807 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:26.807 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:26.808 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:26.808 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:26.808 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:19:26.808 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:19:26.808 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:19:26.808 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:19:26.808 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:19:26.808 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:26.808 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:26.808 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:26.808 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:19:26.808 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:19:26.808 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:26.808 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:19:26.809 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:19:26.809 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:19:26.809 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:26.809 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:19:26.809 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:19:26.809 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:26.809 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:19:26.809 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:19:26.809 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 3 2024-04-22T08:19:26.809 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:26.809 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:19:26.809 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:19:26.809 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:26.809 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:19:26.809 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:19:26.810 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 3 2024-04-22T08:19:26.810 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:26.810 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:19:26.810 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": [ 2024-04-22T08:19:26.810 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:26.810 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:19:26.810 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:19:26.810 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:19:26.810 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:26.810 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:26.810 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:26.810 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:19:26.810 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:19:26.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.791+0000 7f2e1e44b640 1 -- 172.21.15.40:0/4206542068 >> v2:172.21.15.189:6800/30168 conn(0x7f2e04037820 msgr2=0x7f2e04039ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:26.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.791+0000 7f2e1e44b640 1 --2- 172.21.15.40:0/4206542068 >> v2:172.21.15.189:6800/30168 conn(0x7f2e04037820 0x7f2e04039ce0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f2e10035000 tx=0x7f2e10030000 comp rx=0 tx=0).stop 2024-04-22T08:19:26.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.792+0000 7f2e1e44b640 10 monclient: shutdown 2024-04-22T08:19:26.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.792+0000 7f2e1e44b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:19:26.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.792+0000 7f2e1e44b640 1 -- 172.21.15.40:0/4206542068 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e1809abd0 msgr2=0x7f2e1811fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:26.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.792+0000 7f2e1c9e8640 1 -- 172.21.15.40:0/4206542068 reap_dead start 2024-04-22T08:19:26.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.792+0000 7f2e1e44b640 1 --2- 172.21.15.40:0/4206542068 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e1809abd0 0x7f2e1811fc50 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c00c430 tx=0x7f2e0c00c8f0 comp rx=0 tx=0).stop 2024-04-22T08:19:26.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.792+0000 7f2e1e44b640 1 -- 172.21.15.40:0/4206542068 shutdown_connections 2024-04-22T08:19:26.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.792+0000 7f2e1e44b640 1 --2- 172.21.15.40:0/4206542068 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e1809abd0 0x7f2e1811fc50 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:26.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.792+0000 7f2e1e44b640 1 -- 172.21.15.40:0/4206542068 >> 172.21.15.40:0/4206542068 conn(0x7f2e180a1b60 msgr2=0x7f2e18136c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:19:26.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.792+0000 7f2e1e44b640 1 -- 172.21.15.40:0/4206542068 shutdown_connections 2024-04-22T08:19:26.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.792+0000 7f2e1e44b640 1 -- 172.21.15.40:0/4206542068 wait complete. 2024-04-22T08:19:26.813 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:19:26.813 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-04-22T08:19:26.813 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-04-22T08:19:26.813 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-04-22T08:19:26.813 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-04-22T08:19:26.814 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-22T08:19:26.814 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T08:19:26.823 INFO:tasks.ceph.mon.a:Stopped 2024-04-22T08:19:26.823 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-22T08:19:26.824 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:19:26.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.963+0000 7f9536551640 10 monclient: get_monmap_and_config 2024-04-22T08:19:26.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.963+0000 7f9536551640 10 monclient: build_initial_monmap 2024-04-22T08:19:26.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.963+0000 7f9536551640 10 monclient: monmap: 2024-04-22T08:19:26.963 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:19:26.963 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:26.963 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:19:26.963985+0000 2024-04-22T08:19:26.963 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:19:26.963985+0000 2024-04-22T08:19:26.963 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:19:26.963 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:26.963 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:19:26.963 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:19:26.963 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:19:26.963 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:19:26.963 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:19:26.963 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:26.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.963+0000 7f9536551640 1 Processor -- start 2024-04-22T08:19:26.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.964+0000 7f9536551640 1 -- start start 2024-04-22T08:19:26.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.964+0000 7f9536551640 10 monclient: init 2024-04-22T08:19:26.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.964+0000 7f9536551640 10 monclient: _reopen_session rank -1 2024-04-22T08:19:26.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.964+0000 7f9536551640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-04-22T08:19:26.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.964+0000 7f9536551640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f953009ee10 0x7f95300a32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:26.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.964+0000 7f9536551640 10 monclient(hunting): picked mon.noname-a con 0x7f953009ee10 addr v2:172.21.15.40:3300/0 2024-04-22T08:19:26.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.964+0000 7f9536551640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f95300a1d30 0x7f95300a2180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:26.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.964+0000 7f9536551640 10 monclient(hunting): picked mon.noname-c con 0x7f95300a1d30 addr v2:172.21.15.189:3300/0 2024-04-22T08:19:26.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.964+0000 7f9536551640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9530135d90 0x7f95301361a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:26.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.964+0000 7f9536551640 10 monclient(hunting): picked mon.noname-b con 0x7f9530135d90 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:19:26.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.964+0000 7f9536551640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:26.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.964+0000 7f9536551640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f953009f1f0 con 0x7f953009ee10 2024-04-22T08:19:26.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.964+0000 7f9536551640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:26.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.964+0000 7f9536551640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95300a39c0 con 0x7f95300a1d30 2024-04-22T08:19:26.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.964+0000 7f9536551640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:26.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.964+0000 7f9536551640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f953009fd30 con 0x7f9530135d90 2024-04-22T08:19:26.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.964+0000 7f9536551640 10 monclient(hunting): _renew_subs 2024-04-22T08:19:26.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.964+0000 7f9536551640 10 monclient(hunting): authenticate will time out at 1240.958345s 2024-04-22T08:19:26.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.964+0000 7f952ffff640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f953009ee10 msgr2=0x7f95300a32b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:19:26.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.964+0000 7f952ffff640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f953009ee10 0x7f95300a32b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:19:26.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.964+0000 7f9534aee640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9530135d90 0x7f95301361a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:26.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.964+0000 7f9534aee640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9530135d90 0x7f95301361a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:37032/0 (socket says 172.21.15.40:37032) 2024-04-22T08:19:26.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.964+0000 7f9534aee640 1 -- 172.21.15.40:0/922225015 learned_addr learned my addr 172.21.15.40:0/922225015 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:19:26.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.964+0000 7f9534aee640 10 monclient(hunting): get_auth_request con 0x7f9530135d90 auth_method 0 2024-04-22T08:19:26.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.964+0000 7f9534aee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:26.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.964+0000 7f9534aee640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:26.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.964+0000 7f9534aee640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:26.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.964+0000 7f9534aee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:26.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.964+0000 7f9534aee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:26.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.964+0000 7f9534aee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:26.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.964+0000 7f9534aee640 10 monclient(hunting): handle_auth_done global_id 54680 payload 274 2024-04-22T08:19:26.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.964+0000 7f9534aee640 1 -- 172.21.15.40:0/922225015 >> v2:172.21.15.189:3300/0 conn(0x7f95300a1d30 msgr2=0x7f95300a2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:19:26.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.964+0000 7f952f7fe640 1 --2- 172.21.15.40:0/922225015 >> v2:172.21.15.189:3300/0 conn(0x7f95300a1d30 0x7f95300a2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:26.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.964+0000 7f9534aee640 1 --2- 172.21.15.40:0/922225015 >> v2:172.21.15.189:3300/0 conn(0x7f95300a1d30 0x7f95300a2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:26.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.964+0000 7f9534aee640 1 -- 172.21.15.40:0/922225015 >> v2:172.21.15.40:3300/0 conn(0x7f953009ee10 msgr2=0x7f95300a32b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:19:26.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.964+0000 7f9534aee640 1 --2- 172.21.15.40:0/922225015 >> v2:172.21.15.40:3300/0 conn(0x7f953009ee10 0x7f95300a32b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:26.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.964+0000 7f9534aee640 10 monclient: _finish_hunting 0 2024-04-22T08:19:26.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.964+0000 7f9534aee640 1 monclient: found mon.noname-b 2024-04-22T08:19:26.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.964+0000 7f9534aee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:19:26.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.964+0000 7f9534aee640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:19:26.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.965+0000 7f9534aee640 1 -- 172.21.15.40:0/922225015 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95300a02f0 con 0x7f9530135d90 2024-04-22T08:19:26.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.965+0000 7f952f7fe640 10 monclient: get_auth_request con 0x7f95300a1d30 auth_method 0 2024-04-22T08:19:26.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.965+0000 7f952f7fe640 1 --2- 172.21.15.40:0/922225015 >> v2:172.21.15.189:3300/0 conn(0x7f95300a1d30 0x7f95300a2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T08:19:26.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.967+0000 7f9534aee640 1 -- 172.21.15.40:0/922225015 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9530135d90 msgr2=0x7f95301361a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:26.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.967+0000 7f9534aee640 1 --2- 172.21.15.40:0/922225015 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9530135d90 0x7f95301361a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f953006c620 tx=0x7f952802bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c05363a2c41a4a3d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:26.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.967+0000 7f952effd640 1 -- 172.21.15.40:0/922225015 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f952802cb90 con 0x7f9530135d90 2024-04-22T08:19:26.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.967+0000 7f952effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:26.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.967+0000 7f952effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:19:26.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.967+0000 7f952effd640 10 monclient: dump: 2024-04-22T08:19:26.975 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:26.975 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:26.975 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.975 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.975 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:26.975 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:26.975 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:26.975 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:26.975 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:26.975 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:26.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.968+0000 7f952effd640 10 monclient: _finish_auth 0 2024-04-22T08:19:26.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.968+0000 7f952effd640 10 monclient: _check_auth_tickets 2024-04-22T08:19:26.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.968+0000 7f952effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:19:26.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.968+0000 7f9536551640 5 monclient: authenticate success, global_id 54680 2024-04-22T08:19:26.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.968+0000 7f952effd640 1 -- 172.21.15.40:0/922225015 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f952802ccf0 con 0x7f9530135d90 2024-04-22T08:19:26.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.968+0000 7f9536551640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:19:26.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.968+0000 7f952effd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:19:26.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.968+0000 7f9536551640 10 monclient: get_monmap_and_config success 2024-04-22T08:19:26.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.968+0000 7f952effd640 1 -- 172.21.15.40:0/922225015 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9528034610 con 0x7f9530135d90 2024-04-22T08:19:26.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.968+0000 7f9536551640 4 set_mon_vals no callback set 2024-04-22T08:19:26.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.968+0000 7f952effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:26.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.968+0000 7f952effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:19:26.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.968+0000 7f952effd640 10 monclient: dump: 2024-04-22T08:19:26.977 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:26.977 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:26.977 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.977 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.977 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:26.977 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:26.977 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:26.977 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:26.977 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:26.977 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:26.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.968+0000 7f9536551640 10 monclient: shutdown 2024-04-22T08:19:26.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.968+0000 7f9536551640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:19:26.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.968+0000 7f9536551640 1 -- 172.21.15.40:0/922225015 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9530135d90 msgr2=0x7f95301361a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:26.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.968+0000 7f9536551640 1 --2- 172.21.15.40:0/922225015 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9530135d90 0x7f95301361a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f953006c620 tx=0x7f952802bde0 comp rx=0 tx=0).stop 2024-04-22T08:19:26.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.968+0000 7f9536551640 1 -- 172.21.15.40:0/922225015 shutdown_connections 2024-04-22T08:19:26.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.968+0000 7f9536551640 1 --2- 172.21.15.40:0/922225015 >> v2:172.21.15.189:3300/0 conn(0x7f95300a1d30 0x7f95300a2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:26.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.968+0000 7f9536551640 1 --2- 172.21.15.40:0/922225015 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9530135d90 0x7f95301361a0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:26.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.968+0000 7f9536551640 1 --2- 172.21.15.40:0/922225015 >> v2:172.21.15.40:3300/0 conn(0x7f953009ee10 0x7f95300a32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:26.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.968+0000 7f9536551640 1 -- 172.21.15.40:0/922225015 >> 172.21.15.40:0/922225015 conn(0x7f95300a4ef0 msgr2=0x7f95300a5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:19:26.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.968+0000 7f9536551640 1 -- 172.21.15.40:0/922225015 shutdown_connections 2024-04-22T08:19:26.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.968+0000 7f9536551640 1 -- 172.21.15.40:0/922225015 wait complete. 2024-04-22T08:19:26.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.969+0000 7f9536551640 10 monclient: build_initial_monmap 2024-04-22T08:19:26.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.969+0000 7f9536551640 10 monclient: monmap: 2024-04-22T08:19:26.978 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:19:26.978 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:19:26.978 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:19:26.978 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:19:26.979 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:19:26.979 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:26.979 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:19:26.979 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:19:26.979 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:19:26.979 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:26.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.969+0000 7f9536551640 1 Processor -- start 2024-04-22T08:19:26.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.969+0000 7f9536551640 1 -- start start 2024-04-22T08:19:26.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.969+0000 7f9536551640 10 monclient: init 2024-04-22T08:19:26.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.970+0000 7f9536551640 10 monclient: _reopen_session rank -1 2024-04-22T08:19:26.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.970+0000 7f9536551640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-22T08:19:26.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.970+0000 7f9536551640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f953009ee10 0x7f95301ca770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:26.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.970+0000 7f9536551640 10 monclient(hunting): picked mon.noname-a con 0x7f953009ee10 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:19:26.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.970+0000 7f9536551640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f95300a1d30 0x7f9530134eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:26.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.970+0000 7f9536551640 10 monclient(hunting): picked mon.noname-c con 0x7f95300a1d30 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:19:26.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.970+0000 7f9536551640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9530135d90 0x7f95301353f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:26.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.970+0000 7f9536551640 10 monclient(hunting): picked mon.noname-b con 0x7f9530135d90 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:19:26.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.970+0000 7f9536551640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:26.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.970+0000 7f9536551640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9530121fd0 con 0x7f953009ee10 2024-04-22T08:19:26.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.970+0000 7f9536551640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:26.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.970+0000 7f9536551640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9530071670 con 0x7f9530135d90 2024-04-22T08:19:26.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.970+0000 7f9536551640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:26.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.970+0000 7f9536551640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95300a2730 con 0x7f95300a1d30 2024-04-22T08:19:26.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.970+0000 7f9536551640 10 monclient(hunting): _renew_subs 2024-04-22T08:19:26.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.970+0000 7f952ffff640 1 -- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f953009ee10 msgr2=0x7f95301ca770 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:19:26.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.970+0000 7f9536551640 10 monclient(hunting): authenticate will time out at 1240.963865s 2024-04-22T08:19:26.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.970+0000 7f952ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f953009ee10 0x7f95301ca770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:19:26.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.970+0000 7f952f7fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f95300a1d30 0x7f9530134eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:26.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.970+0000 7f952f7fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f95300a1d30 0x7f9530134eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:37048/0 (socket says 172.21.15.40:37048) 2024-04-22T08:19:26.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.970+0000 7f952f7fe640 1 -- 172.21.15.40:0/400020263 learned_addr learned my addr 172.21.15.40:0/400020263 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:19:26.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.970+0000 7f952f7fe640 10 monclient(hunting): get_auth_request con 0x7f95300a1d30 auth_method 0 2024-04-22T08:19:26.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.970+0000 7f952f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:26.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.970+0000 7f952f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:26.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.970+0000 7f952f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:26.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.970+0000 7f952f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:26.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.970+0000 7f952f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:26.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.970+0000 7f952f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:26.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.970+0000 7f9534aee640 1 --2- 172.21.15.40:0/400020263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9530135d90 0x7f95301353f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:26.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.970+0000 7f9534aee640 10 monclient(hunting): get_auth_request con 0x7f9530135d90 auth_method 0 2024-04-22T08:19:26.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.970+0000 7f9534aee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:26.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.970+0000 7f9534aee640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:26.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.970+0000 7f9534aee640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:26.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.970+0000 7f9534aee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:26.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.970+0000 7f9534aee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:26.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.970+0000 7f9534aee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:26.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.970+0000 7f952f7fe640 10 monclient(hunting): handle_auth_done global_id 54683 payload 824 2024-04-22T08:19:26.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.970+0000 7f952f7fe640 1 -- 172.21.15.40:0/400020263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9530135d90 msgr2=0x7f95301353f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:26.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.970+0000 7f952f7fe640 1 --2- 172.21.15.40:0/400020263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9530135d90 0x7f95301353f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:26.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.970+0000 7f952f7fe640 1 -- 172.21.15.40:0/400020263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f953009ee10 msgr2=0x7f95301ca770 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:19:26.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.970+0000 7f952f7fe640 1 --2- 172.21.15.40:0/400020263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f953009ee10 0x7f95301ca770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:26.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.970+0000 7f952f7fe640 10 monclient: _finish_hunting 0 2024-04-22T08:19:26.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.970+0000 7f952f7fe640 1 monclient: found mon.noname-c 2024-04-22T08:19:26.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.970+0000 7f952f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:19:26.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.970+0000 7f952f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:19:26.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.970+0000 7f952f7fe640 1 -- 172.21.15.40:0/400020263 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9528009b30 con 0x7f95300a1d30 2024-04-22T08:19:26.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.972+0000 7f952f7fe640 1 -- 172.21.15.40:0/400020263 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f95300a1d30 msgr2=0x7f9530134eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:26.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.972+0000 7f952f7fe640 1 --2- 172.21.15.40:0/400020263 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f95300a1d30 0x7f9530134eb0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f952000c7f0 tx=0x7f952000cbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:26.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.972+0000 7f952cff9640 1 -- 172.21.15.40:0/400020263 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9520007c40 con 0x7f95300a1d30 2024-04-22T08:19:26.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.972+0000 7f952cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:26.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.972+0000 7f952cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:19:26.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.972+0000 7f952cff9640 10 monclient: dump: 2024-04-22T08:19:26.983 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:26.983 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:26.983 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.983 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.983 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:26.983 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:26.983 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:26.983 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:26.983 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:26.984 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:26.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.972+0000 7f952cff9640 10 monclient: _finish_auth 0 2024-04-22T08:19:26.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.972+0000 7f952cff9640 10 monclient: _check_auth_tickets 2024-04-22T08:19:26.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.972+0000 7f952cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:19:26.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.972+0000 7f9536551640 5 monclient: authenticate success, global_id 54683 2024-04-22T08:19:26.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.972+0000 7f952cff9640 1 -- 172.21.15.40:0/400020263 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9520007da0 con 0x7f95300a1d30 2024-04-22T08:19:26.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.972+0000 7f9536551640 10 monclient: _renew_subs 2024-04-22T08:19:26.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.972+0000 7f9536551640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:19:26.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.972+0000 7f9536551640 1 -- 172.21.15.40:0/400020263 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9530120e60 con 0x7f95300a1d30 2024-04-22T08:19:26.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.973+0000 7f952cff9640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:19:26.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.973+0000 7f952cff9640 1 -- 172.21.15.40:0/400020263 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9520011610 con 0x7f95300a1d30 2024-04-22T08:19:26.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.973+0000 7f952cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:26.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.973+0000 7f952cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:19:26.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.973+0000 7f952cff9640 10 monclient: dump: 2024-04-22T08:19:26.984 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:26.984 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:26.985 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.985 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:26.985 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:26.985 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:26.985 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:26.985 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:26.985 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:26.985 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:26.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.973+0000 7f952e7fc640 4 set_mon_vals no callback set 2024-04-22T08:19:26.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.973+0000 7f9536551640 10 monclient: _renew_subs 2024-04-22T08:19:26.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.973+0000 7f9536551640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:19:26.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.973+0000 7f9536551640 1 -- 172.21.15.40:0/400020263 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9530120550 con 0x7f95300a1d30 2024-04-22T08:19:26.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.973+0000 7f952cff9640 1 -- 172.21.15.40:0/400020263 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f9520011770 con 0x7f95300a1d30 2024-04-22T08:19:26.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.973+0000 7f9536551640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:19:26.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.973+0000 7f9536551640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:19:26.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.973+0000 7f9536551640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:19:26.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.973+0000 7f9536551640 1 -- 172.21.15.40:0/400020263 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f953004e9b0 con 0x7f95300a1d30 2024-04-22T08:19:26.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.974+0000 7f952cff9640 1 --2- 172.21.15.40:0/400020263 >> v2:172.21.15.189:6800/30168 conn(0x7f9510037780 0x7f9510039c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:26.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.974+0000 7f952cff9640 1 -- 172.21.15.40:0/400020263 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f952004a620 con 0x7f95300a1d30 2024-04-22T08:19:26.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.974+0000 7f952ffff640 1 --2- 172.21.15.40:0/400020263 >> v2:172.21.15.189:6800/30168 conn(0x7f9510037780 0x7f9510039c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:26.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.974+0000 7f952ffff640 10 monclient: get_auth_request con 0x7f9510037780 auth_method 0 2024-04-22T08:19:26.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.976+0000 7f952ffff640 1 --2- 172.21.15.40:0/400020263 >> v2:172.21.15.189:6800/30168 conn(0x7f9510037780 0x7f9510039c40 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f9518006fd0 tx=0x7f9518006d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:26.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.978+0000 7f952cff9640 1 -- 172.21.15.40:0/400020263 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f9520018360 con 0x7f95300a1d30 2024-04-22T08:19:26.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.978+0000 7f952cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:19:26.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:26.978+0000 7f952cff9640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:19:27.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:27.236+0000 7f9536551640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:19:27.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:27.236+0000 7f9536551640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:19:27.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:27.236+0000 7f9536551640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:19:27.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:27.236+0000 7f9536551640 1 -- 172.21.15.40:0/400020263 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f953006a1e0 con 0x7f95300a1d30 2024-04-22T08:19:27.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:27.237+0000 7f952cff9640 1 -- 172.21.15.40:0/400020263 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f9520021300 con 0x7f95300a1d30 2024-04-22T08:19:27.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:27.237+0000 7f952cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:19:27.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:27.237+0000 7f952cff9640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:19:27.237 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":66,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:19:27.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:27.239+0000 7f9536551640 1 -- 172.21.15.40:0/400020263 >> v2:172.21.15.189:6800/30168 conn(0x7f9510037780 msgr2=0x7f9510039c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:27.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:27.239+0000 7f9536551640 1 --2- 172.21.15.40:0/400020263 >> v2:172.21.15.189:6800/30168 conn(0x7f9510037780 0x7f9510039c40 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f9518006fd0 tx=0x7f9518006d80 comp rx=0 tx=0).stop 2024-04-22T08:19:27.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:27.239+0000 7f9536551640 10 monclient: shutdown 2024-04-22T08:19:27.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:27.239+0000 7f9536551640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:19:27.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:27.239+0000 7f9536551640 1 -- 172.21.15.40:0/400020263 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f95300a1d30 msgr2=0x7f9530134eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:27.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:27.239+0000 7f9536551640 1 --2- 172.21.15.40:0/400020263 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f95300a1d30 0x7f9530134eb0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f952000c7f0 tx=0x7f952000cbf0 comp rx=0 tx=0).stop 2024-04-22T08:19:27.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:27.239+0000 7f9536551640 1 -- 172.21.15.40:0/400020263 shutdown_connections 2024-04-22T08:19:27.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:27.239+0000 7f9536551640 1 --2- 172.21.15.40:0/400020263 >> v2:172.21.15.189:6800/30168 conn(0x7f9510037780 0x7f9510039c40 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:27.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:27.239+0000 7f9536551640 1 --2- 172.21.15.40:0/400020263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9530135d90 0x7f95301353f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:27.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:27.239+0000 7f9536551640 1 --2- 172.21.15.40:0/400020263 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f95300a1d30 0x7f9530134eb0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:27.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:27.239+0000 7f9536551640 1 --2- 172.21.15.40:0/400020263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f953009ee10 0x7f95301ca770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:27.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:27.239+0000 7f9536551640 1 -- 172.21.15.40:0/400020263 >> 172.21.15.40:0/400020263 conn(0x7f95300a4ef0 msgr2=0x7f95301bddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:19:27.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:27.239+0000 7f9536551640 1 -- 172.21.15.40:0/400020263 shutdown_connections 2024-04-22T08:19:27.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:27.239+0000 7f9536551640 1 -- 172.21.15.40:0/400020263 wait complete. 2024-04-22T08:19:30.250 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:19:30.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.360+0000 7ff7687a7640 10 monclient: get_monmap_and_config 2024-04-22T08:19:30.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.360+0000 7ff7687a7640 10 monclient: build_initial_monmap 2024-04-22T08:19:30.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.360+0000 7ff7687a7640 10 monclient: monmap: 2024-04-22T08:19:30.366 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:19:30.366 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:30.366 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:19:30.361176+0000 2024-04-22T08:19:30.366 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:19:30.361176+0000 2024-04-22T08:19:30.366 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:19:30.366 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:30.366 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:19:30.366 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:19:30.366 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:19:30.366 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:19:30.366 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:19:30.366 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:30.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.360+0000 7ff7687a7640 1 Processor -- start 2024-04-22T08:19:30.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.361+0000 7ff7687a7640 1 -- start start 2024-04-22T08:19:30.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.361+0000 7ff7687a7640 10 monclient: init 2024-04-22T08:19:30.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.361+0000 7ff7687a7640 10 monclient: _reopen_session rank -1 2024-04-22T08:19:30.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.361+0000 7ff7687a7640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-04-22T08:19:30.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.361+0000 7ff7687a7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff760135e30 addr v1:172.21.15.189:6789/0 2024-04-22T08:19:30.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.361+0000 7ff7687a7640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff760071c90 0x7ff76013aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:30.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.361+0000 7ff7687a7640 10 monclient(hunting): picked mon.noname-c con 0x7ff760071c90 addr v2:172.21.15.189:3300/0 2024-04-22T08:19:30.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.361+0000 7ff7687a7640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff76013b410 0x7ff76013d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:30.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.361+0000 7ff7687a7640 10 monclient(hunting): picked mon.noname-b con 0x7ff76013b410 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:19:30.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.361+0000 7ff7687a7640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff76009e340 con 0x7ff760135e30 2024-04-22T08:19:30.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.361+0000 7ff7687a7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:30.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.361+0000 7ff7687a7640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff76009de80 con 0x7ff760071c90 2024-04-22T08:19:30.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.361+0000 7ff7687a7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:30.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.361+0000 7ff7687a7640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff76009f130 con 0x7ff76013b410 2024-04-22T08:19:30.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.361+0000 7ff7687a7640 10 monclient(hunting): _renew_subs 2024-04-22T08:19:30.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.361+0000 7ff7687a7640 10 monclient(hunting): authenticate will time out at 1244.355554s 2024-04-22T08:19:30.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.361+0000 7ff766d44640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff76013b410 0x7ff76013d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:30.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.361+0000 7ff766d44640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff76013b410 0x7ff76013d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:50370/0 (socket says 172.21.15.40:50370) 2024-04-22T08:19:30.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.361+0000 7ff766d44640 1 -- 172.21.15.40:0/3245173055 learned_addr learned my addr 172.21.15.40:0/3245173055 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:19:30.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.361+0000 7ff766d44640 10 monclient(hunting): get_auth_request con 0x7ff76013b410 auth_method 0 2024-04-22T08:19:30.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.361+0000 7ff766d44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:30.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.361+0000 7ff766d44640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:30.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.361+0000 7ff766d44640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:30.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.362+0000 7ff766d44640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:30.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.362+0000 7ff766d44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:30.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.362+0000 7ff766d44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:30.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.362+0000 7ff765d42640 1 --2- 172.21.15.40:0/3245173055 >> v2:172.21.15.189:3300/0 conn(0x7ff760071c90 0x7ff76013aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:30.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.362+0000 7ff765d42640 10 monclient(hunting): get_auth_request con 0x7ff760071c90 auth_method 0 2024-04-22T08:19:30.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.362+0000 7ff765d42640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:30.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.362+0000 7ff765d42640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:30.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.362+0000 7ff765d42640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:30.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.362+0000 7ff766d44640 10 monclient(hunting): handle_auth_done global_id 54686 payload 274 2024-04-22T08:19:30.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.362+0000 7ff766d44640 1 -- 172.21.15.40:0/3245173055 >> v2:172.21.15.189:3300/0 conn(0x7ff760071c90 msgr2=0x7ff76013aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:19:30.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.362+0000 7ff766d44640 1 --2- 172.21.15.40:0/3245173055 >> v2:172.21.15.189:3300/0 conn(0x7ff760071c90 0x7ff76013aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:30.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.362+0000 7ff766d44640 1 -- 172.21.15.40:0/3245173055 >> v1:172.21.15.189:6789/0 conn(0x7ff760135e30 legacy=0x7ff760136210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:19:30.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.362+0000 7ff766d44640 10 monclient: _finish_hunting 0 2024-04-22T08:19:30.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.362+0000 7ff766d44640 1 monclient: found mon.noname-b 2024-04-22T08:19:30.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.362+0000 7ff766d44640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:19:30.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.362+0000 7ff766d44640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:19:30.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.362+0000 7ff766d44640 1 -- 172.21.15.40:0/3245173055 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff76009f410 con 0x7ff76013b410 2024-04-22T08:19:30.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.365+0000 7ff766d44640 1 -- 172.21.15.40:0/3245173055 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff76013b410 msgr2=0x7ff76013d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:30.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.365+0000 7ff766d44640 1 --2- 172.21.15.40:0/3245173055 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff76013b410 0x7ff76013d800 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff758009e80 tx=0x7ff75802bd90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3f9747c4cba795a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:30.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.365+0000 7ff765541640 1 -- 172.21.15.40:0/3245173055 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff75802cb40 con 0x7ff76013b410 2024-04-22T08:19:30.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.365+0000 7ff765541640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:30.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.365+0000 7ff765541640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:19:30.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.365+0000 7ff765541640 10 monclient: dump: 2024-04-22T08:19:30.370 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:30.370 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:30.370 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:30.370 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:30.370 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:30.370 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:30.370 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:30.370 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:30.370 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:30.371 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:30.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.365+0000 7ff765541640 10 monclient: _finish_auth 0 2024-04-22T08:19:30.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.365+0000 7ff765541640 10 monclient: _check_auth_tickets 2024-04-22T08:19:30.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.365+0000 7ff765541640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:19:30.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.365+0000 7ff765541640 1 -- 172.21.15.40:0/3245173055 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff75802cca0 con 0x7ff76013b410 2024-04-22T08:19:30.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.365+0000 7ff765541640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:19:30.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.365+0000 7ff765541640 1 -- 172.21.15.40:0/3245173055 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff758034610 con 0x7ff76013b410 2024-04-22T08:19:30.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.365+0000 7ff7687a7640 5 monclient: authenticate success, global_id 54686 2024-04-22T08:19:30.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.365+0000 7ff7687a7640 10 monclient: get_monmap_and_config success 2024-04-22T08:19:30.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.365+0000 7ff7687a7640 4 set_mon_vals no callback set 2024-04-22T08:19:30.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.365+0000 7ff765541640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:30.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.365+0000 7ff765541640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:19:30.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.365+0000 7ff765541640 10 monclient: dump: 2024-04-22T08:19:30.372 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:30.372 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:30.372 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:30.372 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:30.372 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:30.372 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:30.372 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:30.372 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:30.372 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:30.372 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:30.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.365+0000 7ff7687a7640 10 monclient: shutdown 2024-04-22T08:19:30.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.365+0000 7ff7687a7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:19:30.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.365+0000 7ff7687a7640 1 -- 172.21.15.40:0/3245173055 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff76013b410 msgr2=0x7ff76013d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:30.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.365+0000 7ff7687a7640 1 --2- 172.21.15.40:0/3245173055 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff76013b410 0x7ff76013d800 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff758009e80 tx=0x7ff75802bd90 comp rx=0 tx=0).stop 2024-04-22T08:19:30.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.365+0000 7ff7687a7640 1 -- 172.21.15.40:0/3245173055 shutdown_connections 2024-04-22T08:19:30.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.365+0000 7ff7687a7640 1 --2- 172.21.15.40:0/3245173055 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff76013b410 0x7ff76013d800 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:30.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.365+0000 7ff7687a7640 1 --2- 172.21.15.40:0/3245173055 >> v2:172.21.15.189:3300/0 conn(0x7ff760071c90 0x7ff76013aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:30.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.365+0000 7ff7687a7640 1 -- 172.21.15.40:0/3245173055 >> 172.21.15.40:0/3245173055 conn(0x7ff7600a65c0 msgr2=0x7ff7600a2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:19:30.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.365+0000 7ff7687a7640 1 -- 172.21.15.40:0/3245173055 shutdown_connections 2024-04-22T08:19:30.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.366+0000 7ff7687a7640 1 -- 172.21.15.40:0/3245173055 wait complete. 2024-04-22T08:19:30.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.366+0000 7ff7687a7640 10 monclient: build_initial_monmap 2024-04-22T08:19:30.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.366+0000 7ff7687a7640 10 monclient: monmap: 2024-04-22T08:19:30.373 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:19:30.373 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:19:30.373 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:19:30.373 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:19:30.373 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:19:30.373 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:30.373 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:19:30.373 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:19:30.374 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:19:30.374 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:30.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.366+0000 7ff7687a7640 1 Processor -- start 2024-04-22T08:19:30.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.366+0000 7ff7687a7640 1 -- start start 2024-04-22T08:19:30.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.366+0000 7ff7687a7640 10 monclient: init 2024-04-22T08:19:30.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.367+0000 7ff7687a7640 10 monclient: _reopen_session rank -1 2024-04-22T08:19:30.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.367+0000 7ff7687a7640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-22T08:19:30.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.367+0000 7ff7687a7640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff760071c90 0x7ff7601ca890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:30.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.367+0000 7ff7687a7640 10 monclient(hunting): picked mon.noname-c con 0x7ff760071c90 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:19:30.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.367+0000 7ff7687a7640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff760135e30 0x7ff760134da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:30.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.367+0000 7ff7687a7640 10 monclient(hunting): picked mon.noname-b con 0x7ff760135e30 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:19:30.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.367+0000 7ff7687a7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff76013b410 0x7ff76006ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:30.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.367+0000 7ff7687a7640 10 monclient(hunting): picked mon.noname-a con 0x7ff76013b410 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:19:30.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.367+0000 7ff7687a7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:30.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.367+0000 7ff7687a7640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff760121190 con 0x7ff76013b410 2024-04-22T08:19:30.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.367+0000 7ff7687a7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:30.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.367+0000 7ff7687a7640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7601cadd0 con 0x7ff760135e30 2024-04-22T08:19:30.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.367+0000 7ff7687a7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:30.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.367+0000 7ff7687a7640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff760071020 con 0x7ff760071c90 2024-04-22T08:19:30.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.367+0000 7ff7687a7640 10 monclient(hunting): _renew_subs 2024-04-22T08:19:30.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.367+0000 7ff7687a7640 10 monclient(hunting): authenticate will time out at 1244.361231s 2024-04-22T08:19:30.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.367+0000 7ff766d44640 1 -- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff76013b410 msgr2=0x7ff76006ea70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:19:30.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.367+0000 7ff766d44640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff76013b410 0x7ff76006ea70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:19:30.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.367+0000 7ff766543640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff760071c90 0x7ff7601ca890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:30.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.367+0000 7ff766543640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff760071c90 0x7ff7601ca890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:50384/0 (socket says 172.21.15.40:50384) 2024-04-22T08:19:30.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.367+0000 7ff766543640 1 -- 172.21.15.40:0/3676986205 learned_addr learned my addr 172.21.15.40:0/3676986205 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:19:30.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.367+0000 7ff766543640 10 monclient(hunting): get_auth_request con 0x7ff760071c90 auth_method 0 2024-04-22T08:19:30.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.367+0000 7ff766543640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:30.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.367+0000 7ff766543640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:30.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.367+0000 7ff766543640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:30.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.367+0000 7ff766543640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:30.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.367+0000 7ff766543640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:30.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.367+0000 7ff766543640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:30.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.367+0000 7ff765d42640 1 --2- 172.21.15.40:0/3676986205 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff760135e30 0x7ff760134da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:30.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.367+0000 7ff765d42640 10 monclient(hunting): get_auth_request con 0x7ff760135e30 auth_method 0 2024-04-22T08:19:30.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.367+0000 7ff765d42640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:30.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.367+0000 7ff765d42640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:30.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.367+0000 7ff765d42640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:30.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.367+0000 7ff766543640 10 monclient(hunting): handle_auth_done global_id 54689 payload 824 2024-04-22T08:19:30.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.367+0000 7ff766543640 1 -- 172.21.15.40:0/3676986205 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff760135e30 msgr2=0x7ff760134da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:30.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.367+0000 7ff766543640 1 --2- 172.21.15.40:0/3676986205 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff760135e30 0x7ff760134da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:30.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.367+0000 7ff766543640 1 -- 172.21.15.40:0/3676986205 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff76013b410 msgr2=0x7ff76006ea70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:19:30.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.367+0000 7ff766543640 1 --2- 172.21.15.40:0/3676986205 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff76013b410 0x7ff76006ea70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:30.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.367+0000 7ff766543640 10 monclient: _finish_hunting 0 2024-04-22T08:19:30.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.367+0000 7ff766543640 1 monclient: found mon.noname-c 2024-04-22T08:19:30.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.367+0000 7ff766543640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:19:30.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.367+0000 7ff766543640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:19:30.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.367+0000 7ff766543640 1 -- 172.21.15.40:0/3676986205 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff758009b30 con 0x7ff760071c90 2024-04-22T08:19:30.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.367+0000 7ff765d42640 1 --2- 172.21.15.40:0/3676986205 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff760135e30 0x7ff760134da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T08:19:30.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.370+0000 7ff766543640 1 -- 172.21.15.40:0/3676986205 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff760071c90 msgr2=0x7ff7601ca890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:30.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.370+0000 7ff766543640 1 --2- 172.21.15.40:0/3676986205 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff760071c90 0x7ff7601ca890 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff75c00c460 tx=0x7ff75c00c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:30.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.370+0000 7ff756ffd640 1 -- 172.21.15.40:0/3676986205 DispatchQueue::entry inject delay of 0.002000 2024-04-22T08:19:30.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.372+0000 7ff756ffd640 1 -- 172.21.15.40:0/3676986205 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff75c012a40 con 0x7ff760071c90 2024-04-22T08:19:30.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.372+0000 7ff756ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:30.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.372+0000 7ff756ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:19:30.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.372+0000 7ff756ffd640 10 monclient: dump: 2024-04-22T08:19:30.383 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:30.383 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:30.383 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:30.383 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:30.383 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:30.383 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:30.383 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:30.383 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:30.383 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:30.383 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:30.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.372+0000 7ff756ffd640 10 monclient: _finish_auth 0 2024-04-22T08:19:30.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.372+0000 7ff756ffd640 10 monclient: _check_auth_tickets 2024-04-22T08:19:30.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.372+0000 7ff756ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:19:30.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.372+0000 7ff756ffd640 1 -- 172.21.15.40:0/3676986205 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff75c012be0 con 0x7ff760071c90 2024-04-22T08:19:30.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.372+0000 7ff7687a7640 5 monclient: authenticate success, global_id 54689 2024-04-22T08:19:30.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.372+0000 7ff7687a7640 10 monclient: _renew_subs 2024-04-22T08:19:30.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.372+0000 7ff7687a7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:19:30.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.372+0000 7ff7687a7640 1 -- 172.21.15.40:0/3676986205 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff760072140 con 0x7ff760071c90 2024-04-22T08:19:30.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.372+0000 7ff756ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:19:30.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.372+0000 7ff756ffd640 1 -- 172.21.15.40:0/3676986205 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff75c011380 con 0x7ff760071c90 2024-04-22T08:19:30.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.372+0000 7ff756ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:30.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.372+0000 7ff756ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:19:30.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.372+0000 7ff756ffd640 10 monclient: dump: 2024-04-22T08:19:30.384 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:30.384 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:30.385 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:30.385 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:30.385 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:30.385 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:30.385 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:30.385 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:30.385 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:30.385 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:30.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.372+0000 7ff7687a7640 10 monclient: _renew_subs 2024-04-22T08:19:30.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.372+0000 7ff757fff640 4 set_mon_vals no callback set 2024-04-22T08:19:30.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.372+0000 7ff7687a7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:19:30.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.372+0000 7ff7687a7640 1 -- 172.21.15.40:0/3676986205 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff760120eb0 con 0x7ff760071c90 2024-04-22T08:19:30.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.373+0000 7ff756ffd640 1 -- 172.21.15.40:0/3676986205 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7ff75c011520 con 0x7ff760071c90 2024-04-22T08:19:30.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.373+0000 7ff7687a7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:19:30.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.373+0000 7ff7687a7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:19:30.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.373+0000 7ff7687a7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:19:30.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.373+0000 7ff7687a7640 1 -- 172.21.15.40:0/3676986205 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff760135b90 con 0x7ff760071c90 2024-04-22T08:19:30.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.373+0000 7ff756ffd640 1 --2- 172.21.15.40:0/3676986205 >> v2:172.21.15.189:6800/30168 conn(0x7ff7440377d0 0x7ff744039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:30.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.373+0000 7ff756ffd640 1 -- 172.21.15.40:0/3676986205 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 3898+0+0 (secure 0 0 0) 0x7ff75c04a030 con 0x7ff760071c90 2024-04-22T08:19:30.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.373+0000 7ff765d42640 1 --2- 172.21.15.40:0/3676986205 >> v2:172.21.15.189:6800/30168 conn(0x7ff7440377d0 0x7ff744039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:30.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.373+0000 7ff765d42640 10 monclient: get_auth_request con 0x7ff7440377d0 auth_method 0 2024-04-22T08:19:30.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.376+0000 7ff765d42640 1 --2- 172.21.15.40:0/3676986205 >> v2:172.21.15.189:6800/30168 conn(0x7ff7440377d0 0x7ff744039c90 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff750006fd0 tx=0x7ff750008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:30.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.378+0000 7ff756ffd640 1 -- 172.21.15.40:0/3676986205 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7ff75c07d7f0 con 0x7ff760071c90 2024-04-22T08:19:30.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.378+0000 7ff756ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:19:30.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.378+0000 7ff756ffd640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:19:30.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.630+0000 7ff7687a7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:19:30.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.630+0000 7ff7687a7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:19:30.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.630+0000 7ff7687a7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:19:30.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.630+0000 7ff7687a7640 1 -- 172.21.15.40:0/3676986205 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff7601212d0 con 0x7ff760071c90 2024-04-22T08:19:30.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.631+0000 7ff756ffd640 1 -- 172.21.15.40:0/3676986205 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7ff7601212d0 con 0x7ff760071c90 2024-04-22T08:19:30.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.631+0000 7ff756ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:19:30.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.631+0000 7ff756ffd640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:19:30.631 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":66,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:19:30.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.632+0000 7ff7687a7640 1 -- 172.21.15.40:0/3676986205 >> v2:172.21.15.189:6800/30168 conn(0x7ff7440377d0 msgr2=0x7ff744039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:30.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.632+0000 7ff7687a7640 1 --2- 172.21.15.40:0/3676986205 >> v2:172.21.15.189:6800/30168 conn(0x7ff7440377d0 0x7ff744039c90 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff750006fd0 tx=0x7ff750008040 comp rx=0 tx=0).stop 2024-04-22T08:19:30.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.632+0000 7ff7687a7640 10 monclient: shutdown 2024-04-22T08:19:30.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.632+0000 7ff7687a7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:19:30.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.632+0000 7ff7687a7640 1 -- 172.21.15.40:0/3676986205 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff760071c90 msgr2=0x7ff7601ca890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:30.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.632+0000 7ff7687a7640 1 --2- 172.21.15.40:0/3676986205 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff760071c90 0x7ff7601ca890 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff75c00c460 tx=0x7ff75c00c920 comp rx=0 tx=0).stop 2024-04-22T08:19:30.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.633+0000 7ff7687a7640 1 -- 172.21.15.40:0/3676986205 shutdown_connections 2024-04-22T08:19:30.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.633+0000 7ff7687a7640 1 --2- 172.21.15.40:0/3676986205 >> v2:172.21.15.189:6800/30168 conn(0x7ff7440377d0 0x7ff744039c90 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:30.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.633+0000 7ff7687a7640 1 --2- 172.21.15.40:0/3676986205 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff760135e30 0x7ff760134da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:30.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.633+0000 7ff7687a7640 1 --2- 172.21.15.40:0/3676986205 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff76013b410 0x7ff76006ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:30.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.633+0000 7ff7687a7640 1 --2- 172.21.15.40:0/3676986205 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff760071c90 0x7ff7601ca890 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:30.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.633+0000 7ff7687a7640 1 -- 172.21.15.40:0/3676986205 >> 172.21.15.40:0/3676986205 conn(0x7ff7600a65c0 msgr2=0x7ff7600a3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:19:30.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.633+0000 7ff7687a7640 1 -- 172.21.15.40:0/3676986205 shutdown_connections 2024-04-22T08:19:30.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:30.633+0000 7ff7687a7640 1 -- 172.21.15.40:0/3676986205 wait complete. 2024-04-22T08:19:33.643 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:19:33.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.754+0000 7f3463866640 10 monclient: get_monmap_and_config 2024-04-22T08:19:33.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.754+0000 7f3463866640 10 monclient: build_initial_monmap 2024-04-22T08:19:33.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.754+0000 7f3463866640 10 monclient: monmap: 2024-04-22T08:19:33.754 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:19:33.754 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:33.754 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:19:33.755555+0000 2024-04-22T08:19:33.754 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:19:33.755555+0000 2024-04-22T08:19:33.754 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:19:33.754 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:33.755 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:19:33.755 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:19:33.755 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:19:33.755 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:19:33.755 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:19:33.755 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:33.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.755+0000 7f3463866640 1 Processor -- start 2024-04-22T08:19:33.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.755+0000 7f3463866640 1 -- start start 2024-04-22T08:19:33.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.755+0000 7f3463866640 10 monclient: init 2024-04-22T08:19:33.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.755+0000 7f3463866640 10 monclient: _reopen_session rank -1 2024-04-22T08:19:33.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.756+0000 7f3463866640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-04-22T08:19:33.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.756+0000 7f3463866640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f345c09f0b0 addr v1:172.21.15.189:6789/0 2024-04-22T08:19:33.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.756+0000 7f3463866640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f345c09e2b0 addr v1:172.21.15.40:6789/0 2024-04-22T08:19:33.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.756+0000 7f3463866640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f345c13a560 0x7f345c13c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:33.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.756+0000 7f3463866640 10 monclient(hunting): picked mon.noname-b con 0x7f345c13a560 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:19:33.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.756+0000 7f3463866640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f345c09ef10 con 0x7f345c09e2b0 2024-04-22T08:19:33.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.756+0000 7f3463866640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f345c071670 con 0x7f345c09f0b0 2024-04-22T08:19:33.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.756+0000 7f3463866640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:33.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.756+0000 7f3463866640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f345c09fba0 con 0x7f345c13a560 2024-04-22T08:19:33.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.756+0000 7f3463866640 10 monclient(hunting): _renew_subs 2024-04-22T08:19:33.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.756+0000 7f3463866640 10 monclient(hunting): authenticate will time out at 1247.749857s 2024-04-22T08:19:33.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.756+0000 7f3460e01640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f345c09e2b0 legacy=0x7f345c138e40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-04-22T08:19:33.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.756+0000 7f3461e03640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f345c13a560 0x7f345c13c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:33.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.756+0000 7f3461e03640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f345c13a560 0x7f345c13c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:50400/0 (socket says 172.21.15.40:50400) 2024-04-22T08:19:33.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.756+0000 7f3461e03640 1 -- 172.21.15.40:0/3474754932 learned_addr learned my addr 172.21.15.40:0/3474754932 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:19:33.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.756+0000 7f3461e03640 10 monclient(hunting): get_auth_request con 0x7f345c13a560 auth_method 0 2024-04-22T08:19:33.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.756+0000 7f3461e03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:33.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.756+0000 7f3461e03640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:33.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.756+0000 7f3461e03640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:33.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.756+0000 7f3453fff640 1 -- 172.21.15.40:0/3474754932 DispatchQueue::entry inject delay of 0.002000 2024-04-22T08:19:33.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.756+0000 7f3461e03640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:33.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.756+0000 7f3461e03640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:33.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.756+0000 7f3461e03640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:33.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.756+0000 7f3461e03640 10 monclient(hunting): handle_auth_done global_id 54692 payload 274 2024-04-22T08:19:33.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.756+0000 7f3461e03640 1 -- 172.21.15.40:0/3474754932 >> v1:172.21.15.189:6789/0 conn(0x7f345c09f0b0 legacy=0x7f345c09f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:19:33.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.756+0000 7f3461e03640 1 -- 172.21.15.40:0/3474754932 >> v1:172.21.15.40:6789/0 conn(0x7f345c09e2b0 legacy=0x7f345c138e40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:19:33.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.756+0000 7f3461e03640 10 monclient: _finish_hunting 0 2024-04-22T08:19:33.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.756+0000 7f3461e03640 1 monclient: found mon.noname-b 2024-04-22T08:19:33.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.756+0000 7f3461e03640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:19:33.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.756+0000 7f3461e03640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:19:33.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.756+0000 7f3461e03640 1 -- 172.21.15.40:0/3474754932 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f345c09d9e0 con 0x7f345c13a560 2024-04-22T08:19:33.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.759+0000 7f3461e03640 1 -- 172.21.15.40:0/3474754932 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f345c13a560 msgr2=0x7f345c13c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:33.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.759+0000 7f3461e03640 1 --2- 172.21.15.40:0/3474754932 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f345c13a560 0x7f345c13c950 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3454009e80 tx=0x7f345402bd90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a4a377773d781d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:33.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.759+0000 7f3453fff640 1 -- 172.21.15.40:0/3474754932 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345402cb40 con 0x7f345c13a560 2024-04-22T08:19:33.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.759+0000 7f3453fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:33.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.759+0000 7f3453fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:19:33.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.759+0000 7f3453fff640 10 monclient: dump: 2024-04-22T08:19:33.763 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:33.764 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:33.764 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:33.764 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:33.764 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:33.764 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:33.764 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:33.764 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:33.764 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:33.764 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:33.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.759+0000 7f3453fff640 10 monclient: _finish_auth 0 2024-04-22T08:19:33.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.759+0000 7f3453fff640 10 monclient: _check_auth_tickets 2024-04-22T08:19:33.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.759+0000 7f3453fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:19:33.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.759+0000 7f3453fff640 1 -- 172.21.15.40:0/3474754932 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f345402cca0 con 0x7f345c13a560 2024-04-22T08:19:33.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.759+0000 7f3463866640 5 monclient: authenticate success, global_id 54692 2024-04-22T08:19:33.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.759+0000 7f3463866640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:19:33.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.759+0000 7f3453fff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:19:33.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.759+0000 7f3463866640 10 monclient: get_monmap_and_config success 2024-04-22T08:19:33.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.759+0000 7f3463866640 4 set_mon_vals no callback set 2024-04-22T08:19:33.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.759+0000 7f3453fff640 1 -- 172.21.15.40:0/3474754932 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3454034610 con 0x7f345c13a560 2024-04-22T08:19:33.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.759+0000 7f3453fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:33.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.759+0000 7f3453fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:19:33.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.759+0000 7f3453fff640 10 monclient: dump: 2024-04-22T08:19:33.765 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:33.765 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:33.765 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:33.765 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:33.765 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:33.765 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:33.765 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:33.766 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:33.766 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:33.766 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:33.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.760+0000 7f3463866640 10 monclient: shutdown 2024-04-22T08:19:33.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.760+0000 7f3463866640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:19:33.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.760+0000 7f3463866640 1 -- 172.21.15.40:0/3474754932 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f345c13a560 msgr2=0x7f345c13c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:33.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.760+0000 7f3463866640 1 --2- 172.21.15.40:0/3474754932 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f345c13a560 0x7f345c13c950 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3454009e80 tx=0x7f345402bd90 comp rx=0 tx=0).stop 2024-04-22T08:19:33.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.760+0000 7f3463866640 1 -- 172.21.15.40:0/3474754932 shutdown_connections 2024-04-22T08:19:33.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.760+0000 7f3463866640 1 --2- 172.21.15.40:0/3474754932 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f345c13a560 0x7f345c13c950 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:33.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.760+0000 7f3463866640 1 -- 172.21.15.40:0/3474754932 >> 172.21.15.40:0/3474754932 conn(0x7f345c0a0b50 msgr2=0x7f345c0a0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:19:33.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.760+0000 7f3463866640 1 -- 172.21.15.40:0/3474754932 shutdown_connections 2024-04-22T08:19:33.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.760+0000 7f3463866640 1 -- 172.21.15.40:0/3474754932 wait complete. 2024-04-22T08:19:33.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.761+0000 7f3463866640 10 monclient: build_initial_monmap 2024-04-22T08:19:33.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.761+0000 7f3463866640 10 monclient: monmap: 2024-04-22T08:19:33.767 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:19:33.767 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:19:33.767 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:19:33.767 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:19:33.767 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:19:33.767 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:33.767 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:19:33.767 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:19:33.767 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:19:33.767 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:33.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.761+0000 7f3463866640 1 Processor -- start 2024-04-22T08:19:33.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.761+0000 7f3463866640 1 -- start start 2024-04-22T08:19:33.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.761+0000 7f3463866640 10 monclient: init 2024-04-22T08:19:33.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.762+0000 7f3463866640 10 monclient: _reopen_session rank -1 2024-04-22T08:19:33.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.762+0000 7f3463866640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-22T08:19:33.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.762+0000 7f3463866640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f345c09e2b0 0x7f345c1c8db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:33.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.762+0000 7f3463866640 10 monclient(hunting): picked mon.noname-a con 0x7f345c09e2b0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:19:33.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.762+0000 7f3463866640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f345c09f0b0 0x7f345c120260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:33.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.762+0000 7f3463866640 10 monclient(hunting): picked mon.noname-c con 0x7f345c09f0b0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:19:33.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.762+0000 7f3463866640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f345c13a560 0x7f345c1207a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:33.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.762+0000 7f3463866640 10 monclient(hunting): picked mon.noname-b con 0x7f345c13a560 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:19:33.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.762+0000 7f3463866640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:33.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.762+0000 7f3463866640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f345c122810 con 0x7f345c09e2b0 2024-04-22T08:19:33.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.762+0000 7f3463866640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:33.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.762+0000 7f3463866640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f345c139e60 con 0x7f345c13a560 2024-04-22T08:19:33.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.762+0000 7f3463866640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:33.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.762+0000 7f3463866640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f345c071860 con 0x7f345c09f0b0 2024-04-22T08:19:33.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.762+0000 7f3463866640 10 monclient(hunting): _renew_subs 2024-04-22T08:19:33.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.762+0000 7f3461602640 1 -- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f345c09e2b0 msgr2=0x7f345c1c8db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:19:33.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.762+0000 7f3463866640 10 monclient(hunting): authenticate will time out at 1247.756086s 2024-04-22T08:19:33.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.762+0000 7f3461602640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f345c09e2b0 0x7f345c1c8db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:19:33.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.762+0000 7f3460e01640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f345c09f0b0 0x7f345c120260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:33.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.762+0000 7f3460e01640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f345c09f0b0 0x7f345c120260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:50402/0 (socket says 172.21.15.40:50402) 2024-04-22T08:19:33.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.762+0000 7f3460e01640 1 -- 172.21.15.40:0/878339807 learned_addr learned my addr 172.21.15.40:0/878339807 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:19:33.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.762+0000 7f3460e01640 10 monclient(hunting): get_auth_request con 0x7f345c09f0b0 auth_method 0 2024-04-22T08:19:33.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.762+0000 7f3460e01640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:33.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.762+0000 7f3460e01640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:33.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.762+0000 7f3460e01640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:33.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.762+0000 7f3460e01640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:33.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.762+0000 7f3460e01640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:33.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.762+0000 7f3460e01640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:33.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.762+0000 7f3461e03640 1 --2- 172.21.15.40:0/878339807 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f345c13a560 0x7f345c1207a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:33.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.762+0000 7f3461e03640 10 monclient(hunting): get_auth_request con 0x7f345c13a560 auth_method 0 2024-04-22T08:19:33.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.762+0000 7f3461e03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:33.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.762+0000 7f3461e03640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:33.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.762+0000 7f3461e03640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:33.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.762+0000 7f3460e01640 10 monclient(hunting): handle_auth_done global_id 54695 payload 824 2024-04-22T08:19:33.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.762+0000 7f3460e01640 1 -- 172.21.15.40:0/878339807 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f345c13a560 msgr2=0x7f345c1207a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:33.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.762+0000 7f3460e01640 1 --2- 172.21.15.40:0/878339807 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f345c13a560 0x7f345c1207a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:33.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.762+0000 7f3460e01640 1 -- 172.21.15.40:0/878339807 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f345c09e2b0 msgr2=0x7f345c1c8db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:19:33.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.762+0000 7f3460e01640 1 --2- 172.21.15.40:0/878339807 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f345c09e2b0 0x7f345c1c8db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:33.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.762+0000 7f3460e01640 10 monclient: _finish_hunting 0 2024-04-22T08:19:33.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.762+0000 7f3460e01640 1 monclient: found mon.noname-c 2024-04-22T08:19:33.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.762+0000 7f3460e01640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:19:33.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.762+0000 7f3460e01640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:19:33.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.762+0000 7f3460e01640 1 -- 172.21.15.40:0/878339807 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3454009b30 con 0x7f345c09f0b0 2024-04-22T08:19:33.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.765+0000 7f3460e01640 1 -- 172.21.15.40:0/878339807 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f345c09f0b0 msgr2=0x7f345c120260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:33.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.765+0000 7f3460e01640 1 --2- 172.21.15.40:0/878339807 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f345c09f0b0 0x7f345c120260 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f344c00b580 tx=0x7f344c00ba40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:33.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.765+0000 7f3460e01640 1 --2- 172.21.15.40:0/878339807 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f345c09f0b0 0x7f345c120260 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f344c00b580 tx=0x7f344c00ba40 comp rx=0 tx=0).queue_received will delay after 1.71377e+09 on 0x7f344c005250 config(0 keys) v1 2024-04-22T08:19:33.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.765+0000 7f3451ffb640 1 -- 172.21.15.40:0/878339807 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f344c0050f0 con 0x7f345c09f0b0 2024-04-22T08:19:33.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.765+0000 7f3451ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:33.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.765+0000 7f3451ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:19:33.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.765+0000 7f3451ffb640 10 monclient: dump: 2024-04-22T08:19:33.773 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:33.773 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:33.773 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:33.773 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:33.773 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:33.773 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:33.773 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:33.774 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:33.774 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:33.774 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:33.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.765+0000 7f3451ffb640 10 monclient: _finish_auth 0 2024-04-22T08:19:33.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.765+0000 7f3451ffb640 10 monclient: _check_auth_tickets 2024-04-22T08:19:33.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.765+0000 7f3451ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:19:33.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.765+0000 7f3451ffb640 1 -- 172.21.15.40:0/878339807 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f344c005250 con 0x7f345c09f0b0 2024-04-22T08:19:33.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.765+0000 7f3451ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:19:33.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.765+0000 7f3463866640 5 monclient: authenticate success, global_id 54695 2024-04-22T08:19:33.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.765+0000 7f3463866640 10 monclient: _renew_subs 2024-04-22T08:19:33.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.765+0000 7f3463866640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:19:33.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.765+0000 7f3463866640 1 -- 172.21.15.40:0/878339807 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f345c1241e0 con 0x7f345c09f0b0 2024-04-22T08:19:33.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.765+0000 7f34537fe640 4 set_mon_vals no callback set 2024-04-22T08:19:33.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.765+0000 7f3463866640 10 monclient: _renew_subs 2024-04-22T08:19:33.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.765+0000 7f3463866640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:19:33.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.765+0000 7f3463866640 1 -- 172.21.15.40:0/878339807 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f345c138e70 con 0x7f345c09f0b0 2024-04-22T08:19:33.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.766+0000 7f3451ffb640 1 -- 172.21.15.40:0/878339807 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f344c005600 con 0x7f345c09f0b0 2024-04-22T08:19:33.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.766+0000 7f3451ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:33.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.766+0000 7f3451ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:19:33.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.766+0000 7f3451ffb640 10 monclient: dump: 2024-04-22T08:19:33.775 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:33.775 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:33.775 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:33.775 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:33.775 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:33.775 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:33.775 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:33.775 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:33.776 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:33.776 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:33.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.766+0000 7f3451ffb640 1 -- 172.21.15.40:0/878339807 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f344c010480 con 0x7f345c09f0b0 2024-04-22T08:19:33.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.766+0000 7f3451ffb640 1 --2- 172.21.15.40:0/878339807 >> v2:172.21.15.189:6800/30168 conn(0x7f34400377d0 0x7f3440039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:33.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.766+0000 7f3463866640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:19:33.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.766+0000 7f3463866640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:19:33.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.766+0000 7f3463866640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:19:33.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.766+0000 7f3463866640 1 -- 172.21.15.40:0/878339807 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f345c04e9b0 con 0x7f345c09f0b0 2024-04-22T08:19:33.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.766+0000 7f3461602640 1 --2- 172.21.15.40:0/878339807 >> v2:172.21.15.189:6800/30168 conn(0x7f34400377d0 0x7f3440039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:33.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.766+0000 7f3461602640 10 monclient: get_auth_request con 0x7f34400377d0 auth_method 0 2024-04-22T08:19:33.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.769+0000 7f3461602640 1 --2- 172.21.15.40:0/878339807 >> v2:172.21.15.189:6800/30168 conn(0x7f34400377d0 0x7f3440039c90 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3458003770 tx=0x7f3458007a10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:33.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:33.771+0000 7f3451ffb640 1 -- 172.21.15.40:0/878339807 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f344c04a3f0 con 0x7f345c09f0b0 2024-04-22T08:19:34.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:34.388+0000 7f3451ffb640 1 -- 172.21.15.40:0/878339807 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f344c00d950 con 0x7f345c09f0b0 2024-04-22T08:19:34.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:34.388+0000 7f3451ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:19:34.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:34.388+0000 7f3451ffb640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:19:34.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:34.638+0000 7f3463866640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:19:34.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:34.638+0000 7f3463866640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:19:34.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:34.638+0000 7f3463866640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:19:34.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:34.638+0000 7f3463866640 1 -- 172.21.15.40:0/878339807 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f345c123d20 con 0x7f345c09f0b0 2024-04-22T08:19:34.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:34.639+0000 7f3451ffb640 1 -- 172.21.15.40:0/878339807 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7f344c00d360 con 0x7f345c09f0b0 2024-04-22T08:19:34.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:34.639+0000 7f3451ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:19:34.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:34.639+0000 7f3451ffb640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:19:34.639 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":66,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:19:34.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:34.641+0000 7f3463866640 1 -- 172.21.15.40:0/878339807 >> v2:172.21.15.189:6800/30168 conn(0x7f34400377d0 msgr2=0x7f3440039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:34.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:34.641+0000 7f3463866640 1 --2- 172.21.15.40:0/878339807 >> v2:172.21.15.189:6800/30168 conn(0x7f34400377d0 0x7f3440039c90 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3458003770 tx=0x7f3458007a10 comp rx=0 tx=0).stop 2024-04-22T08:19:34.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:34.641+0000 7f3463866640 10 monclient: shutdown 2024-04-22T08:19:34.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:34.641+0000 7f3463866640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:19:34.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:34.641+0000 7f3463866640 1 -- 172.21.15.40:0/878339807 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f345c09f0b0 msgr2=0x7f345c120260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:34.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:34.641+0000 7f3463866640 1 --2- 172.21.15.40:0/878339807 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f345c09f0b0 0x7f345c120260 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f344c00b580 tx=0x7f344c00ba40 comp rx=0 tx=0).stop 2024-04-22T08:19:34.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:34.641+0000 7f3463866640 1 -- 172.21.15.40:0/878339807 shutdown_connections 2024-04-22T08:19:34.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:34.641+0000 7f3463866640 1 --2- 172.21.15.40:0/878339807 >> v2:172.21.15.189:6800/30168 conn(0x7f34400377d0 0x7f3440039c90 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:34.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:34.641+0000 7f3463866640 1 --2- 172.21.15.40:0/878339807 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f345c13a560 0x7f345c1207a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:34.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:34.641+0000 7f3463866640 1 --2- 172.21.15.40:0/878339807 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f345c09f0b0 0x7f345c120260 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:34.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:34.641+0000 7f3463866640 1 --2- 172.21.15.40:0/878339807 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f345c09e2b0 0x7f345c1c8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:34.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:34.641+0000 7f3463866640 1 -- 172.21.15.40:0/878339807 >> 172.21.15.40:0/878339807 conn(0x7f345c0a0b50 msgr2=0x7f345c06d5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:19:34.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:34.641+0000 7f3463866640 1 -- 172.21.15.40:0/878339807 shutdown_connections 2024-04-22T08:19:34.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:34.641+0000 7f3463866640 1 -- 172.21.15.40:0/878339807 wait complete. 2024-04-22T08:19:37.652 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:19:37.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:37.763+0000 7fad0d1ae640 10 monclient: get_monmap_and_config 2024-04-22T08:19:37.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:37.763+0000 7fad0d1ae640 10 monclient: build_initial_monmap 2024-04-22T08:19:37.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:37.763+0000 7fad0d1ae640 10 monclient: monmap: 2024-04-22T08:19:37.766 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:19:37.766 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:37.766 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:19:37.763803+0000 2024-04-22T08:19:37.766 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:19:37.763803+0000 2024-04-22T08:19:37.767 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:19:37.767 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:37.767 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:19:37.767 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:19:37.767 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:19:37.768 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:19:37.768 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:19:37.768 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:37.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:37.763+0000 7fad0d1ae640 1 Processor -- start 2024-04-22T08:19:37.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:37.763+0000 7fad0d1ae640 1 -- start start 2024-04-22T08:19:37.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:37.763+0000 7fad0d1ae640 10 monclient: init 2024-04-22T08:19:37.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:37.764+0000 7fad0d1ae640 10 monclient: _reopen_session rank -1 2024-04-22T08:19:37.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:37.764+0000 7fad0d1ae640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-04-22T08:19:37.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:37.764+0000 7fad0d1ae640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fad080a20d0 0x7fad080a24b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:37.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:37.764+0000 7fad0d1ae640 10 monclient(hunting): picked mon.noname-a con 0x7fad080a20d0 addr v2:172.21.15.40:3300/0 2024-04-22T08:19:37.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:37.764+0000 7fad0d1ae640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fad080a2d10 addr v1:172.21.15.40:6789/0 2024-04-22T08:19:37.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:37.764+0000 7fad0d1ae640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fad080a38d0 addr v1:172.21.15.189:6789/0 2024-04-22T08:19:37.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:37.764+0000 7fad0d1ae640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad080a45f0 con 0x7fad080a2d10 2024-04-22T08:19:37.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:37.764+0000 7fad0d1ae640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad080a0360 con 0x7fad080a38d0 2024-04-22T08:19:37.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:37.764+0000 7fad0d1ae640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:37.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:37.764+0000 7fad0d1ae640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad080a0640 con 0x7fad080a20d0 2024-04-22T08:19:37.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:37.764+0000 7fad0d1ae640 10 monclient(hunting): _renew_subs 2024-04-22T08:19:37.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:37.764+0000 7fad0d1ae640 10 monclient(hunting): authenticate will time out at 1251.757933s 2024-04-22T08:19:37.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:37.764+0000 7fad0659c640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7fad080a2d10 legacy=0x7fad080a31c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-04-22T08:19:37.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:37.764+0000 7fad06d9d640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7fad080a20d0 msgr2=0x7fad080a24b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:19:37.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:37.764+0000 7fad06d9d640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fad080a20d0 0x7fad080a24b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:19:37.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:37.764+0000 7fad0759e640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fad080a38d0 0x7fad0813db20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.40:35112/0 (socket says 172.21.15.40:35112) 2024-04-22T08:19:37.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:37.764+0000 7fad0759e640 1 -- 172.21.15.40:0/2714390026 learned_addr learned my addr 172.21.15.40:0/2714390026 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:19:37.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:37.766+0000 7fad0759e640 1 -- 172.21.15.40:0/2714390026 >> v1:172.21.15.189:6789/0 conn(0x7fad080a38d0 legacy=0x7fad0813db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:37.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:37.965+0000 7fad0659c640 1 -- 172.21.15.40:0/2714390026 >> v1:172.21.15.40:6789/0 conn(0x7fad080a2d10 legacy=0x7fad080a31c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-04-22T08:19:37.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:37.965+0000 7fad06d9d640 1 -- 172.21.15.40:0/2714390026 >> v2:172.21.15.40:3300/0 conn(0x7fad080a20d0 msgr2=0x7fad080a24b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:19:37.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:37.965+0000 7fad06d9d640 1 --2- 172.21.15.40:0/2714390026 >> v2:172.21.15.40:3300/0 conn(0x7fad080a20d0 0x7fad080a24b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:19:38.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:38.366+0000 7fad0659c640 1 -- 172.21.15.40:0/2714390026 >> v1:172.21.15.40:6789/0 conn(0x7fad080a2d10 legacy=0x7fad080a31c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-04-22T08:19:38.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:38.366+0000 7fad06d9d640 1 -- 172.21.15.40:0/2714390026 >> v2:172.21.15.40:3300/0 conn(0x7fad080a20d0 msgr2=0x7fad080a24b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:19:38.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:38.366+0000 7fad06d9d640 1 --2- 172.21.15.40:0/2714390026 >> v2:172.21.15.40:3300/0 conn(0x7fad080a20d0 0x7fad080a24b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:19:39.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:39.167+0000 7fad0659c640 1 -- 172.21.15.40:0/2714390026 >> v1:172.21.15.40:6789/0 conn(0x7fad080a2d10 legacy=0x7fad080a31c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-04-22T08:19:39.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:39.167+0000 7fad06d9d640 1 -- 172.21.15.40:0/2714390026 >> v2:172.21.15.40:3300/0 conn(0x7fad080a20d0 msgr2=0x7fad080a24b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:19:39.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:39.167+0000 7fad06d9d640 1 --2- 172.21.15.40:0/2714390026 >> v2:172.21.15.40:3300/0 conn(0x7fad080a20d0 0x7fad080a24b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T08:19:40.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.764+0000 7fad04d99640 10 monclient(hunting): tick 2024-04-22T08:19:40.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.764+0000 7fad04d99640 10 monclient(hunting): _check_auth_tickets 2024-04-22T08:19:40.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.764+0000 7fad04d99640 1 monclient(hunting): continuing hunt 2024-04-22T08:19:40.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.764+0000 7fad04d99640 10 monclient(hunting): _reopen_session rank -1 2024-04-22T08:19:40.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.764+0000 7fad04d99640 1 -- 172.21.15.40:0/2714390026 >> v2:172.21.15.40:3300/0 conn(0x7fad080a20d0 msgr2=0x7fad080a24b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:19:40.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.764+0000 7fad04d99640 1 --2- 172.21.15.40:0/2714390026 >> v2:172.21.15.40:3300/0 conn(0x7fad080a20d0 0x7fad080a24b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:40.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.764+0000 7fad04d99640 1 -- 172.21.15.40:0/2714390026 >> v1:172.21.15.189:6789/0 conn(0x7fad080a38d0 legacy=0x7fad0813db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:40.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.764+0000 7fad04d99640 1 -- 172.21.15.40:0/2714390026 >> v1:172.21.15.40:6789/0 conn(0x7fad080a2d10 legacy=0x7fad080a31c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:19:40.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.764+0000 7fad04d99640 10 monclient: _add_conns ranks=[4,2] 2024-04-22T08:19:40.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.764+0000 7fad04d99640 1 --2- 172.21.15.40:0/2714390026 >> v2:172.21.15.189:3300/0 conn(0x7facf4003740 0x7facf4005bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:40.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.764+0000 7fad04d99640 10 monclient(hunting): picked mon.noname-c con 0x7facf4003740 addr v2:172.21.15.189:3300/0 2024-04-22T08:19:40.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.764+0000 7fad04d99640 1 --2- 172.21.15.40:0/2714390026 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7facf4006250 0x7facf4008710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:40.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.764+0000 7fad04d99640 10 monclient(hunting): picked mon.noname-b con 0x7facf4006250 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:19:40.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.764+0000 7fad04d99640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:40.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.764+0000 7fad04d99640 1 -- 172.21.15.40:0/2714390026 --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad080a0640 con 0x7facf4003740 2024-04-22T08:19:40.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.764+0000 7fad04d99640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:40.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.764+0000 7fad04d99640 1 -- 172.21.15.40:0/2714390026 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facf4008e20 con 0x7facf4006250 2024-04-22T08:19:40.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.764+0000 7fad04d99640 10 monclient(hunting): _renew_subs 2024-04-22T08:19:40.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.764+0000 7fad0659c640 1 --2- 172.21.15.40:0/2714390026 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7facf4006250 0x7facf4008710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:40.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.764+0000 7fad0659c640 10 monclient(hunting): get_auth_request con 0x7facf4006250 auth_method 0 2024-04-22T08:19:40.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.764+0000 7fad0659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:40.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.764+0000 7fad0659c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:40.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.764+0000 7fad0659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:40.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.764+0000 7fad0659c640 1 -- 172.21.15.40:0/2714390026 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7facf4006250 msgr2=0x7facf4008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:19:40.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.764+0000 7fad0659c640 1 -- 172.21.15.40:0/2714390026 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7facf4006250 msgr2=0x7facf4008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:19:40.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.764+0000 7fad0659c640 1 --2- 172.21.15.40:0/2714390026 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7facf4006250 0x7facf4008710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:19:40.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.764+0000 7fad06d9d640 1 --2- 172.21.15.40:0/2714390026 >> v2:172.21.15.189:3300/0 conn(0x7facf4003740 0x7facf4005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:40.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.764+0000 7fad0659c640 1 --2- 172.21.15.40:0/2714390026 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7facf4006250 0x7facf4008710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:19:40.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.764+0000 7fad06d9d640 10 monclient(hunting): get_auth_request con 0x7facf4003740 auth_method 0 2024-04-22T08:19:40.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.764+0000 7fad06d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:40.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.764+0000 7fad06d9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:40.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.764+0000 7fad06d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:40.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.764+0000 7fad06d9d640 1 -- 172.21.15.40:0/2714390026 >> v2:172.21.15.189:3300/0 conn(0x7facf4003740 msgr2=0x7facf4005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:19:40.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.764+0000 7fad06d9d640 1 -- 172.21.15.40:0/2714390026 >> v2:172.21.15.189:3300/0 conn(0x7facf4003740 msgr2=0x7facf4005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:19:40.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.764+0000 7fad06d9d640 1 --2- 172.21.15.40:0/2714390026 >> v2:172.21.15.189:3300/0 conn(0x7facf4003740 0x7facf4005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:19:40.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.764+0000 7fad06d9d640 1 --2- 172.21.15.40:0/2714390026 >> v2:172.21.15.189:3300/0 conn(0x7facf4003740 0x7facf4005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:19:40.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.965+0000 7fad0659c640 1 --2- 172.21.15.40:0/2714390026 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7facf4006250 0x7facf4008710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:40.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.965+0000 7fad0659c640 10 monclient(hunting): get_auth_request con 0x7facf4006250 auth_method 0 2024-04-22T08:19:40.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.965+0000 7fad0659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:40.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.965+0000 7fad0659c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:40.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.965+0000 7fad0659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:19:40.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.965+0000 7fad0659c640 1 -- 172.21.15.40:0/2714390026 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7facf4006250 msgr2=0x7facf4008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:19:40.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.965+0000 7fad0659c640 1 -- 172.21.15.40:0/2714390026 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7facf4006250 msgr2=0x7facf4008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:19:40.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.965+0000 7fad0659c640 1 --2- 172.21.15.40:0/2714390026 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7facf4006250 0x7facf4008710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:19:40.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.965+0000 7fad0659c640 1 --2- 172.21.15.40:0/2714390026 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7facf4006250 0x7facf4008710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:19:40.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.965+0000 7fad06d9d640 1 --2- 172.21.15.40:0/2714390026 >> v2:172.21.15.189:3300/0 conn(0x7facf4003740 0x7facf4005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:40.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.965+0000 7fad06d9d640 10 monclient(hunting): get_auth_request con 0x7facf4003740 auth_method 0 2024-04-22T08:19:40.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.965+0000 7fad06d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:40.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.965+0000 7fad06d9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:40.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.965+0000 7fad06d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:19:40.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.965+0000 7fad06d9d640 1 -- 172.21.15.40:0/2714390026 >> v2:172.21.15.189:3300/0 conn(0x7facf4003740 msgr2=0x7facf4005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:19:40.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.965+0000 7fad06d9d640 1 -- 172.21.15.40:0/2714390026 >> v2:172.21.15.189:3300/0 conn(0x7facf4003740 msgr2=0x7facf4005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:19:40.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.965+0000 7fad06d9d640 1 --2- 172.21.15.40:0/2714390026 >> v2:172.21.15.189:3300/0 conn(0x7facf4003740 0x7facf4005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:19:40.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:40.965+0000 7fad06d9d640 1 --2- 172.21.15.40:0/2714390026 >> v2:172.21.15.189:3300/0 conn(0x7facf4003740 0x7facf4005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:19:41.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.366+0000 7fad0659c640 1 --2- 172.21.15.40:0/2714390026 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7facf4006250 0x7facf4008710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:41.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.366+0000 7fad0659c640 10 monclient(hunting): get_auth_request con 0x7facf4006250 auth_method 0 2024-04-22T08:19:41.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.366+0000 7fad0659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:41.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.366+0000 7fad0659c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:41.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.366+0000 7fad0659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:19:41.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.366+0000 7fad0659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:41.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.366+0000 7fad0659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:41.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.366+0000 7fad0659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:41.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.366+0000 7fad0659c640 10 monclient(hunting): handle_auth_done global_id 54698 payload 274 2024-04-22T08:19:41.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.366+0000 7fad0659c640 1 -- 172.21.15.40:0/2714390026 >> v2:172.21.15.189:3300/0 conn(0x7facf4003740 msgr2=0x7facf4005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:19:41.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.366+0000 7fad0659c640 1 --2- 172.21.15.40:0/2714390026 >> v2:172.21.15.189:3300/0 conn(0x7facf4003740 0x7facf4005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:41.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.366+0000 7fad0659c640 10 monclient: _finish_hunting 0 2024-04-22T08:19:41.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.366+0000 7fad0659c640 1 monclient: found mon.noname-b 2024-04-22T08:19:41.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.366+0000 7fad0659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:19:41.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.366+0000 7fad0659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:19:41.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.366+0000 7fad0659c640 1 -- 172.21.15.40:0/2714390026 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad080a0920 con 0x7facf4006250 2024-04-22T08:19:41.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.369+0000 7fad0659c640 1 -- 172.21.15.40:0/2714390026 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7facf4006250 msgr2=0x7facf4008710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:41.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.369+0000 7fad0659c640 1 --2- 172.21.15.40:0/2714390026 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7facf4006250 0x7facf4008710 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7facfc003980 tx=0x7facfc02c730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd039f5ffd9e076a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:41.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.369+0000 7fad05d9b640 1 -- 172.21.15.40:0/2714390026 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facfc031920 con 0x7facf4006250 2024-04-22T08:19:41.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.369+0000 7fad05d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:41.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.369+0000 7fad05d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:19:41.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.369+0000 7fad05d9b640 10 monclient: dump: 2024-04-22T08:19:41.369 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:41.369 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:41.369 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:41.369 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:41.369 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:41.369 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:41.370 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:41.370 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:41.370 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:41.370 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:41.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.369+0000 7fad05d9b640 10 monclient: _finish_auth 0 2024-04-22T08:19:41.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.369+0000 7fad05d9b640 10 monclient: _check_auth_tickets 2024-04-22T08:19:41.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.369+0000 7fad05d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:19:41.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.369+0000 7fad05d9b640 1 -- 172.21.15.40:0/2714390026 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facfc02f050 con 0x7facf4006250 2024-04-22T08:19:41.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.369+0000 7fad05d9b640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:19:41.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.369+0000 7fad05d9b640 1 -- 172.21.15.40:0/2714390026 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facfc031ca0 con 0x7facf4006250 2024-04-22T08:19:41.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.369+0000 7fad0d1ae640 5 monclient: authenticate success, global_id 54698 2024-04-22T08:19:41.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.369+0000 7fad0d1ae640 10 monclient: get_monmap_and_config success 2024-04-22T08:19:41.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.369+0000 7fad0d1ae640 4 set_mon_vals no callback set 2024-04-22T08:19:41.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.369+0000 7fad05d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:41.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.369+0000 7fad05d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:19:41.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.369+0000 7fad05d9b640 10 monclient: dump: 2024-04-22T08:19:41.371 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:41.371 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:41.371 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:41.371 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:41.371 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:41.371 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:41.372 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:41.372 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:41.372 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:41.372 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:41.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.370+0000 7fad0d1ae640 10 monclient: shutdown 2024-04-22T08:19:41.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.370+0000 7fad0d1ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:19:41.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.370+0000 7fad0d1ae640 1 -- 172.21.15.40:0/2714390026 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7facf4006250 msgr2=0x7facf4008710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:41.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.370+0000 7fad0d1ae640 1 --2- 172.21.15.40:0/2714390026 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7facf4006250 0x7facf4008710 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7facfc003980 tx=0x7facfc02c730 comp rx=0 tx=0).stop 2024-04-22T08:19:41.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.370+0000 7fad0759e640 1 -- 172.21.15.40:0/2714390026 reap_dead start 2024-04-22T08:19:41.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.370+0000 7fad0d1ae640 1 -- 172.21.15.40:0/2714390026 shutdown_connections 2024-04-22T08:19:41.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.370+0000 7fad0d1ae640 1 -- 172.21.15.40:0/2714390026 >> 172.21.15.40:0/2714390026 conn(0x7fad080a8e60 msgr2=0x7fad080a4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:19:41.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.370+0000 7fad0d1ae640 1 -- 172.21.15.40:0/2714390026 shutdown_connections 2024-04-22T08:19:41.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.370+0000 7fad0d1ae640 1 -- 172.21.15.40:0/2714390026 wait complete. 2024-04-22T08:19:41.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.371+0000 7fad0d1ae640 10 monclient: build_initial_monmap 2024-04-22T08:19:41.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.371+0000 7fad0d1ae640 10 monclient: monmap: 2024-04-22T08:19:41.378 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:19:41.378 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:19:41.378 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:19:41.378 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:19:41.378 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:19:41.378 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:41.379 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:19:41.379 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:19:41.379 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:19:41.379 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:41.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.372+0000 7fad0d1ae640 1 Processor -- start 2024-04-22T08:19:41.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.372+0000 7fad0d1ae640 1 -- start start 2024-04-22T08:19:41.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.372+0000 7fad0d1ae640 10 monclient: init 2024-04-22T08:19:41.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.372+0000 7fad0d1ae640 10 monclient: _reopen_session rank -1 2024-04-22T08:19:41.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.372+0000 7fad0d1ae640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-22T08:19:41.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.372+0000 7fad0d1ae640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad081ca990 0x7fad081cad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:41.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.372+0000 7fad0d1ae640 10 monclient(hunting): picked mon.noname-b con 0x7fad081ca990 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:19:41.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.372+0000 7fad0d1ae640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fad081c4820 0x7fad081c7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:41.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.372+0000 7fad0d1ae640 10 monclient(hunting): picked mon.noname-c con 0x7fad081c4820 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:19:41.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.372+0000 7fad0d1ae640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad081c4c80 0x7fad081c7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:41.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.372+0000 7fad0d1ae640 10 monclient(hunting): picked mon.noname-a con 0x7fad081c4c80 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:19:41.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.372+0000 7fad0d1ae640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:41.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.372+0000 7fad0d1ae640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad081a9e80 con 0x7fad081c4c80 2024-04-22T08:19:41.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.372+0000 7fad0d1ae640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:41.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.372+0000 7fad0d1ae640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad08070d40 con 0x7fad081ca990 2024-04-22T08:19:41.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.372+0000 7fad0d1ae640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:41.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.372+0000 7fad0d1ae640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad081ab8e0 con 0x7fad081c4820 2024-04-22T08:19:41.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.372+0000 7fad0d1ae640 10 monclient(hunting): _renew_subs 2024-04-22T08:19:41.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.372+0000 7fad0d1ae640 10 monclient(hunting): authenticate will time out at 1255.366624s 2024-04-22T08:19:41.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.372+0000 7fad0759e640 1 -- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad081c4c80 msgr2=0x7fad081c7950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:19:41.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.372+0000 7fad0759e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad081c4c80 0x7fad081c7950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:19:41.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.372+0000 7fad0659c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fad081c4820 0x7fad081c7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:41.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.373+0000 7fad0659c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fad081c4820 0x7fad081c7410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:48262/0 (socket says 172.21.15.40:48262) 2024-04-22T08:19:41.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.373+0000 7fad0659c640 1 -- 172.21.15.40:0/3537987845 learned_addr learned my addr 172.21.15.40:0/3537987845 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:19:41.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.373+0000 7fad0659c640 10 monclient(hunting): get_auth_request con 0x7fad081c4820 auth_method 0 2024-04-22T08:19:41.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.373+0000 7fad0659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:41.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.373+0000 7fad0659c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:41.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.373+0000 7fad0659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:41.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.373+0000 7fad06d9d640 1 --2- 172.21.15.40:0/3537987845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad081ca990 0x7fad081cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:41.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.373+0000 7fad0659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:41.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.373+0000 7fad0659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:41.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.373+0000 7fad0659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:41.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.373+0000 7fad06d9d640 10 monclient(hunting): get_auth_request con 0x7fad081ca990 auth_method 0 2024-04-22T08:19:41.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.373+0000 7fad06d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:41.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.373+0000 7fad06d9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:41.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.373+0000 7fad06d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:41.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.373+0000 7fad06d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:41.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.373+0000 7fad06d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:41.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.373+0000 7fad06d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:41.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.373+0000 7fad0659c640 10 monclient(hunting): handle_auth_done global_id 54701 payload 824 2024-04-22T08:19:41.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.373+0000 7fad0659c640 1 -- 172.21.15.40:0/3537987845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad081ca990 msgr2=0x7fad081cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:41.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.373+0000 7fad0659c640 1 --2- 172.21.15.40:0/3537987845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad081ca990 0x7fad081cad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:41.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.373+0000 7fad0659c640 1 -- 172.21.15.40:0/3537987845 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad081c4c80 msgr2=0x7fad081c7950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:19:41.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.373+0000 7fad0659c640 1 --2- 172.21.15.40:0/3537987845 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad081c4c80 0x7fad081c7950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:41.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.373+0000 7fad0659c640 10 monclient: _finish_hunting 0 2024-04-22T08:19:41.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.373+0000 7fad0659c640 1 monclient: found mon.noname-c 2024-04-22T08:19:41.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.373+0000 7fad0659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:19:41.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.373+0000 7fad0659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:19:41.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.373+0000 7fad0659c640 1 -- 172.21.15.40:0/3537987845 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facfc009a40 con 0x7fad081c4820 2024-04-22T08:19:41.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.375+0000 7fad0659c640 1 -- 172.21.15.40:0/3537987845 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fad081c4820 msgr2=0x7fad081c7410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:41.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.375+0000 7fad0659c640 1 --2- 172.21.15.40:0/3537987845 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fad081c4820 0x7fad081c7410 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7facfc009d30 tx=0x7facfc005d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:41.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.375+0000 7facef7fe640 1 -- 172.21.15.40:0/3537987845 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facfc003960 con 0x7fad081c4820 2024-04-22T08:19:41.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.376+0000 7facef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:41.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.376+0000 7facef7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:19:41.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.376+0000 7facef7fe640 10 monclient: dump: 2024-04-22T08:19:41.383 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:41.383 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:41.383 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:41.383 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:41.383 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:41.383 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:41.383 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:41.383 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:41.383 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:41.383 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:41.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.376+0000 7facef7fe640 10 monclient: _finish_auth 0 2024-04-22T08:19:41.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.376+0000 7facef7fe640 10 monclient: _check_auth_tickets 2024-04-22T08:19:41.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.376+0000 7facef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:19:41.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.376+0000 7facef7fe640 1 -- 172.21.15.40:0/3537987845 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facfc03e840 con 0x7fad081c4820 2024-04-22T08:19:41.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.376+0000 7fad0d1ae640 5 monclient: authenticate success, global_id 54701 2024-04-22T08:19:41.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.376+0000 7fad0d1ae640 10 monclient: _renew_subs 2024-04-22T08:19:41.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.376+0000 7fad0d1ae640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:19:41.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.376+0000 7fad0d1ae640 1 -- 172.21.15.40:0/3537987845 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad081abbc0 con 0x7fad081c4820 2024-04-22T08:19:41.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.376+0000 7facef7fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:19:41.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.376+0000 7facef7fe640 1 -- 172.21.15.40:0/3537987845 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facfc03ebb0 con 0x7fad081c4820 2024-04-22T08:19:41.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.376+0000 7facef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:41.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.376+0000 7facef7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:19:41.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.376+0000 7facef7fe640 10 monclient: dump: 2024-04-22T08:19:41.384 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:41.384 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:41.384 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:41.384 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:41.385 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:41.385 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:41.385 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:41.385 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:41.385 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:41.385 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:41.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.376+0000 7fad0d1ae640 10 monclient: _renew_subs 2024-04-22T08:19:41.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.376+0000 7fad0d1ae640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:19:41.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.376+0000 7fad0d1ae640 1 -- 172.21.15.40:0/3537987845 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad081cb2b0 con 0x7fad081c4820 2024-04-22T08:19:41.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.376+0000 7fad0559a640 4 set_mon_vals no callback set 2024-04-22T08:19:41.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.377+0000 7facef7fe640 1 -- 172.21.15.40:0/3537987845 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7facfc033400 con 0x7fad081c4820 2024-04-22T08:19:41.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.377+0000 7fad0d1ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:19:41.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.377+0000 7fad0d1ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:19:41.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.377+0000 7fad0d1ae640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:19:41.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.377+0000 7fad0d1ae640 1 -- 172.21.15.40:0/3537987845 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad0804e9b0 con 0x7fad081c4820 2024-04-22T08:19:41.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.377+0000 7facef7fe640 1 --2- 172.21.15.40:0/3537987845 >> v2:172.21.15.189:6800/30168 conn(0x7facf4038620 0x7facf403aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:41.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.377+0000 7facef7fe640 1 -- 172.21.15.40:0/3537987845 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 3898+0+0 (secure 0 0 0) 0x7facfc06ce40 con 0x7fad081c4820 2024-04-22T08:19:41.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.378+0000 7fad06d9d640 1 --2- 172.21.15.40:0/3537987845 >> v2:172.21.15.189:6800/30168 conn(0x7facf4038620 0x7facf403aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:41.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.378+0000 7fad06d9d640 10 monclient: get_auth_request con 0x7facf4038620 auth_method 0 2024-04-22T08:19:41.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.380+0000 7fad06d9d640 1 --2- 172.21.15.40:0/3537987845 >> v2:172.21.15.189:6800/30168 conn(0x7facf4038620 0x7facf403aae0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7facf80038b0 tx=0x7facf8003b60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:41.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.382+0000 7facef7fe640 1 -- 172.21.15.40:0/3537987845 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7facfc03ed90 con 0x7fad081c4820 2024-04-22T08:19:41.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.382+0000 7facef7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:19:41.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.382+0000 7facef7fe640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:19:41.572 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[37844]: starting. 2024-04-22T08:19:41.573 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[37844]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-37842 2024-04-22T08:19:41.573 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[37844]: shutting down. 2024-04-22T08:19:41.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.637+0000 7fad0d1ae640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:19:41.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.637+0000 7fad0d1ae640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:19:41.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.637+0000 7fad0d1ae640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:19:41.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.637+0000 7fad0d1ae640 1 -- 172.21.15.40:0/3537987845 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fad081a4d80 con 0x7fad081c4820 2024-04-22T08:19:41.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.638+0000 7facef7fe640 1 -- 172.21.15.40:0/3537987845 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1537 (secure 0 0 0) 0x7facfc042070 con 0x7fad081c4820 2024-04-22T08:19:41.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.638+0000 7facef7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:19:41.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.638+0000 7facef7fe640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:19:41.638 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":68,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:19:41.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.639+0000 7fad0d1ae640 1 -- 172.21.15.40:0/3537987845 >> v2:172.21.15.189:6800/30168 conn(0x7facf4038620 msgr2=0x7facf403aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:41.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.639+0000 7fad0d1ae640 1 --2- 172.21.15.40:0/3537987845 >> v2:172.21.15.189:6800/30168 conn(0x7facf4038620 0x7facf403aae0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7facf80038b0 tx=0x7facf8003b60 comp rx=0 tx=0).stop 2024-04-22T08:19:41.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.639+0000 7fad0d1ae640 10 monclient: shutdown 2024-04-22T08:19:41.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.639+0000 7fad0d1ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:19:41.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.639+0000 7fad0d1ae640 1 -- 172.21.15.40:0/3537987845 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fad081c4820 msgr2=0x7fad081c7410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:41.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.639+0000 7fad0d1ae640 1 --2- 172.21.15.40:0/3537987845 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fad081c4820 0x7fad081c7410 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7facfc009d30 tx=0x7facfc005d00 comp rx=0 tx=0).stop 2024-04-22T08:19:41.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.639+0000 7fad0d1ae640 1 -- 172.21.15.40:0/3537987845 shutdown_connections 2024-04-22T08:19:41.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.639+0000 7fad0d1ae640 1 --2- 172.21.15.40:0/3537987845 >> v2:172.21.15.189:6800/30168 conn(0x7facf4038620 0x7facf403aae0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:41.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.639+0000 7fad0d1ae640 1 --2- 172.21.15.40:0/3537987845 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad081c4c80 0x7fad081c7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:41.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.639+0000 7fad0d1ae640 1 --2- 172.21.15.40:0/3537987845 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fad081c4820 0x7fad081c7410 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:41.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.639+0000 7fad0d1ae640 1 --2- 172.21.15.40:0/3537987845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad081ca990 0x7fad081cad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:41.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.639+0000 7fad0d1ae640 1 -- 172.21.15.40:0/3537987845 >> 172.21.15.40:0/3537987845 conn(0x7fad080a8e60 msgr2=0x7fad080a78e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:19:41.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.640+0000 7fad0d1ae640 1 -- 172.21.15.40:0/3537987845 shutdown_connections 2024-04-22T08:19:41.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.640+0000 7fad0d1ae640 1 -- 172.21.15.40:0/3537987845 wait complete. 2024-04-22T08:19:41.650 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-22T08:19:41.650 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-04-22T08:19:41.650 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-22T08:19:41.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.750+0000 7f2e52208640 10 monclient: get_monmap_and_config 2024-04-22T08:19:41.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.750+0000 7f2e52208640 10 monclient: build_initial_monmap 2024-04-22T08:19:41.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.750+0000 7f2e52208640 10 monclient: monmap: 2024-04-22T08:19:41.750 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:19:41.750 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:41.750 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:19:41.751034+0000 2024-04-22T08:19:41.750 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:19:41.751034+0000 2024-04-22T08:19:41.750 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:19:41.750 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:41.750 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:19:41.750 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:19:41.750 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:19:41.750 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:19:41.750 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:19:41.750 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:41.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.750+0000 7f2e52208640 1 Processor -- start 2024-04-22T08:19:41.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.750+0000 7f2e52208640 1 -- start start 2024-04-22T08:19:41.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.750+0000 7f2e52208640 10 monclient: init 2024-04-22T08:19:41.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.751+0000 7f2e52208640 10 monclient: _reopen_session rank -1 2024-04-22T08:19:41.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.751+0000 7f2e52208640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-04-22T08:19:41.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.751+0000 7f2e52208640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e4c0a1c60 0x7f2e4c0a31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:41.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.751+0000 7f2e52208640 10 monclient(hunting): picked mon.noname-b con 0x7f2e4c0a1c60 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:19:41.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.751+0000 7f2e52208640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2e4c0a2040 addr v1:172.21.15.189:6789/0 2024-04-22T08:19:41.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.751+0000 7f2e52208640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2e4c0a24a0 0x7f2e4c1362c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:41.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.751+0000 7f2e52208640 10 monclient(hunting): picked mon.noname-c con 0x7f2e4c0a24a0 addr v2:172.21.15.189:3300/0 2024-04-22T08:19:41.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.751+0000 7f2e52208640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e4c0a41f0 con 0x7f2e4c0a2040 2024-04-22T08:19:41.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.751+0000 7f2e52208640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:41.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.751+0000 7f2e52208640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e4c09fe90 con 0x7f2e4c0a24a0 2024-04-22T08:19:41.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.751+0000 7f2e52208640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:41.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.751+0000 7f2e52208640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4c0a0170 con 0x7f2e4c0a1c60 2024-04-22T08:19:41.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.751+0000 7f2e52208640 10 monclient(hunting): _renew_subs 2024-04-22T08:19:41.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.751+0000 7f2e52208640 10 monclient(hunting): authenticate will time out at 1255.745206s 2024-04-22T08:19:41.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.751+0000 7f2e4b7fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e4c0a1c60 0x7f2e4c0a31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:41.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.751+0000 7f2e4b7fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e4c0a1c60 0x7f2e4c0a31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:48268/0 (socket says 172.21.15.40:48268) 2024-04-22T08:19:41.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.751+0000 7f2e4b7fe640 1 -- 172.21.15.40:0/4230189774 learned_addr learned my addr 172.21.15.40:0/4230189774 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:19:41.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.751+0000 7f2e4b7fe640 10 monclient(hunting): get_auth_request con 0x7f2e4c0a1c60 auth_method 0 2024-04-22T08:19:41.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.751+0000 7f2e4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:41.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.751+0000 7f2e4b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:41.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.751+0000 7f2e4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:41.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.751+0000 7f2e4b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:41.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.751+0000 7f2e4b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:41.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.751+0000 7f2e4b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:41.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.751+0000 7f2e4bfff640 1 --2- 172.21.15.40:0/4230189774 >> v2:172.21.15.189:3300/0 conn(0x7f2e4c0a24a0 0x7f2e4c1362c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:41.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.751+0000 7f2e4b7fe640 10 monclient(hunting): handle_auth_done global_id 54713 payload 274 2024-04-22T08:19:41.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.751+0000 7f2e4b7fe640 1 -- 172.21.15.40:0/4230189774 >> v2:172.21.15.189:3300/0 conn(0x7f2e4c0a24a0 msgr2=0x7f2e4c1362c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:19:41.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.751+0000 7f2e4b7fe640 1 --2- 172.21.15.40:0/4230189774 >> v2:172.21.15.189:3300/0 conn(0x7f2e4c0a24a0 0x7f2e4c1362c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:41.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.751+0000 7f2e4b7fe640 1 -- 172.21.15.40:0/4230189774 >> v1:172.21.15.189:6789/0 conn(0x7f2e4c0a2040 legacy=0x7f2e4c135bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:19:41.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.751+0000 7f2e4b7fe640 10 monclient: _finish_hunting 0 2024-04-22T08:19:41.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.751+0000 7f2e4b7fe640 1 monclient: found mon.noname-b 2024-04-22T08:19:41.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.751+0000 7f2e4b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:19:41.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.751+0000 7f2e4b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:19:41.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.751+0000 7f2e4b7fe640 1 -- 172.21.15.40:0/4230189774 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e4c0a0450 con 0x7f2e4c0a1c60 2024-04-22T08:19:41.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.751+0000 7f2e4bfff640 10 monclient: get_auth_request con 0x7f2e4c0a24a0 auth_method 0 2024-04-22T08:19:41.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.751+0000 7f2e4bfff640 1 --2- 172.21.15.40:0/4230189774 >> v2:172.21.15.189:3300/0 conn(0x7f2e4c0a24a0 0x7f2e4c1362c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T08:19:41.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.754+0000 7f2e4b7fe640 1 -- 172.21.15.40:0/4230189774 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e4c0a1c60 msgr2=0x7f2e4c0a31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:41.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.754+0000 7f2e4b7fe640 1 --2- 172.21.15.40:0/4230189774 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e4c0a1c60 0x7f2e4c0a31d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c009870 tx=0x7f2e3c02b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de37c9af48588fa9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:41.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.754+0000 7f2e4a7fc640 1 -- 172.21.15.40:0/4230189774 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e3c031a30 con 0x7f2e4c0a1c60 2024-04-22T08:19:41.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.754+0000 7f2e4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:41.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.754+0000 7f2e4a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:19:41.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.754+0000 7f2e4a7fc640 10 monclient: dump: 2024-04-22T08:19:41.761 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:41.762 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:41.762 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:41.762 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:41.762 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:41.762 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:41.762 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:41.762 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:41.762 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:41.762 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:41.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.754+0000 7f2e4a7fc640 10 monclient: _finish_auth 0 2024-04-22T08:19:41.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.754+0000 7f2e4a7fc640 10 monclient: _check_auth_tickets 2024-04-22T08:19:41.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.754+0000 7f2e4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:19:41.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.754+0000 7f2e4a7fc640 1 -- 172.21.15.40:0/4230189774 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e3c031b90 con 0x7f2e4c0a1c60 2024-04-22T08:19:41.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.754+0000 7f2e52208640 5 monclient: authenticate success, global_id 54713 2024-04-22T08:19:41.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.754+0000 7f2e52208640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:19:41.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.754+0000 7f2e4a7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:19:41.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.754+0000 7f2e52208640 10 monclient: get_monmap_and_config success 2024-04-22T08:19:41.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.754+0000 7f2e4a7fc640 1 -- 172.21.15.40:0/4230189774 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e3c033430 con 0x7f2e4c0a1c60 2024-04-22T08:19:41.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.754+0000 7f2e52208640 4 set_mon_vals no callback set 2024-04-22T08:19:41.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.754+0000 7f2e4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:41.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.754+0000 7f2e4a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:19:41.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.754+0000 7f2e4a7fc640 10 monclient: dump: 2024-04-22T08:19:41.763 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:41.763 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:41.763 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:41.763 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:41.763 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:41.763 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:41.764 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:41.764 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:41.764 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:41.764 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:41.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.755+0000 7f2e52208640 10 monclient: shutdown 2024-04-22T08:19:41.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.755+0000 7f2e52208640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:19:41.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.755+0000 7f2e52208640 1 -- 172.21.15.40:0/4230189774 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e4c0a1c60 msgr2=0x7f2e4c0a31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:41.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.755+0000 7f2e52208640 1 --2- 172.21.15.40:0/4230189774 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e4c0a1c60 0x7f2e4c0a31d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c009870 tx=0x7f2e3c02b7e0 comp rx=0 tx=0).stop 2024-04-22T08:19:41.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.755+0000 7f2e52208640 1 -- 172.21.15.40:0/4230189774 shutdown_connections 2024-04-22T08:19:41.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.755+0000 7f2e52208640 1 --2- 172.21.15.40:0/4230189774 >> v2:172.21.15.189:3300/0 conn(0x7f2e4c0a24a0 0x7f2e4c1362c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:41.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.755+0000 7f2e52208640 1 --2- 172.21.15.40:0/4230189774 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e4c0a1c60 0x7f2e4c0a31d0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:41.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.755+0000 7f2e52208640 1 -- 172.21.15.40:0/4230189774 >> 172.21.15.40:0/4230189774 conn(0x7f2e4c0a89a0 msgr2=0x7f2e4c0a4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:19:41.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.755+0000 7f2e52208640 1 -- 172.21.15.40:0/4230189774 shutdown_connections 2024-04-22T08:19:41.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.755+0000 7f2e52208640 1 -- 172.21.15.40:0/4230189774 wait complete. 2024-04-22T08:19:41.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.756+0000 7f2e52208640 10 monclient: build_initial_monmap 2024-04-22T08:19:41.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.756+0000 7f2e52208640 10 monclient: monmap: 2024-04-22T08:19:41.765 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:19:41.765 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:19:41.765 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:19:41.765 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:19:41.765 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:19:41.765 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:41.765 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:19:41.765 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:19:41.765 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:19:41.765 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:41.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.756+0000 7f2e52208640 1 Processor -- start 2024-04-22T08:19:41.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.756+0000 7f2e52208640 1 -- start start 2024-04-22T08:19:41.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.756+0000 7f2e52208640 10 monclient: init 2024-04-22T08:19:41.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.756+0000 7f2e52208640 10 monclient: _reopen_session rank -1 2024-04-22T08:19:41.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.756+0000 7f2e52208640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-22T08:19:41.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.756+0000 7f2e52208640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e4c0a1c60 0x7f2e4c1cad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:41.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.756+0000 7f2e52208640 10 monclient(hunting): picked mon.noname-b con 0x7f2e4c0a1c60 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:19:41.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.756+0000 7f2e52208640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e4c0a2040 0x7f2e4c1cb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:41.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.756+0000 7f2e52208640 10 monclient(hunting): picked mon.noname-c con 0x7f2e4c0a2040 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:19:41.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.756+0000 7f2e52208640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e4c0a24a0 0x7f2e4c122090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:41.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.756+0000 7f2e52208640 10 monclient(hunting): picked mon.noname-a con 0x7f2e4c0a24a0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:19:41.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.756+0000 7f2e52208640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:41.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.756+0000 7f2e52208640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4c124860 con 0x7f2e4c0a24a0 2024-04-22T08:19:41.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.756+0000 7f2e52208640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:41.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.756+0000 7f2e52208640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4c070d00 con 0x7f2e4c0a1c60 2024-04-22T08:19:41.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.756+0000 7f2e52208640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:41.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.756+0000 7f2e52208640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4c0a4790 con 0x7f2e4c0a2040 2024-04-22T08:19:41.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.756+0000 7f2e52208640 10 monclient(hunting): _renew_subs 2024-04-22T08:19:41.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.756+0000 7f2e52208640 10 monclient(hunting): authenticate will time out at 1255.750760s 2024-04-22T08:19:41.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.757+0000 7f2e4bfff640 1 -- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e4c0a24a0 msgr2=0x7f2e4c122090 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:19:41.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.757+0000 7f2e4bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e4c0a24a0 0x7f2e4c122090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:19:41.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.757+0000 7f2e4affd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e4c0a2040 0x7f2e4c1cb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:41.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.757+0000 7f2e4affd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e4c0a2040 0x7f2e4c1cb2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:48278/0 (socket says 172.21.15.40:48278) 2024-04-22T08:19:41.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.757+0000 7f2e4affd640 1 -- 172.21.15.40:0/3903580355 learned_addr learned my addr 172.21.15.40:0/3903580355 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:19:41.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.757+0000 7f2e4affd640 10 monclient(hunting): get_auth_request con 0x7f2e4c0a2040 auth_method 0 2024-04-22T08:19:41.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.757+0000 7f2e4affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:41.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.757+0000 7f2e4affd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:41.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.757+0000 7f2e4affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:41.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.757+0000 7f2e4affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:41.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.757+0000 7f2e4affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:41.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.757+0000 7f2e4affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:41.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.757+0000 7f2e4b7fe640 1 --2- 172.21.15.40:0/3903580355 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e4c0a1c60 0x7f2e4c1cad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:41.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.757+0000 7f2e4b7fe640 10 monclient(hunting): get_auth_request con 0x7f2e4c0a1c60 auth_method 0 2024-04-22T08:19:41.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.757+0000 7f2e4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:41.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.757+0000 7f2e4b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:41.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.757+0000 7f2e4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:41.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.757+0000 7f2e4affd640 10 monclient(hunting): handle_auth_done global_id 54716 payload 824 2024-04-22T08:19:41.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.757+0000 7f2e4affd640 1 -- 172.21.15.40:0/3903580355 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e4c0a1c60 msgr2=0x7f2e4c1cad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:41.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.757+0000 7f2e4affd640 1 --2- 172.21.15.40:0/3903580355 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e4c0a1c60 0x7f2e4c1cad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:41.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.757+0000 7f2e4affd640 1 -- 172.21.15.40:0/3903580355 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e4c0a24a0 msgr2=0x7f2e4c122090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:19:41.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.757+0000 7f2e4affd640 1 --2- 172.21.15.40:0/3903580355 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e4c0a24a0 0x7f2e4c122090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:41.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.757+0000 7f2e4affd640 10 monclient: _finish_hunting 0 2024-04-22T08:19:41.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.757+0000 7f2e4affd640 1 monclient: found mon.noname-c 2024-04-22T08:19:41.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.757+0000 7f2e4affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:19:41.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.757+0000 7f2e4affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:19:41.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.757+0000 7f2e4affd640 1 -- 172.21.15.40:0/3903580355 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e3c0094d0 con 0x7f2e4c0a2040 2024-04-22T08:19:41.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.759+0000 7f2e4affd640 1 -- 172.21.15.40:0/3903580355 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e4c0a2040 msgr2=0x7f2e4c1cb2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:41.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.759+0000 7f2e4affd640 1 --2- 172.21.15.40:0/3903580355 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e4c0a2040 0x7f2e4c1cb2c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2e4000b3f0 tx=0x7f2e4000b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:41.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.759+0000 7f2e33fff640 1 -- 172.21.15.40:0/3903580355 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e40011a40 con 0x7f2e4c0a2040 2024-04-22T08:19:41.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.759+0000 7f2e33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:41.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.759+0000 7f2e33fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:19:41.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.759+0000 7f2e33fff640 10 monclient: dump: 2024-04-22T08:19:41.770 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:41.770 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:41.770 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:41.770 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:41.770 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:41.770 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:41.770 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:41.771 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:41.771 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:41.771 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:41.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.759+0000 7f2e33fff640 10 monclient: _finish_auth 0 2024-04-22T08:19:41.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.759+0000 7f2e33fff640 10 monclient: _check_auth_tickets 2024-04-22T08:19:41.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.759+0000 7f2e33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:19:41.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.759+0000 7f2e33fff640 1 -- 172.21.15.40:0/3903580355 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e40011ba0 con 0x7f2e4c0a2040 2024-04-22T08:19:41.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.759+0000 7f2e33fff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:19:41.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.759+0000 7f2e33fff640 1 -- 172.21.15.40:0/3903580355 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e40010380 con 0x7f2e4c0a2040 2024-04-22T08:19:41.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.759+0000 7f2e52208640 5 monclient: authenticate success, global_id 54716 2024-04-22T08:19:41.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.759+0000 7f2e52208640 10 monclient: _renew_subs 2024-04-22T08:19:41.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.759+0000 7f2e52208640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:19:41.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.759+0000 7f2e52208640 1 -- 172.21.15.40:0/3903580355 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e4c123c90 con 0x7f2e4c0a2040 2024-04-22T08:19:41.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.759+0000 7f2e497fa640 4 set_mon_vals no callback set 2024-04-22T08:19:41.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.759+0000 7f2e33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:41.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.759+0000 7f2e33fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:19:41.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.759+0000 7f2e33fff640 10 monclient: dump: 2024-04-22T08:19:41.772 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:41.772 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:41.772 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:41.772 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:41.772 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:41.773 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:41.773 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:41.773 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:41.773 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:41.773 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:41.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.759+0000 7f2e52208640 10 monclient: _renew_subs 2024-04-22T08:19:41.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.759+0000 7f2e52208640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:19:41.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.759+0000 7f2e52208640 1 -- 172.21.15.40:0/3903580355 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e4c125ff0 con 0x7f2e4c0a2040 2024-04-22T08:19:41.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.760+0000 7f2e52208640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:19:41.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.760+0000 7f2e33fff640 1 -- 172.21.15.40:0/3903580355 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f2e40010560 con 0x7f2e4c0a2040 2024-04-22T08:19:41.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.760+0000 7f2e52208640 1 --2- 172.21.15.40:0/3903580355 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e4c126410 0x7f2e4c1225d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:41.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.760+0000 7f2e52208640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:41.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.760+0000 7f2e52208640 1 -- 172.21.15.40:0/3903580355 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4c066650 con 0x7f2e4c126410 2024-04-22T08:19:41.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.760+0000 7f2e4b7fe640 1 --2- 172.21.15.40:0/3903580355 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e4c126410 0x7f2e4c1225d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:41.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.760+0000 7f2e4b7fe640 10 monclient: get_auth_request con 0x7f2e4c126410 auth_method 0 2024-04-22T08:19:41.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.760+0000 7f2e4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:41.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.760+0000 7f2e4b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:41.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.760+0000 7f2e4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:41.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.760+0000 7f2e33fff640 1 --2- 172.21.15.40:0/3903580355 >> v2:172.21.15.189:6800/30168 conn(0x7f2e380377d0 0x7f2e38039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:41.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.760+0000 7f2e4b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:41.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.760+0000 7f2e4b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:41.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.760+0000 7f2e4b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:41.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.760+0000 7f2e33fff640 1 -- 172.21.15.40:0/3903580355 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f2e4004a0a0 con 0x7f2e4c0a2040 2024-04-22T08:19:41.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.761+0000 7f2e4b7fe640 10 monclient(hunting): handle_auth_done global_id 54719 payload 824 2024-04-22T08:19:41.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.761+0000 7f2e4b7fe640 1 -- 172.21.15.40:0/3903580355 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2e4c136800 con 0x7f2e4c126410 2024-04-22T08:19:41.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.761+0000 7f2e4bfff640 1 --2- 172.21.15.40:0/3903580355 >> v2:172.21.15.189:6800/30168 conn(0x7f2e380377d0 0x7f2e38039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:41.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.761+0000 7f2e4bfff640 10 monclient: get_auth_request con 0x7f2e380377d0 auth_method 0 2024-04-22T08:19:41.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.763+0000 7f2e4b7fe640 1 -- 172.21.15.40:0/3903580355 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e4c126410 msgr2=0x7f2e4c1225d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:41.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.763+0000 7f2e4b7fe640 1 --2- 172.21.15.40:0/3903580355 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e4c126410 0x7f2e4c1225d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c035000 tx=0x7f2e3c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:41.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.763+0000 7f2e33fff640 1 -- 172.21.15.40:0/3903580355 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e3c031f10 con 0x7f2e4c126410 2024-04-22T08:19:41.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.763+0000 7f2e33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:41.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.763+0000 7f2e33fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:19:41.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.763+0000 7f2e33fff640 10 monclient: dump: 2024-04-22T08:19:41.778 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:41.778 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:41.778 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:41.778 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:41.778 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:41.778 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:41.778 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:41.779 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:41.779 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:41.779 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:41.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.763+0000 7f2e33fff640 1 -- 172.21.15.40:0/3903580355 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f2e3c03d640 con 0x7f2e4c126410 2024-04-22T08:19:41.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.763+0000 7f2e33fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:19:41.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.763+0000 7f2e33fff640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:19:41.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.763+0000 7f2e33fff640 1 -- 172.21.15.40:0/3903580355 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e4c126410 msgr2=0x7f2e4c1225d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:41.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.763+0000 7f2e33fff640 1 --2- 172.21.15.40:0/3903580355 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e4c126410 0x7f2e4c1225d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c035000 tx=0x7f2e3c0046c0 comp rx=0 tx=0).stop 2024-04-22T08:19:41.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.763+0000 7f2e33fff640 1 -- 172.21.15.40:0/3903580355 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e4c126410 msgr2=0x7f2e4c1225d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:19:41.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.763+0000 7f2e33fff640 1 --2- 172.21.15.40:0/3903580355 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e4c126410 0x7f2e4c1225d0 secure :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c035000 tx=0x7f2e3c0046c0 comp rx=0 tx=0).stop 2024-04-22T08:19:41.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.763+0000 7f2e4bfff640 1 --2- 172.21.15.40:0/3903580355 >> v2:172.21.15.189:6800/30168 conn(0x7f2e380377d0 0x7f2e38039c90 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f2e440012a0 tx=0x7f2e44008390 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:41.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.769+0000 7f2e52208640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:19:41.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.769+0000 7f2e52208640 1 --2- 172.21.15.40:0/3903580355 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e4c122b10 0x7f2e4c122f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:41.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.770+0000 7f2e52208640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:41.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.770+0000 7f2e52208640 1 -- 172.21.15.40:0/3903580355 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4c123460 con 0x7f2e4c122b10 2024-04-22T08:19:41.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.770+0000 7f2e4b7fe640 1 --2- 172.21.15.40:0/3903580355 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e4c122b10 0x7f2e4c122f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:41.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.770+0000 7f2e4b7fe640 10 monclient: get_auth_request con 0x7f2e4c122b10 auth_method 0 2024-04-22T08:19:41.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.770+0000 7f2e4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:41.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.770+0000 7f2e4b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:41.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.770+0000 7f2e4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:41.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.770+0000 7f2e4b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:41.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.770+0000 7f2e4b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:41.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.770+0000 7f2e4b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:41.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.770+0000 7f2e4b7fe640 10 monclient(hunting): handle_auth_done global_id 54722 payload 824 2024-04-22T08:19:41.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.770+0000 7f2e4b7fe640 1 -- 172.21.15.40:0/3903580355 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2e4c04e9b0 con 0x7f2e4c122b10 2024-04-22T08:19:41.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.772+0000 7f2e4b7fe640 1 -- 172.21.15.40:0/3903580355 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e4c122b10 msgr2=0x7f2e4c122f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:41.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.772+0000 7f2e4b7fe640 1 --2- 172.21.15.40:0/3903580355 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e4c122b10 0x7f2e4c122f20 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c02b820 tx=0x7f2e3c031000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:41.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.772+0000 7f2e33fff640 1 -- 172.21.15.40:0/3903580355 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e3c003810 con 0x7f2e4c122b10 2024-04-22T08:19:41.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.772+0000 7f2e33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:41.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.772+0000 7f2e33fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:19:41.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.772+0000 7f2e33fff640 10 monclient: dump: 2024-04-22T08:19:41.782 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:41.782 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:41.782 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:41.782 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:41.782 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:41.782 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:41.782 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:41.782 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:41.783 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:41.783 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:41.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.772+0000 7f2e33fff640 1 -- 172.21.15.40:0/3903580355 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4620 (secure 0 0 0) 0x7f2e3c03de80 con 0x7f2e4c122b10 2024-04-22T08:19:41.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.772+0000 7f2e33fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:19:41.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.772+0000 7f2e33fff640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:19:41.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.772+0000 7f2e33fff640 1 -- 172.21.15.40:0/3903580355 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e4c122b10 msgr2=0x7f2e4c122f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:41.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.772+0000 7f2e33fff640 1 --2- 172.21.15.40:0/3903580355 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e4c122b10 0x7f2e4c122f20 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c02b820 tx=0x7f2e3c031000 comp rx=0 tx=0).stop 2024-04-22T08:19:41.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.772+0000 7f2e33fff640 1 -- 172.21.15.40:0/3903580355 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e4c122b10 msgr2=0x7f2e4c122f20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:19:41.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.772+0000 7f2e33fff640 1 --2- 172.21.15.40:0/3903580355 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e4c122b10 0x7f2e4c122f20 secure :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c02b820 tx=0x7f2e3c031000 comp rx=0 tx=0).stop 2024-04-22T08:19:41.784 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:19:41.784 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:19:41.784 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:19:41.784 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:19:41.784 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 68, 2024-04-22T08:19:41.784 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:19:41.784 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:19:41.784 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:19:41.784 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:19:41.784 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 0, 2024-04-22T08:19:41.784 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:19:41.785 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:19:41.785 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:19:41.785 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:19:41.785 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:19:41.785 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:19:41.785 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:19:41.785 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:19:41.785 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:19:41.785 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:19:41.785 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:19:41.785 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:19:41.785 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:19:41.785 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:19:41.786 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:19:41.786 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:19:41.786 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:19:41.786 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:19:41.786 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:19:41.786 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:19:41.787 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:19:41.787 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:19:41.787 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:19:41.787 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:19:41.787 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:19:41.787 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:19:41.787 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:41.787 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:41.787 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:19:41.787 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:19:41.788 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:19:41.788 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:19:41.788 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:19:41.788 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:19:41.788 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:19:41.788 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:19:41.788 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:19:41.788 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:19:41.788 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:19:41.788 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:19:41.788 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:19:41.788 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:19:41.788 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:19:41.788 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:19:41.789 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:19:41.789 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:19:41.789 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:19:41.789 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:19:41.789 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:19:41.789 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:19:41.789 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:19:41.789 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:19:41.789 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:19:41.789 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:19:41.789 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:19:41.789 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:19:41.789 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:19:41.790 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:41.790 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:19:41.790 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:41.790 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:19:41.790 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:19:41.790 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:19:41.790 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:19:41.790 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:41.790 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:19:41.790 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:19:41.790 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:41.790 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:41.790 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:41.791 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:19:41.791 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:19:41.791 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:41.791 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:41.791 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:41.791 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:41.791 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:19:41.791 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:19:41.791 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:19:41.791 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:19:41.791 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:19:41.791 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:41.791 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:41.791 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:19:41.792 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:19:41.792 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:19:41.792 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:19:41.792 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:41.792 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:19:41.792 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:19:41.792 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:41.792 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:41.793 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:41.793 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:19:41.793 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:19:41.793 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:41.793 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:41.793 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:41.793 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:41.793 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:19:41.793 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:19:41.793 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:19:41.793 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:19:41.793 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:19:41.793 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:41.794 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:41.794 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:19:41.794 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:19:41.794 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:19:41.794 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:19:41.794 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:41.794 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:19:41.794 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:19:41.794 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:41.794 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:41.794 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:41.794 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:19:41.794 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:19:41.795 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:41.795 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:41.795 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:41.795 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:41.795 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:19:41.795 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:19:41.795 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:19:41.795 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:19:41.795 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:19:41.795 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:41.795 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:41.795 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:41.796 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:19:41.796 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:19:41.796 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:41.796 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:19:41.796 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:19:41.796 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:19:41.796 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:41.796 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:19:41.796 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:19:41.796 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:41.796 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:19:41.796 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:19:41.796 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 3 2024-04-22T08:19:41.796 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:41.797 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:19:41.797 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:19:41.797 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:41.797 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:19:41.797 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:19:41.797 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 7 2024-04-22T08:19:41.797 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:41.797 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:41.797 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:41.797 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:19:41.797 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:19:41.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.773+0000 7f2e52208640 1 -- 172.21.15.40:0/3903580355 >> v2:172.21.15.189:6800/30168 conn(0x7f2e380377d0 msgr2=0x7f2e38039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:41.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.773+0000 7f2e52208640 1 --2- 172.21.15.40:0/3903580355 >> v2:172.21.15.189:6800/30168 conn(0x7f2e380377d0 0x7f2e38039c90 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f2e440012a0 tx=0x7f2e44008390 comp rx=0 tx=0).stop 2024-04-22T08:19:41.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.773+0000 7f2e52208640 10 monclient: shutdown 2024-04-22T08:19:41.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.773+0000 7f2e52208640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:19:41.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.773+0000 7f2e52208640 1 -- 172.21.15.40:0/3903580355 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e4c0a2040 msgr2=0x7f2e4c1cb2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:41.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.773+0000 7f2e52208640 1 --2- 172.21.15.40:0/3903580355 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e4c0a2040 0x7f2e4c1cb2c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2e4000b3f0 tx=0x7f2e4000b8b0 comp rx=0 tx=0).stop 2024-04-22T08:19:41.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.773+0000 7f2e4bfff640 1 -- 172.21.15.40:0/3903580355 reap_dead start 2024-04-22T08:19:41.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.773+0000 7f2e4bfff640 1 -- 172.21.15.40:0/3903580355 reap_dead start 2024-04-22T08:19:41.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.773+0000 7f2e52208640 1 -- 172.21.15.40:0/3903580355 shutdown_connections 2024-04-22T08:19:41.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.773+0000 7f2e52208640 1 -- 172.21.15.40:0/3903580355 >> 172.21.15.40:0/3903580355 conn(0x7f2e4c0a89a0 msgr2=0x7f2e4c0a0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:19:41.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.773+0000 7f2e52208640 1 -- 172.21.15.40:0/3903580355 shutdown_connections 2024-04-22T08:19:41.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.773+0000 7f2e52208640 1 -- 172.21.15.40:0/3903580355 wait complete. 2024-04-22T08:19:41.800 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-22T08:19:41.800 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-04-22T08:19:41.800 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-22T08:19:41.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.900+0000 7fbc85999640 10 monclient: get_monmap_and_config 2024-04-22T08:19:41.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.900+0000 7fbc85999640 10 monclient: build_initial_monmap 2024-04-22T08:19:41.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.900+0000 7fbc85999640 10 monclient: monmap: 2024-04-22T08:19:41.903 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:19:41.903 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:41.903 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:19:41.901579+0000 2024-04-22T08:19:41.903 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:19:41.901579+0000 2024-04-22T08:19:41.903 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:19:41.904 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:41.904 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:19:41.904 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:19:41.904 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:19:41.904 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:19:41.904 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:19:41.905 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:41.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.901+0000 7fbc85999640 1 Processor -- start 2024-04-22T08:19:41.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.901+0000 7fbc85999640 1 -- start start 2024-04-22T08:19:41.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.901+0000 7fbc85999640 10 monclient: init 2024-04-22T08:19:41.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.901+0000 7fbc85999640 10 monclient: _reopen_session rank -1 2024-04-22T08:19:41.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.901+0000 7fbc85999640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-04-22T08:19:41.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.901+0000 7fbc85999640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbc800a1a80 addr v1:172.21.15.189:6789/0 2024-04-22T08:19:41.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.901+0000 7fbc85999640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbc800a2570 0x7fbc800a01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:41.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.901+0000 7fbc85999640 10 monclient(hunting): picked mon.noname-a con 0x7fbc800a2570 addr v2:172.21.15.40:3300/0 2024-04-22T08:19:41.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.901+0000 7fbc85999640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbc8013b0e0 addr v1:172.21.15.40:6789/0 2024-04-22T08:19:41.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.901+0000 7fbc85999640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc8009d860 con 0x7fbc8013b0e0 2024-04-22T08:19:41.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.901+0000 7fbc85999640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc8009ef50 con 0x7fbc800a1a80 2024-04-22T08:19:41.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.901+0000 7fbc85999640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:41.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.901+0000 7fbc85999640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc8009f880 con 0x7fbc800a2570 2024-04-22T08:19:41.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.901+0000 7fbc85999640 10 monclient(hunting): _renew_subs 2024-04-22T08:19:41.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.901+0000 7fbc85999640 10 monclient(hunting): authenticate will time out at 1255.895679s 2024-04-22T08:19:41.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.901+0000 7fbc7f7fe640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7fbc8013b0e0 legacy=0x7fbc8013d4d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-04-22T08:19:41.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.901+0000 7fbc767fc640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7fbc800a2570 msgr2=0x7fbc800a01d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:19:41.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.901+0000 7fbc767fc640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbc800a2570 0x7fbc800a01d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:19:41.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.902+0000 7fbc7effd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fbc800a1a80 0x7fbc800a1e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.40:55824/0 (socket says 172.21.15.40:55824) 2024-04-22T08:19:41.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.902+0000 7fbc7effd640 1 -- 172.21.15.40:0/2865791504 learned_addr learned my addr 172.21.15.40:0/2865791504 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:19:41.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.904+0000 7fbc7effd640 1 -- 172.21.15.40:0/2865791504 >> v1:172.21.15.189:6789/0 conn(0x7fbc800a1a80 legacy=0x7fbc800a1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:41.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.904+0000 7fbc7e7fc640 1 -- 172.21.15.40:0/2865791504 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1887314884 0 0) 0x7fbc8009ef50 con 0x7fbc800a1a80 2024-04-22T08:19:41.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.904+0000 7fbc7e7fc640 10 monclient(hunting): _negotiate 2024-04-22T08:19:41.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.904+0000 7fbc7e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:41.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.904+0000 7fbc7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:41.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.904+0000 7fbc7e7fc640 10 monclient(hunting): my global_id is 64540 2024-04-22T08:19:41.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.904+0000 7fbc7e7fc640 1 -- 172.21.15.40:0/2865791504 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc6c003520 con 0x7fbc800a1a80 2024-04-22T08:19:41.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc7e7fc640 1 -- 172.21.15.40:0/2865791504 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2314531965 0 0) 0x7fbc6c003520 con 0x7fbc800a1a80 2024-04-22T08:19:41.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc7e7fc640 1 -- 172.21.15.40:0/2865791504 >> v2:172.21.15.40:3300/0 conn(0x7fbc800a2570 msgr2=0x7fbc800a01d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:19:41.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc7e7fc640 1 --2- 172.21.15.40:0/2865791504 >> v2:172.21.15.40:3300/0 conn(0x7fbc800a2570 0x7fbc800a01d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:41.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc7e7fc640 1 -- 172.21.15.40:0/2865791504 >> v1:172.21.15.40:6789/0 conn(0x7fbc8013b0e0 legacy=0x7fbc8013d4d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:19:41.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc7e7fc640 10 monclient: _finish_hunting 0 2024-04-22T08:19:41.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc7e7fc640 1 monclient: found mon.noname-c-legacy 2024-04-22T08:19:41.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc7e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:19:41.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc7e7fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.189:6789/0 2024-04-22T08:19:41.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc7e7fc640 1 -- 172.21.15.40:0/2865791504 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc80072160 con 0x7fbc800a1a80 2024-04-22T08:19:41.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc7e7fc640 10 monclient: _finish_auth 0 2024-04-22T08:19:41.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc7e7fc640 10 monclient: _check_auth_tickets 2024-04-22T08:19:41.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc7e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:19:41.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc7e7fc640 1 -- 172.21.15.40:0/2865791504 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1087006497 0 0) 0x7fbc70001920 con 0x7fbc800a1a80 2024-04-22T08:19:41.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:41.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc7e7fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-04-22T08:19:41.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc7e7fc640 10 monclient: dump: 2024-04-22T08:19:41.927 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:41.928 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:41.928 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:41.928 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:41.928 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:41.928 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:41.928 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:41.928 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:41.928 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:41.928 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:41.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc7e7fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] but i'm connected to v1:172.21.15.189:6789/0, reconnecting 2024-04-22T08:19:41.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc7e7fc640 10 monclient: _reopen_session rank -1 2024-04-22T08:19:41.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc7e7fc640 1 -- 172.21.15.40:0/2865791504 >> v1:172.21.15.189:6789/0 conn(0x7fbc800a1a80 legacy=0x7fbc800a1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:41.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc7e7fc640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:19:41.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc7e7fc640 1 --2- 172.21.15.40:0/2865791504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc6c003970 0x7fbc6c003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:41.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc7e7fc640 10 monclient(hunting): picked mon.a con 0x7fbc6c003970 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:19:41.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc7e7fc640 1 --2- 172.21.15.40:0/2865791504 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc6c0047c0 0x7fbc6c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:41.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc7e7fc640 10 monclient(hunting): picked mon.b con 0x7fbc6c0047c0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:19:41.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc7e7fc640 1 --2- 172.21.15.40:0/2865791504 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbc6c005380 0x7fbc6c00b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:41.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc7e7fc640 10 monclient(hunting): picked mon.c con 0x7fbc6c005380 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:19:41.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc7e7fc640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:41.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc7e7fc640 1 -- 172.21.15.40:0/2865791504 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8009f880 con 0x7fbc6c003970 2024-04-22T08:19:41.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc7e7fc640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:41.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc7e7fc640 1 -- 172.21.15.40:0/2865791504 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc6c00c2c0 con 0x7fbc6c0047c0 2024-04-22T08:19:41.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc7e7fc640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:41.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc7e7fc640 1 -- 172.21.15.40:0/2865791504 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc6c00c480 con 0x7fbc6c005380 2024-04-22T08:19:41.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc7e7fc640 10 monclient(hunting): _renew_subs 2024-04-22T08:19:41.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc7e7fc640 10 monclient(hunting): _finish_auth 0 2024-04-22T08:19:41.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc7effd640 1 -- 172.21.15.40:0/2865791504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc6c003970 msgr2=0x7fbc6c003d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:19:41.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc7effd640 1 --2- 172.21.15.40:0/2865791504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc6c003970 0x7fbc6c003d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:19:41.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc7f7fe640 1 --2- 172.21.15.40:0/2865791504 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbc6c005380 0x7fbc6c00b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:41.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc767fc640 1 --2- 172.21.15.40:0/2865791504 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc6c0047c0 0x7fbc6c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:41.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc7f7fe640 10 monclient(hunting): get_auth_request con 0x7fbc6c005380 auth_method 0 2024-04-22T08:19:41.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc7f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:41.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc7f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:41.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc7f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:19:41.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc767fc640 10 monclient(hunting): get_auth_request con 0x7fbc6c0047c0 auth_method 0 2024-04-22T08:19:41.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc767fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:41.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc767fc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:41.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc767fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:19:41.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc7f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:41.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc7f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:41.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc7f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-22T08:19:41.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc767fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:41.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc767fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:41.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc767fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-22T08:19:41.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc767fc640 10 monclient(hunting): handle_auth_done global_id 64540 payload 293 2024-04-22T08:19:41.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc767fc640 1 -- 172.21.15.40:0/2865791504 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbc6c005380 msgr2=0x7fbc6c00b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:19:41.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc767fc640 1 --2- 172.21.15.40:0/2865791504 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbc6c005380 0x7fbc6c00b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:41.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc767fc640 1 -- 172.21.15.40:0/2865791504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc6c003970 msgr2=0x7fbc6c003d70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:19:41.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc767fc640 1 --2- 172.21.15.40:0/2865791504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc6c003970 0x7fbc6c003d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:41.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc767fc640 10 monclient: _finish_hunting 0 2024-04-22T08:19:41.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc767fc640 1 monclient: found mon.b 2024-04-22T08:19:41.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:19:41.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc767fc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.189:3300/0 2024-04-22T08:19:41.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc767fc640 1 -- 172.21.15.40:0/2865791504 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc6c003520 con 0x7fbc6c0047c0 2024-04-22T08:19:41.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc767fc640 10 monclient: _finish_auth 0 2024-04-22T08:19:41.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc767fc640 10 monclient: _check_auth_tickets 2024-04-22T08:19:41.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:19:41.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc85999640 5 monclient: authenticate success, global_id 64540 2024-04-22T08:19:41.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc85999640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:19:41.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc7f7fe640 1 --2- 172.21.15.40:0/2865791504 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbc6c005380 0x7fbc6c00b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:19:41.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.905+0000 7fbc7f7fe640 1 -- 172.21.15.40:0/2865791504 reap_dead start 2024-04-22T08:19:41.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.908+0000 7fbc767fc640 1 -- 172.21.15.40:0/2865791504 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc6c0047c0 msgr2=0x7fbc6c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:41.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.908+0000 7fbc767fc640 1 --2- 172.21.15.40:0/2865791504 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc6c0047c0 0x7fbc6c004c40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fbc68009830 tx=0x7fbc6c0057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97e9cff88f04aeb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:41.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.908+0000 7fbc7e7fc640 1 -- 172.21.15.40:0/2865791504 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc68003ae0 con 0x7fbc6c0047c0 2024-04-22T08:19:41.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.908+0000 7fbc7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:41.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.908+0000 7fbc7e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:19:41.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.908+0000 7fbc7e7fc640 10 monclient: dump: 2024-04-22T08:19:41.932 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:41.932 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:41.932 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:41.932 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:41.932 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:41.933 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:41.933 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:41.933 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:41.933 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:41.933 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:41.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.908+0000 7fbc85999640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:19:41.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.908+0000 7fbc7e7fc640 1 -- 172.21.15.40:0/2865791504 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc6802e070 con 0x7fbc6c0047c0 2024-04-22T08:19:41.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.908+0000 7fbc7e7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:19:41.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.908+0000 7fbc7e7fc640 1 -- 172.21.15.40:0/2865791504 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc68003e80 con 0x7fbc6c0047c0 2024-04-22T08:19:41.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.908+0000 7fbc85999640 10 monclient: get_monmap_and_config success 2024-04-22T08:19:41.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.908+0000 7fbc7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:41.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.908+0000 7fbc85999640 4 set_mon_vals no callback set 2024-04-22T08:19:41.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.908+0000 7fbc7e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:19:41.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.908+0000 7fbc7e7fc640 10 monclient: dump: 2024-04-22T08:19:41.933 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:41.934 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:41.934 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:41.934 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:41.934 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:41.934 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:41.934 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:41.934 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:41.934 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:41.934 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:41.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.909+0000 7fbc85999640 10 monclient: shutdown 2024-04-22T08:19:41.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.909+0000 7fbc85999640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:19:41.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.909+0000 7fbc85999640 1 -- 172.21.15.40:0/2865791504 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc6c0047c0 msgr2=0x7fbc6c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:41.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.909+0000 7fbc85999640 1 --2- 172.21.15.40:0/2865791504 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc6c0047c0 0x7fbc6c004c40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fbc68009830 tx=0x7fbc6c0057f0 comp rx=0 tx=0).stop 2024-04-22T08:19:41.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.909+0000 7fbc85999640 1 -- 172.21.15.40:0/2865791504 shutdown_connections 2024-04-22T08:19:41.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.909+0000 7fbc85999640 1 --2- 172.21.15.40:0/2865791504 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc6c0047c0 0x7fbc6c004c40 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:41.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.909+0000 7fbc85999640 1 -- 172.21.15.40:0/2865791504 >> 172.21.15.40:0/2865791504 conn(0x7fbc800a66a0 msgr2=0x7fbc800a2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:19:41.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.909+0000 7fbc85999640 1 -- 172.21.15.40:0/2865791504 shutdown_connections 2024-04-22T08:19:41.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.909+0000 7fbc85999640 1 -- 172.21.15.40:0/2865791504 wait complete. 2024-04-22T08:19:41.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.910+0000 7fbc85999640 10 monclient: build_initial_monmap 2024-04-22T08:19:41.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.910+0000 7fbc85999640 10 monclient: monmap: 2024-04-22T08:19:41.935 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:19:41.935 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:19:41.935 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:19:41.935 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:19:41.935 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:19:41.935 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:41.935 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:19:41.935 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:19:41.935 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:19:41.935 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:41.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.910+0000 7fbc85999640 1 Processor -- start 2024-04-22T08:19:41.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.910+0000 7fbc85999640 1 -- start start 2024-04-22T08:19:41.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.910+0000 7fbc85999640 10 monclient: init 2024-04-22T08:19:41.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.910+0000 7fbc85999640 10 monclient: _reopen_session rank -1 2024-04-22T08:19:41.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.910+0000 7fbc85999640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-22T08:19:41.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.910+0000 7fbc85999640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbc6c0047c0 0x7fbc800707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:41.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.910+0000 7fbc85999640 10 monclient(hunting): picked mon.noname-c con 0x7fbc6c0047c0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:19:41.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.910+0000 7fbc85999640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc80136c20 0x7fbc801370a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:41.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.910+0000 7fbc85999640 10 monclient(hunting): picked mon.noname-b con 0x7fbc80136c20 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:19:41.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.910+0000 7fbc85999640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc80120ad0 0x7fbc80120f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:41.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.910+0000 7fbc85999640 10 monclient(hunting): picked mon.noname-a con 0x7fbc80120ad0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:19:41.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.910+0000 7fbc85999640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:41.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.910+0000 7fbc85999640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc801375e0 con 0x7fbc80120ad0 2024-04-22T08:19:41.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.910+0000 7fbc85999640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:41.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.910+0000 7fbc85999640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc80138ce0 con 0x7fbc80136c20 2024-04-22T08:19:41.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.910+0000 7fbc85999640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:41.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.910+0000 7fbc85999640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc801cb2c0 con 0x7fbc6c0047c0 2024-04-22T08:19:41.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.910+0000 7fbc85999640 10 monclient(hunting): _renew_subs 2024-04-22T08:19:41.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.910+0000 7fbc85999640 10 monclient(hunting): authenticate will time out at 1255.904724s 2024-04-22T08:19:41.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.910+0000 7fbc7f7fe640 1 -- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc80120ad0 msgr2=0x7fbc80120f50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:19:41.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.911+0000 7fbc7f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc80120ad0 0x7fbc80120f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:19:41.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.911+0000 7fbc7effd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbc6c0047c0 0x7fbc800707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:41.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.911+0000 7fbc7effd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbc6c0047c0 0x7fbc800707d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:48306/0 (socket says 172.21.15.40:48306) 2024-04-22T08:19:41.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.911+0000 7fbc7effd640 1 -- 172.21.15.40:0/360037732 learned_addr learned my addr 172.21.15.40:0/360037732 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:19:41.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.911+0000 7fbc7effd640 10 monclient(hunting): get_auth_request con 0x7fbc6c0047c0 auth_method 0 2024-04-22T08:19:41.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.911+0000 7fbc7effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:41.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.911+0000 7fbc7effd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:41.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.911+0000 7fbc7effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:41.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.911+0000 7fbc7effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:41.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.911+0000 7fbc7effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:41.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.911+0000 7fbc7effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:41.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.911+0000 7fbc767fc640 1 --2- 172.21.15.40:0/360037732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc80136c20 0x7fbc801370a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:41.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.911+0000 7fbc767fc640 10 monclient(hunting): get_auth_request con 0x7fbc80136c20 auth_method 0 2024-04-22T08:19:41.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.911+0000 7fbc767fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:41.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.911+0000 7fbc767fc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:41.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.911+0000 7fbc767fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:41.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.911+0000 7fbc7effd640 10 monclient(hunting): handle_auth_done global_id 54725 payload 824 2024-04-22T08:19:41.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.911+0000 7fbc7effd640 1 -- 172.21.15.40:0/360037732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc80136c20 msgr2=0x7fbc801370a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:41.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.911+0000 7fbc7effd640 1 --2- 172.21.15.40:0/360037732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc80136c20 0x7fbc801370a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:41.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.911+0000 7fbc7effd640 1 -- 172.21.15.40:0/360037732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc80120ad0 msgr2=0x7fbc80120f50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:19:41.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.911+0000 7fbc7effd640 1 --2- 172.21.15.40:0/360037732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc80120ad0 0x7fbc80120f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:19:41.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.911+0000 7fbc7effd640 10 monclient: _finish_hunting 0 2024-04-22T08:19:41.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.911+0000 7fbc7effd640 1 monclient: found mon.noname-c 2024-04-22T08:19:41.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.911+0000 7fbc7effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:19:41.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.911+0000 7fbc7effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:19:41.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.911+0000 7fbc7effd640 1 -- 172.21.15.40:0/360037732 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc80071320 con 0x7fbc6c0047c0 2024-04-22T08:19:41.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.911+0000 7fbc767fc640 1 --2- 172.21.15.40:0/360037732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc80136c20 0x7fbc801370a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T08:19:41.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.913+0000 7fbc7effd640 1 -- 172.21.15.40:0/360037732 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbc6c0047c0 msgr2=0x7fbc800707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:41.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.913+0000 7fbc7effd640 1 --2- 172.21.15.40:0/360037732 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbc6c0047c0 0x7fbc800707d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fbc7000c730 tx=0x7fbc7000cbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:41.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.913+0000 7fbc7cff9640 1 -- 172.21.15.40:0/360037732 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc70004e00 con 0x7fbc6c0047c0 2024-04-22T08:19:41.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.913+0000 7fbc7cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:41.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.913+0000 7fbc7cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:19:41.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.913+0000 7fbc7cff9640 10 monclient: dump: 2024-04-22T08:19:41.939 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:41.939 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:41.939 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:41.940 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:41.940 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:41.940 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:41.940 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:41.940 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:41.940 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:41.940 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:41.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.913+0000 7fbc7cff9640 10 monclient: _finish_auth 0 2024-04-22T08:19:41.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.913+0000 7fbc7cff9640 10 monclient: _check_auth_tickets 2024-04-22T08:19:41.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.913+0000 7fbc7cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:19:41.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.913+0000 7fbc85999640 5 monclient: authenticate success, global_id 54725 2024-04-22T08:19:41.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.913+0000 7fbc85999640 10 monclient: _renew_subs 2024-04-22T08:19:41.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.913+0000 7fbc7cff9640 1 -- 172.21.15.40:0/360037732 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc700070d0 con 0x7fbc6c0047c0 2024-04-22T08:19:41.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.913+0000 7fbc85999640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:19:41.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.913+0000 7fbc85999640 1 -- 172.21.15.40:0/360037732 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc80122d30 con 0x7fbc6c0047c0 2024-04-22T08:19:41.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.913+0000 7fbc7cff9640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:19:41.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.913+0000 7fbc7cff9640 1 -- 172.21.15.40:0/360037732 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc70007480 con 0x7fbc6c0047c0 2024-04-22T08:19:41.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.913+0000 7fbc7cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:41.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.913+0000 7fbc7cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:19:41.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.913+0000 7fbc7cff9640 10 monclient: dump: 2024-04-22T08:19:41.941 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:41.941 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:41.941 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:41.941 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:41.941 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:41.941 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:41.941 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:41.941 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:41.941 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:41.941 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:41.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.913+0000 7fbc7dffb640 4 set_mon_vals no callback set 2024-04-22T08:19:41.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.913+0000 7fbc85999640 10 monclient: _renew_subs 2024-04-22T08:19:41.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.913+0000 7fbc85999640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:19:41.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.913+0000 7fbc85999640 1 -- 172.21.15.40:0/360037732 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc801232f0 con 0x7fbc6c0047c0 2024-04-22T08:19:41.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.914+0000 7fbc85999640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:19:41.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.914+0000 7fbc85999640 1 --2- 172.21.15.40:0/360037732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc8002c380 0x7fbc80121490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:41.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.914+0000 7fbc85999640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:41.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.914+0000 7fbc85999640 1 -- 172.21.15.40:0/360037732 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc80066770 con 0x7fbc8002c380 2024-04-22T08:19:41.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.914+0000 7fbc7cff9640 1 -- 172.21.15.40:0/360037732 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fbc70007620 con 0x7fbc6c0047c0 2024-04-22T08:19:41.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.914+0000 7fbc7cff9640 1 --2- 172.21.15.40:0/360037732 >> v2:172.21.15.189:6800/30168 conn(0x7fbc600377b0 0x7fbc60039c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:41.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.914+0000 7fbc7cff9640 1 -- 172.21.15.40:0/360037732 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 3506+0+0 (secure 0 0 0) 0x7fbc70049670 con 0x7fbc6c0047c0 2024-04-22T08:19:41.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.914+0000 7fbc767fc640 1 --2- 172.21.15.40:0/360037732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc8002c380 0x7fbc80121490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:41.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.914+0000 7fbc767fc640 10 monclient: get_auth_request con 0x7fbc8002c380 auth_method 0 2024-04-22T08:19:41.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.914+0000 7fbc767fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:41.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.914+0000 7fbc767fc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:41.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.914+0000 7fbc767fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:41.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.915+0000 7fbc7f7fe640 1 --2- 172.21.15.40:0/360037732 >> v2:172.21.15.189:6800/30168 conn(0x7fbc600377b0 0x7fbc60039c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:41.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.915+0000 7fbc7f7fe640 10 monclient: get_auth_request con 0x7fbc600377b0 auth_method 0 2024-04-22T08:19:41.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.915+0000 7fbc767fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:41.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.915+0000 7fbc767fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:41.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.915+0000 7fbc767fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:41.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.915+0000 7fbc767fc640 10 monclient(hunting): handle_auth_done global_id 64546 payload 824 2024-04-22T08:19:41.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.915+0000 7fbc767fc640 1 -- 172.21.15.40:0/360037732 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbc8009dc10 con 0x7fbc8002c380 2024-04-22T08:19:41.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.917+0000 7fbc7f7fe640 1 --2- 172.21.15.40:0/360037732 >> v2:172.21.15.189:6800/30168 conn(0x7fbc600377b0 0x7fbc60039c70 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fbc8006a150 tx=0x7fbc78009540 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:41.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.917+0000 7fbc767fc640 1 -- 172.21.15.40:0/360037732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc8002c380 msgr2=0x7fbc80121490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:41.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.917+0000 7fbc767fc640 1 --2- 172.21.15.40:0/360037732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc8002c380 0x7fbc80121490 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fbc68033000 tx=0x7fbc68005940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:41.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.917+0000 7fbc7cff9640 1 -- 172.21.15.40:0/360037732 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc68005280 con 0x7fbc8002c380 2024-04-22T08:19:41.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.917+0000 7fbc7cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:41.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.917+0000 7fbc7cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:19:41.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.917+0000 7fbc7cff9640 10 monclient: dump: 2024-04-22T08:19:41.944 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:41.944 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:41.944 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:41.944 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:41.944 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:41.944 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:41.944 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:41.944 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:41.944 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:41.945 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:41.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.918+0000 7fbc7cff9640 1 -- 172.21.15.40:0/360037732 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7fbc680053e0 con 0x7fbc8002c380 2024-04-22T08:19:41.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.918+0000 7fbc7cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:19:41.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.918+0000 7fbc7cff9640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:19:41.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.918+0000 7fbc7cff9640 1 -- 172.21.15.40:0/360037732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc8002c380 msgr2=0x7fbc80121490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:41.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.918+0000 7fbc7cff9640 1 --2- 172.21.15.40:0/360037732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc8002c380 0x7fbc80121490 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fbc68033000 tx=0x7fbc68005940 comp rx=0 tx=0).stop 2024-04-22T08:19:41.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.918+0000 7fbc7cff9640 1 -- 172.21.15.40:0/360037732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc8002c380 msgr2=0x7fbc80121490 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:19:41.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.918+0000 7fbc7cff9640 1 --2- 172.21.15.40:0/360037732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc8002c380 0x7fbc80121490 secure :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fbc68033000 tx=0x7fbc68005940 comp rx=0 tx=0).stop 2024-04-22T08:19:41.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.924+0000 7fbc85999640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:19:41.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.924+0000 7fbc85999640 1 --2- 172.21.15.40:0/360037732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc80060ff0 0x7fbc80061400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:19:41.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.924+0000 7fbc85999640 10 monclient(hunting): start opening mon connection 2024-04-22T08:19:41.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.924+0000 7fbc85999640 1 -- 172.21.15.40:0/360037732 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8002c760 con 0x7fbc80060ff0 2024-04-22T08:19:41.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.924+0000 7fbc767fc640 1 --2- 172.21.15.40:0/360037732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc80060ff0 0x7fbc80061400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:19:41.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.924+0000 7fbc767fc640 10 monclient: get_auth_request con 0x7fbc80060ff0 auth_method 0 2024-04-22T08:19:41.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.924+0000 7fbc767fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:19:41.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.924+0000 7fbc767fc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:19:41.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.924+0000 7fbc767fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:19:41.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.924+0000 7fbc767fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:19:41.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.924+0000 7fbc767fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:19:41.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.924+0000 7fbc767fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:19:41.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.924+0000 7fbc767fc640 10 monclient(hunting): handle_auth_done global_id 64549 payload 824 2024-04-22T08:19:41.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.924+0000 7fbc767fc640 1 -- 172.21.15.40:0/360037732 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbc8004e9b0 con 0x7fbc80060ff0 2024-04-22T08:19:41.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.927+0000 7fbc767fc640 1 -- 172.21.15.40:0/360037732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc80060ff0 msgr2=0x7fbc80061400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:19:41.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.927+0000 7fbc767fc640 1 --2- 172.21.15.40:0/360037732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc80060ff0 0x7fbc80061400 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fbc68034000 tx=0x7fbc6802e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:19:41.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.927+0000 7fbc7cff9640 1 -- 172.21.15.40:0/360037732 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6803f810 con 0x7fbc80060ff0 2024-04-22T08:19:41.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.927+0000 7fbc7cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:19:41.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.927+0000 7fbc7cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:19:41.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.927+0000 7fbc7cff9640 10 monclient: dump: 2024-04-22T08:19:41.948 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:19:41.948 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:19:41.948 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:41.948 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:19:41.948 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:19:41.948 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:19:41.949 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:19:41.949 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:19:41.949 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:19:41.949 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:19:41.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.927+0000 7fbc7cff9640 1 -- 172.21.15.40:0/360037732 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4792 (secure 0 0 0) 0x7fbc6804d040 con 0x7fbc80060ff0 2024-04-22T08:19:41.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.927+0000 7fbc7cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:19:41.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.927+0000 7fbc7cff9640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:19:41.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.927+0000 7fbc7cff9640 1 -- 172.21.15.40:0/360037732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc80060ff0 msgr2=0x7fbc80061400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:41.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.927+0000 7fbc7cff9640 1 --2- 172.21.15.40:0/360037732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc80060ff0 0x7fbc80061400 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fbc68034000 tx=0x7fbc6802e000 comp rx=0 tx=0).stop 2024-04-22T08:19:41.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.927+0000 7fbc7cff9640 1 -- 172.21.15.40:0/360037732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc80060ff0 msgr2=0x7fbc80061400 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:19:41.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.927+0000 7fbc7cff9640 1 --2- 172.21.15.40:0/360037732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc80060ff0 0x7fbc80061400 secure :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fbc68034000 tx=0x7fbc6802e000 comp rx=0 tx=0).stop 2024-04-22T08:19:41.950 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:19:41.950 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:19:41.950 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:19:41.950 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "leader", 2024-04-22T08:19:41.950 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 68, 2024-04-22T08:19:41.950 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:19:41.950 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:19:41.950 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:19:41.950 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:19:41.950 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 0, 2024-04-22T08:19:41.950 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:19:41.950 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:19:41.950 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:19:41.951 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:19:41.951 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:19:41.951 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:19:41.951 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:19:41.951 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:19:41.951 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:19:41.951 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:19:41.951 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:19:41.951 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:19:41.951 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:19:41.951 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:19:41.951 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:19:41.951 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:19:41.951 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:19:41.951 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:19:41.952 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:19:41.952 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:19:41.952 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:19:41.952 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:19:41.952 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:19:41.952 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:19:41.952 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:19:41.952 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:19:41.952 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:41.952 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:41.952 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:19:41.952 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:19:41.952 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:19:41.952 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:19:41.953 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:19:41.953 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:19:41.953 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:19:41.953 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:19:41.953 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:19:41.953 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:19:41.953 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:19:41.953 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:19:41.953 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:19:41.953 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:19:41.953 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:19:41.953 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:19:41.954 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:19:41.954 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:19:41.954 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:19:41.954 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:19:41.954 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:19:41.954 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:19:41.954 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:19:41.954 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:19:41.954 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:19:41.954 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:19:41.954 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:19:41.954 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:19:41.954 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:19:41.954 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:41.954 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:19:41.955 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:41.955 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:19:41.955 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:19:41.955 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:19:41.955 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:19:41.955 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:41.955 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:19:41.955 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:19:41.955 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:41.955 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:41.955 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:41.955 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:19:41.955 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:19:41.955 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:41.956 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:41.956 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:41.956 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:41.956 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:19:41.956 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:19:41.956 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:19:41.956 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:19:41.956 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:19:41.956 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:41.956 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:41.956 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:19:41.956 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:19:41.956 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:19:41.956 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:19:41.956 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:41.957 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:19:41.957 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:19:41.957 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:41.957 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:41.957 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:41.957 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:19:41.957 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:19:41.957 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:41.957 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:41.957 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:41.957 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:41.957 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:19:41.957 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:19:41.957 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:19:41.957 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:19:41.958 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:19:41.958 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:41.958 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:41.958 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:19:41.958 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:19:41.958 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:19:41.958 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:19:41.958 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:41.958 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:19:41.958 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:19:41.959 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:41.959 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:41.959 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:41.959 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:19:41.959 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:19:41.959 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:19:41.959 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:41.959 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:41.959 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:41.960 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:19:41.960 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:19:41.960 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:19:41.960 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:19:41.960 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:19:41.960 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:41.960 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:41.960 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:41.960 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:19:41.960 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:19:41.961 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:41.961 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:19:41.961 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:19:41.961 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:19:41.961 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:41.961 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:19:41.961 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:19:41.961 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:41.961 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:19:41.961 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:19:41.961 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 3 2024-04-22T08:19:41.961 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:41.961 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:19:41.961 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:19:41.961 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:41.962 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:19:41.962 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:19:41.962 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 3 2024-04-22T08:19:41.962 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:41.962 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:19:41.962 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": [ 2024-04-22T08:19:41.962 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:19:41.962 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:19:41.962 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:19:41.962 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:19:41.962 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:19:41.962 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:19:41.962 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:19:41.962 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:19:41.963 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:19:41.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.928+0000 7fbc85999640 1 -- 172.21.15.40:0/360037732 >> v2:172.21.15.189:6800/30168 conn(0x7fbc600377b0 msgr2=0x7fbc60039c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:41.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.928+0000 7fbc85999640 1 --2- 172.21.15.40:0/360037732 >> v2:172.21.15.189:6800/30168 conn(0x7fbc600377b0 0x7fbc60039c70 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fbc8006a150 tx=0x7fbc78009540 comp rx=0 tx=0).stop 2024-04-22T08:19:41.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.928+0000 7fbc85999640 10 monclient: shutdown 2024-04-22T08:19:41.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.928+0000 7fbc85999640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:19:41.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.928+0000 7fbc85999640 1 -- 172.21.15.40:0/360037732 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbc6c0047c0 msgr2=0x7fbc800707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:19:41.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.928+0000 7fbc85999640 1 --2- 172.21.15.40:0/360037732 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbc6c0047c0 0x7fbc800707d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fbc7000c730 tx=0x7fbc7000cbf0 comp rx=0 tx=0).stop 2024-04-22T08:19:41.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.928+0000 7fbc7f7fe640 1 -- 172.21.15.40:0/360037732 reap_dead start 2024-04-22T08:19:41.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.928+0000 7fbc7f7fe640 1 -- 172.21.15.40:0/360037732 reap_dead start 2024-04-22T08:19:41.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.928+0000 7fbc85999640 1 -- 172.21.15.40:0/360037732 shutdown_connections 2024-04-22T08:19:41.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.928+0000 7fbc85999640 1 -- 172.21.15.40:0/360037732 >> 172.21.15.40:0/360037732 conn(0x7fbc800a66a0 msgr2=0x7fbc80137e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:19:41.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.928+0000 7fbc85999640 1 -- 172.21.15.40:0/360037732 shutdown_connections 2024-04-22T08:19:41.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:19:41.928+0000 7fbc85999640 1 -- 172.21.15.40:0/360037732 wait complete. 2024-04-22T08:19:41.965 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-22T08:19:41.965 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-04-22T08:19:43.702 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:19:43.703 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37877]: starting. 2024-04-22T08:19:43.703 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37877]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-37842 2024-04-22T08:19:43.703 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37877]: created object 0... 2024-04-22T08:19:43.703 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37877]: created object 25... 2024-04-22T08:19:43.703 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37877]: created object 49... 2024-04-22T08:19:43.703 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37877]: finishing. 2024-04-22T08:19:43.703 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37877]: shutting down. 2024-04-22T08:19:43.743 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:19:43.743 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37879]: starting. 2024-04-22T08:19:43.743 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37879]: listing objects. 2024-04-22T08:19:43.743 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37879]: listed object 0... 2024-04-22T08:19:43.743 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37879]: listed object 25... 2024-04-22T08:19:43.744 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37879]: saw 50 objects 2024-04-22T08:19:43.744 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37879]: shutting down. 2024-04-22T08:19:43.954 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:19:43.954 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[37878]: starting. 2024-04-22T08:19:43.954 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[37878]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-37842 2024-04-22T08:19:43.955 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[37878]: shutting down. 2024-04-22T08:19:43.956 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:19:43.956 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:19:43.956 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:19:43.958 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:19:46.072 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37948]: starting. 2024-04-22T08:19:46.072 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37948]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-37947 2024-04-22T08:19:46.072 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37948]: created object 0... 2024-04-22T08:19:46.072 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37948]: created object 25... 2024-04-22T08:19:46.072 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37948]: created object 49... 2024-04-22T08:19:46.072 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37948]: finishing. 2024-04-22T08:19:46.072 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[37948]: shutting down. 2024-04-22T08:19:46.973 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[37949]: starting. 2024-04-22T08:19:46.974 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[37949]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-37947 2024-04-22T08:19:46.974 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[37949]: shutting down. 2024-04-22T08:19:49.093 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:19:49.094 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37984]: starting. 2024-04-22T08:19:49.094 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37984]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-37947 2024-04-22T08:19:49.094 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37984]: created object 0... 2024-04-22T08:19:49.094 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37984]: created object 25... 2024-04-22T08:19:49.094 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37984]: created object 49... 2024-04-22T08:19:49.094 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37984]: finishing. 2024-04-22T08:19:49.094 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[37984]: shutting down. 2024-04-22T08:19:49.133 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:19:49.133 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37986]: starting. 2024-04-22T08:19:49.134 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37986]: listing objects. 2024-04-22T08:19:49.134 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37986]: listed object 0... 2024-04-22T08:19:49.134 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37986]: listed object 25... 2024-04-22T08:19:49.134 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37986]: saw 50 objects 2024-04-22T08:19:49.134 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[37986]: shutting down. 2024-04-22T08:19:49.983 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:19:49.983 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[37985]: starting. 2024-04-22T08:19:49.984 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[37985]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-37947 2024-04-22T08:19:49.984 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[37985]: shutting down. 2024-04-22T08:19:49.985 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:19:49.986 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:19:49.986 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:19:49.988 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:19:52.114 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38055]: starting. 2024-04-22T08:19:52.115 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38055]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-38054 2024-04-22T08:19:52.115 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38055]: created object 0... 2024-04-22T08:19:52.115 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38055]: created object 25... 2024-04-22T08:19:52.115 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38055]: created object 49... 2024-04-22T08:19:52.115 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38055]: finishing. 2024-04-22T08:19:52.115 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38055]: shutting down. 2024-04-22T08:19:53.169 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[38056]: starting. 2024-04-22T08:19:53.169 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[38056]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-38054 2024-04-22T08:19:53.169 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[38056]: shutting down. 2024-04-22T08:19:55.073 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:19:55.074 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38091]: starting. 2024-04-22T08:19:55.074 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38091]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-38054 2024-04-22T08:19:55.074 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38091]: created object 0... 2024-04-22T08:19:55.074 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38091]: created object 25... 2024-04-22T08:19:55.074 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38091]: created object 49... 2024-04-22T08:19:55.074 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38091]: finishing. 2024-04-22T08:19:55.074 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38091]: shutting down. 2024-04-22T08:19:55.119 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:19:55.119 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38093]: starting. 2024-04-22T08:19:55.119 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38093]: listing objects. 2024-04-22T08:19:55.119 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38093]: listed object 0... 2024-04-22T08:19:55.119 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38093]: listed object 25... 2024-04-22T08:19:55.119 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38093]: saw 50 objects 2024-04-22T08:19:55.119 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38093]: shutting down. 2024-04-22T08:19:55.966 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:19:55.978 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[38092]: starting. 2024-04-22T08:19:55.978 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[38092]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-38054 2024-04-22T08:19:55.978 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[38092]: shutting down. 2024-04-22T08:19:55.979 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:19:55.979 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:19:55.979 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:19:55.980 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:19:58.101 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38162]: starting. 2024-04-22T08:19:58.101 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38162]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-38161 2024-04-22T08:19:58.101 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38162]: created object 0... 2024-04-22T08:19:58.102 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38162]: created object 25... 2024-04-22T08:19:58.102 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38162]: created object 49... 2024-04-22T08:19:58.102 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38162]: finishing. 2024-04-22T08:19:58.102 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38162]: shutting down. 2024-04-22T08:19:58.990 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[38163]: starting. 2024-04-22T08:19:58.990 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[38163]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-38161 2024-04-22T08:19:58.991 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[38163]: shutting down. 2024-04-22T08:20:01.127 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:20:01.127 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38198]: starting. 2024-04-22T08:20:01.127 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38198]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-38161 2024-04-22T08:20:01.127 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38198]: created object 0... 2024-04-22T08:20:01.127 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38198]: created object 25... 2024-04-22T08:20:01.128 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38198]: created object 49... 2024-04-22T08:20:01.128 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38198]: finishing. 2024-04-22T08:20:01.128 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38198]: shutting down. 2024-04-22T08:20:01.169 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:20:01.169 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38200]: starting. 2024-04-22T08:20:01.169 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38200]: listing objects. 2024-04-22T08:20:01.169 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38200]: listed object 0... 2024-04-22T08:20:01.169 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38200]: listed object 25... 2024-04-22T08:20:01.169 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38200]: saw 50 objects 2024-04-22T08:20:01.169 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38200]: shutting down. 2024-04-22T08:20:01.965 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-04-22T08:20:01.966 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-04-22T08:20:01.966 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-22T08:20:01.966 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-22T08:20:01.969 INFO:tasks.ceph.mon.a:Started 2024-04-22T08:20:01.969 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-22T08:20:01.970 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:20:02.017 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:20:02.017 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[38199]: starting. 2024-04-22T08:20:02.017 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[38199]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-38161 2024-04-22T08:20:02.017 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[38199]: shutting down. 2024-04-22T08:20:02.018 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:20:02.018 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:20:02.018 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:20:02.021 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:20:02.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.090+0000 7f6e84f99640 10 monclient: get_monmap_and_config 2024-04-22T08:20:02.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.090+0000 7f6e84f99640 10 monclient: build_initial_monmap 2024-04-22T08:20:02.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.090+0000 7f6e84f99640 10 monclient: monmap: 2024-04-22T08:20:02.091 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:20:02.091 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:02.091 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:20:02.092490+0000 2024-04-22T08:20:02.091 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:20:02.092490+0000 2024-04-22T08:20:02.091 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:20:02.091 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:02.091 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:20:02.092 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:20:02.092 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:20:02.092 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:20:02.092 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:20:02.092 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:02.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.091+0000 7f6e84f99640 1 Processor -- start 2024-04-22T08:20:02.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.091+0000 7f6e84f99640 1 -- start start 2024-04-22T08:20:02.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.091+0000 7f6e84f99640 10 monclient: init 2024-04-22T08:20:02.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.091+0000 7f6e84f99640 10 monclient: _reopen_session rank -1 2024-04-22T08:20:02.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.091+0000 7f6e84f99640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-04-22T08:20:02.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.091+0000 7f6e84f99640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6e8009ee10 addr v1:172.21.15.40:6789/0 2024-04-22T08:20:02.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.091+0000 7f6e84f99640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6e800a32b0 0x7f6e8013b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:02.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.091+0000 7f6e84f99640 10 monclient(hunting): picked mon.noname-c con 0x7f6e800a32b0 addr v2:172.21.15.189:3300/0 2024-04-22T08:20:02.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.091+0000 7f6e84f99640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6e8013b5f0 addr v1:172.21.15.189:6789/0 2024-04-22T08:20:02.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.091+0000 7f6e84f99640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e8009f1f0 con 0x7f6e8009ee10 2024-04-22T08:20:02.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.092+0000 7f6e84f99640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e800a39c0 con 0x7f6e8013b5f0 2024-04-22T08:20:02.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.092+0000 7f6e84f99640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:02.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.092+0000 7f6e84f99640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e800a3700 con 0x7f6e800a32b0 2024-04-22T08:20:02.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.092+0000 7f6e84f99640 10 monclient(hunting): _renew_subs 2024-04-22T08:20:02.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.092+0000 7f6e84f99640 10 monclient(hunting): authenticate will time out at 1276.086791s 2024-04-22T08:20:02.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.092+0000 7f6e7e59c640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f6e8009ee10 0x7f6e800a1d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53288/0 (socket says 172.21.15.40:53288) 2024-04-22T08:20:02.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.092+0000 7f6e7e59c640 1 -- 172.21.15.40:0/1089554169 learned_addr learned my addr 172.21.15.40:0/1089554169 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:20:02.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.092+0000 7f6e7dd9b640 1 --2- 172.21.15.40:0/1089554169 >> v2:172.21.15.189:3300/0 conn(0x7f6e800a32b0 0x7f6e8013b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:02.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.092+0000 7f6e7dd9b640 10 monclient(hunting): get_auth_request con 0x7f6e800a32b0 auth_method 0 2024-04-22T08:20:02.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.092+0000 7f6e7dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:02.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.092+0000 7f6e7dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:02.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.092+0000 7f6e7dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:02.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.092+0000 7f6e7dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:02.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.092+0000 7f6e7dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:02.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.092+0000 7f6e7dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:02.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.092+0000 7f6e7dd9b640 10 monclient(hunting): handle_auth_done global_id 64630 payload 274 2024-04-22T08:20:02.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.092+0000 7f6e7dd9b640 1 -- 172.21.15.40:0/1089554169 >> v1:172.21.15.189:6789/0 conn(0x7f6e8013b5f0 legacy=0x7f6e8013d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:20:02.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.092+0000 7f6e7dd9b640 1 -- 172.21.15.40:0/1089554169 >> v1:172.21.15.40:6789/0 conn(0x7f6e8009ee10 legacy=0x7f6e800a1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:20:02.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.092+0000 7f6e7dd9b640 10 monclient: _finish_hunting 0 2024-04-22T08:20:02.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.092+0000 7f6e7dd9b640 1 monclient: found mon.noname-c 2024-04-22T08:20:02.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.092+0000 7f6e7dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:20:02.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.092+0000 7f6e7dd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.189:3300/0 2024-04-22T08:20:02.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.092+0000 7f6e7dd9b640 1 -- 172.21.15.40:0/1089554169 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e8009fd30 con 0x7f6e800a32b0 2024-04-22T08:20:02.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.095+0000 7f6e7dd9b640 1 -- 172.21.15.40:0/1089554169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e800a32b0 msgr2=0x7f6e8013b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:02.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.095+0000 7f6e7dd9b640 1 --2- 172.21.15.40:0/1089554169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e800a32b0 0x7f6e8013b0b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6e80065d80 tx=0x7f6e7402b6e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6cb1746339dc055a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:02.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.097+0000 7f6e7d59a640 1 -- 172.21.15.40:0/1089554169 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e74030a00 con 0x7f6e800a32b0 2024-04-22T08:20:02.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.097+0000 7f6e7d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:02.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.097+0000 7f6e7d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:20:02.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.097+0000 7f6e7d59a640 10 monclient: dump: 2024-04-22T08:20:02.098 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:02.098 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:02.098 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:02.098 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:02.098 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:02.098 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:02.099 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:02.099 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:02.099 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:02.099 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:02.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.097+0000 7f6e7d59a640 10 monclient: _finish_auth 0 2024-04-22T08:20:02.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.097+0000 7f6e7d59a640 10 monclient: _check_auth_tickets 2024-04-22T08:20:02.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.097+0000 7f6e7d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:20:02.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.097+0000 7f6e7d59a640 1 -- 172.21.15.40:0/1089554169 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e7402e050 con 0x7f6e800a32b0 2024-04-22T08:20:02.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.097+0000 7f6e7d59a640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:20:02.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.097+0000 7f6e7d59a640 1 -- 172.21.15.40:0/1089554169 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e74030d80 con 0x7f6e800a32b0 2024-04-22T08:20:02.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.097+0000 7f6e84f99640 5 monclient: authenticate success, global_id 64630 2024-04-22T08:20:02.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.097+0000 7f6e84f99640 10 monclient: get_monmap_and_config success 2024-04-22T08:20:02.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.097+0000 7f6e84f99640 4 set_mon_vals no callback set 2024-04-22T08:20:02.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.097+0000 7f6e7d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:02.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.097+0000 7f6e7d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:20:02.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.097+0000 7f6e7d59a640 10 monclient: dump: 2024-04-22T08:20:02.100 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:02.100 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:02.100 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:02.100 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:02.100 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:02.100 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:02.100 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:02.100 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:02.100 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:02.100 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:02.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.097+0000 7f6e84f99640 10 monclient: shutdown 2024-04-22T08:20:02.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.097+0000 7f6e84f99640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:20:02.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.097+0000 7f6e84f99640 1 -- 172.21.15.40:0/1089554169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e800a32b0 msgr2=0x7f6e8013b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:02.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.097+0000 7f6e84f99640 1 --2- 172.21.15.40:0/1089554169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e800a32b0 0x7f6e8013b0b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6e80065d80 tx=0x7f6e7402b6e0 comp rx=0 tx=0).stop 2024-04-22T08:20:02.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.097+0000 7f6e84f99640 1 -- 172.21.15.40:0/1089554169 shutdown_connections 2024-04-22T08:20:02.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.097+0000 7f6e84f99640 1 --2- 172.21.15.40:0/1089554169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e800a32b0 0x7f6e8013b0b0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:02.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.097+0000 7f6e84f99640 1 -- 172.21.15.40:0/1089554169 >> 172.21.15.40:0/1089554169 conn(0x7f6e800a4ef0 msgr2=0x7f6e800a5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:20:02.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.097+0000 7f6e84f99640 1 -- 172.21.15.40:0/1089554169 shutdown_connections 2024-04-22T08:20:02.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.097+0000 7f6e84f99640 1 -- 172.21.15.40:0/1089554169 wait complete. 2024-04-22T08:20:02.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.098+0000 7f6e84f99640 10 monclient: build_initial_monmap 2024-04-22T08:20:02.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.098+0000 7f6e84f99640 10 monclient: monmap: 2024-04-22T08:20:02.109 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:20:02.109 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:20:02.110 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:20:02.110 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:20:02.110 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:20:02.110 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:02.110 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:20:02.110 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:20:02.110 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:20:02.110 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:02.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.098+0000 7f6e84f99640 1 Processor -- start 2024-04-22T08:20:02.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.098+0000 7f6e84f99640 1 -- start start 2024-04-22T08:20:02.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.098+0000 7f6e84f99640 10 monclient: init 2024-04-22T08:20:02.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e84f99640 10 monclient: _reopen_session rank -1 2024-04-22T08:20:02.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e84f99640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-22T08:20:02.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e84f99640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e8009ee10 0x7f6e8006e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:02.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e84f99640 10 monclient(hunting): picked mon.noname-c con 0x7f6e8009ee10 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:20:02.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e84f99640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e800a32b0 0x7f6e8006ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:02.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e84f99640 10 monclient(hunting): picked mon.noname-b con 0x7f6e800a32b0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:20:02.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e84f99640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e8013b5f0 0x7f6e8006f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:02.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e84f99640 10 monclient(hunting): picked mon.noname-a con 0x7f6e8013b5f0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:20:02.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e84f99640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:02.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e84f99640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e80135be0 con 0x7f6e8013b5f0 2024-04-22T08:20:02.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e84f99640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:02.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e84f99640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8011ebe0 con 0x7f6e800a32b0 2024-04-22T08:20:02.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e84f99640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:02.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e84f99640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e80071aa0 con 0x7f6e8009ee10 2024-04-22T08:20:02.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e84f99640 10 monclient(hunting): _renew_subs 2024-04-22T08:20:02.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e84f99640 10 monclient(hunting): authenticate will time out at 1276.094173s 2024-04-22T08:20:02.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e7e59c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e8009ee10 0x7f6e8006e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:02.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e7ed9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e8013b5f0 0x7f6e8006f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:02.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e7e59c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e8009ee10 0x7f6e8006e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:56518/0 (socket says 172.21.15.40:56518) 2024-04-22T08:20:02.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e7e59c640 1 -- 172.21.15.40:0/2492329502 learned_addr learned my addr 172.21.15.40:0/2492329502 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:20:02.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e7e59c640 10 monclient(hunting): get_auth_request con 0x7f6e8009ee10 auth_method 0 2024-04-22T08:20:02.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e7e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:02.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e7e59c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:02.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e7e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:02.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e7ed9d640 10 monclient(hunting): get_auth_request con 0x7f6e8013b5f0 auth_method 0 2024-04-22T08:20:02.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e7ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:02.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e7ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:02.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e7ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:02.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e7dd9b640 1 --2- 172.21.15.40:0/2492329502 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e800a32b0 0x7f6e8006ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:02.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e7dd9b640 10 monclient(hunting): get_auth_request con 0x7f6e800a32b0 auth_method 0 2024-04-22T08:20:02.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e7dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:02.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e7dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:02.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e7dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:02.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e7e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:02.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e7e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:02.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e7ed9d640 1 -- 172.21.15.40:0/2492329502 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e8013b5f0 msgr2=0x7f6e8006f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-22T08:20:02.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e7ed9d640 1 -- 172.21.15.40:0/2492329502 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e8013b5f0 msgr2=0x7f6e8006f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:20:02.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e7ed9d640 1 --2- 172.21.15.40:0/2492329502 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e8013b5f0 0x7f6e8006f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:20:02.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e7e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:02.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e7ed9d640 1 --2- 172.21.15.40:0/2492329502 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e8013b5f0 0x7f6e8006f180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:20:02.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e7dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:02.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e7dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:02.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e7dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:02.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e7e59c640 10 monclient(hunting): handle_auth_done global_id 54800 payload 824 2024-04-22T08:20:02.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e7e59c640 1 -- 172.21.15.40:0/2492329502 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e800a32b0 msgr2=0x7f6e8006ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:02.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e7e59c640 1 --2- 172.21.15.40:0/2492329502 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e800a32b0 0x7f6e8006ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:02.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e7e59c640 1 -- 172.21.15.40:0/2492329502 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e8013b5f0 msgr2=0x7f6e8006f180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:20:02.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e7e59c640 1 --2- 172.21.15.40:0/2492329502 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e8013b5f0 0x7f6e8006f180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:02.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e7e59c640 10 monclient: _finish_hunting 0 2024-04-22T08:20:02.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e7e59c640 1 monclient: found mon.noname-c 2024-04-22T08:20:02.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e7e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:20:02.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e7e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:20:02.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e7e59c640 1 -- 172.21.15.40:0/2492329502 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e74009400 con 0x7f6e8009ee10 2024-04-22T08:20:02.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.099+0000 7f6e7dd9b640 1 --2- 172.21.15.40:0/2492329502 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e800a32b0 0x7f6e8006ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:20:02.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.102+0000 7f6e7e59c640 1 -- 172.21.15.40:0/2492329502 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e8009ee10 msgr2=0x7f6e8006e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:02.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.102+0000 7f6e7e59c640 1 --2- 172.21.15.40:0/2492329502 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e8009ee10 0x7f6e8006e700 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6e7000b460 tx=0x7f6e7000b920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:02.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.102+0000 7f6e6effd640 1 -- 172.21.15.40:0/2492329502 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e70011aa0 con 0x7f6e8009ee10 2024-04-22T08:20:02.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.102+0000 7f6e6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:02.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.102+0000 7f6e6effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:20:02.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.102+0000 7f6e6effd640 10 monclient: dump: 2024-04-22T08:20:02.115 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:02.115 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:02.115 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:02.115 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:02.115 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:02.115 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:02.115 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:02.115 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:02.115 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:02.115 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:02.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.102+0000 7f6e6effd640 10 monclient: _finish_auth 0 2024-04-22T08:20:02.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.102+0000 7f6e6effd640 10 monclient: _check_auth_tickets 2024-04-22T08:20:02.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.102+0000 7f6e6effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:20:02.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.102+0000 7f6e6effd640 1 -- 172.21.15.40:0/2492329502 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e70011c20 con 0x7f6e8009ee10 2024-04-22T08:20:02.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.102+0000 7f6e84f99640 5 monclient: authenticate success, global_id 54800 2024-04-22T08:20:02.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.102+0000 7f6e84f99640 10 monclient: _renew_subs 2024-04-22T08:20:02.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.102+0000 7f6e84f99640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:20:02.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.102+0000 7f6e84f99640 1 -- 172.21.15.40:0/2492329502 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e800a2730 con 0x7f6e8009ee10 2024-04-22T08:20:02.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.102+0000 7f6e6effd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:20:02.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.102+0000 7f6e6effd640 1 -- 172.21.15.40:0/2492329502 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e700103e0 con 0x7f6e8009ee10 2024-04-22T08:20:02.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.102+0000 7f6e6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:02.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.102+0000 7f6e6effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:20:02.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.102+0000 7f6e6effd640 10 monclient: dump: 2024-04-22T08:20:02.116 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:02.116 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:02.116 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:02.116 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:02.116 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:02.117 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:02.117 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:02.117 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:02.117 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:02.117 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:02.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.102+0000 7f6e84f99640 10 monclient: _renew_subs 2024-04-22T08:20:02.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.102+0000 7f6e84f99640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:20:02.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.102+0000 7f6e84f99640 1 -- 172.21.15.40:0/2492329502 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e800a7cd0 con 0x7f6e8009ee10 2024-04-22T08:20:02.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.102+0000 7f6e7cd99640 4 set_mon_vals no callback set 2024-04-22T08:20:02.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.103+0000 7f6e6effd640 1 -- 172.21.15.40:0/2492329502 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f6e70010580 con 0x7f6e8009ee10 2024-04-22T08:20:02.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.103+0000 7f6e84f99640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:20:02.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.103+0000 7f6e84f99640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:20:02.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.103+0000 7f6e84f99640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:20:02.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.103+0000 7f6e84f99640 1 -- 172.21.15.40:0/2492329502 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e80070030 con 0x7f6e8009ee10 2024-04-22T08:20:02.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.103+0000 7f6e6effd640 1 --2- 172.21.15.40:0/2492329502 >> v2:172.21.15.189:6800/30168 conn(0x7f6e600377d0 0x7f6e60039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:02.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.103+0000 7f6e6effd640 1 -- 172.21.15.40:0/2492329502 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f6e7004a5f0 con 0x7f6e8009ee10 2024-04-22T08:20:02.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.103+0000 7f6e7dd9b640 1 --2- 172.21.15.40:0/2492329502 >> v2:172.21.15.189:6800/30168 conn(0x7f6e600377d0 0x7f6e60039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:02.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.103+0000 7f6e7dd9b640 10 monclient: get_auth_request con 0x7f6e600377d0 auth_method 0 2024-04-22T08:20:02.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.106+0000 7f6e7dd9b640 1 --2- 172.21.15.40:0/2492329502 >> v2:172.21.15.189:6800/30168 conn(0x7f6e600377d0 0x7f6e60039c90 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f6e74032000 tx=0x7f6e74008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:02.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.108+0000 7f6e6effd640 1 -- 172.21.15.40:0/2492329502 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f6e80070030 con 0x7f6e8009ee10 2024-04-22T08:20:02.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.108+0000 7f6e6effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:20:02.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.108+0000 7f6e6effd640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:20:02.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.365+0000 7f6e84f99640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:20:02.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.365+0000 7f6e84f99640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:20:02.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.365+0000 7f6e84f99640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:20:02.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.365+0000 7f6e84f99640 1 -- 172.21.15.40:0/2492329502 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6e8011ea40 con 0x7f6e8009ee10 2024-04-22T08:20:02.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.365+0000 7f6e6effd640 1 -- 172.21.15.40:0/2492329502 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7f6e8011ea40 con 0x7f6e8009ee10 2024-04-22T08:20:02.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.365+0000 7f6e6effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:20:02.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.365+0000 7f6e6effd640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:20:02.368 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":68,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:20:02.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.367+0000 7f6e84f99640 1 -- 172.21.15.40:0/2492329502 >> v2:172.21.15.189:6800/30168 conn(0x7f6e600377d0 msgr2=0x7f6e60039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:02.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.367+0000 7f6e84f99640 1 --2- 172.21.15.40:0/2492329502 >> v2:172.21.15.189:6800/30168 conn(0x7f6e600377d0 0x7f6e60039c90 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f6e74032000 tx=0x7f6e74008030 comp rx=0 tx=0).stop 2024-04-22T08:20:02.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.367+0000 7f6e84f99640 10 monclient: shutdown 2024-04-22T08:20:02.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.367+0000 7f6e84f99640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:20:02.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.367+0000 7f6e84f99640 1 -- 172.21.15.40:0/2492329502 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e8009ee10 msgr2=0x7f6e8006e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:02.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.367+0000 7f6e84f99640 1 --2- 172.21.15.40:0/2492329502 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e8009ee10 0x7f6e8006e700 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6e7000b460 tx=0x7f6e7000b920 comp rx=0 tx=0).stop 2024-04-22T08:20:02.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.368+0000 7f6e84f99640 1 -- 172.21.15.40:0/2492329502 shutdown_connections 2024-04-22T08:20:02.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.368+0000 7f6e84f99640 1 --2- 172.21.15.40:0/2492329502 >> v2:172.21.15.189:6800/30168 conn(0x7f6e600377d0 0x7f6e60039c90 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:02.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.368+0000 7f6e84f99640 1 --2- 172.21.15.40:0/2492329502 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e800a32b0 0x7f6e8006ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:02.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.368+0000 7f6e84f99640 1 --2- 172.21.15.40:0/2492329502 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e8013b5f0 0x7f6e8006f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:02.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.368+0000 7f6e84f99640 1 --2- 172.21.15.40:0/2492329502 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e8009ee10 0x7f6e8006e700 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:02.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.368+0000 7f6e84f99640 1 -- 172.21.15.40:0/2492329502 >> 172.21.15.40:0/2492329502 conn(0x7f6e800a4ef0 msgr2=0x7f6e80136dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:20:02.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.368+0000 7f6e84f99640 1 -- 172.21.15.40:0/2492329502 shutdown_connections 2024-04-22T08:20:02.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:02.368+0000 7f6e84f99640 1 -- 172.21.15.40:0/2492329502 wait complete. 2024-04-22T08:20:04.075 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38270]: starting. 2024-04-22T08:20:04.075 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38270]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-38268 2024-04-22T08:20:04.075 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38270]: created object 0... 2024-04-22T08:20:04.075 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38270]: created object 25... 2024-04-22T08:20:04.075 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38270]: created object 49... 2024-04-22T08:20:04.075 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38270]: finishing. 2024-04-22T08:20:04.075 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38270]: shutting down. 2024-04-22T08:20:04.962 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[38271]: starting. 2024-04-22T08:20:04.962 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[38271]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-38268 2024-04-22T08:20:04.962 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[38271]: shutting down. 2024-04-22T08:20:05.379 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:20:05.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:05.490+0000 7f9af5722640 10 monclient: get_monmap_and_config 2024-04-22T08:20:05.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:05.490+0000 7f9af5722640 10 monclient: build_initial_monmap 2024-04-22T08:20:05.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:05.490+0000 7f9af5722640 10 monclient: monmap: 2024-04-22T08:20:05.491 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:20:05.491 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:05.491 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:20:05.492620+0000 2024-04-22T08:20:05.491 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:20:05.492620+0000 2024-04-22T08:20:05.491 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:20:05.492 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:05.492 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:20:05.492 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:20:05.492 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:20:05.492 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:20:05.492 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:20:05.492 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:05.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:05.491+0000 7f9af5722640 1 Processor -- start 2024-04-22T08:20:05.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:05.491+0000 7f9af5722640 1 -- start start 2024-04-22T08:20:05.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:05.491+0000 7f9af5722640 10 monclient: init 2024-04-22T08:20:05.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:05.491+0000 7f9af5722640 10 monclient: _reopen_session rank -1 2024-04-22T08:20:05.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:05.491+0000 7f9af5722640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-04-22T08:20:05.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:05.491+0000 7f9af5722640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9af0135e30 0x7f9af0136210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:05.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:05.491+0000 7f9af5722640 10 monclient(hunting): picked mon.noname-a con 0x7f9af0135e30 addr v2:172.21.15.40:3300/0 2024-04-22T08:20:05.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:05.491+0000 7f9af5722640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9af0071c90 addr v1:172.21.15.189:6789/0 2024-04-22T08:20:05.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:05.491+0000 7f9af5722640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9af0136750 addr v1:172.21.15.40:6789/0 2024-04-22T08:20:05.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:05.492+0000 7f9af5722640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9af00a27c0 con 0x7f9af0136750 2024-04-22T08:20:05.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:05.492+0000 7f9af5722640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9af009e310 con 0x7f9af0071c90 2024-04-22T08:20:05.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:05.492+0000 7f9af5722640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:05.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:05.492+0000 7f9af5722640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9af009de80 con 0x7f9af0135e30 2024-04-22T08:20:05.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:05.492+0000 7f9af5722640 10 monclient(hunting): _renew_subs 2024-04-22T08:20:05.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:05.492+0000 7f9af5722640 10 monclient(hunting): authenticate will time out at 1279.486809s 2024-04-22T08:20:05.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:05.492+0000 7f9aeeffd640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9af0135e30 0x7f9af0136210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:05.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:05.492+0000 7f9aef7fe640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f9af0136750 0x7f9af00a20b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53292/0 (socket says 172.21.15.40:53292) 2024-04-22T08:20:05.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:05.492+0000 7f9aef7fe640 1 -- 172.21.15.40:0/3156528183 learned_addr learned my addr 172.21.15.40:0/3156528183 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:20:05.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:05.492+0000 7f9aeeffd640 10 monclient(hunting): get_auth_request con 0x7f9af0135e30 auth_method 0 2024-04-22T08:20:05.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:05.492+0000 7f9aeeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:05.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:05.492+0000 7f9aeeffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:05.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:05.492+0000 7f9aeeffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:05.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:05.492+0000 7f9aeeffd640 1 -- 172.21.15.40:0/3156528183 >> v2:172.21.15.40:3300/0 conn(0x7f9af0135e30 msgr2=0x7f9af0136210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:20:05.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:05.492+0000 7f9aeeffd640 1 -- 172.21.15.40:0/3156528183 >> v2:172.21.15.40:3300/0 conn(0x7f9af0135e30 msgr2=0x7f9af0136210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:20:05.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:05.492+0000 7f9aeeffd640 1 --2- 172.21.15.40:0/3156528183 >> v2:172.21.15.40:3300/0 conn(0x7f9af0135e30 0x7f9af0136210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:20:05.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:05.492+0000 7f9aeeffd640 1 --2- 172.21.15.40:0/3156528183 >> v2:172.21.15.40:3300/0 conn(0x7f9af0135e30 0x7f9af0136210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:20:05.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:05.494+0000 7f9aef7fe640 1 -- 172.21.15.40:0/3156528183 >> v1:172.21.15.40:6789/0 conn(0x7f9af0136750 legacy=0x7f9af00a20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:05.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:05.494+0000 7f9aee7fc640 1 -- 172.21.15.40:0/3156528183 >> v1:172.21.15.189:6789/0 conn(0x7f9af0071c90 legacy=0x7f9af00a19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:05.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:05.693+0000 7f9aeeffd640 1 --2- 172.21.15.40:0/3156528183 >> v2:172.21.15.40:3300/0 conn(0x7f9af0135e30 0x7f9af0136210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:05.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:05.693+0000 7f9aeeffd640 10 monclient(hunting): get_auth_request con 0x7f9af0135e30 auth_method 0 2024-04-22T08:20:05.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:05.693+0000 7f9aeeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:05.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:05.693+0000 7f9aeeffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:05.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:05.693+0000 7f9aeeffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:20:05.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:05.693+0000 7f9aeeffd640 1 -- 172.21.15.40:0/3156528183 >> v2:172.21.15.40:3300/0 conn(0x7f9af0135e30 msgr2=0x7f9af0136210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:20:05.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:05.693+0000 7f9aeeffd640 1 -- 172.21.15.40:0/3156528183 >> v2:172.21.15.40:3300/0 conn(0x7f9af0135e30 msgr2=0x7f9af0136210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:20:05.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:05.693+0000 7f9aeeffd640 1 --2- 172.21.15.40:0/3156528183 >> v2:172.21.15.40:3300/0 conn(0x7f9af0135e30 0x7f9af0136210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:20:05.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:05.693+0000 7f9aeeffd640 1 --2- 172.21.15.40:0/3156528183 >> v2:172.21.15.40:3300/0 conn(0x7f9af0135e30 0x7f9af0136210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:20:06.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:06.094+0000 7f9aeeffd640 1 --2- 172.21.15.40:0/3156528183 >> v2:172.21.15.40:3300/0 conn(0x7f9af0135e30 0x7f9af0136210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:06.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:06.094+0000 7f9aeeffd640 10 monclient(hunting): get_auth_request con 0x7f9af0135e30 auth_method 0 2024-04-22T08:20:06.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:06.094+0000 7f9aeeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:06.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:06.094+0000 7f9aeeffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:06.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:06.094+0000 7f9aeeffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:20:06.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:06.094+0000 7f9aeeffd640 1 -- 172.21.15.40:0/3156528183 >> v2:172.21.15.40:3300/0 conn(0x7f9af0135e30 msgr2=0x7f9af0136210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:20:06.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:06.094+0000 7f9aeeffd640 1 -- 172.21.15.40:0/3156528183 >> v2:172.21.15.40:3300/0 conn(0x7f9af0135e30 msgr2=0x7f9af0136210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:20:06.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:06.094+0000 7f9aeeffd640 1 --2- 172.21.15.40:0/3156528183 >> v2:172.21.15.40:3300/0 conn(0x7f9af0135e30 0x7f9af0136210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:20:06.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:06.094+0000 7f9aeeffd640 1 --2- 172.21.15.40:0/3156528183 >> v2:172.21.15.40:3300/0 conn(0x7f9af0135e30 0x7f9af0136210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:20:06.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:06.896+0000 7f9aeeffd640 1 --2- 172.21.15.40:0/3156528183 >> v2:172.21.15.40:3300/0 conn(0x7f9af0135e30 0x7f9af0136210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:06.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:06.896+0000 7f9aeeffd640 10 monclient(hunting): get_auth_request con 0x7f9af0135e30 auth_method 0 2024-04-22T08:20:06.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:06.896+0000 7f9aeeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:06.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:06.896+0000 7f9aeeffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:06.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:06.896+0000 7f9aeeffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:20:06.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:06.896+0000 7f9aeeffd640 1 -- 172.21.15.40:0/3156528183 >> v2:172.21.15.40:3300/0 conn(0x7f9af0135e30 msgr2=0x7f9af0136210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:20:06.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:06.896+0000 7f9aeeffd640 1 -- 172.21.15.40:0/3156528183 >> v2:172.21.15.40:3300/0 conn(0x7f9af0135e30 msgr2=0x7f9af0136210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:20:06.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:06.896+0000 7f9aeeffd640 1 --2- 172.21.15.40:0/3156528183 >> v2:172.21.15.40:3300/0 conn(0x7f9af0135e30 0x7f9af0136210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:20:06.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:06.896+0000 7f9aeeffd640 1 --2- 172.21.15.40:0/3156528183 >> v2:172.21.15.40:3300/0 conn(0x7f9af0135e30 0x7f9af0136210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T08:20:08.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.298+0000 7f9aedffb640 1 -- 172.21.15.40:0/3156528183 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2218714268 0 0) 0x7f9af009e310 con 0x7f9af0071c90 2024-04-22T08:20:08.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.298+0000 7f9aedffb640 10 monclient(hunting): _negotiate 2024-04-22T08:20:08.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.298+0000 7f9aedffb640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:08.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.298+0000 7f9aedffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:08.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.298+0000 7f9aedffb640 10 monclient(hunting): my global_id is 64651 2024-04-22T08:20:08.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.298+0000 7f9aedffb640 1 -- 172.21.15.40:0/3156528183 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ad8003520 con 0x7f9af0071c90 2024-04-22T08:20:08.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aedffb640 1 -- 172.21.15.40:0/3156528183 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3896618515 0 0) 0x7f9ad8003520 con 0x7f9af0071c90 2024-04-22T08:20:08.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aedffb640 1 -- 172.21.15.40:0/3156528183 >> v2:172.21.15.40:3300/0 conn(0x7f9af0135e30 msgr2=0x7f9af0136210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:20:08.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aedffb640 1 --2- 172.21.15.40:0/3156528183 >> v2:172.21.15.40:3300/0 conn(0x7f9af0135e30 0x7f9af0136210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:08.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aedffb640 1 -- 172.21.15.40:0/3156528183 >> v1:172.21.15.40:6789/0 conn(0x7f9af0136750 legacy=0x7f9af00a20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:08.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aedffb640 10 monclient: _finish_hunting 0 2024-04-22T08:20:08.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aedffb640 1 monclient: found mon.noname-c-legacy 2024-04-22T08:20:08.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aedffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:20:08.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aedffb640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.189:6789/0 2024-04-22T08:20:08.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aedffb640 1 -- 172.21.15.40:0/3156528183 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9af009f130 con 0x7f9af0071c90 2024-04-22T08:20:08.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aedffb640 10 monclient: _finish_auth 0 2024-04-22T08:20:08.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aedffb640 10 monclient: _check_auth_tickets 2024-04-22T08:20:08.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aedffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:20:08.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aedffb640 1 -- 172.21.15.40:0/3156528183 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1087006497 0 0) 0x7f9ae4001970 con 0x7f9af0071c90 2024-04-22T08:20:08.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aedffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:08.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aedffb640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-04-22T08:20:08.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aedffb640 10 monclient: dump: 2024-04-22T08:20:08.303 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:08.304 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:08.304 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:08.304 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:08.304 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:08.304 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:08.304 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:08.304 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:08.304 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:08.304 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:08.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aedffb640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] but i'm connected to v1:172.21.15.189:6789/0, reconnecting 2024-04-22T08:20:08.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aedffb640 10 monclient: _reopen_session rank -1 2024-04-22T08:20:08.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aedffb640 1 -- 172.21.15.40:0/3156528183 >> v1:172.21.15.189:6789/0 conn(0x7f9af0071c90 legacy=0x7f9af00a19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:08.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aedffb640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-22T08:20:08.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aedffb640 1 --2- 172.21.15.40:0/3156528183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ad8003a40 0x7f9ad8003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:08.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aedffb640 10 monclient(hunting): picked mon.b con 0x7f9ad8003a40 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:20:08.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aedffb640 1 --2- 172.21.15.40:0/3156528183 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ad80047c0 0x7f9ad8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:08.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aedffb640 10 monclient(hunting): picked mon.c con 0x7f9ad80047c0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:20:08.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aedffb640 1 --2- 172.21.15.40:0/3156528183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ad80053d0 0x7f9ad800ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:08.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aedffb640 10 monclient(hunting): picked mon.a con 0x7f9ad80053d0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:20:08.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aedffb640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:08.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aedffb640 1 -- 172.21.15.40:0/3156528183 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af009de80 con 0x7f9ad80053d0 2024-04-22T08:20:08.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aedffb640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:08.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aedffb640 1 -- 172.21.15.40:0/3156528183 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad800c2e0 con 0x7f9ad8003a40 2024-04-22T08:20:08.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aedffb640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:08.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aedffb640 1 -- 172.21.15.40:0/3156528183 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad800c4a0 con 0x7f9ad80047c0 2024-04-22T08:20:08.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aedffb640 10 monclient(hunting): _renew_subs 2024-04-22T08:20:08.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aedffb640 10 monclient(hunting): _finish_auth 0 2024-04-22T08:20:08.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aee7fc640 1 --2- 172.21.15.40:0/3156528183 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ad80047c0 0x7f9ad8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:08.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aeeffd640 1 --2- 172.21.15.40:0/3156528183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ad8003a40 0x7f9ad8003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:08.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aeeffd640 10 monclient(hunting): get_auth_request con 0x7f9ad8003a40 auth_method 0 2024-04-22T08:20:08.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aeeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:08.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aeeffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:08.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aeeffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:20:08.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aee7fc640 10 monclient(hunting): get_auth_request con 0x7f9ad80047c0 auth_method 0 2024-04-22T08:20:08.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aee7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:08.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aee7fc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:08.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aee7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:20:08.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aeeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:08.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aeeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:08.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aeeffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-22T08:20:08.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aee7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:08.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aee7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:08.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.299+0000 7f9aee7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-22T08:20:08.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.300+0000 7f9aeeffd640 10 monclient(hunting): handle_auth_done global_id 64651 payload 293 2024-04-22T08:20:08.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.300+0000 7f9aef7fe640 1 --2- 172.21.15.40:0/3156528183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ad80053d0 0x7f9ad800ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:08.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.300+0000 7f9aeeffd640 1 -- 172.21.15.40:0/3156528183 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ad80047c0 msgr2=0x7f9ad8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:20:08.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.300+0000 7f9aeeffd640 1 --2- 172.21.15.40:0/3156528183 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ad80047c0 0x7f9ad8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:08.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.300+0000 7f9aeeffd640 1 -- 172.21.15.40:0/3156528183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ad80053d0 msgr2=0x7f9ad800ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:20:08.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.300+0000 7f9aeeffd640 1 --2- 172.21.15.40:0/3156528183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ad80053d0 0x7f9ad800ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:08.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.300+0000 7f9aeeffd640 10 monclient: _finish_hunting 0 2024-04-22T08:20:08.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.300+0000 7f9aeeffd640 1 monclient: found mon.b 2024-04-22T08:20:08.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.300+0000 7f9aeeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:20:08.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.300+0000 7f9aeeffd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.189:3300/0 2024-04-22T08:20:08.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.300+0000 7f9aeeffd640 1 -- 172.21.15.40:0/3156528183 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad8003520 con 0x7f9ad8003a40 2024-04-22T08:20:08.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.300+0000 7f9aeeffd640 10 monclient: _finish_auth 0 2024-04-22T08:20:08.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.300+0000 7f9aeeffd640 10 monclient: _check_auth_tickets 2024-04-22T08:20:08.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.300+0000 7f9aeeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:20:08.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.300+0000 7f9af5722640 5 monclient: authenticate success, global_id 64651 2024-04-22T08:20:08.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.300+0000 7f9af5722640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:20:08.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.300+0000 7f9aef7fe640 10 monclient: get_auth_request con 0x7f9ad80053d0 auth_method 0 2024-04-22T08:20:08.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.300+0000 7f9aee7fc640 1 --2- 172.21.15.40:0/3156528183 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ad80047c0 0x7f9ad8004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:20:08.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.300+0000 7f9aef7fe640 1 --2- 172.21.15.40:0/3156528183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ad80053d0 0x7f9ad800ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T08:20:08.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.300+0000 7f9aef7fe640 1 -- 172.21.15.40:0/3156528183 reap_dead start 2024-04-22T08:20:08.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.303+0000 7f9aeeffd640 1 -- 172.21.15.40:0/3156528183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ad8003a40 msgr2=0x7f9ad8003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:08.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.303+0000 7f9aeeffd640 1 --2- 172.21.15.40:0/3156528183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ad8003a40 0x7f9ad8003e60 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f9ae0011b90 tx=0x7f9ad80057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a5ad06676e70a97 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:08.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.303+0000 7f9aedffb640 1 -- 172.21.15.40:0/3156528183 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae0035410 con 0x7f9ad8003a40 2024-04-22T08:20:08.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.303+0000 7f9aedffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:08.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.303+0000 7f9aedffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:20:08.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.303+0000 7f9aedffb640 10 monclient: dump: 2024-04-22T08:20:08.313 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:08.313 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:08.313 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:08.313 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:08.313 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:08.313 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:08.313 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:08.313 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:08.313 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:08.313 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:08.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.303+0000 7f9aedffb640 1 -- 172.21.15.40:0/3156528183 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ae002f030 con 0x7f9ad8003a40 2024-04-22T08:20:08.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.303+0000 7f9aedffb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:20:08.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.303+0000 7f9aedffb640 1 -- 172.21.15.40:0/3156528183 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae00355d0 con 0x7f9ad8003a40 2024-04-22T08:20:08.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.303+0000 7f9af5722640 10 monclient: get_monmap_and_config success 2024-04-22T08:20:08.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.303+0000 7f9af5722640 4 set_mon_vals no callback set 2024-04-22T08:20:08.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.303+0000 7f9aedffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:08.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.303+0000 7f9aedffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:20:08.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.303+0000 7f9aedffb640 10 monclient: dump: 2024-04-22T08:20:08.314 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:08.314 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:08.314 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:08.314 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:08.314 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:08.314 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:08.314 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:08.315 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:08.315 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:08.315 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:08.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.305+0000 7f9af5722640 10 monclient: shutdown 2024-04-22T08:20:08.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.305+0000 7f9af5722640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:20:08.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.305+0000 7f9af5722640 1 -- 172.21.15.40:0/3156528183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ad8003a40 msgr2=0x7f9ad8003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:08.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.305+0000 7f9af5722640 1 --2- 172.21.15.40:0/3156528183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ad8003a40 0x7f9ad8003e60 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f9ae0011b90 tx=0x7f9ad80057f0 comp rx=0 tx=0).stop 2024-04-22T08:20:08.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.305+0000 7f9af5722640 1 -- 172.21.15.40:0/3156528183 shutdown_connections 2024-04-22T08:20:08.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.305+0000 7f9af5722640 1 --2- 172.21.15.40:0/3156528183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ad8003a40 0x7f9ad8003e60 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:08.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.305+0000 7f9af5722640 1 -- 172.21.15.40:0/3156528183 >> 172.21.15.40:0/3156528183 conn(0x7f9af00a65c0 msgr2=0x7f9af00a2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:20:08.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.305+0000 7f9af5722640 1 -- 172.21.15.40:0/3156528183 shutdown_connections 2024-04-22T08:20:08.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.305+0000 7f9af5722640 1 -- 172.21.15.40:0/3156528183 wait complete. 2024-04-22T08:20:08.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.307+0000 7f9af5722640 10 monclient: build_initial_monmap 2024-04-22T08:20:08.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.307+0000 7f9af5722640 10 monclient: monmap: 2024-04-22T08:20:08.316 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:20:08.316 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:20:08.316 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:20:08.316 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:20:08.316 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:20:08.316 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:08.316 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:20:08.316 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:20:08.316 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:20:08.316 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:08.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.308+0000 7f9af5722640 1 Processor -- start 2024-04-22T08:20:08.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.309+0000 7f9af5722640 1 -- start start 2024-04-22T08:20:08.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.309+0000 7f9af5722640 10 monclient: init 2024-04-22T08:20:08.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9af5722640 10 monclient: _reopen_session rank -1 2024-04-22T08:20:08.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9af5722640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-22T08:20:08.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9af5722640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ad8003a40 0x7f9af01209f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:08.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9af5722640 10 monclient(hunting): picked mon.noname-b con 0x7f9ad8003a40 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:20:08.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9af5722640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9af0120f30 0x7f9af01213b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:08.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9af5722640 10 monclient(hunting): picked mon.noname-a con 0x7f9af0120f30 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:20:08.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9af5722640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9af01caaa0 0x7f9af01caf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:08.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9af5722640 10 monclient(hunting): picked mon.noname-c con 0x7f9af01caaa0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:20:08.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9af5722640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:08.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9af5722640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af01238e0 con 0x7f9af0120f30 2024-04-22T08:20:08.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9af5722640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:08.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9af5722640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af00720e0 con 0x7f9ad8003a40 2024-04-22T08:20:08.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9af5722640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:08.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9af5722640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9af0121ad0 con 0x7f9af01caaa0 2024-04-22T08:20:08.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9af5722640 10 monclient(hunting): _renew_subs 2024-04-22T08:20:08.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9af5722640 10 monclient(hunting): authenticate will time out at 1282.305044s 2024-04-22T08:20:08.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9aef7fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9af01caaa0 0x7f9af01caf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:08.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9aef7fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9af01caaa0 0x7f9af01caf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:56564/0 (socket says 172.21.15.40:56564) 2024-04-22T08:20:08.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9aef7fe640 1 -- 172.21.15.40:0/3005055413 learned_addr learned my addr 172.21.15.40:0/3005055413 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:20:08.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9aef7fe640 10 monclient(hunting): get_auth_request con 0x7f9af01caaa0 auth_method 0 2024-04-22T08:20:08.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9aeeffd640 1 --2- 172.21.15.40:0/3005055413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ad8003a40 0x7f9af01209f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:08.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9aef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:08.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9aef7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:08.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9aef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:08.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9aeeffd640 10 monclient(hunting): get_auth_request con 0x7f9ad8003a40 auth_method 0 2024-04-22T08:20:08.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9aeeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:08.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9aeeffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:08.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9aeeffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:08.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9aee7fc640 1 --2- 172.21.15.40:0/3005055413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9af0120f30 0x7f9af01213b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:08.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9aef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:08.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9aef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:08.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9aef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:08.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9aeeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:08.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9aeeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:08.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9aeeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:08.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9aee7fc640 10 monclient(hunting): get_auth_request con 0x7f9af0120f30 auth_method 0 2024-04-22T08:20:08.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9aee7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:08.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9aee7fc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:08.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9aee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:08.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9aeeffd640 10 monclient(hunting): handle_auth_done global_id 64654 payload 824 2024-04-22T08:20:08.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9aee7fc640 1 -- 172.21.15.40:0/3005055413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9af0120f30 msgr2=0x7f9af01213b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-22T08:20:08.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9aee7fc640 1 -- 172.21.15.40:0/3005055413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9af0120f30 msgr2=0x7f9af01213b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:20:08.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9aee7fc640 1 --2- 172.21.15.40:0/3005055413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9af0120f30 0x7f9af01213b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:20:08.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9aeeffd640 1 -- 172.21.15.40:0/3005055413 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9af01caaa0 msgr2=0x7f9af01caf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:08.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9aeeffd640 1 --2- 172.21.15.40:0/3005055413 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9af01caaa0 0x7f9af01caf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:08.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9aeeffd640 1 -- 172.21.15.40:0/3005055413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9af0120f30 msgr2=0x7f9af01213b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:08.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9aee7fc640 1 --2- 172.21.15.40:0/3005055413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9af0120f30 0x7f9af01213b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:20:08.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9aeeffd640 1 --2- 172.21.15.40:0/3005055413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9af0120f30 0x7f9af01213b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:08.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9aeeffd640 10 monclient: _finish_hunting 0 2024-04-22T08:20:08.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9aeeffd640 1 monclient: found mon.noname-b 2024-04-22T08:20:08.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9aeeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:20:08.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9aeeffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.189:3300/0 2024-04-22T08:20:08.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9aeeffd640 1 -- 172.21.15.40:0/3005055413 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9af0134f80 con 0x7f9ad8003a40 2024-04-22T08:20:08.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.310+0000 7f9aef7fe640 1 --2- 172.21.15.40:0/3005055413 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9af01caaa0 0x7f9af01caf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:20:08.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.313+0000 7f9aeeffd640 1 -- 172.21.15.40:0/3005055413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ad8003a40 msgr2=0x7f9af01209f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:08.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.313+0000 7f9aeeffd640 1 --2- 172.21.15.40:0/3005055413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ad8003a40 0x7f9af01209f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9ae0037000 tx=0x7f9ae00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:08.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.313+0000 7f9ad77fe640 1 -- 172.21.15.40:0/3005055413 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae003f820 con 0x7f9ad8003a40 2024-04-22T08:20:08.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.313+0000 7f9ad77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:08.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.313+0000 7f9ad77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:20:08.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.313+0000 7f9ad77fe640 10 monclient: dump: 2024-04-22T08:20:08.324 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:08.324 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:08.324 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:08.324 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:08.324 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:08.324 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:08.324 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:08.325 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:08.325 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:08.325 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:08.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.313+0000 7f9ad77fe640 10 monclient: _finish_auth 0 2024-04-22T08:20:08.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.313+0000 7f9ad77fe640 10 monclient: _check_auth_tickets 2024-04-22T08:20:08.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.313+0000 7f9ad77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:20:08.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.313+0000 7f9ad77fe640 1 -- 172.21.15.40:0/3005055413 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ae003f9a0 con 0x7f9ad8003a40 2024-04-22T08:20:08.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.313+0000 7f9ad77fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:20:08.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.313+0000 7f9ad77fe640 1 -- 172.21.15.40:0/3005055413 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae003fd50 con 0x7f9ad8003a40 2024-04-22T08:20:08.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.313+0000 7f9af5722640 5 monclient: authenticate success, global_id 64654 2024-04-22T08:20:08.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.313+0000 7f9af5722640 10 monclient: _renew_subs 2024-04-22T08:20:08.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.313+0000 7f9af5722640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.189:3300/0 2024-04-22T08:20:08.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.313+0000 7f9af5722640 1 -- 172.21.15.40:0/3005055413 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9af0123e10 con 0x7f9ad8003a40 2024-04-22T08:20:08.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.313+0000 7f9aed7fa640 4 set_mon_vals no callback set 2024-04-22T08:20:08.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.313+0000 7f9ad77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:08.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.313+0000 7f9ad77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:20:08.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.313+0000 7f9ad77fe640 10 monclient: dump: 2024-04-22T08:20:08.326 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:08.326 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:08.326 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:08.326 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:08.326 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:08.326 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:08.326 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:08.326 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:08.326 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:08.326 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:08.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.313+0000 7f9af5722640 10 monclient: _renew_subs 2024-04-22T08:20:08.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.313+0000 7f9af5722640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.189:3300/0 2024-04-22T08:20:08.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.313+0000 7f9af5722640 1 -- 172.21.15.40:0/3005055413 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9af0124230 con 0x7f9ad8003a40 2024-04-22T08:20:08.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.314+0000 7f9ad77fe640 1 -- 172.21.15.40:0/3005055413 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f9ae003c050 con 0x7f9ad8003a40 2024-04-22T08:20:08.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.314+0000 7f9af5722640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:20:08.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.315+0000 7f9af5722640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:20:08.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.315+0000 7f9af5722640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.189:3300/0 2024-04-22T08:20:08.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.315+0000 7f9ad77fe640 1 --2- 172.21.15.40:0/3005055413 >> v2:172.21.15.189:6800/30168 conn(0x7f9adc037820 0x7f9adc039ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:08.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.315+0000 7f9af5722640 1 -- 172.21.15.40:0/3005055413 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9af004e9b0 con 0x7f9ad8003a40 2024-04-22T08:20:08.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.315+0000 7f9ad77fe640 1 -- 172.21.15.40:0/3005055413 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f9ae006c880 con 0x7f9ad8003a40 2024-04-22T08:20:08.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.315+0000 7f9aee7fc640 1 --2- 172.21.15.40:0/3005055413 >> v2:172.21.15.189:6800/30168 conn(0x7f9adc037820 0x7f9adc039ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:08.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.315+0000 7f9aee7fc640 10 monclient: get_auth_request con 0x7f9adc037820 auth_method 0 2024-04-22T08:20:08.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.318+0000 7f9aee7fc640 1 --2- 172.21.15.40:0/3005055413 >> v2:172.21.15.189:6800/30168 conn(0x7f9adc037820 0x7f9adc039ce0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f9af0071910 tx=0x7f9ae4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:08.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.320+0000 7f9ad77fe640 1 -- 172.21.15.40:0/3005055413 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f9ae003e370 con 0x7f9ad8003a40 2024-04-22T08:20:08.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.320+0000 7f9ad77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:20:08.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.320+0000 7f9ad77fe640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:20:08.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.578+0000 7f9af5722640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:20:08.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.578+0000 7f9af5722640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:20:08.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.578+0000 7f9af5722640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.189:3300/0 2024-04-22T08:20:08.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.578+0000 7f9af5722640 1 -- 172.21.15.40:0/3005055413 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9af006e280 con 0x7f9ad8003a40 2024-04-22T08:20:08.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.578+0000 7f9ad77fe640 1 -- 172.21.15.40:0/3005055413 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f9ae003a030 con 0x7f9ad8003a40 2024-04-22T08:20:08.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.578+0000 7f9ad77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:20:08.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.578+0000 7f9ad77fe640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:20:08.579 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":72,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:20:08.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.580+0000 7f9af5722640 1 -- 172.21.15.40:0/3005055413 >> v2:172.21.15.189:6800/30168 conn(0x7f9adc037820 msgr2=0x7f9adc039ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:08.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.580+0000 7f9af5722640 1 --2- 172.21.15.40:0/3005055413 >> v2:172.21.15.189:6800/30168 conn(0x7f9adc037820 0x7f9adc039ce0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f9af0071910 tx=0x7f9ae4009040 comp rx=0 tx=0).stop 2024-04-22T08:20:08.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.580+0000 7f9af5722640 10 monclient: shutdown 2024-04-22T08:20:08.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.580+0000 7f9af5722640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:20:08.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.580+0000 7f9af5722640 1 -- 172.21.15.40:0/3005055413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ad8003a40 msgr2=0x7f9af01209f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:08.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.580+0000 7f9af5722640 1 --2- 172.21.15.40:0/3005055413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ad8003a40 0x7f9af01209f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9ae0037000 tx=0x7f9ae00046c0 comp rx=0 tx=0).stop 2024-04-22T08:20:08.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.580+0000 7f9af5722640 1 -- 172.21.15.40:0/3005055413 shutdown_connections 2024-04-22T08:20:08.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.580+0000 7f9af5722640 1 --2- 172.21.15.40:0/3005055413 >> v2:172.21.15.189:6800/30168 conn(0x7f9adc037820 0x7f9adc039ce0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:08.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.580+0000 7f9af5722640 1 --2- 172.21.15.40:0/3005055413 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9af01caaa0 0x7f9af01caf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:08.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.580+0000 7f9af5722640 1 --2- 172.21.15.40:0/3005055413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9af0120f30 0x7f9af01213b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:08.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.580+0000 7f9af5722640 1 --2- 172.21.15.40:0/3005055413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ad8003a40 0x7f9af01209f0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:08.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.580+0000 7f9af5722640 1 -- 172.21.15.40:0/3005055413 >> 172.21.15.40:0/3005055413 conn(0x7f9af00a65c0 msgr2=0x7f9af009c2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:20:08.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.580+0000 7f9af5722640 1 -- 172.21.15.40:0/3005055413 shutdown_connections 2024-04-22T08:20:08.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.580+0000 7f9af5722640 1 -- 172.21.15.40:0/3005055413 wait complete. 2024-04-22T08:20:08.591 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-22T08:20:08.591 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-22T08:20:08.591 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-22T08:20:08.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.690+0000 7fb16b7fd640 10 monclient: get_monmap_and_config 2024-04-22T08:20:08.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.690+0000 7fb16b7fd640 10 monclient: build_initial_monmap 2024-04-22T08:20:08.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.690+0000 7fb16b7fd640 10 monclient: monmap: 2024-04-22T08:20:08.691 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:20:08.691 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:08.691 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:20:08.692340+0000 2024-04-22T08:20:08.691 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:20:08.692340+0000 2024-04-22T08:20:08.691 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:20:08.691 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:08.691 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:20:08.692 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:20:08.692 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:20:08.692 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:20:08.692 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:20:08.692 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:08.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.691+0000 7fb16b7fd640 1 Processor -- start 2024-04-22T08:20:08.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.691+0000 7fb16b7fd640 1 -- start start 2024-04-22T08:20:08.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.691+0000 7fb16b7fd640 10 monclient: init 2024-04-22T08:20:08.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.691+0000 7fb16b7fd640 10 monclient: _reopen_session rank -1 2024-04-22T08:20:08.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.691+0000 7fb16b7fd640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-04-22T08:20:08.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.691+0000 7fb16b7fd640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb16412fad0 0x7fb16412feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:08.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.691+0000 7fb16b7fd640 10 monclient(hunting): picked mon.noname-c con 0x7fb16412fad0 addr v2:172.21.15.189:3300/0 2024-04-22T08:20:08.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.691+0000 7fb16b7fd640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb164132920 0x7fb164132d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:08.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.691+0000 7fb16b7fd640 10 monclient(hunting): picked mon.noname-a con 0x7fb164132920 addr v2:172.21.15.40:3300/0 2024-04-22T08:20:08.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.691+0000 7fb16b7fd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb164133290 0x7fb164139c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:08.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.691+0000 7fb16b7fd640 10 monclient(hunting): picked mon.noname-b con 0x7fb164133290 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:20:08.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.691+0000 7fb16b7fd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:08.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.691+0000 7fb16b7fd640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb16409ae50 con 0x7fb164132920 2024-04-22T08:20:08.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.691+0000 7fb16b7fd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:08.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.691+0000 7fb16b7fd640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb16409efc0 con 0x7fb16412fad0 2024-04-22T08:20:08.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.691+0000 7fb16b7fd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:08.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.691+0000 7fb16b7fd640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb16409d070 con 0x7fb164133290 2024-04-22T08:20:08.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.691+0000 7fb16b7fd640 10 monclient(hunting): _renew_subs 2024-04-22T08:20:08.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.691+0000 7fb16b7fd640 10 monclient(hunting): authenticate will time out at 1282.686676s 2024-04-22T08:20:08.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.692+0000 7fb169d9a640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb164133290 0x7fb164139c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:08.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.692+0000 7fb168d98640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb164132920 0x7fb164132d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:08.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.692+0000 7fb169d9a640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb164133290 0x7fb164139c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:56596/0 (socket says 172.21.15.40:56596) 2024-04-22T08:20:08.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.692+0000 7fb168d98640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb164132920 0x7fb164132d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43350/0 (socket says 172.21.15.40:43350) 2024-04-22T08:20:08.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.692+0000 7fb169d9a640 1 -- 172.21.15.40:0/615155733 learned_addr learned my addr 172.21.15.40:0/615155733 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:20:08.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.692+0000 7fb169d9a640 10 monclient(hunting): get_auth_request con 0x7fb164133290 auth_method 0 2024-04-22T08:20:08.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.692+0000 7fb169d9a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:08.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.692+0000 7fb169d9a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:08.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.692+0000 7fb169d9a640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:08.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.692+0000 7fb168d98640 10 monclient(hunting): get_auth_request con 0x7fb164132920 auth_method 0 2024-04-22T08:20:08.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.692+0000 7fb168d98640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:08.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.692+0000 7fb168d98640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:08.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.692+0000 7fb168d98640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:08.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.692+0000 7fb169d9a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:08.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.692+0000 7fb169d9a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:08.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.692+0000 7fb169d9a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:08.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.692+0000 7fb168d98640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:08.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.692+0000 7fb168d98640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:08.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.692+0000 7fb168d98640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:08.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.692+0000 7fb169599640 1 --2- 172.21.15.40:0/615155733 >> v2:172.21.15.189:3300/0 conn(0x7fb16412fad0 0x7fb16412feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:08.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.692+0000 7fb169599640 10 monclient(hunting): get_auth_request con 0x7fb16412fad0 auth_method 0 2024-04-22T08:20:08.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.692+0000 7fb169599640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:08.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.692+0000 7fb169599640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:08.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.692+0000 7fb169599640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:08.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.692+0000 7fb169d9a640 10 monclient(hunting): handle_auth_done global_id 54821 payload 274 2024-04-22T08:20:08.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.692+0000 7fb169d9a640 1 -- 172.21.15.40:0/615155733 >> v2:172.21.15.189:3300/0 conn(0x7fb16412fad0 msgr2=0x7fb16412feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:20:08.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.692+0000 7fb169d9a640 1 --2- 172.21.15.40:0/615155733 >> v2:172.21.15.189:3300/0 conn(0x7fb16412fad0 0x7fb16412feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:08.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.692+0000 7fb169d9a640 1 -- 172.21.15.40:0/615155733 >> v2:172.21.15.40:3300/0 conn(0x7fb164132920 msgr2=0x7fb164132d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:20:08.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.692+0000 7fb169d9a640 1 --2- 172.21.15.40:0/615155733 >> v2:172.21.15.40:3300/0 conn(0x7fb164132920 0x7fb164132d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:08.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.692+0000 7fb169d9a640 10 monclient: _finish_hunting 0 2024-04-22T08:20:08.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.692+0000 7fb169d9a640 1 monclient: found mon.noname-b 2024-04-22T08:20:08.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.692+0000 7fb169d9a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:20:08.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.692+0000 7fb169d9a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:20:08.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.692+0000 7fb169d9a640 1 -- 172.21.15.40:0/615155733 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb16409d350 con 0x7fb164133290 2024-04-22T08:20:08.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.692+0000 7fb168d98640 1 --2- 172.21.15.40:0/615155733 >> v2:172.21.15.40:3300/0 conn(0x7fb164132920 0x7fb164132d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:20:08.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.695+0000 7fb169d9a640 1 -- 172.21.15.40:0/615155733 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb164133290 msgr2=0x7fb164139c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:08.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.695+0000 7fb169d9a640 1 --2- 172.21.15.40:0/615155733 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb164133290 0x7fb164139c50 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb16406bb30 tx=0x7fb15c02bdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a828e7ce8a7466df server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:08.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.695+0000 7fb15bfff640 1 -- 172.21.15.40:0/615155733 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15c02cb60 con 0x7fb164133290 2024-04-22T08:20:08.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.695+0000 7fb15bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:08.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.695+0000 7fb15bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:20:08.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.695+0000 7fb15bfff640 10 monclient: dump: 2024-04-22T08:20:08.705 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:08.706 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:08.706 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:08.706 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:08.706 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:08.706 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:08.706 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:08.706 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:08.706 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:08.706 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:08.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.695+0000 7fb15bfff640 10 monclient: _finish_auth 0 2024-04-22T08:20:08.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.695+0000 7fb15bfff640 10 monclient: _check_auth_tickets 2024-04-22T08:20:08.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.695+0000 7fb15bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:20:08.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.695+0000 7fb15bfff640 1 -- 172.21.15.40:0/615155733 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb15c02ccc0 con 0x7fb164133290 2024-04-22T08:20:08.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.695+0000 7fb16b7fd640 5 monclient: authenticate success, global_id 54821 2024-04-22T08:20:08.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.695+0000 7fb16b7fd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:20:08.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.695+0000 7fb15bfff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:20:08.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.695+0000 7fb15bfff640 1 -- 172.21.15.40:0/615155733 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15c034610 con 0x7fb164133290 2024-04-22T08:20:08.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.695+0000 7fb16b7fd640 10 monclient: get_monmap_and_config success 2024-04-22T08:20:08.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.695+0000 7fb16b7fd640 4 set_mon_vals no callback set 2024-04-22T08:20:08.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.695+0000 7fb15bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:08.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.695+0000 7fb15bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:20:08.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.695+0000 7fb15bfff640 10 monclient: dump: 2024-04-22T08:20:08.707 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:08.707 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:08.707 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:08.707 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:08.707 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:08.707 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:08.708 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:08.708 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:08.708 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:08.708 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:08.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.695+0000 7fb16b7fd640 10 monclient: shutdown 2024-04-22T08:20:08.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.695+0000 7fb16b7fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:20:08.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.695+0000 7fb16b7fd640 1 -- 172.21.15.40:0/615155733 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb164133290 msgr2=0x7fb164139c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:08.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.695+0000 7fb16b7fd640 1 --2- 172.21.15.40:0/615155733 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb164133290 0x7fb164139c50 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb16406bb30 tx=0x7fb15c02bdb0 comp rx=0 tx=0).stop 2024-04-22T08:20:08.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.696+0000 7fb16b7fd640 1 -- 172.21.15.40:0/615155733 shutdown_connections 2024-04-22T08:20:08.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.696+0000 7fb16b7fd640 1 --2- 172.21.15.40:0/615155733 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb164133290 0x7fb164139c50 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:08.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.696+0000 7fb16b7fd640 1 --2- 172.21.15.40:0/615155733 >> v2:172.21.15.40:3300/0 conn(0x7fb164132920 0x7fb164132d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:08.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.696+0000 7fb16b7fd640 1 --2- 172.21.15.40:0/615155733 >> v2:172.21.15.189:3300/0 conn(0x7fb16412fad0 0x7fb16412feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:08.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.696+0000 7fb16b7fd640 1 -- 172.21.15.40:0/615155733 >> 172.21.15.40:0/615155733 conn(0x7fb1640a0ae0 msgr2=0x7fb1640a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:20:08.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.696+0000 7fb16b7fd640 1 -- 172.21.15.40:0/615155733 shutdown_connections 2024-04-22T08:20:08.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.696+0000 7fb16b7fd640 1 -- 172.21.15.40:0/615155733 wait complete. 2024-04-22T08:20:08.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.696+0000 7fb16b7fd640 10 monclient: build_initial_monmap 2024-04-22T08:20:08.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.696+0000 7fb16b7fd640 10 monclient: monmap: 2024-04-22T08:20:08.709 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:20:08.709 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:20:08.709 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:20:08.709 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:20:08.709 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:20:08.709 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:08.709 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:20:08.709 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:20:08.709 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:20:08.709 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:08.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb16b7fd640 1 Processor -- start 2024-04-22T08:20:08.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb16b7fd640 1 -- start start 2024-04-22T08:20:08.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb16b7fd640 10 monclient: init 2024-04-22T08:20:08.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb16b7fd640 10 monclient: _reopen_session rank -1 2024-04-22T08:20:08.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb16b7fd640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:20:08.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb16b7fd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb16412fad0 0x7fb1641ca530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:08.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb16b7fd640 10 monclient(hunting): picked mon.noname-a con 0x7fb16412fad0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:20:08.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb16b7fd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb164132920 0x7fb16412eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:08.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb16b7fd640 10 monclient(hunting): picked mon.noname-b con 0x7fb164132920 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:20:08.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb16b7fd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb164133290 0x7fb164137bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:08.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb16b7fd640 10 monclient(hunting): picked mon.noname-c con 0x7fb164133290 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:20:08.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb16b7fd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:08.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb16b7fd640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb16413b800 con 0x7fb16412fad0 2024-04-22T08:20:08.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb16b7fd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:08.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb16b7fd640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1641caa70 con 0x7fb164132920 2024-04-22T08:20:08.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb16b7fd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:08.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb16b7fd640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb16413bdb0 con 0x7fb164133290 2024-04-22T08:20:08.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb16b7fd640 10 monclient(hunting): _renew_subs 2024-04-22T08:20:08.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb16b7fd640 10 monclient(hunting): authenticate will time out at 1282.692385s 2024-04-22T08:20:08.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb169599640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb16412fad0 0x7fb1641ca530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:08.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb169599640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb16412fad0 0x7fb1641ca530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43366/0 (socket says 172.21.15.40:43366) 2024-04-22T08:20:08.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb169599640 1 -- 172.21.15.40:0/1168981831 learned_addr learned my addr 172.21.15.40:0/1168981831 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:20:08.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb169599640 10 monclient(hunting): get_auth_request con 0x7fb16412fad0 auth_method 0 2024-04-22T08:20:08.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb169599640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:08.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb169599640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:08.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb169599640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:08.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb169d9a640 1 --2- 172.21.15.40:0/1168981831 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb164133290 0x7fb164137bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:08.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb169d9a640 10 monclient(hunting): get_auth_request con 0x7fb164133290 auth_method 0 2024-04-22T08:20:08.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb169d9a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:08.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb169d9a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:08.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb169d9a640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:08.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb168d98640 1 --2- 172.21.15.40:0/1168981831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb164132920 0x7fb16412eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:08.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb169599640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:08.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb169599640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:08.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb169599640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:08.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb169d9a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:08.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb169d9a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:08.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb169d9a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:08.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb168d98640 10 monclient(hunting): get_auth_request con 0x7fb164132920 auth_method 0 2024-04-22T08:20:08.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb168d98640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:08.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb168d98640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:08.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb168d98640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:08.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb168d98640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:08.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb168d98640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:08.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb168d98640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:08.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb169d9a640 10 monclient(hunting): handle_auth_done global_id 54824 payload 824 2024-04-22T08:20:08.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb169d9a640 1 -- 172.21.15.40:0/1168981831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb164132920 msgr2=0x7fb16412eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:08.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.697+0000 7fb169d9a640 1 --2- 172.21.15.40:0/1168981831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb164132920 0x7fb16412eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:08.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.698+0000 7fb169d9a640 1 -- 172.21.15.40:0/1168981831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb16412fad0 msgr2=0x7fb1641ca530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:08.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.698+0000 7fb169d9a640 1 --2- 172.21.15.40:0/1168981831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb16412fad0 0x7fb1641ca530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:08.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.698+0000 7fb169d9a640 10 monclient: _finish_hunting 0 2024-04-22T08:20:08.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.698+0000 7fb169d9a640 1 monclient: found mon.noname-c 2024-04-22T08:20:08.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.698+0000 7fb169d9a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:20:08.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.698+0000 7fb169d9a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:20:08.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.698+0000 7fb169d9a640 1 -- 172.21.15.40:0/1168981831 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb154009400 con 0x7fb164133290 2024-04-22T08:20:08.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.698+0000 7fb169599640 1 --2- 172.21.15.40:0/1168981831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb16412fad0 0x7fb1641ca530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:20:08.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.700+0000 7fb169d9a640 1 -- 172.21.15.40:0/1168981831 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb164133290 msgr2=0x7fb164137bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:08.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.700+0000 7fb169d9a640 1 --2- 172.21.15.40:0/1168981831 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb164133290 0x7fb164137bc0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb16409d1b0 tx=0x7fb15c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:08.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.700+0000 7fb159ffb640 1 -- 172.21.15.40:0/1168981831 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15c034ca0 con 0x7fb164133290 2024-04-22T08:20:08.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.700+0000 7fb159ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:08.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.700+0000 7fb159ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:20:08.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.700+0000 7fb159ffb640 10 monclient: dump: 2024-04-22T08:20:08.714 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:08.714 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:08.714 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:08.714 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:08.714 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:08.714 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:08.715 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:08.715 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:08.715 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:08.715 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:08.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.700+0000 7fb159ffb640 10 monclient: _finish_auth 0 2024-04-22T08:20:08.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.700+0000 7fb159ffb640 10 monclient: _check_auth_tickets 2024-04-22T08:20:08.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.700+0000 7fb159ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:20:08.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.700+0000 7fb16b7fd640 5 monclient: authenticate success, global_id 54824 2024-04-22T08:20:08.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.700+0000 7fb159ffb640 1 -- 172.21.15.40:0/1168981831 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb15c034e20 con 0x7fb164133290 2024-04-22T08:20:08.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.700+0000 7fb159ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:20:08.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.700+0000 7fb16b7fd640 10 monclient: _renew_subs 2024-04-22T08:20:08.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.700+0000 7fb159ffb640 1 -- 172.21.15.40:0/1168981831 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15c03e690 con 0x7fb164133290 2024-04-22T08:20:08.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.700+0000 7fb16b7fd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:20:08.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.700+0000 7fb16b7fd640 1 -- 172.21.15.40:0/1168981831 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb15c009b30 con 0x7fb164133290 2024-04-22T08:20:08.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.700+0000 7fb159ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:08.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.700+0000 7fb15b7fe640 4 set_mon_vals no callback set 2024-04-22T08:20:08.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.700+0000 7fb159ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:20:08.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.700+0000 7fb159ffb640 10 monclient: dump: 2024-04-22T08:20:08.716 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:08.716 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:08.716 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:08.716 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:08.716 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:08.716 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:08.716 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:08.716 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:08.716 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:08.716 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:08.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.700+0000 7fb16b7fd640 10 monclient: _renew_subs 2024-04-22T08:20:08.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.700+0000 7fb16b7fd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:20:08.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.700+0000 7fb16b7fd640 1 -- 172.21.15.40:0/1168981831 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb164139680 con 0x7fb164133290 2024-04-22T08:20:08.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.701+0000 7fb16b7fd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:20:08.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.701+0000 7fb16b7fd640 1 --2- 172.21.15.40:0/1168981831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb16412f090 0x7fb16402b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:08.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.701+0000 7fb16b7fd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:08.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.701+0000 7fb16b7fd640 1 -- 172.21.15.40:0/1168981831 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1640667a0 con 0x7fb16412f090 2024-04-22T08:20:08.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.701+0000 7fb159ffb640 1 -- 172.21.15.40:0/1168981831 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fb15c03e870 con 0x7fb164133290 2024-04-22T08:20:08.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.701+0000 7fb169599640 1 --2- 172.21.15.40:0/1168981831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb16412f090 0x7fb16402b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:08.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.701+0000 7fb169599640 10 monclient: get_auth_request con 0x7fb16412f090 auth_method 0 2024-04-22T08:20:08.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.701+0000 7fb169599640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:08.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.701+0000 7fb169599640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:08.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.701+0000 7fb169599640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:08.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.701+0000 7fb169599640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:08.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.701+0000 7fb169599640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:08.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.701+0000 7fb169599640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:08.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.701+0000 7fb169599640 10 monclient(hunting): handle_auth_done global_id 94104 payload 824 2024-04-22T08:20:08.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.701+0000 7fb169599640 1 -- 172.21.15.40:0/1168981831 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb16409c9b0 con 0x7fb16412f090 2024-04-22T08:20:08.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.701+0000 7fb159ffb640 1 --2- 172.21.15.40:0/1168981831 >> v2:172.21.15.189:6800/30168 conn(0x7fb1480377d0 0x7fb148039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:08.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.702+0000 7fb169599640 1 -- 172.21.15.40:0/1168981831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb16412f090 msgr2=0x7fb16402b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-22T08:20:08.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.702+0000 7fb169599640 1 -- 172.21.15.40:0/1168981831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb16412f090 msgr2=0x7fb16402b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:20:08.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.702+0000 7fb169599640 1 --2- 172.21.15.40:0/1168981831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb16412f090 0x7fb16402b200 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7fb160010c80 tx=0x7fb16000e5a0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:20:08.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.702+0000 7fb159ffb640 1 -- 172.21.15.40:0/1168981831 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 3506+0+0 (secure 0 0 0) 0x7fb15c06cd10 con 0x7fb164133290 2024-04-22T08:20:08.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.702+0000 7fb169599640 1 --2- 172.21.15.40:0/1168981831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb16412f090 0x7fb16402b200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:20:08.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.702+0000 7fb168d98640 1 --2- 172.21.15.40:0/1168981831 >> v2:172.21.15.189:6800/30168 conn(0x7fb1480377d0 0x7fb148039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:08.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.702+0000 7fb168d98640 10 monclient: get_auth_request con 0x7fb1480377d0 auth_method 0 2024-04-22T08:20:08.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.704+0000 7fb168d98640 1 --2- 172.21.15.40:0/1168981831 >> v2:172.21.15.189:6800/30168 conn(0x7fb1480377d0 0x7fb148039c90 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fb1640a3f30 tx=0x7fb154009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:08.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.903+0000 7fb169599640 1 --2- 172.21.15.40:0/1168981831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb16412f090 0x7fb16402b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:08.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.903+0000 7fb169599640 10 monclient: get_auth_request con 0x7fb16412f090 auth_method 0 2024-04-22T08:20:08.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.903+0000 7fb169599640 10 monclient: get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:08.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.903+0000 7fb169599640 10 monclient: _init_auth method 2 2024-04-22T08:20:08.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.903+0000 7fb169599640 10 monclient: _init_auth already have auth, reseting 2024-04-22T08:20:08.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.903+0000 7fb169599640 10 monclient: handle_auth_reply_more payload 9 2024-04-22T08:20:08.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.903+0000 7fb169599640 10 monclient: handle_auth_reply_more payload_len 9 2024-04-22T08:20:08.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.903+0000 7fb169599640 10 monclient: handle_auth_reply_more responding with 132 bytes 2024-04-22T08:20:08.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.903+0000 7fb169599640 10 monclient: handle_auth_done global_id 94104 payload 293 2024-04-22T08:20:08.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.904+0000 7fb169599640 1 -- 172.21.15.40:0/1168981831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb16412f090 msgr2=0x7fb16402b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:08.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.904+0000 7fb169599640 1 --2- 172.21.15.40:0/1168981831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb16412f090 0x7fb16402b200 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb160010c80 tx=0x7fb160008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:08.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.905+0000 7fb159ffb640 1 -- 172.21.15.40:0/1168981831 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb16000eec0 con 0x7fb16412f090 2024-04-22T08:20:08.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.905+0000 7fb159ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:08.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.905+0000 7fb159ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:20:08.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.905+0000 7fb159ffb640 10 monclient: dump: 2024-04-22T08:20:08.905 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:08.906 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:08.906 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:08.906 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:08.906 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:08.906 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:08.906 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:08.906 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:08.906 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:08.906 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:08.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.905+0000 7fb159ffb640 1 -- 172.21.15.40:0/1168981831 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7fb160004240 con 0x7fb16412f090 2024-04-22T08:20:08.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.905+0000 7fb159ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:20:08.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.905+0000 7fb159ffb640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:20:08.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.905+0000 7fb159ffb640 1 -- 172.21.15.40:0/1168981831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb16412f090 msgr2=0x7fb16402b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:08.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.905+0000 7fb159ffb640 1 --2- 172.21.15.40:0/1168981831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb16412f090 0x7fb16402b200 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb160010c80 tx=0x7fb160008040 comp rx=0 tx=0).stop 2024-04-22T08:20:08.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.905+0000 7fb159ffb640 1 -- 172.21.15.40:0/1168981831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb16412f090 msgr2=0x7fb16402b200 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:20:08.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.905+0000 7fb159ffb640 1 --2- 172.21.15.40:0/1168981831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb16412f090 0x7fb16402b200 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb160010c80 tx=0x7fb160008040 comp rx=0 tx=0).stop 2024-04-22T08:20:08.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.911+0000 7fb16b7fd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:20:08.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.911+0000 7fb16b7fd640 1 --2- 172.21.15.40:0/1168981831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb16402b740 0x7fb16402bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:08.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.911+0000 7fb16b7fd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:08.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.911+0000 7fb16b7fd640 1 -- 172.21.15.40:0/1168981831 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb16412f470 con 0x7fb16402b740 2024-04-22T08:20:08.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.912+0000 7fb169599640 1 --2- 172.21.15.40:0/1168981831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb16402b740 0x7fb16402bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:08.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.912+0000 7fb169599640 10 monclient: get_auth_request con 0x7fb16402b740 auth_method 0 2024-04-22T08:20:08.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.912+0000 7fb169599640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:08.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.912+0000 7fb169599640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:08.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.912+0000 7fb169599640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:08.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.912+0000 7fb169599640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:08.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.912+0000 7fb169599640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:08.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.912+0000 7fb169599640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:08.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.912+0000 7fb169599640 10 monclient(hunting): handle_auth_done global_id 94107 payload 824 2024-04-22T08:20:08.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.912+0000 7fb169599640 1 -- 172.21.15.40:0/1168981831 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb16413b2d0 con 0x7fb16402b740 2024-04-22T08:20:08.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.914+0000 7fb169599640 1 -- 172.21.15.40:0/1168981831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb16402b740 msgr2=0x7fb16402bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:08.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.914+0000 7fb169599640 1 --2- 172.21.15.40:0/1168981831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb16402b740 0x7fb16402bb50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb16001e000 tx=0x7fb1600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:08.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.914+0000 7fb159ffb640 1 -- 172.21.15.40:0/1168981831 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb160003a30 con 0x7fb16402b740 2024-04-22T08:20:08.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.914+0000 7fb159ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:08.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.914+0000 7fb159ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:20:08.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.914+0000 7fb159ffb640 10 monclient: dump: 2024-04-22T08:20:08.916 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:08.916 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:08.916 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:08.916 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:08.916 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:08.916 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:08.916 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:08.916 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:08.916 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:08.916 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:08.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.915+0000 7fb159ffb640 1 -- 172.21.15.40:0/1168981831 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4463 (secure 0 0 0) 0x7fb160003b90 con 0x7fb16402b740 2024-04-22T08:20:08.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.915+0000 7fb159ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:20:08.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.915+0000 7fb159ffb640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:20:08.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.915+0000 7fb159ffb640 1 -- 172.21.15.40:0/1168981831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb16402b740 msgr2=0x7fb16402bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:08.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.915+0000 7fb159ffb640 1 --2- 172.21.15.40:0/1168981831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb16402b740 0x7fb16402bb50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb16001e000 tx=0x7fb1600046c0 comp rx=0 tx=0).stop 2024-04-22T08:20:08.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.915+0000 7fb159ffb640 1 -- 172.21.15.40:0/1168981831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb16402b740 msgr2=0x7fb16402bb50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:20:08.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.915+0000 7fb159ffb640 1 --2- 172.21.15.40:0/1168981831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb16402b740 0x7fb16402bb50 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb16001e000 tx=0x7fb1600046c0 comp rx=0 tx=0).stop 2024-04-22T08:20:08.917 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:20:08.917 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:20:08.917 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:20:08.917 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "leader", 2024-04-22T08:20:08.917 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 72, 2024-04-22T08:20:08.917 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:20:08.918 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:20:08.918 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:20:08.918 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:20:08.918 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:20:08.918 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 0, 2024-04-22T08:20:08.918 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:20:08.918 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:20:08.918 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:20:08.918 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:20:08.918 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:20:08.918 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:20:08.918 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:20:08.918 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:20:08.918 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:20:08.919 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:20:08.919 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:20:08.919 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:20:08.919 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:20:08.919 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:20:08.919 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:20:08.919 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:20:08.919 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:20:08.919 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:20:08.919 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:20:08.919 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:20:08.919 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:20:08.919 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:20:08.919 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:20:08.919 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:20:08.920 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:20:08.920 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:20:08.920 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:08.920 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:08.920 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:20:08.920 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:20:08.920 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:20:08.920 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:20:08.920 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:20:08.920 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:20:08.920 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:20:08.920 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:20:08.920 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:20:08.920 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:20:08.920 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:20:08.921 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:20:08.921 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:20:08.921 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:20:08.921 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:20:08.921 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:20:08.921 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:20:08.921 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:20:08.921 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:20:08.921 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:20:08.921 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:20:08.921 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:20:08.921 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:20:08.921 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:20:08.921 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:20:08.922 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:20:08.922 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:20:08.922 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:20:08.922 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:20:08.922 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:08.922 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:20:08.922 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:08.922 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:20:08.922 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:20:08.922 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:20:08.922 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:20:08.922 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:08.922 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:20:08.923 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:20:08.923 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:08.923 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:08.923 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:08.923 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:20:08.923 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:20:08.923 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:08.923 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:08.923 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:08.923 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:08.923 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:20:08.923 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:20:08.923 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:20:08.923 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:20:08.924 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:20:08.924 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:08.924 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:08.924 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:20:08.924 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:20:08.924 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:20:08.924 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:20:08.924 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:08.924 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:20:08.924 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:20:08.924 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:08.924 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:08.924 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:08.924 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:20:08.925 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:20:08.925 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:08.925 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:08.925 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:08.925 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:08.925 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:20:08.925 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:20:08.925 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:20:08.925 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:20:08.925 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:20:08.925 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:08.925 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:08.925 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:20:08.925 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:20:08.926 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:20:08.926 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:20:08.926 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:08.926 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:20:08.926 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:20:08.926 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:08.926 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:08.926 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:08.926 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:20:08.926 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:20:08.926 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:08.926 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:08.926 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:08.927 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:08.927 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:20:08.927 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:20:08.927 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:20:08.927 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:20:08.927 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:20:08.927 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:08.927 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:08.927 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:08.927 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:20:08.927 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:20:08.927 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:08.927 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:20:08.928 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:20:08.928 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:20:08.928 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:08.928 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:20:08.928 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:20:08.928 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:08.928 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:20:08.928 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:20:08.928 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:20:08.928 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:08.928 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:08.928 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:08.928 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:20:08.928 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:20:08.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.915+0000 7fb16b7fd640 1 -- 172.21.15.40:0/1168981831 >> v2:172.21.15.189:6800/30168 conn(0x7fb1480377d0 msgr2=0x7fb148039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:08.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.915+0000 7fb16b7fd640 1 --2- 172.21.15.40:0/1168981831 >> v2:172.21.15.189:6800/30168 conn(0x7fb1480377d0 0x7fb148039c90 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fb1640a3f30 tx=0x7fb154009190 comp rx=0 tx=0).stop 2024-04-22T08:20:08.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.915+0000 7fb16b7fd640 10 monclient: shutdown 2024-04-22T08:20:08.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.915+0000 7fb16b7fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:20:08.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.915+0000 7fb16b7fd640 1 -- 172.21.15.40:0/1168981831 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb164133290 msgr2=0x7fb164137bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:08.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.915+0000 7fb16b7fd640 1 --2- 172.21.15.40:0/1168981831 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb164133290 0x7fb164137bc0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb16409d1b0 tx=0x7fb15c004a40 comp rx=0 tx=0).stop 2024-04-22T08:20:08.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.915+0000 7fb169d9a640 1 -- 172.21.15.40:0/1168981831 reap_dead start 2024-04-22T08:20:08.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.915+0000 7fb169d9a640 1 -- 172.21.15.40:0/1168981831 reap_dead start 2024-04-22T08:20:08.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.915+0000 7fb16b7fd640 1 -- 172.21.15.40:0/1168981831 shutdown_connections 2024-04-22T08:20:08.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.915+0000 7fb16b7fd640 1 -- 172.21.15.40:0/1168981831 >> 172.21.15.40:0/1168981831 conn(0x7fb1640a0ae0 msgr2=0x7fb1640a2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:20:08.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.915+0000 7fb16b7fd640 1 -- 172.21.15.40:0/1168981831 shutdown_connections 2024-04-22T08:20:08.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:08.915+0000 7fb16b7fd640 1 -- 172.21.15.40:0/1168981831 wait complete. 2024-04-22T08:20:08.931 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:20:08.931 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-22T08:20:08.931 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-22T08:20:09.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.032+0000 7f7b9cd75640 10 monclient: get_monmap_and_config 2024-04-22T08:20:09.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.032+0000 7f7b9cd75640 10 monclient: build_initial_monmap 2024-04-22T08:20:09.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.032+0000 7f7b9cd75640 10 monclient: monmap: 2024-04-22T08:20:09.033 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:20:09.033 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:09.033 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:20:09.034468+0000 2024-04-22T08:20:09.033 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:20:09.034468+0000 2024-04-22T08:20:09.033 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:20:09.033 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:09.033 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:20:09.034 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:20:09.034 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:20:09.034 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:20:09.034 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:20:09.034 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:09.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.033+0000 7f7b9cd75640 1 Processor -- start 2024-04-22T08:20:09.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.033+0000 7f7b9cd75640 1 -- start start 2024-04-22T08:20:09.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.033+0000 7f7b9cd75640 10 monclient: init 2024-04-22T08:20:09.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.033+0000 7f7b9cd75640 10 monclient: _reopen_session rank -1 2024-04-22T08:20:09.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.033+0000 7f7b9cd75640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-04-22T08:20:09.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.033+0000 7f7b9cd75640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7b980a1a80 0x7f7b980a01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:09.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.033+0000 7f7b9cd75640 10 monclient(hunting): picked mon.noname-a con 0x7f7b980a1a80 addr v2:172.21.15.40:3300/0 2024-04-22T08:20:09.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.033+0000 7f7b9cd75640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7b980a1e60 0x7f7b980a2260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:09.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.033+0000 7f7b9cd75640 10 monclient(hunting): picked mon.noname-b con 0x7f7b980a1e60 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:20:09.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.033+0000 7f7b9cd75640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7b98135c20 0x7f7b981360b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:09.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.033+0000 7f7b9cd75640 10 monclient(hunting): picked mon.noname-c con 0x7f7b98135c20 addr v2:172.21.15.189:3300/0 2024-04-22T08:20:09.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.033+0000 7f7b9cd75640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:09.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.033+0000 7f7b9cd75640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b9809f880 con 0x7f7b980a1a80 2024-04-22T08:20:09.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.033+0000 7f7b9cd75640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:09.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.033+0000 7f7b9cd75640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b9809d860 con 0x7f7b98135c20 2024-04-22T08:20:09.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.033+0000 7f7b9cd75640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:09.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.033+0000 7f7b9cd75640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9809ef50 con 0x7f7b980a1e60 2024-04-22T08:20:09.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.033+0000 7f7b9cd75640 10 monclient(hunting): _renew_subs 2024-04-22T08:20:09.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.033+0000 7f7b9cd75640 10 monclient(hunting): authenticate will time out at 1283.028554s 2024-04-22T08:20:09.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.033+0000 7f7b9659c640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7b980a1a80 0x7f7b980a01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:09.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.033+0000 7f7b8dd9b640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7b980a1e60 0x7f7b980a2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:09.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.033+0000 7f7b8dd9b640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7b980a1e60 0x7f7b980a2260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:57970/0 (socket says 172.21.15.40:57970) 2024-04-22T08:20:09.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.033+0000 7f7b9659c640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7b980a1a80 0x7f7b980a01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46304/0 (socket says 172.21.15.40:46304) 2024-04-22T08:20:09.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.033+0000 7f7b8dd9b640 1 -- 172.21.15.40:0/213261739 learned_addr learned my addr 172.21.15.40:0/213261739 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:20:09.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.033+0000 7f7b8dd9b640 10 monclient(hunting): get_auth_request con 0x7f7b980a1e60 auth_method 0 2024-04-22T08:20:09.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.033+0000 7f7b8dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:09.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.033+0000 7f7b8dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:09.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.033+0000 7f7b8dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:09.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.033+0000 7f7b9659c640 10 monclient(hunting): get_auth_request con 0x7f7b980a1a80 auth_method 0 2024-04-22T08:20:09.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.033+0000 7f7b9659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:09.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.033+0000 7f7b9659c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:09.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.033+0000 7f7b9659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:09.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.034+0000 7f7b8dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:09.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.034+0000 7f7b8dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:09.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.034+0000 7f7b8dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:09.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.034+0000 7f7b9659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:09.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.034+0000 7f7b9659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:09.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.034+0000 7f7b9659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:09.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.034+0000 7f7b96d9d640 1 --2- 172.21.15.40:0/213261739 >> v2:172.21.15.189:3300/0 conn(0x7f7b98135c20 0x7f7b981360b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:09.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.034+0000 7f7b96d9d640 10 monclient(hunting): get_auth_request con 0x7f7b98135c20 auth_method 0 2024-04-22T08:20:09.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.034+0000 7f7b96d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:09.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.034+0000 7f7b96d9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:09.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.034+0000 7f7b96d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:09.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.034+0000 7f7b8dd9b640 10 monclient(hunting): handle_auth_done global_id 54827 payload 274 2024-04-22T08:20:09.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.034+0000 7f7b8dd9b640 1 -- 172.21.15.40:0/213261739 >> v2:172.21.15.189:3300/0 conn(0x7f7b98135c20 msgr2=0x7f7b981360b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:20:09.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.034+0000 7f7b8dd9b640 1 --2- 172.21.15.40:0/213261739 >> v2:172.21.15.189:3300/0 conn(0x7f7b98135c20 0x7f7b981360b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:09.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.034+0000 7f7b8dd9b640 1 -- 172.21.15.40:0/213261739 >> v2:172.21.15.40:3300/0 conn(0x7f7b980a1a80 msgr2=0x7f7b980a01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:20:09.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.034+0000 7f7b8dd9b640 1 --2- 172.21.15.40:0/213261739 >> v2:172.21.15.40:3300/0 conn(0x7f7b980a1a80 0x7f7b980a01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:09.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.034+0000 7f7b8dd9b640 10 monclient: _finish_hunting 0 2024-04-22T08:20:09.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.034+0000 7f7b8dd9b640 1 monclient: found mon.noname-b 2024-04-22T08:20:09.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.034+0000 7f7b8dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:20:09.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.034+0000 7f7b8dd9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:20:09.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.034+0000 7f7b8dd9b640 1 -- 172.21.15.40:0/213261739 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b98072160 con 0x7f7b980a1e60 2024-04-22T08:20:09.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.034+0000 7f7b9659c640 1 --2- 172.21.15.40:0/213261739 >> v2:172.21.15.40:3300/0 conn(0x7f7b980a1a80 0x7f7b980a01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:20:09.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.037+0000 7f7b8dd9b640 1 -- 172.21.15.40:0/213261739 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7b980a1e60 msgr2=0x7f7b980a2260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:09.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.037+0000 7f7b8dd9b640 1 --2- 172.21.15.40:0/213261739 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7b980a1e60 0x7f7b980a2260 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7b9809fca0 tx=0x7f7b8002b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea07f34237cad945 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:09.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.037+0000 7f7b95d9b640 1 -- 172.21.15.40:0/213261739 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b80031950 con 0x7f7b980a1e60 2024-04-22T08:20:09.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.037+0000 7f7b95d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:09.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.037+0000 7f7b95d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:20:09.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.037+0000 7f7b95d9b640 10 monclient: dump: 2024-04-22T08:20:09.041 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:09.041 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:09.042 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:09.042 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:09.042 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:09.042 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:09.042 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:09.042 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:09.042 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:09.042 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:09.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.037+0000 7f7b95d9b640 10 monclient: _finish_auth 0 2024-04-22T08:20:09.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.037+0000 7f7b95d9b640 10 monclient: _check_auth_tickets 2024-04-22T08:20:09.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.037+0000 7f7b95d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:20:09.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.037+0000 7f7b95d9b640 1 -- 172.21.15.40:0/213261739 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b80031ab0 con 0x7f7b980a1e60 2024-04-22T08:20:09.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.037+0000 7f7b9cd75640 5 monclient: authenticate success, global_id 54827 2024-04-22T08:20:09.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.037+0000 7f7b9cd75640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:20:09.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.037+0000 7f7b95d9b640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:20:09.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.037+0000 7f7b9cd75640 10 monclient: get_monmap_and_config success 2024-04-22T08:20:09.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.037+0000 7f7b95d9b640 1 -- 172.21.15.40:0/213261739 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b80031e40 con 0x7f7b980a1e60 2024-04-22T08:20:09.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.037+0000 7f7b95d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:09.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.037+0000 7f7b9cd75640 4 set_mon_vals no callback set 2024-04-22T08:20:09.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.037+0000 7f7b95d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:20:09.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.037+0000 7f7b95d9b640 10 monclient: dump: 2024-04-22T08:20:09.043 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:09.043 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:09.043 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:09.043 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:09.043 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:09.043 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:09.043 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:09.043 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:09.044 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:09.044 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:09.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.037+0000 7f7b9cd75640 10 monclient: shutdown 2024-04-22T08:20:09.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.037+0000 7f7b9cd75640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:20:09.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.037+0000 7f7b9cd75640 1 -- 172.21.15.40:0/213261739 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7b980a1e60 msgr2=0x7f7b980a2260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:09.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.037+0000 7f7b9cd75640 1 --2- 172.21.15.40:0/213261739 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7b980a1e60 0x7f7b980a2260 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7b9809fca0 tx=0x7f7b8002b7e0 comp rx=0 tx=0).stop 2024-04-22T08:20:09.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.037+0000 7f7b9cd75640 1 -- 172.21.15.40:0/213261739 shutdown_connections 2024-04-22T08:20:09.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.037+0000 7f7b9cd75640 1 --2- 172.21.15.40:0/213261739 >> v2:172.21.15.189:3300/0 conn(0x7f7b98135c20 0x7f7b981360b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:09.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.037+0000 7f7b9cd75640 1 --2- 172.21.15.40:0/213261739 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7b980a1e60 0x7f7b980a2260 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:09.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.037+0000 7f7b9cd75640 1 --2- 172.21.15.40:0/213261739 >> v2:172.21.15.40:3300/0 conn(0x7f7b980a1a80 0x7f7b980a01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:09.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.037+0000 7f7b9cd75640 1 -- 172.21.15.40:0/213261739 >> 172.21.15.40:0/213261739 conn(0x7f7b980a66a0 msgr2=0x7f7b980a2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:20:09.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.037+0000 7f7b9cd75640 1 -- 172.21.15.40:0/213261739 shutdown_connections 2024-04-22T08:20:09.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.037+0000 7f7b9cd75640 1 -- 172.21.15.40:0/213261739 wait complete. 2024-04-22T08:20:09.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.038+0000 7f7b9cd75640 10 monclient: build_initial_monmap 2024-04-22T08:20:09.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.038+0000 7f7b9cd75640 10 monclient: monmap: 2024-04-22T08:20:09.050 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:20:09.050 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:20:09.050 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:20:09.051 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:20:09.051 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:20:09.051 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:09.051 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:20:09.051 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:20:09.051 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:20:09.051 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:09.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.038+0000 7f7b9cd75640 1 Processor -- start 2024-04-22T08:20:09.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b9cd75640 1 -- start start 2024-04-22T08:20:09.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b9cd75640 10 monclient: init 2024-04-22T08:20:09.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b9cd75640 10 monclient: _reopen_session rank -1 2024-04-22T08:20:09.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b9cd75640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:20:09.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b9cd75640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b980a1a80 0x7f7b981caaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:09.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b9cd75640 10 monclient(hunting): picked mon.noname-a con 0x7f7b980a1a80 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:20:09.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b9cd75640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b980a1e60 0x7f7b981cb030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:09.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b9cd75640 10 monclient(hunting): picked mon.noname-b con 0x7f7b980a1e60 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:20:09.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b9cd75640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7b98135c20 0x7f7b981220a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:09.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b9cd75640 10 monclient(hunting): picked mon.noname-c con 0x7f7b98135c20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:20:09.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b9cd75640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:09.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b9cd75640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b98124290 con 0x7f7b980a1a80 2024-04-22T08:20:09.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b9cd75640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:09.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b9cd75640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b98125e00 con 0x7f7b980a1e60 2024-04-22T08:20:09.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b9cd75640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:09.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b9cd75640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b98126050 con 0x7f7b98135c20 2024-04-22T08:20:09.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b9cd75640 10 monclient(hunting): _renew_subs 2024-04-22T08:20:09.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b9cd75640 10 monclient(hunting): authenticate will time out at 1283.034227s 2024-04-22T08:20:09.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b96d9d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7b98135c20 0x7f7b981220a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:09.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b9659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b980a1a80 0x7f7b981caaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:09.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b9659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b980a1a80 0x7f7b981caaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46308/0 (socket says 172.21.15.40:46308) 2024-04-22T08:20:09.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b96d9d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7b98135c20 0x7f7b981220a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:57976/0 (socket says 172.21.15.40:57976) 2024-04-22T08:20:09.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b9659c640 1 -- 172.21.15.40:0/544092400 learned_addr learned my addr 172.21.15.40:0/544092400 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:20:09.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b9659c640 10 monclient(hunting): get_auth_request con 0x7f7b980a1a80 auth_method 0 2024-04-22T08:20:09.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b9659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:09.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b9659c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:09.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b9659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:09.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b96d9d640 10 monclient(hunting): get_auth_request con 0x7f7b98135c20 auth_method 0 2024-04-22T08:20:09.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b96d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:09.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b96d9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:09.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b96d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:09.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b9659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:09.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b9659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:09.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b9659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:09.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b8dd9b640 1 --2- 172.21.15.40:0/544092400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b980a1e60 0x7f7b981cb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:09.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b8dd9b640 10 monclient(hunting): get_auth_request con 0x7f7b980a1e60 auth_method 0 2024-04-22T08:20:09.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b8dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:09.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b8dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:09.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b8dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:09.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b96d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:09.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b96d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:09.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b96d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:09.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b8dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:09.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b8dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:09.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b8dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:09.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b9659c640 10 monclient(hunting): handle_auth_done global_id 94113 payload 824 2024-04-22T08:20:09.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b9659c640 1 -- 172.21.15.40:0/544092400 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7b98135c20 msgr2=0x7f7b981220a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:09.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b9659c640 1 --2- 172.21.15.40:0/544092400 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7b98135c20 0x7f7b981220a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:09.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b9659c640 1 -- 172.21.15.40:0/544092400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b980a1e60 msgr2=0x7f7b981cb030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:09.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b9659c640 1 --2- 172.21.15.40:0/544092400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b980a1e60 0x7f7b981cb030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:09.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b9659c640 10 monclient: _finish_hunting 0 2024-04-22T08:20:09.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b9659c640 1 monclient: found mon.noname-a 2024-04-22T08:20:09.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b9659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:20:09.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b9659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:20:09.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.039+0000 7f7b9659c640 1 -- 172.21.15.40:0/544092400 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b88009400 con 0x7f7b980a1a80 2024-04-22T08:20:09.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.042+0000 7f7b9659c640 1 -- 172.21.15.40:0/544092400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b980a1a80 msgr2=0x7f7b981caaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:09.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.042+0000 7f7b9659c640 1 --2- 172.21.15.40:0/544092400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b980a1a80 0x7f7b981caaf0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7b8800ebe0 tx=0x7f7b8800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:09.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.042+0000 7f7b8f7fe640 1 -- 172.21.15.40:0/544092400 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b88009760 con 0x7f7b980a1a80 2024-04-22T08:20:09.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.042+0000 7f7b8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:09.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.042+0000 7f7b8f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:20:09.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.042+0000 7f7b8f7fe640 10 monclient: dump: 2024-04-22T08:20:09.055 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:09.055 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:09.056 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:09.056 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:09.056 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:09.056 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:09.056 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:09.056 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:09.056 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:09.056 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:09.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.042+0000 7f7b8f7fe640 10 monclient: _finish_auth 0 2024-04-22T08:20:09.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.042+0000 7f7b8f7fe640 10 monclient: _check_auth_tickets 2024-04-22T08:20:09.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.042+0000 7f7b8f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:20:09.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.042+0000 7f7b8f7fe640 1 -- 172.21.15.40:0/544092400 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b8800ce40 con 0x7f7b980a1a80 2024-04-22T08:20:09.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.042+0000 7f7b8f7fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:20:09.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.042+0000 7f7b8f7fe640 1 -- 172.21.15.40:0/544092400 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b8800eee0 con 0x7f7b980a1a80 2024-04-22T08:20:09.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.042+0000 7f7b8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:09.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.042+0000 7f7b8f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:20:09.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.042+0000 7f7b8f7fe640 10 monclient: dump: 2024-04-22T08:20:09.057 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:09.057 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:09.057 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:09.057 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:09.057 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:09.057 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:09.057 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:09.057 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:09.057 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:09.057 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:09.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.042+0000 7f7b9cd75640 5 monclient: authenticate success, global_id 94113 2024-04-22T08:20:09.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.042+0000 7f7b9cd75640 10 monclient: _renew_subs 2024-04-22T08:20:09.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.042+0000 7f7b9cd75640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:20:09.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.042+0000 7f7b9cd75640 1 -- 172.21.15.40:0/544092400 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b800094d0 con 0x7f7b980a1a80 2024-04-22T08:20:09.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.042+0000 7f7b9559a640 4 set_mon_vals no callback set 2024-04-22T08:20:09.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.042+0000 7f7b9cd75640 10 monclient: _renew_subs 2024-04-22T08:20:09.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.042+0000 7f7b9cd75640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:20:09.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.042+0000 7f7b9cd75640 1 -- 172.21.15.40:0/544092400 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b9809c180 con 0x7f7b980a1a80 2024-04-22T08:20:09.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.043+0000 7f7b9cd75640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:20:09.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.043+0000 7f7b8f7fe640 1 -- 172.21.15.40:0/544092400 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f7b88013690 con 0x7f7b980a1a80 2024-04-22T08:20:09.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.043+0000 7f7b9cd75640 1 --2- 172.21.15.40:0/544092400 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7b981225e0 0x7f7b981229c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:09.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.043+0000 7f7b9cd75640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:09.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.043+0000 7f7b9cd75640 1 -- 172.21.15.40:0/544092400 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b981261e0 con 0x7f7b981225e0 2024-04-22T08:20:09.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.043+0000 7f7b8dd9b640 1 --2- 172.21.15.40:0/544092400 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7b981225e0 0x7f7b981229c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:09.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.043+0000 7f7b8dd9b640 10 monclient: get_auth_request con 0x7f7b981225e0 auth_method 0 2024-04-22T08:20:09.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.043+0000 7f7b8dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:09.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.043+0000 7f7b8dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:09.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.043+0000 7f7b8dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:09.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.043+0000 7f7b8dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:09.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.043+0000 7f7b8f7fe640 1 --2- 172.21.15.40:0/544092400 >> v2:172.21.15.189:6800/30168 conn(0x7f7b7c0377d0 0x7f7b7c039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:09.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.043+0000 7f7b8dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:09.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.043+0000 7f7b8dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:09.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.043+0000 7f7b8f7fe640 1 -- 172.21.15.40:0/544092400 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f7b8804a630 con 0x7f7b980a1a80 2024-04-22T08:20:09.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.043+0000 7f7b8dd9b640 10 monclient(hunting): handle_auth_done global_id 54833 payload 824 2024-04-22T08:20:09.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.043+0000 7f7b8dd9b640 1 -- 172.21.15.40:0/544092400 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7b980a27a0 con 0x7f7b981225e0 2024-04-22T08:20:09.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.043+0000 7f7b96d9d640 1 --2- 172.21.15.40:0/544092400 >> v2:172.21.15.189:6800/30168 conn(0x7f7b7c0377d0 0x7f7b7c039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:09.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.043+0000 7f7b96d9d640 10 monclient: get_auth_request con 0x7f7b7c0377d0 auth_method 0 2024-04-22T08:20:09.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.045+0000 7f7b8dd9b640 1 -- 172.21.15.40:0/544092400 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7b981225e0 msgr2=0x7f7b981229c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:09.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.045+0000 7f7b8dd9b640 1 --2- 172.21.15.40:0/544092400 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7b981225e0 0x7f7b981229c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7b80035000 tx=0x7f7b800046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:09.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.046+0000 7f7b8f7fe640 1 -- 172.21.15.40:0/544092400 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b80033c90 con 0x7f7b981225e0 2024-04-22T08:20:09.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.046+0000 7f7b8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:09.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.046+0000 7f7b8f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:20:09.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.046+0000 7f7b8f7fe640 10 monclient: dump: 2024-04-22T08:20:09.060 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:09.060 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:09.060 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:09.060 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:09.060 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:09.060 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:09.060 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:09.060 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:09.060 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:09.060 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:09.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.046+0000 7f7b8f7fe640 1 -- 172.21.15.40:0/544092400 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f7b80041070 con 0x7f7b981225e0 2024-04-22T08:20:09.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.046+0000 7f7b8f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:20:09.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.046+0000 7f7b8f7fe640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:20:09.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.046+0000 7f7b8f7fe640 1 -- 172.21.15.40:0/544092400 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7b981225e0 msgr2=0x7f7b981229c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:09.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.046+0000 7f7b8f7fe640 1 --2- 172.21.15.40:0/544092400 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7b981225e0 0x7f7b981229c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7b80035000 tx=0x7f7b800046c0 comp rx=0 tx=0).stop 2024-04-22T08:20:09.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.046+0000 7f7b8f7fe640 1 -- 172.21.15.40:0/544092400 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7b981225e0 msgr2=0x7f7b981229c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:20:09.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.046+0000 7f7b8f7fe640 1 --2- 172.21.15.40:0/544092400 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7b981225e0 0x7f7b981229c0 secure :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7b80035000 tx=0x7f7b800046c0 comp rx=0 tx=0).stop 2024-04-22T08:20:09.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.046+0000 7f7b96d9d640 1 --2- 172.21.15.40:0/544092400 >> v2:172.21.15.189:6800/30168 conn(0x7f7b7c0377d0 0x7f7b7c039c90 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f7b98135bf0 tx=0x7f7b90009820 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:09.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.052+0000 7f7b9cd75640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:20:09.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.052+0000 7f7b9cd75640 1 --2- 172.21.15.40:0/544092400 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7b98122f00 0x7f7b9806fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:09.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.052+0000 7f7b9cd75640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:09.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.052+0000 7f7b9cd75640 1 -- 172.21.15.40:0/544092400 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b98066770 con 0x7f7b98122f00 2024-04-22T08:20:09.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.052+0000 7f7b8dd9b640 1 --2- 172.21.15.40:0/544092400 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7b98122f00 0x7f7b9806fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:09.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.052+0000 7f7b8dd9b640 10 monclient: get_auth_request con 0x7f7b98122f00 auth_method 0 2024-04-22T08:20:09.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.052+0000 7f7b8dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:09.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.052+0000 7f7b8dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:09.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.052+0000 7f7b8dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:09.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.052+0000 7f7b8dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:09.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.052+0000 7f7b8dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:09.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.052+0000 7f7b8dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:09.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.052+0000 7f7b8dd9b640 10 monclient(hunting): handle_auth_done global_id 54836 payload 824 2024-04-22T08:20:09.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.052+0000 7f7b8dd9b640 1 -- 172.21.15.40:0/544092400 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7b9809dc10 con 0x7f7b98122f00 2024-04-22T08:20:09.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.056+0000 7f7b8dd9b640 1 -- 172.21.15.40:0/544092400 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7b98122f00 msgr2=0x7f7b9806fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:09.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.056+0000 7f7b8dd9b640 1 --2- 172.21.15.40:0/544092400 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7b98122f00 0x7f7b9806fb90 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7b8002b820 tx=0x7f7b80031000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:09.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.056+0000 7f7b8f7fe640 1 -- 172.21.15.40:0/544092400 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b80050480 con 0x7f7b98122f00 2024-04-22T08:20:09.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.056+0000 7f7b8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:09.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.056+0000 7f7b8f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:20:09.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.056+0000 7f7b8f7fe640 10 monclient: dump: 2024-04-22T08:20:09.064 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:09.064 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:09.064 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:09.064 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:09.064 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:09.065 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:09.065 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:09.065 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:09.065 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:09.065 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:09.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.056+0000 7f7b8f7fe640 1 -- 172.21.15.40:0/544092400 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4631 (secure 0 0 0) 0x7f7b800505e0 con 0x7f7b98122f00 2024-04-22T08:20:09.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.056+0000 7f7b8f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:20:09.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.056+0000 7f7b8f7fe640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:20:09.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.056+0000 7f7b8f7fe640 1 -- 172.21.15.40:0/544092400 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7b98122f00 msgr2=0x7f7b9806fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:09.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.056+0000 7f7b8f7fe640 1 --2- 172.21.15.40:0/544092400 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7b98122f00 0x7f7b9806fb90 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7b8002b820 tx=0x7f7b80031000 comp rx=0 tx=0).stop 2024-04-22T08:20:09.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.056+0000 7f7b8f7fe640 1 -- 172.21.15.40:0/544092400 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7b98122f00 msgr2=0x7f7b9806fb90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:20:09.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.056+0000 7f7b8f7fe640 1 --2- 172.21.15.40:0/544092400 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7b98122f00 0x7f7b9806fb90 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:09.066 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:20:09.066 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:20:09.066 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:20:09.066 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:20:09.066 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 72, 2024-04-22T08:20:09.066 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:20:09.066 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:20:09.066 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:20:09.066 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:20:09.066 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:20:09.066 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 0, 2024-04-22T08:20:09.066 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:20:09.067 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:20:09.067 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:20:09.067 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:20:09.067 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:20:09.067 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:20:09.067 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:20:09.067 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:20:09.067 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:20:09.067 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:20:09.067 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:20:09.067 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:20:09.067 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:20:09.067 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:20:09.067 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:20:09.068 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:20:09.068 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:20:09.068 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:20:09.068 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:20:09.068 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:20:09.068 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:20:09.068 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:20:09.068 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:20:09.068 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:20:09.068 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:20:09.068 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:20:09.068 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:09.068 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:09.068 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:20:09.068 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:20:09.069 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:20:09.069 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:20:09.069 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:20:09.069 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:20:09.069 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:20:09.069 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:20:09.069 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:20:09.069 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:20:09.069 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:20:09.069 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:20:09.069 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:20:09.069 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:20:09.069 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:20:09.069 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:20:09.069 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:20:09.070 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:20:09.070 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:20:09.070 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:20:09.070 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:20:09.070 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:20:09.070 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:20:09.070 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:20:09.070 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:20:09.070 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:20:09.070 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:20:09.070 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:20:09.070 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:20:09.070 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:09.070 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:20:09.071 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:09.071 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:20:09.071 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:20:09.071 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:20:09.071 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:20:09.071 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:09.071 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:20:09.071 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:20:09.071 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:09.071 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:09.071 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:09.071 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:20:09.071 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:20:09.071 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:09.071 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:09.072 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:09.072 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:09.072 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:20:09.072 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:20:09.072 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:20:09.072 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:20:09.072 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:20:09.072 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:09.072 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:09.072 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:20:09.072 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:20:09.072 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:20:09.072 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:20:09.072 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:09.073 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:20:09.073 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:20:09.073 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:09.073 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:09.073 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:09.073 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:20:09.073 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:20:09.073 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:09.073 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:09.073 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:09.073 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:09.073 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:20:09.073 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:20:09.073 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:20:09.073 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:20:09.074 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:20:09.074 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:09.074 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:09.074 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:20:09.074 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:20:09.074 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:20:09.074 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:20:09.074 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:09.074 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:20:09.074 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:20:09.074 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:09.074 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:09.074 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:09.074 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:20:09.075 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:20:09.075 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:09.075 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:09.075 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:09.075 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:09.075 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:20:09.075 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:20:09.075 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:20:09.075 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:20:09.075 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:20:09.075 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:09.075 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:09.075 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:09.075 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:20:09.075 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:20:09.076 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:09.076 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:20:09.076 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:20:09.076 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:20:09.076 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:09.076 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:20:09.076 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:20:09.076 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:09.076 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:20:09.076 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:20:09.076 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 3 2024-04-22T08:20:09.076 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:09.076 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:20:09.076 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:20:09.076 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:09.077 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:20:09.077 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:20:09.077 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 4 2024-04-22T08:20:09.077 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:09.077 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:09.077 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:09.077 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:20:09.077 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:20:09.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.056+0000 7f7b9cd75640 1 -- 172.21.15.40:0/544092400 >> v2:172.21.15.189:6800/30168 conn(0x7f7b7c0377d0 msgr2=0x7f7b7c039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:09.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.056+0000 7f7b9cd75640 1 --2- 172.21.15.40:0/544092400 >> v2:172.21.15.189:6800/30168 conn(0x7f7b7c0377d0 0x7f7b7c039c90 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f7b98135bf0 tx=0x7f7b90009820 comp rx=0 tx=0).stop 2024-04-22T08:20:09.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.056+0000 7f7b9cd75640 10 monclient: shutdown 2024-04-22T08:20:09.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.056+0000 7f7b9cd75640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:20:09.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.056+0000 7f7b9cd75640 1 -- 172.21.15.40:0/544092400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b980a1a80 msgr2=0x7f7b981caaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:09.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.056+0000 7f7b9cd75640 1 --2- 172.21.15.40:0/544092400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b980a1a80 0x7f7b981caaf0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7b8800ebe0 tx=0x7f7b8800c520 comp rx=0 tx=0).stop 2024-04-22T08:20:09.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.056+0000 7f7b96d9d640 1 -- 172.21.15.40:0/544092400 reap_dead start 2024-04-22T08:20:09.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.056+0000 7f7b96d9d640 1 -- 172.21.15.40:0/544092400 reap_dead start 2024-04-22T08:20:09.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.056+0000 7f7b9cd75640 1 -- 172.21.15.40:0/544092400 shutdown_connections 2024-04-22T08:20:09.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.056+0000 7f7b9cd75640 1 -- 172.21.15.40:0/544092400 >> 172.21.15.40:0/544092400 conn(0x7f7b980a66a0 msgr2=0x7f7b9809a650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:20:09.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.056+0000 7f7b9cd75640 1 -- 172.21.15.40:0/544092400 shutdown_connections 2024-04-22T08:20:09.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.056+0000 7f7b9cd75640 1 -- 172.21.15.40:0/544092400 wait complete. 2024-04-22T08:20:09.079 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:20:09.080 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-22T08:20:09.080 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-22T08:20:09.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.183+0000 7fe2c51d7640 10 monclient: get_monmap_and_config 2024-04-22T08:20:09.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.183+0000 7fe2c51d7640 10 monclient: build_initial_monmap 2024-04-22T08:20:09.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.183+0000 7fe2c51d7640 10 monclient: monmap: 2024-04-22T08:20:09.184 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:20:09.184 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:09.184 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:20:09.185563+0000 2024-04-22T08:20:09.184 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:20:09.185563+0000 2024-04-22T08:20:09.185 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:20:09.185 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:09.185 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:20:09.185 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:20:09.185 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:20:09.185 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:20:09.185 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:20:09.185 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:09.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.184+0000 7fe2c51d7640 1 Processor -- start 2024-04-22T08:20:09.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.184+0000 7fe2c51d7640 1 -- start start 2024-04-22T08:20:09.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.184+0000 7fe2c51d7640 10 monclient: init 2024-04-22T08:20:09.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.184+0000 7fe2c51d7640 10 monclient: _reopen_session rank -1 2024-04-22T08:20:09.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.184+0000 7fe2c51d7640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-04-22T08:20:09.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.184+0000 7fe2c51d7640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe2c012fad0 0x7fe2c012feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:09.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.184+0000 7fe2c51d7640 10 monclient(hunting): picked mon.noname-c con 0x7fe2c012fad0 addr v2:172.21.15.189:3300/0 2024-04-22T08:20:09.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.184+0000 7fe2c51d7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe2c0132920 addr v1:172.21.15.40:6789/0 2024-04-22T08:20:09.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.184+0000 7fe2c51d7640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe2c0133460 0x7fe2c013ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:09.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.184+0000 7fe2c51d7640 10 monclient(hunting): picked mon.noname-a con 0x7fe2c0133460 addr v2:172.21.15.40:3300/0 2024-04-22T08:20:09.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.184+0000 7fe2c51d7640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2c009ae50 con 0x7fe2c0132920 2024-04-22T08:20:09.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.184+0000 7fe2c51d7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:09.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.184+0000 7fe2c51d7640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2c009efc0 con 0x7fe2c0133460 2024-04-22T08:20:09.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.184+0000 7fe2c51d7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:09.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.184+0000 7fe2c51d7640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2c009d070 con 0x7fe2c012fad0 2024-04-22T08:20:09.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.184+0000 7fe2c51d7640 10 monclient(hunting): _renew_subs 2024-04-22T08:20:09.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.184+0000 7fe2c51d7640 10 monclient(hunting): authenticate will time out at 1283.179686s 2024-04-22T08:20:09.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.185+0000 7fe2bf59e640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe2c0133460 0x7fe2c013ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:09.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.185+0000 7fe2be59c640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fe2c0132920 0x7fe2c0132d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53782/0 (socket says 172.21.15.40:53782) 2024-04-22T08:20:09.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.185+0000 7fe2bf59e640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe2c0133460 0x7fe2c013ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46320/0 (socket says 172.21.15.40:46320) 2024-04-22T08:20:09.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.185+0000 7fe2be59c640 1 -- 172.21.15.40:0/2929802211 learned_addr learned my addr 172.21.15.40:0/2929802211 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:20:09.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.185+0000 7fe2bf59e640 10 monclient(hunting): get_auth_request con 0x7fe2c0133460 auth_method 0 2024-04-22T08:20:09.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.185+0000 7fe2bf59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:09.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.185+0000 7fe2bf59e640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:09.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.185+0000 7fe2bf59e640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:09.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.185+0000 7fe2bf59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:09.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.185+0000 7fe2bf59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:09.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.185+0000 7fe2bf59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:09.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.185+0000 7fe2bed9d640 1 --2- 172.21.15.40:0/2929802211 >> v2:172.21.15.189:3300/0 conn(0x7fe2c012fad0 0x7fe2c012feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:09.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.185+0000 7fe2bf59e640 10 monclient(hunting): handle_auth_done global_id 94116 payload 274 2024-04-22T08:20:09.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.185+0000 7fe2bf59e640 1 -- 172.21.15.40:0/2929802211 >> v2:172.21.15.189:3300/0 conn(0x7fe2c012fad0 msgr2=0x7fe2c012feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:20:09.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.185+0000 7fe2bf59e640 1 --2- 172.21.15.40:0/2929802211 >> v2:172.21.15.189:3300/0 conn(0x7fe2c012fad0 0x7fe2c012feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:09.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.185+0000 7fe2bf59e640 1 -- 172.21.15.40:0/2929802211 >> v1:172.21.15.40:6789/0 conn(0x7fe2c0132920 legacy=0x7fe2c0132d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:20:09.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.185+0000 7fe2bf59e640 10 monclient: _finish_hunting 0 2024-04-22T08:20:09.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.185+0000 7fe2bf59e640 1 monclient: found mon.noname-a 2024-04-22T08:20:09.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.185+0000 7fe2bf59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:20:09.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.185+0000 7fe2bf59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:20:09.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.185+0000 7fe2bf59e640 1 -- 172.21.15.40:0/2929802211 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c009d350 con 0x7fe2c0133460 2024-04-22T08:20:09.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.185+0000 7fe2bed9d640 10 monclient: get_auth_request con 0x7fe2c012fad0 auth_method 0 2024-04-22T08:20:09.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.185+0000 7fe2bed9d640 1 --2- 172.21.15.40:0/2929802211 >> v2:172.21.15.189:3300/0 conn(0x7fe2c012fad0 0x7fe2c012feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T08:20:09.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.188+0000 7fe2bf59e640 1 -- 172.21.15.40:0/2929802211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2c0133460 msgr2=0x7fe2c013ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:09.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.188+0000 7fe2bf59e640 1 --2- 172.21.15.40:0/2929802211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2c0133460 0x7fe2c013ac60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe2c006bb30 tx=0x7fe2b802bd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d782191fde8d0ad server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:09.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.188+0000 7fe2bdd9b640 1 -- 172.21.15.40:0/2929802211 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b802cdc0 con 0x7fe2c0133460 2024-04-22T08:20:09.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.188+0000 7fe2bdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:09.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.188+0000 7fe2bdd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:20:09.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.188+0000 7fe2bdd9b640 10 monclient: dump: 2024-04-22T08:20:09.198 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:09.198 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:09.199 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:09.199 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:09.199 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:09.199 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:09.199 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:09.199 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:09.199 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:09.199 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:09.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.188+0000 7fe2bdd9b640 10 monclient: _finish_auth 0 2024-04-22T08:20:09.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.188+0000 7fe2bdd9b640 10 monclient: _check_auth_tickets 2024-04-22T08:20:09.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.188+0000 7fe2bdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:20:09.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.188+0000 7fe2c51d7640 5 monclient: authenticate success, global_id 94116 2024-04-22T08:20:09.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.188+0000 7fe2bdd9b640 1 -- 172.21.15.40:0/2929802211 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2b8034440 con 0x7fe2c0133460 2024-04-22T08:20:09.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.188+0000 7fe2c51d7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:20:09.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.188+0000 7fe2bdd9b640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:20:09.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.188+0000 7fe2c51d7640 10 monclient: get_monmap_and_config success 2024-04-22T08:20:09.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.188+0000 7fe2c51d7640 4 set_mon_vals no callback set 2024-04-22T08:20:09.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.188+0000 7fe2bdd9b640 1 -- 172.21.15.40:0/2929802211 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b80347d0 con 0x7fe2c0133460 2024-04-22T08:20:09.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.188+0000 7fe2bdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:09.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.188+0000 7fe2bdd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:20:09.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.188+0000 7fe2bdd9b640 10 monclient: dump: 2024-04-22T08:20:09.200 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:09.200 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:09.200 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:09.200 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:09.200 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:09.201 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:09.201 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:09.201 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:09.201 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:09.201 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:09.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.188+0000 7fe2c51d7640 10 monclient: shutdown 2024-04-22T08:20:09.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.188+0000 7fe2c51d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:20:09.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.188+0000 7fe2c51d7640 1 -- 172.21.15.40:0/2929802211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2c0133460 msgr2=0x7fe2c013ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:09.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.188+0000 7fe2c51d7640 1 --2- 172.21.15.40:0/2929802211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2c0133460 0x7fe2c013ac60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe2c006bb30 tx=0x7fe2b802bd20 comp rx=0 tx=0).stop 2024-04-22T08:20:09.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.188+0000 7fe2c51d7640 1 -- 172.21.15.40:0/2929802211 shutdown_connections 2024-04-22T08:20:09.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.188+0000 7fe2c51d7640 1 --2- 172.21.15.40:0/2929802211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2c0133460 0x7fe2c013ac60 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:09.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.188+0000 7fe2c51d7640 1 --2- 172.21.15.40:0/2929802211 >> v2:172.21.15.189:3300/0 conn(0x7fe2c012fad0 0x7fe2c012feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:09.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.188+0000 7fe2c51d7640 1 -- 172.21.15.40:0/2929802211 >> 172.21.15.40:0/2929802211 conn(0x7fe2c00a0ae0 msgr2=0x7fe2c00a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:20:09.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.188+0000 7fe2c51d7640 1 -- 172.21.15.40:0/2929802211 shutdown_connections 2024-04-22T08:20:09.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.188+0000 7fe2c51d7640 1 -- 172.21.15.40:0/2929802211 wait complete. 2024-04-22T08:20:09.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.189+0000 7fe2c51d7640 10 monclient: build_initial_monmap 2024-04-22T08:20:09.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.189+0000 7fe2c51d7640 10 monclient: monmap: 2024-04-22T08:20:09.202 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:20:09.202 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:20:09.202 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:20:09.202 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:20:09.202 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:20:09.202 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:09.202 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:20:09.202 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:20:09.202 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:20:09.202 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:09.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.189+0000 7fe2c51d7640 1 Processor -- start 2024-04-22T08:20:09.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2c51d7640 1 -- start start 2024-04-22T08:20:09.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2c51d7640 10 monclient: init 2024-04-22T08:20:09.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2c51d7640 10 monclient: _reopen_session rank -1 2024-04-22T08:20:09.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2c51d7640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-22T08:20:09.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2c51d7640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2c012fad0 0x7fe2c013b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:09.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2c51d7640 10 monclient(hunting): picked mon.noname-b con 0x7fe2c012fad0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:20:09.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2c51d7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2c0132920 0x7fe2c0139df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:09.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2c51d7640 10 monclient(hunting): picked mon.noname-a con 0x7fe2c0132920 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:20:09.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2c51d7640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe2c0133460 0x7fe2c013a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:09.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2c51d7640 10 monclient(hunting): picked mon.noname-c con 0x7fe2c0133460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:20:09.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2c51d7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:09.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2c51d7640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c013a870 con 0x7fe2c0132920 2024-04-22T08:20:09.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2c51d7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:09.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2c51d7640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c013cb90 con 0x7fe2c012fad0 2024-04-22T08:20:09.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2c51d7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:09.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2c51d7640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c0071870 con 0x7fe2c0133460 2024-04-22T08:20:09.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2c51d7640 10 monclient(hunting): _renew_subs 2024-04-22T08:20:09.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2c51d7640 10 monclient(hunting): authenticate will time out at 1283.185268s 2024-04-22T08:20:09.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2bf59e640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe2c0133460 0x7fe2c013a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:09.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2be59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2c0132920 0x7fe2c0139df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:09.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2bf59e640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe2c0133460 0x7fe2c013a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:57992/0 (socket says 172.21.15.40:57992) 2024-04-22T08:20:09.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2bf59e640 1 -- 172.21.15.40:0/272787860 learned_addr learned my addr 172.21.15.40:0/272787860 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:20:09.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2bf59e640 10 monclient(hunting): get_auth_request con 0x7fe2c0133460 auth_method 0 2024-04-22T08:20:09.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2bf59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:09.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2bf59e640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:09.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2bf59e640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:09.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2be59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2c0132920 0x7fe2c0139df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46324/0 (socket says 172.21.15.40:46324) 2024-04-22T08:20:09.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2be59c640 10 monclient(hunting): get_auth_request con 0x7fe2c0132920 auth_method 0 2024-04-22T08:20:09.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2be59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:09.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2be59c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:09.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2be59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:09.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2bed9d640 1 --2- 172.21.15.40:0/272787860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2c012fad0 0x7fe2c013b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:09.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2bf59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:09.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2bf59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:09.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2bf59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:09.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2be59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:09.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2be59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:09.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2be59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:09.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2bed9d640 10 monclient(hunting): get_auth_request con 0x7fe2c012fad0 auth_method 0 2024-04-22T08:20:09.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2bed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:09.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2bed9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:09.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2bed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:09.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2bed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:09.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2bed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:09.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2bed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:09.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2bf59e640 10 monclient(hunting): handle_auth_done global_id 54839 payload 824 2024-04-22T08:20:09.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2bf59e640 1 -- 172.21.15.40:0/272787860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2c012fad0 msgr2=0x7fe2c013b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:09.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2bf59e640 1 --2- 172.21.15.40:0/272787860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2c012fad0 0x7fe2c013b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:09.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2bf59e640 1 -- 172.21.15.40:0/272787860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2c0132920 msgr2=0x7fe2c0139df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:09.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2bf59e640 1 --2- 172.21.15.40:0/272787860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2c0132920 0x7fe2c0139df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:09.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2bf59e640 10 monclient: _finish_hunting 0 2024-04-22T08:20:09.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2bf59e640 1 monclient: found mon.noname-c 2024-04-22T08:20:09.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2bf59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:20:09.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2bf59e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:20:09.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2bf59e640 1 -- 172.21.15.40:0/272787860 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b8009a70 con 0x7fe2c0133460 2024-04-22T08:20:09.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.190+0000 7fe2be59c640 1 --2- 172.21.15.40:0/272787860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2c0132920 0x7fe2c0139df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:20:09.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.193+0000 7fe2bf59e640 1 -- 172.21.15.40:0/272787860 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe2c0133460 msgr2=0x7fe2c013a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:09.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.193+0000 7fe2bf59e640 1 --2- 172.21.15.40:0/272787860 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe2c0133460 0x7fe2c013a330 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fe2b8009e80 tx=0x7fe2b8004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:09.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.193+0000 7fe2a77fe640 1 -- 172.21.15.40:0/272787860 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b8040650 con 0x7fe2c0133460 2024-04-22T08:20:09.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.193+0000 7fe2a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:09.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.193+0000 7fe2a77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:20:09.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.193+0000 7fe2a77fe640 10 monclient: dump: 2024-04-22T08:20:09.207 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:09.207 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:09.207 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:09.207 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:09.207 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:09.207 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:09.208 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:09.208 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:09.208 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:09.208 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:09.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.193+0000 7fe2a77fe640 10 monclient: _finish_auth 0 2024-04-22T08:20:09.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.193+0000 7fe2a77fe640 10 monclient: _check_auth_tickets 2024-04-22T08:20:09.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.193+0000 7fe2a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:20:09.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.193+0000 7fe2a77fe640 1 -- 172.21.15.40:0/272787860 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2b80407b0 con 0x7fe2c0133460 2024-04-22T08:20:09.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.193+0000 7fe2a77fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:20:09.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.193+0000 7fe2a77fe640 1 -- 172.21.15.40:0/272787860 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b8040b60 con 0x7fe2c0133460 2024-04-22T08:20:09.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.193+0000 7fe2c51d7640 5 monclient: authenticate success, global_id 54839 2024-04-22T08:20:09.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.193+0000 7fe2c51d7640 10 monclient: _renew_subs 2024-04-22T08:20:09.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.193+0000 7fe2c51d7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:20:09.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.193+0000 7fe2c51d7640 1 -- 172.21.15.40:0/272787860 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2c013ab10 con 0x7fe2c0133460 2024-04-22T08:20:09.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.193+0000 7fe2a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:09.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.193+0000 7fe2a77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:20:09.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.193+0000 7fe2a77fe640 10 monclient: dump: 2024-04-22T08:20:09.209 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:09.209 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:09.209 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:09.209 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:09.209 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:09.209 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:09.209 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:09.209 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:09.209 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:09.209 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:09.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.193+0000 7fe2bd59a640 4 set_mon_vals no callback set 2024-04-22T08:20:09.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.193+0000 7fe2c51d7640 10 monclient: _renew_subs 2024-04-22T08:20:09.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.193+0000 7fe2c51d7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:20:09.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.193+0000 7fe2c51d7640 1 -- 172.21.15.40:0/272787860 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2c013c910 con 0x7fe2c0133460 2024-04-22T08:20:09.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.194+0000 7fe2a77fe640 1 -- 172.21.15.40:0/272787860 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fe2b8040d40 con 0x7fe2c0133460 2024-04-22T08:20:09.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.194+0000 7fe2c51d7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:20:09.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.194+0000 7fe2c51d7640 1 --2- 172.21.15.40:0/272787860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2c01ca630 0x7fe2c01caa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:09.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.194+0000 7fe2c51d7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:09.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.194+0000 7fe2c51d7640 1 -- 172.21.15.40:0/272787860 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c00667a0 con 0x7fe2c01ca630 2024-04-22T08:20:09.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.194+0000 7fe2a77fe640 1 --2- 172.21.15.40:0/272787860 >> v2:172.21.15.189:6800/30168 conn(0x7fe2ac0377d0 0x7fe2ac039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:09.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.194+0000 7fe2a77fe640 1 -- 172.21.15.40:0/272787860 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(241..241 src has 1..241) v4 ==== 3898+0+0 (secure 0 0 0) 0x7fe2b806d110 con 0x7fe2c0133460 2024-04-22T08:20:09.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.194+0000 7fe2bed9d640 1 --2- 172.21.15.40:0/272787860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2c01ca630 0x7fe2c01caa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:09.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.194+0000 7fe2bed9d640 10 monclient: get_auth_request con 0x7fe2c01ca630 auth_method 0 2024-04-22T08:20:09.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.194+0000 7fe2bed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:09.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.194+0000 7fe2bed9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:09.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.194+0000 7fe2bed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:09.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.194+0000 7fe2bed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:09.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.194+0000 7fe2bed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:09.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.194+0000 7fe2bed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:09.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.194+0000 7fe2be59c640 1 --2- 172.21.15.40:0/272787860 >> v2:172.21.15.189:6800/30168 conn(0x7fe2ac0377d0 0x7fe2ac039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:09.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.194+0000 7fe2be59c640 10 monclient: get_auth_request con 0x7fe2ac0377d0 auth_method 0 2024-04-22T08:20:09.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.194+0000 7fe2bed9d640 10 monclient(hunting): handle_auth_done global_id 64672 payload 824 2024-04-22T08:20:09.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.194+0000 7fe2bed9d640 1 -- 172.21.15.40:0/272787860 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe2c009e2f0 con 0x7fe2c01ca630 2024-04-22T08:20:09.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.197+0000 7fe2bed9d640 1 -- 172.21.15.40:0/272787860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2c01ca630 msgr2=0x7fe2c01caa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:09.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.197+0000 7fe2bed9d640 1 --2- 172.21.15.40:0/272787860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2c01ca630 0x7fe2c01caa10 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe2b000f880 tx=0x7fe2b000fd40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:09.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.197+0000 7fe2be59c640 1 --2- 172.21.15.40:0/272787860 >> v2:172.21.15.189:6800/30168 conn(0x7fe2ac0377d0 0x7fe2ac039c90 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fe2b4007c90 tx=0x7fe2b4007be0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:09.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.197+0000 7fe2a77fe640 1 -- 172.21.15.40:0/272787860 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b000de30 con 0x7fe2c01ca630 2024-04-22T08:20:09.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.197+0000 7fe2a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:09.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.197+0000 7fe2a77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:20:09.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.197+0000 7fe2a77fe640 10 monclient: dump: 2024-04-22T08:20:09.212 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:09.212 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:09.212 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:09.212 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:09.212 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:09.212 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:09.212 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:09.212 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:09.212 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:09.212 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:09.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.197+0000 7fe2a77fe640 1 -- 172.21.15.40:0/272787860 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7fe2b000c040 con 0x7fe2c01ca630 2024-04-22T08:20:09.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.197+0000 7fe2a77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:20:09.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.197+0000 7fe2a77fe640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:20:09.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.197+0000 7fe2a77fe640 1 -- 172.21.15.40:0/272787860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2c01ca630 msgr2=0x7fe2c01caa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:09.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.197+0000 7fe2a77fe640 1 --2- 172.21.15.40:0/272787860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2c01ca630 0x7fe2c01caa10 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe2b000f880 tx=0x7fe2b000fd40 comp rx=0 tx=0).stop 2024-04-22T08:20:09.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.197+0000 7fe2a77fe640 1 -- 172.21.15.40:0/272787860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2c01ca630 msgr2=0x7fe2c01caa10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:20:09.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.197+0000 7fe2a77fe640 1 --2- 172.21.15.40:0/272787860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2c01ca630 0x7fe2c01caa10 secure :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe2b000f880 tx=0x7fe2b000fd40 comp rx=0 tx=0).stop 2024-04-22T08:20:09.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.203+0000 7fe2c51d7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:20:09.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.204+0000 7fe2c51d7640 1 --2- 172.21.15.40:0/272787860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2c01caf50 0x7fe2c012b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:09.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.204+0000 7fe2c51d7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:09.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.204+0000 7fe2c51d7640 1 -- 172.21.15.40:0/272787860 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c0134a10 con 0x7fe2c01caf50 2024-04-22T08:20:09.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.204+0000 7fe2bed9d640 1 --2- 172.21.15.40:0/272787860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2c01caf50 0x7fe2c012b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:09.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.204+0000 7fe2bed9d640 10 monclient: get_auth_request con 0x7fe2c01caf50 auth_method 0 2024-04-22T08:20:09.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.204+0000 7fe2bed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:09.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.204+0000 7fe2bed9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:09.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.204+0000 7fe2bed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:09.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.204+0000 7fe2bed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:09.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.204+0000 7fe2bed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:09.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.204+0000 7fe2bed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:09.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.204+0000 7fe2bed9d640 10 monclient(hunting): handle_auth_done global_id 64675 payload 824 2024-04-22T08:20:09.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.204+0000 7fe2bed9d640 1 -- 172.21.15.40:0/272787860 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe2c0133fe0 con 0x7fe2c01caf50 2024-04-22T08:20:09.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.208+0000 7fe2bed9d640 1 -- 172.21.15.40:0/272787860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2c01caf50 msgr2=0x7fe2c012b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:09.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.208+0000 7fe2bed9d640 1 --2- 172.21.15.40:0/272787860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2c01caf50 0x7fe2c012b950 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fe2b000f880 tx=0x7fe2b00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:09.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.208+0000 7fe2a77fe640 1 -- 172.21.15.40:0/272787860 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b0022070 con 0x7fe2c01caf50 2024-04-22T08:20:09.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.208+0000 7fe2a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:09.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.208+0000 7fe2a77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:20:09.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.208+0000 7fe2a77fe640 10 monclient: dump: 2024-04-22T08:20:09.216 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:09.216 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:09.216 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:09.216 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:09.216 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:09.216 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:09.216 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:09.216 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:09.217 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:09.217 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:09.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.208+0000 7fe2a77fe640 1 -- 172.21.15.40:0/272787860 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4801 (secure 0 0 0) 0x7fe2b0003a30 con 0x7fe2c01caf50 2024-04-22T08:20:09.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.208+0000 7fe2a77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:20:09.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.208+0000 7fe2a77fe640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:20:09.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.208+0000 7fe2a77fe640 1 -- 172.21.15.40:0/272787860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2c01caf50 msgr2=0x7fe2c012b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:09.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.208+0000 7fe2a77fe640 1 --2- 172.21.15.40:0/272787860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2c01caf50 0x7fe2c012b950 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fe2b000f880 tx=0x7fe2b00046c0 comp rx=0 tx=0).stop 2024-04-22T08:20:09.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.208+0000 7fe2a77fe640 1 -- 172.21.15.40:0/272787860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2c01caf50 msgr2=0x7fe2c012b950 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:20:09.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.208+0000 7fe2a77fe640 1 --2- 172.21.15.40:0/272787860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2c01caf50 0x7fe2c012b950 secure :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fe2b000f880 tx=0x7fe2b00046c0 comp rx=0 tx=0).stop 2024-04-22T08:20:09.217 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:20:09.217 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:20:09.218 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:20:09.218 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:20:09.218 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 72, 2024-04-22T08:20:09.218 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:20:09.218 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:20:09.218 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:20:09.218 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:20:09.218 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:20:09.218 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 0, 2024-04-22T08:20:09.218 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:20:09.218 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:20:09.218 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:20:09.218 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:20:09.219 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:20:09.219 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:20:09.219 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:20:09.219 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:20:09.219 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:20:09.219 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:20:09.219 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:20:09.219 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:20:09.219 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:20:09.219 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:20:09.219 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:20:09.219 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:20:09.219 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:20:09.219 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:20:09.219 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:20:09.220 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:20:09.220 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:20:09.220 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:20:09.220 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:20:09.220 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:20:09.220 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:20:09.220 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:20:09.220 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:09.220 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:09.220 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:20:09.220 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:20:09.220 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:20:09.220 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:20:09.220 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:20:09.221 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:20:09.221 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:20:09.221 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:20:09.221 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:20:09.221 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:20:09.221 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:20:09.221 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:20:09.221 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:20:09.221 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:20:09.221 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:20:09.221 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:20:09.221 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:20:09.221 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:20:09.221 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:20:09.222 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:20:09.222 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:20:09.222 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:20:09.222 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:20:09.222 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:20:09.222 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:20:09.222 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:20:09.222 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:20:09.222 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:20:09.222 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:20:09.222 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:09.222 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:20:09.222 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:09.223 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:20:09.223 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:20:09.223 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:20:09.223 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:20:09.223 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:09.223 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:20:09.223 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:20:09.223 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:09.223 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:09.223 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:09.223 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:20:09.223 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:20:09.223 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:09.223 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:09.223 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:09.224 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:09.224 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:20:09.224 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:20:09.224 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:20:09.224 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:20:09.224 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:20:09.224 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:09.224 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:09.224 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:20:09.224 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:20:09.224 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:20:09.224 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:20:09.224 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:09.224 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:20:09.225 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:20:09.225 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:09.225 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:09.225 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:09.225 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:20:09.225 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:20:09.225 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:09.225 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:09.225 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:09.225 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:09.225 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:20:09.225 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:20:09.225 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:20:09.226 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:20:09.226 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:20:09.226 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:09.226 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:09.226 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:20:09.226 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:20:09.226 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:20:09.226 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:20:09.226 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:09.226 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:20:09.226 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:20:09.226 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:09.226 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:09.226 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:09.227 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:20:09.227 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:20:09.227 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:09.227 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:09.227 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:09.227 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:09.227 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:20:09.227 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:20:09.227 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:20:09.227 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:20:09.227 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:20:09.227 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:09.227 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:09.227 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:09.228 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:20:09.228 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:20:09.228 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:09.228 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:20:09.228 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:20:09.228 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:20:09.228 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:09.228 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:20:09.228 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:20:09.228 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:09.228 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:20:09.228 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:20:09.228 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 3 2024-04-22T08:20:09.229 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:09.229 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:20:09.229 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:20:09.229 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:09.229 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:20:09.229 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:20:09.229 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 5 2024-04-22T08:20:09.229 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:09.229 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:20:09.229 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": [ 2024-04-22T08:20:09.229 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:09.229 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:20:09.229 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:20:09.229 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:20:09.229 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:09.229 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:09.230 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:09.230 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:20:09.230 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:20:09.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.208+0000 7fe2c51d7640 1 -- 172.21.15.40:0/272787860 >> v2:172.21.15.189:6800/30168 conn(0x7fe2ac0377d0 msgr2=0x7fe2ac039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:09.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.208+0000 7fe2c51d7640 1 --2- 172.21.15.40:0/272787860 >> v2:172.21.15.189:6800/30168 conn(0x7fe2ac0377d0 0x7fe2ac039c90 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fe2b4007c90 tx=0x7fe2b4007be0 comp rx=0 tx=0).stop 2024-04-22T08:20:09.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.208+0000 7fe2c51d7640 10 monclient: shutdown 2024-04-22T08:20:09.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.208+0000 7fe2c51d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:20:09.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.208+0000 7fe2c51d7640 1 -- 172.21.15.40:0/272787860 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe2c0133460 msgr2=0x7fe2c013a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:09.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.208+0000 7fe2bf59e640 1 -- 172.21.15.40:0/272787860 reap_dead start 2024-04-22T08:20:09.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.208+0000 7fe2c51d7640 1 --2- 172.21.15.40:0/272787860 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe2c0133460 0x7fe2c013a330 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fe2b8009e80 tx=0x7fe2b8004a40 comp rx=0 tx=0).stop 2024-04-22T08:20:09.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.208+0000 7fe2c51d7640 1 -- 172.21.15.40:0/272787860 shutdown_connections 2024-04-22T08:20:09.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.208+0000 7fe2c51d7640 1 --2- 172.21.15.40:0/272787860 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe2c0133460 0x7fe2c013a330 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:09.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.208+0000 7fe2c51d7640 1 -- 172.21.15.40:0/272787860 >> 172.21.15.40:0/272787860 conn(0x7fe2c00a0ae0 msgr2=0x7fe2c009b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:20:09.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.208+0000 7fe2c51d7640 1 -- 172.21.15.40:0/272787860 shutdown_connections 2024-04-22T08:20:09.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.208+0000 7fe2c51d7640 1 -- 172.21.15.40:0/272787860 wait complete. 2024-04-22T08:20:09.232 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:20:09.232 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-22T08:20:09.232 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-22T08:20:09.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.330+0000 7fe736639640 10 monclient: get_monmap_and_config 2024-04-22T08:20:09.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.330+0000 7fe736639640 10 monclient: build_initial_monmap 2024-04-22T08:20:09.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.330+0000 7fe736639640 10 monclient: monmap: 2024-04-22T08:20:09.331 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:20:09.332 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:09.332 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:20:09.332631+0000 2024-04-22T08:20:09.332 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:20:09.332631+0000 2024-04-22T08:20:09.332 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:20:09.332 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:09.332 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:20:09.332 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:20:09.332 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:20:09.332 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:20:09.332 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:20:09.332 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:09.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.331+0000 7fe736639640 1 Processor -- start 2024-04-22T08:20:09.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.331+0000 7fe736639640 1 -- start start 2024-04-22T08:20:09.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.331+0000 7fe736639640 10 monclient: init 2024-04-22T08:20:09.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.332+0000 7fe736639640 10 monclient: _reopen_session rank -1 2024-04-22T08:20:09.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.332+0000 7fe736639640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-04-22T08:20:09.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.332+0000 7fe736639640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe73009f0b0 addr v1:172.21.15.189:6789/0 2024-04-22T08:20:09.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.332+0000 7fe736639640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe73009e2b0 0x7fe730138e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:09.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.332+0000 7fe736639640 10 monclient(hunting): picked mon.noname-b con 0x7fe73009e2b0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:20:09.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.332+0000 7fe736639640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe730139380 0x7fe73013b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:09.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.332+0000 7fe736639640 10 monclient(hunting): picked mon.noname-a con 0x7fe730139380 addr v2:172.21.15.40:3300/0 2024-04-22T08:20:09.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.332+0000 7fe736639640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe73009ef10 con 0x7fe73009f0b0 2024-04-22T08:20:09.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.332+0000 7fe736639640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:09.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.332+0000 7fe736639640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe730071670 con 0x7fe730139380 2024-04-22T08:20:09.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.332+0000 7fe736639640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:09.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.332+0000 7fe736639640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe73009fba0 con 0x7fe73009e2b0 2024-04-22T08:20:09.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.332+0000 7fe736639640 10 monclient(hunting): _renew_subs 2024-04-22T08:20:09.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.332+0000 7fe736639640 10 monclient(hunting): authenticate will time out at 1283.326932s 2024-04-22T08:20:09.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.332+0000 7fe734bd6640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe730139380 0x7fe73013b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:09.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.332+0000 7fe72f7fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe73009e2b0 0x7fe730138e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:09.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.332+0000 7fe734bd6640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe730139380 0x7fe73013b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46336/0 (socket says 172.21.15.40:46336) 2024-04-22T08:20:09.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.332+0000 7fe72f7fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe73009e2b0 0x7fe730138e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:58008/0 (socket says 172.21.15.40:58008) 2024-04-22T08:20:09.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.332+0000 7fe734bd6640 1 -- 172.21.15.40:0/1939118061 learned_addr learned my addr 172.21.15.40:0/1939118061 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:20:09.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.332+0000 7fe734bd6640 10 monclient(hunting): get_auth_request con 0x7fe730139380 auth_method 0 2024-04-22T08:20:09.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.332+0000 7fe734bd6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:09.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.332+0000 7fe734bd6640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:09.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.332+0000 7fe734bd6640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:09.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.332+0000 7fe72f7fe640 10 monclient(hunting): get_auth_request con 0x7fe73009e2b0 auth_method 0 2024-04-22T08:20:09.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.332+0000 7fe72f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:09.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.332+0000 7fe72f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:09.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.332+0000 7fe72f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:09.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.332+0000 7fe734bd6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:09.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.332+0000 7fe734bd6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:09.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.332+0000 7fe734bd6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:09.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.332+0000 7fe72f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:09.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.332+0000 7fe72f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:09.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.332+0000 7fe72f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:09.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.332+0000 7fe734bd6640 10 monclient(hunting): handle_auth_done global_id 94122 payload 274 2024-04-22T08:20:09.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.332+0000 7fe734bd6640 1 -- 172.21.15.40:0/1939118061 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe73009e2b0 msgr2=0x7fe730138e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:20:09.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.332+0000 7fe734bd6640 1 --2- 172.21.15.40:0/1939118061 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe73009e2b0 0x7fe730138e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:09.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.332+0000 7fe734bd6640 1 -- 172.21.15.40:0/1939118061 >> v1:172.21.15.189:6789/0 conn(0x7fe73009f0b0 legacy=0x7fe73009f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:20:09.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.332+0000 7fe734bd6640 10 monclient: _finish_hunting 0 2024-04-22T08:20:09.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.332+0000 7fe734bd6640 1 monclient: found mon.noname-a 2024-04-22T08:20:09.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.332+0000 7fe734bd6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:20:09.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.332+0000 7fe734bd6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:20:09.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.332+0000 7fe734bd6640 1 -- 172.21.15.40:0/1939118061 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe73009d9e0 con 0x7fe730139380 2024-04-22T08:20:09.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.335+0000 7fe734bd6640 1 -- 172.21.15.40:0/1939118061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe730139380 msgr2=0x7fe73013b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:09.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.335+0000 7fe734bd6640 1 --2- 172.21.15.40:0/1939118061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe730139380 0x7fe73013b770 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe728009e50 tx=0x7fe72802bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ce1215ecd280435 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:09.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.335+0000 7fe72effd640 1 -- 172.21.15.40:0/1939118061 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe72802cb60 con 0x7fe730139380 2024-04-22T08:20:09.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.335+0000 7fe72effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:09.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.335+0000 7fe72effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:20:09.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.335+0000 7fe72effd640 10 monclient: dump: 2024-04-22T08:20:09.340 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:09.340 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:09.340 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:09.340 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:09.340 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:09.340 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:09.340 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:09.340 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:09.340 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:09.340 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:09.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.335+0000 7fe72effd640 10 monclient: _finish_auth 0 2024-04-22T08:20:09.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.335+0000 7fe72effd640 10 monclient: _check_auth_tickets 2024-04-22T08:20:09.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.335+0000 7fe72effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:20:09.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.335+0000 7fe72effd640 1 -- 172.21.15.40:0/1939118061 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe72802ccc0 con 0x7fe730139380 2024-04-22T08:20:09.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.335+0000 7fe72effd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:20:09.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.335+0000 7fe72effd640 1 -- 172.21.15.40:0/1939118061 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe728034610 con 0x7fe730139380 2024-04-22T08:20:09.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.335+0000 7fe736639640 5 monclient: authenticate success, global_id 94122 2024-04-22T08:20:09.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.335+0000 7fe736639640 10 monclient: get_monmap_and_config success 2024-04-22T08:20:09.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.335+0000 7fe736639640 4 set_mon_vals no callback set 2024-04-22T08:20:09.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.335+0000 7fe72effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:09.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.335+0000 7fe72effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:20:09.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.335+0000 7fe72effd640 10 monclient: dump: 2024-04-22T08:20:09.342 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:09.342 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:09.342 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:09.342 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:09.342 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:09.342 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:09.342 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:09.342 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:09.343 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:09.343 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:09.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.336+0000 7fe736639640 10 monclient: shutdown 2024-04-22T08:20:09.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.336+0000 7fe736639640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:20:09.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.336+0000 7fe736639640 1 -- 172.21.15.40:0/1939118061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe730139380 msgr2=0x7fe73013b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:09.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.336+0000 7fe736639640 1 --2- 172.21.15.40:0/1939118061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe730139380 0x7fe73013b770 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe728009e50 tx=0x7fe72802bdb0 comp rx=0 tx=0).stop 2024-04-22T08:20:09.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.336+0000 7fe736639640 1 -- 172.21.15.40:0/1939118061 shutdown_connections 2024-04-22T08:20:09.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.336+0000 7fe736639640 1 --2- 172.21.15.40:0/1939118061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe730139380 0x7fe73013b770 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:09.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.336+0000 7fe736639640 1 --2- 172.21.15.40:0/1939118061 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe73009e2b0 0x7fe730138e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:09.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.336+0000 7fe736639640 1 -- 172.21.15.40:0/1939118061 >> 172.21.15.40:0/1939118061 conn(0x7fe7300a0b50 msgr2=0x7fe7300a0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:20:09.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.336+0000 7fe736639640 1 -- 172.21.15.40:0/1939118061 shutdown_connections 2024-04-22T08:20:09.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.336+0000 7fe736639640 1 -- 172.21.15.40:0/1939118061 wait complete. 2024-04-22T08:20:09.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.337+0000 7fe736639640 10 monclient: build_initial_monmap 2024-04-22T08:20:09.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.337+0000 7fe736639640 10 monclient: monmap: 2024-04-22T08:20:09.348 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:20:09.348 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:20:09.348 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:20:09.348 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:20:09.349 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:20:09.349 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:09.349 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:20:09.349 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:20:09.349 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:20:09.349 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:09.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.337+0000 7fe736639640 1 Processor -- start 2024-04-22T08:20:09.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.337+0000 7fe736639640 1 -- start start 2024-04-22T08:20:09.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.337+0000 7fe736639640 10 monclient: init 2024-04-22T08:20:09.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.337+0000 7fe736639640 10 monclient: _reopen_session rank -1 2024-04-22T08:20:09.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.337+0000 7fe736639640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-22T08:20:09.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.337+0000 7fe736639640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe73009e2b0 0x7fe730137b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:09.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.337+0000 7fe736639640 10 monclient(hunting): picked mon.noname-b con 0x7fe73009e2b0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:20:09.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.337+0000 7fe736639640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe73009f0b0 0x7fe7301380b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:09.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.337+0000 7fe736639640 10 monclient(hunting): picked mon.noname-a con 0x7fe73009f0b0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:20:09.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.338+0000 7fe736639640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe730139380 0x7fe7301205c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:09.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.338+0000 7fe736639640 10 monclient(hunting): picked mon.noname-c con 0x7fe730139380 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:20:09.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.338+0000 7fe736639640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:09.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.338+0000 7fe736639640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7301c96b0 con 0x7fe73009f0b0 2024-04-22T08:20:09.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.338+0000 7fe736639640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:09.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.338+0000 7fe736639640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe730071860 con 0x7fe73009e2b0 2024-04-22T08:20:09.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.338+0000 7fe736639640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:09.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.338+0000 7fe736639640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe730138de0 con 0x7fe730139380 2024-04-22T08:20:09.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.338+0000 7fe736639640 10 monclient(hunting): _renew_subs 2024-04-22T08:20:09.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.338+0000 7fe736639640 10 monclient(hunting): authenticate will time out at 1283.332814s 2024-04-22T08:20:09.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.338+0000 7fe72f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe73009f0b0 0x7fe7301380b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:09.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.338+0000 7fe734bd6640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe730139380 0x7fe7301205c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:09.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.338+0000 7fe72f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe73009f0b0 0x7fe7301380b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46348/0 (socket says 172.21.15.40:46348) 2024-04-22T08:20:09.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.338+0000 7fe72f7fe640 1 -- 172.21.15.40:0/1667973160 learned_addr learned my addr 172.21.15.40:0/1667973160 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:20:09.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.338+0000 7fe72f7fe640 10 monclient(hunting): get_auth_request con 0x7fe73009f0b0 auth_method 0 2024-04-22T08:20:09.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.338+0000 7fe72f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:09.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.338+0000 7fe72f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:09.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.338+0000 7fe72f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:09.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.338+0000 7fe734bd6640 10 monclient(hunting): get_auth_request con 0x7fe730139380 auth_method 0 2024-04-22T08:20:09.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.338+0000 7fe734bd6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:09.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.338+0000 7fe734bd6640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:09.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.338+0000 7fe734bd6640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:09.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.338+0000 7fe734bd6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:09.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.338+0000 7fe734bd6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:09.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.338+0000 7fe734bd6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:09.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.338+0000 7fe72f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:09.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.338+0000 7fe72f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:09.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.338+0000 7fe72f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:09.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.338+0000 7fe72ffff640 1 --2- 172.21.15.40:0/1667973160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe73009e2b0 0x7fe730137b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:09.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.338+0000 7fe734bd6640 10 monclient(hunting): handle_auth_done global_id 54845 payload 824 2024-04-22T08:20:09.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.338+0000 7fe734bd6640 1 -- 172.21.15.40:0/1667973160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe73009e2b0 msgr2=0x7fe730137b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:09.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.338+0000 7fe734bd6640 1 --2- 172.21.15.40:0/1667973160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe73009e2b0 0x7fe730137b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:09.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.338+0000 7fe734bd6640 1 -- 172.21.15.40:0/1667973160 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe73009f0b0 msgr2=0x7fe7301380b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:09.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.338+0000 7fe734bd6640 1 --2- 172.21.15.40:0/1667973160 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe73009f0b0 0x7fe7301380b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:09.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.338+0000 7fe734bd6640 10 monclient: _finish_hunting 0 2024-04-22T08:20:09.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.338+0000 7fe734bd6640 1 monclient: found mon.noname-c 2024-04-22T08:20:09.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.338+0000 7fe734bd6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:20:09.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.338+0000 7fe734bd6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:20:09.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.338+0000 7fe734bd6640 1 -- 172.21.15.40:0/1667973160 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe728009b30 con 0x7fe730139380 2024-04-22T08:20:09.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.338+0000 7fe72ffff640 10 monclient: get_auth_request con 0x7fe73009e2b0 auth_method 0 2024-04-22T08:20:09.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.338+0000 7fe72ffff640 1 --2- 172.21.15.40:0/1667973160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe73009e2b0 0x7fe730137b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T08:20:09.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.338+0000 7fe72f7fe640 1 --2- 172.21.15.40:0/1667973160 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe73009f0b0 0x7fe7301380b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:20:09.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.340+0000 7fe734bd6640 1 -- 172.21.15.40:0/1667973160 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe730139380 msgr2=0x7fe7301205c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:09.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.340+0000 7fe734bd6640 1 --2- 172.21.15.40:0/1667973160 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe730139380 0x7fe7301205c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe728035000 tx=0x7fe728004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:09.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.340+0000 7fe72cff9640 1 -- 172.21.15.40:0/1667973160 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe72803e450 con 0x7fe730139380 2024-04-22T08:20:09.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.340+0000 7fe72cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:09.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.340+0000 7fe72cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:20:09.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.340+0000 7fe72cff9640 10 monclient: dump: 2024-04-22T08:20:09.354 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:09.354 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:09.354 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:09.354 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:09.354 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:09.354 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:09.354 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:09.354 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:09.354 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:09.354 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:09.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.340+0000 7fe72cff9640 10 monclient: _finish_auth 0 2024-04-22T08:20:09.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.340+0000 7fe72cff9640 10 monclient: _check_auth_tickets 2024-04-22T08:20:09.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.340+0000 7fe72cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:20:09.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.340+0000 7fe72cff9640 1 -- 172.21.15.40:0/1667973160 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe72803e5b0 con 0x7fe730139380 2024-04-22T08:20:09.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.340+0000 7fe736639640 5 monclient: authenticate success, global_id 54845 2024-04-22T08:20:09.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.340+0000 7fe736639640 10 monclient: _renew_subs 2024-04-22T08:20:09.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.340+0000 7fe736639640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:20:09.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.340+0000 7fe736639640 1 -- 172.21.15.40:0/1667973160 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe730123c80 con 0x7fe730139380 2024-04-22T08:20:09.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.340+0000 7fe72cff9640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:20:09.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.340+0000 7fe72cff9640 1 -- 172.21.15.40:0/1667973160 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe72803e960 con 0x7fe730139380 2024-04-22T08:20:09.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.340+0000 7fe72cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:09.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.340+0000 7fe72cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:20:09.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.340+0000 7fe72cff9640 10 monclient: dump: 2024-04-22T08:20:09.355 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:09.355 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:09.356 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:09.356 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:09.356 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:09.356 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:09.356 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:09.356 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:09.356 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:09.356 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:09.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.340+0000 7fe72e7fc640 4 set_mon_vals no callback set 2024-04-22T08:20:09.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.341+0000 7fe736639640 10 monclient: _renew_subs 2024-04-22T08:20:09.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.341+0000 7fe736639640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:20:09.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.341+0000 7fe736639640 1 -- 172.21.15.40:0/1667973160 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7301221c0 con 0x7fe730139380 2024-04-22T08:20:09.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.341+0000 7fe72cff9640 1 -- 172.21.15.40:0/1667973160 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fe72803eb40 con 0x7fe730139380 2024-04-22T08:20:09.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.341+0000 7fe736639640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:20:09.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.341+0000 7fe736639640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:20:09.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.341+0000 7fe736639640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:20:09.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.341+0000 7fe736639640 1 -- 172.21.15.40:0/1667973160 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe73006b180 con 0x7fe730139380 2024-04-22T08:20:09.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.341+0000 7fe72cff9640 1 --2- 172.21.15.40:0/1667973160 >> v2:172.21.15.189:6800/30168 conn(0x7fe71c037820 0x7fe71c039ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:09.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.342+0000 7fe72cff9640 1 -- 172.21.15.40:0/1667973160 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(241..241 src has 1..241) v4 ==== 3898+0+0 (secure 0 0 0) 0x7fe72806ce10 con 0x7fe730139380 2024-04-22T08:20:09.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.342+0000 7fe72ffff640 1 --2- 172.21.15.40:0/1667973160 >> v2:172.21.15.189:6800/30168 conn(0x7fe71c037820 0x7fe71c039ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:09.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.342+0000 7fe72ffff640 10 monclient: get_auth_request con 0x7fe71c037820 auth_method 0 2024-04-22T08:20:09.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.344+0000 7fe72ffff640 1 --2- 172.21.15.40:0/1667973160 >> v2:172.21.15.189:6800/30168 conn(0x7fe71c037820 0x7fe71c039ce0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fe730121dc0 tx=0x7fe724009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:09.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.346+0000 7fe72cff9640 1 -- 172.21.15.40:0/1667973160 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7fe72803c350 con 0x7fe730139380 2024-04-22T08:20:09.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.346+0000 7fe72cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:20:09.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.346+0000 7fe72cff9640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:20:09.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.594+0000 7fe736639640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-22T08:20:09.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.595+0000 7fe736639640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-22T08:20:09.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.595+0000 7fe736639640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:20:09.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.595+0000 7fe736639640 1 -- 172.21.15.40:0/1667973160 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fe73009eef0 con 0x7fe730139380 2024-04-22T08:20:09.605 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.605+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:20:09.605 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.605+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=15,config=2,health=12,kv=6,logm=65} crc {auth=942960952,config=1618278320,health=1271409132,kv=2937709213,logm=1167190491}) 2024-04-22T08:20:09.605 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.605+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=15,config=2,health=11,kv=9,logm=63} crc {auth=4088143761,config=1618278320,health=1619211808,kv=1350343662,logm=2472827155}) 2024-04-22T08:20:09.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.605+0000 7fe72cff9640 1 -- 172.21.15.40:0/1667973160 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fe72806e020 con 0x7fe730139380 2024-04-22T08:20:09.606 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.605+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:20:09.606 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.605+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=15,config=2,health=12,kv=6,logm=65} crc {auth=942960952,config=1618278320,health=1271409132,kv=2937709213,logm=1167190491}) 2024-04-22T08:20:09.606 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.605+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=16,config=2,health=12,kv=9,logm=61} crc {auth=3840600155,config=1618278320,health=1271409132,kv=1350343662,logm=2185513102}) 2024-04-22T08:20:09.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.605+0000 7fe72cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-22T08:20:09.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.605+0000 7fe72cff9640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:20:09.607 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.606+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:20:09.608 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.606+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2485490089}) 2024-04-22T08:20:09.608 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.606+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1117119765}) 2024-04-22T08:20:09.608 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.606+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:20:09.608 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.606+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2485490089}) 2024-04-22T08:20:09.608 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.606+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=402476331}) 2024-04-22T08:20:09.608 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.607+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:20:09.608 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.607+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=182188139}) 2024-04-22T08:20:09.608 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.607+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2041548017}) 2024-04-22T08:20:09.608 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.607+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:20:09.608 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.607+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=182188139}) 2024-04-22T08:20:09.608 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.607+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1374820843}) 2024-04-22T08:20:09.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.608+0000 7fe736639640 1 -- 172.21.15.40:0/1667973160 >> v2:172.21.15.189:6800/30168 conn(0x7fe71c037820 msgr2=0x7fe71c039ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:09.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.608+0000 7fe736639640 1 --2- 172.21.15.40:0/1667973160 >> v2:172.21.15.189:6800/30168 conn(0x7fe71c037820 0x7fe71c039ce0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fe730121dc0 tx=0x7fe724009040 comp rx=0 tx=0).stop 2024-04-22T08:20:09.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.608+0000 7fe736639640 10 monclient: shutdown 2024-04-22T08:20:09.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.608+0000 7fe736639640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:20:09.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.608+0000 7fe736639640 1 -- 172.21.15.40:0/1667973160 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe730139380 msgr2=0x7fe7301205c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:09.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.608+0000 7fe736639640 1 --2- 172.21.15.40:0/1667973160 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe730139380 0x7fe7301205c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe728035000 tx=0x7fe728004a40 comp rx=0 tx=0).stop 2024-04-22T08:20:09.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.608+0000 7fe736639640 1 -- 172.21.15.40:0/1667973160 shutdown_connections 2024-04-22T08:20:09.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.608+0000 7fe736639640 1 --2- 172.21.15.40:0/1667973160 >> v2:172.21.15.189:6800/30168 conn(0x7fe71c037820 0x7fe71c039ce0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:09.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.608+0000 7fe736639640 1 --2- 172.21.15.40:0/1667973160 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe730139380 0x7fe7301205c0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:09.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.608+0000 7fe736639640 1 --2- 172.21.15.40:0/1667973160 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe73009f0b0 0x7fe7301380b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:09.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.608+0000 7fe736639640 1 --2- 172.21.15.40:0/1667973160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe73009e2b0 0x7fe730137b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:09.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.608+0000 7fe736639640 1 -- 172.21.15.40:0/1667973160 >> 172.21.15.40:0/1667973160 conn(0x7fe7300a0b50 msgr2=0x7fe73011a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:20:09.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.608+0000 7fe736639640 1 -- 172.21.15.40:0/1667973160 shutdown_connections 2024-04-22T08:20:09.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:09.608+0000 7fe736639640 1 -- 172.21.15.40:0/1667973160 wait complete. 2024-04-22T08:20:09.613 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.608+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:20:09.613 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.608+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2335858937}) 2024-04-22T08:20:09.614 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.608+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1007623718}) 2024-04-22T08:20:09.614 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.608+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:20:09.614 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.608+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2335858937}) 2024-04-22T08:20:09.614 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.608+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=194667606}) 2024-04-22T08:20:09.614 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.609+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:20:09.614 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.609+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2465004151}) 2024-04-22T08:20:09.614 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.609+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1402920605}) 2024-04-22T08:20:09.614 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.609+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:20:09.614 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.609+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2465004151}) 2024-04-22T08:20:09.614 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.609+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=792951550}) 2024-04-22T08:20:09.615 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.610+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:20:09.615 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.610+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2516909858}) 2024-04-22T08:20:09.615 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.610+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2941157402}) 2024-04-22T08:20:09.615 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.610+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:20:09.615 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.610+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2516909858}) 2024-04-22T08:20:09.615 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.610+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=535108635}) 2024-04-22T08:20:09.615 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.611+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:20:09.615 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.611+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1901400098}) 2024-04-22T08:20:09.615 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.611+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1438643446}) 2024-04-22T08:20:09.615 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.611+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:20:09.615 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.611+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1901400098}) 2024-04-22T08:20:09.615 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.611+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3606878005}) 2024-04-22T08:20:09.615 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.611+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:20:09.615 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.611+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2304132754}) 2024-04-22T08:20:09.616 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.611+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1581136249}) 2024-04-22T08:20:09.616 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.611+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:20:09.616 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.611+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2304132754}) 2024-04-22T08:20:09.616 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.611+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2399335307}) 2024-04-22T08:20:09.616 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.612+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:20:09.616 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.612+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=609599291}) 2024-04-22T08:20:09.616 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.612+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1725995742}) 2024-04-22T08:20:09.616 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.612+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:20:09.616 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.612+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=609599291}) 2024-04-22T08:20:09.616 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.612+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2204540106}) 2024-04-22T08:20:09.616 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.613+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:20:09.616 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.613+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=296431005}) 2024-04-22T08:20:09.616 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.613+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=456977927}) 2024-04-22T08:20:09.617 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.613+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:20:09.617 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.613+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=296431005}) 2024-04-22T08:20:09.617 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.613+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=182267148}) 2024-04-22T08:20:09.620 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.614+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:20:09.621 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.614+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2497052373}) 2024-04-22T08:20:09.621 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.614+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1985498272}) 2024-04-22T08:20:09.621 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.614+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:20:09.621 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.614+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2497052373}) 2024-04-22T08:20:09.621 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.614+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3567779255}) 2024-04-22T08:20:09.621 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.614+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:20:09.621 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.614+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=662580378}) 2024-04-22T08:20:09.621 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.614+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=464483090}) 2024-04-22T08:20:09.621 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.614+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:20:09.621 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.614+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=662580378}) 2024-04-22T08:20:09.621 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.614+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2684164240}) 2024-04-22T08:20:09.621 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.615+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:20:09.622 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.615+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=244495245}) 2024-04-22T08:20:09.622 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.615+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=390738885}) 2024-04-22T08:20:09.622 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.615+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:20:09.622 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.615+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=244495245}) 2024-04-22T08:20:09.622 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.615+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=957219561}) 2024-04-22T08:20:09.622 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.616+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:20:09.622 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.616+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1590907446}) 2024-04-22T08:20:09.622 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.616+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2551121667}) 2024-04-22T08:20:09.622 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.616+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:20:09.622 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.616+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1590907446}) 2024-04-22T08:20:09.622 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.616+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4175720990}) 2024-04-22T08:20:09.622 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.616+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:20:09.623 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.616+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3781086018}) 2024-04-22T08:20:09.623 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.616+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1958585592}) 2024-04-22T08:20:09.623 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.616+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:20:09.623 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.616+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3781086018}) 2024-04-22T08:20:09.623 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.616+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3468518049}) 2024-04-22T08:20:09.623 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.617+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:20:09.623 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.617+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3922323369}) 2024-04-22T08:20:09.623 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.617+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3018752419}) 2024-04-22T08:20:09.623 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.617+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:20:09.623 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.617+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3922323369}) 2024-04-22T08:20:09.623 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.617+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2259598269}) 2024-04-22T08:20:09.623 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.618+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:20:09.623 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.618+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3010017687}) 2024-04-22T08:20:09.624 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.618+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2916161360}) 2024-04-22T08:20:09.624 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.618+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:20:09.624 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.618+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3010017687}) 2024-04-22T08:20:09.624 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.618+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2119481385}) 2024-04-22T08:20:09.624 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.619+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:20:09.624 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.619+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=50,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=8,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=7} crc {logm=3658039469,mdsmap=1531151339,mgr=878880162,mgr_command_descs=470045235,mgr_metadata=4222303120,mgrstat=3005651047,mon_config_key=4255067121,monmap=3469086636,osd_metadata=4151711904,osd_pg_creating=3569137452,osdmap=727016054}) 2024-04-22T08:20:09.624 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.619+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=50,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=8,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=6} crc {logm=1363314334,mdsmap=1531151339,mgr=878880162,mgr_command_descs=470045235,mgr_metadata=4222303120,mgrstat=2563080513,mon_config_key=4255067121,monmap=3469086636,osd_metadata=4151711905,osd_pg_creating=3569137452,osdmap=1555372396}) 2024-04-22T08:20:09.624 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.619+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:20:09.624 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.619+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=50,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=8,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=7} crc {logm=3658039469,mdsmap=1531151339,mgr=878880162,mgr_command_descs=470045235,mgr_metadata=4222303120,mgrstat=3005651047,mon_config_key=4255067121,monmap=3469086636,osd_metadata=4151711904,osd_pg_creating=3569137452,osdmap=727016054}) 2024-04-22T08:20:09.624 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.619+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=51,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=8,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=6} crc {logm=779203187,mdsmap=1531151339,mgr=3080842177,mgr_command_descs=470045235,mgr_metadata=4222303120,mgrstat=2890705216,mon_config_key=4255067121,monmap=3469086636,osd_metadata=4151711904,osd_pg_creating=3569137452,osdmap=873066774}) 2024-04-22T08:20:09.624 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.620+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:20:09.624 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.620+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1210114476}) 2024-04-22T08:20:09.624 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.620+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3725969275}) 2024-04-22T08:20:09.625 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.620+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:20:09.625 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.620+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1210114476}) 2024-04-22T08:20:09.625 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.620+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2486630062}) 2024-04-22T08:20:09.626 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-22T08:20:09.627 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.620+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:20:09.627 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.620+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2091858892}) 2024-04-22T08:20:09.627 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.620+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1557862738}) 2024-04-22T08:20:09.627 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.620+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:20:09.627 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.620+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2091858892}) 2024-04-22T08:20:09.628 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.620+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1870406474}) 2024-04-22T08:20:09.628 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.622+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:20:09.628 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.622+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=985741910}) 2024-04-22T08:20:09.628 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.622+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2232870928}) 2024-04-22T08:20:09.628 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.622+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:20:09.628 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.622+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=985741910}) 2024-04-22T08:20:09.628 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.622+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1925939793}) 2024-04-22T08:20:09.628 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.626+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:20:09.628 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.626+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=56} crc {osdmap=456175423}) 2024-04-22T08:20:09.628 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.626+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=57} crc {osdmap=3691198145}) 2024-04-22T08:20:09.628 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.626+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:20:09.628 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.626+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=56} crc {osdmap=456175423}) 2024-04-22T08:20:09.628 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:20:09.626+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=56} crc {osdmap=3019273785}) 2024-04-22T08:20:10.283 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:20:10.283 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38308]: starting. 2024-04-22T08:20:10.284 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38308]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-38268 2024-04-22T08:20:10.284 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38308]: created object 0... 2024-04-22T08:20:10.284 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38308]: created object 25... 2024-04-22T08:20:10.284 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38308]: created object 49... 2024-04-22T08:20:10.284 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38308]: finishing. 2024-04-22T08:20:10.284 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38308]: shutting down. 2024-04-22T08:20:10.323 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:20:10.324 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38310]: starting. 2024-04-22T08:20:10.324 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38310]: listing objects. 2024-04-22T08:20:10.324 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38310]: listed object 0... 2024-04-22T08:20:10.324 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38310]: listed object 25... 2024-04-22T08:20:10.324 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38310]: saw 50 objects 2024-04-22T08:20:10.324 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38310]: shutting down. 2024-04-22T08:20:10.627 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-22T08:20:10.627 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:20:10.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.738+0000 7fa7f04db640 10 monclient: get_monmap_and_config 2024-04-22T08:20:10.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.738+0000 7fa7f04db640 10 monclient: build_initial_monmap 2024-04-22T08:20:10.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.738+0000 7fa7f04db640 10 monclient: monmap: 2024-04-22T08:20:10.739 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:20:10.739 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:10.739 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:20:10.739894+0000 2024-04-22T08:20:10.739 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:20:10.739894+0000 2024-04-22T08:20:10.739 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:20:10.739 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:10.739 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:20:10.739 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:20:10.739 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:20:10.739 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:20:10.739 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:20:10.740 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:10.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.738+0000 7fa7f04db640 1 Processor -- start 2024-04-22T08:20:10.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.738+0000 7fa7f04db640 1 -- start start 2024-04-22T08:20:10.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.738+0000 7fa7f04db640 10 monclient: init 2024-04-22T08:20:10.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.739+0000 7fa7f04db640 10 monclient: _reopen_session rank -1 2024-04-22T08:20:10.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.739+0000 7fa7f04db640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-04-22T08:20:10.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.739+0000 7fa7f04db640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7e8130810 0x7fa7e8130bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:10.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.739+0000 7fa7f04db640 10 monclient(hunting): picked mon.noname-b con 0x7fa7e8130810 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:20:10.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.739+0000 7fa7f04db640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa7e8071290 addr v1:172.21.15.189:6789/0 2024-04-22T08:20:10.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.739+0000 7fa7f04db640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa7e8130350 addr v1:172.21.15.40:6789/0 2024-04-22T08:20:10.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.739+0000 7fa7f04db640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7e8070bc0 con 0x7fa7e8130350 2024-04-22T08:20:10.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.739+0000 7fa7f04db640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7e80a0ff0 con 0x7fa7e8071290 2024-04-22T08:20:10.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.739+0000 7fa7f04db640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:10.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.739+0000 7fa7f04db640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e80708e0 con 0x7fa7e8130810 2024-04-22T08:20:10.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.739+0000 7fa7f04db640 10 monclient(hunting): _renew_subs 2024-04-22T08:20:10.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.739+0000 7fa7f04db640 10 monclient(hunting): authenticate will time out at 1284.734236s 2024-04-22T08:20:10.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.739+0000 7fa7ee277640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7e8130810 0x7fa7e8130bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:10.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.739+0000 7fa7eea78640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa7e8130350 0x7fa7e80a08e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53786/0 (socket says 172.21.15.40:53786) 2024-04-22T08:20:10.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.739+0000 7fa7ee277640 10 monclient(hunting): get_auth_request con 0x7fa7e8130810 auth_method 0 2024-04-22T08:20:10.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.739+0000 7fa7eea78640 1 -- 172.21.15.40:0/3639932188 learned_addr learned my addr 172.21.15.40:0/3639932188 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:20:10.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.739+0000 7fa7ee277640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:10.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.739+0000 7fa7ee277640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:10.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.739+0000 7fa7ee277640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:10.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.739+0000 7fa7ee277640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:10.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.739+0000 7fa7ee277640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:10.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.739+0000 7fa7ee277640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:10.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.739+0000 7fa7ee277640 10 monclient(hunting): handle_auth_done global_id 54848 payload 274 2024-04-22T08:20:10.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.739+0000 7fa7ee277640 1 -- 172.21.15.40:0/3639932188 >> v1:172.21.15.189:6789/0 conn(0x7fa7e8071290 legacy=0x7fa7e80a01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:20:10.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.739+0000 7fa7ee277640 1 -- 172.21.15.40:0/3639932188 >> v1:172.21.15.40:6789/0 conn(0x7fa7e8130350 legacy=0x7fa7e80a08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:20:10.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.739+0000 7fa7ee277640 10 monclient: _finish_hunting 0 2024-04-22T08:20:10.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.739+0000 7fa7ee277640 1 monclient: found mon.noname-b 2024-04-22T08:20:10.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.739+0000 7fa7ee277640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:20:10.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.739+0000 7fa7ee277640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:20:10.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.739+0000 7fa7ee277640 1 -- 172.21.15.40:0/3639932188 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e8070320 con 0x7fa7e8130810 2024-04-22T08:20:10.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.742+0000 7fa7ee277640 1 -- 172.21.15.40:0/3639932188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7e8130810 msgr2=0x7fa7e8130bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:10.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.742+0000 7fa7ee277640 1 --2- 172.21.15.40:0/3639932188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7e8130810 0x7fa7e8130bf0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa7d80097d0 tx=0x7fa7d802b6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e27a54b210b74be6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:10.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.743+0000 7fa7ed275640 1 -- 172.21.15.40:0/3639932188 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d8030a00 con 0x7fa7e8130810 2024-04-22T08:20:10.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.743+0000 7fa7ed275640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:10.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.743+0000 7fa7ed275640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:20:10.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.743+0000 7fa7ed275640 10 monclient: dump: 2024-04-22T08:20:10.752 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:10.752 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:10.752 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:10.753 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:10.753 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:10.753 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:10.753 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:10.753 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:10.753 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:10.753 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:10.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.743+0000 7fa7ed275640 10 monclient: _finish_auth 0 2024-04-22T08:20:10.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.743+0000 7fa7ed275640 10 monclient: _check_auth_tickets 2024-04-22T08:20:10.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.743+0000 7fa7ed275640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:20:10.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.743+0000 7fa7ed275640 1 -- 172.21.15.40:0/3639932188 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7d802e050 con 0x7fa7e8130810 2024-04-22T08:20:10.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.743+0000 7fa7f04db640 5 monclient: authenticate success, global_id 54848 2024-04-22T08:20:10.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.743+0000 7fa7f04db640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:20:10.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.743+0000 7fa7ed275640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:20:10.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.743+0000 7fa7f04db640 10 monclient: get_monmap_and_config success 2024-04-22T08:20:10.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.743+0000 7fa7f04db640 4 set_mon_vals no callback set 2024-04-22T08:20:10.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.743+0000 7fa7ed275640 1 -- 172.21.15.40:0/3639932188 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d8030d80 con 0x7fa7e8130810 2024-04-22T08:20:10.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.743+0000 7fa7ed275640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:10.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.743+0000 7fa7ed275640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:20:10.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.743+0000 7fa7ed275640 10 monclient: dump: 2024-04-22T08:20:10.754 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:10.754 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:10.754 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:10.754 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:10.754 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:10.754 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:10.754 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:10.755 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:10.755 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:10.755 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:10.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.743+0000 7fa7f04db640 10 monclient: shutdown 2024-04-22T08:20:10.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.743+0000 7fa7f04db640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:20:10.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.743+0000 7fa7f04db640 1 -- 172.21.15.40:0/3639932188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7e8130810 msgr2=0x7fa7e8130bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:10.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.743+0000 7fa7f04db640 1 --2- 172.21.15.40:0/3639932188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7e8130810 0x7fa7e8130bf0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa7d80097d0 tx=0x7fa7d802b6f0 comp rx=0 tx=0).stop 2024-04-22T08:20:10.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.743+0000 7fa7f04db640 1 -- 172.21.15.40:0/3639932188 shutdown_connections 2024-04-22T08:20:10.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.743+0000 7fa7f04db640 1 --2- 172.21.15.40:0/3639932188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7e8130810 0x7fa7e8130bf0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:10.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.743+0000 7fa7f04db640 1 -- 172.21.15.40:0/3639932188 >> 172.21.15.40:0/3639932188 conn(0x7fa7e80a18f0 msgr2=0x7fa7e806ba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:20:10.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.743+0000 7fa7f04db640 1 -- 172.21.15.40:0/3639932188 shutdown_connections 2024-04-22T08:20:10.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.743+0000 7fa7f04db640 1 -- 172.21.15.40:0/3639932188 wait complete. 2024-04-22T08:20:10.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.744+0000 7fa7f04db640 10 monclient: build_initial_monmap 2024-04-22T08:20:10.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.744+0000 7fa7f04db640 10 monclient: monmap: 2024-04-22T08:20:10.756 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:20:10.756 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:20:10.756 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:20:10.756 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:20:10.756 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:20:10.756 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:10.756 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:20:10.756 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:20:10.756 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:20:10.756 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:10.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.744+0000 7fa7f04db640 1 Processor -- start 2024-04-22T08:20:10.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.744+0000 7fa7f04db640 1 -- start start 2024-04-22T08:20:10.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.744+0000 7fa7f04db640 10 monclient: init 2024-04-22T08:20:10.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7f04db640 10 monclient: _reopen_session rank -1 2024-04-22T08:20:10.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7f04db640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-22T08:20:10.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7f04db640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7e8071290 0x7fa7e806dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:10.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7f04db640 10 monclient(hunting): picked mon.noname-c con 0x7fa7e8071290 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:20:10.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7f04db640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7e8130350 0x7fa7e806e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:10.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7f04db640 10 monclient(hunting): picked mon.noname-a con 0x7fa7e8130350 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:20:10.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7f04db640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa7e8130810 0x7fa7e806ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:10.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7f04db640 10 monclient(hunting): picked mon.noname-b con 0x7fa7e8130810 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:20:10.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7f04db640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:10.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7f04db640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e81235d0 con 0x7fa7e8130350 2024-04-22T08:20:10.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7f04db640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:10.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7f04db640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e806d8f0 con 0x7fa7e8130810 2024-04-22T08:20:10.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7f04db640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:10.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7f04db640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e812db50 con 0x7fa7e8071290 2024-04-22T08:20:10.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7f04db640 10 monclient(hunting): _renew_subs 2024-04-22T08:20:10.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7f04db640 10 monclient(hunting): authenticate will time out at 1284.740142s 2024-04-22T08:20:10.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7ee277640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7e8071290 0x7fa7e806dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:10.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7eda76640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7e8130350 0x7fa7e806e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:10.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7ee277640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7e8071290 0x7fa7e806dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:58024/0 (socket says 172.21.15.40:58024) 2024-04-22T08:20:10.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7eda76640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7e8130350 0x7fa7e806e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46354/0 (socket says 172.21.15.40:46354) 2024-04-22T08:20:10.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7ee277640 1 -- 172.21.15.40:0/843048408 learned_addr learned my addr 172.21.15.40:0/843048408 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:20:10.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7ee277640 10 monclient(hunting): get_auth_request con 0x7fa7e8071290 auth_method 0 2024-04-22T08:20:10.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7ee277640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:10.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7ee277640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:10.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7ee277640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:10.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7eda76640 10 monclient(hunting): get_auth_request con 0x7fa7e8130350 auth_method 0 2024-04-22T08:20:10.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7eda76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:10.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7eda76640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:10.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7eda76640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:10.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7ee277640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:10.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7ee277640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:10.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7ee277640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:10.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7eda76640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:10.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7eda76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:10.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7eda76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:10.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7eea78640 1 --2- 172.21.15.40:0/843048408 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa7e8130810 0x7fa7e806ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:10.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7eea78640 10 monclient(hunting): get_auth_request con 0x7fa7e8130810 auth_method 0 2024-04-22T08:20:10.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7eea78640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:10.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7eea78640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:10.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7eea78640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:10.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7ee277640 10 monclient(hunting): handle_auth_done global_id 54851 payload 824 2024-04-22T08:20:10.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7ee277640 1 -- 172.21.15.40:0/843048408 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa7e8130810 msgr2=0x7fa7e806ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:10.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7ee277640 1 --2- 172.21.15.40:0/843048408 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa7e8130810 0x7fa7e806ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:10.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7ee277640 1 -- 172.21.15.40:0/843048408 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7e8130350 msgr2=0x7fa7e806e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:10.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7ee277640 1 --2- 172.21.15.40:0/843048408 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7e8130350 0x7fa7e806e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:10.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7ee277640 10 monclient: _finish_hunting 0 2024-04-22T08:20:10.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7ee277640 1 monclient: found mon.noname-c 2024-04-22T08:20:10.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7ee277640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:20:10.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7ee277640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:20:10.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7ee277640 1 -- 172.21.15.40:0/843048408 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7d8009400 con 0x7fa7e8071290 2024-04-22T08:20:10.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.745+0000 7fa7eda76640 1 --2- 172.21.15.40:0/843048408 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7e8130350 0x7fa7e806e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:20:10.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.748+0000 7fa7ee277640 1 -- 172.21.15.40:0/843048408 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7e8071290 msgr2=0x7fa7e806dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:10.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.748+0000 7fa7ee277640 1 --2- 172.21.15.40:0/843048408 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7e8071290 0x7fa7e806dfd0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa7d802bd00 tx=0x7fa7d80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:10.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.748+0000 7fa7deffd640 1 -- 172.21.15.40:0/843048408 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d803db60 con 0x7fa7e8071290 2024-04-22T08:20:10.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.748+0000 7fa7deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:10.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.748+0000 7fa7deffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:20:10.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.748+0000 7fa7deffd640 10 monclient: dump: 2024-04-22T08:20:10.761 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:10.761 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:10.761 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:10.761 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:10.761 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:10.761 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:10.762 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:10.762 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:10.762 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:10.762 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:10.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.748+0000 7fa7deffd640 10 monclient: _finish_auth 0 2024-04-22T08:20:10.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.748+0000 7fa7deffd640 10 monclient: _check_auth_tickets 2024-04-22T08:20:10.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.748+0000 7fa7deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:20:10.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.748+0000 7fa7deffd640 1 -- 172.21.15.40:0/843048408 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7d803dcc0 con 0x7fa7e8071290 2024-04-22T08:20:10.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.748+0000 7fa7deffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:20:10.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.748+0000 7fa7deffd640 1 -- 172.21.15.40:0/843048408 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d80326c0 con 0x7fa7e8071290 2024-04-22T08:20:10.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.748+0000 7fa7f04db640 5 monclient: authenticate success, global_id 54851 2024-04-22T08:20:10.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.748+0000 7fa7f04db640 10 monclient: _renew_subs 2024-04-22T08:20:10.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.748+0000 7fa7f04db640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:20:10.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.748+0000 7fa7f04db640 1 -- 172.21.15.40:0/843048408 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7e81312b0 con 0x7fa7e8071290 2024-04-22T08:20:10.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.748+0000 7fa7eca74640 4 set_mon_vals no callback set 2024-04-22T08:20:10.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.748+0000 7fa7deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:10.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.748+0000 7fa7deffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:20:10.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.748+0000 7fa7deffd640 10 monclient: dump: 2024-04-22T08:20:10.763 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:10.763 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:10.763 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:10.763 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:10.763 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:10.763 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:10.763 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:10.763 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:10.763 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:10.764 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:10.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.748+0000 7fa7f04db640 10 monclient: _renew_subs 2024-04-22T08:20:10.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.748+0000 7fa7f04db640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:20:10.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.748+0000 7fa7f04db640 1 -- 172.21.15.40:0/843048408 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7e81232f0 con 0x7fa7e8071290 2024-04-22T08:20:10.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.749+0000 7fa7deffd640 1 -- 172.21.15.40:0/843048408 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fa7d8039040 con 0x7fa7e8071290 2024-04-22T08:20:10.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.749+0000 7fa7deffd640 1 --2- 172.21.15.40:0/843048408 >> v2:172.21.15.189:6800/30168 conn(0x7fa7cc037820 0x7fa7cc039ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:10.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.749+0000 7fa7f04db640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:20:10.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.749+0000 7fa7f04db640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:20:10.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.749+0000 7fa7deffd640 1 -- 172.21.15.40:0/843048408 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(242..242 src has 1..242) v4 ==== 3898+0+0 (secure 0 0 0) 0x7fa7d806c470 con 0x7fa7e8071290 2024-04-22T08:20:10.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.749+0000 7fa7f04db640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:20:10.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.749+0000 7fa7f04db640 1 -- 172.21.15.40:0/843048408 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7e8123a40 con 0x7fa7e8071290 2024-04-22T08:20:10.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.749+0000 7fa7eda76640 1 --2- 172.21.15.40:0/843048408 >> v2:172.21.15.189:6800/30168 conn(0x7fa7cc037820 0x7fa7cc039ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:10.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.749+0000 7fa7eda76640 10 monclient: get_auth_request con 0x7fa7cc037820 auth_method 0 2024-04-22T08:20:10.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.752+0000 7fa7eda76640 1 --2- 172.21.15.40:0/843048408 >> v2:172.21.15.189:6800/30168 conn(0x7fa7cc037820 0x7fa7cc039ce0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fa7e81302e0 tx=0x7fa7e4007e30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:10.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.754+0000 7fa7deffd640 1 -- 172.21.15.40:0/843048408 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7fa7d8037070 con 0x7fa7e8071290 2024-04-22T08:20:10.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.754+0000 7fa7deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:20:10.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:10.754+0000 7fa7deffd640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:20:11.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.004+0000 7fa7f04db640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:20:11.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.004+0000 7fa7f04db640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:20:11.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.004+0000 7fa7f04db640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:20:11.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.004+0000 7fa7f04db640 1 -- 172.21.15.40:0/843048408 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa7e804e9b0 con 0x7fa7e8071290 2024-04-22T08:20:11.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.004+0000 7fa7deffd640 1 -- 172.21.15.40:0/843048408 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7fa7d803de30 con 0x7fa7e8071290 2024-04-22T08:20:11.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.004+0000 7fa7deffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:20:11.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.004+0000 7fa7deffd640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:20:11.006 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":72,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:20:11.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.006+0000 7fa7f04db640 1 -- 172.21.15.40:0/843048408 >> v2:172.21.15.189:6800/30168 conn(0x7fa7cc037820 msgr2=0x7fa7cc039ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:11.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.006+0000 7fa7f04db640 1 --2- 172.21.15.40:0/843048408 >> v2:172.21.15.189:6800/30168 conn(0x7fa7cc037820 0x7fa7cc039ce0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fa7e81302e0 tx=0x7fa7e4007e30 comp rx=0 tx=0).stop 2024-04-22T08:20:11.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.006+0000 7fa7f04db640 10 monclient: shutdown 2024-04-22T08:20:11.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.006+0000 7fa7f04db640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:20:11.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.006+0000 7fa7f04db640 1 -- 172.21.15.40:0/843048408 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7e8071290 msgr2=0x7fa7e806dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:11.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.006+0000 7fa7f04db640 1 --2- 172.21.15.40:0/843048408 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7e8071290 0x7fa7e806dfd0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa7d802bd00 tx=0x7fa7d80046c0 comp rx=0 tx=0).stop 2024-04-22T08:20:11.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.006+0000 7fa7f04db640 1 -- 172.21.15.40:0/843048408 shutdown_connections 2024-04-22T08:20:11.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.006+0000 7fa7f04db640 1 --2- 172.21.15.40:0/843048408 >> v2:172.21.15.189:6800/30168 conn(0x7fa7cc037820 0x7fa7cc039ce0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:11.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.006+0000 7fa7f04db640 1 --2- 172.21.15.40:0/843048408 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa7e8130810 0x7fa7e806ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:11.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.006+0000 7fa7f04db640 1 --2- 172.21.15.40:0/843048408 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7e8130350 0x7fa7e806e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:11.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.006+0000 7fa7f04db640 1 --2- 172.21.15.40:0/843048408 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7e8071290 0x7fa7e806dfd0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:11.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.006+0000 7fa7f04db640 1 -- 172.21.15.40:0/843048408 >> 172.21.15.40:0/843048408 conn(0x7fa7e80a18f0 msgr2=0x7fa7e80a08a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:20:11.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.006+0000 7fa7f04db640 1 -- 172.21.15.40:0/843048408 shutdown_connections 2024-04-22T08:20:11.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.006+0000 7fa7f04db640 1 -- 172.21.15.40:0/843048408 wait complete. 2024-04-22T08:20:11.018 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-22T08:20:11.018 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-22T08:20:11.018 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-22T08:20:11.019 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-22T08:20:11.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.118+0000 7f4fb8fb9640 10 monclient: get_monmap_and_config 2024-04-22T08:20:11.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.118+0000 7f4fb8fb9640 10 monclient: build_initial_monmap 2024-04-22T08:20:11.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.118+0000 7f4fb8fb9640 10 monclient: monmap: 2024-04-22T08:20:11.119 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:20:11.119 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:11.119 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:20:11.120153+0000 2024-04-22T08:20:11.119 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:20:11.120153+0000 2024-04-22T08:20:11.120 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:20:11.120 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:11.120 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:20:11.120 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:20:11.120 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:20:11.120 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:20:11.120 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:20:11.120 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:11.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.118+0000 7f4fb8fb9640 1 Processor -- start 2024-04-22T08:20:11.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.118+0000 7f4fb8fb9640 1 -- start start 2024-04-22T08:20:11.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.118+0000 7f4fb8fb9640 10 monclient: init 2024-04-22T08:20:11.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.119+0000 7f4fb8fb9640 10 monclient: _reopen_session rank -1 2024-04-22T08:20:11.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.119+0000 7f4fb8fb9640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-04-22T08:20:11.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.119+0000 7f4fb8fb9640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4fb40736d0 0x7f4fb4135f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:11.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.119+0000 7f4fb8fb9640 10 monclient(hunting): picked mon.noname-b con 0x7f4fb40736d0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:20:11.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.119+0000 7f4fb8fb9640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4fb4070eb0 0x7f4fb40712f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:11.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.119+0000 7f4fb8fb9640 10 monclient(hunting): picked mon.noname-a con 0x7f4fb4070eb0 addr v2:172.21.15.40:3300/0 2024-04-22T08:20:11.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.119+0000 7f4fb8fb9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4fb409de00 addr v1:172.21.15.40:6789/0 2024-04-22T08:20:11.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.119+0000 7f4fb8fb9640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fb4073ab0 con 0x7f4fb409de00 2024-04-22T08:20:11.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.119+0000 7f4fb8fb9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:11.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.119+0000 7f4fb8fb9640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fb4071830 con 0x7f4fb4070eb0 2024-04-22T08:20:11.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.119+0000 7f4fb8fb9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:11.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.119+0000 7f4fb8fb9640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb4136510 con 0x7f4fb40736d0 2024-04-22T08:20:11.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.119+0000 7f4fb8fb9640 10 monclient(hunting): _renew_subs 2024-04-22T08:20:11.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.119+0000 7f4fb8fb9640 10 monclient(hunting): authenticate will time out at 1285.114388s 2024-04-22T08:20:11.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.119+0000 7f4fb259c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4fb40736d0 0x7f4fb4135f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:11.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.119+0000 7f4fb1d9b640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4fb4070eb0 0x7f4fb40712f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:11.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.119+0000 7f4fb1d9b640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4fb4070eb0 0x7f4fb40712f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46356/0 (socket says 172.21.15.40:46356) 2024-04-22T08:20:11.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.119+0000 7f4fb259c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4fb40736d0 0x7f4fb4135f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:58032/0 (socket says 172.21.15.40:58032) 2024-04-22T08:20:11.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.119+0000 7f4fb1d9b640 1 -- 172.21.15.40:0/1533863741 learned_addr learned my addr 172.21.15.40:0/1533863741 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:20:11.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.119+0000 7f4fb1d9b640 10 monclient(hunting): get_auth_request con 0x7f4fb4070eb0 auth_method 0 2024-04-22T08:20:11.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.119+0000 7f4fb1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:11.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.119+0000 7f4fb1d9b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:11.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.119+0000 7f4fb1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:11.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.119+0000 7f4fb259c640 10 monclient(hunting): get_auth_request con 0x7f4fb40736d0 auth_method 0 2024-04-22T08:20:11.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.119+0000 7f4fb259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:11.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.119+0000 7f4fb259c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:11.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.119+0000 7f4fb259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:11.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.119+0000 7f4fb1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:11.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.119+0000 7f4fb1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:11.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.119+0000 7f4fb1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:11.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.119+0000 7f4fb259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:11.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.119+0000 7f4fb259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:11.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.119+0000 7f4fb259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:11.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.120+0000 7f4fb1d9b640 10 monclient(hunting): handle_auth_done global_id 94131 payload 274 2024-04-22T08:20:11.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.120+0000 7f4fb1d9b640 1 -- 172.21.15.40:0/1533863741 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4fb40736d0 msgr2=0x7f4fb4135f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:20:11.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.120+0000 7f4fb1d9b640 1 --2- 172.21.15.40:0/1533863741 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4fb40736d0 0x7f4fb4135f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:11.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.120+0000 7f4fb1d9b640 1 -- 172.21.15.40:0/1533863741 >> v1:172.21.15.40:6789/0 conn(0x7f4fb409de00 legacy=0x7f4fb409e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:20:11.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.120+0000 7f4fb1d9b640 10 monclient: _finish_hunting 0 2024-04-22T08:20:11.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.120+0000 7f4fb1d9b640 1 monclient: found mon.noname-a 2024-04-22T08:20:11.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.120+0000 7f4fb1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:20:11.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.120+0000 7f4fb1d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:20:11.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.120+0000 7f4fb1d9b640 1 -- 172.21.15.40:0/1533863741 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fb406ce90 con 0x7f4fb4070eb0 2024-04-22T08:20:11.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.120+0000 7f4fb259c640 1 --2- 172.21.15.40:0/1533863741 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4fb40736d0 0x7f4fb4135f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:20:11.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.122+0000 7f4fb1d9b640 1 -- 172.21.15.40:0/1533863741 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fb4070eb0 msgr2=0x7f4fb40712f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:11.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.122+0000 7f4fb1d9b640 1 --2- 172.21.15.40:0/1533863741 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fb4070eb0 0x7f4fb40712f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4fb409ef20 tx=0x7f4f9c02b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c56359406566e775 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:11.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.123+0000 7f4fb159a640 1 -- 172.21.15.40:0/1533863741 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f9c02f050 con 0x7f4fb4070eb0 2024-04-22T08:20:11.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.123+0000 7f4fb159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:11.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.123+0000 7f4fb159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:20:11.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.123+0000 7f4fb159a640 10 monclient: dump: 2024-04-22T08:20:11.133 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:11.133 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:11.133 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:11.133 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:11.133 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:11.133 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:11.133 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:11.133 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:11.133 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:11.133 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:11.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.123+0000 7f4fb159a640 10 monclient: _finish_auth 0 2024-04-22T08:20:11.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.123+0000 7f4fb159a640 10 monclient: _check_auth_tickets 2024-04-22T08:20:11.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.123+0000 7f4fb159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:20:11.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.123+0000 7f4fb159a640 1 -- 172.21.15.40:0/1533863741 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f9c031910 con 0x7f4fb4070eb0 2024-04-22T08:20:11.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.123+0000 7f4fb8fb9640 5 monclient: authenticate success, global_id 94131 2024-04-22T08:20:11.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.123+0000 7f4fb8fb9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:20:11.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.123+0000 7f4fb159a640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:20:11.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.123+0000 7f4fb8fb9640 10 monclient: get_monmap_and_config success 2024-04-22T08:20:11.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.123+0000 7f4fb159a640 1 -- 172.21.15.40:0/1533863741 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f9c031ca0 con 0x7f4fb4070eb0 2024-04-22T08:20:11.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.123+0000 7f4fb159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:11.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.123+0000 7f4fb8fb9640 4 set_mon_vals no callback set 2024-04-22T08:20:11.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.123+0000 7f4fb159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:20:11.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.123+0000 7f4fb159a640 10 monclient: dump: 2024-04-22T08:20:11.134 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:11.134 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:11.134 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:11.134 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:11.135 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:11.135 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:11.135 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:11.135 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:11.135 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:11.135 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:11.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.123+0000 7f4fb8fb9640 10 monclient: shutdown 2024-04-22T08:20:11.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.123+0000 7f4fb8fb9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:20:11.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.123+0000 7f4fb8fb9640 1 -- 172.21.15.40:0/1533863741 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fb4070eb0 msgr2=0x7f4fb40712f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:11.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.123+0000 7f4fb8fb9640 1 --2- 172.21.15.40:0/1533863741 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fb4070eb0 0x7f4fb40712f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4fb409ef20 tx=0x7f4f9c02b7a0 comp rx=0 tx=0).stop 2024-04-22T08:20:11.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.123+0000 7f4fb8fb9640 1 -- 172.21.15.40:0/1533863741 shutdown_connections 2024-04-22T08:20:11.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.123+0000 7f4fb8fb9640 1 --2- 172.21.15.40:0/1533863741 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fb4070eb0 0x7f4fb40712f0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:11.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.123+0000 7f4fb8fb9640 1 --2- 172.21.15.40:0/1533863741 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4fb40736d0 0x7f4fb4135f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:11.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.123+0000 7f4fb8fb9640 1 -- 172.21.15.40:0/1533863741 >> 172.21.15.40:0/1533863741 conn(0x7f4fb40a1780 msgr2=0x7f4fb40a1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:20:11.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.123+0000 7f4fb8fb9640 1 -- 172.21.15.40:0/1533863741 shutdown_connections 2024-04-22T08:20:11.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.123+0000 7f4fb8fb9640 1 -- 172.21.15.40:0/1533863741 wait complete. 2024-04-22T08:20:11.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb8fb9640 10 monclient: build_initial_monmap 2024-04-22T08:20:11.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb8fb9640 10 monclient: monmap: 2024-04-22T08:20:11.136 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:20:11.136 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:20:11.136 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:20:11.136 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:20:11.136 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:20:11.136 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:11.136 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:20:11.136 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:20:11.136 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:20:11.136 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:11.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb8fb9640 1 Processor -- start 2024-04-22T08:20:11.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb8fb9640 1 -- start start 2024-04-22T08:20:11.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb8fb9640 10 monclient: init 2024-04-22T08:20:11.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb8fb9640 10 monclient: _reopen_session rank -1 2024-04-22T08:20:11.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb8fb9640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-22T08:20:11.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb8fb9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fb4070eb0 0x7f4fb41cadc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:11.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb8fb9640 10 monclient(hunting): picked mon.noname-b con 0x7f4fb4070eb0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:20:11.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb8fb9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fb40736d0 0x7f4fb411c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:11.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb8fb9640 10 monclient(hunting): picked mon.noname-a con 0x7f4fb40736d0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:20:11.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb8fb9640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4fb409de00 0x7f4fb411cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:11.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb8fb9640 10 monclient(hunting): picked mon.noname-c con 0x7f4fb409de00 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:20:11.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb8fb9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:11.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb8fb9640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb411ffa0 con 0x7f4fb40736d0 2024-04-22T08:20:11.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb8fb9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:11.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb8fb9640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb40a4090 con 0x7f4fb4070eb0 2024-04-22T08:20:11.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb8fb9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:11.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb8fb9640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb4135260 con 0x7f4fb409de00 2024-04-22T08:20:11.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb8fb9640 10 monclient(hunting): _renew_subs 2024-04-22T08:20:11.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb8fb9640 10 monclient(hunting): authenticate will time out at 1285.119990s 2024-04-22T08:20:11.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb1d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fb40736d0 0x7f4fb411c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:11.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb2d9d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4fb409de00 0x7f4fb411cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:11.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb1d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fb40736d0 0x7f4fb411c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46366/0 (socket says 172.21.15.40:46366) 2024-04-22T08:20:11.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb1d9b640 1 -- 172.21.15.40:0/3624849470 learned_addr learned my addr 172.21.15.40:0/3624849470 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:20:11.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb1d9b640 10 monclient(hunting): get_auth_request con 0x7f4fb40736d0 auth_method 0 2024-04-22T08:20:11.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:11.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb1d9b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:11.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:11.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb2d9d640 10 monclient(hunting): get_auth_request con 0x7f4fb409de00 auth_method 0 2024-04-22T08:20:11.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:11.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb2d9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:11.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:11.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:11.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:11.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:11.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:11.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:11.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:11.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb259c640 1 --2- 172.21.15.40:0/3624849470 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fb4070eb0 0x7f4fb41cadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:11.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb259c640 10 monclient(hunting): get_auth_request con 0x7f4fb4070eb0 auth_method 0 2024-04-22T08:20:11.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:11.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb259c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:11.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:11.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb2d9d640 10 monclient(hunting): handle_auth_done global_id 54857 payload 824 2024-04-22T08:20:11.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb2d9d640 1 -- 172.21.15.40:0/3624849470 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fb4070eb0 msgr2=0x7f4fb41cadc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:11.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb2d9d640 1 --2- 172.21.15.40:0/3624849470 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fb4070eb0 0x7f4fb41cadc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:11.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb2d9d640 1 -- 172.21.15.40:0/3624849470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fb40736d0 msgr2=0x7f4fb411c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:11.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb2d9d640 1 --2- 172.21.15.40:0/3624849470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fb40736d0 0x7f4fb411c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:11.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb2d9d640 10 monclient: _finish_hunting 0 2024-04-22T08:20:11.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb2d9d640 1 monclient: found mon.noname-c 2024-04-22T08:20:11.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:20:11.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb2d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:20:11.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.125+0000 7f4fb2d9d640 1 -- 172.21.15.40:0/3624849470 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fac009400 con 0x7f4fb409de00 2024-04-22T08:20:11.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.127+0000 7f4fb2d9d640 1 -- 172.21.15.40:0/3624849470 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4fb409de00 msgr2=0x7f4fb411cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:11.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.127+0000 7f4fb2d9d640 1 --2- 172.21.15.40:0/3624849470 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4fb409de00 0x7f4fb411cdf0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4fa400c7b0 tx=0x7f4fa400cc70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:11.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.128+0000 7f4faaffd640 1 -- 172.21.15.40:0/3624849470 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fa400ac60 con 0x7f4fb409de00 2024-04-22T08:20:11.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.128+0000 7f4faaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:11.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.128+0000 7f4faaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:20:11.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.128+0000 7f4faaffd640 10 monclient: dump: 2024-04-22T08:20:11.141 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:11.141 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:11.141 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:11.141 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:11.141 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:11.141 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:11.141 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:11.141 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:11.141 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:11.141 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:11.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.128+0000 7f4faaffd640 10 monclient: _finish_auth 0 2024-04-22T08:20:11.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.128+0000 7f4faaffd640 10 monclient: _check_auth_tickets 2024-04-22T08:20:11.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.128+0000 7f4faaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:20:11.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.128+0000 7f4faaffd640 1 -- 172.21.15.40:0/3624849470 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fa400adc0 con 0x7f4fb409de00 2024-04-22T08:20:11.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.128+0000 7f4fb8fb9640 5 monclient: authenticate success, global_id 54857 2024-04-22T08:20:11.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.128+0000 7f4fb8fb9640 10 monclient: _renew_subs 2024-04-22T08:20:11.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.128+0000 7f4fb8fb9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:20:11.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.128+0000 7f4fb8fb9640 1 -- 172.21.15.40:0/3624849470 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f9c0094d0 con 0x7f4fb409de00 2024-04-22T08:20:11.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.128+0000 7f4faaffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:20:11.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.128+0000 7f4faaffd640 1 -- 172.21.15.40:0/3624849470 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fa4012660 con 0x7f4fb409de00 2024-04-22T08:20:11.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.128+0000 7f4faaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:11.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.128+0000 7f4faaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:20:11.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.128+0000 7f4faaffd640 10 monclient: dump: 2024-04-22T08:20:11.142 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:11.142 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:11.142 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:11.142 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:11.143 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:11.143 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:11.143 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:11.143 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:11.143 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:11.143 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:11.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.128+0000 7f4fb0d99640 4 set_mon_vals no callback set 2024-04-22T08:20:11.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.128+0000 7f4fb8fb9640 10 monclient: _renew_subs 2024-04-22T08:20:11.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.128+0000 7f4fb8fb9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:20:11.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.128+0000 7f4fb8fb9640 1 -- 172.21.15.40:0/3624849470 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fb412fdf0 con 0x7f4fb409de00 2024-04-22T08:20:11.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.128+0000 7f4fb8fb9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:20:11.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.128+0000 7f4fb8fb9640 1 --2- 172.21.15.40:0/3624849470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fb41cb300 0x7f4fb402b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:11.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.128+0000 7f4fb8fb9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:11.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.128+0000 7f4fb8fb9640 1 -- 172.21.15.40:0/3624849470 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb4066900 con 0x7f4fb41cb300 2024-04-22T08:20:11.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.128+0000 7f4faaffd640 1 -- 172.21.15.40:0/3624849470 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f4fa4012840 con 0x7f4fb409de00 2024-04-22T08:20:11.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.129+0000 7f4fb259c640 1 --2- 172.21.15.40:0/3624849470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fb41cb300 0x7f4fb402b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:11.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.129+0000 7f4fb259c640 10 monclient: get_auth_request con 0x7f4fb41cb300 auth_method 0 2024-04-22T08:20:11.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.129+0000 7f4fb259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:11.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.129+0000 7f4fb259c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:11.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.129+0000 7f4fb259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:11.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.129+0000 7f4fb259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:11.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.129+0000 7f4fb259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:11.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.129+0000 7f4fb259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:11.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.129+0000 7f4faaffd640 1 --2- 172.21.15.40:0/3624849470 >> v2:172.21.15.189:6800/30168 conn(0x7f4f94037820 0x7f4f94039ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:11.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.129+0000 7f4faaffd640 1 -- 172.21.15.40:0/3624849470 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(242..242 src has 1..242) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f4fa404a490 con 0x7f4fb409de00 2024-04-22T08:20:11.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.129+0000 7f4fb259c640 10 monclient(hunting): handle_auth_done global_id 94137 payload 824 2024-04-22T08:20:11.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.129+0000 7f4fb259c640 1 -- 172.21.15.40:0/3624849470 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4fb40a3980 con 0x7f4fb41cb300 2024-04-22T08:20:11.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.129+0000 7f4fb1d9b640 1 --2- 172.21.15.40:0/3624849470 >> v2:172.21.15.189:6800/30168 conn(0x7f4f94037820 0x7f4f94039ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:11.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.129+0000 7f4fb1d9b640 10 monclient: get_auth_request con 0x7f4f94037820 auth_method 0 2024-04-22T08:20:11.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.131+0000 7f4fb259c640 1 -- 172.21.15.40:0/3624849470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fb41cb300 msgr2=0x7f4fb402b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:11.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.131+0000 7f4fb259c640 1 --2- 172.21.15.40:0/3624849470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fb41cb300 0x7f4fb402b200 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4fac011830 tx=0x7f4fac011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:11.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.131+0000 7f4faaffd640 1 -- 172.21.15.40:0/3624849470 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fac00fe50 con 0x7f4fb41cb300 2024-04-22T08:20:11.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.131+0000 7f4faaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:11.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.131+0000 7f4faaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:20:11.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.131+0000 7f4faaffd640 10 monclient: dump: 2024-04-22T08:20:11.145 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:11.145 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:11.145 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:11.145 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:11.145 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:11.145 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:11.145 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:11.146 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:11.146 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:11.146 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:11.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.131+0000 7f4faaffd640 1 -- 172.21.15.40:0/3624849470 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f4fac004030 con 0x7f4fb41cb300 2024-04-22T08:20:11.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.131+0000 7f4faaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:20:11.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.131+0000 7f4faaffd640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:20:11.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.131+0000 7f4faaffd640 1 -- 172.21.15.40:0/3624849470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fb41cb300 msgr2=0x7f4fb402b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:11.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.131+0000 7f4faaffd640 1 --2- 172.21.15.40:0/3624849470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fb41cb300 0x7f4fb402b200 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4fac011830 tx=0x7f4fac011cf0 comp rx=0 tx=0).stop 2024-04-22T08:20:11.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.131+0000 7f4faaffd640 1 -- 172.21.15.40:0/3624849470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fb41cb300 msgr2=0x7f4fb402b200 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:20:11.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.131+0000 7f4faaffd640 1 --2- 172.21.15.40:0/3624849470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fb41cb300 0x7f4fb402b200 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4fac011830 tx=0x7f4fac011cf0 comp rx=0 tx=0).stop 2024-04-22T08:20:11.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.131+0000 7f4fb1d9b640 1 --2- 172.21.15.40:0/3624849470 >> v2:172.21.15.189:6800/30168 conn(0x7f4f94037820 0x7f4f94039ce0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f4fb41344e0 tx=0x7f4f9c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:11.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.138+0000 7f4fb8fb9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:20:11.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.138+0000 7f4fb8fb9640 1 --2- 172.21.15.40:0/3624849470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fb402b740 0x7f4fb402bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:11.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.138+0000 7f4fb8fb9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:11.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.138+0000 7f4fb8fb9640 1 -- 172.21.15.40:0/3624849470 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb41205e0 con 0x7f4fb402b740 2024-04-22T08:20:11.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.138+0000 7f4fb259c640 1 --2- 172.21.15.40:0/3624849470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fb402b740 0x7f4fb402bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:11.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.138+0000 7f4fb259c640 10 monclient: get_auth_request con 0x7f4fb402b740 auth_method 0 2024-04-22T08:20:11.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.138+0000 7f4fb259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:11.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.138+0000 7f4fb259c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:11.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.138+0000 7f4fb259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:11.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.138+0000 7f4fb259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:11.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.138+0000 7f4fb259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:11.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.138+0000 7f4fb259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:11.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.138+0000 7f4fb259c640 10 monclient(hunting): handle_auth_done global_id 94140 payload 824 2024-04-22T08:20:11.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.138+0000 7f4fb259c640 1 -- 172.21.15.40:0/3624849470 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4fb406d450 con 0x7f4fb402b740 2024-04-22T08:20:11.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.141+0000 7f4fb259c640 1 -- 172.21.15.40:0/3624849470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fb402b740 msgr2=0x7f4fb402bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:11.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.141+0000 7f4fb259c640 1 --2- 172.21.15.40:0/3624849470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fb402b740 0x7f4fb402bb50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4fac01b000 tx=0x7f4fac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:11.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.141+0000 7f4faaffd640 1 -- 172.21.15.40:0/3624849470 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fac005da0 con 0x7f4fb402b740 2024-04-22T08:20:11.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.141+0000 7f4faaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:11.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.141+0000 7f4faaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:20:11.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.141+0000 7f4faaffd640 10 monclient: dump: 2024-04-22T08:20:11.150 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:11.150 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:11.150 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:11.150 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:11.150 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:11.150 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:11.150 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:11.150 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:11.150 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:11.150 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:11.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.141+0000 7f4faaffd640 1 -- 172.21.15.40:0/3624849470 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4463 (secure 0 0 0) 0x7f4fac003e80 con 0x7f4fb402b740 2024-04-22T08:20:11.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.141+0000 7f4faaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:20:11.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.141+0000 7f4faaffd640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:20:11.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.141+0000 7f4faaffd640 1 -- 172.21.15.40:0/3624849470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fb402b740 msgr2=0x7f4fb402bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:11.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.141+0000 7f4faaffd640 1 --2- 172.21.15.40:0/3624849470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fb402b740 0x7f4fb402bb50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4fac01b000 tx=0x7f4fac0046c0 comp rx=0 tx=0).stop 2024-04-22T08:20:11.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.141+0000 7f4faaffd640 1 -- 172.21.15.40:0/3624849470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fb402b740 msgr2=0x7f4fb402bb50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:20:11.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.141+0000 7f4faaffd640 1 --2- 172.21.15.40:0/3624849470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fb402b740 0x7f4fb402bb50 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4fac01b000 tx=0x7f4fac0046c0 comp rx=0 tx=0).stop 2024-04-22T08:20:11.151 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:20:11.151 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:20:11.151 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:20:11.151 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "leader", 2024-04-22T08:20:11.151 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 72, 2024-04-22T08:20:11.152 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:20:11.152 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:20:11.152 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:20:11.152 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:20:11.152 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:20:11.152 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 2, 2024-04-22T08:20:11.152 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:20:11.152 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:20:11.152 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:20:11.152 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:20:11.152 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:20:11.152 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:20:11.152 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:20:11.153 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:20:11.153 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:20:11.153 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:20:11.153 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:20:11.153 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:20:11.153 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:20:11.153 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:20:11.153 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:20:11.153 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:20:11.153 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:20:11.153 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:20:11.153 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:20:11.153 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:20:11.153 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:20:11.153 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:20:11.154 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:20:11.154 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:20:11.154 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:20:11.154 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:20:11.154 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:11.154 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:11.154 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:20:11.154 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:20:11.154 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:20:11.154 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:20:11.154 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:20:11.154 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:20:11.154 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:20:11.155 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:20:11.155 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:20:11.155 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:20:11.155 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:20:11.155 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:20:11.155 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:20:11.155 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:20:11.155 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:20:11.155 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:20:11.155 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:20:11.155 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:20:11.155 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:20:11.155 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:20:11.155 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:20:11.156 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:20:11.156 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:20:11.156 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:20:11.156 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:20:11.156 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:20:11.156 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:20:11.156 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:20:11.156 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:20:11.156 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:11.156 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:20:11.156 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:11.156 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:20:11.156 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:20:11.156 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:20:11.157 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:20:11.157 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:11.157 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:20:11.157 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:20:11.157 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:11.157 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:11.157 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:11.157 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:20:11.157 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:20:11.157 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:11.157 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:11.157 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:11.157 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:11.157 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:20:11.158 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:20:11.158 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:20:11.158 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:20:11.158 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:20:11.158 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:11.158 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:11.158 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:20:11.158 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:20:11.158 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:20:11.158 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:20:11.158 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:11.158 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:20:11.158 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:20:11.158 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:11.159 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:11.159 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:11.159 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:20:11.159 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:20:11.159 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:11.159 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:11.159 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:11.159 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:11.159 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:20:11.159 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:20:11.159 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:20:11.159 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:20:11.159 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:20:11.159 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:11.160 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:11.160 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:20:11.160 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:20:11.160 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:20:11.160 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:20:11.160 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:11.160 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:20:11.160 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:20:11.160 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:11.160 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:11.160 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:11.160 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:20:11.160 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:20:11.160 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:11.161 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:11.161 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:11.161 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:11.161 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:20:11.161 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:20:11.161 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:20:11.161 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:20:11.161 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:20:11.161 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:11.161 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:11.161 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:11.161 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:20:11.161 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:20:11.161 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:11.162 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:20:11.162 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:20:11.162 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:20:11.162 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:11.162 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:20:11.162 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:20:11.162 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:11.162 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:20:11.162 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:20:11.162 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:20:11.162 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:11.162 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:11.162 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:11.162 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:20:11.163 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:20:11.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.141+0000 7f4fb8fb9640 1 -- 172.21.15.40:0/3624849470 >> v2:172.21.15.189:6800/30168 conn(0x7f4f94037820 msgr2=0x7f4f94039ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:11.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.141+0000 7f4fb8fb9640 1 --2- 172.21.15.40:0/3624849470 >> v2:172.21.15.189:6800/30168 conn(0x7f4f94037820 0x7f4f94039ce0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f4fb41344e0 tx=0x7f4f9c008030 comp rx=0 tx=0).stop 2024-04-22T08:20:11.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.141+0000 7f4fb8fb9640 10 monclient: shutdown 2024-04-22T08:20:11.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.141+0000 7f4fb8fb9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:20:11.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.141+0000 7f4fb8fb9640 1 -- 172.21.15.40:0/3624849470 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4fb409de00 msgr2=0x7f4fb411cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:11.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.141+0000 7f4fb2d9d640 1 -- 172.21.15.40:0/3624849470 reap_dead start 2024-04-22T08:20:11.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.141+0000 7f4fb8fb9640 1 --2- 172.21.15.40:0/3624849470 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4fb409de00 0x7f4fb411cdf0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4fa400c7b0 tx=0x7f4fa400cc70 comp rx=0 tx=0).stop 2024-04-22T08:20:11.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.141+0000 7f4fb8fb9640 1 -- 172.21.15.40:0/3624849470 shutdown_connections 2024-04-22T08:20:11.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.141+0000 7f4fb8fb9640 1 --2- 172.21.15.40:0/3624849470 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4fb409de00 0x7f4fb411cdf0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:11.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.141+0000 7f4fb8fb9640 1 -- 172.21.15.40:0/3624849470 >> 172.21.15.40:0/3624849470 conn(0x7f4fb40a1780 msgr2=0x7f4fb4133b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:20:11.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.141+0000 7f4fb8fb9640 1 -- 172.21.15.40:0/3624849470 shutdown_connections 2024-04-22T08:20:11.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.141+0000 7f4fb8fb9640 1 -- 172.21.15.40:0/3624849470 wait complete. 2024-04-22T08:20:11.166 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:20:11.166 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-22T08:20:11.166 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-22T08:20:11.166 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:20:11.167 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[38309]: starting. 2024-04-22T08:20:11.167 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[38309]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-38268 2024-04-22T08:20:11.167 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[38309]: shutting down. 2024-04-22T08:20:11.167 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:20:11.167 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:20:11.167 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:20:11.167 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:20:11.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.268+0000 7fd83718d640 10 monclient: get_monmap_and_config 2024-04-22T08:20:11.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.268+0000 7fd83718d640 10 monclient: build_initial_monmap 2024-04-22T08:20:11.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.268+0000 7fd83718d640 10 monclient: monmap: 2024-04-22T08:20:11.268 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:20:11.268 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:11.268 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:20:11.269243+0000 2024-04-22T08:20:11.268 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:20:11.269243+0000 2024-04-22T08:20:11.268 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:20:11.268 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:11.268 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:20:11.268 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:20:11.269 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:20:11.269 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:20:11.269 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:20:11.269 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:11.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.268+0000 7fd83718d640 1 Processor -- start 2024-04-22T08:20:11.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.268+0000 7fd83718d640 1 -- start start 2024-04-22T08:20:11.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.268+0000 7fd83718d640 10 monclient: init 2024-04-22T08:20:11.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.268+0000 7fd83718d640 10 monclient: _reopen_session rank -1 2024-04-22T08:20:11.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.268+0000 7fd83718d640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-04-22T08:20:11.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.268+0000 7fd83718d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd83009f270 0x7fd83009d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:11.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.268+0000 7fd83718d640 10 monclient(hunting): picked mon.noname-b con 0x7fd83009f270 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:20:11.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.268+0000 7fd83718d640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd83009f650 0x7fd83009fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:11.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.268+0000 7fd83718d640 10 monclient(hunting): picked mon.noname-c con 0x7fd83009f650 addr v2:172.21.15.189:3300/0 2024-04-22T08:20:11.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.268+0000 7fd83718d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd8301367d0 addr v1:172.21.15.189:6789/0 2024-04-22T08:20:11.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.268+0000 7fd83718d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd83009e510 con 0x7fd8301367d0 2024-04-22T08:20:11.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.268+0000 7fd83718d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:11.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.268+0000 7fd83718d640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd83009ba30 con 0x7fd83009f650 2024-04-22T08:20:11.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.268+0000 7fd83718d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:11.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.268+0000 7fd83718d640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd83009b760 con 0x7fd83009f270 2024-04-22T08:20:11.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.268+0000 7fd83718d640 10 monclient(hunting): _renew_subs 2024-04-22T08:20:11.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.268+0000 7fd83718d640 10 monclient(hunting): authenticate will time out at 1285.263595s 2024-04-22T08:20:11.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.268+0000 7fd834f29640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd83009f270 0x7fd83009d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:11.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.268+0000 7fd834f29640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd83009f270 0x7fd83009d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:58052/0 (socket says 172.21.15.40:58052) 2024-04-22T08:20:11.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.268+0000 7fd834f29640 1 -- 172.21.15.40:0/1964310437 learned_addr learned my addr 172.21.15.40:0/1964310437 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:20:11.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.268+0000 7fd834f29640 10 monclient(hunting): get_auth_request con 0x7fd83009f270 auth_method 0 2024-04-22T08:20:11.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.268+0000 7fd834f29640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:11.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.268+0000 7fd834f29640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:11.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.268+0000 7fd834f29640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:11.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.268+0000 7fd82ffff640 1 --2- 172.21.15.40:0/1964310437 >> v2:172.21.15.189:3300/0 conn(0x7fd83009f650 0x7fd83009fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:11.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.269+0000 7fd82ffff640 10 monclient(hunting): get_auth_request con 0x7fd83009f650 auth_method 0 2024-04-22T08:20:11.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.269+0000 7fd82ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:11.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.269+0000 7fd82ffff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:11.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.269+0000 7fd82ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:11.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.269+0000 7fd834f29640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:11.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.269+0000 7fd834f29640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:11.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.269+0000 7fd834f29640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:11.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.269+0000 7fd82ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:11.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.269+0000 7fd82ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:11.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.269+0000 7fd82ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:11.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.269+0000 7fd834f29640 10 monclient(hunting): handle_auth_done global_id 54872 payload 274 2024-04-22T08:20:11.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.269+0000 7fd834f29640 1 -- 172.21.15.40:0/1964310437 >> v2:172.21.15.189:3300/0 conn(0x7fd83009f650 msgr2=0x7fd83009fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:20:11.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.269+0000 7fd834f29640 1 --2- 172.21.15.40:0/1964310437 >> v2:172.21.15.189:3300/0 conn(0x7fd83009f650 0x7fd83009fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:11.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.269+0000 7fd834f29640 1 -- 172.21.15.40:0/1964310437 >> v1:172.21.15.189:6789/0 conn(0x7fd8301367d0 legacy=0x7fd830138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:20:11.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.269+0000 7fd834f29640 10 monclient: _finish_hunting 0 2024-04-22T08:20:11.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.269+0000 7fd834f29640 1 monclient: found mon.noname-b 2024-04-22T08:20:11.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.269+0000 7fd834f29640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:20:11.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.269+0000 7fd834f29640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:20:11.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.269+0000 7fd834f29640 1 -- 172.21.15.40:0/1964310437 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd83009c790 con 0x7fd83009f270 2024-04-22T08:20:11.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.272+0000 7fd834f29640 1 -- 172.21.15.40:0/1964310437 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd83009f270 msgr2=0x7fd83009d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:11.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.272+0000 7fd834f29640 1 --2- 172.21.15.40:0/1964310437 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd83009f270 0x7fd83009d450 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd83009b060 tx=0x7fd82402df80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db55099aea828946 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:11.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.272+0000 7fd82f7fe640 1 -- 172.21.15.40:0/1964310437 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd824036440 con 0x7fd83009f270 2024-04-22T08:20:11.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.272+0000 7fd82f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:11.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.272+0000 7fd82f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:20:11.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.272+0000 7fd82f7fe640 10 monclient: dump: 2024-04-22T08:20:11.286 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:11.286 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:11.286 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:11.286 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:11.287 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:11.287 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:11.287 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:11.287 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:11.287 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:11.287 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:11.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.272+0000 7fd82f7fe640 10 monclient: _finish_auth 0 2024-04-22T08:20:11.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.272+0000 7fd82f7fe640 10 monclient: _check_auth_tickets 2024-04-22T08:20:11.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.272+0000 7fd82f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:20:11.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.272+0000 7fd82f7fe640 1 -- 172.21.15.40:0/1964310437 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8240365a0 con 0x7fd83009f270 2024-04-22T08:20:11.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.272+0000 7fd83718d640 5 monclient: authenticate success, global_id 54872 2024-04-22T08:20:11.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.272+0000 7fd83718d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:20:11.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.272+0000 7fd82f7fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:20:11.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.272+0000 7fd83718d640 10 monclient: get_monmap_and_config success 2024-04-22T08:20:11.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.272+0000 7fd82f7fe640 1 -- 172.21.15.40:0/1964310437 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd824036930 con 0x7fd83009f270 2024-04-22T08:20:11.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.272+0000 7fd82f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:11.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.272+0000 7fd83718d640 4 set_mon_vals no callback set 2024-04-22T08:20:11.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.272+0000 7fd82f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:20:11.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.272+0000 7fd82f7fe640 10 monclient: dump: 2024-04-22T08:20:11.290 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:11.290 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:11.290 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:11.290 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:11.290 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:11.291 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:11.291 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:11.291 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:11.291 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:11.292 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:11.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.272+0000 7fd83718d640 10 monclient: shutdown 2024-04-22T08:20:11.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.272+0000 7fd83718d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:20:11.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.272+0000 7fd83718d640 1 -- 172.21.15.40:0/1964310437 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd83009f270 msgr2=0x7fd83009d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:11.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.272+0000 7fd83718d640 1 --2- 172.21.15.40:0/1964310437 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd83009f270 0x7fd83009d450 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd83009b060 tx=0x7fd82402df80 comp rx=0 tx=0).stop 2024-04-22T08:20:11.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.272+0000 7fd83718d640 1 -- 172.21.15.40:0/1964310437 shutdown_connections 2024-04-22T08:20:11.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.272+0000 7fd83718d640 1 --2- 172.21.15.40:0/1964310437 >> v2:172.21.15.189:3300/0 conn(0x7fd83009f650 0x7fd83009fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:11.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.272+0000 7fd83718d640 1 --2- 172.21.15.40:0/1964310437 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd83009f270 0x7fd83009d450 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:11.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.272+0000 7fd83718d640 1 -- 172.21.15.40:0/1964310437 >> 172.21.15.40:0/1964310437 conn(0x7fd8300a0ae0 msgr2=0x7fd8300a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:20:11.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.272+0000 7fd83718d640 1 -- 172.21.15.40:0/1964310437 shutdown_connections 2024-04-22T08:20:11.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.272+0000 7fd83718d640 1 -- 172.21.15.40:0/1964310437 wait complete. 2024-04-22T08:20:11.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.273+0000 7fd83718d640 10 monclient: build_initial_monmap 2024-04-22T08:20:11.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.273+0000 7fd83718d640 10 monclient: monmap: 2024-04-22T08:20:11.294 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:20:11.294 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:20:11.294 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:20:11.294 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:20:11.294 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:20:11.295 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:11.295 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:20:11.295 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:20:11.295 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:20:11.295 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:11.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.273+0000 7fd83718d640 1 Processor -- start 2024-04-22T08:20:11.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.273+0000 7fd83718d640 1 -- start start 2024-04-22T08:20:11.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.273+0000 7fd83718d640 10 monclient: init 2024-04-22T08:20:11.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd83718d640 10 monclient: _reopen_session rank -1 2024-04-22T08:20:11.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd83718d640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-22T08:20:11.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd83718d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd83009f270 0x7fd8301ca9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:11.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd83718d640 10 monclient(hunting): picked mon.noname-a con 0x7fd83009f270 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:20:11.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd83718d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd83009f650 0x7fd8301caef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:11.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd83718d640 10 monclient(hunting): picked mon.noname-c con 0x7fd83009f650 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:20:11.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd83718d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8301367d0 0x7fd8301cb430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:11.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd83718d640 10 monclient(hunting): picked mon.noname-b con 0x7fd8301367d0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:20:11.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd83718d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:11.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd83718d640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd830122610 con 0x7fd83009f270 2024-04-22T08:20:11.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd83718d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:11.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd83718d640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd830122950 con 0x7fd8301367d0 2024-04-22T08:20:11.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd83718d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:11.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd83718d640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd830123870 con 0x7fd83009f650 2024-04-22T08:20:11.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd83718d640 10 monclient(hunting): _renew_subs 2024-04-22T08:20:11.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd83718d640 10 monclient(hunting): authenticate will time out at 1285.269137s 2024-04-22T08:20:11.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd82ffff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd83009f650 0x7fd8301caef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:11.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd834f29640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd83009f270 0x7fd8301ca9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:11.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd82ffff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd83009f650 0x7fd8301caef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:58066/0 (socket says 172.21.15.40:58066) 2024-04-22T08:20:11.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd82ffff640 1 -- 172.21.15.40:0/229393316 learned_addr learned my addr 172.21.15.40:0/229393316 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:20:11.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd82ffff640 10 monclient(hunting): get_auth_request con 0x7fd83009f650 auth_method 0 2024-04-22T08:20:11.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd82ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:11.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd82ffff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:11.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd82ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:11.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd834f29640 10 monclient(hunting): get_auth_request con 0x7fd83009f270 auth_method 0 2024-04-22T08:20:11.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd834f29640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:11.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd834f29640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:11.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd834f29640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:11.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd83572a640 1 --2- 172.21.15.40:0/229393316 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8301367d0 0x7fd8301cb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:11.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd83572a640 10 monclient(hunting): get_auth_request con 0x7fd8301367d0 auth_method 0 2024-04-22T08:20:11.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd83572a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:11.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd83572a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:11.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd83572a640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:11.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd834f29640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:11.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd834f29640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:11.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd834f29640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:11.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd82ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:11.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd82ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:11.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd82ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:11.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd83572a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:11.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd83572a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:11.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd83572a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:11.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd834f29640 10 monclient(hunting): handle_auth_done global_id 94155 payload 824 2024-04-22T08:20:11.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd834f29640 1 -- 172.21.15.40:0/229393316 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd83009f650 msgr2=0x7fd8301caef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:11.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd834f29640 1 --2- 172.21.15.40:0/229393316 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd83009f650 0x7fd8301caef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:11.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd834f29640 1 -- 172.21.15.40:0/229393316 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8301367d0 msgr2=0x7fd8301cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:11.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd834f29640 1 --2- 172.21.15.40:0/229393316 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8301367d0 0x7fd8301cb430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:11.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd834f29640 10 monclient: _finish_hunting 0 2024-04-22T08:20:11.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd834f29640 1 monclient: found mon.noname-a 2024-04-22T08:20:11.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd834f29640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:20:11.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd834f29640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:20:11.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd834f29640 1 -- 172.21.15.40:0/229393316 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd82400b770 con 0x7fd83009f270 2024-04-22T08:20:11.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.274+0000 7fd82ffff640 1 --2- 172.21.15.40:0/229393316 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd83009f650 0x7fd8301caef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:20:11.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.277+0000 7fd834f29640 1 -- 172.21.15.40:0/229393316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd83009f270 msgr2=0x7fd8301ca9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:11.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.277+0000 7fd834f29640 1 --2- 172.21.15.40:0/229393316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd83009f270 0x7fd8301ca9b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd824035000 tx=0x7fd8240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:11.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.277+0000 7fd82d7fa640 1 -- 172.21.15.40:0/229393316 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82403f960 con 0x7fd83009f270 2024-04-22T08:20:11.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.277+0000 7fd82d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:11.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.277+0000 7fd82d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:20:11.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.277+0000 7fd82d7fa640 10 monclient: dump: 2024-04-22T08:20:11.301 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:11.301 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:11.301 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:11.301 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:11.301 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:11.301 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:11.301 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:11.301 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:11.301 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:11.301 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:11.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.277+0000 7fd82d7fa640 10 monclient: _finish_auth 0 2024-04-22T08:20:11.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.277+0000 7fd82d7fa640 10 monclient: _check_auth_tickets 2024-04-22T08:20:11.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.277+0000 7fd82d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:20:11.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.277+0000 7fd82d7fa640 1 -- 172.21.15.40:0/229393316 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd82403fac0 con 0x7fd83009f270 2024-04-22T08:20:11.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.277+0000 7fd82d7fa640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:20:11.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.277+0000 7fd82d7fa640 1 -- 172.21.15.40:0/229393316 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82403fe70 con 0x7fd83009f270 2024-04-22T08:20:11.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.277+0000 7fd82d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:11.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.277+0000 7fd82d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:20:11.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.277+0000 7fd82d7fa640 10 monclient: dump: 2024-04-22T08:20:11.302 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:11.302 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:11.302 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:11.302 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:11.302 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:11.302 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:11.303 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:11.303 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:11.303 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:11.303 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:11.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.277+0000 7fd83718d640 5 monclient: authenticate success, global_id 94155 2024-04-22T08:20:11.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.277+0000 7fd82effd640 4 set_mon_vals no callback set 2024-04-22T08:20:11.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.277+0000 7fd83718d640 10 monclient: _renew_subs 2024-04-22T08:20:11.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.277+0000 7fd83718d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:20:11.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.277+0000 7fd83718d640 1 -- 172.21.15.40:0/229393316 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd83009bc10 con 0x7fd83009f270 2024-04-22T08:20:11.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.277+0000 7fd83718d640 10 monclient: _renew_subs 2024-04-22T08:20:11.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.277+0000 7fd83718d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:20:11.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.277+0000 7fd83718d640 1 -- 172.21.15.40:0/229393316 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd830123e00 con 0x7fd83009f270 2024-04-22T08:20:11.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.278+0000 7fd82d7fa640 1 -- 172.21.15.40:0/229393316 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fd8240403e0 con 0x7fd83009f270 2024-04-22T08:20:11.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.278+0000 7fd83718d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:20:11.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.278+0000 7fd83718d640 1 --2- 172.21.15.40:0/229393316 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd830133850 0x7fd830133c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:11.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.278+0000 7fd83718d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:11.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.278+0000 7fd83718d640 1 -- 172.21.15.40:0/229393316 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8300667a0 con 0x7fd830133850 2024-04-22T08:20:11.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.278+0000 7fd82ffff640 1 --2- 172.21.15.40:0/229393316 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd830133850 0x7fd830133c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:11.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.278+0000 7fd82ffff640 10 monclient: get_auth_request con 0x7fd830133850 auth_method 0 2024-04-22T08:20:11.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.278+0000 7fd82ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:11.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.278+0000 7fd82ffff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:11.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.278+0000 7fd82ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:11.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.278+0000 7fd82d7fa640 1 --2- 172.21.15.40:0/229393316 >> v2:172.21.15.189:6800/30168 conn(0x7fd814037850 0x7fd814039d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:11.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.278+0000 7fd82ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:11.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.278+0000 7fd82ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:11.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.278+0000 7fd82ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:11.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.278+0000 7fd82d7fa640 1 -- 172.21.15.40:0/229393316 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 3506+0+0 (secure 0 0 0) 0x7fd82406c600 con 0x7fd83009f270 2024-04-22T08:20:11.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.278+0000 7fd83572a640 1 --2- 172.21.15.40:0/229393316 >> v2:172.21.15.189:6800/30168 conn(0x7fd814037850 0x7fd814039d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:11.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.278+0000 7fd82ffff640 10 monclient(hunting): handle_auth_done global_id 54878 payload 824 2024-04-22T08:20:11.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.278+0000 7fd82ffff640 1 -- 172.21.15.40:0/229393316 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd83009ff90 con 0x7fd830133850 2024-04-22T08:20:11.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.278+0000 7fd83572a640 10 monclient: get_auth_request con 0x7fd814037850 auth_method 0 2024-04-22T08:20:11.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.281+0000 7fd82ffff640 1 -- 172.21.15.40:0/229393316 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd830133850 msgr2=0x7fd830133c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:11.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.281+0000 7fd82ffff640 1 --2- 172.21.15.40:0/229393316 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd830133850 0x7fd830133c30 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd83004e210 tx=0x7fd81c00f610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:11.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.281+0000 7fd82d7fa640 1 -- 172.21.15.40:0/229393316 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd81c00fe40 con 0x7fd830133850 2024-04-22T08:20:11.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.281+0000 7fd82d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:11.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.281+0000 7fd82d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:20:11.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.281+0000 7fd82d7fa640 10 monclient: dump: 2024-04-22T08:20:11.307 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:11.307 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:11.307 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:11.307 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:11.307 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:11.307 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:11.307 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:11.307 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:11.307 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:11.307 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:11.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.281+0000 7fd83572a640 1 --2- 172.21.15.40:0/229393316 >> v2:172.21.15.189:6800/30168 conn(0x7fd814037850 0x7fd814039d10 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fd8280055e0 tx=0x7fd828003d10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:11.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.281+0000 7fd82d7fa640 1 -- 172.21.15.40:0/229393316 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7fd81c004450 con 0x7fd830133850 2024-04-22T08:20:11.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.281+0000 7fd82d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:20:11.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.281+0000 7fd82d7fa640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:20:11.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.281+0000 7fd82d7fa640 1 -- 172.21.15.40:0/229393316 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd830133850 msgr2=0x7fd830133c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:11.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.281+0000 7fd82d7fa640 1 --2- 172.21.15.40:0/229393316 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd830133850 0x7fd830133c30 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd83004e210 tx=0x7fd81c00f610 comp rx=0 tx=0).stop 2024-04-22T08:20:11.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.281+0000 7fd82d7fa640 1 -- 172.21.15.40:0/229393316 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd830133850 msgr2=0x7fd830133c30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:20:11.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.281+0000 7fd82d7fa640 1 --2- 172.21.15.40:0/229393316 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd830133850 0x7fd830133c30 secure :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd83004e210 tx=0x7fd81c00f610 comp rx=0 tx=0).stop 2024-04-22T08:20:11.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.288+0000 7fd83718d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:20:11.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.288+0000 7fd83718d640 1 --2- 172.21.15.40:0/229393316 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd830134170 0x7fd83011fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:11.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.288+0000 7fd83718d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:11.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.288+0000 7fd83718d640 1 -- 172.21.15.40:0/229393316 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8301239e0 con 0x7fd830134170 2024-04-22T08:20:11.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.288+0000 7fd82ffff640 1 --2- 172.21.15.40:0/229393316 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd830134170 0x7fd83011fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:11.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.288+0000 7fd82ffff640 10 monclient: get_auth_request con 0x7fd830134170 auth_method 0 2024-04-22T08:20:11.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.288+0000 7fd82ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:11.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.288+0000 7fd82ffff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:11.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.288+0000 7fd82ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:11.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.288+0000 7fd82ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:11.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.288+0000 7fd82ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:11.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.288+0000 7fd82ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:11.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.288+0000 7fd82ffff640 10 monclient(hunting): handle_auth_done global_id 54881 payload 824 2024-04-22T08:20:11.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.288+0000 7fd82ffff640 1 -- 172.21.15.40:0/229393316 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd83009d990 con 0x7fd830134170 2024-04-22T08:20:11.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.290+0000 7fd82ffff640 1 -- 172.21.15.40:0/229393316 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd830134170 msgr2=0x7fd83011fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:11.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.290+0000 7fd82ffff640 1 --2- 172.21.15.40:0/229393316 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd830134170 0x7fd83011fee0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fd83004e210 tx=0x7fd81c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:11.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.290+0000 7fd82d7fa640 1 -- 172.21.15.40:0/229393316 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd81c01eb50 con 0x7fd830134170 2024-04-22T08:20:11.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.290+0000 7fd82d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:11.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.290+0000 7fd82d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:20:11.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.290+0000 7fd82d7fa640 10 monclient: dump: 2024-04-22T08:20:11.311 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:11.313 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:11.313 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:11.313 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:11.313 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:11.313 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:11.313 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:11.313 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:11.313 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:11.313 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:11.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.290+0000 7fd82d7fa640 1 -- 172.21.15.40:0/229393316 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4631 (secure 0 0 0) 0x7fd81c0133a0 con 0x7fd830134170 2024-04-22T08:20:11.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.290+0000 7fd82d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:20:11.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.290+0000 7fd82d7fa640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:20:11.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.290+0000 7fd82d7fa640 1 -- 172.21.15.40:0/229393316 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd830134170 msgr2=0x7fd83011fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:11.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.290+0000 7fd82d7fa640 1 --2- 172.21.15.40:0/229393316 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd830134170 0x7fd83011fee0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fd83004e210 tx=0x7fd81c0046c0 comp rx=0 tx=0).stop 2024-04-22T08:20:11.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.290+0000 7fd82d7fa640 1 -- 172.21.15.40:0/229393316 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd830134170 msgr2=0x7fd83011fee0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:20:11.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.290+0000 7fd82d7fa640 1 --2- 172.21.15.40:0/229393316 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd830134170 0x7fd83011fee0 secure :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fd83004e210 tx=0x7fd81c0046c0 comp rx=0 tx=0).stop 2024-04-22T08:20:11.315 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:20:11.315 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:20:11.315 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:20:11.315 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:20:11.315 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 72, 2024-04-22T08:20:11.316 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:20:11.316 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:20:11.316 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:20:11.316 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:20:11.316 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:20:11.316 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 2, 2024-04-22T08:20:11.316 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:20:11.316 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:20:11.316 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:20:11.317 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:20:11.317 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:20:11.317 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:20:11.317 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:20:11.317 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:20:11.317 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:20:11.317 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:20:11.317 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:20:11.318 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:20:11.318 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:20:11.318 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:20:11.318 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:20:11.319 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:20:11.319 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:20:11.319 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:20:11.319 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:20:11.319 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:20:11.319 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:20:11.319 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:20:11.320 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:20:11.320 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:20:11.320 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:20:11.320 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:20:11.320 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:11.320 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:11.320 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:20:11.320 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:20:11.321 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:20:11.321 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:20:11.321 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:20:11.321 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:20:11.321 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:20:11.321 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:20:11.321 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:20:11.321 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:20:11.322 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:20:11.322 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:20:11.322 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:20:11.322 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:20:11.322 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:20:11.322 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:20:11.323 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:20:11.323 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:20:11.323 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:20:11.323 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:20:11.323 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:20:11.323 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:20:11.323 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:20:11.323 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:20:11.324 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:20:11.324 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:20:11.324 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:20:11.324 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:20:11.324 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:20:11.324 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:11.324 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:20:11.324 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:11.325 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:20:11.325 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:20:11.325 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:20:11.325 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:20:11.325 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:11.325 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:20:11.325 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:20:11.326 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:11.326 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:11.326 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:11.326 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:20:11.327 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:20:11.327 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:11.327 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:11.327 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:11.327 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:11.327 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:20:11.328 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:20:11.328 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:20:11.328 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:20:11.328 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:20:11.328 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:11.328 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:11.328 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:20:11.328 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:20:11.329 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:20:11.329 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:20:11.329 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:11.329 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:20:11.329 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:20:11.329 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:11.329 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:11.329 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:11.330 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:20:11.330 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:20:11.330 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:11.330 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:11.330 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:11.330 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:11.330 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:20:11.330 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:20:11.331 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:20:11.331 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:20:11.331 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:20:11.331 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:11.331 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:11.331 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:20:11.331 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:20:11.331 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:20:11.332 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:20:11.332 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:11.332 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:20:11.332 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:20:11.332 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:11.332 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:11.332 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:11.333 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:20:11.333 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:20:11.333 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:11.333 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:11.333 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:11.333 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:11.333 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:20:11.333 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:20:11.334 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:20:11.334 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:20:11.334 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:20:11.334 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:11.334 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:11.334 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:11.334 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:20:11.335 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:20:11.335 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:11.335 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:20:11.335 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:20:11.335 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:20:11.335 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:11.336 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:20:11.336 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:20:11.336 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:11.336 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:20:11.336 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:20:11.336 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 3 2024-04-22T08:20:11.336 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:11.336 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:20:11.337 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:20:11.337 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:11.337 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:20:11.337 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:20:11.337 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 5 2024-04-22T08:20:11.337 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:11.337 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:11.338 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:11.338 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:20:11.338 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:20:11.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.290+0000 7fd83718d640 1 -- 172.21.15.40:0/229393316 >> v2:172.21.15.189:6800/30168 conn(0x7fd814037850 msgr2=0x7fd814039d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:11.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.290+0000 7fd83718d640 1 --2- 172.21.15.40:0/229393316 >> v2:172.21.15.189:6800/30168 conn(0x7fd814037850 0x7fd814039d10 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fd8280055e0 tx=0x7fd828003d10 comp rx=0 tx=0).stop 2024-04-22T08:20:11.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.290+0000 7fd83718d640 10 monclient: shutdown 2024-04-22T08:20:11.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.290+0000 7fd83718d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:20:11.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.290+0000 7fd83718d640 1 -- 172.21.15.40:0/229393316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd83009f270 msgr2=0x7fd8301ca9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:11.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.290+0000 7fd83718d640 1 --2- 172.21.15.40:0/229393316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd83009f270 0x7fd8301ca9b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd824035000 tx=0x7fd8240046c0 comp rx=0 tx=0).stop 2024-04-22T08:20:11.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.290+0000 7fd83572a640 1 -- 172.21.15.40:0/229393316 reap_dead start 2024-04-22T08:20:11.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.290+0000 7fd83718d640 1 -- 172.21.15.40:0/229393316 shutdown_connections 2024-04-22T08:20:11.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.290+0000 7fd83718d640 1 --2- 172.21.15.40:0/229393316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd83009f270 0x7fd8301ca9b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:11.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.291+0000 7fd83718d640 1 -- 172.21.15.40:0/229393316 >> 172.21.15.40:0/229393316 conn(0x7fd8300a0ae0 msgr2=0x7fd8300a3670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:20:11.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.291+0000 7fd83718d640 1 -- 172.21.15.40:0/229393316 shutdown_connections 2024-04-22T08:20:11.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.291+0000 7fd83718d640 1 -- 172.21.15.40:0/229393316 wait complete. 2024-04-22T08:20:11.341 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:20:11.342 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-22T08:20:11.342 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-22T08:20:11.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.445+0000 7f9da6a08640 10 monclient: get_monmap_and_config 2024-04-22T08:20:11.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.445+0000 7f9da6a08640 10 monclient: build_initial_monmap 2024-04-22T08:20:11.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.445+0000 7f9da6a08640 10 monclient: monmap: 2024-04-22T08:20:11.446 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:20:11.446 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:11.446 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:20:11.446469+0000 2024-04-22T08:20:11.446 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:20:11.446469+0000 2024-04-22T08:20:11.446 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:20:11.446 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:11.446 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:20:11.446 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:20:11.446 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:20:11.447 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:20:11.447 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:20:11.447 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:11.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.445+0000 7f9da6a08640 1 Processor -- start 2024-04-22T08:20:11.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.445+0000 7f9da6a08640 1 -- start start 2024-04-22T08:20:11.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.445+0000 7f9da6a08640 10 monclient: init 2024-04-22T08:20:11.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.445+0000 7f9da6a08640 10 monclient: _reopen_session rank -1 2024-04-22T08:20:11.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.445+0000 7f9da6a08640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-04-22T08:20:11.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.445+0000 7f9da6a08640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9da0136070 addr v1:172.21.15.189:6789/0 2024-04-22T08:20:11.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.445+0000 7f9da6a08640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9da0136450 addr v1:172.21.15.40:6789/0 2024-04-22T08:20:11.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.445+0000 7f9da6a08640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9da01368b0 0x7f9da013e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:11.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.445+0000 7f9da6a08640 10 monclient(hunting): picked mon.noname-c con 0x7f9da01368b0 addr v2:172.21.15.189:3300/0 2024-04-22T08:20:11.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.445+0000 7f9da6a08640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9da006bb50 con 0x7f9da0136450 2024-04-22T08:20:11.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.445+0000 7f9da6a08640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9da0099dd0 con 0x7f9da0136070 2024-04-22T08:20:11.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.445+0000 7f9da6a08640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:11.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.445+0000 7f9da6a08640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9da009a390 con 0x7f9da01368b0 2024-04-22T08:20:11.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.445+0000 7f9da6a08640 10 monclient(hunting): _renew_subs 2024-04-22T08:20:11.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.445+0000 7f9da6a08640 10 monclient(hunting): authenticate will time out at 1285.440690s 2024-04-22T08:20:11.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.446+0000 7f9d9f7fe640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f9da0136450 0x7f9da013aed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53810/0 (socket says 172.21.15.40:53810) 2024-04-22T08:20:11.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.446+0000 7f9d9f7fe640 1 -- 172.21.15.40:0/3222908153 learned_addr learned my addr 172.21.15.40:0/3222908153 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:20:11.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.446+0000 7f9da4fa5640 1 --2- 172.21.15.40:0/3222908153 >> v2:172.21.15.189:3300/0 conn(0x7f9da01368b0 0x7f9da013e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:11.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.446+0000 7f9da4fa5640 10 monclient(hunting): get_auth_request con 0x7f9da01368b0 auth_method 0 2024-04-22T08:20:11.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.446+0000 7f9da4fa5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:11.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.446+0000 7f9da4fa5640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:11.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.446+0000 7f9da4fa5640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:11.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.446+0000 7f9da4fa5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:11.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.446+0000 7f9da4fa5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:11.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.446+0000 7f9da4fa5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:11.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.446+0000 7f9da4fa5640 10 monclient(hunting): handle_auth_done global_id 64696 payload 274 2024-04-22T08:20:11.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.446+0000 7f9da4fa5640 1 -- 172.21.15.40:0/3222908153 >> v1:172.21.15.189:6789/0 conn(0x7f9da0136070 legacy=0x7f9da009aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:20:11.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.446+0000 7f9da4fa5640 1 -- 172.21.15.40:0/3222908153 >> v1:172.21.15.40:6789/0 conn(0x7f9da0136450 legacy=0x7f9da013aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:20:11.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.446+0000 7f9da4fa5640 10 monclient: _finish_hunting 0 2024-04-22T08:20:11.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.446+0000 7f9da4fa5640 1 monclient: found mon.noname-c 2024-04-22T08:20:11.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.446+0000 7f9da4fa5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:20:11.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.446+0000 7f9da4fa5640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.189:3300/0 2024-04-22T08:20:11.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.446+0000 7f9da4fa5640 1 -- 172.21.15.40:0/3222908153 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9da009a0b0 con 0x7f9da01368b0 2024-04-22T08:20:11.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.450+0000 7f9da4fa5640 1 -- 172.21.15.40:0/3222908153 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9da01368b0 msgr2=0x7f9da013e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:11.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.450+0000 7f9da4fa5640 1 --2- 172.21.15.40:0/3222908153 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9da01368b0 0x7f9da013e600 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f9d98009d90 tx=0x7f9d9802bd90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5745a70bde9ecd96 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:11.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.450+0000 7f9d9effd640 1 -- 172.21.15.40:0/3222908153 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d9802ce30 con 0x7f9da01368b0 2024-04-22T08:20:11.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.450+0000 7f9d9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:11.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.450+0000 7f9d9effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:20:11.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.450+0000 7f9d9effd640 10 monclient: dump: 2024-04-22T08:20:11.456 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:11.456 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:11.456 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:11.456 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:11.456 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:11.456 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:11.456 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:11.456 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:11.456 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:11.456 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:11.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.450+0000 7f9d9effd640 10 monclient: _finish_auth 0 2024-04-22T08:20:11.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.450+0000 7f9d9effd640 10 monclient: _check_auth_tickets 2024-04-22T08:20:11.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.450+0000 7f9d9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:20:11.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.450+0000 7f9da6a08640 5 monclient: authenticate success, global_id 64696 2024-04-22T08:20:11.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.450+0000 7f9d9effd640 1 -- 172.21.15.40:0/3222908153 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d98034440 con 0x7f9da01368b0 2024-04-22T08:20:11.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.450+0000 7f9da6a08640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:20:11.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.450+0000 7f9d9effd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:20:11.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.450+0000 7f9da6a08640 10 monclient: get_monmap_and_config success 2024-04-22T08:20:11.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.450+0000 7f9da6a08640 4 set_mon_vals no callback set 2024-04-22T08:20:11.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.450+0000 7f9d9effd640 1 -- 172.21.15.40:0/3222908153 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d980347d0 con 0x7f9da01368b0 2024-04-22T08:20:11.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.450+0000 7f9d9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:11.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.450+0000 7f9d9effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:20:11.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.450+0000 7f9d9effd640 10 monclient: dump: 2024-04-22T08:20:11.457 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:11.457 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:11.457 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:11.458 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:11.458 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:11.458 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:11.458 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:11.458 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:11.458 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:11.458 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:11.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.450+0000 7f9da6a08640 10 monclient: shutdown 2024-04-22T08:20:11.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.450+0000 7f9da6a08640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:20:11.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.450+0000 7f9da6a08640 1 -- 172.21.15.40:0/3222908153 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9da01368b0 msgr2=0x7f9da013e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:11.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.450+0000 7f9da6a08640 1 --2- 172.21.15.40:0/3222908153 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9da01368b0 0x7f9da013e600 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f9d98009d90 tx=0x7f9d9802bd90 comp rx=0 tx=0).stop 2024-04-22T08:20:11.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.450+0000 7f9da6a08640 1 -- 172.21.15.40:0/3222908153 shutdown_connections 2024-04-22T08:20:11.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.450+0000 7f9da6a08640 1 --2- 172.21.15.40:0/3222908153 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9da01368b0 0x7f9da013e600 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:11.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.450+0000 7f9da6a08640 1 -- 172.21.15.40:0/3222908153 >> 172.21.15.40:0/3222908153 conn(0x7f9da00a1b60 msgr2=0x7f9da00a1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:20:11.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.450+0000 7f9da6a08640 1 -- 172.21.15.40:0/3222908153 shutdown_connections 2024-04-22T08:20:11.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.450+0000 7f9da6a08640 1 -- 172.21.15.40:0/3222908153 wait complete. 2024-04-22T08:20:11.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.451+0000 7f9da6a08640 10 monclient: build_initial_monmap 2024-04-22T08:20:11.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.451+0000 7f9da6a08640 10 monclient: monmap: 2024-04-22T08:20:11.459 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:20:11.459 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:20:11.459 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:20:11.459 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:20:11.459 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:20:11.459 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:11.459 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:20:11.459 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:20:11.459 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:20:11.459 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:11.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.451+0000 7f9da6a08640 1 Processor -- start 2024-04-22T08:20:11.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.451+0000 7f9da6a08640 1 -- start start 2024-04-22T08:20:11.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.451+0000 7f9da6a08640 10 monclient: init 2024-04-22T08:20:11.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.451+0000 7f9da6a08640 10 monclient: _reopen_session rank -1 2024-04-22T08:20:11.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.451+0000 7f9da6a08640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-22T08:20:11.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.451+0000 7f9da6a08640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9da0136070 0x7f9da013cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:11.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.451+0000 7f9da6a08640 10 monclient(hunting): picked mon.noname-c con 0x7f9da0136070 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:20:11.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.451+0000 7f9da6a08640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9da0136450 0x7f9da01cac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:11.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.451+0000 7f9da6a08640 10 monclient(hunting): picked mon.noname-a con 0x7f9da0136450 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:20:11.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.451+0000 7f9da6a08640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9da01cb1a0 0x7f9da01cb630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:11.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.451+0000 7f9da6a08640 10 monclient(hunting): picked mon.noname-b con 0x7f9da01cb1a0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:20:11.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.451+0000 7f9da6a08640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:11.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.451+0000 7f9da6a08640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da013e0d0 con 0x7f9da0136450 2024-04-22T08:20:11.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.451+0000 7f9da6a08640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:11.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.451+0000 7f9da6a08640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da009b470 con 0x7f9da01cb1a0 2024-04-22T08:20:11.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.451+0000 7f9da6a08640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:11.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.451+0000 7f9da6a08640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9da013c3b0 con 0x7f9da0136070 2024-04-22T08:20:11.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.451+0000 7f9da6a08640 10 monclient(hunting): _renew_subs 2024-04-22T08:20:11.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.451+0000 7f9da6a08640 10 monclient(hunting): authenticate will time out at 1285.446761s 2024-04-22T08:20:11.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.452+0000 7f9d9ffff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9da0136070 0x7f9da013cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:11.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.452+0000 7f9d9f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9da0136450 0x7f9da01cac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:11.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.452+0000 7f9d9ffff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9da0136070 0x7f9da013cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:58078/0 (socket says 172.21.15.40:58078) 2024-04-22T08:20:11.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.452+0000 7f9d9ffff640 1 -- 172.21.15.40:0/3961739916 learned_addr learned my addr 172.21.15.40:0/3961739916 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:20:11.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.452+0000 7f9d9ffff640 10 monclient(hunting): get_auth_request con 0x7f9da0136070 auth_method 0 2024-04-22T08:20:11.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.452+0000 7f9d9ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:11.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.452+0000 7f9d9ffff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:11.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.452+0000 7f9d9f7fe640 1 --2- 172.21.15.40:0/3961739916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9da0136450 0x7f9da01cac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46418/0 (socket says 172.21.15.40:46418) 2024-04-22T08:20:11.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.452+0000 7f9d9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:11.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.452+0000 7f9d9f7fe640 10 monclient(hunting): get_auth_request con 0x7f9da0136450 auth_method 0 2024-04-22T08:20:11.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.452+0000 7f9d9f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:11.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.452+0000 7f9d9f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:11.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.452+0000 7f9d9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:11.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.452+0000 7f9d9ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:11.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.452+0000 7f9d9ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:11.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.452+0000 7f9d9ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:11.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.452+0000 7f9d9f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:11.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.452+0000 7f9d9f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:11.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.452+0000 7f9d9f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:11.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.452+0000 7f9da4fa5640 1 --2- 172.21.15.40:0/3961739916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9da01cb1a0 0x7f9da01cb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:11.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.452+0000 7f9da4fa5640 10 monclient(hunting): get_auth_request con 0x7f9da01cb1a0 auth_method 0 2024-04-22T08:20:11.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.452+0000 7f9da4fa5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:11.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.452+0000 7f9da4fa5640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:11.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.452+0000 7f9da4fa5640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:11.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.452+0000 7f9d9ffff640 10 monclient(hunting): handle_auth_done global_id 54884 payload 824 2024-04-22T08:20:11.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.452+0000 7f9d9ffff640 1 -- 172.21.15.40:0/3961739916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9da01cb1a0 msgr2=0x7f9da01cb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:11.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.452+0000 7f9d9ffff640 1 --2- 172.21.15.40:0/3961739916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9da01cb1a0 0x7f9da01cb630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:11.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.452+0000 7f9d9ffff640 1 -- 172.21.15.40:0/3961739916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9da0136450 msgr2=0x7f9da01cac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:11.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.452+0000 7f9d9ffff640 1 --2- 172.21.15.40:0/3961739916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9da0136450 0x7f9da01cac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:11.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.452+0000 7f9d9ffff640 10 monclient: _finish_hunting 0 2024-04-22T08:20:11.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.452+0000 7f9d9ffff640 1 monclient: found mon.noname-c 2024-04-22T08:20:11.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.452+0000 7f9d9ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:20:11.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.452+0000 7f9d9ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:20:11.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.452+0000 7f9d9ffff640 1 -- 172.21.15.40:0/3961739916 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d98009a70 con 0x7f9da0136070 2024-04-22T08:20:11.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.452+0000 7f9d9f7fe640 1 --2- 172.21.15.40:0/3961739916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9da0136450 0x7f9da01cac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:20:11.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.452+0000 7f9da4fa5640 1 --2- 172.21.15.40:0/3961739916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9da01cb1a0 0x7f9da01cb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T08:20:11.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.455+0000 7f9d9ffff640 1 -- 172.21.15.40:0/3961739916 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9da0136070 msgr2=0x7f9da013cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:11.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.455+0000 7f9d9ffff640 1 --2- 172.21.15.40:0/3961739916 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9da0136070 0x7f9da013cb40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9d9000c430 tx=0x7f9d9000c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:11.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.455+0000 7f9d9cff9640 1 -- 172.21.15.40:0/3961739916 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d900119a0 con 0x7f9da0136070 2024-04-22T08:20:11.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.455+0000 7f9d9cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:11.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.455+0000 7f9d9cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:20:11.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.455+0000 7f9d9cff9640 10 monclient: dump: 2024-04-22T08:20:11.464 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:11.464 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:11.464 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:11.464 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:11.464 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:11.464 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:11.464 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:11.464 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:11.464 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:11.464 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:11.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.455+0000 7f9d9cff9640 10 monclient: _finish_auth 0 2024-04-22T08:20:11.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.455+0000 7f9d9cff9640 10 monclient: _check_auth_tickets 2024-04-22T08:20:11.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.455+0000 7f9d9cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:20:11.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.455+0000 7f9da6a08640 5 monclient: authenticate success, global_id 54884 2024-04-22T08:20:11.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.455+0000 7f9d9cff9640 1 -- 172.21.15.40:0/3961739916 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d90011b00 con 0x7f9da0136070 2024-04-22T08:20:11.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.455+0000 7f9da6a08640 10 monclient: _renew_subs 2024-04-22T08:20:11.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.455+0000 7f9da6a08640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:20:11.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.455+0000 7f9da6a08640 1 -- 172.21.15.40:0/3961739916 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9da013bb10 con 0x7f9da0136070 2024-04-22T08:20:11.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.455+0000 7f9d9cff9640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:20:11.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.455+0000 7f9d9cff9640 1 -- 172.21.15.40:0/3961739916 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d90012420 con 0x7f9da0136070 2024-04-22T08:20:11.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.455+0000 7f9d9cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:11.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.455+0000 7f9d9cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:20:11.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.455+0000 7f9d9cff9640 10 monclient: dump: 2024-04-22T08:20:11.469 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:11.469 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:11.469 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:11.469 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:11.469 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:11.469 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:11.469 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:11.469 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:11.469 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:11.469 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:11.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.455+0000 7f9da6a08640 10 monclient: _renew_subs 2024-04-22T08:20:11.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.455+0000 7f9da6a08640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:20:11.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.455+0000 7f9da6a08640 1 -- 172.21.15.40:0/3961739916 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9da013a050 con 0x7f9da0136070 2024-04-22T08:20:11.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.455+0000 7f9d9e7fc640 4 set_mon_vals no callback set 2024-04-22T08:20:11.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.455+0000 7f9da6a08640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:20:11.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.455+0000 7f9da6a08640 1 --2- 172.21.15.40:0/3961739916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9da0136870 0x7f9da0138c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:11.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.455+0000 7f9da6a08640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:11.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.455+0000 7f9d9cff9640 1 -- 172.21.15.40:0/3961739916 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f9d90012640 con 0x7f9da0136070 2024-04-22T08:20:11.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.455+0000 7f9da6a08640 1 -- 172.21.15.40:0/3961739916 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da00667a0 con 0x7f9da0136870 2024-04-22T08:20:11.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.456+0000 7f9d9f7fe640 1 --2- 172.21.15.40:0/3961739916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9da0136870 0x7f9da0138c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:11.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.456+0000 7f9d9f7fe640 10 monclient: get_auth_request con 0x7f9da0136870 auth_method 0 2024-04-22T08:20:11.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.456+0000 7f9d9f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:11.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.456+0000 7f9d9f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:11.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.456+0000 7f9d9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:11.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.456+0000 7f9d9cff9640 1 --2- 172.21.15.40:0/3961739916 >> v2:172.21.15.189:6800/30168 conn(0x7f9d8c0377d0 0x7f9d8c039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:11.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.456+0000 7f9d9cff9640 1 -- 172.21.15.40:0/3961739916 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f9d9004a1b0 con 0x7f9da0136070 2024-04-22T08:20:11.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.456+0000 7f9d9f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:11.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.456+0000 7f9d9f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:11.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.456+0000 7f9d9f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:11.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.456+0000 7f9da4fa5640 1 --2- 172.21.15.40:0/3961739916 >> v2:172.21.15.189:6800/30168 conn(0x7f9d8c0377d0 0x7f9d8c039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:11.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.456+0000 7f9da4fa5640 10 monclient: get_auth_request con 0x7f9d8c0377d0 auth_method 0 2024-04-22T08:20:11.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.456+0000 7f9d9f7fe640 10 monclient(hunting): handle_auth_done global_id 64702 payload 824 2024-04-22T08:20:11.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.456+0000 7f9d9f7fe640 1 -- 172.21.15.40:0/3961739916 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9da013c520 con 0x7f9da0136870 2024-04-22T08:20:11.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.458+0000 7f9d9f7fe640 1 -- 172.21.15.40:0/3961739916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9da0136870 msgr2=0x7f9da0138c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:11.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.458+0000 7f9d9f7fe640 1 --2- 172.21.15.40:0/3961739916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9da0136870 0x7f9da0138c10 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9da0139c50 tx=0x7f9d94007700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:11.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.458+0000 7f9da4fa5640 1 --2- 172.21.15.40:0/3961739916 >> v2:172.21.15.189:6800/30168 conn(0x7f9d8c0377d0 0x7f9d8c039c90 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f9d98033000 tx=0x7f9d98030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:11.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.458+0000 7f9d9cff9640 1 -- 172.21.15.40:0/3961739916 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d9400fc10 con 0x7f9da0136870 2024-04-22T08:20:11.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.458+0000 7f9d9cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:11.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.458+0000 7f9d9cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:20:11.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.458+0000 7f9d9cff9640 10 monclient: dump: 2024-04-22T08:20:11.472 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:11.472 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:11.472 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:11.472 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:11.472 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:11.472 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:11.472 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:11.472 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:11.472 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:11.472 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:11.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.459+0000 7f9d9cff9640 1 -- 172.21.15.40:0/3961739916 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f9d9400fd70 con 0x7f9da0136870 2024-04-22T08:20:11.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.459+0000 7f9d9cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:20:11.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.459+0000 7f9d9cff9640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:20:11.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.459+0000 7f9d9cff9640 1 -- 172.21.15.40:0/3961739916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9da0136870 msgr2=0x7f9da0138c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:11.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.459+0000 7f9d9cff9640 1 --2- 172.21.15.40:0/3961739916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9da0136870 0x7f9da0138c10 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9da0139c50 tx=0x7f9d94007700 comp rx=0 tx=0).stop 2024-04-22T08:20:11.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.459+0000 7f9d9cff9640 1 -- 172.21.15.40:0/3961739916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9da0136870 msgr2=0x7f9da0138c10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:20:11.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.459+0000 7f9d9cff9640 1 --2- 172.21.15.40:0/3961739916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9da0136870 0x7f9da0138c10 secure :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9da0139c50 tx=0x7f9d94007700 comp rx=0 tx=0).stop 2024-04-22T08:20:11.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.465+0000 7f9da6a08640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:20:11.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.465+0000 7f9da6a08640 1 --2- 172.21.15.40:0/3961739916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9da0139150 0x7f9da002b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:11.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.465+0000 7f9da6a08640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:11.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.465+0000 7f9da6a08640 1 -- 172.21.15.40:0/3961739916 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da013eb40 con 0x7f9da0139150 2024-04-22T08:20:11.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.466+0000 7f9d9f7fe640 1 --2- 172.21.15.40:0/3961739916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9da0139150 0x7f9da002b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:11.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.466+0000 7f9d9f7fe640 10 monclient: get_auth_request con 0x7f9da0139150 auth_method 0 2024-04-22T08:20:11.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.466+0000 7f9d9f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:11.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.466+0000 7f9d9f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:11.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.466+0000 7f9d9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:11.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.466+0000 7f9d9f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:11.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.466+0000 7f9d9f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:11.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.466+0000 7f9d9f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:11.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.466+0000 7f9d9f7fe640 10 monclient(hunting): handle_auth_done global_id 64705 payload 824 2024-04-22T08:20:11.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.466+0000 7f9d9f7fe640 1 -- 172.21.15.40:0/3961739916 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9da004e9b0 con 0x7f9da0139150 2024-04-22T08:20:11.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.469+0000 7f9d9f7fe640 1 -- 172.21.15.40:0/3961739916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9da0139150 msgr2=0x7f9da002b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:11.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.469+0000 7f9d9f7fe640 1 --2- 172.21.15.40:0/3961739916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9da0139150 0x7f9da002b200 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9d94018000 tx=0x7f9d94004940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:11.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.469+0000 7f9d9cff9640 1 -- 172.21.15.40:0/3961739916 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d94024070 con 0x7f9da0139150 2024-04-22T08:20:11.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.469+0000 7f9d9cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:11.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.469+0000 7f9d9cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:20:11.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.469+0000 7f9d9cff9640 10 monclient: dump: 2024-04-22T08:20:11.475 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:11.475 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:11.475 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:11.475 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:11.475 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:11.475 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:11.475 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:11.476 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:11.476 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:11.476 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:11.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.469+0000 7f9d9cff9640 1 -- 172.21.15.40:0/3961739916 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4801 (secure 0 0 0) 0x7f9d9401c840 con 0x7f9da0139150 2024-04-22T08:20:11.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.469+0000 7f9d9cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:20:11.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.469+0000 7f9d9cff9640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:20:11.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.469+0000 7f9d9cff9640 1 -- 172.21.15.40:0/3961739916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9da0139150 msgr2=0x7f9da002b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:11.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.469+0000 7f9d9cff9640 1 --2- 172.21.15.40:0/3961739916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9da0139150 0x7f9da002b200 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9d94018000 tx=0x7f9d94004940 comp rx=0 tx=0).stop 2024-04-22T08:20:11.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.469+0000 7f9d9cff9640 1 -- 172.21.15.40:0/3961739916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9da0139150 msgr2=0x7f9da002b200 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:20:11.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.469+0000 7f9d9cff9640 1 --2- 172.21.15.40:0/3961739916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9da0139150 0x7f9da002b200 secure :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9d94018000 tx=0x7f9d94004940 comp rx=0 tx=0).stop 2024-04-22T08:20:11.476 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:20:11.477 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:20:11.477 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:20:11.477 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:20:11.477 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 72, 2024-04-22T08:20:11.477 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:20:11.477 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:20:11.477 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:20:11.477 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:20:11.477 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:20:11.477 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 3, 2024-04-22T08:20:11.477 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:20:11.477 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:20:11.477 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:20:11.477 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:20:11.478 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:20:11.478 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:20:11.478 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:20:11.478 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:20:11.478 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:20:11.478 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:20:11.478 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:20:11.478 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:20:11.478 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:20:11.478 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:20:11.478 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:20:11.478 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:20:11.478 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:20:11.479 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:20:11.479 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:20:11.479 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:20:11.479 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:20:11.479 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:20:11.479 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:20:11.479 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:20:11.479 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:20:11.479 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:20:11.479 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:11.479 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:11.479 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:20:11.479 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:20:11.479 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:20:11.479 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:20:11.480 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:20:11.480 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:20:11.480 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:20:11.480 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:20:11.480 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:20:11.480 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:20:11.480 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:20:11.480 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:20:11.480 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:20:11.480 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:20:11.480 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:20:11.480 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:20:11.480 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:20:11.480 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:20:11.481 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:20:11.481 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:20:11.481 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:20:11.481 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:20:11.481 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:20:11.481 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:20:11.481 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:20:11.481 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:20:11.481 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:20:11.481 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:20:11.481 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:20:11.481 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:11.481 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:20:11.481 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:11.481 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:20:11.482 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:20:11.482 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:20:11.482 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:20:11.482 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:11.482 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:20:11.482 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:20:11.482 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:11.482 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:11.482 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:11.482 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:20:11.482 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:20:11.482 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:11.482 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:11.482 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:11.483 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:11.483 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:20:11.483 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:20:11.483 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:20:11.483 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:20:11.483 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:20:11.483 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:11.483 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:11.483 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:20:11.483 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:20:11.483 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:20:11.483 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:20:11.483 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:11.483 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:20:11.484 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:20:11.484 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:11.484 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:11.484 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:11.484 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:20:11.484 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:20:11.484 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:11.484 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:11.484 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:11.484 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:11.484 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:20:11.484 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:20:11.484 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:20:11.484 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:20:11.484 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:20:11.485 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:11.485 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:11.485 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:20:11.485 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:20:11.485 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:20:11.485 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:20:11.485 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:11.485 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:20:11.485 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:20:11.485 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:11.485 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:11.485 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:11.485 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:20:11.485 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:20:11.485 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:11.486 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:11.486 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:11.486 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:11.486 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:20:11.486 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:20:11.486 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:20:11.486 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:20:11.486 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:20:11.486 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:11.486 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:11.486 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:11.486 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:20:11.486 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:20:11.486 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:11.487 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:20:11.487 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:20:11.487 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:20:11.487 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:11.487 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:20:11.487 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:20:11.487 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:11.487 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:20:11.487 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:20:11.487 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 3 2024-04-22T08:20:11.487 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:11.487 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:20:11.487 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:20:11.487 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:11.488 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:20:11.488 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:20:11.488 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 2 2024-04-22T08:20:11.488 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:11.488 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:20:11.488 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": [ 2024-04-22T08:20:11.488 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:11.488 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:20:11.488 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:20:11.488 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:20:11.488 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:11.488 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:11.488 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:11.488 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:20:11.488 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:20:11.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.469+0000 7f9da6a08640 1 -- 172.21.15.40:0/3961739916 >> v2:172.21.15.189:6800/30168 conn(0x7f9d8c0377d0 msgr2=0x7f9d8c039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:11.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.469+0000 7f9da6a08640 1 --2- 172.21.15.40:0/3961739916 >> v2:172.21.15.189:6800/30168 conn(0x7f9d8c0377d0 0x7f9d8c039c90 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f9d98033000 tx=0x7f9d98030000 comp rx=0 tx=0).stop 2024-04-22T08:20:11.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.469+0000 7f9da6a08640 10 monclient: shutdown 2024-04-22T08:20:11.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.469+0000 7f9da6a08640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:20:11.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.469+0000 7f9da6a08640 1 -- 172.21.15.40:0/3961739916 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9da0136070 msgr2=0x7f9da013cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:11.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.469+0000 7f9da6a08640 1 --2- 172.21.15.40:0/3961739916 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9da0136070 0x7f9da013cb40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9d9000c430 tx=0x7f9d9000c8f0 comp rx=0 tx=0).stop 2024-04-22T08:20:11.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.469+0000 7f9da4fa5640 1 -- 172.21.15.40:0/3961739916 reap_dead start 2024-04-22T08:20:11.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.470+0000 7f9da6a08640 1 -- 172.21.15.40:0/3961739916 shutdown_connections 2024-04-22T08:20:11.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.470+0000 7f9da6a08640 1 --2- 172.21.15.40:0/3961739916 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9da0136070 0x7f9da013cb40 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:11.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.470+0000 7f9da6a08640 1 -- 172.21.15.40:0/3961739916 >> 172.21.15.40:0/3961739916 conn(0x7f9da00a1b60 msgr2=0x7f9da006e030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:20:11.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.470+0000 7f9da6a08640 1 -- 172.21.15.40:0/3961739916 shutdown_connections 2024-04-22T08:20:11.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.470+0000 7f9da6a08640 1 -- 172.21.15.40:0/3961739916 wait complete. 2024-04-22T08:20:11.491 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:20:11.491 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-04-22T08:20:11.491 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-04-22T08:20:11.491 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-04-22T08:20:11.491 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-04-22T08:20:11.491 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-22T08:20:11.492 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T08:20:11.507 INFO:tasks.ceph.mon.c:Stopped 2024-04-22T08:20:11.507 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-22T08:20:11.508 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:20:11.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.649+0000 7fb2182d5640 10 monclient: get_monmap_and_config 2024-04-22T08:20:11.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.649+0000 7fb2182d5640 10 monclient: build_initial_monmap 2024-04-22T08:20:11.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.649+0000 7fb2182d5640 10 monclient: monmap: 2024-04-22T08:20:11.650 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:20:11.650 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:11.650 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:20:11.650391+0000 2024-04-22T08:20:11.650 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:20:11.650391+0000 2024-04-22T08:20:11.650 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:20:11.650 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:11.650 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:20:11.650 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:20:11.650 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:20:11.650 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:20:11.650 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:20:11.650 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:11.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.650+0000 7fb2182d5640 1 Processor -- start 2024-04-22T08:20:11.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.650+0000 7fb2182d5640 1 -- start start 2024-04-22T08:20:11.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.650+0000 7fb2182d5640 10 monclient: init 2024-04-22T08:20:11.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.650+0000 7fb2182d5640 10 monclient: _reopen_session rank -1 2024-04-22T08:20:11.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.650+0000 7fb2182d5640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-04-22T08:20:11.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.650+0000 7fb2182d5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb210135e30 addr v1:172.21.15.189:6789/0 2024-04-22T08:20:11.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.650+0000 7fb2182d5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb210071c90 addr v1:172.21.15.40:6789/0 2024-04-22T08:20:11.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.650+0000 7fb2182d5640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb21013c5f0 0x7fb21013e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:11.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.650+0000 7fb2182d5640 10 monclient(hunting): picked mon.noname-c con 0x7fb21013c5f0 addr v2:172.21.15.189:3300/0 2024-04-22T08:20:11.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.650+0000 7fb2182d5640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb21009de80 con 0x7fb210071c90 2024-04-22T08:20:11.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.650+0000 7fb2182d5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb21009f130 con 0x7fb210135e30 2024-04-22T08:20:11.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.650+0000 7fb2182d5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:11.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.650+0000 7fb2182d5640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb21009e3d0 con 0x7fb21013c5f0 2024-04-22T08:20:11.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.650+0000 7fb2182d5640 10 monclient(hunting): _renew_subs 2024-04-22T08:20:11.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.650+0000 7fb2182d5640 10 monclient(hunting): authenticate will time out at 1285.644589s 2024-04-22T08:20:11.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.650+0000 7fb215870640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fb210071c90 0x7fb21013aed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53814/0 (socket says 172.21.15.40:53814) 2024-04-22T08:20:11.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.650+0000 7fb215870640 1 -- 172.21.15.40:0/1156190040 learned_addr learned my addr 172.21.15.40:0/1156190040 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:20:11.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.650+0000 7fb216872640 1 --2- 172.21.15.40:0/1156190040 >> v2:172.21.15.189:3300/0 conn(0x7fb21013c5f0 0x7fb21013e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:11.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.650+0000 7fb216872640 10 monclient(hunting): get_auth_request con 0x7fb21013c5f0 auth_method 0 2024-04-22T08:20:11.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.650+0000 7fb216872640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:11.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.650+0000 7fb216872640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:11.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.650+0000 7fb216872640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:11.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.650+0000 7fb216872640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:11.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.650+0000 7fb216872640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:11.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.650+0000 7fb216872640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:11.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb215870640 1 -- 172.21.15.40:0/1156190040 >> v1:172.21.15.40:6789/0 conn(0x7fb210071c90 legacy=0x7fb21013aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:11.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb21506f640 1 -- 172.21.15.40:0/1156190040 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1688203916 0 0) 0x7fb21009de80 con 0x7fb210071c90 2024-04-22T08:20:11.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb21506f640 10 monclient(hunting): _negotiate 2024-04-22T08:20:11.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb21506f640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:11.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb21506f640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:11.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb21506f640 10 monclient(hunting): my global_id is 94161 2024-04-22T08:20:11.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb21506f640 1 -- 172.21.15.40:0/1156190040 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1fc003520 con 0x7fb210071c90 2024-04-22T08:20:11.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb216071640 1 -- 172.21.15.40:0/1156190040 >> v1:172.21.15.189:6789/0 conn(0x7fb210135e30 legacy=0x7fb210136210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:11.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb21506f640 1 -- 172.21.15.40:0/1156190040 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1970677255 0 0) 0x7fb1fc003520 con 0x7fb210071c90 2024-04-22T08:20:11.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb21506f640 1 -- 172.21.15.40:0/1156190040 >> v2:172.21.15.189:3300/0 conn(0x7fb21013c5f0 msgr2=0x7fb21013e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:20:11.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb21506f640 1 --2- 172.21.15.40:0/1156190040 >> v2:172.21.15.189:3300/0 conn(0x7fb21013c5f0 0x7fb21013e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:11.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb21506f640 1 -- 172.21.15.40:0/1156190040 >> v1:172.21.15.189:6789/0 conn(0x7fb210135e30 legacy=0x7fb210136210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:11.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb21506f640 10 monclient: _finish_hunting 0 2024-04-22T08:20:11.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb21506f640 1 monclient: found mon.noname-a-legacy 2024-04-22T08:20:11.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb21506f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:20:11.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb21506f640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.40:6789/0 2024-04-22T08:20:11.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb21506f640 1 -- 172.21.15.40:0/1156190040 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb21009f410 con 0x7fb210071c90 2024-04-22T08:20:11.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb21506f640 10 monclient: _finish_auth 0 2024-04-22T08:20:11.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb21506f640 10 monclient: _check_auth_tickets 2024-04-22T08:20:11.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb21506f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:20:11.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb21506f640 1 -- 172.21.15.40:0/1156190040 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1087006497 0 0) 0x7fb200001920 con 0x7fb210071c90 2024-04-22T08:20:11.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb2182d5640 5 monclient: authenticate success, global_id 94161 2024-04-22T08:20:11.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb2182d5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:20:11.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb21506f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:11.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb21506f640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-22T08:20:11.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb21506f640 10 monclient: dump: 2024-04-22T08:20:11.659 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:11.659 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:11.659 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:11.660 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:11.660 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:11.660 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:11.660 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:11.660 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:11.660 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:11.660 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:11.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb21506f640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] but i'm connected to v1:172.21.15.40:6789/0, reconnecting 2024-04-22T08:20:11.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb21506f640 10 monclient: _reopen_session rank -1 2024-04-22T08:20:11.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb21506f640 1 -- 172.21.15.40:0/1156190040 >> v1:172.21.15.40:6789/0 conn(0x7fb210071c90 legacy=0x7fb21013aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:11.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb21506f640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-22T08:20:11.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb21506f640 1 --2- 172.21.15.40:0/1156190040 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1fc003a40 0x7fb1fc003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:11.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb21506f640 10 monclient(hunting): picked mon.b con 0x7fb1fc003a40 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:20:11.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb21506f640 1 --2- 172.21.15.40:0/1156190040 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb1fc0047c0 0x7fb1fc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:11.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb21506f640 10 monclient(hunting): picked mon.a con 0x7fb1fc0047c0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:20:11.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb21506f640 1 --2- 172.21.15.40:0/1156190040 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb1fc005380 0x7fb1fc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:11.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb21506f640 10 monclient(hunting): picked mon.c con 0x7fb1fc005380 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:20:11.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb21506f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:11.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb21506f640 1 -- 172.21.15.40:0/1156190040 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1fc00c1f0 con 0x7fb1fc0047c0 2024-04-22T08:20:11.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb21506f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:11.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb21506f640 1 -- 172.21.15.40:0/1156190040 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1fc00c360 con 0x7fb1fc003a40 2024-04-22T08:20:11.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb21506f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:11.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb21506f640 1 -- 172.21.15.40:0/1156190040 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1fc00c520 con 0x7fb1fc005380 2024-04-22T08:20:11.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb21506f640 10 monclient(hunting): _renew_subs 2024-04-22T08:20:11.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb21506f640 10 monclient(hunting): _finish_auth 0 2024-04-22T08:20:11.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb2182d5640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-04-22T08:20:11.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb216872640 1 --2- 172.21.15.40:0/1156190040 >> v2:172.21.15.189:3300/0 conn(0x7fb21013c5f0 0x7fb21013e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:20:11.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb216872640 1 -- 172.21.15.40:0/1156190040 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb1fc005380 msgr2=0x7fb1fc00b980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:20:11.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb216872640 1 --2- 172.21.15.40:0/1156190040 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb1fc005380 0x7fb1fc00b980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:20:11.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb215870640 1 --2- 172.21.15.40:0/1156190040 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb1fc0047c0 0x7fb1fc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:11.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb215870640 10 monclient(hunting): get_auth_request con 0x7fb1fc0047c0 auth_method 0 2024-04-22T08:20:11.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb215870640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:11.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb215870640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:11.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb215870640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:20:11.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb216071640 1 --2- 172.21.15.40:0/1156190040 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1fc003a40 0x7fb1fc003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:11.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb216071640 10 monclient(hunting): get_auth_request con 0x7fb1fc003a40 auth_method 0 2024-04-22T08:20:11.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb216071640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:11.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb216071640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:11.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.652+0000 7fb216071640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:20:11.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.653+0000 7fb215870640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:11.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.653+0000 7fb215870640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:11.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.653+0000 7fb215870640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-22T08:20:11.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.653+0000 7fb216071640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:11.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.653+0000 7fb216071640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:11.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.653+0000 7fb216071640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-22T08:20:11.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.653+0000 7fb215870640 10 monclient(hunting): handle_auth_done global_id 94161 payload 293 2024-04-22T08:20:11.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.653+0000 7fb215870640 1 -- 172.21.15.40:0/1156190040 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb1fc005380 msgr2=0x7fb1fc00b980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:20:11.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.653+0000 7fb215870640 1 --2- 172.21.15.40:0/1156190040 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb1fc005380 0x7fb1fc00b980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:11.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.653+0000 7fb215870640 1 -- 172.21.15.40:0/1156190040 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1fc003a40 msgr2=0x7fb1fc003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:20:11.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.653+0000 7fb215870640 1 --2- 172.21.15.40:0/1156190040 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1fc003a40 0x7fb1fc003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:11.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.653+0000 7fb215870640 10 monclient: _finish_hunting 0 2024-04-22T08:20:11.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.653+0000 7fb215870640 1 monclient: found mon.a 2024-04-22T08:20:11.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.653+0000 7fb216872640 1 -- 172.21.15.40:0/1156190040 reap_dead start 2024-04-22T08:20:11.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.653+0000 7fb215870640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:20:11.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.653+0000 7fb215870640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:20:11.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.653+0000 7fb215870640 1 -- 172.21.15.40:0/1156190040 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1fc003520 con 0x7fb1fc0047c0 2024-04-22T08:20:11.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.653+0000 7fb215870640 10 monclient: _finish_auth 0 2024-04-22T08:20:11.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.653+0000 7fb215870640 10 monclient: _check_auth_tickets 2024-04-22T08:20:11.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.653+0000 7fb215870640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:20:11.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.656+0000 7fb215870640 1 -- 172.21.15.40:0/1156190040 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb1fc0047c0 msgr2=0x7fb1fc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:11.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.656+0000 7fb215870640 1 --2- 172.21.15.40:0/1156190040 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb1fc0047c0 0x7fb1fc004c40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb200001b00 tx=0x7fb1fc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c3af38353a70d1f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:11.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.656+0000 7fb21506f640 1 -- 172.21.15.40:0/1156190040 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb20002cb30 con 0x7fb1fc0047c0 2024-04-22T08:20:11.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.656+0000 7fb21506f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:11.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.656+0000 7fb21506f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:20:11.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.656+0000 7fb21506f640 10 monclient: dump: 2024-04-22T08:20:11.670 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:11.670 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:11.670 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:11.670 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:11.670 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:11.670 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:11.670 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:11.671 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:11.671 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:11.671 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:11.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.656+0000 7fb2182d5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:20:11.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.656+0000 7fb21506f640 1 -- 172.21.15.40:0/1156190040 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb20002ccb0 con 0x7fb1fc0047c0 2024-04-22T08:20:11.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.656+0000 7fb21506f640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:20:11.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.656+0000 7fb21506f640 1 -- 172.21.15.40:0/1156190040 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb200005b10 con 0x7fb1fc0047c0 2024-04-22T08:20:11.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.656+0000 7fb21506f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:11.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.656+0000 7fb21506f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:20:11.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.656+0000 7fb21506f640 10 monclient: dump: 2024-04-22T08:20:11.671 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:11.671 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:11.671 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:11.671 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:11.671 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:11.672 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:11.672 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:11.672 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:11.672 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:11.672 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:11.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.656+0000 7fb2182d5640 10 monclient: get_monmap_and_config success 2024-04-22T08:20:11.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.656+0000 7fb2182d5640 4 set_mon_vals no callback set 2024-04-22T08:20:11.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.656+0000 7fb2182d5640 10 monclient: shutdown 2024-04-22T08:20:11.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.656+0000 7fb2182d5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:20:11.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.656+0000 7fb2182d5640 1 -- 172.21.15.40:0/1156190040 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb1fc0047c0 msgr2=0x7fb1fc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:11.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.656+0000 7fb2182d5640 1 --2- 172.21.15.40:0/1156190040 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb1fc0047c0 0x7fb1fc004c40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb200001b00 tx=0x7fb1fc0057f0 comp rx=0 tx=0).stop 2024-04-22T08:20:11.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.656+0000 7fb2182d5640 1 -- 172.21.15.40:0/1156190040 shutdown_connections 2024-04-22T08:20:11.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.656+0000 7fb2182d5640 1 --2- 172.21.15.40:0/1156190040 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb1fc0047c0 0x7fb1fc004c40 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:11.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.656+0000 7fb2182d5640 1 -- 172.21.15.40:0/1156190040 >> 172.21.15.40:0/1156190040 conn(0x7fb2100a65c0 msgr2=0x7fb2100a2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:20:11.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.656+0000 7fb2182d5640 1 -- 172.21.15.40:0/1156190040 shutdown_connections 2024-04-22T08:20:11.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.656+0000 7fb2182d5640 1 -- 172.21.15.40:0/1156190040 wait complete. 2024-04-22T08:20:11.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.657+0000 7fb2182d5640 10 monclient: build_initial_monmap 2024-04-22T08:20:11.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.657+0000 7fb2182d5640 10 monclient: monmap: 2024-04-22T08:20:11.673 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:20:11.673 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:20:11.673 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:20:11.673 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:20:11.673 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:20:11.673 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:11.673 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:20:11.673 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:20:11.673 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:20:11.673 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:11.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.657+0000 7fb2182d5640 1 Processor -- start 2024-04-22T08:20:11.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.657+0000 7fb2182d5640 1 -- start start 2024-04-22T08:20:11.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.657+0000 7fb2182d5640 10 monclient: init 2024-04-22T08:20:11.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.658+0000 7fb2182d5640 10 monclient: _reopen_session rank -1 2024-04-22T08:20:11.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.658+0000 7fb2182d5640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-22T08:20:11.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.658+0000 7fb2182d5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1fc0047c0 0x7fb2101cadd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:11.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.658+0000 7fb2182d5640 10 monclient(hunting): picked mon.noname-b con 0x7fb1fc0047c0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:20:11.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.658+0000 7fb2182d5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2101cb310 0x7fb210137c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:11.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.658+0000 7fb2182d5640 10 monclient(hunting): picked mon.noname-a con 0x7fb2101cb310 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:20:11.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.658+0000 7fb2182d5640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2101cb790 0x7fb210138150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:11.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.658+0000 7fb2182d5640 10 monclient(hunting): picked mon.noname-c con 0x7fb2101cb790 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:20:11.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.658+0000 7fb2182d5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:11.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.658+0000 7fb2182d5640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb21013a910 con 0x7fb2101cb310 2024-04-22T08:20:11.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.658+0000 7fb2182d5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:11.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.658+0000 7fb2182d5640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb21013bf90 con 0x7fb1fc0047c0 2024-04-22T08:20:11.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.658+0000 7fb2182d5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:11.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.658+0000 7fb2182d5640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb21013bb70 con 0x7fb2101cb790 2024-04-22T08:20:11.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.658+0000 7fb2182d5640 10 monclient(hunting): _renew_subs 2024-04-22T08:20:11.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.658+0000 7fb2182d5640 10 monclient(hunting): authenticate will time out at 1285.653158s 2024-04-22T08:20:11.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.658+0000 7fb216872640 1 -- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2101cb790 msgr2=0x7fb210138150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:20:11.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.658+0000 7fb216872640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2101cb790 0x7fb210138150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:20:11.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.658+0000 7fb215870640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2101cb310 0x7fb210137c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:11.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.658+0000 7fb215870640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2101cb310 0x7fb210137c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46460/0 (socket says 172.21.15.40:46460) 2024-04-22T08:20:11.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.658+0000 7fb215870640 1 -- 172.21.15.40:0/2441334864 learned_addr learned my addr 172.21.15.40:0/2441334864 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:20:11.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.658+0000 7fb215870640 10 monclient(hunting): get_auth_request con 0x7fb2101cb310 auth_method 0 2024-04-22T08:20:11.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.658+0000 7fb216071640 1 --2- 172.21.15.40:0/2441334864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1fc0047c0 0x7fb2101cadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:11.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.658+0000 7fb215870640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:11.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.658+0000 7fb215870640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:11.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.658+0000 7fb215870640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:11.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.658+0000 7fb216071640 10 monclient(hunting): get_auth_request con 0x7fb1fc0047c0 auth_method 0 2024-04-22T08:20:11.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.658+0000 7fb216071640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:11.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.658+0000 7fb216071640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:11.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.658+0000 7fb216071640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:11.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.658+0000 7fb215870640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:11.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.658+0000 7fb215870640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:11.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.658+0000 7fb215870640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:11.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.658+0000 7fb216071640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:11.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.658+0000 7fb216071640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:11.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.658+0000 7fb216071640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:11.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.658+0000 7fb215870640 10 monclient(hunting): handle_auth_done global_id 94164 payload 824 2024-04-22T08:20:11.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.658+0000 7fb215870640 1 -- 172.21.15.40:0/2441334864 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2101cb790 msgr2=0x7fb210138150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:20:11.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.658+0000 7fb215870640 1 --2- 172.21.15.40:0/2441334864 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2101cb790 0x7fb210138150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:11.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.658+0000 7fb215870640 1 -- 172.21.15.40:0/2441334864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1fc0047c0 msgr2=0x7fb2101cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:11.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.658+0000 7fb215870640 1 --2- 172.21.15.40:0/2441334864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1fc0047c0 0x7fb2101cadd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:11.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.658+0000 7fb215870640 10 monclient: _finish_hunting 0 2024-04-22T08:20:11.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.658+0000 7fb215870640 1 monclient: found mon.noname-a 2024-04-22T08:20:11.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.658+0000 7fb215870640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:20:11.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.658+0000 7fb215870640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:20:11.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.658+0000 7fb215870640 1 -- 172.21.15.40:0/2441334864 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2100720e0 con 0x7fb2101cb310 2024-04-22T08:20:11.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.661+0000 7fb215870640 1 -- 172.21.15.40:0/2441334864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2101cb310 msgr2=0x7fb210137c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:11.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.661+0000 7fb215870640 1 --2- 172.21.15.40:0/2441334864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2101cb310 0x7fb210137c10 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb20000c470 tx=0x7fb20002e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:11.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.661+0000 7fb206ffd640 1 -- 172.21.15.40:0/2441334864 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb20003c040 con 0x7fb2101cb310 2024-04-22T08:20:11.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.661+0000 7fb206ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:11.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.661+0000 7fb206ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:20:11.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.661+0000 7fb206ffd640 10 monclient: dump: 2024-04-22T08:20:11.677 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:11.677 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:11.677 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:11.678 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:11.678 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:11.678 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:11.678 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:11.678 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:11.678 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:11.678 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:11.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.661+0000 7fb206ffd640 10 monclient: _finish_auth 0 2024-04-22T08:20:11.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.661+0000 7fb206ffd640 10 monclient: _check_auth_tickets 2024-04-22T08:20:11.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.661+0000 7fb206ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:20:11.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.661+0000 7fb206ffd640 1 -- 172.21.15.40:0/2441334864 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb20002cb40 con 0x7fb2101cb310 2024-04-22T08:20:11.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.661+0000 7fb2182d5640 5 monclient: authenticate success, global_id 94164 2024-04-22T08:20:11.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.661+0000 7fb206ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:20:11.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.661+0000 7fb206ffd640 1 -- 172.21.15.40:0/2441334864 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb200007470 con 0x7fb2101cb310 2024-04-22T08:20:11.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.661+0000 7fb206ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:11.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.661+0000 7fb206ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:20:11.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.661+0000 7fb206ffd640 10 monclient: dump: 2024-04-22T08:20:11.679 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:11.679 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:11.679 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:11.679 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:11.679 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:11.679 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:11.679 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:11.679 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:11.679 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:11.679 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:11.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.661+0000 7fb2182d5640 10 monclient: _renew_subs 2024-04-22T08:20:11.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.661+0000 7fb2182d5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:20:11.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.661+0000 7fb207fff640 4 set_mon_vals no callback set 2024-04-22T08:20:11.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.661+0000 7fb2182d5640 1 -- 172.21.15.40:0/2441334864 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb21009c270 con 0x7fb2101cb310 2024-04-22T08:20:11.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.661+0000 7fb2182d5640 10 monclient: _renew_subs 2024-04-22T08:20:11.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.661+0000 7fb2182d5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:20:11.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.661+0000 7fb2182d5640 1 -- 172.21.15.40:0/2441334864 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb21013a160 con 0x7fb2101cb310 2024-04-22T08:20:11.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.662+0000 7fb206ffd640 1 -- 172.21.15.40:0/2441334864 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fb200032070 con 0x7fb2101cb310 2024-04-22T08:20:11.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.662+0000 7fb2182d5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:20:11.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.662+0000 7fb2182d5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:20:11.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.662+0000 7fb2182d5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:20:11.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.662+0000 7fb2182d5640 1 -- 172.21.15.40:0/2441334864 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb21013b2d0 con 0x7fb2101cb310 2024-04-22T08:20:11.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.662+0000 7fb206ffd640 1 --2- 172.21.15.40:0/2441334864 >> v2:172.21.15.189:6800/30168 conn(0x7fb1f4037780 0x7fb1f4039c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:11.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.662+0000 7fb206ffd640 1 -- 172.21.15.40:0/2441334864 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 3506+0+0 (secure 0 0 0) 0x7fb20006b6d0 con 0x7fb2101cb310 2024-04-22T08:20:11.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.662+0000 7fb216071640 1 --2- 172.21.15.40:0/2441334864 >> v2:172.21.15.189:6800/30168 conn(0x7fb1f4037780 0x7fb1f4039c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:11.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.662+0000 7fb216071640 10 monclient: get_auth_request con 0x7fb1f4037780 auth_method 0 2024-04-22T08:20:11.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.666+0000 7fb216071640 1 --2- 172.21.15.40:0/2441334864 >> v2:172.21.15.189:6800/30168 conn(0x7fb1f4037780 0x7fb1f4039c40 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fb21009dc10 tx=0x7fb20c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:11.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.667+0000 7fb206ffd640 1 -- 172.21.15.40:0/2441334864 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7fb2000343d0 con 0x7fb2101cb310 2024-04-22T08:20:11.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.667+0000 7fb206ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:20:11.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.667+0000 7fb206ffd640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:20:11.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.916+0000 7fb2182d5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:20:11.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.916+0000 7fb2182d5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:20:11.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.916+0000 7fb2182d5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:20:11.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.916+0000 7fb2182d5640 1 -- 172.21.15.40:0/2441334864 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb21004e9b0 con 0x7fb2101cb310 2024-04-22T08:20:11.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.917+0000 7fb206ffd640 1 -- 172.21.15.40:0/2441334864 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7fb200033b20 con 0x7fb2101cb310 2024-04-22T08:20:11.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.917+0000 7fb206ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:20:11.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.917+0000 7fb206ffd640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:20:11.918 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":72,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:20:11.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.918+0000 7fb2182d5640 1 -- 172.21.15.40:0/2441334864 >> v2:172.21.15.189:6800/30168 conn(0x7fb1f4037780 msgr2=0x7fb1f4039c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:11.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.918+0000 7fb2182d5640 1 --2- 172.21.15.40:0/2441334864 >> v2:172.21.15.189:6800/30168 conn(0x7fb1f4037780 0x7fb1f4039c40 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fb21009dc10 tx=0x7fb20c009040 comp rx=0 tx=0).stop 2024-04-22T08:20:11.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.918+0000 7fb2182d5640 10 monclient: shutdown 2024-04-22T08:20:11.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.918+0000 7fb2182d5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:20:11.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.918+0000 7fb2182d5640 1 -- 172.21.15.40:0/2441334864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2101cb310 msgr2=0x7fb210137c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:11.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.918+0000 7fb2182d5640 1 --2- 172.21.15.40:0/2441334864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2101cb310 0x7fb210137c10 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb20000c470 tx=0x7fb20002e070 comp rx=0 tx=0).stop 2024-04-22T08:20:11.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.918+0000 7fb2182d5640 1 -- 172.21.15.40:0/2441334864 shutdown_connections 2024-04-22T08:20:11.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.918+0000 7fb2182d5640 1 --2- 172.21.15.40:0/2441334864 >> v2:172.21.15.189:6800/30168 conn(0x7fb1f4037780 0x7fb1f4039c40 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:11.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.918+0000 7fb2182d5640 1 --2- 172.21.15.40:0/2441334864 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2101cb790 0x7fb210138150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:11.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.918+0000 7fb2182d5640 1 --2- 172.21.15.40:0/2441334864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2101cb310 0x7fb210137c10 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:11.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.918+0000 7fb2182d5640 1 --2- 172.21.15.40:0/2441334864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1fc0047c0 0x7fb2101cadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:11.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.918+0000 7fb2182d5640 1 -- 172.21.15.40:0/2441334864 >> 172.21.15.40:0/2441334864 conn(0x7fb2100a65c0 msgr2=0x7fb21009a480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:20:11.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.918+0000 7fb2182d5640 1 -- 172.21.15.40:0/2441334864 shutdown_connections 2024-04-22T08:20:11.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:11.919+0000 7fb2182d5640 1 -- 172.21.15.40:0/2441334864 wait complete. 2024-04-22T08:20:14.931 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:20:15.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.041+0000 7ff20c0ce640 10 monclient: get_monmap_and_config 2024-04-22T08:20:15.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.042+0000 7ff20c0ce640 10 monclient: build_initial_monmap 2024-04-22T08:20:15.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.042+0000 7ff20c0ce640 10 monclient: monmap: 2024-04-22T08:20:15.041 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:20:15.042 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:15.042 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:20:15.042796+0000 2024-04-22T08:20:15.042 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:20:15.042796+0000 2024-04-22T08:20:15.042 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:20:15.042 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:15.042 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:20:15.042 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:20:15.042 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:20:15.042 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:20:15.042 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:20:15.042 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:15.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.042+0000 7ff20c0ce640 1 Processor -- start 2024-04-22T08:20:15.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.042+0000 7ff20c0ce640 1 -- start start 2024-04-22T08:20:15.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.042+0000 7ff20c0ce640 10 monclient: init 2024-04-22T08:20:15.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.043+0000 7ff20c0ce640 10 monclient: _reopen_session rank -1 2024-04-22T08:20:15.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.043+0000 7ff20c0ce640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-04-22T08:20:15.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.043+0000 7ff20c0ce640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff2040a0a00 0x7ff20409eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:15.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.043+0000 7ff20c0ce640 10 monclient(hunting): picked mon.noname-a con 0x7ff2040a0a00 addr v2:172.21.15.40:3300/0 2024-04-22T08:20:15.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.043+0000 7ff20c0ce640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff204132990 addr v1:172.21.15.40:6789/0 2024-04-22T08:20:15.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.043+0000 7ff20c0ce640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff20409f0a0 0x7ff20409a100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:15.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.043+0000 7ff20c0ce640 10 monclient(hunting): picked mon.noname-b con 0x7ff20409f0a0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:20:15.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.043+0000 7ff20c0ce640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff20409b460 con 0x7ff204132990 2024-04-22T08:20:15.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.043+0000 7ff20c0ce640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:15.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.043+0000 7ff20c0ce640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2040a0de0 con 0x7ff2040a0a00 2024-04-22T08:20:15.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.043+0000 7ff20c0ce640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:15.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.043+0000 7ff20c0ce640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2040a13f0 con 0x7ff20409f0a0 2024-04-22T08:20:15.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.043+0000 7ff20c0ce640 10 monclient(hunting): _renew_subs 2024-04-22T08:20:15.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.043+0000 7ff20c0ce640 10 monclient(hunting): authenticate will time out at 1289.037208s 2024-04-22T08:20:15.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.043+0000 7ff20a66b640 1 -- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff20409f0a0 msgr2=0x7ff20409a100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:20:15.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.043+0000 7ff20a66b640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff20409f0a0 0x7ff20409a100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:20:15.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.043+0000 7ff209e6a640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff2040a0a00 0x7ff20409eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:15.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.043+0000 7ff209669640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7ff204132990 0x7ff2040999f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53824/0 (socket says 172.21.15.40:53824) 2024-04-22T08:20:15.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.043+0000 7ff209669640 1 -- 172.21.15.40:0/710327720 learned_addr learned my addr 172.21.15.40:0/710327720 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:20:15.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.043+0000 7ff209e6a640 10 monclient(hunting): get_auth_request con 0x7ff2040a0a00 auth_method 0 2024-04-22T08:20:15.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.043+0000 7ff209e6a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:15.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.043+0000 7ff209e6a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:15.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.043+0000 7ff209e6a640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:15.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.043+0000 7ff209e6a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:15.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.043+0000 7ff209e6a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:15.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.043+0000 7ff209e6a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:15.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.043+0000 7ff209e6a640 10 monclient(hunting): handle_auth_done global_id 94167 payload 274 2024-04-22T08:20:15.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.043+0000 7ff209e6a640 1 -- 172.21.15.40:0/710327720 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff20409f0a0 msgr2=0x7ff20409a100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:20:15.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.043+0000 7ff209e6a640 1 --2- 172.21.15.40:0/710327720 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff20409f0a0 0x7ff20409a100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:15.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.043+0000 7ff209e6a640 1 -- 172.21.15.40:0/710327720 >> v1:172.21.15.40:6789/0 conn(0x7ff204132990 legacy=0x7ff2040999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:20:15.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.043+0000 7ff209e6a640 10 monclient: _finish_hunting 0 2024-04-22T08:20:15.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.043+0000 7ff209e6a640 1 monclient: found mon.noname-a 2024-04-22T08:20:15.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.043+0000 7ff209e6a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:20:15.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.043+0000 7ff209e6a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:20:15.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.043+0000 7ff209e6a640 1 -- 172.21.15.40:0/710327720 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff204132ed0 con 0x7ff2040a0a00 2024-04-22T08:20:15.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.046+0000 7ff209e6a640 1 -- 172.21.15.40:0/710327720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2040a0a00 msgr2=0x7ff20409eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:15.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.046+0000 7ff209e6a640 1 --2- 172.21.15.40:0/710327720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2040a0a00 0x7ff20409eb60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff2000097a0 tx=0x7ff20002b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=669e71653dd527b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:15.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.046+0000 7ff208e68640 1 -- 172.21.15.40:0/710327720 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff200031a00 con 0x7ff2040a0a00 2024-04-22T08:20:15.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.046+0000 7ff208e68640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:15.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.047+0000 7ff208e68640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:20:15.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.047+0000 7ff208e68640 10 monclient: dump: 2024-04-22T08:20:15.053 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:15.053 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:15.053 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:15.053 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:15.053 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:15.053 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:15.053 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:15.053 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:15.053 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:15.053 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:15.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.047+0000 7ff208e68640 10 monclient: _finish_auth 0 2024-04-22T08:20:15.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.047+0000 7ff208e68640 10 monclient: _check_auth_tickets 2024-04-22T08:20:15.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.047+0000 7ff208e68640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:20:15.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.047+0000 7ff208e68640 1 -- 172.21.15.40:0/710327720 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff20002f050 con 0x7ff2040a0a00 2024-04-22T08:20:15.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.047+0000 7ff20c0ce640 5 monclient: authenticate success, global_id 94167 2024-04-22T08:20:15.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.047+0000 7ff20c0ce640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:20:15.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.047+0000 7ff208e68640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:20:15.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.047+0000 7ff208e68640 1 -- 172.21.15.40:0/710327720 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff200031d80 con 0x7ff2040a0a00 2024-04-22T08:20:15.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.047+0000 7ff20c0ce640 10 monclient: get_monmap_and_config success 2024-04-22T08:20:15.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.047+0000 7ff20c0ce640 4 set_mon_vals no callback set 2024-04-22T08:20:15.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.047+0000 7ff208e68640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:15.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.047+0000 7ff208e68640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:20:15.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.047+0000 7ff208e68640 10 monclient: dump: 2024-04-22T08:20:15.054 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:15.054 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:15.054 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:15.054 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:15.055 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:15.055 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:15.055 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:15.055 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:15.055 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:15.055 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:15.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.047+0000 7ff20c0ce640 10 monclient: shutdown 2024-04-22T08:20:15.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.047+0000 7ff20c0ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:20:15.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.047+0000 7ff20c0ce640 1 -- 172.21.15.40:0/710327720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2040a0a00 msgr2=0x7ff20409eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:15.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.047+0000 7ff20c0ce640 1 --2- 172.21.15.40:0/710327720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2040a0a00 0x7ff20409eb60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff2000097a0 tx=0x7ff20002b7b0 comp rx=0 tx=0).stop 2024-04-22T08:20:15.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.047+0000 7ff20c0ce640 1 -- 172.21.15.40:0/710327720 shutdown_connections 2024-04-22T08:20:15.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.047+0000 7ff20c0ce640 1 --2- 172.21.15.40:0/710327720 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff20409f0a0 0x7ff20409a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:15.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.047+0000 7ff20c0ce640 1 --2- 172.21.15.40:0/710327720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2040a0a00 0x7ff20409eb60 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:15.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.047+0000 7ff20c0ce640 1 -- 172.21.15.40:0/710327720 >> 172.21.15.40:0/710327720 conn(0x7ff2040a5960 msgr2=0x7ff2040a1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:20:15.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.047+0000 7ff20c0ce640 1 -- 172.21.15.40:0/710327720 shutdown_connections 2024-04-22T08:20:15.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.047+0000 7ff20c0ce640 1 -- 172.21.15.40:0/710327720 wait complete. 2024-04-22T08:20:15.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.048+0000 7ff20c0ce640 10 monclient: build_initial_monmap 2024-04-22T08:20:15.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.048+0000 7ff20c0ce640 10 monclient: monmap: 2024-04-22T08:20:15.056 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:20:15.056 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:20:15.056 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:20:15.056 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:20:15.056 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:20:15.056 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:15.056 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:20:15.056 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:20:15.056 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:20:15.056 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:15.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.048+0000 7ff20c0ce640 1 Processor -- start 2024-04-22T08:20:15.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.048+0000 7ff20c0ce640 1 -- start start 2024-04-22T08:20:15.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.048+0000 7ff20c0ce640 10 monclient: init 2024-04-22T08:20:15.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.049+0000 7ff20c0ce640 10 monclient: _reopen_session rank -1 2024-04-22T08:20:15.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.049+0000 7ff20c0ce640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-22T08:20:15.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.049+0000 7ff20c0ce640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff20409f0a0 0x7ff2041c8c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:15.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.049+0000 7ff20c0ce640 10 monclient(hunting): picked mon.noname-b con 0x7ff20409f0a0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:20:15.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.049+0000 7ff20c0ce640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff2040a0a00 0x7ff2041c91a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:15.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.049+0000 7ff20c0ce640 10 monclient(hunting): picked mon.noname-c con 0x7ff2040a0a00 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:20:15.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.049+0000 7ff20c0ce640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff204132990 0x7ff2041c96e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:15.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.049+0000 7ff20c0ce640 10 monclient(hunting): picked mon.noname-a con 0x7ff204132990 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:20:15.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.049+0000 7ff20c0ce640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:15.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.049+0000 7ff20c0ce640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff204071420 con 0x7ff204132990 2024-04-22T08:20:15.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.049+0000 7ff20c0ce640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:15.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.049+0000 7ff20c0ce640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2040a4610 con 0x7ff20409f0a0 2024-04-22T08:20:15.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.049+0000 7ff20c0ce640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:15.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.049+0000 7ff20c0ce640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff20406f030 con 0x7ff2040a0a00 2024-04-22T08:20:15.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.049+0000 7ff20c0ce640 10 monclient(hunting): _renew_subs 2024-04-22T08:20:15.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.049+0000 7ff20c0ce640 10 monclient(hunting): authenticate will time out at 1289.043041s 2024-04-22T08:20:15.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.049+0000 7ff209669640 1 -- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff2040a0a00 msgr2=0x7ff2041c91a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:20:15.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.049+0000 7ff209669640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff2040a0a00 0x7ff2041c91a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:20:15.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.049+0000 7ff20a66b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff204132990 0x7ff2041c96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:15.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.049+0000 7ff20a66b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff204132990 0x7ff2041c96e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46476/0 (socket says 172.21.15.40:46476) 2024-04-22T08:20:15.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.049+0000 7ff20a66b640 1 -- 172.21.15.40:0/3007730202 learned_addr learned my addr 172.21.15.40:0/3007730202 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:20:15.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.049+0000 7ff20a66b640 10 monclient(hunting): get_auth_request con 0x7ff204132990 auth_method 0 2024-04-22T08:20:15.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.049+0000 7ff20a66b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:15.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.049+0000 7ff20a66b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:15.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.049+0000 7ff20a66b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:15.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.049+0000 7ff20a66b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:15.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.049+0000 7ff20a66b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:15.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.049+0000 7ff20a66b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:15.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.049+0000 7ff20a66b640 1 -- 172.21.15.40:0/3007730202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff204132990 msgr2=0x7ff2041c96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-22T08:20:15.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.049+0000 7ff20a66b640 1 -- 172.21.15.40:0/3007730202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff204132990 msgr2=0x7ff2041c96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:20:15.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.049+0000 7ff20a66b640 1 --2- 172.21.15.40:0/3007730202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff204132990 0x7ff2041c96e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:20:15.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.049+0000 7ff20a66b640 1 --2- 172.21.15.40:0/3007730202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff204132990 0x7ff2041c96e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:20:15.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.049+0000 7ff209e6a640 1 --2- 172.21.15.40:0/3007730202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff20409f0a0 0x7ff2041c8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:15.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.049+0000 7ff209e6a640 10 monclient(hunting): get_auth_request con 0x7ff20409f0a0 auth_method 0 2024-04-22T08:20:15.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.049+0000 7ff209e6a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:15.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.049+0000 7ff209e6a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:15.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.049+0000 7ff209e6a640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:15.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.049+0000 7ff209e6a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:15.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.049+0000 7ff209e6a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:15.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.049+0000 7ff209e6a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:15.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.049+0000 7ff209e6a640 10 monclient(hunting): handle_auth_done global_id 64717 payload 824 2024-04-22T08:20:15.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.049+0000 7ff209e6a640 1 -- 172.21.15.40:0/3007730202 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff2040a0a00 msgr2=0x7ff2041c91a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:20:15.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.050+0000 7ff209e6a640 1 --2- 172.21.15.40:0/3007730202 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff2040a0a00 0x7ff2041c91a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:15.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.050+0000 7ff209e6a640 1 -- 172.21.15.40:0/3007730202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff204132990 msgr2=0x7ff2041c96e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:20:15.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.050+0000 7ff209e6a640 1 --2- 172.21.15.40:0/3007730202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff204132990 0x7ff2041c96e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:15.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.050+0000 7ff209e6a640 10 monclient: _finish_hunting 0 2024-04-22T08:20:15.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.050+0000 7ff209e6a640 1 monclient: found mon.noname-b 2024-04-22T08:20:15.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.050+0000 7ff209e6a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:20:15.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.050+0000 7ff209e6a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.189:3300/0 2024-04-22T08:20:15.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.050+0000 7ff209e6a640 1 -- 172.21.15.40:0/3007730202 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff200009400 con 0x7ff20409f0a0 2024-04-22T08:20:15.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.052+0000 7ff209e6a640 1 -- 172.21.15.40:0/3007730202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff20409f0a0 msgr2=0x7ff2041c8c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:15.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.052+0000 7ff209e6a640 1 --2- 172.21.15.40:0/3007730202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff20409f0a0 0x7ff2041c8c60 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ff2040a2af0 tx=0x7ff2000046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:15.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.052+0000 7ff1fa7fc640 1 -- 172.21.15.40:0/3007730202 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff200033b60 con 0x7ff20409f0a0 2024-04-22T08:20:15.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.052+0000 7ff1fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:15.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.052+0000 7ff1fa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:20:15.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.052+0000 7ff1fa7fc640 10 monclient: dump: 2024-04-22T08:20:15.061 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:15.061 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:15.061 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:15.061 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:15.061 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:15.061 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:15.061 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:15.061 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:15.061 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:15.061 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:15.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.052+0000 7ff1fa7fc640 10 monclient: _finish_auth 0 2024-04-22T08:20:15.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.052+0000 7ff1fa7fc640 10 monclient: _check_auth_tickets 2024-04-22T08:20:15.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.052+0000 7ff1fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:20:15.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.052+0000 7ff1fa7fc640 1 -- 172.21.15.40:0/3007730202 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff200033cc0 con 0x7ff20409f0a0 2024-04-22T08:20:15.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.052+0000 7ff1fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:20:15.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.052+0000 7ff1fa7fc640 1 -- 172.21.15.40:0/3007730202 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20003d690 con 0x7ff20409f0a0 2024-04-22T08:20:15.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.052+0000 7ff1fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:15.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.052+0000 7ff1fa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:20:15.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.052+0000 7ff1fbfff640 4 set_mon_vals no callback set 2024-04-22T08:20:15.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.052+0000 7ff1fa7fc640 10 monclient: dump: 2024-04-22T08:20:15.062 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:15.062 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:15.062 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:15.062 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:15.062 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:15.062 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:15.063 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:15.063 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:15.063 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:15.063 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:15.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.052+0000 7ff20c0ce640 5 monclient: authenticate success, global_id 64717 2024-04-22T08:20:15.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.052+0000 7ff20c0ce640 10 monclient: _renew_subs 2024-04-22T08:20:15.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.052+0000 7ff20c0ce640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.189:3300/0 2024-04-22T08:20:15.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.052+0000 7ff20c0ce640 1 -- 172.21.15.40:0/3007730202 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2040717b0 con 0x7ff20409f0a0 2024-04-22T08:20:15.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.052+0000 7ff20c0ce640 10 monclient: _renew_subs 2024-04-22T08:20:15.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.052+0000 7ff20c0ce640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.189:3300/0 2024-04-22T08:20:15.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.052+0000 7ff20c0ce640 1 -- 172.21.15.40:0/3007730202 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff204070dd0 con 0x7ff20409f0a0 2024-04-22T08:20:15.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.053+0000 7ff20c0ce640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:20:15.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.053+0000 7ff20c0ce640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:20:15.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.053+0000 7ff20c0ce640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.189:3300/0 2024-04-22T08:20:15.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.053+0000 7ff20c0ce640 1 -- 172.21.15.40:0/3007730202 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff20409f480 con 0x7ff20409f0a0 2024-04-22T08:20:15.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.053+0000 7ff1fa7fc640 1 -- 172.21.15.40:0/3007730202 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7ff20003a040 con 0x7ff20409f0a0 2024-04-22T08:20:15.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.054+0000 7ff1fa7fc640 1 --2- 172.21.15.40:0/3007730202 >> v2:172.21.15.189:6800/30168 conn(0x7ff1e80377b0 0x7ff1e8039c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:15.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.054+0000 7ff1fa7fc640 1 -- 172.21.15.40:0/3007730202 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 3506+0+0 (secure 0 0 0) 0x7ff20006bb50 con 0x7ff20409f0a0 2024-04-22T08:20:15.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.054+0000 7ff209669640 1 --2- 172.21.15.40:0/3007730202 >> v2:172.21.15.189:6800/30168 conn(0x7ff1e80377b0 0x7ff1e8039c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:15.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.054+0000 7ff209669640 10 monclient: get_auth_request con 0x7ff1e80377b0 auth_method 0 2024-04-22T08:20:15.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.056+0000 7ff209669640 1 --2- 172.21.15.40:0/3007730202 >> v2:172.21.15.189:6800/30168 conn(0x7ff1e80377b0 0x7ff1e8039c70 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff1f4007fd0 tx=0x7ff1f4007b20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:15.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.059+0000 7ff1fa7fc640 1 -- 172.21.15.40:0/3007730202 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7ff20409f480 con 0x7ff20409f0a0 2024-04-22T08:20:15.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.059+0000 7ff1fa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:20:15.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.059+0000 7ff1fa7fc640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:20:15.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.317+0000 7ff20c0ce640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:20:15.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.317+0000 7ff20c0ce640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:20:15.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.317+0000 7ff20c0ce640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.189:3300/0 2024-04-22T08:20:15.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.317+0000 7ff20c0ce640 1 -- 172.21.15.40:0/3007730202 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff20409b460 con 0x7ff20409f0a0 2024-04-22T08:20:15.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.317+0000 7ff1fa7fc640 1 -- 172.21.15.40:0/3007730202 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7ff20409b460 con 0x7ff20409f0a0 2024-04-22T08:20:15.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.317+0000 7ff1fa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:20:15.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.317+0000 7ff1fa7fc640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:20:15.317 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":72,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:20:15.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.320+0000 7ff20c0ce640 1 -- 172.21.15.40:0/3007730202 >> v2:172.21.15.189:6800/30168 conn(0x7ff1e80377b0 msgr2=0x7ff1e8039c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:15.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.320+0000 7ff20c0ce640 1 --2- 172.21.15.40:0/3007730202 >> v2:172.21.15.189:6800/30168 conn(0x7ff1e80377b0 0x7ff1e8039c70 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff1f4007fd0 tx=0x7ff1f4007b20 comp rx=0 tx=0).stop 2024-04-22T08:20:15.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.320+0000 7ff20c0ce640 10 monclient: shutdown 2024-04-22T08:20:15.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.320+0000 7ff20c0ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:20:15.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.320+0000 7ff20c0ce640 1 -- 172.21.15.40:0/3007730202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff20409f0a0 msgr2=0x7ff2041c8c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:15.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.320+0000 7ff20c0ce640 1 --2- 172.21.15.40:0/3007730202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff20409f0a0 0x7ff2041c8c60 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ff2040a2af0 tx=0x7ff2000046c0 comp rx=0 tx=0).stop 2024-04-22T08:20:15.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.320+0000 7ff20c0ce640 1 -- 172.21.15.40:0/3007730202 shutdown_connections 2024-04-22T08:20:15.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.320+0000 7ff20c0ce640 1 --2- 172.21.15.40:0/3007730202 >> v2:172.21.15.189:6800/30168 conn(0x7ff1e80377b0 0x7ff1e8039c70 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:15.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.320+0000 7ff20c0ce640 1 --2- 172.21.15.40:0/3007730202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff204132990 0x7ff2041c96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:15.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.320+0000 7ff20c0ce640 1 --2- 172.21.15.40:0/3007730202 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff2040a0a00 0x7ff2041c91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:15.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.320+0000 7ff20c0ce640 1 --2- 172.21.15.40:0/3007730202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff20409f0a0 0x7ff2041c8c60 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:15.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.320+0000 7ff20c0ce640 1 -- 172.21.15.40:0/3007730202 >> 172.21.15.40:0/3007730202 conn(0x7ff2040a5960 msgr2=0x7ff20406ba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:20:15.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.320+0000 7ff20c0ce640 1 -- 172.21.15.40:0/3007730202 shutdown_connections 2024-04-22T08:20:15.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:15.320+0000 7ff20c0ce640 1 -- 172.21.15.40:0/3007730202 wait complete. 2024-04-22T08:20:18.331 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:20:18.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:18.442+0000 7fd3ea766640 10 monclient: get_monmap_and_config 2024-04-22T08:20:18.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:18.442+0000 7fd3ea766640 10 monclient: build_initial_monmap 2024-04-22T08:20:18.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:18.442+0000 7fd3ea766640 10 monclient: monmap: 2024-04-22T08:20:18.442 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:20:18.442 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:18.442 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:20:18.442964+0000 2024-04-22T08:20:18.442 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:20:18.442964+0000 2024-04-22T08:20:18.442 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:20:18.442 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:18.442 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:20:18.442 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:20:18.442 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:20:18.442 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:20:18.443 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:20:18.443 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:18.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:18.442+0000 7fd3ea766640 1 Processor -- start 2024-04-22T08:20:18.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:18.442+0000 7fd3ea766640 1 -- start start 2024-04-22T08:20:18.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:18.442+0000 7fd3ea766640 10 monclient: init 2024-04-22T08:20:18.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:18.443+0000 7fd3ea766640 10 monclient: _reopen_session rank -1 2024-04-22T08:20:18.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:18.443+0000 7fd3ea766640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-04-22T08:20:18.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:18.443+0000 7fd3ea766640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3e409ee10 0x7fd3e40a32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:18.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:18.443+0000 7fd3ea766640 10 monclient(hunting): picked mon.noname-b con 0x7fd3e409ee10 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:20:18.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:18.443+0000 7fd3ea766640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd3e40a1d30 addr v1:172.21.15.189:6789/0 2024-04-22T08:20:18.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:18.443+0000 7fd3ea766640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd3e40a2150 addr v1:172.21.15.40:6789/0 2024-04-22T08:20:18.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:18.443+0000 7fd3ea766640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3e40a2580 con 0x7fd3e40a2150 2024-04-22T08:20:18.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:18.443+0000 7fd3ea766640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3e409f1f0 con 0x7fd3e40a1d30 2024-04-22T08:20:18.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:18.443+0000 7fd3ea766640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:18.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:18.443+0000 7fd3ea766640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e4136bb0 con 0x7fd3e409ee10 2024-04-22T08:20:18.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:18.443+0000 7fd3ea766640 10 monclient(hunting): _renew_subs 2024-04-22T08:20:18.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:18.443+0000 7fd3ea766640 10 monclient(hunting): authenticate will time out at 1292.437261s 2024-04-22T08:20:18.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:18.443+0000 7fd3e3fff640 1 -- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3e409ee10 msgr2=0x7fd3e40a32b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:20:18.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:18.443+0000 7fd3e3fff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3e409ee10 0x7fd3e40a32b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:20:18.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:18.443+0000 7fd3e8d03640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fd3e40a2150 0x7fd3e41364a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53840/0 (socket says 172.21.15.40:53840) 2024-04-22T08:20:18.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:18.443+0000 7fd3e8d03640 1 -- 172.21.15.40:0/3358357769 learned_addr learned my addr 172.21.15.40:0/3358357769 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:20:18.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:18.445+0000 7fd3e8d03640 1 -- 172.21.15.40:0/3358357769 >> v1:172.21.15.40:6789/0 conn(0x7fd3e40a2150 legacy=0x7fd3e41364a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:18.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:18.445+0000 7fd3e8d03640 1 -- 172.21.15.40:0/3358357769 >> v1:172.21.15.40:6789/0 conn(0x7fd3e40a2150 legacy=0x7fd3e41364a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-04-22T08:20:18.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:18.445+0000 7fd3e8d03640 1 -- 172.21.15.40:0/3358357769 >> v1:172.21.15.40:6789/0 conn(0x7fd3e40a2150 legacy=0x7fd3e41364a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:20:18.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:18.445+0000 7fd3e8d03640 1 --1- 172.21.15.40:0/3358357769 >> v1:172.21.15.40:6789/0 conn(0x7fd3e40a2150 0x7fd3e41364a0 :-1 s=OPENED pgs=56 cs=1 l=1).handle_message read tag failed 2024-04-22T08:20:18.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:18.445+0000 7fd3e8d03640 1 --1- 172.21.15.40:0/3358357769 >> v1:172.21.15.40:6789/0 conn(0x7fd3e40a2150 0x7fd3e41364a0 :-1 s=OPENED pgs=56 cs=1 l=1).fault on lossy channel, failing 2024-04-22T08:20:18.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:18.445+0000 7fd3e2ffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.40:6789/0 2024-04-22T08:20:18.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:18.446+0000 7fd3e37fe640 1 -- 172.21.15.40:0/3358357769 >> v1:172.21.15.189:6789/0 conn(0x7fd3e40a1d30 legacy=0x7fd3e4135d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:18.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:18.644+0000 7fd3e3fff640 1 -- 172.21.15.40:0/3358357769 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3e409ee10 msgr2=0x7fd3e40a32b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:20:18.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:18.644+0000 7fd3e3fff640 1 --2- 172.21.15.40:0/3358357769 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3e409ee10 0x7fd3e40a32b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:20:19.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:19.045+0000 7fd3e3fff640 1 -- 172.21.15.40:0/3358357769 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3e409ee10 msgr2=0x7fd3e40a32b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:20:19.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:19.045+0000 7fd3e3fff640 1 --2- 172.21.15.40:0/3358357769 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3e409ee10 0x7fd3e40a32b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:20:19.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:19.846+0000 7fd3e3fff640 1 -- 172.21.15.40:0/3358357769 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3e409ee10 msgr2=0x7fd3e40a32b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:20:19.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:19.846+0000 7fd3e3fff640 1 --2- 172.21.15.40:0/3358357769 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3e409ee10 0x7fd3e40a32b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T08:20:21.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.443+0000 7fd3e1ffb640 10 monclient(hunting): tick 2024-04-22T08:20:21.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.443+0000 7fd3e1ffb640 10 monclient(hunting): _check_auth_tickets 2024-04-22T08:20:21.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.443+0000 7fd3e1ffb640 1 monclient(hunting): continuing hunt 2024-04-22T08:20:21.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.443+0000 7fd3e1ffb640 10 monclient(hunting): _reopen_session rank -1 2024-04-22T08:20:21.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.443+0000 7fd3e1ffb640 1 -- 172.21.15.40:0/3358357769 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3e409ee10 msgr2=0x7fd3e40a32b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:20:21.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.443+0000 7fd3e1ffb640 1 --2- 172.21.15.40:0/3358357769 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3e409ee10 0x7fd3e40a32b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:21.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.443+0000 7fd3e1ffb640 1 -- 172.21.15.40:0/3358357769 >> v1:172.21.15.189:6789/0 conn(0x7fd3e40a1d30 legacy=0x7fd3e4135d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:21.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.443+0000 7fd3e1ffb640 1 -- 172.21.15.40:0/3358357769 >> v1:172.21.15.40:6789/0 conn(0x7fd3e40a2150 legacy=0x7fd3e41364a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:20:21.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.443+0000 7fd3e1ffb640 10 monclient: _add_conns ranks=[4,3] 2024-04-22T08:20:21.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.443+0000 7fd3e1ffb640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.189:3300/0 conn(0x7fd3d00036c0 0x7fd3d0005b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:21.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.443+0000 7fd3e1ffb640 10 monclient(hunting): picked mon.noname-c con 0x7fd3d00036c0 addr v2:172.21.15.189:3300/0 2024-04-22T08:20:21.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.443+0000 7fd3e1ffb640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.40:3300/0 conn(0x7fd3d0006230 0x7fd3d00086d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:21.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.443+0000 7fd3e1ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fd3d0006230 addr v2:172.21.15.40:3300/0 2024-04-22T08:20:21.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.443+0000 7fd3e1ffb640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:21.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.443+0000 7fd3e1ffb640 1 -- 172.21.15.40:0/3358357769 --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3e4136bb0 con 0x7fd3d0006230 2024-04-22T08:20:21.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.443+0000 7fd3e1ffb640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:21.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.443+0000 7fd3e1ffb640 1 -- 172.21.15.40:0/3358357769 --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3d0008d60 con 0x7fd3d00036c0 2024-04-22T08:20:21.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.443+0000 7fd3e1ffb640 10 monclient(hunting): _renew_subs 2024-04-22T08:20:21.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.444+0000 7fd3e37fe640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.40:3300/0 conn(0x7fd3d0006230 0x7fd3d00086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:21.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.444+0000 7fd3e37fe640 10 monclient(hunting): get_auth_request con 0x7fd3d0006230 auth_method 0 2024-04-22T08:20:21.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.444+0000 7fd3e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:21.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.444+0000 7fd3e37fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:21.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.444+0000 7fd3e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:21.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.444+0000 7fd3e37fe640 1 -- 172.21.15.40:0/3358357769 >> v2:172.21.15.40:3300/0 conn(0x7fd3d0006230 msgr2=0x7fd3d00086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:20:21.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.444+0000 7fd3e37fe640 1 -- 172.21.15.40:0/3358357769 >> v2:172.21.15.40:3300/0 conn(0x7fd3d0006230 msgr2=0x7fd3d00086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:20:21.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.444+0000 7fd3e37fe640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.40:3300/0 conn(0x7fd3d0006230 0x7fd3d00086d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:20:21.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.444+0000 7fd3e3fff640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.189:3300/0 conn(0x7fd3d00036c0 0x7fd3d0005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:21.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.444+0000 7fd3e3fff640 10 monclient(hunting): get_auth_request con 0x7fd3d00036c0 auth_method 0 2024-04-22T08:20:21.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.444+0000 7fd3e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:21.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.444+0000 7fd3e37fe640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.40:3300/0 conn(0x7fd3d0006230 0x7fd3d00086d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:20:21.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.444+0000 7fd3e3fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:21.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.444+0000 7fd3e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:21.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.444+0000 7fd3e3fff640 1 -- 172.21.15.40:0/3358357769 >> v2:172.21.15.189:3300/0 conn(0x7fd3d00036c0 msgr2=0x7fd3d0005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:20:21.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.444+0000 7fd3e3fff640 1 -- 172.21.15.40:0/3358357769 >> v2:172.21.15.189:3300/0 conn(0x7fd3d00036c0 msgr2=0x7fd3d0005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:20:21.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.444+0000 7fd3e3fff640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.189:3300/0 conn(0x7fd3d00036c0 0x7fd3d0005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:20:21.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.444+0000 7fd3e3fff640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.189:3300/0 conn(0x7fd3d00036c0 0x7fd3d0005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:20:21.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.645+0000 7fd3e37fe640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.40:3300/0 conn(0x7fd3d0006230 0x7fd3d00086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:21.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.645+0000 7fd3e37fe640 10 monclient(hunting): get_auth_request con 0x7fd3d0006230 auth_method 0 2024-04-22T08:20:21.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.645+0000 7fd3e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:21.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.645+0000 7fd3e37fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:21.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.645+0000 7fd3e37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:20:21.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.645+0000 7fd3e37fe640 1 -- 172.21.15.40:0/3358357769 >> v2:172.21.15.40:3300/0 conn(0x7fd3d0006230 msgr2=0x7fd3d00086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:20:21.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.645+0000 7fd3e37fe640 1 -- 172.21.15.40:0/3358357769 >> v2:172.21.15.40:3300/0 conn(0x7fd3d0006230 msgr2=0x7fd3d00086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:20:21.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.645+0000 7fd3e37fe640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.40:3300/0 conn(0x7fd3d0006230 0x7fd3d00086d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:20:21.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.645+0000 7fd3e37fe640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.40:3300/0 conn(0x7fd3d0006230 0x7fd3d00086d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:20:21.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.645+0000 7fd3e3fff640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.189:3300/0 conn(0x7fd3d00036c0 0x7fd3d0005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:21.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.645+0000 7fd3e3fff640 10 monclient(hunting): get_auth_request con 0x7fd3d00036c0 auth_method 0 2024-04-22T08:20:21.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.645+0000 7fd3e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:21.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.645+0000 7fd3e3fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:21.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.645+0000 7fd3e3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:20:21.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.645+0000 7fd3e3fff640 1 -- 172.21.15.40:0/3358357769 >> v2:172.21.15.189:3300/0 conn(0x7fd3d00036c0 msgr2=0x7fd3d0005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:20:21.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.645+0000 7fd3e3fff640 1 -- 172.21.15.40:0/3358357769 >> v2:172.21.15.189:3300/0 conn(0x7fd3d00036c0 msgr2=0x7fd3d0005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:20:21.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.645+0000 7fd3e3fff640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.189:3300/0 conn(0x7fd3d00036c0 0x7fd3d0005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:20:21.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:21.645+0000 7fd3e3fff640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.189:3300/0 conn(0x7fd3d00036c0 0x7fd3d0005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:20:22.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:22.046+0000 7fd3e37fe640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.40:3300/0 conn(0x7fd3d0006230 0x7fd3d00086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:22.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:22.046+0000 7fd3e37fe640 10 monclient(hunting): get_auth_request con 0x7fd3d0006230 auth_method 0 2024-04-22T08:20:22.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:22.046+0000 7fd3e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:22.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:22.046+0000 7fd3e37fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:22.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:22.046+0000 7fd3e37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:20:22.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:22.046+0000 7fd3e37fe640 1 -- 172.21.15.40:0/3358357769 >> v2:172.21.15.40:3300/0 conn(0x7fd3d0006230 msgr2=0x7fd3d00086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:20:22.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:22.046+0000 7fd3e37fe640 1 -- 172.21.15.40:0/3358357769 >> v2:172.21.15.40:3300/0 conn(0x7fd3d0006230 msgr2=0x7fd3d00086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:20:22.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:22.046+0000 7fd3e37fe640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.40:3300/0 conn(0x7fd3d0006230 0x7fd3d00086d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:20:22.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:22.046+0000 7fd3e37fe640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.40:3300/0 conn(0x7fd3d0006230 0x7fd3d00086d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:20:22.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:22.046+0000 7fd3e3fff640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.189:3300/0 conn(0x7fd3d00036c0 0x7fd3d0005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:22.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:22.046+0000 7fd3e3fff640 10 monclient(hunting): get_auth_request con 0x7fd3d00036c0 auth_method 0 2024-04-22T08:20:22.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:22.046+0000 7fd3e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:22.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:22.046+0000 7fd3e3fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:22.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:22.046+0000 7fd3e3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:20:22.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:22.046+0000 7fd3e3fff640 1 -- 172.21.15.40:0/3358357769 >> v2:172.21.15.189:3300/0 conn(0x7fd3d00036c0 msgr2=0x7fd3d0005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:20:22.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:22.046+0000 7fd3e3fff640 1 -- 172.21.15.40:0/3358357769 >> v2:172.21.15.189:3300/0 conn(0x7fd3d00036c0 msgr2=0x7fd3d0005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:20:22.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:22.046+0000 7fd3e3fff640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.189:3300/0 conn(0x7fd3d00036c0 0x7fd3d0005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:20:22.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:22.046+0000 7fd3e3fff640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.189:3300/0 conn(0x7fd3d00036c0 0x7fd3d0005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:20:22.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:22.847+0000 7fd3e37fe640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.40:3300/0 conn(0x7fd3d0006230 0x7fd3d00086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:22.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:22.847+0000 7fd3e37fe640 10 monclient(hunting): get_auth_request con 0x7fd3d0006230 auth_method 0 2024-04-22T08:20:22.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:22.847+0000 7fd3e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:22.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:22.847+0000 7fd3e37fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:22.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:22.847+0000 7fd3e37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:20:22.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:22.847+0000 7fd3e37fe640 1 -- 172.21.15.40:0/3358357769 >> v2:172.21.15.40:3300/0 conn(0x7fd3d0006230 msgr2=0x7fd3d00086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:20:22.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:22.847+0000 7fd3e37fe640 1 -- 172.21.15.40:0/3358357769 >> v2:172.21.15.40:3300/0 conn(0x7fd3d0006230 msgr2=0x7fd3d00086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:20:22.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:22.847+0000 7fd3e37fe640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.40:3300/0 conn(0x7fd3d0006230 0x7fd3d00086d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:20:22.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:22.847+0000 7fd3e37fe640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.40:3300/0 conn(0x7fd3d0006230 0x7fd3d00086d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T08:20:22.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:22.847+0000 7fd3e3fff640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.189:3300/0 conn(0x7fd3d00036c0 0x7fd3d0005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:22.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:22.847+0000 7fd3e3fff640 10 monclient(hunting): get_auth_request con 0x7fd3d00036c0 auth_method 0 2024-04-22T08:20:22.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:22.847+0000 7fd3e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:22.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:22.847+0000 7fd3e3fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:22.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:22.847+0000 7fd3e3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:20:22.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:22.847+0000 7fd3e3fff640 1 -- 172.21.15.40:0/3358357769 >> v2:172.21.15.189:3300/0 conn(0x7fd3d00036c0 msgr2=0x7fd3d0005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:20:22.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:22.847+0000 7fd3e3fff640 1 -- 172.21.15.40:0/3358357769 >> v2:172.21.15.189:3300/0 conn(0x7fd3d00036c0 msgr2=0x7fd3d0005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:20:22.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:22.847+0000 7fd3e3fff640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.189:3300/0 conn(0x7fd3d00036c0 0x7fd3d0005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:20:22.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:22.847+0000 7fd3e3fff640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.189:3300/0 conn(0x7fd3d00036c0 0x7fd3d0005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T08:20:24.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.444+0000 7fd3e1ffb640 10 monclient(hunting): tick 2024-04-22T08:20:24.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.444+0000 7fd3e1ffb640 10 monclient(hunting): _check_auth_tickets 2024-04-22T08:20:24.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.444+0000 7fd3e1ffb640 1 monclient(hunting): continuing hunt 2024-04-22T08:20:24.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.444+0000 7fd3e1ffb640 10 monclient(hunting): _reopen_session rank -1 2024-04-22T08:20:24.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.444+0000 7fd3e1ffb640 1 -- 172.21.15.40:0/3358357769 >> v2:172.21.15.189:3300/0 conn(0x7fd3d00036c0 msgr2=0x7fd3d0005b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:20:24.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.444+0000 7fd3e1ffb640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.189:3300/0 conn(0x7fd3d00036c0 0x7fd3d0005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:24.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.444+0000 7fd3e1ffb640 1 -- 172.21.15.40:0/3358357769 >> v2:172.21.15.40:3300/0 conn(0x7fd3d0006230 msgr2=0x7fd3d00086d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:20:24.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.444+0000 7fd3e1ffb640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.40:3300/0 conn(0x7fd3d0006230 0x7fd3d00086d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:24.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.444+0000 7fd3e8d03640 1 -- 172.21.15.40:0/3358357769 reap_dead start 2024-04-22T08:20:24.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.444+0000 7fd3e1ffb640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-04-22T08:20:24.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.444+0000 7fd3e1ffb640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.40:3300/0 conn(0x7fd3d0003b30 0x7fd3d0006610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:24.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.444+0000 7fd3e1ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fd3d0003b30 addr v2:172.21.15.40:3300/0 2024-04-22T08:20:24.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.444+0000 7fd3e1ffb640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.189:3300/0 conn(0x7fd3d0006b50 0x7fd3d0006fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:24.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.444+0000 7fd3e1ffb640 10 monclient(hunting): picked mon.noname-c con 0x7fd3d0006b50 addr v2:172.21.15.189:3300/0 2024-04-22T08:20:24.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.444+0000 7fd3e1ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd3d0007520 addr v1:172.21.15.40:6789/0 2024-04-22T08:20:24.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.444+0000 7fd3e1ffb640 1 -- 172.21.15.40:0/3358357769 --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3e40a39c0 con 0x7fd3d0007520 2024-04-22T08:20:24.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.444+0000 7fd3e1ffb640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:24.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.444+0000 7fd3e1ffb640 1 -- 172.21.15.40:0/3358357769 --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3e4136bb0 con 0x7fd3d0003b30 2024-04-22T08:20:24.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.444+0000 7fd3e1ffb640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:24.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.444+0000 7fd3e1ffb640 1 -- 172.21.15.40:0/3358357769 --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3d0008d60 con 0x7fd3d0006b50 2024-04-22T08:20:24.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.444+0000 7fd3e1ffb640 10 monclient(hunting): _renew_subs 2024-04-22T08:20:24.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.444+0000 7fd3e3fff640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.40:3300/0 conn(0x7fd3d0003b30 0x7fd3d0006610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:24.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.444+0000 7fd3e3fff640 10 monclient(hunting): get_auth_request con 0x7fd3d0003b30 auth_method 0 2024-04-22T08:20:24.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.444+0000 7fd3e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:24.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.444+0000 7fd3e3fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:24.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.444+0000 7fd3e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:24.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.444+0000 7fd3e3fff640 1 -- 172.21.15.40:0/3358357769 >> v2:172.21.15.40:3300/0 conn(0x7fd3d0003b30 msgr2=0x7fd3d0006610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:20:24.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.444+0000 7fd3e3fff640 1 -- 172.21.15.40:0/3358357769 >> v2:172.21.15.40:3300/0 conn(0x7fd3d0003b30 msgr2=0x7fd3d0006610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:20:24.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.444+0000 7fd3e3fff640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.40:3300/0 conn(0x7fd3d0003b30 0x7fd3d0006610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:20:24.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.444+0000 7fd3e3fff640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.40:3300/0 conn(0x7fd3d0003b30 0x7fd3d0006610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:20:24.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.444+0000 7fd3e37fe640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.189:3300/0 conn(0x7fd3d0006b50 0x7fd3d0006fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:24.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.444+0000 7fd3e37fe640 10 monclient(hunting): get_auth_request con 0x7fd3d0006b50 auth_method 0 2024-04-22T08:20:24.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.444+0000 7fd3e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:24.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.444+0000 7fd3e37fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:24.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.444+0000 7fd3e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:24.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.444+0000 7fd3e37fe640 1 -- 172.21.15.40:0/3358357769 >> v2:172.21.15.189:3300/0 conn(0x7fd3d0006b50 msgr2=0x7fd3d0006fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:20:24.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.444+0000 7fd3e37fe640 1 -- 172.21.15.40:0/3358357769 >> v2:172.21.15.189:3300/0 conn(0x7fd3d0006b50 msgr2=0x7fd3d0006fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:20:24.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.444+0000 7fd3e37fe640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.189:3300/0 conn(0x7fd3d0006b50 0x7fd3d0006fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:20:24.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.444+0000 7fd3e37fe640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.189:3300/0 conn(0x7fd3d0006b50 0x7fd3d0006fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:20:24.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.446+0000 7fd3e8d03640 1 -- 172.21.15.40:0/3358357769 >> v1:172.21.15.40:6789/0 conn(0x7fd3d0007520 legacy=0x7fd3d00079d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:24.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.645+0000 7fd3e3fff640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.40:3300/0 conn(0x7fd3d0003b30 0x7fd3d0006610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:24.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.645+0000 7fd3e3fff640 10 monclient(hunting): get_auth_request con 0x7fd3d0003b30 auth_method 0 2024-04-22T08:20:24.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.645+0000 7fd3e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:24.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.645+0000 7fd3e3fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:24.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.645+0000 7fd3e3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:20:24.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.645+0000 7fd3e3fff640 1 -- 172.21.15.40:0/3358357769 >> v2:172.21.15.40:3300/0 conn(0x7fd3d0003b30 msgr2=0x7fd3d0006610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:20:24.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.645+0000 7fd3e3fff640 1 -- 172.21.15.40:0/3358357769 >> v2:172.21.15.40:3300/0 conn(0x7fd3d0003b30 msgr2=0x7fd3d0006610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:20:24.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.645+0000 7fd3e3fff640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.40:3300/0 conn(0x7fd3d0003b30 0x7fd3d0006610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:20:24.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.645+0000 7fd3e3fff640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.40:3300/0 conn(0x7fd3d0003b30 0x7fd3d0006610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:20:24.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.645+0000 7fd3e37fe640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.189:3300/0 conn(0x7fd3d0006b50 0x7fd3d0006fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:24.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.645+0000 7fd3e37fe640 10 monclient(hunting): get_auth_request con 0x7fd3d0006b50 auth_method 0 2024-04-22T08:20:24.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.645+0000 7fd3e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:24.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.645+0000 7fd3e37fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:24.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.645+0000 7fd3e37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:20:24.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.645+0000 7fd3e37fe640 1 -- 172.21.15.40:0/3358357769 >> v2:172.21.15.189:3300/0 conn(0x7fd3d0006b50 msgr2=0x7fd3d0006fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:20:24.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.645+0000 7fd3e37fe640 1 -- 172.21.15.40:0/3358357769 >> v2:172.21.15.189:3300/0 conn(0x7fd3d0006b50 msgr2=0x7fd3d0006fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:20:24.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.645+0000 7fd3e37fe640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.189:3300/0 conn(0x7fd3d0006b50 0x7fd3d0006fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:20:24.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:24.645+0000 7fd3e37fe640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.189:3300/0 conn(0x7fd3d0006b50 0x7fd3d0006fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:20:25.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:25.046+0000 7fd3e3fff640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.40:3300/0 conn(0x7fd3d0003b30 0x7fd3d0006610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:25.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:25.046+0000 7fd3e3fff640 10 monclient(hunting): get_auth_request con 0x7fd3d0003b30 auth_method 0 2024-04-22T08:20:25.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:25.046+0000 7fd3e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:25.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:25.046+0000 7fd3e3fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:25.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:25.046+0000 7fd3e3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:20:25.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:25.046+0000 7fd3e3fff640 1 -- 172.21.15.40:0/3358357769 >> v2:172.21.15.40:3300/0 conn(0x7fd3d0003b30 msgr2=0x7fd3d0006610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:20:25.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:25.046+0000 7fd3e3fff640 1 -- 172.21.15.40:0/3358357769 >> v2:172.21.15.40:3300/0 conn(0x7fd3d0003b30 msgr2=0x7fd3d0006610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:20:25.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:25.046+0000 7fd3e3fff640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.40:3300/0 conn(0x7fd3d0003b30 0x7fd3d0006610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:20:25.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:25.046+0000 7fd3e3fff640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.40:3300/0 conn(0x7fd3d0003b30 0x7fd3d0006610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:20:25.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:25.046+0000 7fd3e37fe640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.189:3300/0 conn(0x7fd3d0006b50 0x7fd3d0006fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:25.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:25.046+0000 7fd3e37fe640 10 monclient(hunting): get_auth_request con 0x7fd3d0006b50 auth_method 0 2024-04-22T08:20:25.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:25.046+0000 7fd3e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:25.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:25.046+0000 7fd3e37fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:25.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:25.046+0000 7fd3e37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:20:25.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:25.046+0000 7fd3e37fe640 1 -- 172.21.15.40:0/3358357769 >> v2:172.21.15.189:3300/0 conn(0x7fd3d0006b50 msgr2=0x7fd3d0006fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:20:25.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:25.046+0000 7fd3e37fe640 1 -- 172.21.15.40:0/3358357769 >> v2:172.21.15.189:3300/0 conn(0x7fd3d0006b50 msgr2=0x7fd3d0006fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:20:25.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:25.046+0000 7fd3e37fe640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.189:3300/0 conn(0x7fd3d0006b50 0x7fd3d0006fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:20:25.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:25.046+0000 7fd3e37fe640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.189:3300/0 conn(0x7fd3d0006b50 0x7fd3d0006fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:20:25.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:25.847+0000 7fd3e3fff640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.40:3300/0 conn(0x7fd3d0003b30 0x7fd3d0006610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:25.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:25.847+0000 7fd3e3fff640 10 monclient(hunting): get_auth_request con 0x7fd3d0003b30 auth_method 0 2024-04-22T08:20:25.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:25.847+0000 7fd3e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:25.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:25.847+0000 7fd3e3fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:25.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:25.847+0000 7fd3e3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:20:25.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:25.847+0000 7fd3e3fff640 1 -- 172.21.15.40:0/3358357769 >> v2:172.21.15.40:3300/0 conn(0x7fd3d0003b30 msgr2=0x7fd3d0006610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:20:25.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:25.847+0000 7fd3e3fff640 1 -- 172.21.15.40:0/3358357769 >> v2:172.21.15.40:3300/0 conn(0x7fd3d0003b30 msgr2=0x7fd3d0006610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:20:25.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:25.847+0000 7fd3e3fff640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.40:3300/0 conn(0x7fd3d0003b30 0x7fd3d0006610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:20:25.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:25.847+0000 7fd3e3fff640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.40:3300/0 conn(0x7fd3d0003b30 0x7fd3d0006610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T08:20:25.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:25.847+0000 7fd3e37fe640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.189:3300/0 conn(0x7fd3d0006b50 0x7fd3d0006fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:25.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:25.847+0000 7fd3e37fe640 10 monclient(hunting): get_auth_request con 0x7fd3d0006b50 auth_method 0 2024-04-22T08:20:25.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:25.847+0000 7fd3e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:25.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:25.847+0000 7fd3e37fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:25.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:25.847+0000 7fd3e37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:20:25.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:25.848+0000 7fd3e37fe640 1 -- 172.21.15.40:0/3358357769 >> v2:172.21.15.189:3300/0 conn(0x7fd3d0006b50 msgr2=0x7fd3d0006fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:20:25.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:25.848+0000 7fd3e37fe640 1 -- 172.21.15.40:0/3358357769 >> v2:172.21.15.189:3300/0 conn(0x7fd3d0006b50 msgr2=0x7fd3d0006fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:20:25.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:25.848+0000 7fd3e37fe640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.189:3300/0 conn(0x7fd3d0006b50 0x7fd3d0006fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:20:25.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:25.848+0000 7fd3e37fe640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.189:3300/0 conn(0x7fd3d0006b50 0x7fd3d0006fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T08:20:26.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.164+0000 7fd3e2ffd640 1 -- 172.21.15.40:0/3358357769 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 334305443 0 0) 0x7fd3e40a39c0 con 0x7fd3d0007520 2024-04-22T08:20:26.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.164+0000 7fd3e2ffd640 10 monclient(hunting): _negotiate 2024-04-22T08:20:26.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.164+0000 7fd3e2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:26.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.164+0000 7fd3e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:26.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.164+0000 7fd3e2ffd640 10 monclient(hunting): my global_id is 94173 2024-04-22T08:20:26.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.164+0000 7fd3e2ffd640 1 -- 172.21.15.40:0/3358357769 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3cc003520 con 0x7fd3d0007520 2024-04-22T08:20:26.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e2ffd640 1 -- 172.21.15.40:0/3358357769 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 45565151 0 0) 0x7fd3cc003520 con 0x7fd3d0007520 2024-04-22T08:20:26.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e2ffd640 1 -- 172.21.15.40:0/3358357769 >> v2:172.21.15.189:3300/0 conn(0x7fd3d0006b50 msgr2=0x7fd3d0006fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:20:26.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e2ffd640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.189:3300/0 conn(0x7fd3d0006b50 0x7fd3d0006fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:26.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e2ffd640 1 -- 172.21.15.40:0/3358357769 >> v2:172.21.15.40:3300/0 conn(0x7fd3d0003b30 msgr2=0x7fd3d0006610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:20:26.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e2ffd640 1 --2- 172.21.15.40:0/3358357769 >> v2:172.21.15.40:3300/0 conn(0x7fd3d0003b30 0x7fd3d0006610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:26.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e2ffd640 10 monclient: _finish_hunting 0 2024-04-22T08:20:26.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e2ffd640 1 monclient: found mon.noname-a-legacy 2024-04-22T08:20:26.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:20:26.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e2ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.40:6789/0 2024-04-22T08:20:26.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e2ffd640 1 -- 172.21.15.40:0/3358357769 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3d0008350 con 0x7fd3d0007520 2024-04-22T08:20:26.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e2ffd640 10 monclient: _finish_auth 0 2024-04-22T08:20:26.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e2ffd640 10 monclient: _check_auth_tickets 2024-04-22T08:20:26.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:20:26.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e2ffd640 1 -- 172.21.15.40:0/3358357769 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1087006497 0 0) 0x7fd3dc001ce0 con 0x7fd3d0007520 2024-04-22T08:20:26.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3ea766640 5 monclient: authenticate success, global_id 94173 2024-04-22T08:20:26.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3ea766640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:20:26.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:26.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e2ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-22T08:20:26.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e2ffd640 10 monclient: dump: 2024-04-22T08:20:26.231 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:26.231 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:26.231 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:26.231 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:26.231 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:26.231 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:26.231 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:26.231 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:26.231 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:26.231 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:26.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e2ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] but i'm connected to v1:172.21.15.40:6789/0, reconnecting 2024-04-22T08:20:26.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e2ffd640 10 monclient: _reopen_session rank -1 2024-04-22T08:20:26.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e2ffd640 1 -- 172.21.15.40:0/3358357769 >> v1:172.21.15.40:6789/0 conn(0x7fd3d0007520 legacy=0x7fd3d00079d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:26.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e2ffd640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-22T08:20:26.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e2ffd640 1 --2- 172.21.15.40:0/3358357769 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3cc003970 0x7fd3cc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:26.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e2ffd640 10 monclient(hunting): picked mon.c con 0x7fd3cc003970 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:20:26.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e2ffd640 1 --2- 172.21.15.40:0/3358357769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3cc0047c0 0x7fd3cc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:26.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e2ffd640 10 monclient(hunting): picked mon.a con 0x7fd3cc0047c0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:20:26.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e2ffd640 1 --2- 172.21.15.40:0/3358357769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3cc005180 0x7fd3cc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:26.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e3fff640 1 -- 172.21.15.40:0/3358357769 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3cc003970 msgr2=0x7fd3cc003d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:20:26.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e2ffd640 10 monclient(hunting): picked mon.b con 0x7fd3cc005180 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:20:26.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e2ffd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:26.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e2ffd640 1 -- 172.21.15.40:0/3358357769 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e4136bb0 con 0x7fd3cc0047c0 2024-04-22T08:20:26.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e2ffd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:26.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e3fff640 1 --2- 172.21.15.40:0/3358357769 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3cc003970 0x7fd3cc003d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:20:26.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e2ffd640 1 -- 172.21.15.40:0/3358357769 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d0008d60 con 0x7fd3cc005180 2024-04-22T08:20:26.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e2ffd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:26.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e2ffd640 1 -- 172.21.15.40:0/3358357769 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3cc00c0d0 con 0x7fd3cc003970 2024-04-22T08:20:26.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e2ffd640 10 monclient(hunting): _renew_subs 2024-04-22T08:20:26.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e2ffd640 10 monclient(hunting): _finish_auth 0 2024-04-22T08:20:26.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3ea766640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-04-22T08:20:26.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e37fe640 1 --2- 172.21.15.40:0/3358357769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3cc0047c0 0x7fd3cc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:26.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e37fe640 10 monclient(hunting): get_auth_request con 0x7fd3cc0047c0 auth_method 0 2024-04-22T08:20:26.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:26.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e37fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:26.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:20:26.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e8d03640 1 --2- 172.21.15.40:0/3358357769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3cc005180 0x7fd3cc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:26.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e8d03640 10 monclient(hunting): get_auth_request con 0x7fd3cc005180 auth_method 0 2024-04-22T08:20:26.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e8d03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:26.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e8d03640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:26.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e8d03640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:20:26.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:26.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:26.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e37fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-22T08:20:26.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e8d03640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:26.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e8d03640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:26.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e8d03640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-22T08:20:26.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e37fe640 10 monclient(hunting): handle_auth_done global_id 94173 payload 293 2024-04-22T08:20:26.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e37fe640 1 -- 172.21.15.40:0/3358357769 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3cc003970 msgr2=0x7fd3cc003d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:20:26.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.228+0000 7fd3e37fe640 1 --2- 172.21.15.40:0/3358357769 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3cc003970 0x7fd3cc003d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:26.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.229+0000 7fd3e37fe640 1 -- 172.21.15.40:0/3358357769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3cc005180 msgr2=0x7fd3cc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:20:26.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.229+0000 7fd3e37fe640 1 --2- 172.21.15.40:0/3358357769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3cc005180 0x7fd3cc00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:26.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.229+0000 7fd3e37fe640 10 monclient: _finish_hunting 0 2024-04-22T08:20:26.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.229+0000 7fd3e37fe640 1 monclient: found mon.a 2024-04-22T08:20:26.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.229+0000 7fd3e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:20:26.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.229+0000 7fd3e8d03640 1 -- 172.21.15.40:0/3358357769 reap_dead start 2024-04-22T08:20:26.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.229+0000 7fd3e37fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:20:26.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.229+0000 7fd3e37fe640 1 -- 172.21.15.40:0/3358357769 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3cc003520 con 0x7fd3cc0047c0 2024-04-22T08:20:26.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.229+0000 7fd3e37fe640 10 monclient: _finish_auth 0 2024-04-22T08:20:26.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.229+0000 7fd3e37fe640 10 monclient: _check_auth_tickets 2024-04-22T08:20:26.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.229+0000 7fd3e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:20:26.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.232+0000 7fd3e37fe640 1 -- 172.21.15.40:0/3358357769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3cc0047c0 msgr2=0x7fd3cc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:26.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.232+0000 7fd3e37fe640 1 --2- 172.21.15.40:0/3358357769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3cc0047c0 0x7fd3cc004c40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd3d40073e0 tx=0x7fd3cc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=342f9ca2f2eb68f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:26.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.232+0000 7fd3e2ffd640 1 -- 172.21.15.40:0/3358357769 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d402d6c0 con 0x7fd3cc0047c0 2024-04-22T08:20:26.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.232+0000 7fd3e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:26.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.232+0000 7fd3e2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:20:26.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.232+0000 7fd3e2ffd640 10 monclient: dump: 2024-04-22T08:20:26.241 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:26.242 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:26.242 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:26.242 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:26.242 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:26.242 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:26.242 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:26.242 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:26.242 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:26.242 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:26.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.232+0000 7fd3ea766640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:20:26.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.233+0000 7fd3e2ffd640 1 -- 172.21.15.40:0/3358357769 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3d4031070 con 0x7fd3cc0047c0 2024-04-22T08:20:26.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.233+0000 7fd3e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:20:26.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.233+0000 7fd3e2ffd640 1 -- 172.21.15.40:0/3358357769 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d402da60 con 0x7fd3cc0047c0 2024-04-22T08:20:26.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.233+0000 7fd3e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:26.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.233+0000 7fd3e2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:20:26.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.233+0000 7fd3e2ffd640 10 monclient: dump: 2024-04-22T08:20:26.243 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:26.243 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:26.243 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:26.243 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:26.243 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:26.243 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:26.243 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:26.243 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:26.243 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:26.243 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:26.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.233+0000 7fd3ea766640 10 monclient: get_monmap_and_config success 2024-04-22T08:20:26.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.233+0000 7fd3ea766640 4 set_mon_vals no callback set 2024-04-22T08:20:26.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.233+0000 7fd3ea766640 10 monclient: shutdown 2024-04-22T08:20:26.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.233+0000 7fd3ea766640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:20:26.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.233+0000 7fd3ea766640 1 -- 172.21.15.40:0/3358357769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3cc0047c0 msgr2=0x7fd3cc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:26.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.233+0000 7fd3ea766640 1 --2- 172.21.15.40:0/3358357769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3cc0047c0 0x7fd3cc004c40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd3d40073e0 tx=0x7fd3cc0057f0 comp rx=0 tx=0).stop 2024-04-22T08:20:26.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.233+0000 7fd3ea766640 1 -- 172.21.15.40:0/3358357769 shutdown_connections 2024-04-22T08:20:26.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.233+0000 7fd3ea766640 1 --2- 172.21.15.40:0/3358357769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3cc0047c0 0x7fd3cc004c40 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:26.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.233+0000 7fd3ea766640 1 -- 172.21.15.40:0/3358357769 >> 172.21.15.40:0/3358357769 conn(0x7fd3e40a4ef0 msgr2=0x7fd3e40a5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:20:26.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.233+0000 7fd3ea766640 1 -- 172.21.15.40:0/3358357769 shutdown_connections 2024-04-22T08:20:26.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.233+0000 7fd3ea766640 1 -- 172.21.15.40:0/3358357769 wait complete. 2024-04-22T08:20:26.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.234+0000 7fd3ea766640 10 monclient: build_initial_monmap 2024-04-22T08:20:26.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.234+0000 7fd3ea766640 10 monclient: monmap: 2024-04-22T08:20:26.244 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:20:26.245 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:20:26.245 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:20:26.245 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:20:26.245 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:20:26.245 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:26.245 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:20:26.245 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:20:26.245 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:20:26.245 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:26.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.235+0000 7fd3ea766640 1 Processor -- start 2024-04-22T08:20:26.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.235+0000 7fd3ea766640 1 -- start start 2024-04-22T08:20:26.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.235+0000 7fd3ea766640 10 monclient: init 2024-04-22T08:20:26.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.235+0000 7fd3ea766640 10 monclient: _reopen_session rank -1 2024-04-22T08:20:26.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.235+0000 7fd3ea766640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-22T08:20:26.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.235+0000 7fd3ea766640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3cc0047c0 0x7fd3e41cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:26.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.235+0000 7fd3ea766640 10 monclient(hunting): picked mon.noname-c con 0x7fd3cc0047c0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:20:26.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.235+0000 7fd3ea766640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3e41cb350 0x7fd3e41211a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:26.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.235+0000 7fd3ea766640 10 monclient(hunting): picked mon.noname-b con 0x7fd3e41cb350 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:20:26.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.235+0000 7fd3ea766640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3e41cb730 0x7fd3e41216e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:26.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.235+0000 7fd3ea766640 10 monclient(hunting): picked mon.noname-a con 0x7fd3e41cb730 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:20:26.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.235+0000 7fd3ea766640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:26.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.235+0000 7fd3ea766640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e4124e80 con 0x7fd3e41cb730 2024-04-22T08:20:26.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.235+0000 7fd3ea766640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:26.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.235+0000 7fd3ea766640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e41250d0 con 0x7fd3e41cb350 2024-04-22T08:20:26.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.235+0000 7fd3ea766640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:26.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.235+0000 7fd3ea766640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e40a2730 con 0x7fd3cc0047c0 2024-04-22T08:20:26.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.235+0000 7fd3ea766640 10 monclient(hunting): _renew_subs 2024-04-22T08:20:26.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.235+0000 7fd3ea766640 10 monclient(hunting): authenticate will time out at 1300.229535s 2024-04-22T08:20:26.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.235+0000 7fd3e3fff640 1 -- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3cc0047c0 msgr2=0x7fd3e41cae10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:20:26.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.235+0000 7fd3e3fff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3cc0047c0 0x7fd3e41cae10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:20:26.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.235+0000 7fd3e8d03640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3e41cb730 0x7fd3e41216e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:26.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.235+0000 7fd3e8d03640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3e41cb730 0x7fd3e41216e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58662/0 (socket says 172.21.15.40:58662) 2024-04-22T08:20:26.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.235+0000 7fd3e8d03640 1 -- 172.21.15.40:0/1722712926 learned_addr learned my addr 172.21.15.40:0/1722712926 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:20:26.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.235+0000 7fd3e8d03640 10 monclient(hunting): get_auth_request con 0x7fd3e41cb730 auth_method 0 2024-04-22T08:20:26.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.235+0000 7fd3e8d03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:26.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.235+0000 7fd3e8d03640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:26.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.235+0000 7fd3e8d03640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:26.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.236+0000 7fd3e8d03640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:26.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.236+0000 7fd3e8d03640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:26.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.236+0000 7fd3e8d03640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:26.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.236+0000 7fd3e37fe640 1 --2- 172.21.15.40:0/1722712926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3e41cb350 0x7fd3e41211a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:26.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.236+0000 7fd3e37fe640 10 monclient(hunting): get_auth_request con 0x7fd3e41cb350 auth_method 0 2024-04-22T08:20:26.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.236+0000 7fd3e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:26.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.236+0000 7fd3e37fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:26.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.236+0000 7fd3e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:26.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.236+0000 7fd3e8d03640 10 monclient(hunting): handle_auth_done global_id 94176 payload 824 2024-04-22T08:20:26.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.236+0000 7fd3e8d03640 1 -- 172.21.15.40:0/1722712926 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3cc0047c0 msgr2=0x7fd3e41cae10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:20:26.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.236+0000 7fd3e8d03640 1 --2- 172.21.15.40:0/1722712926 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3cc0047c0 0x7fd3e41cae10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:26.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.236+0000 7fd3e8d03640 1 -- 172.21.15.40:0/1722712926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3e41cb350 msgr2=0x7fd3e41211a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:26.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.236+0000 7fd3e8d03640 1 --2- 172.21.15.40:0/1722712926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3e41cb350 0x7fd3e41211a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:26.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.236+0000 7fd3e8d03640 10 monclient: _finish_hunting 0 2024-04-22T08:20:26.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.236+0000 7fd3e8d03640 1 monclient: found mon.noname-a 2024-04-22T08:20:26.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.236+0000 7fd3e8d03640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:20:26.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.236+0000 7fd3e8d03640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:20:26.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.236+0000 7fd3e8d03640 1 -- 172.21.15.40:0/1722712926 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e41238d0 con 0x7fd3e41cb730 2024-04-22T08:20:26.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.238+0000 7fd3e8d03640 1 -- 172.21.15.40:0/1722712926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3e41cb730 msgr2=0x7fd3e41216e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:26.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.238+0000 7fd3e8d03640 1 --2- 172.21.15.40:0/1722712926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3e41cb730 0x7fd3e41216e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd3e406c050 tx=0x7fd3dc007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:26.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.238+0000 7fd3e0ff9640 1 -- 172.21.15.40:0/1722712926 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc00a880 con 0x7fd3e41cb730 2024-04-22T08:20:26.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.238+0000 7fd3e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:26.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.238+0000 7fd3e0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:20:26.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.238+0000 7fd3e0ff9640 10 monclient: dump: 2024-04-22T08:20:26.249 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:26.249 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:26.249 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:26.249 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:26.249 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:26.249 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:26.249 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:26.249 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:26.250 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:26.250 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:26.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.238+0000 7fd3e0ff9640 10 monclient: _finish_auth 0 2024-04-22T08:20:26.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.238+0000 7fd3e0ff9640 10 monclient: _check_auth_tickets 2024-04-22T08:20:26.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.238+0000 7fd3e0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:20:26.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.238+0000 7fd3ea766640 5 monclient: authenticate success, global_id 94176 2024-04-22T08:20:26.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.238+0000 7fd3e0ff9640 1 -- 172.21.15.40:0/1722712926 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3dc00aa00 con 0x7fd3e41cb730 2024-04-22T08:20:26.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.238+0000 7fd3ea766640 10 monclient: _renew_subs 2024-04-22T08:20:26.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.238+0000 7fd3ea766640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:20:26.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.238+0000 7fd3ea766640 1 -- 172.21.15.40:0/1722712926 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3e4135fe0 con 0x7fd3e41cb730 2024-04-22T08:20:26.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.238+0000 7fd3e0ff9640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:20:26.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.238+0000 7fd3e0ff9640 1 -- 172.21.15.40:0/1722712926 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc00adb0 con 0x7fd3e41cb730 2024-04-22T08:20:26.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.238+0000 7fd3e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:26.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.238+0000 7fd3e0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:20:26.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.238+0000 7fd3e0ff9640 10 monclient: dump: 2024-04-22T08:20:26.251 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:26.251 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:26.251 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:26.251 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:26.251 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:26.251 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:26.251 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:26.251 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:26.251 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:26.251 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:26.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.238+0000 7fd3ea766640 10 monclient: _renew_subs 2024-04-22T08:20:26.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.238+0000 7fd3ea766640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:20:26.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.238+0000 7fd3ea766640 1 -- 172.21.15.40:0/1722712926 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3e41254f0 con 0x7fd3e41cb730 2024-04-22T08:20:26.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.238+0000 7fd3e27fc640 4 set_mon_vals no callback set 2024-04-22T08:20:26.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.239+0000 7fd3e0ff9640 1 -- 172.21.15.40:0/1722712926 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fd3dc01e460 con 0x7fd3e41cb730 2024-04-22T08:20:26.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.239+0000 7fd3ea766640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:20:26.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.239+0000 7fd3ea766640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:20:26.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.239+0000 7fd3ea766640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:20:26.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.239+0000 7fd3ea766640 1 -- 172.21.15.40:0/1722712926 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3e4124950 con 0x7fd3e41cb730 2024-04-22T08:20:26.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.240+0000 7fd3e0ff9640 1 --2- 172.21.15.40:0/1722712926 >> v2:172.21.15.189:6800/30168 conn(0x7fd3d0038da0 0x7fd3d003b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:26.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.240+0000 7fd3e0ff9640 1 -- 172.21.15.40:0/1722712926 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 3506+0+0 (secure 0 0 0) 0x7fd3dc04a520 con 0x7fd3e41cb730 2024-04-22T08:20:26.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.240+0000 7fd3e3fff640 1 --2- 172.21.15.40:0/1722712926 >> v2:172.21.15.189:6800/30168 conn(0x7fd3d0038da0 0x7fd3d003b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:26.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.240+0000 7fd3e3fff640 10 monclient: get_auth_request con 0x7fd3d0038da0 auth_method 0 2024-04-22T08:20:26.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.243+0000 7fd3e3fff640 1 --2- 172.21.15.40:0/1722712926 >> v2:172.21.15.189:6800/30168 conn(0x7fd3d0038da0 0x7fd3d003b260 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fd3d8003980 tx=0x7fd3d800d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:26.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.245+0000 7fd3e0ff9640 1 -- 172.21.15.40:0/1722712926 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7fd3dc01b6d0 con 0x7fd3e41cb730 2024-04-22T08:20:26.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.245+0000 7fd3e0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:20:26.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.245+0000 7fd3e0ff9640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:20:26.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.493+0000 7fd3ea766640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:20:26.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.493+0000 7fd3ea766640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:20:26.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.493+0000 7fd3ea766640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:20:26.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.493+0000 7fd3ea766640 1 -- 172.21.15.40:0/1722712926 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd3e404e9b0 con 0x7fd3e41cb730 2024-04-22T08:20:26.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.494+0000 7fd3e0ff9640 1 -- 172.21.15.40:0/1722712926 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1537 (secure 0 0 0) 0x7fd3dc012350 con 0x7fd3e41cb730 2024-04-22T08:20:26.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.494+0000 7fd3e0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:20:26.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.494+0000 7fd3e0ff9640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:20:26.494 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":76,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:20:26.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.496+0000 7fd3ea766640 1 -- 172.21.15.40:0/1722712926 >> v2:172.21.15.189:6800/30168 conn(0x7fd3d0038da0 msgr2=0x7fd3d003b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:26.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.496+0000 7fd3ea766640 1 --2- 172.21.15.40:0/1722712926 >> v2:172.21.15.189:6800/30168 conn(0x7fd3d0038da0 0x7fd3d003b260 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fd3d8003980 tx=0x7fd3d800d040 comp rx=0 tx=0).stop 2024-04-22T08:20:26.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.496+0000 7fd3ea766640 10 monclient: shutdown 2024-04-22T08:20:26.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.496+0000 7fd3ea766640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:20:26.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.496+0000 7fd3ea766640 1 -- 172.21.15.40:0/1722712926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3e41cb730 msgr2=0x7fd3e41216e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:26.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.496+0000 7fd3ea766640 1 --2- 172.21.15.40:0/1722712926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3e41cb730 0x7fd3e41216e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd3e406c050 tx=0x7fd3dc007720 comp rx=0 tx=0).stop 2024-04-22T08:20:26.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.496+0000 7fd3ea766640 1 -- 172.21.15.40:0/1722712926 shutdown_connections 2024-04-22T08:20:26.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.496+0000 7fd3ea766640 1 --2- 172.21.15.40:0/1722712926 >> v2:172.21.15.189:6800/30168 conn(0x7fd3d0038da0 0x7fd3d003b260 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:26.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.496+0000 7fd3ea766640 1 --2- 172.21.15.40:0/1722712926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3e41cb350 0x7fd3e41211a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:26.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.496+0000 7fd3ea766640 1 --2- 172.21.15.40:0/1722712926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3e41cb730 0x7fd3e41216e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:26.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.496+0000 7fd3ea766640 1 --2- 172.21.15.40:0/1722712926 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3cc0047c0 0x7fd3e41cae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:26.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.496+0000 7fd3ea766640 1 -- 172.21.15.40:0/1722712926 >> 172.21.15.40:0/1722712926 conn(0x7fd3e40a4ef0 msgr2=0x7fd3e4135490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:20:26.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.496+0000 7fd3ea766640 1 -- 172.21.15.40:0/1722712926 shutdown_connections 2024-04-22T08:20:26.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.496+0000 7fd3ea766640 1 -- 172.21.15.40:0/1722712926 wait complete. 2024-04-22T08:20:26.506 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-22T08:20:26.506 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-04-22T08:20:26.506 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-22T08:20:26.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.606+0000 7f644e7d7640 10 monclient: get_monmap_and_config 2024-04-22T08:20:26.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.606+0000 7f644e7d7640 10 monclient: build_initial_monmap 2024-04-22T08:20:26.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.606+0000 7f644e7d7640 10 monclient: monmap: 2024-04-22T08:20:26.606 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:20:26.606 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:26.606 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:20:26.607502+0000 2024-04-22T08:20:26.606 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:20:26.607502+0000 2024-04-22T08:20:26.606 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:20:26.606 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:26.607 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:20:26.607 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:20:26.607 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:20:26.607 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:20:26.607 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:20:26.607 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:26.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.607+0000 7f644e7d7640 1 Processor -- start 2024-04-22T08:20:26.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.607+0000 7f644e7d7640 1 -- start start 2024-04-22T08:20:26.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.607+0000 7f644e7d7640 10 monclient: init 2024-04-22T08:20:26.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.607+0000 7f644e7d7640 10 monclient: _reopen_session rank -1 2024-04-22T08:20:26.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.607+0000 7f644e7d7640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-04-22T08:20:26.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.607+0000 7f644e7d7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f644813a300 addr v1:172.21.15.189:6789/0 2024-04-22T08:20:26.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.607+0000 7f644e7d7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f644809cbd0 addr v1:172.21.15.40:6789/0 2024-04-22T08:20:26.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.607+0000 7f644e7d7640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f64480a0620 0x7f6448142990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:26.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.607+0000 7f644e7d7640 10 monclient(hunting): picked mon.noname-b con 0x7f64480a0620 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:20:26.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.607+0000 7f644e7d7640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64480a0190 con 0x7f644809cbd0 2024-04-22T08:20:26.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.607+0000 7f644e7d7640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6448071980 con 0x7f644813a300 2024-04-22T08:20:26.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.607+0000 7f644e7d7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:26.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.607+0000 7f644e7d7640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f644809c2f0 con 0x7f64480a0620 2024-04-22T08:20:26.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.607+0000 7f644e7d7640 10 monclient(hunting): _renew_subs 2024-04-22T08:20:26.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.607+0000 7f644e7d7640 10 monclient(hunting): authenticate will time out at 1300.601717s 2024-04-22T08:20:26.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.607+0000 7f644cd74640 1 -- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f64480a0620 msgr2=0x7f6448142990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:20:26.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.608+0000 7f644cd74640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f64480a0620 0x7f6448142990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:20:26.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.608+0000 7f64477fe640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f644809cbd0 0x7f644813f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:54018/0 (socket says 172.21.15.40:54018) 2024-04-22T08:20:26.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.608+0000 7f64477fe640 1 -- 172.21.15.40:0/2023127710 learned_addr learned my addr 172.21.15.40:0/2023127710 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:20:26.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f64477fe640 1 -- 172.21.15.40:0/2023127710 >> v1:172.21.15.40:6789/0 conn(0x7f644809cbd0 legacy=0x7f644813f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:26.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6447fff640 1 -- 172.21.15.40:0/2023127710 >> v1:172.21.15.189:6789/0 conn(0x7f644813a300 legacy=0x7f644813a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:26.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6446ffd640 1 -- 172.21.15.40:0/2023127710 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 215094881 0 0) 0x7f64480a0190 con 0x7f644809cbd0 2024-04-22T08:20:26.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6446ffd640 10 monclient(hunting): _negotiate 2024-04-22T08:20:26.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6446ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:26.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6446ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:26.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6446ffd640 10 monclient(hunting): my global_id is 94179 2024-04-22T08:20:26.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6446ffd640 1 -- 172.21.15.40:0/2023127710 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6430003520 con 0x7f644809cbd0 2024-04-22T08:20:26.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6446ffd640 1 -- 172.21.15.40:0/2023127710 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2213040352 0 0) 0x7f6448071980 con 0x7f644813a300 2024-04-22T08:20:26.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6446ffd640 10 monclient(hunting): _negotiate 2024-04-22T08:20:26.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6446ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:26.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6446ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:26.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6446ffd640 10 monclient(hunting): my global_id is 64723 2024-04-22T08:20:26.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6446ffd640 1 -- 172.21.15.40:0/2023127710 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64480a0190 con 0x7f644813a300 2024-04-22T08:20:26.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6446ffd640 1 -- 172.21.15.40:0/2023127710 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3943267086 0 0) 0x7f6430003520 con 0x7f644809cbd0 2024-04-22T08:20:26.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6446ffd640 1 -- 172.21.15.40:0/2023127710 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f64480a0620 msgr2=0x7f6448142990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:20:26.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6446ffd640 1 --2- 172.21.15.40:0/2023127710 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f64480a0620 0x7f6448142990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:26.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6446ffd640 1 -- 172.21.15.40:0/2023127710 >> v1:172.21.15.189:6789/0 conn(0x7f644813a300 legacy=0x7f644813a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:26.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6446ffd640 10 monclient: _finish_hunting 0 2024-04-22T08:20:26.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6446ffd640 1 monclient: found mon.noname-a-legacy 2024-04-22T08:20:26.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6446ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:20:26.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6446ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.40:6789/0 2024-04-22T08:20:26.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6446ffd640 1 -- 172.21.15.40:0/2023127710 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f644809bf10 con 0x7f644809cbd0 2024-04-22T08:20:26.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6446ffd640 10 monclient: _finish_auth 0 2024-04-22T08:20:26.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6446ffd640 10 monclient: _check_auth_tickets 2024-04-22T08:20:26.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6446ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:20:26.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f644e7d7640 5 monclient: authenticate success, global_id 94179 2024-04-22T08:20:26.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6446ffd640 1 -- 172.21.15.40:0/2023127710 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1087006497 0 0) 0x7f643c001920 con 0x7f644809cbd0 2024-04-22T08:20:26.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f644e7d7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:20:26.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6446ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:26.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6446ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-22T08:20:26.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6446ffd640 10 monclient: dump: 2024-04-22T08:20:26.619 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:26.619 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:26.619 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:26.619 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:26.619 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:26.619 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:26.619 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:26.619 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:26.619 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:26.619 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:26.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6446ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] but i'm connected to v1:172.21.15.40:6789/0, reconnecting 2024-04-22T08:20:26.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6446ffd640 10 monclient: _reopen_session rank -1 2024-04-22T08:20:26.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6446ffd640 1 -- 172.21.15.40:0/2023127710 >> v1:172.21.15.40:6789/0 conn(0x7f644809cbd0 legacy=0x7f644813f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:26.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6446ffd640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-22T08:20:26.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6446ffd640 1 --2- 172.21.15.40:0/2023127710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6430003b30 0x7f6430003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:26.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6446ffd640 10 monclient(hunting): picked mon.b con 0x7f6430003b30 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:20:26.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6446ffd640 1 --2- 172.21.15.40:0/2023127710 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f64480a0620 0x7f64300047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:26.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6446ffd640 10 monclient(hunting): picked mon.c con 0x7f64480a0620 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:20:26.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6446ffd640 1 --2- 172.21.15.40:0/2023127710 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6430004e20 0x7f643000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:26.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6446ffd640 10 monclient(hunting): picked mon.a con 0x7f6430004e20 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:20:26.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6446ffd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:26.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6446ffd640 1 -- 172.21.15.40:0/2023127710 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f644809c2f0 con 0x7f6430004e20 2024-04-22T08:20:26.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6446ffd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:26.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6446ffd640 1 -- 172.21.15.40:0/2023127710 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6430005610 con 0x7f6430003b30 2024-04-22T08:20:26.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6446ffd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:26.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6446ffd640 1 -- 172.21.15.40:0/2023127710 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f643000bec0 con 0x7f64480a0620 2024-04-22T08:20:26.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6446ffd640 10 monclient(hunting): _renew_subs 2024-04-22T08:20:26.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6446ffd640 10 monclient(hunting): _finish_auth 0 2024-04-22T08:20:26.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f644e7d7640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-04-22T08:20:26.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f64477fe640 1 -- 172.21.15.40:0/2023127710 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f64480a0620 msgr2=0x7f64300047c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:20:26.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f64477fe640 1 --2- 172.21.15.40:0/2023127710 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f64480a0620 0x7f64300047c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:20:26.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f644cd74640 1 --2- 172.21.15.40:0/2023127710 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6430004e20 0x7f643000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:26.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f644cd74640 10 monclient(hunting): get_auth_request con 0x7f6430004e20 auth_method 0 2024-04-22T08:20:26.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f644cd74640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:26.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f644cd74640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:26.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f644cd74640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:20:26.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f644cd74640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:26.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f644cd74640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:26.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.611+0000 7f644cd74640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-22T08:20:26.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.610+0000 7f6447fff640 1 --2- 172.21.15.40:0/2023127710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6430003b30 0x7f6430003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:26.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.611+0000 7f6447fff640 10 monclient(hunting): get_auth_request con 0x7f6430003b30 auth_method 0 2024-04-22T08:20:26.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.611+0000 7f6447fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:26.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.611+0000 7f6447fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:26.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.611+0000 7f6447fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:20:26.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.611+0000 7f644cd74640 10 monclient(hunting): handle_auth_done global_id 94179 payload 293 2024-04-22T08:20:26.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.611+0000 7f644cd74640 1 -- 172.21.15.40:0/2023127710 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f64480a0620 msgr2=0x7f64300047c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:20:26.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.611+0000 7f644cd74640 1 --2- 172.21.15.40:0/2023127710 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f64480a0620 0x7f64300047c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:26.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.611+0000 7f644cd74640 1 -- 172.21.15.40:0/2023127710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6430003b30 msgr2=0x7f6430003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:20:26.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.611+0000 7f644cd74640 1 --2- 172.21.15.40:0/2023127710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6430003b30 0x7f6430003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:26.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.611+0000 7f644cd74640 10 monclient: _finish_hunting 0 2024-04-22T08:20:26.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.611+0000 7f644cd74640 1 monclient: found mon.a 2024-04-22T08:20:26.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.611+0000 7f644cd74640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:20:26.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.611+0000 7f644cd74640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:20:26.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.611+0000 7f644cd74640 1 -- 172.21.15.40:0/2023127710 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6430003520 con 0x7f6430004e20 2024-04-22T08:20:26.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.611+0000 7f644cd74640 10 monclient: _finish_auth 0 2024-04-22T08:20:26.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.611+0000 7f644cd74640 10 monclient: _check_auth_tickets 2024-04-22T08:20:26.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.611+0000 7f644cd74640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:20:26.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.611+0000 7f6447fff640 1 --2- 172.21.15.40:0/2023127710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6430003b30 0x7f6430003f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T08:20:26.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.613+0000 7f644cd74640 1 -- 172.21.15.40:0/2023127710 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6430004e20 msgr2=0x7f643000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:26.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.613+0000 7f644cd74640 1 --2- 172.21.15.40:0/2023127710 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6430004e20 0x7f643000b980 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6440009e30 tx=0x7f64300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8c7f5ee3ca9f3c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:26.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.614+0000 7f6446ffd640 1 -- 172.21.15.40:0/2023127710 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6440035470 con 0x7f6430004e20 2024-04-22T08:20:26.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.614+0000 7f6446ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:26.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.614+0000 7f6446ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:20:26.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.614+0000 7f6446ffd640 10 monclient: dump: 2024-04-22T08:20:26.623 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:26.623 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:26.623 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:26.624 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:26.624 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:26.624 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:26.624 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:26.624 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:26.624 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:26.624 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:26.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.614+0000 7f6446ffd640 1 -- 172.21.15.40:0/2023127710 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64400355d0 con 0x7f6430004e20 2024-04-22T08:20:26.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.614+0000 7f644e7d7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:20:26.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.614+0000 7f6446ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:20:26.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.614+0000 7f644e7d7640 10 monclient: get_monmap_and_config success 2024-04-22T08:20:26.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.614+0000 7f644e7d7640 4 set_mon_vals no callback set 2024-04-22T08:20:26.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.614+0000 7f6446ffd640 1 -- 172.21.15.40:0/2023127710 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6440035980 con 0x7f6430004e20 2024-04-22T08:20:26.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.614+0000 7f6446ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:26.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.614+0000 7f6446ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:20:26.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.614+0000 7f6446ffd640 10 monclient: dump: 2024-04-22T08:20:26.625 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:26.625 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:26.625 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:26.625 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:26.625 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:26.625 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:26.625 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:26.625 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:26.625 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:26.625 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:26.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.614+0000 7f644e7d7640 10 monclient: shutdown 2024-04-22T08:20:26.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.614+0000 7f644e7d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:20:26.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.614+0000 7f644e7d7640 1 -- 172.21.15.40:0/2023127710 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6430004e20 msgr2=0x7f643000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:26.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.614+0000 7f644e7d7640 1 --2- 172.21.15.40:0/2023127710 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6430004e20 0x7f643000b980 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6440009e30 tx=0x7f64300057f0 comp rx=0 tx=0).stop 2024-04-22T08:20:26.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.614+0000 7f644cd74640 1 -- 172.21.15.40:0/2023127710 reap_dead start 2024-04-22T08:20:26.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.614+0000 7f644e7d7640 1 -- 172.21.15.40:0/2023127710 shutdown_connections 2024-04-22T08:20:26.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.614+0000 7f644e7d7640 1 -- 172.21.15.40:0/2023127710 >> 172.21.15.40:0/2023127710 conn(0x7f64480a0ae0 msgr2=0x7f64480a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:20:26.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.614+0000 7f644e7d7640 1 -- 172.21.15.40:0/2023127710 shutdown_connections 2024-04-22T08:20:26.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.614+0000 7f644e7d7640 1 -- 172.21.15.40:0/2023127710 wait complete. 2024-04-22T08:20:26.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.615+0000 7f644e7d7640 10 monclient: build_initial_monmap 2024-04-22T08:20:26.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.615+0000 7f644e7d7640 10 monclient: monmap: 2024-04-22T08:20:26.626 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:20:26.626 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:20:26.627 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:20:26.627 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:20:26.627 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:20:26.627 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:26.627 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:20:26.627 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:20:26.627 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:20:26.627 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:26.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.615+0000 7f644e7d7640 1 Processor -- start 2024-04-22T08:20:26.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.615+0000 7f644e7d7640 1 -- start start 2024-04-22T08:20:26.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.615+0000 7f644e7d7640 10 monclient: init 2024-04-22T08:20:26.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.616+0000 7f644e7d7640 10 monclient: _reopen_session rank -1 2024-04-22T08:20:26.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.616+0000 7f644e7d7640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-22T08:20:26.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.616+0000 7f644e7d7640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f64481233e0 0x7f644813e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:26.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.616+0000 7f644e7d7640 10 monclient(hunting): picked mon.noname-c con 0x7f64481233e0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:20:26.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.616+0000 7f644e7d7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6448139240 0x7f64481396a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:26.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.616+0000 7f644e7d7640 10 monclient(hunting): picked mon.noname-a con 0x7f6448139240 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:20:26.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.616+0000 7f644e7d7640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64481cecb0 0x7f64481cf130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:26.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.616+0000 7f644e7d7640 10 monclient(hunting): picked mon.noname-b con 0x7f64481cecb0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:20:26.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.616+0000 7f644e7d7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:26.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.616+0000 7f644e7d7640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6448140310 con 0x7f6448139240 2024-04-22T08:20:26.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.616+0000 7f644e7d7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:26.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.616+0000 7f644e7d7640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f644813e9e0 con 0x7f64481cecb0 2024-04-22T08:20:26.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.616+0000 7f644e7d7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:26.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.616+0000 7f644e7d7640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f644813fb70 con 0x7f64481233e0 2024-04-22T08:20:26.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.616+0000 7f644e7d7640 10 monclient(hunting): _renew_subs 2024-04-22T08:20:26.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.616+0000 7f6447fff640 1 -- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f64481233e0 msgr2=0x7f644813e4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:20:26.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.616+0000 7f644e7d7640 10 monclient(hunting): authenticate will time out at 1300.609969s 2024-04-22T08:20:26.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.616+0000 7f6447fff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f64481233e0 0x7f644813e4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:20:26.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.616+0000 7f64477fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6448139240 0x7f64481396a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:26.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.616+0000 7f64477fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6448139240 0x7f64481396a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58680/0 (socket says 172.21.15.40:58680) 2024-04-22T08:20:26.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.616+0000 7f64477fe640 1 -- 172.21.15.40:0/715929310 learned_addr learned my addr 172.21.15.40:0/715929310 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:20:26.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.616+0000 7f64477fe640 10 monclient(hunting): get_auth_request con 0x7f6448139240 auth_method 0 2024-04-22T08:20:26.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.616+0000 7f64477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:26.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.616+0000 7f64477fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:26.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.616+0000 7f64477fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:26.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.616+0000 7f64477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:26.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.616+0000 7f64477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:26.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.616+0000 7f64477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:26.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.616+0000 7f644cd74640 1 --2- 172.21.15.40:0/715929310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64481cecb0 0x7f64481cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:26.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.616+0000 7f644cd74640 10 monclient(hunting): get_auth_request con 0x7f64481cecb0 auth_method 0 2024-04-22T08:20:26.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.616+0000 7f644cd74640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:26.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.616+0000 7f644cd74640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:26.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.616+0000 7f644cd74640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:26.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.616+0000 7f64477fe640 10 monclient(hunting): handle_auth_done global_id 94182 payload 824 2024-04-22T08:20:26.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.616+0000 7f64477fe640 1 -- 172.21.15.40:0/715929310 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f64481233e0 msgr2=0x7f644813e4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:20:26.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.616+0000 7f64477fe640 1 --2- 172.21.15.40:0/715929310 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f64481233e0 0x7f644813e4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:26.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.616+0000 7f64477fe640 1 -- 172.21.15.40:0/715929310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64481cecb0 msgr2=0x7f64481cf130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:26.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.616+0000 7f64477fe640 1 --2- 172.21.15.40:0/715929310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64481cecb0 0x7f64481cf130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:26.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.616+0000 7f64477fe640 10 monclient: _finish_hunting 0 2024-04-22T08:20:26.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.616+0000 7f64477fe640 1 monclient: found mon.noname-a 2024-04-22T08:20:26.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.616+0000 7f64477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:20:26.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.616+0000 7f64477fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:20:26.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.616+0000 7f64477fe640 1 -- 172.21.15.40:0/715929310 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64481237c0 con 0x7f6448139240 2024-04-22T08:20:26.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.616+0000 7f644cd74640 1 --2- 172.21.15.40:0/715929310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64481cecb0 0x7f64481cf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T08:20:26.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.618+0000 7f64477fe640 1 -- 172.21.15.40:0/715929310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6448139240 msgr2=0x7f64481396a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:26.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.618+0000 7f64477fe640 1 --2- 172.21.15.40:0/715929310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6448139240 0x7f64481396a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f643c00c690 tx=0x7f643c00cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:26.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.619+0000 7f6444ff9640 1 -- 172.21.15.40:0/715929310 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f643c004cb0 con 0x7f6448139240 2024-04-22T08:20:26.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.619+0000 7f6444ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:26.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.619+0000 7f6444ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:20:26.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.619+0000 7f6444ff9640 10 monclient: dump: 2024-04-22T08:20:26.636 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:26.636 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:26.636 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:26.636 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:26.636 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:26.636 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:26.636 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:26.636 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:26.636 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:26.637 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:26.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.619+0000 7f6444ff9640 10 monclient: _finish_auth 0 2024-04-22T08:20:26.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.619+0000 7f6444ff9640 10 monclient: _check_auth_tickets 2024-04-22T08:20:26.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.619+0000 7f6444ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:20:26.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.619+0000 7f644e7d7640 5 monclient: authenticate success, global_id 94182 2024-04-22T08:20:26.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.619+0000 7f6444ff9640 1 -- 172.21.15.40:0/715929310 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f643c004e30 con 0x7f6448139240 2024-04-22T08:20:26.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.619+0000 7f6444ff9640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:20:26.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.619+0000 7f6444ff9640 1 -- 172.21.15.40:0/715929310 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f643c007310 con 0x7f6448139240 2024-04-22T08:20:26.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.619+0000 7f644e7d7640 10 monclient: _renew_subs 2024-04-22T08:20:26.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.619+0000 7f644e7d7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:20:26.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.619+0000 7f644e7d7640 1 -- 172.21.15.40:0/715929310 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6448124340 con 0x7f6448139240 2024-04-22T08:20:26.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.619+0000 7f64467fc640 4 set_mon_vals no callback set 2024-04-22T08:20:26.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.619+0000 7f6444ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:26.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.619+0000 7f6444ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:20:26.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.619+0000 7f6444ff9640 10 monclient: dump: 2024-04-22T08:20:26.638 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:26.638 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:26.638 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:26.638 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:26.638 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:26.638 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:26.638 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:26.638 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:26.638 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:26.638 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:26.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.619+0000 7f644e7d7640 10 monclient: _renew_subs 2024-04-22T08:20:26.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.619+0000 7f644e7d7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:20:26.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.619+0000 7f644e7d7640 1 -- 172.21.15.40:0/715929310 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6448124900 con 0x7f6448139240 2024-04-22T08:20:26.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.619+0000 7f6444ff9640 1 -- 172.21.15.40:0/715929310 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f643c0074b0 con 0x7f6448139240 2024-04-22T08:20:26.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.619+0000 7f644e7d7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:20:26.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.620+0000 7f644e7d7640 1 --2- 172.21.15.40:0/715929310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64481cf670 0x7f644806f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:26.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.620+0000 7f644e7d7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:26.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.620+0000 7f644e7d7640 1 -- 172.21.15.40:0/715929310 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64480667a0 con 0x7f64481cf670 2024-04-22T08:20:26.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.620+0000 7f6444ff9640 1 --2- 172.21.15.40:0/715929310 >> v2:172.21.15.189:6800/30168 conn(0x7f6434037780 0x7f6434039c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:26.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.620+0000 7f6444ff9640 1 -- 172.21.15.40:0/715929310 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f643c049560 con 0x7f6448139240 2024-04-22T08:20:26.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.620+0000 7f6447fff640 1 --2- 172.21.15.40:0/715929310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64481cf670 0x7f644806f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:26.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.620+0000 7f6447fff640 10 monclient: get_auth_request con 0x7f64481cf670 auth_method 0 2024-04-22T08:20:26.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.620+0000 7f6447fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:26.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.620+0000 7f6447fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:26.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.620+0000 7f6447fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:26.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.620+0000 7f6447fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:26.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.620+0000 7f6447fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:26.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.620+0000 7f6447fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:26.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.620+0000 7f6447fff640 10 monclient(hunting): handle_auth_done global_id 94185 payload 824 2024-04-22T08:20:26.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.620+0000 7f6447fff640 1 -- 172.21.15.40:0/715929310 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6448140980 con 0x7f64481cf670 2024-04-22T08:20:26.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.620+0000 7f644cd74640 1 --2- 172.21.15.40:0/715929310 >> v2:172.21.15.189:6800/30168 conn(0x7f6434037780 0x7f6434039c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:26.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.620+0000 7f644cd74640 10 monclient: get_auth_request con 0x7f6434037780 auth_method 0 2024-04-22T08:20:26.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.622+0000 7f6447fff640 1 -- 172.21.15.40:0/715929310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64481cf670 msgr2=0x7f644806f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:26.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.622+0000 7f6447fff640 1 --2- 172.21.15.40:0/715929310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64481cf670 0x7f644806f740 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f643800e700 tx=0x7f643800ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:26.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.622+0000 7f6444ff9640 1 -- 172.21.15.40:0/715929310 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6438007ea0 con 0x7f64481cf670 2024-04-22T08:20:26.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.622+0000 7f6444ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:26.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.622+0000 7f6444ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:20:26.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.622+0000 7f6444ff9640 10 monclient: dump: 2024-04-22T08:20:26.641 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:26.641 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:26.641 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:26.641 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:26.641 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:26.641 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:26.641 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:26.641 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:26.641 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:26.641 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:26.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.623+0000 7f6444ff9640 1 -- 172.21.15.40:0/715929310 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f64380072e0 con 0x7f64481cf670 2024-04-22T08:20:26.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.623+0000 7f6444ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:20:26.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.623+0000 7f6444ff9640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:20:26.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.623+0000 7f6444ff9640 1 -- 172.21.15.40:0/715929310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64481cf670 msgr2=0x7f644806f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:26.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.623+0000 7f6444ff9640 1 --2- 172.21.15.40:0/715929310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64481cf670 0x7f644806f740 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f643800e700 tx=0x7f643800ebc0 comp rx=0 tx=0).stop 2024-04-22T08:20:26.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.623+0000 7f644cd74640 1 --2- 172.21.15.40:0/715929310 >> v2:172.21.15.189:6800/30168 conn(0x7f6434037780 0x7f6434039c40 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f6440031000 tx=0x7f644002f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:26.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.623+0000 7f6444ff9640 1 -- 172.21.15.40:0/715929310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64481cf670 msgr2=0x7f644806f740 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:20:26.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.623+0000 7f6444ff9640 1 --2- 172.21.15.40:0/715929310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64481cf670 0x7f644806f740 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f643800e700 tx=0x7f643800ebc0 comp rx=0 tx=0).stop 2024-04-22T08:20:26.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.629+0000 7f644e7d7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:20:26.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.629+0000 7f644e7d7640 1 --2- 172.21.15.40:0/715929310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f644806fc80 0x7f6448070090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:26.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.629+0000 7f644e7d7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:26.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.629+0000 7f644e7d7640 1 -- 172.21.15.40:0/715929310 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6448124b20 con 0x7f644806fc80 2024-04-22T08:20:26.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.629+0000 7f6447fff640 1 --2- 172.21.15.40:0/715929310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f644806fc80 0x7f6448070090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:26.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.629+0000 7f6447fff640 10 monclient: get_auth_request con 0x7f644806fc80 auth_method 0 2024-04-22T08:20:26.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.629+0000 7f6447fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:26.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.629+0000 7f6447fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:26.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.629+0000 7f6447fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:26.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.629+0000 7f6447fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:26.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.629+0000 7f6447fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:26.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.629+0000 7f6447fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:26.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.629+0000 7f6447fff640 10 monclient(hunting): handle_auth_done global_id 94188 payload 824 2024-04-22T08:20:26.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.629+0000 7f6447fff640 1 -- 172.21.15.40:0/715929310 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f644809c4d0 con 0x7f644806fc80 2024-04-22T08:20:26.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.631+0000 7f6447fff640 1 -- 172.21.15.40:0/715929310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f644806fc80 msgr2=0x7f6448070090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:26.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.631+0000 7f6447fff640 1 --2- 172.21.15.40:0/715929310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f644806fc80 0x7f6448070090 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f643800e700 tx=0x7f6438001700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:26.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.631+0000 7f6444ff9640 1 -- 172.21.15.40:0/715929310 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6438022070 con 0x7f644806fc80 2024-04-22T08:20:26.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.631+0000 7f6444ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:26.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.631+0000 7f6444ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:20:26.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.631+0000 7f6444ff9640 10 monclient: dump: 2024-04-22T08:20:26.643 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:26.643 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:26.643 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:26.644 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:26.644 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:26.644 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:26.644 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:26.644 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:26.644 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:26.644 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:26.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.632+0000 7f6444ff9640 1 -- 172.21.15.40:0/715929310 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4622 (secure 0 0 0) 0x7f643800c6e0 con 0x7f644806fc80 2024-04-22T08:20:26.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.632+0000 7f6444ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:20:26.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.632+0000 7f6444ff9640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:20:26.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.632+0000 7f6444ff9640 1 -- 172.21.15.40:0/715929310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f644806fc80 msgr2=0x7f6448070090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:26.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.632+0000 7f6444ff9640 1 --2- 172.21.15.40:0/715929310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f644806fc80 0x7f6448070090 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f643800e700 tx=0x7f6438001700 comp rx=0 tx=0).stop 2024-04-22T08:20:26.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.632+0000 7f6444ff9640 1 -- 172.21.15.40:0/715929310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f644806fc80 msgr2=0x7f6448070090 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:20:26.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.632+0000 7f6444ff9640 1 --2- 172.21.15.40:0/715929310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f644806fc80 0x7f6448070090 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:26.645 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:20:26.645 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:20:26.645 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:20:26.645 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "leader", 2024-04-22T08:20:26.645 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 76, 2024-04-22T08:20:26.645 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:20:26.645 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:20:26.645 INFO:teuthology.orchestra.run.smithi040.stdout: 1 2024-04-22T08:20:26.645 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:20:26.645 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 0, 2024-04-22T08:20:26.646 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:20:26.646 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:20:26.646 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:20:26.646 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:20:26.646 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:20:26.646 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:20:26.646 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:20:26.646 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:20:26.646 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:20:26.646 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:20:26.646 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:20:26.646 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:20:26.646 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:20:26.646 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:20:26.647 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:20:26.647 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:20:26.647 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:20:26.647 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:20:26.647 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:20:26.647 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:20:26.647 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:20:26.647 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:20:26.647 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:20:26.647 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:20:26.647 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:20:26.647 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:20:26.647 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:26.647 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:26.647 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:20:26.648 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:20:26.648 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:20:26.648 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:20:26.648 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:20:26.648 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:20:26.648 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:20:26.648 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:20:26.648 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:20:26.648 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:20:26.648 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:20:26.648 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:20:26.648 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:20:26.648 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:20:26.648 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:20:26.648 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:20:26.649 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:20:26.649 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:20:26.649 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:20:26.649 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:20:26.649 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:20:26.649 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:20:26.649 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:20:26.649 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:20:26.649 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:20:26.649 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:20:26.649 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:20:26.649 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:20:26.649 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:20:26.649 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:26.649 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:20:26.650 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:26.650 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:20:26.650 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:20:26.650 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:20:26.650 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:20:26.650 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:26.650 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:20:26.650 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:20:26.650 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:26.650 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:26.650 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:26.650 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:20:26.650 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:20:26.650 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:26.651 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:26.651 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:26.651 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:26.651 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:20:26.651 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:20:26.651 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:20:26.651 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:20:26.651 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:20:26.651 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:26.651 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:26.651 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:20:26.651 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:20:26.651 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:20:26.651 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:20:26.651 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:26.652 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:20:26.652 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:20:26.652 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:26.652 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:26.652 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:26.652 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:20:26.652 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:20:26.652 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:26.652 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:26.652 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:26.652 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:26.652 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:20:26.652 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:20:26.652 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:20:26.652 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:20:26.653 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:20:26.653 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:26.653 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:26.653 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:20:26.653 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:20:26.653 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:20:26.653 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:20:26.653 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:26.653 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:20:26.653 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:20:26.653 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:26.653 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:26.653 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:26.653 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:20:26.654 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:20:26.654 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:26.654 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:26.654 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:26.654 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:26.654 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:20:26.654 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:20:26.654 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:20:26.654 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:20:26.654 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:20:26.654 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:26.654 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:26.654 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:26.654 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:20:26.654 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:20:26.655 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:26.655 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:20:26.655 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:20:26.655 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:20:26.655 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:26.655 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:20:26.655 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:20:26.655 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:26.655 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:20:26.655 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:20:26.656 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 2 2024-04-22T08:20:26.656 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:26.656 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:20:26.656 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:20:26.656 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:26.656 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:20:26.656 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:20:26.656 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 5 2024-04-22T08:20:26.656 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:26.656 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:26.656 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:26.656 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:20:26.656 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:20:26.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.632+0000 7f644e7d7640 1 -- 172.21.15.40:0/715929310 >> v2:172.21.15.189:6800/30168 conn(0x7f6434037780 msgr2=0x7f6434039c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:26.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.632+0000 7f644e7d7640 1 --2- 172.21.15.40:0/715929310 >> v2:172.21.15.189:6800/30168 conn(0x7f6434037780 0x7f6434039c40 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f6440031000 tx=0x7f644002f000 comp rx=0 tx=0).stop 2024-04-22T08:20:26.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.632+0000 7f644e7d7640 10 monclient: shutdown 2024-04-22T08:20:26.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.632+0000 7f644e7d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:20:26.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.632+0000 7f644e7d7640 1 -- 172.21.15.40:0/715929310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6448139240 msgr2=0x7f64481396a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:26.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.632+0000 7f644e7d7640 1 --2- 172.21.15.40:0/715929310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6448139240 0x7f64481396a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f643c00c690 tx=0x7f643c00cb50 comp rx=0 tx=0).stop 2024-04-22T08:20:26.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.632+0000 7f644cd74640 1 -- 172.21.15.40:0/715929310 reap_dead start 2024-04-22T08:20:26.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.632+0000 7f644cd74640 1 -- 172.21.15.40:0/715929310 reap_dead start 2024-04-22T08:20:26.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.632+0000 7f644e7d7640 1 -- 172.21.15.40:0/715929310 shutdown_connections 2024-04-22T08:20:26.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.632+0000 7f644e7d7640 1 -- 172.21.15.40:0/715929310 >> 172.21.15.40:0/715929310 conn(0x7f64480a0ae0 msgr2=0x7f644809e8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:20:26.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.632+0000 7f644e7d7640 1 -- 172.21.15.40:0/715929310 shutdown_connections 2024-04-22T08:20:26.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.632+0000 7f644e7d7640 1 -- 172.21.15.40:0/715929310 wait complete. 2024-04-22T08:20:26.659 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-22T08:20:26.659 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-04-22T08:20:26.659 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-22T08:20:26.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.759+0000 7f10116ce640 10 monclient: get_monmap_and_config 2024-04-22T08:20:26.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.759+0000 7f10116ce640 10 monclient: build_initial_monmap 2024-04-22T08:20:26.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.759+0000 7f10116ce640 10 monclient: monmap: 2024-04-22T08:20:26.759 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:20:26.759 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:26.759 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:20:26.760230+0000 2024-04-22T08:20:26.759 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:20:26.760230+0000 2024-04-22T08:20:26.759 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:20:26.759 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:26.759 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:20:26.759 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:20:26.759 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:20:26.759 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:20:26.760 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:20:26.760 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:26.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.760+0000 7f10116ce640 1 Processor -- start 2024-04-22T08:20:26.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.760+0000 7f10116ce640 1 -- start start 2024-04-22T08:20:26.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.760+0000 7f10116ce640 10 monclient: init 2024-04-22T08:20:26.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.760+0000 7f10116ce640 10 monclient: _reopen_session rank -1 2024-04-22T08:20:26.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.760+0000 7f10116ce640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-04-22T08:20:26.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.760+0000 7f10116ce640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f100c12fad0 addr v1:172.21.15.189:6789/0 2024-04-22T08:20:26.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.760+0000 7f10116ce640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f100c135c30 0x7f100c138020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:26.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.760+0000 7f10116ce640 10 monclient(hunting): picked mon.noname-a con 0x7f100c135c30 addr v2:172.21.15.40:3300/0 2024-04-22T08:20:26.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.760+0000 7f10116ce640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f100c138560 0x7f100c13a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:26.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.760+0000 7f10116ce640 10 monclient(hunting): picked mon.noname-b con 0x7f100c138560 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:20:26.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.760+0000 7f10116ce640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f100c1305c0 con 0x7f100c12fad0 2024-04-22T08:20:26.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.760+0000 7f10116ce640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:26.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.760+0000 7f10116ce640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f100c09ae50 con 0x7f100c135c30 2024-04-22T08:20:26.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.760+0000 7f10116ce640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:26.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.760+0000 7f10116ce640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f100c09efc0 con 0x7f100c138560 2024-04-22T08:20:26.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.760+0000 7f10116ce640 10 monclient(hunting): _renew_subs 2024-04-22T08:20:26.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.760+0000 7f10116ce640 10 monclient(hunting): authenticate will time out at 1300.754420s 2024-04-22T08:20:26.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.760+0000 7f100b7fe640 1 -- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f100c138560 msgr2=0x7f100c13a950 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:20:26.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.760+0000 7f100b7fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f100c138560 0x7f100c13a950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:20:26.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.760+0000 7f1003fff640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f100c135c30 0x7f100c138020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:26.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.760+0000 7f1003fff640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f100c135c30 0x7f100c138020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58710/0 (socket says 172.21.15.40:58710) 2024-04-22T08:20:26.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.760+0000 7f1003fff640 1 -- 172.21.15.40:0/844187986 learned_addr learned my addr 172.21.15.40:0/844187986 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:20:26.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.760+0000 7f1003fff640 10 monclient(hunting): get_auth_request con 0x7f100c135c30 auth_method 0 2024-04-22T08:20:26.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.760+0000 7f1003fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:26.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.760+0000 7f1003fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:26.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.760+0000 7f1003fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:26.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.760+0000 7f1003fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:26.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.760+0000 7f1003fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:26.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.760+0000 7f1003fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:26.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.761+0000 7f1003fff640 10 monclient(hunting): handle_auth_done global_id 94191 payload 274 2024-04-22T08:20:26.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.761+0000 7f1003fff640 1 -- 172.21.15.40:0/844187986 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f100c138560 msgr2=0x7f100c13a950 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:20:26.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.761+0000 7f1003fff640 1 --2- 172.21.15.40:0/844187986 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f100c138560 0x7f100c13a950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:26.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.761+0000 7f1003fff640 1 -- 172.21.15.40:0/844187986 >> v1:172.21.15.189:6789/0 conn(0x7f100c12fad0 legacy=0x7f100c12feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:20:26.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.761+0000 7f1003fff640 10 monclient: _finish_hunting 0 2024-04-22T08:20:26.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.761+0000 7f1003fff640 1 monclient: found mon.noname-a 2024-04-22T08:20:26.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.761+0000 7f1003fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:20:26.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.761+0000 7f1003fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:20:26.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.761+0000 7f1003fff640 1 -- 172.21.15.40:0/844187986 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f100c09d070 con 0x7f100c135c30 2024-04-22T08:20:26.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.763+0000 7f1003fff640 1 -- 172.21.15.40:0/844187986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f100c135c30 msgr2=0x7f100c138020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:26.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.763+0000 7f1003fff640 1 --2- 172.21.15.40:0/844187986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f100c135c30 0x7f100c138020 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0ff4009870 tx=0x7f0ff402b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49c430b6b7a4ca84 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:26.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.764+0000 7f100a7fc640 1 -- 172.21.15.40:0/844187986 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff4031a40 con 0x7f100c135c30 2024-04-22T08:20:26.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.764+0000 7f100a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:26.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.764+0000 7f100a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:20:26.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.764+0000 7f100a7fc640 10 monclient: dump: 2024-04-22T08:20:26.771 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:26.771 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:26.771 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:26.771 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:26.771 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:26.771 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:26.771 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:26.771 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:26.771 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:26.771 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:26.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.764+0000 7f100a7fc640 10 monclient: _finish_auth 0 2024-04-22T08:20:26.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.764+0000 7f100a7fc640 10 monclient: _check_auth_tickets 2024-04-22T08:20:26.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.764+0000 7f100a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:20:26.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.764+0000 7f100a7fc640 1 -- 172.21.15.40:0/844187986 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ff4031ba0 con 0x7f100c135c30 2024-04-22T08:20:26.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.764+0000 7f10116ce640 5 monclient: authenticate success, global_id 94191 2024-04-22T08:20:26.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.764+0000 7f10116ce640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:20:26.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.764+0000 7f100a7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:20:26.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.764+0000 7f10116ce640 10 monclient: get_monmap_and_config success 2024-04-22T08:20:26.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.764+0000 7f10116ce640 4 set_mon_vals no callback set 2024-04-22T08:20:26.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.764+0000 7f100a7fc640 1 -- 172.21.15.40:0/844187986 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff4033440 con 0x7f100c135c30 2024-04-22T08:20:26.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.764+0000 7f100a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:26.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.764+0000 7f100a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:20:26.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.764+0000 7f100a7fc640 10 monclient: dump: 2024-04-22T08:20:26.772 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:26.772 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:26.772 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:26.772 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:26.773 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:26.773 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:26.773 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:26.773 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:26.773 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:26.773 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:26.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.764+0000 7f10116ce640 10 monclient: shutdown 2024-04-22T08:20:26.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.764+0000 7f10116ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:20:26.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.764+0000 7f10116ce640 1 -- 172.21.15.40:0/844187986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f100c135c30 msgr2=0x7f100c138020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:26.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.764+0000 7f10116ce640 1 --2- 172.21.15.40:0/844187986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f100c135c30 0x7f100c138020 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0ff4009870 tx=0x7f0ff402b7f0 comp rx=0 tx=0).stop 2024-04-22T08:20:26.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.764+0000 7f10116ce640 1 -- 172.21.15.40:0/844187986 shutdown_connections 2024-04-22T08:20:26.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.764+0000 7f10116ce640 1 --2- 172.21.15.40:0/844187986 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f100c138560 0x7f100c13a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:26.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.764+0000 7f10116ce640 1 --2- 172.21.15.40:0/844187986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f100c135c30 0x7f100c138020 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:26.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.764+0000 7f10116ce640 1 -- 172.21.15.40:0/844187986 >> 172.21.15.40:0/844187986 conn(0x7f100c0a0ae0 msgr2=0x7f100c0a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:20:26.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.764+0000 7f10116ce640 1 -- 172.21.15.40:0/844187986 shutdown_connections 2024-04-22T08:20:26.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.764+0000 7f10116ce640 1 -- 172.21.15.40:0/844187986 wait complete. 2024-04-22T08:20:26.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.765+0000 7f10116ce640 10 monclient: build_initial_monmap 2024-04-22T08:20:26.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.765+0000 7f10116ce640 10 monclient: monmap: 2024-04-22T08:20:26.774 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:20:26.774 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:20:26.774 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:20:26.774 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:20:26.774 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:20:26.774 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:26.774 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:20:26.774 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:20:26.774 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:20:26.774 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:26.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.765+0000 7f10116ce640 1 Processor -- start 2024-04-22T08:20:26.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.765+0000 7f10116ce640 1 -- start start 2024-04-22T08:20:26.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.765+0000 7f10116ce640 10 monclient: init 2024-04-22T08:20:26.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.766+0000 7f10116ce640 10 monclient: _reopen_session rank -1 2024-04-22T08:20:26.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.766+0000 7f10116ce640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-22T08:20:26.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.766+0000 7f10116ce640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f100c12fad0 0x7f100c1c6e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:26.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.766+0000 7f10116ce640 10 monclient(hunting): picked mon.noname-c con 0x7f100c12fad0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:20:26.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.766+0000 7f10116ce640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f100c135c30 0x7f100c1312f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:26.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.766+0000 7f10116ce640 10 monclient(hunting): picked mon.noname-a con 0x7f100c135c30 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:20:26.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.766+0000 7f10116ce640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f100c138560 0x7f100c131830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:26.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.766+0000 7f10116ce640 10 monclient(hunting): picked mon.noname-b con 0x7f100c138560 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:20:26.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.766+0000 7f10116ce640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:26.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.766+0000 7f10116ce640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f100c12eda0 con 0x7f100c135c30 2024-04-22T08:20:26.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.766+0000 7f10116ce640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:26.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.766+0000 7f10116ce640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f100c071870 con 0x7f100c138560 2024-04-22T08:20:26.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.766+0000 7f10116ce640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:26.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.766+0000 7f10116ce640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f100c0a1ec0 con 0x7f100c12fad0 2024-04-22T08:20:26.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.766+0000 7f10116ce640 10 monclient(hunting): _renew_subs 2024-04-22T08:20:26.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.766+0000 7f10116ce640 10 monclient(hunting): authenticate will time out at 1300.759891s 2024-04-22T08:20:26.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.766+0000 7f100affd640 1 -- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f100c12fad0 msgr2=0x7f100c1c6e60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:20:26.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.766+0000 7f100affd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f100c12fad0 0x7f100c1c6e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:20:26.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.766+0000 7f1003fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f100c135c30 0x7f100c1312f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:26.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.766+0000 7f1003fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f100c135c30 0x7f100c1312f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58712/0 (socket says 172.21.15.40:58712) 2024-04-22T08:20:26.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.766+0000 7f1003fff640 1 -- 172.21.15.40:0/1285597493 learned_addr learned my addr 172.21.15.40:0/1285597493 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:20:26.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.766+0000 7f1003fff640 10 monclient(hunting): get_auth_request con 0x7f100c135c30 auth_method 0 2024-04-22T08:20:26.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.766+0000 7f1003fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:26.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.766+0000 7f1003fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:26.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.766+0000 7f1003fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:26.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.766+0000 7f1003fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:26.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.766+0000 7f1003fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:26.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.766+0000 7f1003fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:26.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.766+0000 7f100b7fe640 1 --2- 172.21.15.40:0/1285597493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f100c138560 0x7f100c131830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:26.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.766+0000 7f100b7fe640 10 monclient(hunting): get_auth_request con 0x7f100c138560 auth_method 0 2024-04-22T08:20:26.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.766+0000 7f100b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:26.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.766+0000 7f100b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:26.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.766+0000 7f100b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:26.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.766+0000 7f1003fff640 10 monclient(hunting): handle_auth_done global_id 94194 payload 824 2024-04-22T08:20:26.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.766+0000 7f1003fff640 1 -- 172.21.15.40:0/1285597493 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f100c12fad0 msgr2=0x7f100c1c6e60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:20:26.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.766+0000 7f1003fff640 1 --2- 172.21.15.40:0/1285597493 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f100c12fad0 0x7f100c1c6e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:26.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.766+0000 7f1003fff640 1 -- 172.21.15.40:0/1285597493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f100c138560 msgr2=0x7f100c131830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:26.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.766+0000 7f1003fff640 1 --2- 172.21.15.40:0/1285597493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f100c138560 0x7f100c131830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:26.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.766+0000 7f1003fff640 10 monclient: _finish_hunting 0 2024-04-22T08:20:26.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.766+0000 7f1003fff640 1 monclient: found mon.noname-a 2024-04-22T08:20:26.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.766+0000 7f1003fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:20:26.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.766+0000 7f1003fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:20:26.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.766+0000 7f1003fff640 1 -- 172.21.15.40:0/1285597493 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff40094d0 con 0x7f100c135c30 2024-04-22T08:20:26.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.766+0000 7f100b7fe640 1 --2- 172.21.15.40:0/1285597493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f100c138560 0x7f100c131830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T08:20:26.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.768+0000 7f1003fff640 1 -- 172.21.15.40:0/1285597493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f100c135c30 msgr2=0x7f100c1312f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:26.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.768+0000 7f1003fff640 1 --2- 172.21.15.40:0/1285597493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f100c135c30 0x7f100c1312f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0ff4009870 tx=0x7f0ff40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:26.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.768+0000 7f10037fe640 1 -- 172.21.15.40:0/1285597493 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff4031f10 con 0x7f100c135c30 2024-04-22T08:20:26.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.768+0000 7f10037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:26.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.768+0000 7f10037fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:20:26.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.768+0000 7f10037fe640 10 monclient: dump: 2024-04-22T08:20:26.778 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:26.778 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:26.778 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:26.778 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:26.779 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:26.779 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:26.779 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:26.779 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:26.779 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:26.779 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:26.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.768+0000 7f10037fe640 10 monclient: _finish_auth 0 2024-04-22T08:20:26.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.768+0000 7f10037fe640 10 monclient: _check_auth_tickets 2024-04-22T08:20:26.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.768+0000 7f10037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:20:26.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.768+0000 7f10116ce640 5 monclient: authenticate success, global_id 94194 2024-04-22T08:20:26.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.768+0000 7f10037fe640 1 -- 172.21.15.40:0/1285597493 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ff403d470 con 0x7f100c135c30 2024-04-22T08:20:26.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.768+0000 7f10116ce640 10 monclient: _renew_subs 2024-04-22T08:20:26.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.768+0000 7f10116ce640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:20:26.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.768+0000 7f10116ce640 1 -- 172.21.15.40:0/1285597493 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f100c135250 con 0x7f100c135c30 2024-04-22T08:20:26.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.768+0000 7f10037fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:20:26.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.768+0000 7f10037fe640 1 -- 172.21.15.40:0/1285597493 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff403d800 con 0x7f100c135c30 2024-04-22T08:20:26.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.768+0000 7f10037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:26.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.769+0000 7f10037fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:20:26.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.769+0000 7f10037fe640 10 monclient: dump: 2024-04-22T08:20:26.780 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:26.780 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:26.780 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:26.780 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:26.780 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:26.780 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:26.780 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:26.780 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:26.780 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:26.780 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:26.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.768+0000 7f1009ffb640 4 set_mon_vals no callback set 2024-04-22T08:20:26.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.769+0000 7f10116ce640 10 monclient: _renew_subs 2024-04-22T08:20:26.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.769+0000 7f10116ce640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:20:26.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.769+0000 7f10116ce640 1 -- 172.21.15.40:0/1285597493 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f100c1349b0 con 0x7f100c135c30 2024-04-22T08:20:26.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.769+0000 7f10116ce640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:20:26.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.769+0000 7f10116ce640 1 --2- 172.21.15.40:0/1285597493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f100c131d70 0x7f100c132150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:26.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.769+0000 7f10116ce640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:26.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.769+0000 7f10116ce640 1 -- 172.21.15.40:0/1285597493 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f100c134d50 con 0x7f100c131d70 2024-04-22T08:20:26.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.769+0000 7f10037fe640 1 -- 172.21.15.40:0/1285597493 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f0ff403a040 con 0x7f100c135c30 2024-04-22T08:20:26.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.770+0000 7f10037fe640 1 --2- 172.21.15.40:0/1285597493 >> v2:172.21.15.189:6800/30168 conn(0x7f0fec037780 0x7f0fec039c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:26.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.770+0000 7f10037fe640 1 -- 172.21.15.40:0/1285597493 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f0ff406c420 con 0x7f100c135c30 2024-04-22T08:20:26.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.770+0000 7f100affd640 1 --2- 172.21.15.40:0/1285597493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f100c131d70 0x7f100c132150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:26.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.770+0000 7f100affd640 10 monclient: get_auth_request con 0x7f100c131d70 auth_method 0 2024-04-22T08:20:26.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.770+0000 7f100affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:26.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.770+0000 7f100affd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:26.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.770+0000 7f100affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:26.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.770+0000 7f100affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:26.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.770+0000 7f100affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:26.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.770+0000 7f100affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:26.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.770+0000 7f100b7fe640 1 --2- 172.21.15.40:0/1285597493 >> v2:172.21.15.189:6800/30168 conn(0x7f0fec037780 0x7f0fec039c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:26.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.770+0000 7f100b7fe640 10 monclient: get_auth_request con 0x7f0fec037780 auth_method 0 2024-04-22T08:20:26.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.770+0000 7f100affd640 10 monclient(hunting): handle_auth_done global_id 64732 payload 824 2024-04-22T08:20:26.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.770+0000 7f100affd640 1 -- 172.21.15.40:0/1285597493 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f100c09c9b0 con 0x7f100c131d70 2024-04-22T08:20:26.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.772+0000 7f100affd640 1 -- 172.21.15.40:0/1285597493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f100c131d70 msgr2=0x7f100c132150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:26.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.772+0000 7f100affd640 1 --2- 172.21.15.40:0/1285597493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f100c131d70 0x7f100c132150 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc00c380 tx=0x7f0ffc00c840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:26.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.772+0000 7f100b7fe640 1 --2- 172.21.15.40:0/1285597493 >> v2:172.21.15.189:6800/30168 conn(0x7f0fec037780 0x7f0fec039c40 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f100c133c90 tx=0x7f10040094f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:26.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.773+0000 7f10037fe640 1 -- 172.21.15.40:0/1285597493 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc0057c0 con 0x7f100c131d70 2024-04-22T08:20:26.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.773+0000 7f10037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:26.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.773+0000 7f10037fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:20:26.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.773+0000 7f10037fe640 10 monclient: dump: 2024-04-22T08:20:26.785 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:26.785 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:26.785 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:26.785 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:26.785 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:26.785 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:26.785 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:26.785 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:26.786 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:26.786 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:26.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.773+0000 7f10037fe640 1 -- 172.21.15.40:0/1285597493 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f0ffc005920 con 0x7f100c131d70 2024-04-22T08:20:26.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.773+0000 7f10037fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:20:26.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.773+0000 7f10037fe640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:20:26.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.773+0000 7f10037fe640 1 -- 172.21.15.40:0/1285597493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f100c131d70 msgr2=0x7f100c132150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:26.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.773+0000 7f10037fe640 1 --2- 172.21.15.40:0/1285597493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f100c131d70 0x7f100c132150 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc00c380 tx=0x7f0ffc00c840 comp rx=0 tx=0).stop 2024-04-22T08:20:26.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.773+0000 7f10037fe640 1 -- 172.21.15.40:0/1285597493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f100c131d70 msgr2=0x7f100c132150 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:20:26.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.773+0000 7f10037fe640 1 --2- 172.21.15.40:0/1285597493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f100c131d70 0x7f100c132150 secure :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc00c380 tx=0x7f0ffc00c840 comp rx=0 tx=0).stop 2024-04-22T08:20:26.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.779+0000 7f10116ce640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:20:26.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.779+0000 7f10116ce640 1 --2- 172.21.15.40:0/1285597493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f100c06e730 0x7f100c06eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:26.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.779+0000 7f10116ce640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:26.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.779+0000 7f10116ce640 1 -- 172.21.15.40:0/1285597493 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f100c134b40 con 0x7f100c06e730 2024-04-22T08:20:26.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.779+0000 7f100affd640 1 --2- 172.21.15.40:0/1285597493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f100c06e730 0x7f100c06eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:26.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.779+0000 7f100affd640 10 monclient: get_auth_request con 0x7f100c06e730 auth_method 0 2024-04-22T08:20:26.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.779+0000 7f100affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:26.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.779+0000 7f100affd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:26.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.779+0000 7f100affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:26.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.780+0000 7f100affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:26.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.780+0000 7f100affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:26.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.780+0000 7f100affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:26.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.780+0000 7f100affd640 10 monclient(hunting): handle_auth_done global_id 64735 payload 824 2024-04-22T08:20:26.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.780+0000 7f100affd640 1 -- 172.21.15.40:0/1285597493 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f100c04e9b0 con 0x7f100c06e730 2024-04-22T08:20:26.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.782+0000 7f100affd640 1 -- 172.21.15.40:0/1285597493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f100c06e730 msgr2=0x7f100c06eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:26.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.782+0000 7f100affd640 1 --2- 172.21.15.40:0/1285597493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f100c06e730 0x7f100c06eb40 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc00d010 tx=0x7f0ffc0037e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:26.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.783+0000 7f10037fe640 1 -- 172.21.15.40:0/1285597493 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc013b00 con 0x7f100c06e730 2024-04-22T08:20:26.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.783+0000 7f10037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:26.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.783+0000 7f10037fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:20:26.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.783+0000 7f10037fe640 10 monclient: dump: 2024-04-22T08:20:26.788 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:26.788 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:26.788 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:26.788 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:26.788 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:26.788 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:26.788 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:26.788 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:26.788 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:26.788 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:26.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.783+0000 7f10037fe640 1 -- 172.21.15.40:0/1285597493 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4790 (secure 0 0 0) 0x7f0ffc013c60 con 0x7f100c06e730 2024-04-22T08:20:26.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.783+0000 7f10037fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:20:26.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.783+0000 7f10037fe640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:20:26.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.783+0000 7f10037fe640 1 -- 172.21.15.40:0/1285597493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f100c06e730 msgr2=0x7f100c06eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:26.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.783+0000 7f10037fe640 1 --2- 172.21.15.40:0/1285597493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f100c06e730 0x7f100c06eb40 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc00d010 tx=0x7f0ffc0037e0 comp rx=0 tx=0).stop 2024-04-22T08:20:26.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.783+0000 7f10037fe640 1 -- 172.21.15.40:0/1285597493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f100c06e730 msgr2=0x7f100c06eb40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:20:26.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.783+0000 7f10037fe640 1 --2- 172.21.15.40:0/1285597493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f100c06e730 0x7f100c06eb40 secure :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc00d010 tx=0x7f0ffc0037e0 comp rx=0 tx=0).stop 2024-04-22T08:20:26.789 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:20:26.789 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:20:26.789 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:20:26.789 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:20:26.789 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 76, 2024-04-22T08:20:26.789 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:20:26.789 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:20:26.789 INFO:teuthology.orchestra.run.smithi040.stdout: 1 2024-04-22T08:20:26.789 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:20:26.790 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 0, 2024-04-22T08:20:26.790 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:20:26.790 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:20:26.790 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:20:26.790 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:20:26.790 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:20:26.790 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:20:26.790 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:20:26.790 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:20:26.790 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:20:26.790 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:20:26.790 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:20:26.790 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:20:26.790 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:20:26.791 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:20:26.791 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:20:26.791 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:20:26.791 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:20:26.791 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:20:26.791 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:20:26.791 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:20:26.791 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:20:26.791 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:20:26.791 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:20:26.791 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:20:26.791 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:20:26.791 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:20:26.791 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:26.791 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:26.791 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:20:26.792 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:20:26.792 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:20:26.792 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:20:26.792 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:20:26.792 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:20:26.792 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:20:26.792 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:20:26.792 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:20:26.792 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:20:26.792 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:20:26.792 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:20:26.792 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:20:26.792 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:20:26.792 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:20:26.793 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:20:26.793 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:20:26.793 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:20:26.793 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:20:26.793 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:20:26.793 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:20:26.793 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:20:26.793 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:20:26.793 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:20:26.793 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:20:26.793 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:20:26.793 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:20:26.793 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:20:26.793 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:20:26.793 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:26.794 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:20:26.794 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:26.794 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:20:26.794 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:20:26.794 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:20:26.794 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:20:26.794 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:26.859 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:20:26.860 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:20:26.860 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:26.860 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:26.860 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:26.860 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:20:26.860 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:20:26.860 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:26.860 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:26.860 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:26.860 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:26.860 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:20:26.860 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:20:26.861 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:20:26.861 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:20:26.861 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:20:26.861 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:26.861 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:26.861 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:20:26.861 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:20:26.861 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:20:26.861 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:20:26.861 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:26.861 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:20:26.861 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:20:26.861 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:26.861 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:26.861 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:26.862 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:20:26.862 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:20:26.862 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:26.862 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:26.862 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:26.862 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:26.862 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:20:26.862 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:20:26.862 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:20:26.862 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:20:26.862 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:20:26.862 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:26.862 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:26.862 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:20:26.863 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:20:26.863 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:20:26.863 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:20:26.863 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:26.863 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:20:26.863 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:20:26.863 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:26.863 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:26.863 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:26.863 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:20:26.863 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:20:26.863 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:20:26.863 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:26.864 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:26.864 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:26.864 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:20:26.864 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:20:26.864 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:20:26.864 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:20:26.864 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:20:26.864 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:26.864 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:26.864 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:26.864 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:20:26.864 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:20:26.864 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:26.864 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:20:26.865 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:20:26.865 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:20:26.865 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:26.865 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:20:26.865 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:20:26.865 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:26.865 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:20:26.865 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:20:26.865 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 4 2024-04-22T08:20:26.865 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:26.865 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:20:26.865 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:20:26.865 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:26.865 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:20:26.865 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:20:26.866 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 4 2024-04-22T08:20:26.866 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:26.866 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:20:26.866 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": [ 2024-04-22T08:20:26.866 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:20:26.866 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:20:26.866 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:20:26.866 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:20:26.866 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:20:26.866 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:20:26.866 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:20:26.866 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:20:26.866 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:20:26.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.783+0000 7f10116ce640 1 -- 172.21.15.40:0/1285597493 >> v2:172.21.15.189:6800/30168 conn(0x7f0fec037780 msgr2=0x7f0fec039c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:26.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.783+0000 7f10116ce640 1 --2- 172.21.15.40:0/1285597493 >> v2:172.21.15.189:6800/30168 conn(0x7f0fec037780 0x7f0fec039c40 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f100c133c90 tx=0x7f10040094f0 comp rx=0 tx=0).stop 2024-04-22T08:20:26.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.783+0000 7f10116ce640 10 monclient: shutdown 2024-04-22T08:20:26.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.783+0000 7f10116ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:20:26.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.783+0000 7f10116ce640 1 -- 172.21.15.40:0/1285597493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f100c135c30 msgr2=0x7f100c1312f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:26.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.783+0000 7f10116ce640 1 --2- 172.21.15.40:0/1285597493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f100c135c30 0x7f100c1312f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0ff4009870 tx=0x7f0ff40046c0 comp rx=0 tx=0).stop 2024-04-22T08:20:26.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.783+0000 7f100b7fe640 1 -- 172.21.15.40:0/1285597493 reap_dead start 2024-04-22T08:20:26.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.783+0000 7f100b7fe640 1 -- 172.21.15.40:0/1285597493 reap_dead start 2024-04-22T08:20:26.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.783+0000 7f10116ce640 1 -- 172.21.15.40:0/1285597493 shutdown_connections 2024-04-22T08:20:26.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.783+0000 7f10116ce640 1 -- 172.21.15.40:0/1285597493 >> 172.21.15.40:0/1285597493 conn(0x7f100c0a0ae0 msgr2=0x7f100c09b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:20:26.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.783+0000 7f10116ce640 1 -- 172.21.15.40:0/1285597493 shutdown_connections 2024-04-22T08:20:26.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:26.783+0000 7f10116ce640 1 -- 172.21.15.40:0/1285597493 wait complete. 2024-04-22T08:20:26.870 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-22T08:20:26.870 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-04-22T08:20:28.310 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38379]: starting. 2024-04-22T08:20:28.310 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38379]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-38378 2024-04-22T08:20:28.310 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38379]: created object 0... 2024-04-22T08:20:28.310 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38379]: created object 25... 2024-04-22T08:20:28.310 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38379]: created object 49... 2024-04-22T08:20:28.310 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38379]: finishing. 2024-04-22T08:20:28.310 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38379]: shutting down. 2024-04-22T08:20:29.180 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[38380]: starting. 2024-04-22T08:20:29.180 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[38380]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-38378 2024-04-22T08:20:29.180 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[38380]: shutting down. 2024-04-22T08:20:31.403 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:20:31.404 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38413]: starting. 2024-04-22T08:20:31.404 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38413]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-38378 2024-04-22T08:20:31.404 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38413]: created object 0... 2024-04-22T08:20:31.404 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38413]: created object 25... 2024-04-22T08:20:31.404 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38413]: created object 49... 2024-04-22T08:20:31.404 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38413]: finishing. 2024-04-22T08:20:31.404 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38413]: shutting down. 2024-04-22T08:20:31.445 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:20:31.445 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38415]: starting. 2024-04-22T08:20:31.445 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38415]: listing objects. 2024-04-22T08:20:31.445 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38415]: listed object 0... 2024-04-22T08:20:31.445 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38415]: listed object 25... 2024-04-22T08:20:31.445 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38415]: saw 50 objects 2024-04-22T08:20:31.445 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38415]: shutting down. 2024-04-22T08:20:32.077 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:20:32.078 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[38414]: starting. 2024-04-22T08:20:32.078 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[38414]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-38378 2024-04-22T08:20:32.078 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[38414]: shutting down. 2024-04-22T08:20:32.079 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:20:32.079 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:20:32.079 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:20:32.081 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:20:34.220 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38482]: starting. 2024-04-22T08:20:34.220 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38482]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-38481 2024-04-22T08:20:34.220 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38482]: created object 0... 2024-04-22T08:20:34.221 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38482]: created object 25... 2024-04-22T08:20:34.221 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38482]: created object 49... 2024-04-22T08:20:34.221 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38482]: finishing. 2024-04-22T08:20:34.221 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38482]: shutting down. 2024-04-22T08:20:35.092 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[38483]: starting. 2024-04-22T08:20:35.092 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[38483]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-38481 2024-04-22T08:20:35.092 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[38483]: shutting down. 2024-04-22T08:20:37.216 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:20:37.216 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38516]: starting. 2024-04-22T08:20:37.217 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38516]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-38481 2024-04-22T08:20:37.217 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38516]: created object 0... 2024-04-22T08:20:37.217 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38516]: created object 25... 2024-04-22T08:20:37.217 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38516]: created object 49... 2024-04-22T08:20:37.217 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38516]: finishing. 2024-04-22T08:20:37.217 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38516]: shutting down. 2024-04-22T08:20:37.262 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:20:37.263 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38518]: starting. 2024-04-22T08:20:37.263 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38518]: listing objects. 2024-04-22T08:20:37.263 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38518]: listed object 0... 2024-04-22T08:20:37.263 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38518]: listed object 25... 2024-04-22T08:20:37.263 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38518]: saw 50 objects 2024-04-22T08:20:37.263 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38518]: shutting down. 2024-04-22T08:20:38.108 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:20:38.108 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[38517]: starting. 2024-04-22T08:20:38.109 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[38517]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-38481 2024-04-22T08:20:38.109 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[38517]: shutting down. 2024-04-22T08:20:38.110 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:20:38.110 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:20:38.110 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:20:38.113 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:20:40.258 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38585]: starting. 2024-04-22T08:20:40.258 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38585]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-38584 2024-04-22T08:20:40.258 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38585]: created object 0... 2024-04-22T08:20:40.258 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38585]: created object 25... 2024-04-22T08:20:40.258 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38585]: created object 49... 2024-04-22T08:20:40.258 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38585]: finishing. 2024-04-22T08:20:40.258 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38585]: shutting down. 2024-04-22T08:20:41.132 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[38586]: starting. 2024-04-22T08:20:41.132 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[38586]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-38584 2024-04-22T08:20:41.132 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[38586]: shutting down. 2024-04-22T08:20:43.216 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:20:43.217 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38619]: starting. 2024-04-22T08:20:43.217 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38619]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-38584 2024-04-22T08:20:43.217 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38619]: created object 0... 2024-04-22T08:20:43.217 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38619]: created object 25... 2024-04-22T08:20:43.217 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38619]: created object 49... 2024-04-22T08:20:43.217 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38619]: finishing. 2024-04-22T08:20:43.217 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38619]: shutting down. 2024-04-22T08:20:43.258 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:20:43.258 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38621]: starting. 2024-04-22T08:20:43.258 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38621]: listing objects. 2024-04-22T08:20:43.258 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38621]: listed object 0... 2024-04-22T08:20:43.258 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38621]: listed object 25... 2024-04-22T08:20:43.258 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38621]: saw 50 objects 2024-04-22T08:20:43.259 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38621]: shutting down. 2024-04-22T08:20:44.087 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:20:44.087 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[38620]: starting. 2024-04-22T08:20:44.088 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[38620]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-38584 2024-04-22T08:20:44.088 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[38620]: shutting down. 2024-04-22T08:20:44.089 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:20:44.089 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:20:44.089 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:20:44.092 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:20:46.227 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38688]: starting. 2024-04-22T08:20:46.227 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38688]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-38687 2024-04-22T08:20:46.227 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38688]: created object 0... 2024-04-22T08:20:46.227 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38688]: created object 25... 2024-04-22T08:20:46.227 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38688]: created object 49... 2024-04-22T08:20:46.227 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38688]: finishing. 2024-04-22T08:20:46.227 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38688]: shutting down. 2024-04-22T08:20:46.871 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-04-22T08:20:46.871 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-04-22T08:20:46.871 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-22T08:20:46.872 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-22T08:20:46.874 INFO:tasks.ceph.mon.c:Started 2024-04-22T08:20:46.874 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-22T08:20:46.875 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:20:46.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.985+0000 7f415d2af640 10 monclient: get_monmap_and_config 2024-04-22T08:20:46.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.985+0000 7f415d2af640 10 monclient: build_initial_monmap 2024-04-22T08:20:46.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.985+0000 7f415d2af640 10 monclient: monmap: 2024-04-22T08:20:46.985 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:20:46.985 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:46.985 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:20:46.986104+0000 2024-04-22T08:20:46.985 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:20:46.986104+0000 2024-04-22T08:20:46.985 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:20:46.985 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:46.985 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:20:46.985 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:20:46.985 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:20:46.985 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:20:46.986 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:20:46.986 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:46.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.985+0000 7f415d2af640 1 Processor -- start 2024-04-22T08:20:46.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.985+0000 7f415d2af640 1 -- start start 2024-04-22T08:20:46.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.985+0000 7f415d2af640 10 monclient: init 2024-04-22T08:20:46.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f415d2af640 10 monclient: _reopen_session rank -1 2024-04-22T08:20:46.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f415d2af640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-04-22T08:20:46.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f415d2af640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f415809ee10 0x7f41580a32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:46.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f415d2af640 10 monclient(hunting): picked mon.noname-b con 0x7f415809ee10 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:20:46.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f415d2af640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f41580a1d30 0x7f41580a2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:46.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f415d2af640 10 monclient(hunting): picked mon.noname-c con 0x7f41580a1d30 addr v2:172.21.15.189:3300/0 2024-04-22T08:20:46.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f415d2af640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4158135d90 0x7f41581361c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:46.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f415d2af640 10 monclient(hunting): picked mon.noname-a con 0x7f4158135d90 addr v2:172.21.15.40:3300/0 2024-04-22T08:20:46.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f415d2af640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:46.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f415d2af640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f415809f1f0 con 0x7f4158135d90 2024-04-22T08:20:46.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f415d2af640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:46.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f415d2af640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41580a39c0 con 0x7f41580a1d30 2024-04-22T08:20:46.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f415d2af640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:46.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f415d2af640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f415809fd30 con 0x7f415809ee10 2024-04-22T08:20:46.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f415d2af640 10 monclient(hunting): _renew_subs 2024-04-22T08:20:46.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f415d2af640 10 monclient(hunting): authenticate will time out at 1320.980258s 2024-04-22T08:20:46.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f4156ffd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f415809ee10 0x7f41580a32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:46.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f41577fe640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4158135d90 0x7f41581361c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:46.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f4156ffd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f415809ee10 0x7f41580a32b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:53474/0 (socket says 172.21.15.40:53474) 2024-04-22T08:20:46.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f41577fe640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4158135d90 0x7f41581361c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50772/0 (socket says 172.21.15.40:50772) 2024-04-22T08:20:46.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f4156ffd640 1 -- 172.21.15.40:0/1977169448 learned_addr learned my addr 172.21.15.40:0/1977169448 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:20:46.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f4156ffd640 10 monclient(hunting): get_auth_request con 0x7f415809ee10 auth_method 0 2024-04-22T08:20:46.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f4156ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:46.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f4156ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:46.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f4156ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:46.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f41577fe640 10 monclient(hunting): get_auth_request con 0x7f4158135d90 auth_method 0 2024-04-22T08:20:46.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f41577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:46.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f41577fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:46.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f41577fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:46.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f41567fc640 1 --2- 172.21.15.40:0/1977169448 >> v2:172.21.15.189:3300/0 conn(0x7f41580a1d30 0x7f41580a2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:46.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f4156ffd640 1 -- 172.21.15.40:0/1977169448 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f415809ee10 msgr2=0x7f41580a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:20:46.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f4156ffd640 1 -- 172.21.15.40:0/1977169448 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f415809ee10 msgr2=0x7f41580a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:20:46.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f4156ffd640 1 --2- 172.21.15.40:0/1977169448 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f415809ee10 0x7f41580a32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:20:46.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f41577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:46.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f41577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:46.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f41577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:46.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f41567fc640 10 monclient(hunting): get_auth_request con 0x7f41580a1d30 auth_method 0 2024-04-22T08:20:46.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f4156ffd640 1 --2- 172.21.15.40:0/1977169448 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f415809ee10 0x7f41580a32b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:20:46.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f41567fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:46.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f41567fc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:46.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f41567fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:46.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f41567fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:46.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f41567fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:46.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f41567fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:46.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f41577fe640 10 monclient(hunting): handle_auth_done global_id 94278 payload 274 2024-04-22T08:20:46.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f41577fe640 1 -- 172.21.15.40:0/1977169448 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f415809ee10 msgr2=0x7f41580a32b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:20:46.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f41577fe640 1 --2- 172.21.15.40:0/1977169448 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f415809ee10 0x7f41580a32b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:46.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f41577fe640 1 -- 172.21.15.40:0/1977169448 >> v2:172.21.15.189:3300/0 conn(0x7f41580a1d30 msgr2=0x7f41580a2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:20:46.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f41577fe640 1 --2- 172.21.15.40:0/1977169448 >> v2:172.21.15.189:3300/0 conn(0x7f41580a1d30 0x7f41580a2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:46.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.986+0000 7f41577fe640 10 monclient: _finish_hunting 0 2024-04-22T08:20:46.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.987+0000 7f41577fe640 1 monclient: found mon.noname-a 2024-04-22T08:20:46.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.987+0000 7f41577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:20:46.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.987+0000 7f41577fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:20:46.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.987+0000 7f41577fe640 1 -- 172.21.15.40:0/1977169448 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41580a02f0 con 0x7f4158135d90 2024-04-22T08:20:46.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.989+0000 7f41577fe640 1 -- 172.21.15.40:0/1977169448 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4158135d90 msgr2=0x7f41581361c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:46.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.989+0000 7f41577fe640 1 --2- 172.21.15.40:0/1977169448 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4158135d90 0x7f41581361c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f415806c020 tx=0x7f415002bd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db71ded233f391ef server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:46.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.990+0000 7f4155ffb640 1 -- 172.21.15.40:0/1977169448 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f415002ccc0 con 0x7f4158135d90 2024-04-22T08:20:47.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.990+0000 7f4155ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:47.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.990+0000 7f4155ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:20:47.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.990+0000 7f4155ffb640 10 monclient: dump: 2024-04-22T08:20:47.000 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:47.000 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:47.000 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:47.000 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:47.000 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:47.000 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:47.000 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:47.000 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:47.000 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:47.000 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:47.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.990+0000 7f4155ffb640 10 monclient: _finish_auth 0 2024-04-22T08:20:47.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.990+0000 7f4155ffb640 10 monclient: _check_auth_tickets 2024-04-22T08:20:47.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.990+0000 7f4155ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:20:47.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.990+0000 7f415d2af640 5 monclient: authenticate success, global_id 94278 2024-04-22T08:20:47.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.990+0000 7f415d2af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:20:47.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.990+0000 7f4155ffb640 1 -- 172.21.15.40:0/1977169448 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f415002ce20 con 0x7f4158135d90 2024-04-22T08:20:47.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.990+0000 7f4155ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:20:47.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.990+0000 7f415d2af640 10 monclient: get_monmap_and_config success 2024-04-22T08:20:47.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.990+0000 7f4155ffb640 1 -- 172.21.15.40:0/1977169448 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4150034660 con 0x7f4158135d90 2024-04-22T08:20:47.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.990+0000 7f4155ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:47.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.990+0000 7f415d2af640 4 set_mon_vals no callback set 2024-04-22T08:20:47.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.990+0000 7f4155ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:20:47.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.990+0000 7f4155ffb640 10 monclient: dump: 2024-04-22T08:20:47.001 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:47.001 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:47.001 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:47.001 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:47.002 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:47.002 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:47.002 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:47.002 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:47.002 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:47.002 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:47.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.990+0000 7f415d2af640 10 monclient: shutdown 2024-04-22T08:20:47.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.990+0000 7f415d2af640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:20:47.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.990+0000 7f415d2af640 1 -- 172.21.15.40:0/1977169448 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4158135d90 msgr2=0x7f41581361c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:47.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.990+0000 7f415d2af640 1 --2- 172.21.15.40:0/1977169448 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4158135d90 0x7f41581361c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f415806c020 tx=0x7f415002bd00 comp rx=0 tx=0).stop 2024-04-22T08:20:47.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.990+0000 7f415d2af640 1 -- 172.21.15.40:0/1977169448 shutdown_connections 2024-04-22T08:20:47.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.990+0000 7f415d2af640 1 --2- 172.21.15.40:0/1977169448 >> v2:172.21.15.189:3300/0 conn(0x7f41580a1d30 0x7f41580a2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:47.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.990+0000 7f415d2af640 1 --2- 172.21.15.40:0/1977169448 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4158135d90 0x7f41581361c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:47.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.990+0000 7f415d2af640 1 --2- 172.21.15.40:0/1977169448 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f415809ee10 0x7f41580a32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:47.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.990+0000 7f415d2af640 1 -- 172.21.15.40:0/1977169448 >> 172.21.15.40:0/1977169448 conn(0x7f41580a4ef0 msgr2=0x7f41580a5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:20:47.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.990+0000 7f415d2af640 1 -- 172.21.15.40:0/1977169448 shutdown_connections 2024-04-22T08:20:47.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.990+0000 7f415d2af640 1 -- 172.21.15.40:0/1977169448 wait complete. 2024-04-22T08:20:47.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.991+0000 7f415d2af640 10 monclient: build_initial_monmap 2024-04-22T08:20:47.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.991+0000 7f415d2af640 10 monclient: monmap: 2024-04-22T08:20:47.003 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:20:47.003 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:20:47.003 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:20:47.003 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:20:47.003 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:20:47.003 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:47.003 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:20:47.003 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:20:47.003 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:20:47.003 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:47.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.991+0000 7f415d2af640 1 Processor -- start 2024-04-22T08:20:47.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.991+0000 7f415d2af640 1 -- start start 2024-04-22T08:20:47.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.991+0000 7f415d2af640 10 monclient: init 2024-04-22T08:20:47.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f415d2af640 10 monclient: _reopen_session rank -1 2024-04-22T08:20:47.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f415d2af640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:20:47.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f415d2af640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f415809ee10 0x7f41581cad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:47.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f415d2af640 10 monclient(hunting): picked mon.noname-a con 0x7f415809ee10 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:20:47.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f415d2af640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41580a1d30 0x7f41581cb260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:47.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f415d2af640 10 monclient(hunting): picked mon.noname-b con 0x7f41580a1d30 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:20:47.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f415d2af640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4158135d90 0x7f4158121160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:47.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f415d2af640 10 monclient(hunting): picked mon.noname-c con 0x7f4158135d90 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:20:47.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f415d2af640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:47.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f415d2af640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4158124840 con 0x7f415809ee10 2024-04-22T08:20:47.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f415d2af640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:47.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f415d2af640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4158123950 con 0x7f41580a1d30 2024-04-22T08:20:47.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f415d2af640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:47.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f415d2af640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41580a2730 con 0x7f4158135d90 2024-04-22T08:20:47.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f415d2af640 10 monclient(hunting): _renew_subs 2024-04-22T08:20:47.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f415d2af640 10 monclient(hunting): authenticate will time out at 1320.986243s 2024-04-22T08:20:47.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f4156ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f415809ee10 0x7f41581cad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:47.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f41577fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4158135d90 0x7f4158121160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:47.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f4156ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f415809ee10 0x7f41581cad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50774/0 (socket says 172.21.15.40:50774) 2024-04-22T08:20:47.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f41577fe640 10 monclient(hunting): get_auth_request con 0x7f4158135d90 auth_method 0 2024-04-22T08:20:47.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f4156ffd640 1 -- 172.21.15.40:0/813963293 learned_addr learned my addr 172.21.15.40:0/813963293 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:20:47.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f41577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:47.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f41577fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:47.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f41577fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:47.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f4156ffd640 10 monclient(hunting): get_auth_request con 0x7f415809ee10 auth_method 0 2024-04-22T08:20:47.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f4156ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:47.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f4156ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:47.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f4156ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:47.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f41577fe640 1 -- 172.21.15.40:0/813963293 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4158135d90 msgr2=0x7f4158121160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-22T08:20:47.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f41577fe640 1 -- 172.21.15.40:0/813963293 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4158135d90 msgr2=0x7f4158121160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:20:47.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f41577fe640 1 --2- 172.21.15.40:0/813963293 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4158135d90 0x7f4158121160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:20:47.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f4156ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:47.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f4156ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:47.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f4156ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:47.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f41577fe640 1 --2- 172.21.15.40:0/813963293 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4158135d90 0x7f4158121160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:20:47.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f41567fc640 1 --2- 172.21.15.40:0/813963293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41580a1d30 0x7f41581cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:47.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f41567fc640 10 monclient(hunting): get_auth_request con 0x7f41580a1d30 auth_method 0 2024-04-22T08:20:47.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f41567fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:47.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f41567fc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:47.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f41567fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:47.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f41567fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:47.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f41567fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:47.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f41567fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:47.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f4156ffd640 10 monclient(hunting): handle_auth_done global_id 94281 payload 824 2024-04-22T08:20:47.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f4156ffd640 1 -- 172.21.15.40:0/813963293 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4158135d90 msgr2=0x7f4158121160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:20:47.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f4156ffd640 1 --2- 172.21.15.40:0/813963293 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4158135d90 0x7f4158121160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:47.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f4156ffd640 1 -- 172.21.15.40:0/813963293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41580a1d30 msgr2=0x7f41581cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:47.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f4156ffd640 1 --2- 172.21.15.40:0/813963293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41580a1d30 0x7f41581cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:47.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f4156ffd640 10 monclient: _finish_hunting 0 2024-04-22T08:20:47.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f4156ffd640 1 monclient: found mon.noname-a 2024-04-22T08:20:47.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f4156ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:20:47.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f4156ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:20:47.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.992+0000 7f4156ffd640 1 -- 172.21.15.40:0/813963293 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4150009a70 con 0x7f415809ee10 2024-04-22T08:20:47.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.995+0000 7f4156ffd640 1 -- 172.21.15.40:0/813963293 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f415809ee10 msgr2=0x7f41581cad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:47.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.995+0000 7f4156ffd640 1 --2- 172.21.15.40:0/813963293 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f415809ee10 0x7f41581cad20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f414800e840 tx=0x7f414800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:47.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.995+0000 7f413f7fe640 1 -- 172.21.15.40:0/813963293 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4148004220 con 0x7f415809ee10 2024-04-22T08:20:47.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.995+0000 7f413f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:47.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.995+0000 7f413f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:20:47.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.995+0000 7f413f7fe640 10 monclient: dump: 2024-04-22T08:20:47.008 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:47.008 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:47.008 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:47.008 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:47.008 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:47.008 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:47.008 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:47.008 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:47.008 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:47.009 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:47.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.995+0000 7f413f7fe640 10 monclient: _finish_auth 0 2024-04-22T08:20:47.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.995+0000 7f413f7fe640 10 monclient: _check_auth_tickets 2024-04-22T08:20:47.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.995+0000 7f413f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:20:47.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.995+0000 7f413f7fe640 1 -- 172.21.15.40:0/813963293 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4148004380 con 0x7f415809ee10 2024-04-22T08:20:47.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.995+0000 7f415d2af640 5 monclient: authenticate success, global_id 94281 2024-04-22T08:20:47.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.995+0000 7f413f7fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:20:47.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.995+0000 7f413f7fe640 1 -- 172.21.15.40:0/813963293 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414800c910 con 0x7f415809ee10 2024-04-22T08:20:47.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.995+0000 7f413f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:47.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.995+0000 7f413f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:20:47.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.995+0000 7f413f7fe640 10 monclient: dump: 2024-04-22T08:20:47.009 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:47.009 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:47.009 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:47.009 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:47.010 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:47.010 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:47.010 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:47.010 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:47.010 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:47.010 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:47.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.995+0000 7f415d2af640 10 monclient: _renew_subs 2024-04-22T08:20:47.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.995+0000 7f415d2af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:20:47.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.995+0000 7f415d2af640 1 -- 172.21.15.40:0/813963293 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4158071750 con 0x7f415809ee10 2024-04-22T08:20:47.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.995+0000 7f4154ff9640 4 set_mon_vals no callback set 2024-04-22T08:20:47.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.995+0000 7f415d2af640 10 monclient: _renew_subs 2024-04-22T08:20:47.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.995+0000 7f415d2af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:20:47.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.995+0000 7f415d2af640 1 -- 172.21.15.40:0/813963293 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4158122d10 con 0x7f415809ee10 2024-04-22T08:20:47.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.996+0000 7f413f7fe640 1 -- 172.21.15.40:0/813963293 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f414800cab0 con 0x7f415809ee10 2024-04-22T08:20:47.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.996+0000 7f415d2af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:20:47.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.996+0000 7f415d2af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:20:47.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.996+0000 7f415d2af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:20:47.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.996+0000 7f415d2af640 1 -- 172.21.15.40:0/813963293 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4158124cb0 con 0x7f415809ee10 2024-04-22T08:20:47.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.996+0000 7f413f7fe640 1 --2- 172.21.15.40:0/813963293 >> v2:172.21.15.189:6800/30168 conn(0x7f41440377d0 0x7f4144039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:47.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.996+0000 7f413f7fe640 1 -- 172.21.15.40:0/813963293 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(263..263 src has 1..263) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f4148049f30 con 0x7f415809ee10 2024-04-22T08:20:47.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.996+0000 7f41567fc640 1 --2- 172.21.15.40:0/813963293 >> v2:172.21.15.189:6800/30168 conn(0x7f41440377d0 0x7f4144039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:47.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.996+0000 7f41567fc640 10 monclient: get_auth_request con 0x7f41440377d0 auth_method 0 2024-04-22T08:20:47.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:46.999+0000 7f41567fc640 1 --2- 172.21.15.40:0/813963293 >> v2:172.21.15.189:6800/30168 conn(0x7f41440377d0 0x7f4144039c90 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f41580a4320 tx=0x7f414c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:47.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:47.001+0000 7f413f7fe640 1 -- 172.21.15.40:0/813963293 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f414800cd60 con 0x7f415809ee10 2024-04-22T08:20:47.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:47.001+0000 7f413f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:20:47.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:47.001+0000 7f413f7fe640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:20:47.103 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[38689]: starting. 2024-04-22T08:20:47.103 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[38689]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-38687 2024-04-22T08:20:47.103 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[38689]: shutting down. 2024-04-22T08:20:47.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:47.260+0000 7f415d2af640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:20:47.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:47.260+0000 7f415d2af640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:20:47.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:47.260+0000 7f415d2af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:20:47.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:47.260+0000 7f415d2af640 1 -- 172.21.15.40:0/813963293 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f415804e160 con 0x7f415809ee10 2024-04-22T08:20:47.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:47.260+0000 7f413f7fe640 1 -- 172.21.15.40:0/813963293 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7f4148022300 con 0x7f415809ee10 2024-04-22T08:20:47.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:47.260+0000 7f413f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:20:47.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:47.260+0000 7f413f7fe640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:20:47.260 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":76,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:20:47.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:47.262+0000 7f415d2af640 1 -- 172.21.15.40:0/813963293 >> v2:172.21.15.189:6800/30168 conn(0x7f41440377d0 msgr2=0x7f4144039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:47.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:47.262+0000 7f415d2af640 1 --2- 172.21.15.40:0/813963293 >> v2:172.21.15.189:6800/30168 conn(0x7f41440377d0 0x7f4144039c90 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f41580a4320 tx=0x7f414c009190 comp rx=0 tx=0).stop 2024-04-22T08:20:47.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:47.262+0000 7f415d2af640 10 monclient: shutdown 2024-04-22T08:20:47.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:47.262+0000 7f415d2af640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:20:47.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:47.262+0000 7f415d2af640 1 -- 172.21.15.40:0/813963293 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f415809ee10 msgr2=0x7f41581cad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:47.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:47.262+0000 7f415d2af640 1 --2- 172.21.15.40:0/813963293 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f415809ee10 0x7f41581cad20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f414800e840 tx=0x7f414800ed00 comp rx=0 tx=0).stop 2024-04-22T08:20:47.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:47.262+0000 7f415d2af640 1 -- 172.21.15.40:0/813963293 shutdown_connections 2024-04-22T08:20:47.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:47.262+0000 7f415d2af640 1 --2- 172.21.15.40:0/813963293 >> v2:172.21.15.189:6800/30168 conn(0x7f41440377d0 0x7f4144039c90 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:47.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:47.262+0000 7f415d2af640 1 --2- 172.21.15.40:0/813963293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41580a1d30 0x7f41581cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:47.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:47.262+0000 7f415d2af640 1 --2- 172.21.15.40:0/813963293 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4158135d90 0x7f4158121160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:47.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:47.262+0000 7f415d2af640 1 --2- 172.21.15.40:0/813963293 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f415809ee10 0x7f41581cad20 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:47.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:47.262+0000 7f415d2af640 1 -- 172.21.15.40:0/813963293 >> 172.21.15.40:0/813963293 conn(0x7f41580a4ef0 msgr2=0x7f41580a52d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:20:47.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:47.262+0000 7f415d2af640 1 -- 172.21.15.40:0/813963293 shutdown_connections 2024-04-22T08:20:47.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:47.262+0000 7f415d2af640 1 -- 172.21.15.40:0/813963293 wait complete. 2024-04-22T08:20:49.247 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:20:49.247 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38722]: starting. 2024-04-22T08:20:49.247 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38722]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-38687 2024-04-22T08:20:49.247 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38722]: created object 0... 2024-04-22T08:20:49.247 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38722]: created object 25... 2024-04-22T08:20:49.247 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38722]: created object 49... 2024-04-22T08:20:49.247 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38722]: finishing. 2024-04-22T08:20:49.247 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38722]: shutting down. 2024-04-22T08:20:49.287 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:20:49.287 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38724]: starting. 2024-04-22T08:20:49.287 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38724]: listing objects. 2024-04-22T08:20:49.287 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38724]: listed object 0... 2024-04-22T08:20:49.288 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38724]: listed object 25... 2024-04-22T08:20:49.288 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38724]: saw 50 objects 2024-04-22T08:20:49.288 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38724]: shutting down. 2024-04-22T08:20:50.116 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:20:50.116 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[38723]: starting. 2024-04-22T08:20:50.116 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[38723]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-38687 2024-04-22T08:20:50.116 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[38723]: shutting down. 2024-04-22T08:20:50.117 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:20:50.118 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:20:50.118 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:20:50.120 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:20:50.274 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:20:50.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.383+0000 7f295e08e640 10 monclient: get_monmap_and_config 2024-04-22T08:20:50.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.383+0000 7f295e08e640 10 monclient: build_initial_monmap 2024-04-22T08:20:50.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.383+0000 7f295e08e640 10 monclient: monmap: 2024-04-22T08:20:50.382 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:20:50.383 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:50.383 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:20:50.383846+0000 2024-04-22T08:20:50.383 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:20:50.383846+0000 2024-04-22T08:20:50.383 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:20:50.383 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:50.383 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:20:50.383 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:20:50.383 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:20:50.383 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:20:50.383 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:20:50.383 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:50.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.383+0000 7f295e08e640 1 Processor -- start 2024-04-22T08:20:50.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.383+0000 7f295e08e640 1 -- start start 2024-04-22T08:20:50.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.383+0000 7f295e08e640 10 monclient: init 2024-04-22T08:20:50.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.384+0000 7f295e08e640 10 monclient: _reopen_session rank -1 2024-04-22T08:20:50.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.384+0000 7f295e08e640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-04-22T08:20:50.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.384+0000 7f295e08e640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f295809ee10 0x7f29580a32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:50.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.384+0000 7f295e08e640 10 monclient(hunting): picked mon.noname-c con 0x7f295809ee10 addr v2:172.21.15.189:3300/0 2024-04-22T08:20:50.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.384+0000 7f295e08e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f29580a1d30 addr v1:172.21.15.40:6789/0 2024-04-22T08:20:50.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.384+0000 7f295e08e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f29580a2180 addr v1:172.21.15.189:6789/0 2024-04-22T08:20:50.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.384+0000 7f295e08e640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29580a2590 con 0x7f29580a1d30 2024-04-22T08:20:50.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.384+0000 7f295e08e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f295809f1f0 con 0x7f29580a2180 2024-04-22T08:20:50.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.384+0000 7f295e08e640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:50.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.384+0000 7f295e08e640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2958136bb0 con 0x7f295809ee10 2024-04-22T08:20:50.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.384+0000 7f295e08e640 10 monclient(hunting): _renew_subs 2024-04-22T08:20:50.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.384+0000 7f295e08e640 10 monclient(hunting): authenticate will time out at 1324.378254s 2024-04-22T08:20:50.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.384+0000 7f2956ffd640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f29580a1d30 0x7f2958135d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:56480/0 (socket says 172.21.15.40:56480) 2024-04-22T08:20:50.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.384+0000 7f2956ffd640 1 -- 172.21.15.40:0/897934275 learned_addr learned my addr 172.21.15.40:0/897934275 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:20:50.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.384+0000 7f29577fe640 1 --2- 172.21.15.40:0/897934275 >> v2:172.21.15.189:3300/0 conn(0x7f295809ee10 0x7f29580a32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:50.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.384+0000 7f29577fe640 10 monclient(hunting): get_auth_request con 0x7f295809ee10 auth_method 0 2024-04-22T08:20:50.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.384+0000 7f29577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:50.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.384+0000 7f29577fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:50.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.384+0000 7f29577fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:50.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.384+0000 7f29577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:50.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.384+0000 7f29577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:50.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.384+0000 7f29577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:50.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.385+0000 7f29577fe640 10 monclient(hunting): handle_auth_done global_id 64864 payload 274 2024-04-22T08:20:50.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.385+0000 7f29577fe640 1 -- 172.21.15.40:0/897934275 >> v1:172.21.15.189:6789/0 conn(0x7f29580a2180 legacy=0x7f29581364a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:20:50.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.385+0000 7f29577fe640 1 -- 172.21.15.40:0/897934275 >> v1:172.21.15.40:6789/0 conn(0x7f29580a1d30 legacy=0x7f2958135d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:20:50.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.385+0000 7f29577fe640 10 monclient: _finish_hunting 0 2024-04-22T08:20:50.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.385+0000 7f29577fe640 1 monclient: found mon.noname-c 2024-04-22T08:20:50.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.385+0000 7f29577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:20:50.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.385+0000 7f29577fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.189:3300/0 2024-04-22T08:20:50.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.385+0000 7f29577fe640 1 -- 172.21.15.40:0/897934275 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29580a39c0 con 0x7f295809ee10 2024-04-22T08:20:50.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.388+0000 7f29577fe640 1 -- 172.21.15.40:0/897934275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f295809ee10 msgr2=0x7f29580a32b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:50.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.388+0000 7f29577fe640 1 --2- 172.21.15.40:0/897934275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f295809ee10 0x7f29580a32b0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f295809f380 tx=0x7f294802b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=109b21c6af20a990 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:50.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.388+0000 7f29567fc640 1 -- 172.21.15.40:0/897934275 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2948030a00 con 0x7f295809ee10 2024-04-22T08:20:50.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.388+0000 7f29567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:50.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.388+0000 7f29567fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:20:50.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.388+0000 7f29567fc640 10 monclient: dump: 2024-04-22T08:20:50.392 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:50.392 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:50.392 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:50.392 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:50.392 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:50.392 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:50.392 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:50.392 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:50.392 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:50.392 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:50.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.388+0000 7f29567fc640 10 monclient: _finish_auth 0 2024-04-22T08:20:50.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.388+0000 7f29567fc640 10 monclient: _check_auth_tickets 2024-04-22T08:20:50.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.388+0000 7f29567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:20:50.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.388+0000 7f29567fc640 1 -- 172.21.15.40:0/897934275 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f294802e050 con 0x7f295809ee10 2024-04-22T08:20:50.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.388+0000 7f29567fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:20:50.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.388+0000 7f29567fc640 1 -- 172.21.15.40:0/897934275 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2948030d80 con 0x7f295809ee10 2024-04-22T08:20:50.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.388+0000 7f295e08e640 5 monclient: authenticate success, global_id 64864 2024-04-22T08:20:50.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.388+0000 7f295e08e640 10 monclient: get_monmap_and_config success 2024-04-22T08:20:50.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.388+0000 7f295e08e640 4 set_mon_vals no callback set 2024-04-22T08:20:50.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.388+0000 7f29567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:50.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.388+0000 7f29567fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:20:50.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.388+0000 7f29567fc640 10 monclient: dump: 2024-04-22T08:20:50.393 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:50.393 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:50.394 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:50.394 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:50.394 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:50.394 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:50.394 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:50.394 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:50.394 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:50.394 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:50.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.388+0000 7f295e08e640 10 monclient: shutdown 2024-04-22T08:20:50.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.388+0000 7f295e08e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:20:50.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.388+0000 7f295e08e640 1 -- 172.21.15.40:0/897934275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f295809ee10 msgr2=0x7f29580a32b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:50.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.388+0000 7f295e08e640 1 --2- 172.21.15.40:0/897934275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f295809ee10 0x7f29580a32b0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f295809f380 tx=0x7f294802b710 comp rx=0 tx=0).stop 2024-04-22T08:20:50.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.389+0000 7f295e08e640 1 -- 172.21.15.40:0/897934275 shutdown_connections 2024-04-22T08:20:50.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.389+0000 7f295e08e640 1 --2- 172.21.15.40:0/897934275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f295809ee10 0x7f29580a32b0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:50.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.389+0000 7f295e08e640 1 -- 172.21.15.40:0/897934275 >> 172.21.15.40:0/897934275 conn(0x7f29580a4ef0 msgr2=0x7f29580a5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:20:50.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.389+0000 7f295e08e640 1 -- 172.21.15.40:0/897934275 shutdown_connections 2024-04-22T08:20:50.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.389+0000 7f295e08e640 1 -- 172.21.15.40:0/897934275 wait complete. 2024-04-22T08:20:50.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.389+0000 7f295e08e640 10 monclient: build_initial_monmap 2024-04-22T08:20:50.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.389+0000 7f295e08e640 10 monclient: monmap: 2024-04-22T08:20:50.395 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:20:50.395 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:20:50.395 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:20:50.395 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:20:50.395 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:20:50.395 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:50.395 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:20:50.395 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:20:50.396 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:20:50.396 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:50.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.390+0000 7f295e08e640 1 Processor -- start 2024-04-22T08:20:50.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.390+0000 7f295e08e640 1 -- start start 2024-04-22T08:20:50.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.390+0000 7f295e08e640 10 monclient: init 2024-04-22T08:20:50.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.390+0000 7f295e08e640 10 monclient: _reopen_session rank -1 2024-04-22T08:20:50.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.390+0000 7f295e08e640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:20:50.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.390+0000 7f295e08e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f295809ee10 0x7f29581cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:50.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.390+0000 7f295e08e640 10 monclient(hunting): picked mon.noname-a con 0x7f295809ee10 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:20:50.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.390+0000 7f295e08e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29580a1d30 0x7f29581cb350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:50.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.390+0000 7f295e08e640 10 monclient(hunting): picked mon.noname-b con 0x7f29580a1d30 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:20:50.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.390+0000 7f295e08e640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29580a2180 0x7f29581234a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:50.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.390+0000 7f295e08e640 10 monclient(hunting): picked mon.noname-c con 0x7f29580a2180 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:20:50.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.390+0000 7f295e08e640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:50.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.390+0000 7f295e08e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2958071810 con 0x7f295809ee10 2024-04-22T08:20:50.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.390+0000 7f295e08e640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:50.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.390+0000 7f295e08e640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2958125530 con 0x7f29580a1d30 2024-04-22T08:20:50.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.390+0000 7f295e08e640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:50.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.390+0000 7f295e08e640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2958125b80 con 0x7f29580a2180 2024-04-22T08:20:50.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.390+0000 7f295e08e640 10 monclient(hunting): _renew_subs 2024-04-22T08:20:50.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.390+0000 7f295e08e640 10 monclient(hunting): authenticate will time out at 1324.384519s 2024-04-22T08:20:50.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.390+0000 7f29577fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f295809ee10 0x7f29581cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:50.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.390+0000 7f29577fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f295809ee10 0x7f29581cae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57242/0 (socket says 172.21.15.40:57242) 2024-04-22T08:20:50.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.390+0000 7f29577fe640 1 -- 172.21.15.40:0/3565097578 learned_addr learned my addr 172.21.15.40:0/3565097578 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:20:50.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.390+0000 7f29577fe640 10 monclient(hunting): get_auth_request con 0x7f295809ee10 auth_method 0 2024-04-22T08:20:50.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.390+0000 7f29577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:50.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.390+0000 7f29577fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:50.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.390+0000 7f29577fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:50.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.390+0000 7f2957fff640 1 --2- 172.21.15.40:0/3565097578 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29580a2180 0x7f29581234a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:50.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.390+0000 7f2957fff640 10 monclient(hunting): get_auth_request con 0x7f29580a2180 auth_method 0 2024-04-22T08:20:50.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.390+0000 7f2957fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:50.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.390+0000 7f2957fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:50.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.390+0000 7f2957fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:50.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.390+0000 7f29577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:50.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.390+0000 7f29577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:50.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.391+0000 7f29577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:50.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.391+0000 7f2957fff640 1 -- 172.21.15.40:0/3565097578 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29580a2180 msgr2=0x7f29581234a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-22T08:20:50.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.391+0000 7f2957fff640 1 -- 172.21.15.40:0/3565097578 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29580a2180 msgr2=0x7f29581234a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:20:50.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.391+0000 7f2957fff640 1 --2- 172.21.15.40:0/3565097578 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29580a2180 0x7f29581234a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:20:50.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.391+0000 7f2957fff640 1 --2- 172.21.15.40:0/3565097578 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29580a2180 0x7f29581234a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:20:50.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.391+0000 7f2956ffd640 1 --2- 172.21.15.40:0/3565097578 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29580a1d30 0x7f29581cb350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:50.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.391+0000 7f2956ffd640 10 monclient(hunting): get_auth_request con 0x7f29580a1d30 auth_method 0 2024-04-22T08:20:50.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.391+0000 7f2956ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:50.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.391+0000 7f2956ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:50.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.391+0000 7f2956ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:50.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.391+0000 7f29577fe640 10 monclient(hunting): handle_auth_done global_id 94308 payload 824 2024-04-22T08:20:50.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.391+0000 7f29577fe640 1 -- 172.21.15.40:0/3565097578 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29580a2180 msgr2=0x7f29581234a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:20:50.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.391+0000 7f29577fe640 1 --2- 172.21.15.40:0/3565097578 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29580a2180 0x7f29581234a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:50.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.391+0000 7f29577fe640 1 -- 172.21.15.40:0/3565097578 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29580a1d30 msgr2=0x7f29581cb350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:50.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.391+0000 7f29577fe640 1 --2- 172.21.15.40:0/3565097578 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29580a1d30 0x7f29581cb350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:50.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.391+0000 7f29577fe640 10 monclient: _finish_hunting 0 2024-04-22T08:20:50.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.391+0000 7f29577fe640 1 monclient: found mon.noname-a 2024-04-22T08:20:50.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.391+0000 7f29577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:20:50.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.391+0000 7f29577fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:20:50.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.391+0000 7f29577fe640 1 -- 172.21.15.40:0/3565097578 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2948009400 con 0x7f295809ee10 2024-04-22T08:20:50.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.393+0000 7f29577fe640 1 -- 172.21.15.40:0/3565097578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f295809ee10 msgr2=0x7f29581cae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:50.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.393+0000 7f29577fe640 1 --2- 172.21.15.40:0/3565097578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f295809ee10 0x7f29581cae10 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2948009890 tx=0x7f29480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:50.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.393+0000 7f293ffff640 1 -- 172.21.15.40:0/3565097578 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f294803db60 con 0x7f295809ee10 2024-04-22T08:20:50.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.393+0000 7f293ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:50.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.393+0000 7f293ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:20:50.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.393+0000 7f293ffff640 10 monclient: dump: 2024-04-22T08:20:50.402 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:50.402 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:50.402 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:50.402 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:50.402 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:50.403 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:50.403 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:50.403 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:50.403 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:50.403 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:50.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.393+0000 7f293ffff640 10 monclient: _finish_auth 0 2024-04-22T08:20:50.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.393+0000 7f293ffff640 10 monclient: _check_auth_tickets 2024-04-22T08:20:50.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.393+0000 7f293ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:20:50.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.393+0000 7f293ffff640 1 -- 172.21.15.40:0/3565097578 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f294803dcc0 con 0x7f295809ee10 2024-04-22T08:20:50.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.393+0000 7f295e08e640 5 monclient: authenticate success, global_id 94308 2024-04-22T08:20:50.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.393+0000 7f295e08e640 10 monclient: _renew_subs 2024-04-22T08:20:50.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.393+0000 7f295e08e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:20:50.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.393+0000 7f295e08e640 1 -- 172.21.15.40:0/3565097578 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29580a2730 con 0x7f295809ee10 2024-04-22T08:20:50.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.393+0000 7f293ffff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:20:50.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.393+0000 7f293ffff640 1 -- 172.21.15.40:0/3565097578 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29480326a0 con 0x7f295809ee10 2024-04-22T08:20:50.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.393+0000 7f293ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:50.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.393+0000 7f293ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:20:50.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.393+0000 7f293ffff640 10 monclient: dump: 2024-04-22T08:20:50.404 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:50.404 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:50.404 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:50.404 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:50.404 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:50.404 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:50.404 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:50.404 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:50.404 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:50.404 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:50.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.393+0000 7f295e08e640 10 monclient: _renew_subs 2024-04-22T08:20:50.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.393+0000 7f2955ffb640 4 set_mon_vals no callback set 2024-04-22T08:20:50.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.393+0000 7f295e08e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:20:50.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.393+0000 7f295e08e640 1 -- 172.21.15.40:0/3565097578 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2958124fb0 con 0x7f295809ee10 2024-04-22T08:20:50.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.394+0000 7f293ffff640 1 -- 172.21.15.40:0/3565097578 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f2948039040 con 0x7f295809ee10 2024-04-22T08:20:50.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.394+0000 7f295e08e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:20:50.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.394+0000 7f295e08e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:20:50.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.394+0000 7f295e08e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:20:50.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.394+0000 7f295e08e640 1 -- 172.21.15.40:0/3565097578 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2958126c00 con 0x7f295809ee10 2024-04-22T08:20:50.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.394+0000 7f293ffff640 1 --2- 172.21.15.40:0/3565097578 >> v2:172.21.15.189:6800/30168 conn(0x7f29440377d0 0x7f2944039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:50.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.394+0000 7f293ffff640 1 -- 172.21.15.40:0/3565097578 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(267..267 src has 1..267) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f294806c1f0 con 0x7f295809ee10 2024-04-22T08:20:50.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.395+0000 7f2956ffd640 1 --2- 172.21.15.40:0/3565097578 >> v2:172.21.15.189:6800/30168 conn(0x7f29440377d0 0x7f2944039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:50.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.395+0000 7f2956ffd640 10 monclient: get_auth_request con 0x7f29440377d0 auth_method 0 2024-04-22T08:20:50.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.397+0000 7f2956ffd640 1 --2- 172.21.15.40:0/3565097578 >> v2:172.21.15.189:6800/30168 conn(0x7f29440377d0 0x7f2944039c90 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f2958124c50 tx=0x7f294c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:50.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.399+0000 7f293ffff640 1 -- 172.21.15.40:0/3565097578 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f2948050d20 con 0x7f295809ee10 2024-04-22T08:20:50.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.399+0000 7f293ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:20:50.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.399+0000 7f293ffff640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:20:50.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.652+0000 7f295e08e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:20:50.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.652+0000 7f295e08e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:20:50.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.652+0000 7f295e08e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:20:50.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.652+0000 7f295e08e640 1 -- 172.21.15.40:0/3565097578 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f29580a2590 con 0x7f295809ee10 2024-04-22T08:20:50.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.652+0000 7f293ffff640 1 -- 172.21.15.40:0/3565097578 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7f29580a2590 con 0x7f295809ee10 2024-04-22T08:20:50.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.652+0000 7f293ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:20:50.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.652+0000 7f293ffff640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:20:50.652 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":76,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:20:50.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.654+0000 7f295e08e640 1 -- 172.21.15.40:0/3565097578 >> v2:172.21.15.189:6800/30168 conn(0x7f29440377d0 msgr2=0x7f2944039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:50.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.654+0000 7f295e08e640 1 --2- 172.21.15.40:0/3565097578 >> v2:172.21.15.189:6800/30168 conn(0x7f29440377d0 0x7f2944039c90 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f2958124c50 tx=0x7f294c009040 comp rx=0 tx=0).stop 2024-04-22T08:20:50.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.654+0000 7f295e08e640 10 monclient: shutdown 2024-04-22T08:20:50.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.654+0000 7f295e08e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:20:50.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.654+0000 7f295e08e640 1 -- 172.21.15.40:0/3565097578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f295809ee10 msgr2=0x7f29581cae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:50.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.654+0000 7f295e08e640 1 --2- 172.21.15.40:0/3565097578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f295809ee10 0x7f29581cae10 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2948009890 tx=0x7f29480046c0 comp rx=0 tx=0).stop 2024-04-22T08:20:50.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.654+0000 7f295e08e640 1 -- 172.21.15.40:0/3565097578 shutdown_connections 2024-04-22T08:20:50.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.654+0000 7f295e08e640 1 --2- 172.21.15.40:0/3565097578 >> v2:172.21.15.189:6800/30168 conn(0x7f29440377d0 0x7f2944039c90 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:50.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.654+0000 7f295e08e640 1 --2- 172.21.15.40:0/3565097578 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29580a1d30 0x7f29581cb350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:50.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.654+0000 7f295e08e640 1 --2- 172.21.15.40:0/3565097578 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29580a2180 0x7f29581234a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:50.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.654+0000 7f295e08e640 1 --2- 172.21.15.40:0/3565097578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f295809ee10 0x7f29581cae10 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:50.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.654+0000 7f295e08e640 1 -- 172.21.15.40:0/3565097578 >> 172.21.15.40:0/3565097578 conn(0x7f29580a4ef0 msgr2=0x7f29580a6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:20:50.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.654+0000 7f295e08e640 1 -- 172.21.15.40:0/3565097578 shutdown_connections 2024-04-22T08:20:50.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:50.654+0000 7f295e08e640 1 -- 172.21.15.40:0/3565097578 wait complete. 2024-04-22T08:20:52.207 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38791]: starting. 2024-04-22T08:20:52.208 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38791]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-38790 2024-04-22T08:20:52.208 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38791]: created object 0... 2024-04-22T08:20:52.208 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38791]: created object 25... 2024-04-22T08:20:52.208 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38791]: created object 49... 2024-04-22T08:20:52.208 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38791]: finishing. 2024-04-22T08:20:52.208 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38791]: shutting down. 2024-04-22T08:20:53.665 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:20:53.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:53.777+0000 7f554b20b640 10 monclient: get_monmap_and_config 2024-04-22T08:20:53.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:53.777+0000 7f554b20b640 10 monclient: build_initial_monmap 2024-04-22T08:20:53.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:53.777+0000 7f554b20b640 10 monclient: monmap: 2024-04-22T08:20:53.776 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:20:53.777 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:53.777 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:20:53.777899+0000 2024-04-22T08:20:53.777 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:20:53.777899+0000 2024-04-22T08:20:53.777 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:20:53.777 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:53.777 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:20:53.777 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:20:53.777 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:20:53.777 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:20:53.777 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:20:53.777 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:53.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:53.777+0000 7f554b20b640 1 Processor -- start 2024-04-22T08:20:53.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:53.777+0000 7f554b20b640 1 -- start start 2024-04-22T08:20:53.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:53.777+0000 7f554b20b640 10 monclient: init 2024-04-22T08:20:53.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:53.778+0000 7f554b20b640 10 monclient: _reopen_session rank -1 2024-04-22T08:20:53.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:53.778+0000 7f554b20b640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-04-22T08:20:53.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:53.778+0000 7f554b20b640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f554409ee10 0x7f55440a32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:53.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:53.778+0000 7f554b20b640 10 monclient(hunting): picked mon.noname-b con 0x7f554409ee10 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:20:53.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:53.778+0000 7f554b20b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f55440a1d30 addr v1:172.21.15.40:6789/0 2024-04-22T08:20:53.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:53.778+0000 7f554b20b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f55440a2150 addr v1:172.21.15.189:6789/0 2024-04-22T08:20:53.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:53.778+0000 7f554b20b640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55440a2580 con 0x7f55440a1d30 2024-04-22T08:20:53.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:53.778+0000 7f554b20b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f554409f1f0 con 0x7f55440a2150 2024-04-22T08:20:53.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:53.778+0000 7f554b20b640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:53.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:53.778+0000 7f554b20b640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5544136bb0 con 0x7f554409ee10 2024-04-22T08:20:53.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:53.778+0000 7f554b20b640 10 monclient(hunting): _renew_subs 2024-04-22T08:20:53.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:53.778+0000 7f554b20b640 10 monclient(hunting): authenticate will time out at 1327.772122s 2024-04-22T08:20:53.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:53.778+0000 7f5548fa7640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f554409ee10 0x7f55440a32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:53.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:53.778+0000 7f5543fff640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f55440a1d30 0x7f5544135d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:56494/0 (socket says 172.21.15.40:56494) 2024-04-22T08:20:53.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:53.778+0000 7f5543fff640 1 -- 172.21.15.40:0/1038544173 learned_addr learned my addr 172.21.15.40:0/1038544173 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:20:53.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:53.778+0000 7f5548fa7640 10 monclient(hunting): get_auth_request con 0x7f554409ee10 auth_method 0 2024-04-22T08:20:53.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:53.778+0000 7f5548fa7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:53.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:53.778+0000 7f5548fa7640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:53.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:53.778+0000 7f5548fa7640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:53.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:53.778+0000 7f5548fa7640 1 -- 172.21.15.40:0/1038544173 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f554409ee10 msgr2=0x7f55440a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:20:53.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:53.778+0000 7f5548fa7640 1 -- 172.21.15.40:0/1038544173 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f554409ee10 msgr2=0x7f55440a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:20:53.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:53.778+0000 7f5548fa7640 1 --2- 172.21.15.40:0/1038544173 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f554409ee10 0x7f55440a32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:20:53.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:53.778+0000 7f5548fa7640 1 --2- 172.21.15.40:0/1038544173 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f554409ee10 0x7f55440a32b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:20:53.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:53.780+0000 7f5543fff640 1 -- 172.21.15.40:0/1038544173 >> v1:172.21.15.40:6789/0 conn(0x7f55440a1d30 legacy=0x7f5544135d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:53.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:53.780+0000 7f55497a8640 1 -- 172.21.15.40:0/1038544173 >> v1:172.21.15.189:6789/0 conn(0x7f55440a2150 legacy=0x7f55441364a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:53.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:53.979+0000 7f5548fa7640 1 --2- 172.21.15.40:0/1038544173 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f554409ee10 0x7f55440a32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:53.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:53.979+0000 7f5548fa7640 10 monclient(hunting): get_auth_request con 0x7f554409ee10 auth_method 0 2024-04-22T08:20:53.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:53.979+0000 7f5548fa7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:53.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:53.979+0000 7f5548fa7640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:53.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:53.979+0000 7f5548fa7640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:20:53.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:53.979+0000 7f5548fa7640 1 -- 172.21.15.40:0/1038544173 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f554409ee10 msgr2=0x7f55440a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:20:53.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:53.979+0000 7f5548fa7640 1 -- 172.21.15.40:0/1038544173 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f554409ee10 msgr2=0x7f55440a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:20:53.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:53.979+0000 7f5548fa7640 1 --2- 172.21.15.40:0/1038544173 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f554409ee10 0x7f55440a32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:20:53.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:53.979+0000 7f5548fa7640 1 --2- 172.21.15.40:0/1038544173 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f554409ee10 0x7f55440a32b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:20:54.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:54.380+0000 7f5548fa7640 1 --2- 172.21.15.40:0/1038544173 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f554409ee10 0x7f55440a32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:54.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:54.380+0000 7f5548fa7640 10 monclient(hunting): get_auth_request con 0x7f554409ee10 auth_method 0 2024-04-22T08:20:54.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:54.380+0000 7f5548fa7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:54.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:54.380+0000 7f5548fa7640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:54.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:54.380+0000 7f5548fa7640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:20:54.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:54.380+0000 7f5548fa7640 1 -- 172.21.15.40:0/1038544173 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f554409ee10 msgr2=0x7f55440a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:20:54.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:54.380+0000 7f5548fa7640 1 -- 172.21.15.40:0/1038544173 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f554409ee10 msgr2=0x7f55440a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:20:54.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:54.380+0000 7f5548fa7640 1 --2- 172.21.15.40:0/1038544173 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f554409ee10 0x7f55440a32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:20:54.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:54.380+0000 7f5548fa7640 1 --2- 172.21.15.40:0/1038544173 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f554409ee10 0x7f55440a32b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:20:55.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:55.181+0000 7f5548fa7640 1 --2- 172.21.15.40:0/1038544173 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f554409ee10 0x7f55440a32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:55.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:55.181+0000 7f5548fa7640 10 monclient(hunting): get_auth_request con 0x7f554409ee10 auth_method 0 2024-04-22T08:20:55.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:55.181+0000 7f5548fa7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:55.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:55.181+0000 7f5548fa7640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:55.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:55.181+0000 7f5548fa7640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:20:55.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:55.181+0000 7f5548fa7640 1 -- 172.21.15.40:0/1038544173 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f554409ee10 msgr2=0x7f55440a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:20:55.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:55.181+0000 7f5548fa7640 1 -- 172.21.15.40:0/1038544173 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f554409ee10 msgr2=0x7f55440a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:20:55.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:55.181+0000 7f5548fa7640 1 --2- 172.21.15.40:0/1038544173 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f554409ee10 0x7f55440a32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:20:55.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:55.181+0000 7f5548fa7640 1 --2- 172.21.15.40:0/1038544173 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f554409ee10 0x7f55440a32b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T08:20:56.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.778+0000 7f55427fc640 10 monclient(hunting): tick 2024-04-22T08:20:56.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.778+0000 7f55427fc640 10 monclient(hunting): _check_auth_tickets 2024-04-22T08:20:56.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.778+0000 7f55427fc640 1 monclient(hunting): continuing hunt 2024-04-22T08:20:56.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.778+0000 7f55427fc640 10 monclient(hunting): _reopen_session rank -1 2024-04-22T08:20:56.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.778+0000 7f55427fc640 1 -- 172.21.15.40:0/1038544173 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f554409ee10 msgr2=0x7f55440a32b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:20:56.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.778+0000 7f55427fc640 1 --2- 172.21.15.40:0/1038544173 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f554409ee10 0x7f55440a32b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:56.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.778+0000 7f55427fc640 1 -- 172.21.15.40:0/1038544173 >> v1:172.21.15.189:6789/0 conn(0x7f55440a2150 legacy=0x7f55441364a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:56.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.778+0000 7f55427fc640 1 -- 172.21.15.40:0/1038544173 >> v1:172.21.15.40:6789/0 conn(0x7f55440a1d30 legacy=0x7f5544135d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:56.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.778+0000 7f55427fc640 10 monclient: _add_conns ranks=[4,3] 2024-04-22T08:20:56.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.778+0000 7f55427fc640 1 --2- 172.21.15.40:0/1038544173 >> v2:172.21.15.189:3300/0 conn(0x7f5530003810 0x7f5530005c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:56.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.778+0000 7f55427fc640 10 monclient(hunting): picked mon.noname-c con 0x7f5530003810 addr v2:172.21.15.189:3300/0 2024-04-22T08:20:56.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.778+0000 7f55427fc640 1 --2- 172.21.15.40:0/1038544173 >> v2:172.21.15.40:3300/0 conn(0x7f5530006360 0x7f5530008820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:56.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.778+0000 7f55427fc640 10 monclient(hunting): picked mon.noname-a con 0x7f5530006360 addr v2:172.21.15.40:3300/0 2024-04-22T08:20:56.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.778+0000 7f55427fc640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:56.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.778+0000 7f55427fc640 1 -- 172.21.15.40:0/1038544173 --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5544136bb0 con 0x7f5530006360 2024-04-22T08:20:56.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.778+0000 7f55427fc640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:56.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.778+0000 7f55427fc640 1 -- 172.21.15.40:0/1038544173 --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5530008eb0 con 0x7f5530003810 2024-04-22T08:20:56.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.778+0000 7f55427fc640 10 monclient(hunting): _renew_subs 2024-04-22T08:20:56.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.778+0000 7f5543fff640 1 --2- 172.21.15.40:0/1038544173 >> v2:172.21.15.40:3300/0 conn(0x7f5530006360 0x7f5530008820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:56.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.778+0000 7f5543fff640 10 monclient(hunting): get_auth_request con 0x7f5530006360 auth_method 0 2024-04-22T08:20:56.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.778+0000 7f5543fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:56.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.778+0000 7f5543fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:56.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.778+0000 7f5543fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:56.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.778+0000 7f5548fa7640 1 --2- 172.21.15.40:0/1038544173 >> v2:172.21.15.189:3300/0 conn(0x7f5530003810 0x7f5530005c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:56.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.779+0000 7f5548fa7640 10 monclient(hunting): get_auth_request con 0x7f5530003810 auth_method 0 2024-04-22T08:20:56.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.779+0000 7f5548fa7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:56.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.779+0000 7f5543fff640 1 -- 172.21.15.40:0/1038544173 >> v2:172.21.15.40:3300/0 conn(0x7f5530006360 msgr2=0x7f5530008820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:20:56.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.779+0000 7f5548fa7640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:56.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.779+0000 7f5548fa7640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:56.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.779+0000 7f5543fff640 1 -- 172.21.15.40:0/1038544173 >> v2:172.21.15.40:3300/0 conn(0x7f5530006360 msgr2=0x7f5530008820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:20:56.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.779+0000 7f5543fff640 1 --2- 172.21.15.40:0/1038544173 >> v2:172.21.15.40:3300/0 conn(0x7f5530006360 0x7f5530008820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:20:56.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.779+0000 7f5543fff640 1 --2- 172.21.15.40:0/1038544173 >> v2:172.21.15.40:3300/0 conn(0x7f5530006360 0x7f5530008820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:20:56.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.779+0000 7f5548fa7640 1 -- 172.21.15.40:0/1038544173 >> v2:172.21.15.189:3300/0 conn(0x7f5530003810 msgr2=0x7f5530005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:20:56.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.779+0000 7f5548fa7640 1 -- 172.21.15.40:0/1038544173 >> v2:172.21.15.189:3300/0 conn(0x7f5530003810 msgr2=0x7f5530005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:20:56.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.779+0000 7f5548fa7640 1 --2- 172.21.15.40:0/1038544173 >> v2:172.21.15.189:3300/0 conn(0x7f5530003810 0x7f5530005c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:20:56.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.779+0000 7f5548fa7640 1 --2- 172.21.15.40:0/1038544173 >> v2:172.21.15.189:3300/0 conn(0x7f5530003810 0x7f5530005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:20:56.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.980+0000 7f5543fff640 1 --2- 172.21.15.40:0/1038544173 >> v2:172.21.15.40:3300/0 conn(0x7f5530006360 0x7f5530008820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:56.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.980+0000 7f5543fff640 10 monclient(hunting): get_auth_request con 0x7f5530006360 auth_method 0 2024-04-22T08:20:56.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.980+0000 7f5543fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:56.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.980+0000 7f5543fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:56.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.980+0000 7f5543fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:20:56.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.980+0000 7f5548fa7640 1 --2- 172.21.15.40:0/1038544173 >> v2:172.21.15.189:3300/0 conn(0x7f5530003810 0x7f5530005c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:56.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.980+0000 7f5543fff640 1 -- 172.21.15.40:0/1038544173 >> v2:172.21.15.40:3300/0 conn(0x7f5530006360 msgr2=0x7f5530008820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:20:56.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.980+0000 7f5543fff640 1 -- 172.21.15.40:0/1038544173 >> v2:172.21.15.40:3300/0 conn(0x7f5530006360 msgr2=0x7f5530008820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:20:56.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.980+0000 7f5543fff640 1 --2- 172.21.15.40:0/1038544173 >> v2:172.21.15.40:3300/0 conn(0x7f5530006360 0x7f5530008820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:20:56.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.980+0000 7f5548fa7640 10 monclient(hunting): get_auth_request con 0x7f5530003810 auth_method 0 2024-04-22T08:20:56.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.980+0000 7f5548fa7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:56.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.980+0000 7f5548fa7640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:56.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.980+0000 7f5548fa7640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:20:56.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.980+0000 7f5543fff640 1 --2- 172.21.15.40:0/1038544173 >> v2:172.21.15.40:3300/0 conn(0x7f5530006360 0x7f5530008820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:20:56.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.980+0000 7f5548fa7640 1 -- 172.21.15.40:0/1038544173 >> v2:172.21.15.189:3300/0 conn(0x7f5530003810 msgr2=0x7f5530005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:20:56.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.980+0000 7f5548fa7640 1 -- 172.21.15.40:0/1038544173 >> v2:172.21.15.189:3300/0 conn(0x7f5530003810 msgr2=0x7f5530005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:20:56.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.980+0000 7f5548fa7640 1 --2- 172.21.15.40:0/1038544173 >> v2:172.21.15.189:3300/0 conn(0x7f5530003810 0x7f5530005c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:20:56.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:56.980+0000 7f5548fa7640 1 --2- 172.21.15.40:0/1038544173 >> v2:172.21.15.189:3300/0 conn(0x7f5530003810 0x7f5530005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:20:57.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.381+0000 7f5543fff640 1 --2- 172.21.15.40:0/1038544173 >> v2:172.21.15.40:3300/0 conn(0x7f5530006360 0x7f5530008820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:57.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.381+0000 7f5543fff640 10 monclient(hunting): get_auth_request con 0x7f5530006360 auth_method 0 2024-04-22T08:20:57.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.381+0000 7f5543fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:57.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.381+0000 7f5543fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:57.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.381+0000 7f5543fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:20:57.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.381+0000 7f5548fa7640 1 --2- 172.21.15.40:0/1038544173 >> v2:172.21.15.189:3300/0 conn(0x7f5530003810 0x7f5530005c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:57.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.381+0000 7f5548fa7640 10 monclient(hunting): get_auth_request con 0x7f5530003810 auth_method 0 2024-04-22T08:20:57.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.381+0000 7f5548fa7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:57.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.381+0000 7f5548fa7640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:57.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.381+0000 7f5548fa7640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:20:57.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.381+0000 7f5543fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:57.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.381+0000 7f5543fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:57.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.381+0000 7f5543fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:57.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.381+0000 7f5548fa7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:57.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.381+0000 7f5548fa7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:57.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.381+0000 7f5548fa7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:57.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.381+0000 7f5543fff640 10 monclient(hunting): handle_auth_done global_id 94311 payload 274 2024-04-22T08:20:57.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.381+0000 7f5543fff640 1 -- 172.21.15.40:0/1038544173 >> v2:172.21.15.189:3300/0 conn(0x7f5530003810 msgr2=0x7f5530005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:20:57.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.381+0000 7f5543fff640 1 --2- 172.21.15.40:0/1038544173 >> v2:172.21.15.189:3300/0 conn(0x7f5530003810 0x7f5530005c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:57.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.381+0000 7f5543fff640 10 monclient: _finish_hunting 0 2024-04-22T08:20:57.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.381+0000 7f5543fff640 1 monclient: found mon.noname-a 2024-04-22T08:20:57.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.381+0000 7f5543fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:20:57.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.381+0000 7f5543fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:20:57.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.381+0000 7f5543fff640 1 -- 172.21.15.40:0/1038544173 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55440a39c0 con 0x7f5530006360 2024-04-22T08:20:57.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.384+0000 7f5543fff640 1 -- 172.21.15.40:0/1038544173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5530006360 msgr2=0x7f5530008820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:57.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.384+0000 7f5543fff640 1 --2- 172.21.15.40:0/1038544173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5530006360 0x7f5530008820 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5530003780 tx=0x7f553802bd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cc0b0b52a21bd2d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:57.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.384+0000 7f55437fe640 1 -- 172.21.15.40:0/1038544173 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5538004b60 con 0x7f5530006360 2024-04-22T08:20:57.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.384+0000 7f55437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:57.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.384+0000 7f55437fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:20:57.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.384+0000 7f55437fe640 10 monclient: dump: 2024-04-22T08:20:57.385 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:57.385 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:57.385 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:57.386 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:57.386 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:57.386 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:57.386 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:57.386 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:57.386 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:57.386 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:57.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.385+0000 7f55437fe640 10 monclient: _finish_auth 0 2024-04-22T08:20:57.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.385+0000 7f55437fe640 10 monclient: _check_auth_tickets 2024-04-22T08:20:57.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.385+0000 7f55437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:20:57.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.385+0000 7f55437fe640 1 -- 172.21.15.40:0/1038544173 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5538030070 con 0x7f5530006360 2024-04-22T08:20:57.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.385+0000 7f554b20b640 5 monclient: authenticate success, global_id 94311 2024-04-22T08:20:57.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.385+0000 7f554b20b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:20:57.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.385+0000 7f55437fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:20:57.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.385+0000 7f554b20b640 10 monclient: get_monmap_and_config success 2024-04-22T08:20:57.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.385+0000 7f554b20b640 4 set_mon_vals no callback set 2024-04-22T08:20:57.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.385+0000 7f55437fe640 1 -- 172.21.15.40:0/1038544173 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5538005050 con 0x7f5530006360 2024-04-22T08:20:57.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.385+0000 7f55437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:57.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.385+0000 7f55437fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:20:57.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.385+0000 7f55437fe640 10 monclient: dump: 2024-04-22T08:20:57.387 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:57.387 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:57.387 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:57.387 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:57.387 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:57.387 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:57.387 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:57.387 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:57.387 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:57.387 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:57.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.385+0000 7f554b20b640 10 monclient: shutdown 2024-04-22T08:20:57.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.385+0000 7f554b20b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:20:57.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.385+0000 7f554b20b640 1 -- 172.21.15.40:0/1038544173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5530006360 msgr2=0x7f5530008820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:57.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.385+0000 7f554b20b640 1 --2- 172.21.15.40:0/1038544173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5530006360 0x7f5530008820 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5530003780 tx=0x7f553802bd20 comp rx=0 tx=0).stop 2024-04-22T08:20:57.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.385+0000 7f55497a8640 1 -- 172.21.15.40:0/1038544173 reap_dead start 2024-04-22T08:20:57.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.385+0000 7f554b20b640 1 -- 172.21.15.40:0/1038544173 shutdown_connections 2024-04-22T08:20:57.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.385+0000 7f554b20b640 1 --2- 172.21.15.40:0/1038544173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5530006360 0x7f5530008820 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:57.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.385+0000 7f554b20b640 1 -- 172.21.15.40:0/1038544173 >> 172.21.15.40:0/1038544173 conn(0x7f55440a4ef0 msgr2=0x7f55440a5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:20:57.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.385+0000 7f554b20b640 1 -- 172.21.15.40:0/1038544173 shutdown_connections 2024-04-22T08:20:57.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.385+0000 7f554b20b640 1 -- 172.21.15.40:0/1038544173 wait complete. 2024-04-22T08:20:57.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.386+0000 7f554b20b640 10 monclient: build_initial_monmap 2024-04-22T08:20:57.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.386+0000 7f554b20b640 10 monclient: monmap: 2024-04-22T08:20:57.392 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:20:57.393 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:20:57.393 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:20:57.393 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:20:57.393 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:20:57.393 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:57.393 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:20:57.393 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:20:57.393 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:20:57.393 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:57.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.387+0000 7f554b20b640 1 Processor -- start 2024-04-22T08:20:57.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.387+0000 7f554b20b640 1 -- start start 2024-04-22T08:20:57.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.387+0000 7f554b20b640 10 monclient: init 2024-04-22T08:20:57.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.387+0000 7f554b20b640 10 monclient: _reopen_session rank -1 2024-04-22T08:20:57.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.387+0000 7f554b20b640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-22T08:20:57.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.387+0000 7f554b20b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5530006360 0x7f55441321f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:57.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.387+0000 7f554b20b640 10 monclient(hunting): picked mon.noname-b con 0x7f5530006360 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:20:57.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.387+0000 7f554b20b640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5544135e00 0x7f5544132730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:57.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.387+0000 7f554b20b640 10 monclient(hunting): picked mon.noname-c con 0x7f5544135e00 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:20:57.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.387+0000 7f554b20b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5544132c70 0x7f55441d0f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:57.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.387+0000 7f554b20b640 10 monclient(hunting): picked mon.noname-a con 0x7f5544132c70 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:20:57.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.387+0000 7f554b20b640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:57.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.387+0000 7f554b20b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55440a2730 con 0x7f5544132c70 2024-04-22T08:20:57.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.387+0000 7f554b20b640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:57.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.387+0000 7f554b20b640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5544134fd0 con 0x7f5530006360 2024-04-22T08:20:57.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.387+0000 7f554b20b640 10 monclient(hunting): start opening mon connection 2024-04-22T08:20:57.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.387+0000 7f554b20b640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55441352b0 con 0x7f5544135e00 2024-04-22T08:20:57.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.387+0000 7f554b20b640 10 monclient(hunting): _renew_subs 2024-04-22T08:20:57.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.387+0000 7f554b20b640 10 monclient(hunting): authenticate will time out at 1331.381764s 2024-04-22T08:20:57.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.388+0000 7f5543fff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5544135e00 0x7f5544132730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:57.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.388+0000 7f55497a8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5544132c70 0x7f55441d0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:57.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.388+0000 7f5543fff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5544135e00 0x7f5544132730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:45012/0 (socket says 172.21.15.40:45012) 2024-04-22T08:20:57.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.388+0000 7f5543fff640 1 -- 172.21.15.40:0/4074605644 learned_addr learned my addr 172.21.15.40:0/4074605644 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:20:57.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.388+0000 7f55497a8640 10 monclient(hunting): get_auth_request con 0x7f5544132c70 auth_method 0 2024-04-22T08:20:57.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.388+0000 7f55497a8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:57.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.388+0000 7f55497a8640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:57.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.388+0000 7f55497a8640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:57.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.388+0000 7f5543fff640 10 monclient(hunting): get_auth_request con 0x7f5544135e00 auth_method 0 2024-04-22T08:20:57.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.388+0000 7f5543fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:57.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.388+0000 7f5543fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:57.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.388+0000 7f5543fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:57.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.388+0000 7f55497a8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:57.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.388+0000 7f55497a8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:57.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.388+0000 7f55497a8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:57.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.388+0000 7f5543fff640 1 -- 172.21.15.40:0/4074605644 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5544135e00 msgr2=0x7f5544132730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-22T08:20:57.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.388+0000 7f5543fff640 1 -- 172.21.15.40:0/4074605644 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5544135e00 msgr2=0x7f5544132730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:20:57.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.388+0000 7f5543fff640 1 --2- 172.21.15.40:0/4074605644 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5544135e00 0x7f5544132730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:20:57.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.388+0000 7f5548fa7640 1 --2- 172.21.15.40:0/4074605644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5530006360 0x7f55441321f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:57.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.388+0000 7f5548fa7640 10 monclient(hunting): get_auth_request con 0x7f5530006360 auth_method 0 2024-04-22T08:20:57.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.388+0000 7f5548fa7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:20:57.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.388+0000 7f5548fa7640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:20:57.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.388+0000 7f5543fff640 1 --2- 172.21.15.40:0/4074605644 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5544135e00 0x7f5544132730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:20:57.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.388+0000 7f5548fa7640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:20:57.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.388+0000 7f5548fa7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:20:57.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.388+0000 7f5548fa7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:20:57.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.388+0000 7f5548fa7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:20:57.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.388+0000 7f55497a8640 10 monclient(hunting): handle_auth_done global_id 94314 payload 824 2024-04-22T08:20:57.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.388+0000 7f55497a8640 1 -- 172.21.15.40:0/4074605644 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5544135e00 msgr2=0x7f5544132730 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:20:57.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.388+0000 7f55497a8640 1 --2- 172.21.15.40:0/4074605644 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5544135e00 0x7f5544132730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:57.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.388+0000 7f55497a8640 1 -- 172.21.15.40:0/4074605644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5530006360 msgr2=0x7f55441321f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:57.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.388+0000 7f55497a8640 1 --2- 172.21.15.40:0/4074605644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5530006360 0x7f55441321f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:57.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.388+0000 7f55497a8640 10 monclient: _finish_hunting 0 2024-04-22T08:20:57.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.388+0000 7f55497a8640 1 monclient: found mon.noname-a 2024-04-22T08:20:57.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.388+0000 7f55497a8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:20:57.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.388+0000 7f55497a8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:20:57.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.388+0000 7f55497a8640 1 -- 172.21.15.40:0/4074605644 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f553800f040 con 0x7f5544132c70 2024-04-22T08:20:57.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.390+0000 7f55497a8640 1 -- 172.21.15.40:0/4074605644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5544132c70 msgr2=0x7f55441d0f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:20:57.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.390+0000 7f55497a8640 1 --2- 172.21.15.40:0/4074605644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5544132c70 0x7f55441d0f20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f554409fa30 tx=0x7f553c00cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:57.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.391+0000 7f55417fa640 1 -- 172.21.15.40:0/4074605644 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f553c00ced0 con 0x7f5544132c70 2024-04-22T08:20:57.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.391+0000 7f55417fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:57.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.391+0000 7f55417fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:20:57.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.391+0000 7f55417fa640 10 monclient: dump: 2024-04-22T08:20:57.401 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:57.401 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:57.401 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:57.401 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:57.401 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:57.401 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:57.401 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:57.401 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:57.401 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:57.401 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:57.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.391+0000 7f55417fa640 10 monclient: _finish_auth 0 2024-04-22T08:20:57.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.391+0000 7f55417fa640 10 monclient: _check_auth_tickets 2024-04-22T08:20:57.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.391+0000 7f55417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:20:57.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.391+0000 7f55417fa640 1 -- 172.21.15.40:0/4074605644 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f553c00a610 con 0x7f5544132c70 2024-04-22T08:20:57.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.391+0000 7f554b20b640 5 monclient: authenticate success, global_id 94314 2024-04-22T08:20:57.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.391+0000 7f554b20b640 10 monclient: _renew_subs 2024-04-22T08:20:57.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.391+0000 7f554b20b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:20:57.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.391+0000 7f554b20b640 1 -- 172.21.15.40:0/4074605644 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5544130f90 con 0x7f5544132c70 2024-04-22T08:20:57.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.391+0000 7f55417fa640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:20:57.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.391+0000 7f55417fa640 1 -- 172.21.15.40:0/4074605644 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f553c00a9a0 con 0x7f5544132c70 2024-04-22T08:20:57.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.391+0000 7f55417fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:20:57.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.391+0000 7f55417fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:20:57.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.391+0000 7f55417fa640 10 monclient: dump: 2024-04-22T08:20:57.402 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:20:57.402 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:20:57.402 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:57.403 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:20:57.403 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:20:57.403 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:20:57.403 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:20:57.403 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:20:57.403 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:20:57.403 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:20:57.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.391+0000 7f554b20b640 10 monclient: _renew_subs 2024-04-22T08:20:57.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.391+0000 7f5542ffd640 4 set_mon_vals no callback set 2024-04-22T08:20:57.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.391+0000 7f554b20b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:20:57.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.391+0000 7f554b20b640 1 -- 172.21.15.40:0/4074605644 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5544133fd0 con 0x7f5544132c70 2024-04-22T08:20:57.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.392+0000 7f55417fa640 1 -- 172.21.15.40:0/4074605644 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f553c00ab80 con 0x7f5544132c70 2024-04-22T08:20:57.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.392+0000 7f55417fa640 1 --2- 172.21.15.40:0/4074605644 >> v2:172.21.15.189:6800/30168 conn(0x7f5530038800 0x7f553003acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:20:57.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.392+0000 7f55417fa640 1 -- 172.21.15.40:0/4074605644 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f553c049870 con 0x7f5544132c70 2024-04-22T08:20:57.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.392+0000 7f554b20b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:20:57.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.392+0000 7f554b20b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:20:57.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.392+0000 7f554b20b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:20:57.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.392+0000 7f554b20b640 1 -- 172.21.15.40:0/4074605644 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f554404e9b0 con 0x7f5544132c70 2024-04-22T08:20:57.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.392+0000 7f5548fa7640 1 --2- 172.21.15.40:0/4074605644 >> v2:172.21.15.189:6800/30168 conn(0x7f5530038800 0x7f553003acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:20:57.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.392+0000 7f5548fa7640 10 monclient: get_auth_request con 0x7f5530038800 auth_method 0 2024-04-22T08:20:57.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.395+0000 7f5548fa7640 1 --2- 172.21.15.40:0/4074605644 >> v2:172.21.15.189:6800/30168 conn(0x7f5530038800 0x7f553003acc0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f5534006fd0 tx=0x7f5534008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:20:57.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.398+0000 7f55417fa640 1 -- 172.21.15.40:0/4074605644 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f553c017080 con 0x7f5544132c70 2024-04-22T08:20:57.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.398+0000 7f55417fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:20:57.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.398+0000 7f55417fa640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:20:57.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.654+0000 7f554b20b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:20:57.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.654+0000 7f554b20b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:20:57.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.654+0000 7f554b20b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:20:57.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.654+0000 7f554b20b640 1 -- 172.21.15.40:0/4074605644 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f554406f610 con 0x7f5544132c70 2024-04-22T08:20:57.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.654+0000 7f55417fa640 1 -- 172.21.15.40:0/4074605644 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1537 (secure 0 0 0) 0x7f553c00ae30 con 0x7f5544132c70 2024-04-22T08:20:57.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.654+0000 7f55417fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:20:57.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.654+0000 7f55417fa640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:20:57.654 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":80,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:20:57.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.656+0000 7f554b20b640 1 -- 172.21.15.40:0/4074605644 >> v2:172.21.15.189:6800/30168 conn(0x7f5530038800 msgr2=0x7f553003acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:57.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.656+0000 7f554b20b640 1 --2- 172.21.15.40:0/4074605644 >> v2:172.21.15.189:6800/30168 conn(0x7f5530038800 0x7f553003acc0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f5534006fd0 tx=0x7f5534008040 comp rx=0 tx=0).stop 2024-04-22T08:20:57.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.656+0000 7f554b20b640 10 monclient: shutdown 2024-04-22T08:20:57.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.656+0000 7f554b20b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:20:57.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.656+0000 7f554b20b640 1 -- 172.21.15.40:0/4074605644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5544132c70 msgr2=0x7f55441d0f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:20:57.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.656+0000 7f554b20b640 1 --2- 172.21.15.40:0/4074605644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5544132c70 0x7f55441d0f20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f554409fa30 tx=0x7f553c00cb90 comp rx=0 tx=0).stop 2024-04-22T08:20:57.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.656+0000 7f554b20b640 1 -- 172.21.15.40:0/4074605644 shutdown_connections 2024-04-22T08:20:57.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.656+0000 7f554b20b640 1 --2- 172.21.15.40:0/4074605644 >> v2:172.21.15.189:6800/30168 conn(0x7f5530038800 0x7f553003acc0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:57.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.656+0000 7f554b20b640 1 --2- 172.21.15.40:0/4074605644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5544132c70 0x7f55441d0f20 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:57.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.656+0000 7f554b20b640 1 --2- 172.21.15.40:0/4074605644 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5544135e00 0x7f5544132730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:57.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.656+0000 7f554b20b640 1 --2- 172.21.15.40:0/4074605644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5530006360 0x7f55441321f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:20:57.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.656+0000 7f554b20b640 1 -- 172.21.15.40:0/4074605644 >> 172.21.15.40:0/4074605644 conn(0x7f55440a4ef0 msgr2=0x7f5544136400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:20:57.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.656+0000 7f554b20b640 1 -- 172.21.15.40:0/4074605644 shutdown_connections 2024-04-22T08:20:57.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:20:57.656+0000 7f554b20b640 1 -- 172.21.15.40:0/4074605644 wait complete. 2024-04-22T08:20:58.256 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[38792]: starting. 2024-04-22T08:20:58.257 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[38792]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-38790 2024-04-22T08:20:58.257 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[38792]: shutting down. 2024-04-22T08:21:00.667 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:21:00.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.780+0000 7f03b1175640 10 monclient: get_monmap_and_config 2024-04-22T08:21:00.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.780+0000 7f03b1175640 10 monclient: build_initial_monmap 2024-04-22T08:21:00.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.780+0000 7f03b1175640 10 monclient: monmap: 2024-04-22T08:21:00.780 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:00.780 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:00.780 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:21:00.781731+0000 2024-04-22T08:21:00.781 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:21:00.781731+0000 2024-04-22T08:21:00.781 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:00.781 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:00.781 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:21:00.781 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:21:00.781 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:21:00.781 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:21:00.781 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:21:00.781 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:00.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.781+0000 7f03b1175640 1 Processor -- start 2024-04-22T08:21:00.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.781+0000 7f03b1175640 1 -- start start 2024-04-22T08:21:00.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.781+0000 7f03b1175640 10 monclient: init 2024-04-22T08:21:00.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.782+0000 7f03b1175640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:00.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.782+0000 7f03b1175640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-04-22T08:21:00.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.782+0000 7f03b1175640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f03ac09ee10 addr v1:172.21.15.189:6789/0 2024-04-22T08:21:00.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.782+0000 7f03b1175640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03ac0a32b0 0x7f03ac13b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:00.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.782+0000 7f03b1175640 10 monclient(hunting): picked mon.noname-b con 0x7f03ac0a32b0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:00.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.782+0000 7f03b1175640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f03ac13b5f0 0x7f03ac13d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:00.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.782+0000 7f03b1175640 10 monclient(hunting): picked mon.noname-a con 0x7f03ac13b5f0 addr v2:172.21.15.40:3300/0 2024-04-22T08:21:00.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.782+0000 7f03b1175640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03ac09f1f0 con 0x7f03ac09ee10 2024-04-22T08:21:00.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.782+0000 7f03b1175640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:00.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.782+0000 7f03b1175640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03ac0a39c0 con 0x7f03ac13b5f0 2024-04-22T08:21:00.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.782+0000 7f03b1175640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:00.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.782+0000 7f03b1175640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03ac09fd30 con 0x7f03ac0a32b0 2024-04-22T08:21:00.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.782+0000 7f03b1175640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:00.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.782+0000 7f03b1175640 10 monclient(hunting): authenticate will time out at 1334.776060s 2024-04-22T08:21:00.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.782+0000 7f03aa59c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03ac0a32b0 0x7f03ac13b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:00.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.782+0000 7f03ab59e640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f03ac13b5f0 0x7f03ac13d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:00.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.782+0000 7f03ab59e640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f03ac13b5f0 0x7f03ac13d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45870/0 (socket says 172.21.15.40:45870) 2024-04-22T08:21:00.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.782+0000 7f03aa59c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03ac0a32b0 0x7f03ac13b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:51732/0 (socket says 172.21.15.40:51732) 2024-04-22T08:21:00.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.782+0000 7f03ab59e640 1 -- 172.21.15.40:0/1941514888 learned_addr learned my addr 172.21.15.40:0/1941514888 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:21:00.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.782+0000 7f03ab59e640 10 monclient(hunting): get_auth_request con 0x7f03ac13b5f0 auth_method 0 2024-04-22T08:21:00.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.782+0000 7f03ab59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:00.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.782+0000 7f03ab59e640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:00.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.782+0000 7f03ab59e640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:00.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.782+0000 7f03aa59c640 10 monclient(hunting): get_auth_request con 0x7f03ac0a32b0 auth_method 0 2024-04-22T08:21:00.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.782+0000 7f03aa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:00.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.782+0000 7f03aa59c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:00.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.782+0000 7f03aa59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:00.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.782+0000 7f03ab59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:00.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.782+0000 7f03ab59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:00.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.782+0000 7f03ab59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:00.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.782+0000 7f03aa59c640 1 -- 172.21.15.40:0/1941514888 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03ac0a32b0 msgr2=0x7f03ac13b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:21:00.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.782+0000 7f03aa59c640 1 -- 172.21.15.40:0/1941514888 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03ac0a32b0 msgr2=0x7f03ac13b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:21:00.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.782+0000 7f03aa59c640 1 --2- 172.21.15.40:0/1941514888 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03ac0a32b0 0x7f03ac13b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:21:00.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.782+0000 7f03aa59c640 1 --2- 172.21.15.40:0/1941514888 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03ac0a32b0 0x7f03ac13b0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:21:00.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.782+0000 7f03ab59e640 10 monclient(hunting): handle_auth_done global_id 94332 payload 274 2024-04-22T08:21:00.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.782+0000 7f03ab59e640 1 -- 172.21.15.40:0/1941514888 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03ac0a32b0 msgr2=0x7f03ac13b0b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:21:00.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.782+0000 7f03ab59e640 1 --2- 172.21.15.40:0/1941514888 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03ac0a32b0 0x7f03ac13b0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:00.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.782+0000 7f03ab59e640 1 -- 172.21.15.40:0/1941514888 >> v1:172.21.15.189:6789/0 conn(0x7f03ac09ee10 legacy=0x7f03ac0a1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:21:00.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.782+0000 7f03ab59e640 10 monclient: _finish_hunting 0 2024-04-22T08:21:00.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.782+0000 7f03ab59e640 1 monclient: found mon.noname-a 2024-04-22T08:21:00.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.782+0000 7f03ab59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:00.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.782+0000 7f03ab59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:21:00.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.782+0000 7f03ab59e640 1 -- 172.21.15.40:0/1941514888 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03ac0a02f0 con 0x7f03ac13b5f0 2024-04-22T08:21:00.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.785+0000 7f03ab59e640 1 -- 172.21.15.40:0/1941514888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03ac13b5f0 msgr2=0x7f03ac13d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:00.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.785+0000 7f03ab59e640 1 --2- 172.21.15.40:0/1941514888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03ac13b5f0 0x7f03ac13d9e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f03ac074690 tx=0x7f03a402bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60595bb593c40e0a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:00.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.785+0000 7f03a9d9b640 1 -- 172.21.15.40:0/1941514888 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a402cb20 con 0x7f03ac13b5f0 2024-04-22T08:21:00.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.785+0000 7f03a9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:00.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.785+0000 7f03a9d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:21:00.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.785+0000 7f03a9d9b640 10 monclient: dump: 2024-04-22T08:21:00.793 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:00.793 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:00.793 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:00.793 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:00.794 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:00.794 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:00.794 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:00.794 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:00.794 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:00.794 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:00.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.786+0000 7f03a9d9b640 10 monclient: _finish_auth 0 2024-04-22T08:21:00.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.786+0000 7f03a9d9b640 10 monclient: _check_auth_tickets 2024-04-22T08:21:00.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.786+0000 7f03a9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:00.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.786+0000 7f03a9d9b640 1 -- 172.21.15.40:0/1941514888 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03a402cc80 con 0x7f03ac13b5f0 2024-04-22T08:21:00.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.786+0000 7f03b1175640 5 monclient: authenticate success, global_id 94332 2024-04-22T08:21:00.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.786+0000 7f03b1175640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:21:00.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.786+0000 7f03a9d9b640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:00.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.786+0000 7f03b1175640 10 monclient: get_monmap_and_config success 2024-04-22T08:21:00.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.786+0000 7f03a9d9b640 1 -- 172.21.15.40:0/1941514888 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a40343f0 con 0x7f03ac13b5f0 2024-04-22T08:21:00.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.786+0000 7f03a9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:00.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.786+0000 7f03b1175640 4 set_mon_vals no callback set 2024-04-22T08:21:00.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.786+0000 7f03a9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:21:00.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.786+0000 7f03a9d9b640 10 monclient: dump: 2024-04-22T08:21:00.795 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:00.795 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:00.795 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:00.795 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:00.795 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:00.795 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:00.795 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:00.795 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:00.795 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:00.796 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:00.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.786+0000 7f03b1175640 10 monclient: shutdown 2024-04-22T08:21:00.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.786+0000 7f03b1175640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:00.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.786+0000 7f03b1175640 1 -- 172.21.15.40:0/1941514888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03ac13b5f0 msgr2=0x7f03ac13d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:00.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.786+0000 7f03b1175640 1 --2- 172.21.15.40:0/1941514888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03ac13b5f0 0x7f03ac13d9e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f03ac074690 tx=0x7f03a402bd70 comp rx=0 tx=0).stop 2024-04-22T08:21:00.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.786+0000 7f03b1175640 1 -- 172.21.15.40:0/1941514888 shutdown_connections 2024-04-22T08:21:00.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.786+0000 7f03b1175640 1 --2- 172.21.15.40:0/1941514888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03ac13b5f0 0x7f03ac13d9e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:00.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.786+0000 7f03b1175640 1 --2- 172.21.15.40:0/1941514888 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03ac0a32b0 0x7f03ac13b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:00.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.786+0000 7f03b1175640 1 -- 172.21.15.40:0/1941514888 >> 172.21.15.40:0/1941514888 conn(0x7f03ac0a4ef0 msgr2=0x7f03ac0a5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:00.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.786+0000 7f03b1175640 1 -- 172.21.15.40:0/1941514888 shutdown_connections 2024-04-22T08:21:00.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.786+0000 7f03b1175640 1 -- 172.21.15.40:0/1941514888 wait complete. 2024-04-22T08:21:00.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.787+0000 7f03b1175640 10 monclient: build_initial_monmap 2024-04-22T08:21:00.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.787+0000 7f03b1175640 10 monclient: monmap: 2024-04-22T08:21:00.796 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:00.797 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:21:00.797 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:21:00.797 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:21:00.797 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:00.797 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:00.797 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:21:00.797 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:21:00.797 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:21:00.797 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:00.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.788+0000 7f03b1175640 1 Processor -- start 2024-04-22T08:21:00.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.788+0000 7f03b1175640 1 -- start start 2024-04-22T08:21:00.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.788+0000 7f03b1175640 10 monclient: init 2024-04-22T08:21:00.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.788+0000 7f03b1175640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:00.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.788+0000 7f03b1175640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-22T08:21:00.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.788+0000 7f03b1175640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03ac09ee10 0x7f03ac1cafb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:00.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.788+0000 7f03b1175640 10 monclient(hunting): picked mon.noname-b con 0x7f03ac09ee10 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:21:00.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.788+0000 7f03b1175640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03ac0a32b0 0x7f03ac1cb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:00.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.788+0000 7f03b1175640 10 monclient(hunting): picked mon.noname-c con 0x7f03ac0a32b0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:00.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.788+0000 7f03b1175640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03ac13b5f0 0x7f03ac06f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:00.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.788+0000 7f03b1175640 10 monclient(hunting): picked mon.noname-a con 0x7f03ac13b5f0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:21:00.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.788+0000 7f03b1175640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:00.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.788+0000 7f03b1175640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03ac135c90 con 0x7f03ac13b5f0 2024-04-22T08:21:00.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.788+0000 7f03b1175640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:00.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.788+0000 7f03b1175640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03ac11f390 con 0x7f03ac09ee10 2024-04-22T08:21:00.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.788+0000 7f03b1175640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:00.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.788+0000 7f03b1175640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03ac0710c0 con 0x7f03ac0a32b0 2024-04-22T08:21:00.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.788+0000 7f03b1175640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:00.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.788+0000 7f03b1175640 10 monclient(hunting): authenticate will time out at 1334.782461s 2024-04-22T08:21:00.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.788+0000 7f03ab59e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03ac13b5f0 0x7f03ac06f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:00.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.788+0000 7f03aa59c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03ac0a32b0 0x7f03ac1cb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:00.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.788+0000 7f03ab59e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03ac13b5f0 0x7f03ac06f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45880/0 (socket says 172.21.15.40:45880) 2024-04-22T08:21:00.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.788+0000 7f03ab59e640 1 -- 172.21.15.40:0/487216550 learned_addr learned my addr 172.21.15.40:0/487216550 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:21:00.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.788+0000 7f03aad9d640 1 --2- 172.21.15.40:0/487216550 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03ac09ee10 0x7f03ac1cafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:00.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.788+0000 7f03aa59c640 10 monclient(hunting): get_auth_request con 0x7f03ac0a32b0 auth_method 0 2024-04-22T08:21:00.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.788+0000 7f03aa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:00.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.788+0000 7f03aa59c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:00.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.788+0000 7f03aa59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:00.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.788+0000 7f03ab59e640 10 monclient(hunting): get_auth_request con 0x7f03ac13b5f0 auth_method 0 2024-04-22T08:21:00.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.788+0000 7f03ab59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:00.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.788+0000 7f03ab59e640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:00.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.788+0000 7f03ab59e640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:00.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.788+0000 7f03aad9d640 10 monclient(hunting): get_auth_request con 0x7f03ac09ee10 auth_method 0 2024-04-22T08:21:00.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.788+0000 7f03aad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:00.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.788+0000 7f03aad9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:00.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.788+0000 7f03aad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:00.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.788+0000 7f03aa59c640 1 -- 172.21.15.40:0/487216550 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03ac0a32b0 msgr2=0x7f03ac1cb4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-22T08:21:00.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.788+0000 7f03ab59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:00.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.788+0000 7f03ab59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:00.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.788+0000 7f03aa59c640 1 -- 172.21.15.40:0/487216550 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03ac0a32b0 msgr2=0x7f03ac1cb4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:21:00.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.788+0000 7f03aa59c640 1 --2- 172.21.15.40:0/487216550 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03ac0a32b0 0x7f03ac1cb4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:21:00.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.789+0000 7f03ab59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:00.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.789+0000 7f03aad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:00.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.789+0000 7f03aad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:00.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.789+0000 7f03aad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:00.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.789+0000 7f03aa59c640 1 --2- 172.21.15.40:0/487216550 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03ac0a32b0 0x7f03ac1cb4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:21:00.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.789+0000 7f03ab59e640 10 monclient(hunting): handle_auth_done global_id 94335 payload 824 2024-04-22T08:21:00.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.789+0000 7f03ab59e640 1 -- 172.21.15.40:0/487216550 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03ac0a32b0 msgr2=0x7f03ac1cb4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:21:00.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.789+0000 7f03ab59e640 1 --2- 172.21.15.40:0/487216550 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03ac0a32b0 0x7f03ac1cb4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:00.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.789+0000 7f03ab59e640 1 -- 172.21.15.40:0/487216550 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03ac09ee10 msgr2=0x7f03ac1cafb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:00.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.789+0000 7f03ab59e640 1 --2- 172.21.15.40:0/487216550 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03ac09ee10 0x7f03ac1cafb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:00.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.789+0000 7f03ab59e640 10 monclient: _finish_hunting 0 2024-04-22T08:21:00.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.789+0000 7f03ab59e640 1 monclient: found mon.noname-a 2024-04-22T08:21:00.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.789+0000 7f03ab59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:00.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.789+0000 7f03ab59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:21:00.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.789+0000 7f03ab59e640 1 -- 172.21.15.40:0/487216550 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a4009b30 con 0x7f03ac13b5f0 2024-04-22T08:21:00.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.789+0000 7f03aad9d640 1 --2- 172.21.15.40:0/487216550 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03ac09ee10 0x7f03ac1cafb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:21:00.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.791+0000 7f03ab59e640 1 -- 172.21.15.40:0/487216550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03ac13b5f0 msgr2=0x7f03ac06f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:00.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.791+0000 7f03ab59e640 1 --2- 172.21.15.40:0/487216550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03ac13b5f0 0x7f03ac06f590 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f03a4009f40 tx=0x7f03a4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:00.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.791+0000 7f03937fe640 1 -- 172.21.15.40:0/487216550 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a403e450 con 0x7f03ac13b5f0 2024-04-22T08:21:00.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.791+0000 7f03937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:00.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.791+0000 7f03937fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:21:00.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.791+0000 7f03937fe640 10 monclient: dump: 2024-04-22T08:21:00.802 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:00.802 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:00.802 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:00.802 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:00.802 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:00.802 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:00.802 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:00.802 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:00.802 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:00.802 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:00.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.791+0000 7f03937fe640 10 monclient: _finish_auth 0 2024-04-22T08:21:00.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.791+0000 7f03937fe640 10 monclient: _check_auth_tickets 2024-04-22T08:21:00.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.791+0000 7f03937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:00.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.791+0000 7f03937fe640 1 -- 172.21.15.40:0/487216550 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03a403e5b0 con 0x7f03ac13b5f0 2024-04-22T08:21:00.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.791+0000 7f03b1175640 5 monclient: authenticate success, global_id 94335 2024-04-22T08:21:00.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.791+0000 7f03b1175640 10 monclient: _renew_subs 2024-04-22T08:21:00.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.791+0000 7f03b1175640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:21:00.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.791+0000 7f03b1175640 1 -- 172.21.15.40:0/487216550 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03ac11f670 con 0x7f03ac13b5f0 2024-04-22T08:21:00.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.792+0000 7f03937fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:00.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.792+0000 7f03937fe640 1 -- 172.21.15.40:0/487216550 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a403e960 con 0x7f03ac13b5f0 2024-04-22T08:21:00.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.792+0000 7f03937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:00.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.792+0000 7f03937fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:21:00.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.792+0000 7f03937fe640 10 monclient: dump: 2024-04-22T08:21:00.803 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:00.803 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:00.803 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:00.804 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:00.804 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:00.804 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:00.804 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:00.804 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:00.804 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:00.804 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:00.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.792+0000 7f03b1175640 10 monclient: _renew_subs 2024-04-22T08:21:00.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.792+0000 7f03b1175640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:21:00.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.792+0000 7f03b1175640 1 -- 172.21.15.40:0/487216550 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03ac1cba30 con 0x7f03ac13b5f0 2024-04-22T08:21:00.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.792+0000 7f03a959a640 4 set_mon_vals no callback set 2024-04-22T08:21:00.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.793+0000 7f03937fe640 1 -- 172.21.15.40:0/487216550 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f03a403eb40 con 0x7f03ac13b5f0 2024-04-22T08:21:00.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.793+0000 7f03b1175640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:00.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.793+0000 7f03b1175640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:00.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.793+0000 7f03b1175640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:21:00.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.793+0000 7f03b1175640 1 -- 172.21.15.40:0/487216550 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03ac04e9b0 con 0x7f03ac13b5f0 2024-04-22T08:21:00.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.793+0000 7f03937fe640 1 --2- 172.21.15.40:0/487216550 >> v2:172.21.15.189:6800/30168 conn(0x7f0398037820 0x7f0398039ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:00.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.793+0000 7f03937fe640 1 -- 172.21.15.40:0/487216550 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(272..272 src has 1..272) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f03a406cdf0 con 0x7f03ac13b5f0 2024-04-22T08:21:00.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.793+0000 7f03aad9d640 1 --2- 172.21.15.40:0/487216550 >> v2:172.21.15.189:6800/30168 conn(0x7f0398037820 0x7f0398039ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:00.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.793+0000 7f03aad9d640 10 monclient: get_auth_request con 0x7f0398037820 auth_method 0 2024-04-22T08:21:00.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.796+0000 7f03aad9d640 1 --2- 172.21.15.40:0/487216550 >> v2:172.21.15.189:6800/30168 conn(0x7f0398037820 0x7f0398039ce0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f039c006fd0 tx=0x7f039c009060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:00.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.798+0000 7f03937fe640 1 -- 172.21.15.40:0/487216550 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f03a4070030 con 0x7f03ac13b5f0 2024-04-22T08:21:00.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.798+0000 7f03937fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:00.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:00.798+0000 7f03937fe640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:21:00.843 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:21:00.843 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38825]: starting. 2024-04-22T08:21:00.843 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38825]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-38790 2024-04-22T08:21:00.843 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38825]: created object 0... 2024-04-22T08:21:00.843 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38825]: created object 25... 2024-04-22T08:21:00.843 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38825]: created object 49... 2024-04-22T08:21:00.843 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38825]: finishing. 2024-04-22T08:21:00.843 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38825]: shutting down. 2024-04-22T08:21:00.878 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:21:00.879 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38827]: starting. 2024-04-22T08:21:00.879 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38827]: listing objects. 2024-04-22T08:21:00.881 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38827]: listed object 0... 2024-04-22T08:21:00.881 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38827]: listed object 25... 2024-04-22T08:21:00.881 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38827]: saw 50 objects 2024-04-22T08:21:00.881 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38827]: shutting down. 2024-04-22T08:21:01.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.052+0000 7f03b1175640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:21:01.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.052+0000 7f03b1175640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:21:01.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.052+0000 7f03b1175640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:21:01.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.052+0000 7f03b1175640 1 -- 172.21.15.40:0/487216550 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f03ac06a1e0 con 0x7f03ac13b5f0 2024-04-22T08:21:01.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.052+0000 7f03937fe640 1 -- 172.21.15.40:0/487216550 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f03a4043070 con 0x7f03ac13b5f0 2024-04-22T08:21:01.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.052+0000 7f03937fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:21:01.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.052+0000 7f03937fe640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:21:01.053 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":82,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:21:01.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.054+0000 7f03b1175640 1 -- 172.21.15.40:0/487216550 >> v2:172.21.15.189:6800/30168 conn(0x7f0398037820 msgr2=0x7f0398039ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:01.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.054+0000 7f03b1175640 1 --2- 172.21.15.40:0/487216550 >> v2:172.21.15.189:6800/30168 conn(0x7f0398037820 0x7f0398039ce0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f039c006fd0 tx=0x7f039c009060 comp rx=0 tx=0).stop 2024-04-22T08:21:01.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.054+0000 7f03b1175640 10 monclient: shutdown 2024-04-22T08:21:01.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.054+0000 7f03b1175640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:01.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.054+0000 7f03b1175640 1 -- 172.21.15.40:0/487216550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03ac13b5f0 msgr2=0x7f03ac06f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:01.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.054+0000 7f03b1175640 1 --2- 172.21.15.40:0/487216550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03ac13b5f0 0x7f03ac06f590 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f03a4009f40 tx=0x7f03a4004a40 comp rx=0 tx=0).stop 2024-04-22T08:21:01.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.054+0000 7f03b1175640 1 -- 172.21.15.40:0/487216550 shutdown_connections 2024-04-22T08:21:01.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.054+0000 7f03b1175640 1 --2- 172.21.15.40:0/487216550 >> v2:172.21.15.189:6800/30168 conn(0x7f0398037820 0x7f0398039ce0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:01.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.054+0000 7f03b1175640 1 --2- 172.21.15.40:0/487216550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03ac13b5f0 0x7f03ac06f590 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:01.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.054+0000 7f03b1175640 1 --2- 172.21.15.40:0/487216550 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03ac0a32b0 0x7f03ac1cb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:01.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.054+0000 7f03b1175640 1 --2- 172.21.15.40:0/487216550 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03ac09ee10 0x7f03ac1cafb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:01.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.054+0000 7f03b1175640 1 -- 172.21.15.40:0/487216550 >> 172.21.15.40:0/487216550 conn(0x7f03ac0a4ef0 msgr2=0x7f03ac13d360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:01.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.054+0000 7f03b1175640 1 -- 172.21.15.40:0/487216550 shutdown_connections 2024-04-22T08:21:01.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.054+0000 7f03b1175640 1 -- 172.21.15.40:0/487216550 wait complete. 2024-04-22T08:21:01.064 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-22T08:21:01.064 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-22T08:21:01.064 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-22T08:21:01.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.164+0000 7f9d69cfc640 10 monclient: get_monmap_and_config 2024-04-22T08:21:01.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.164+0000 7f9d69cfc640 10 monclient: build_initial_monmap 2024-04-22T08:21:01.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.164+0000 7f9d69cfc640 10 monclient: monmap: 2024-04-22T08:21:01.164 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:01.164 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:01.164 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:21:01.165285+0000 2024-04-22T08:21:01.164 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:21:01.165285+0000 2024-04-22T08:21:01.164 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:01.164 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:01.164 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:21:01.164 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:21:01.164 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:21:01.165 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:21:01.165 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:21:01.165 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:01.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.165+0000 7f9d69cfc640 1 Processor -- start 2024-04-22T08:21:01.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.165+0000 7f9d69cfc640 1 -- start start 2024-04-22T08:21:01.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.165+0000 7f9d69cfc640 10 monclient: init 2024-04-22T08:21:01.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.165+0000 7f9d69cfc640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:01.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.165+0000 7f9d69cfc640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-04-22T08:21:01.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.165+0000 7f9d69cfc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9d640a1a80 addr v1:172.21.15.40:6789/0 2024-04-22T08:21:01.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.165+0000 7f9d69cfc640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9d640a2570 0x7f9d640a01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:01.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.165+0000 7f9d69cfc640 10 monclient(hunting): picked mon.noname-c con 0x7f9d640a2570 addr v2:172.21.15.189:3300/0 2024-04-22T08:21:01.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.165+0000 7f9d69cfc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9d6413b0e0 addr v1:172.21.15.189:6789/0 2024-04-22T08:21:01.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.165+0000 7f9d69cfc640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d6409d860 con 0x7f9d640a1a80 2024-04-22T08:21:01.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.165+0000 7f9d69cfc640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d6409ef50 con 0x7f9d6413b0e0 2024-04-22T08:21:01.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.165+0000 7f9d69cfc640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:01.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.165+0000 7f9d69cfc640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d6409f880 con 0x7f9d640a2570 2024-04-22T08:21:01.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.165+0000 7f9d69cfc640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:01.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.165+0000 7f9d69cfc640 10 monclient(hunting): authenticate will time out at 1335.159624s 2024-04-22T08:21:01.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.165+0000 7f9d637fe640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f9d640a1a80 0x7f9d640a1e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:36888/0 (socket says 172.21.15.40:36888) 2024-04-22T08:21:01.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.165+0000 7f9d637fe640 1 -- 172.21.15.40:0/3531363307 learned_addr learned my addr 172.21.15.40:0/3531363307 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:21:01.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.166+0000 7f9d62ffd640 1 --2- 172.21.15.40:0/3531363307 >> v2:172.21.15.189:3300/0 conn(0x7f9d640a2570 0x7f9d640a01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:01.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.166+0000 7f9d62ffd640 10 monclient(hunting): get_auth_request con 0x7f9d640a2570 auth_method 0 2024-04-22T08:21:01.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.166+0000 7f9d62ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:01.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.166+0000 7f9d62ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:01.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.166+0000 7f9d62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:01.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.166+0000 7f9d62ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:01.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.166+0000 7f9d62ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:01.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.166+0000 7f9d62ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:01.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.166+0000 7f9d62ffd640 10 monclient(hunting): handle_auth_done global_id 64894 payload 274 2024-04-22T08:21:01.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.166+0000 7f9d62ffd640 1 -- 172.21.15.40:0/3531363307 >> v1:172.21.15.189:6789/0 conn(0x7f9d6413b0e0 legacy=0x7f9d6413d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:21:01.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.166+0000 7f9d62ffd640 1 -- 172.21.15.40:0/3531363307 >> v1:172.21.15.40:6789/0 conn(0x7f9d640a1a80 legacy=0x7f9d640a1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:21:01.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.166+0000 7f9d62ffd640 10 monclient: _finish_hunting 0 2024-04-22T08:21:01.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.166+0000 7f9d62ffd640 1 monclient: found mon.noname-c 2024-04-22T08:21:01.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.166+0000 7f9d62ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:01.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.166+0000 7f9d62ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.189:3300/0 2024-04-22T08:21:01.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.166+0000 7f9d62ffd640 1 -- 172.21.15.40:0/3531363307 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d64072160 con 0x7f9d640a2570 2024-04-22T08:21:01.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.169+0000 7f9d62ffd640 1 -- 172.21.15.40:0/3531363307 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d640a2570 msgr2=0x7f9d640a01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:01.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.169+0000 7f9d62ffd640 1 --2- 172.21.15.40:0/3531363307 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d640a2570 0x7f9d640a01d0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f9d580097a0 tx=0x7f9d5802b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83e79d40266d8758 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:01.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.169+0000 7f9d627fc640 1 -- 172.21.15.40:0/3531363307 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d58030a00 con 0x7f9d640a2570 2024-04-22T08:21:01.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.169+0000 7f9d627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:01.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.169+0000 7f9d627fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:21:01.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.169+0000 7f9d627fc640 10 monclient: dump: 2024-04-22T08:21:01.179 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:01.179 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:01.179 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:01.179 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:01.179 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:01.179 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:01.179 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:01.179 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:01.180 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:01.180 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:01.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.169+0000 7f9d627fc640 10 monclient: _finish_auth 0 2024-04-22T08:21:01.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.169+0000 7f9d627fc640 10 monclient: _check_auth_tickets 2024-04-22T08:21:01.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.169+0000 7f9d627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:01.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.169+0000 7f9d627fc640 1 -- 172.21.15.40:0/3531363307 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d5802e050 con 0x7f9d640a2570 2024-04-22T08:21:01.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.169+0000 7f9d627fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:01.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.169+0000 7f9d627fc640 1 -- 172.21.15.40:0/3531363307 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d58030d80 con 0x7f9d640a2570 2024-04-22T08:21:01.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.169+0000 7f9d627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:01.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.169+0000 7f9d627fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:21:01.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.169+0000 7f9d627fc640 10 monclient: dump: 2024-04-22T08:21:01.180 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:01.180 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:01.180 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:01.180 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:01.181 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:01.181 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:01.181 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:01.181 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:01.181 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:01.181 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:01.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.169+0000 7f9d69cfc640 5 monclient: authenticate success, global_id 64894 2024-04-22T08:21:01.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.169+0000 7f9d69cfc640 10 monclient: get_monmap_and_config success 2024-04-22T08:21:01.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.169+0000 7f9d69cfc640 4 set_mon_vals no callback set 2024-04-22T08:21:01.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.170+0000 7f9d69cfc640 10 monclient: shutdown 2024-04-22T08:21:01.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.170+0000 7f9d69cfc640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:01.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.170+0000 7f9d69cfc640 1 -- 172.21.15.40:0/3531363307 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d640a2570 msgr2=0x7f9d640a01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:01.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.170+0000 7f9d69cfc640 1 --2- 172.21.15.40:0/3531363307 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d640a2570 0x7f9d640a01d0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f9d580097a0 tx=0x7f9d5802b710 comp rx=0 tx=0).stop 2024-04-22T08:21:01.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.170+0000 7f9d69cfc640 1 -- 172.21.15.40:0/3531363307 shutdown_connections 2024-04-22T08:21:01.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.170+0000 7f9d69cfc640 1 --2- 172.21.15.40:0/3531363307 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d640a2570 0x7f9d640a01d0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:01.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.170+0000 7f9d69cfc640 1 -- 172.21.15.40:0/3531363307 >> 172.21.15.40:0/3531363307 conn(0x7f9d640a66a0 msgr2=0x7f9d640a2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:01.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.170+0000 7f9d69cfc640 1 -- 172.21.15.40:0/3531363307 shutdown_connections 2024-04-22T08:21:01.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.170+0000 7f9d69cfc640 1 -- 172.21.15.40:0/3531363307 wait complete. 2024-04-22T08:21:01.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.171+0000 7f9d69cfc640 10 monclient: build_initial_monmap 2024-04-22T08:21:01.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.171+0000 7f9d69cfc640 10 monclient: monmap: 2024-04-22T08:21:01.182 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:01.182 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:21:01.182 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:21:01.182 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:21:01.182 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:01.182 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:01.182 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:21:01.183 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:21:01.183 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:21:01.183 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:01.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.171+0000 7f9d69cfc640 1 Processor -- start 2024-04-22T08:21:01.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.171+0000 7f9d69cfc640 1 -- start start 2024-04-22T08:21:01.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.171+0000 7f9d69cfc640 10 monclient: init 2024-04-22T08:21:01.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.171+0000 7f9d69cfc640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:01.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.171+0000 7f9d69cfc640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:21:01.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.171+0000 7f9d69cfc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d640a1a80 0x7f9d640705e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:01.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.171+0000 7f9d69cfc640 10 monclient(hunting): picked mon.noname-a con 0x7f9d640a1a80 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:21:01.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.171+0000 7f9d69cfc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d640a2570 0x7f9d64120ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:01.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.171+0000 7f9d69cfc640 10 monclient(hunting): picked mon.noname-b con 0x7f9d640a2570 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:21:01.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.172+0000 7f9d69cfc640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d6413b0e0 0x7f9d64121010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:01.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.172+0000 7f9d69cfc640 10 monclient(hunting): picked mon.noname-c con 0x7f9d6413b0e0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:01.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.172+0000 7f9d69cfc640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:01.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.172+0000 7f9d69cfc640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d64123190 con 0x7f9d640a1a80 2024-04-22T08:21:01.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.172+0000 7f9d69cfc640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:01.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.172+0000 7f9d69cfc640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d64121ca0 con 0x7f9d640a2570 2024-04-22T08:21:01.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.172+0000 7f9d69cfc640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:01.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.172+0000 7f9d69cfc640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6409cc20 con 0x7f9d6413b0e0 2024-04-22T08:21:01.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.172+0000 7f9d69cfc640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:01.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.172+0000 7f9d69cfc640 10 monclient(hunting): authenticate will time out at 1335.165808s 2024-04-22T08:21:01.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.172+0000 7f9d637fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d640a1a80 0x7f9d640705e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:01.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.172+0000 7f9d63fff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d6413b0e0 0x7f9d64121010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:01.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.172+0000 7f9d637fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d640a1a80 0x7f9d640705e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45886/0 (socket says 172.21.15.40:45886) 2024-04-22T08:21:01.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.172+0000 7f9d637fe640 1 -- 172.21.15.40:0/84553312 learned_addr learned my addr 172.21.15.40:0/84553312 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:21:01.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.172+0000 7f9d637fe640 10 monclient(hunting): get_auth_request con 0x7f9d640a1a80 auth_method 0 2024-04-22T08:21:01.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.172+0000 7f9d637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:01.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.172+0000 7f9d637fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:01.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.172+0000 7f9d637fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:01.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.172+0000 7f9d63fff640 10 monclient(hunting): get_auth_request con 0x7f9d6413b0e0 auth_method 0 2024-04-22T08:21:01.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.172+0000 7f9d63fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:01.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.172+0000 7f9d63fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:01.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.172+0000 7f9d63fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:01.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.172+0000 7f9d637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:01.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.172+0000 7f9d637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:01.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.172+0000 7f9d637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:01.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.172+0000 7f9d63fff640 1 -- 172.21.15.40:0/84553312 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d6413b0e0 msgr2=0x7f9d64121010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-22T08:21:01.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.172+0000 7f9d63fff640 1 -- 172.21.15.40:0/84553312 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d6413b0e0 msgr2=0x7f9d64121010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:21:01.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.172+0000 7f9d63fff640 1 --2- 172.21.15.40:0/84553312 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d6413b0e0 0x7f9d64121010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:21:01.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.172+0000 7f9d63fff640 1 --2- 172.21.15.40:0/84553312 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d6413b0e0 0x7f9d64121010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:21:01.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.172+0000 7f9d62ffd640 1 --2- 172.21.15.40:0/84553312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d640a2570 0x7f9d64120ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:01.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.172+0000 7f9d62ffd640 10 monclient(hunting): get_auth_request con 0x7f9d640a2570 auth_method 0 2024-04-22T08:21:01.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.172+0000 7f9d62ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:01.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.172+0000 7f9d62ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:01.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.172+0000 7f9d62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:01.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.172+0000 7f9d637fe640 10 monclient(hunting): handle_auth_done global_id 94338 payload 824 2024-04-22T08:21:01.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.172+0000 7f9d637fe640 1 -- 172.21.15.40:0/84553312 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d6413b0e0 msgr2=0x7f9d64121010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:21:01.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.172+0000 7f9d637fe640 1 --2- 172.21.15.40:0/84553312 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d6413b0e0 0x7f9d64121010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:01.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.172+0000 7f9d637fe640 1 -- 172.21.15.40:0/84553312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d640a2570 msgr2=0x7f9d64120ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:01.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.172+0000 7f9d637fe640 1 --2- 172.21.15.40:0/84553312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d640a2570 0x7f9d64120ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:01.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.172+0000 7f9d637fe640 10 monclient: _finish_hunting 0 2024-04-22T08:21:01.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.172+0000 7f9d637fe640 1 monclient: found mon.noname-a 2024-04-22T08:21:01.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.172+0000 7f9d637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:01.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.172+0000 7f9d637fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:21:01.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.172+0000 7f9d637fe640 1 -- 172.21.15.40:0/84553312 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d58009400 con 0x7f9d640a1a80 2024-04-22T08:21:01.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.174+0000 7f9d637fe640 1 -- 172.21.15.40:0/84553312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d640a1a80 msgr2=0x7f9d640705e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:01.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.174+0000 7f9d637fe640 1 --2- 172.21.15.40:0/84553312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d640a1a80 0x7f9d640705e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9d5400c460 tx=0x7f9d5400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:01.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.174+0000 7f9d4bfff640 1 -- 172.21.15.40:0/84553312 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d54007070 con 0x7f9d640a1a80 2024-04-22T08:21:01.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.174+0000 7f9d4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:01.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.174+0000 7f9d4bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:21:01.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.174+0000 7f9d4bfff640 10 monclient: dump: 2024-04-22T08:21:01.187 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:01.187 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:01.187 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:01.187 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:01.187 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:01.188 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:01.188 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:01.188 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:01.188 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:01.188 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:01.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.174+0000 7f9d4bfff640 10 monclient: _finish_auth 0 2024-04-22T08:21:01.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.174+0000 7f9d4bfff640 10 monclient: _check_auth_tickets 2024-04-22T08:21:01.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.174+0000 7f9d4bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:01.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.174+0000 7f9d69cfc640 5 monclient: authenticate success, global_id 94338 2024-04-22T08:21:01.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.174+0000 7f9d4bfff640 1 -- 172.21.15.40:0/84553312 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d540071f0 con 0x7f9d640a1a80 2024-04-22T08:21:01.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.174+0000 7f9d4bfff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:01.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.174+0000 7f9d4bfff640 1 -- 172.21.15.40:0/84553312 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d540075a0 con 0x7f9d640a1a80 2024-04-22T08:21:01.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.174+0000 7f9d69cfc640 10 monclient: _renew_subs 2024-04-22T08:21:01.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.174+0000 7f9d69cfc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:21:01.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.174+0000 7f9d69cfc640 1 -- 172.21.15.40:0/84553312 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d641233e0 con 0x7f9d640a1a80 2024-04-22T08:21:01.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.174+0000 7f9d4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:01.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.174+0000 7f9d4bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:21:01.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.174+0000 7f9d4bfff640 10 monclient: dump: 2024-04-22T08:21:01.189 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:01.189 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:01.189 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:01.189 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:01.189 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:01.189 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:01.189 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:01.189 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:01.189 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:01.189 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:01.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.174+0000 7f9d617fa640 4 set_mon_vals no callback set 2024-04-22T08:21:01.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.175+0000 7f9d69cfc640 10 monclient: _renew_subs 2024-04-22T08:21:01.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.175+0000 7f9d69cfc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:21:01.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.175+0000 7f9d69cfc640 1 -- 172.21.15.40:0/84553312 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d640710f0 con 0x7f9d640a1a80 2024-04-22T08:21:01.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.175+0000 7f9d4bfff640 1 -- 172.21.15.40:0/84553312 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f9d5401b400 con 0x7f9d640a1a80 2024-04-22T08:21:01.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.175+0000 7f9d69cfc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:01.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.175+0000 7f9d69cfc640 1 --2- 172.21.15.40:0/84553312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d64121550 0x7f9d641cac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:01.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.175+0000 7f9d69cfc640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:01.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.175+0000 7f9d69cfc640 1 -- 172.21.15.40:0/84553312 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d64066770 con 0x7f9d64121550 2024-04-22T08:21:01.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.175+0000 7f9d4bfff640 1 --2- 172.21.15.40:0/84553312 >> v2:172.21.15.189:6800/30168 conn(0x7f9d500377d0 0x7f9d50039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:01.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.175+0000 7f9d4bfff640 1 -- 172.21.15.40:0/84553312 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(272..272 src has 1..272) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f9d5404a050 con 0x7f9d640a1a80 2024-04-22T08:21:01.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.176+0000 7f9d62ffd640 1 --2- 172.21.15.40:0/84553312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d64121550 0x7f9d641cac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:01.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.176+0000 7f9d62ffd640 10 monclient: get_auth_request con 0x7f9d64121550 auth_method 0 2024-04-22T08:21:01.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.176+0000 7f9d62ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:01.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.176+0000 7f9d62ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:01.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.176+0000 7f9d62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:01.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.176+0000 7f9d62ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:01.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.176+0000 7f9d62ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:01.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.176+0000 7f9d62ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:01.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.176+0000 7f9d62ffd640 10 monclient(hunting): handle_auth_done global_id 94341 payload 824 2024-04-22T08:21:01.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.176+0000 7f9d63fff640 1 --2- 172.21.15.40:0/84553312 >> v2:172.21.15.189:6800/30168 conn(0x7f9d500377d0 0x7f9d50039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:01.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.176+0000 7f9d62ffd640 1 -- 172.21.15.40:0/84553312 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9d64122eb0 con 0x7f9d64121550 2024-04-22T08:21:01.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.176+0000 7f9d63fff640 10 monclient: get_auth_request con 0x7f9d500377d0 auth_method 0 2024-04-22T08:21:01.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.178+0000 7f9d62ffd640 1 -- 172.21.15.40:0/84553312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d64121550 msgr2=0x7f9d641cac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:01.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.178+0000 7f9d62ffd640 1 --2- 172.21.15.40:0/84553312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d64121550 0x7f9d641cac70 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9d640a0730 tx=0x7f9d580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:01.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.178+0000 7f9d4bfff640 1 -- 172.21.15.40:0/84553312 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d5803dc10 con 0x7f9d64121550 2024-04-22T08:21:01.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.178+0000 7f9d4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:01.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.178+0000 7f9d4bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:21:01.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.178+0000 7f9d4bfff640 10 monclient: dump: 2024-04-22T08:21:01.192 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:01.192 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:01.192 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:01.192 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:01.192 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:01.192 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:01.192 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:01.192 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:01.192 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:01.192 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:01.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.178+0000 7f9d63fff640 1 --2- 172.21.15.40:0/84553312 >> v2:172.21.15.189:6800/30168 conn(0x7f9d500377d0 0x7f9d50039c90 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f9d640a3f40 tx=0x7f9d5c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:01.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.179+0000 7f9d4bfff640 1 -- 172.21.15.40:0/84553312 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f9d58041070 con 0x7f9d64121550 2024-04-22T08:21:01.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.179+0000 7f9d4bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:01.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.179+0000 7f9d4bfff640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:21:01.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.179+0000 7f9d4bfff640 1 -- 172.21.15.40:0/84553312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d64121550 msgr2=0x7f9d641cac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:01.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.179+0000 7f9d4bfff640 1 --2- 172.21.15.40:0/84553312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d64121550 0x7f9d641cac70 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9d640a0730 tx=0x7f9d580046c0 comp rx=0 tx=0).stop 2024-04-22T08:21:01.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.179+0000 7f9d4bfff640 1 -- 172.21.15.40:0/84553312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d64121550 msgr2=0x7f9d641cac70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:21:01.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.179+0000 7f9d4bfff640 1 --2- 172.21.15.40:0/84553312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d64121550 0x7f9d641cac70 secure :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9d640a0730 tx=0x7f9d580046c0 comp rx=0 tx=0).stop 2024-04-22T08:21:01.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.185+0000 7f9d69cfc640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:21:01.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.185+0000 7f9d69cfc640 1 --2- 172.21.15.40:0/84553312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d641cb1b0 0x7f9d641cb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:01.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.185+0000 7f9d69cfc640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:01.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.185+0000 7f9d69cfc640 1 -- 172.21.15.40:0/84553312 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6409dc10 con 0x7f9d641cb1b0 2024-04-22T08:21:01.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.185+0000 7f9d62ffd640 1 --2- 172.21.15.40:0/84553312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d641cb1b0 0x7f9d641cb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:01.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.185+0000 7f9d62ffd640 10 monclient: get_auth_request con 0x7f9d641cb1b0 auth_method 0 2024-04-22T08:21:01.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.185+0000 7f9d62ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:01.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.185+0000 7f9d62ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:01.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.185+0000 7f9d62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:01.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.185+0000 7f9d62ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:01.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.185+0000 7f9d62ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:01.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.185+0000 7f9d62ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:01.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.185+0000 7f9d62ffd640 10 monclient(hunting): handle_auth_done global_id 94344 payload 824 2024-04-22T08:21:01.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.185+0000 7f9d62ffd640 1 -- 172.21.15.40:0/84553312 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9d641cbb00 con 0x7f9d641cb1b0 2024-04-22T08:21:01.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.187+0000 7f9d62ffd640 1 -- 172.21.15.40:0/84553312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d641cb1b0 msgr2=0x7f9d641cb5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:01.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.188+0000 7f9d62ffd640 1 --2- 172.21.15.40:0/84553312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d641cb1b0 0x7f9d641cb5c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9d5802b750 tx=0x7f9d58008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:01.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.188+0000 7f9d4bfff640 1 -- 172.21.15.40:0/84553312 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d58051620 con 0x7f9d641cb1b0 2024-04-22T08:21:01.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.188+0000 7f9d4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:01.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.188+0000 7f9d4bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:21:01.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.188+0000 7f9d4bfff640 10 monclient: dump: 2024-04-22T08:21:01.196 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:01.197 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:01.197 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:01.197 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:01.197 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:01.197 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:01.197 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:01.197 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:01.197 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:01.197 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:01.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.188+0000 7f9d4bfff640 1 -- 172.21.15.40:0/84553312 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4633 (secure 0 0 0) 0x7f9d58051780 con 0x7f9d641cb1b0 2024-04-22T08:21:01.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.188+0000 7f9d4bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:21:01.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.188+0000 7f9d4bfff640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:21:01.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.188+0000 7f9d4bfff640 1 -- 172.21.15.40:0/84553312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d641cb1b0 msgr2=0x7f9d641cb5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:01.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.188+0000 7f9d4bfff640 1 --2- 172.21.15.40:0/84553312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d641cb1b0 0x7f9d641cb5c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9d5802b750 tx=0x7f9d58008030 comp rx=0 tx=0).stop 2024-04-22T08:21:01.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.188+0000 7f9d4bfff640 1 -- 172.21.15.40:0/84553312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d641cb1b0 msgr2=0x7f9d641cb5c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:21:01.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.188+0000 7f9d4bfff640 1 --2- 172.21.15.40:0/84553312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d641cb1b0 0x7f9d641cb5c0 secure :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9d5802b750 tx=0x7f9d58008030 comp rx=0 tx=0).stop 2024-04-22T08:21:01.198 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:21:01.198 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:21:01.198 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:21:01.198 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "leader", 2024-04-22T08:21:01.198 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 82, 2024-04-22T08:21:01.198 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:21:01.198 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:21:01.198 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:21:01.198 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:21:01.199 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:01.199 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 3, 2024-04-22T08:21:01.199 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:21:01.199 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:21:01.199 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:21:01.199 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:21:01.199 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:21:01.199 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:21:01.199 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:21:01.199 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:21:01.199 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:21:01.199 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:21:01.199 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:21:01.200 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:21:01.200 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:21:01.200 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:01.200 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:21:01.200 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:21:01.200 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:21:01.200 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:21:01.200 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:21:01.200 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:21:01.200 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:21:01.201 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:21:01.201 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:21:01.201 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:21:01.201 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:21:01.201 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:21:01.201 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:01.201 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:01.201 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:21:01.201 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:21:01.201 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:21:01.201 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:21:01.201 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:21:01.201 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:21:01.201 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:21:01.202 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:21:01.202 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:21:01.202 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:21:01.202 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:21:01.202 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:21:01.202 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:21:01.202 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:21:01.202 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:21:01.202 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:21:01.202 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:21:01.202 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:21:01.202 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:21:01.203 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:21:01.203 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:21:01.203 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:21:01.203 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:21:01.203 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:21:01.203 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:21:01.203 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:21:01.203 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:21:01.203 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:01.203 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:21:01.203 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:01.203 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:21:01.203 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:01.203 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:21:01.204 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:21:01.204 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:21:01.204 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:21:01.204 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:01.204 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:21:01.204 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:21:01.204 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:01.204 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:01.204 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:01.204 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:21:01.204 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:21:01.205 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:01.205 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:01.205 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:01.205 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:01.205 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:21:01.205 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:21:01.205 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:21:01.205 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:21:01.205 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:21:01.205 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:01.205 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:01.206 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:21:01.206 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:21:01.206 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:21:01.206 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:21:01.206 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:01.206 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:21:01.206 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:21:01.206 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:01.206 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:01.207 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:01.207 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:21:01.207 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:21:01.207 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:01.207 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:01.207 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:01.207 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:01.207 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:21:01.207 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:21:01.207 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:21:01.208 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:21:01.208 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:21:01.208 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:01.208 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:01.208 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:21:01.208 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:21:01.208 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:21:01.208 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:21:01.208 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:01.208 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:21:01.208 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:21:01.208 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:01.208 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:01.209 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:01.209 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:21:01.209 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:21:01.209 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:01.209 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:01.209 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:01.209 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:01.209 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:21:01.209 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:21:01.210 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:21:01.210 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:21:01.210 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:21:01.210 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:01.210 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:01.210 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:01.210 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:21:01.210 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:21:01.210 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:01.210 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:21:01.210 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:21:01.210 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:21:01.211 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:01.211 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:01.211 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:21:01.211 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:01.211 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:21:01.211 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:21:01.211 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 2 2024-04-22T08:21:01.211 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:01.211 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:01.211 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:21:01.211 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:01.211 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:21:01.211 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:21:01.211 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 3 2024-04-22T08:21:01.211 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:01.212 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:01.212 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:01.212 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:21:01.212 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:21:01.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.188+0000 7f9d69cfc640 1 -- 172.21.15.40:0/84553312 >> v2:172.21.15.189:6800/30168 conn(0x7f9d500377d0 msgr2=0x7f9d50039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:01.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.188+0000 7f9d69cfc640 1 --2- 172.21.15.40:0/84553312 >> v2:172.21.15.189:6800/30168 conn(0x7f9d500377d0 0x7f9d50039c90 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f9d640a3f40 tx=0x7f9d5c009040 comp rx=0 tx=0).stop 2024-04-22T08:21:01.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.188+0000 7f9d69cfc640 10 monclient: shutdown 2024-04-22T08:21:01.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.188+0000 7f9d69cfc640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:01.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.188+0000 7f9d69cfc640 1 -- 172.21.15.40:0/84553312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d640a1a80 msgr2=0x7f9d640705e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:01.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.188+0000 7f9d69cfc640 1 --2- 172.21.15.40:0/84553312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d640a1a80 0x7f9d640705e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9d5400c460 tx=0x7f9d5400c920 comp rx=0 tx=0).stop 2024-04-22T08:21:01.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.188+0000 7f9d63fff640 1 -- 172.21.15.40:0/84553312 reap_dead start 2024-04-22T08:21:01.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.188+0000 7f9d63fff640 1 -- 172.21.15.40:0/84553312 reap_dead start 2024-04-22T08:21:01.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.188+0000 7f9d69cfc640 1 -- 172.21.15.40:0/84553312 shutdown_connections 2024-04-22T08:21:01.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.188+0000 7f9d69cfc640 1 -- 172.21.15.40:0/84553312 >> 172.21.15.40:0/84553312 conn(0x7f9d640a66a0 msgr2=0x7f9d6409b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:01.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.188+0000 7f9d69cfc640 1 -- 172.21.15.40:0/84553312 shutdown_connections 2024-04-22T08:21:01.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.188+0000 7f9d69cfc640 1 -- 172.21.15.40:0/84553312 wait complete. 2024-04-22T08:21:01.214 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:21:01.214 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-22T08:21:01.215 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-22T08:21:01.260 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:21:01.260 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[38826]: starting. 2024-04-22T08:21:01.261 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[38826]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-38790 2024-04-22T08:21:01.261 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[38826]: shutting down. 2024-04-22T08:21:01.262 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:21:01.262 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:21:01.262 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:21:01.265 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:21:01.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.318+0000 7f23196af640 10 monclient: get_monmap_and_config 2024-04-22T08:21:01.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.318+0000 7f23196af640 10 monclient: build_initial_monmap 2024-04-22T08:21:01.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.318+0000 7f23196af640 10 monclient: monmap: 2024-04-22T08:21:01.318 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:01.318 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:01.318 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:21:01.319384+0000 2024-04-22T08:21:01.318 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:21:01.319384+0000 2024-04-22T08:21:01.318 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:01.318 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:01.318 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:21:01.318 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:21:01.319 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:21:01.319 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:21:01.319 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:21:01.319 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:01.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.319+0000 7f23196af640 1 Processor -- start 2024-04-22T08:21:01.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.319+0000 7f23196af640 1 -- start start 2024-04-22T08:21:01.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.319+0000 7f23196af640 10 monclient: init 2024-04-22T08:21:01.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.319+0000 7f23196af640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:01.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.319+0000 7f23196af640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-04-22T08:21:01.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.319+0000 7f23196af640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f23140a1c60 0x7f23140a31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:01.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.319+0000 7f23196af640 10 monclient(hunting): picked mon.noname-a con 0x7f23140a1c60 addr v2:172.21.15.40:3300/0 2024-04-22T08:21:01.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.319+0000 7f23196af640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f23140a2040 0x7f2314135bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:01.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.319+0000 7f23196af640 10 monclient(hunting): picked mon.noname-b con 0x7f23140a2040 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:01.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.319+0000 7f23196af640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f23140a24a0 addr v1:172.21.15.189:6789/0 2024-04-22T08:21:01.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.319+0000 7f23196af640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23140a41f0 con 0x7f23140a24a0 2024-04-22T08:21:01.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.319+0000 7f23196af640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:01.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.319+0000 7f23196af640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f231409fe90 con 0x7f23140a1c60 2024-04-22T08:21:01.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.319+0000 7f23196af640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:01.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.319+0000 7f23196af640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23140a0170 con 0x7f23140a2040 2024-04-22T08:21:01.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.319+0000 7f23196af640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:01.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.319+0000 7f23196af640 10 monclient(hunting): authenticate will time out at 1335.313611s 2024-04-22T08:21:01.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.319+0000 7f230bfff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f23140a2040 0x7f2314135bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:01.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.319+0000 7f230bfff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f23140a2040 0x7f2314135bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:51758/0 (socket says 172.21.15.40:51758) 2024-04-22T08:21:01.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.319+0000 7f230bfff640 1 -- 172.21.15.40:0/1598142598 learned_addr learned my addr 172.21.15.40:0/1598142598 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:21:01.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.319+0000 7f230bfff640 10 monclient(hunting): get_auth_request con 0x7f23140a2040 auth_method 0 2024-04-22T08:21:01.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.319+0000 7f230bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:01.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.319+0000 7f230bfff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:01.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.319+0000 7f230bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:01.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.320+0000 7f230bfff640 1 -- 172.21.15.40:0/1598142598 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f23140a2040 msgr2=0x7f2314135bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:21:01.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.320+0000 7f230bfff640 1 -- 172.21.15.40:0/1598142598 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f23140a2040 msgr2=0x7f2314135bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:21:01.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.320+0000 7f230bfff640 1 --2- 172.21.15.40:0/1598142598 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f23140a2040 0x7f2314135bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:21:01.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.320+0000 7f230bfff640 1 --2- 172.21.15.40:0/1598142598 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f23140a2040 0x7f2314135bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:21:01.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.320+0000 7f2312ffd640 1 --2- 172.21.15.40:0/1598142598 >> v2:172.21.15.40:3300/0 conn(0x7f23140a1c60 0x7f23140a31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:01.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.320+0000 7f2312ffd640 10 monclient(hunting): get_auth_request con 0x7f23140a1c60 auth_method 0 2024-04-22T08:21:01.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.320+0000 7f2312ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:01.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.320+0000 7f2312ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:01.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.320+0000 7f2312ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:01.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.320+0000 7f2312ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:01.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.320+0000 7f2312ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:01.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.320+0000 7f2312ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:01.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.320+0000 7f2312ffd640 10 monclient(hunting): handle_auth_done global_id 94356 payload 274 2024-04-22T08:21:01.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.320+0000 7f2312ffd640 1 -- 172.21.15.40:0/1598142598 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f23140a2040 msgr2=0x7f2314135bb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:21:01.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.320+0000 7f2312ffd640 1 --2- 172.21.15.40:0/1598142598 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f23140a2040 0x7f2314135bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:01.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.320+0000 7f2312ffd640 1 -- 172.21.15.40:0/1598142598 >> v1:172.21.15.189:6789/0 conn(0x7f23140a24a0 legacy=0x7f23141360f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:21:01.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.320+0000 7f2312ffd640 10 monclient: _finish_hunting 0 2024-04-22T08:21:01.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.320+0000 7f2312ffd640 1 monclient: found mon.noname-a 2024-04-22T08:21:01.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.320+0000 7f2312ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:01.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.320+0000 7f2312ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:21:01.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.320+0000 7f2312ffd640 1 -- 172.21.15.40:0/1598142598 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23140a0450 con 0x7f23140a1c60 2024-04-22T08:21:01.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.323+0000 7f2312ffd640 1 -- 172.21.15.40:0/1598142598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23140a1c60 msgr2=0x7f23140a31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:01.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.323+0000 7f2312ffd640 1 --2- 172.21.15.40:0/1598142598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23140a1c60 0x7f23140a31d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f23040097a0 tx=0x7f230402b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aac21f74b2387185 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:01.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.352+0000 7f23127fc640 1 -- 172.21.15.40:0/1598142598 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2304030920 con 0x7f23140a1c60 2024-04-22T08:21:01.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.352+0000 7f23127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:01.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.352+0000 7f23127fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:21:01.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.352+0000 7f23127fc640 10 monclient: dump: 2024-04-22T08:21:01.353 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:01.353 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:01.353 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:01.353 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:01.354 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:01.354 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:01.354 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:01.354 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:01.354 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:01.354 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:01.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.352+0000 7f23127fc640 10 monclient: _finish_auth 0 2024-04-22T08:21:01.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.352+0000 7f23127fc640 10 monclient: _check_auth_tickets 2024-04-22T08:21:01.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.352+0000 7f23127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:01.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.352+0000 7f23127fc640 1 -- 172.21.15.40:0/1598142598 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f230402e050 con 0x7f23140a1c60 2024-04-22T08:21:01.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.352+0000 7f23127fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:01.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.352+0000 7f23127fc640 1 -- 172.21.15.40:0/1598142598 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2304030ca0 con 0x7f23140a1c60 2024-04-22T08:21:01.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.352+0000 7f23196af640 5 monclient: authenticate success, global_id 94356 2024-04-22T08:21:01.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.352+0000 7f23196af640 10 monclient: get_monmap_and_config success 2024-04-22T08:21:01.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.352+0000 7f23196af640 4 set_mon_vals no callback set 2024-04-22T08:21:01.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.352+0000 7f23127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:01.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.352+0000 7f23127fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:21:01.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.352+0000 7f23127fc640 10 monclient: dump: 2024-04-22T08:21:01.355 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:01.355 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:01.355 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:01.355 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:01.355 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:01.355 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:01.355 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:01.355 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:01.355 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:01.355 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:01.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.353+0000 7f23196af640 10 monclient: shutdown 2024-04-22T08:21:01.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.353+0000 7f23196af640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:01.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.353+0000 7f23196af640 1 -- 172.21.15.40:0/1598142598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23140a1c60 msgr2=0x7f23140a31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:01.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.353+0000 7f23196af640 1 --2- 172.21.15.40:0/1598142598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23140a1c60 0x7f23140a31d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f23040097a0 tx=0x7f230402b710 comp rx=0 tx=0).stop 2024-04-22T08:21:01.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.353+0000 7f23196af640 1 -- 172.21.15.40:0/1598142598 shutdown_connections 2024-04-22T08:21:01.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.353+0000 7f23196af640 1 --2- 172.21.15.40:0/1598142598 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f23140a2040 0x7f2314135bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:01.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.353+0000 7f23196af640 1 --2- 172.21.15.40:0/1598142598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23140a1c60 0x7f23140a31d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:01.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.353+0000 7f23196af640 1 -- 172.21.15.40:0/1598142598 >> 172.21.15.40:0/1598142598 conn(0x7f23140a89a0 msgr2=0x7f23140a4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:01.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.353+0000 7f23196af640 1 -- 172.21.15.40:0/1598142598 shutdown_connections 2024-04-22T08:21:01.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.353+0000 7f23196af640 1 -- 172.21.15.40:0/1598142598 wait complete. 2024-04-22T08:21:01.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.353+0000 7f23196af640 10 monclient: build_initial_monmap 2024-04-22T08:21:01.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.353+0000 7f23196af640 10 monclient: monmap: 2024-04-22T08:21:01.356 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:01.356 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:21:01.356 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:21:01.356 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:21:01.357 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:01.357 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:01.357 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:21:01.357 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:21:01.357 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:21:01.357 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:01.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.354+0000 7f23196af640 1 Processor -- start 2024-04-22T08:21:01.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.354+0000 7f23196af640 1 -- start start 2024-04-22T08:21:01.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.354+0000 7f23196af640 10 monclient: init 2024-04-22T08:21:01.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.354+0000 7f23196af640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:01.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.354+0000 7f23196af640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-22T08:21:01.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.354+0000 7f23196af640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23140a1c60 0x7f23141cac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:01.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.354+0000 7f23196af640 10 monclient(hunting): picked mon.noname-a con 0x7f23140a1c60 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:21:01.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.354+0000 7f23196af640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f23140a2040 0x7f23141cb150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:01.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.354+0000 7f23196af640 10 monclient(hunting): picked mon.noname-c con 0x7f23140a2040 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:01.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.354+0000 7f23196af640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23140a24a0 0x7f2314122330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:01.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.354+0000 7f23196af640 10 monclient(hunting): picked mon.noname-b con 0x7f23140a24a0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:21:01.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.354+0000 7f23196af640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:01.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.354+0000 7f23196af640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23141243f0 con 0x7f23140a1c60 2024-04-22T08:21:01.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.354+0000 7f23196af640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:01.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.354+0000 7f23196af640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2314124ab0 con 0x7f23140a24a0 2024-04-22T08:21:01.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.354+0000 7f23196af640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:01.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.354+0000 7f23196af640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23140a4790 con 0x7f23140a2040 2024-04-22T08:21:01.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.354+0000 7f23196af640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:01.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.354+0000 7f23196af640 10 monclient(hunting): authenticate will time out at 1335.348448s 2024-04-22T08:21:01.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.354+0000 7f2312ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23140a1c60 0x7f23141cac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:01.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.354+0000 7f230bfff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f23140a2040 0x7f23141cb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:01.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.354+0000 7f2312ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23140a1c60 0x7f23141cac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45924/0 (socket says 172.21.15.40:45924) 2024-04-22T08:21:01.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.354+0000 7f230bfff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f23140a2040 0x7f23141cb150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:51760/0 (socket says 172.21.15.40:51760) 2024-04-22T08:21:01.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.354+0000 7f2312ffd640 1 -- 172.21.15.40:0/2355624116 learned_addr learned my addr 172.21.15.40:0/2355624116 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:21:01.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.354+0000 7f2312ffd640 10 monclient(hunting): get_auth_request con 0x7f23140a1c60 auth_method 0 2024-04-22T08:21:01.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.354+0000 7f2312ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:01.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.354+0000 7f2312ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:01.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.354+0000 7f2312ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:01.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.354+0000 7f230bfff640 10 monclient(hunting): get_auth_request con 0x7f23140a2040 auth_method 0 2024-04-22T08:21:01.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.354+0000 7f230bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:01.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.354+0000 7f230bfff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:01.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.354+0000 7f230bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:01.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.354+0000 7f2312ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:01.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.354+0000 7f2312ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:01.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.354+0000 7f2312ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:01.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.354+0000 7f230bfff640 1 -- 172.21.15.40:0/2355624116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f23140a2040 msgr2=0x7f23141cb150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-22T08:21:01.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.354+0000 7f230bfff640 1 -- 172.21.15.40:0/2355624116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f23140a2040 msgr2=0x7f23141cb150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:21:01.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.354+0000 7f230bfff640 1 --2- 172.21.15.40:0/2355624116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f23140a2040 0x7f23141cb150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:21:01.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.354+0000 7f230bfff640 1 --2- 172.21.15.40:0/2355624116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f23140a2040 0x7f23141cb150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:21:01.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.355+0000 7f23137fe640 1 --2- 172.21.15.40:0/2355624116 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23140a24a0 0x7f2314122330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:01.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.355+0000 7f2312ffd640 10 monclient(hunting): handle_auth_done global_id 94359 payload 824 2024-04-22T08:21:01.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.355+0000 7f2312ffd640 1 -- 172.21.15.40:0/2355624116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f23140a2040 msgr2=0x7f23141cb150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:21:01.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.355+0000 7f2312ffd640 1 --2- 172.21.15.40:0/2355624116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f23140a2040 0x7f23141cb150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:01.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.355+0000 7f2312ffd640 1 -- 172.21.15.40:0/2355624116 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23140a24a0 msgr2=0x7f2314122330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:01.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.355+0000 7f2312ffd640 1 --2- 172.21.15.40:0/2355624116 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23140a24a0 0x7f2314122330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:01.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.355+0000 7f2312ffd640 10 monclient: _finish_hunting 0 2024-04-22T08:21:01.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.355+0000 7f2312ffd640 1 monclient: found mon.noname-a 2024-04-22T08:21:01.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.355+0000 7f2312ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:01.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.355+0000 7f2312ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:21:01.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.355+0000 7f2312ffd640 1 -- 172.21.15.40:0/2355624116 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2304009400 con 0x7f23140a1c60 2024-04-22T08:21:01.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.355+0000 7f23137fe640 10 monclient: get_auth_request con 0x7f23140a24a0 auth_method 0 2024-04-22T08:21:01.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.355+0000 7f23137fe640 1 --2- 172.21.15.40:0/2355624116 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23140a24a0 0x7f2314122330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T08:21:01.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.357+0000 7f2312ffd640 1 -- 172.21.15.40:0/2355624116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23140a1c60 msgr2=0x7f23141cac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:01.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.357+0000 7f2312ffd640 1 --2- 172.21.15.40:0/2355624116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23140a1c60 0x7f23141cac10 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f23040098c0 tx=0x7f23040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:01.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.357+0000 7f230b7fe640 1 -- 172.21.15.40:0/2355624116 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f230403caf0 con 0x7f23140a1c60 2024-04-22T08:21:01.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.357+0000 7f230b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:01.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.357+0000 7f230b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:21:01.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.357+0000 7f230b7fe640 10 monclient: dump: 2024-04-22T08:21:01.365 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:01.365 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:01.366 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:01.366 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:01.366 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:01.366 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:01.366 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:01.366 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:01.366 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:01.366 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:01.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.357+0000 7f230b7fe640 10 monclient: _finish_auth 0 2024-04-22T08:21:01.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.357+0000 7f230b7fe640 10 monclient: _check_auth_tickets 2024-04-22T08:21:01.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.357+0000 7f230b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:01.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.357+0000 7f230b7fe640 1 -- 172.21.15.40:0/2355624116 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f230403cc70 con 0x7f23140a1c60 2024-04-22T08:21:01.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.357+0000 7f23196af640 5 monclient: authenticate success, global_id 94359 2024-04-22T08:21:01.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.357+0000 7f23196af640 10 monclient: _renew_subs 2024-04-22T08:21:01.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.357+0000 7f23196af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:21:01.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.357+0000 7f23196af640 1 -- 172.21.15.40:0/2355624116 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2314123ee0 con 0x7f23140a1c60 2024-04-22T08:21:01.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.357+0000 7f230b7fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:01.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.357+0000 7f230b7fe640 1 -- 172.21.15.40:0/2355624116 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2304032400 con 0x7f23140a1c60 2024-04-22T08:21:01.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.357+0000 7f230b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:01.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.357+0000 7f230b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:21:01.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.357+0000 7f230b7fe640 10 monclient: dump: 2024-04-22T08:21:01.367 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:01.367 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:01.367 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:01.367 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:01.367 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:01.367 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:01.367 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:01.368 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:01.368 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:01.368 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:01.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.357+0000 7f2311ffb640 4 set_mon_vals no callback set 2024-04-22T08:21:01.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.357+0000 7f23196af640 10 monclient: _renew_subs 2024-04-22T08:21:01.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.357+0000 7f23196af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:21:01.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.357+0000 7f23196af640 1 -- 172.21.15.40:0/2355624116 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23141cb690 con 0x7f23140a1c60 2024-04-22T08:21:01.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.358+0000 7f230b7fe640 1 -- 172.21.15.40:0/2355624116 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f2304039070 con 0x7f23140a1c60 2024-04-22T08:21:01.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.358+0000 7f23196af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:01.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.358+0000 7f23196af640 1 --2- 172.21.15.40:0/2355624116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2314122870 0x7f2314122c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:01.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.358+0000 7f23196af640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:01.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.358+0000 7f23196af640 1 -- 172.21.15.40:0/2355624116 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2314066650 con 0x7f2314122870 2024-04-22T08:21:01.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.358+0000 7f230bfff640 1 --2- 172.21.15.40:0/2355624116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2314122870 0x7f2314122c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:01.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.358+0000 7f230bfff640 10 monclient: get_auth_request con 0x7f2314122870 auth_method 0 2024-04-22T08:21:01.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.358+0000 7f230bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:01.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.358+0000 7f230bfff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:01.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.358+0000 7f230bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:01.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.358+0000 7f230b7fe640 1 --2- 172.21.15.40:0/2355624116 >> v2:172.21.15.189:6800/30168 conn(0x7f22f40377d0 0x7f22f4039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:01.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.358+0000 7f230bfff640 1 -- 172.21.15.40:0/2355624116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2314122870 msgr2=0x7f2314122c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-22T08:21:01.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.358+0000 7f230b7fe640 1 -- 172.21.15.40:0/2355624116 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(273..273 src has 1..273) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f230406bdc0 con 0x7f23140a1c60 2024-04-22T08:21:01.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.358+0000 7f230bfff640 1 -- 172.21.15.40:0/2355624116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2314122870 msgr2=0x7f2314122c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:21:01.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.358+0000 7f230bfff640 1 --2- 172.21.15.40:0/2355624116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2314122870 0x7f2314122c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:21:01.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.358+0000 7f230bfff640 1 --2- 172.21.15.40:0/2355624116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2314122870 0x7f2314122c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:21:01.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.359+0000 7f23137fe640 1 --2- 172.21.15.40:0/2355624116 >> v2:172.21.15.189:6800/30168 conn(0x7f22f40377d0 0x7f22f4039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:01.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.359+0000 7f23137fe640 10 monclient: get_auth_request con 0x7f22f40377d0 auth_method 0 2024-04-22T08:21:01.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.361+0000 7f23137fe640 1 --2- 172.21.15.40:0/2355624116 >> v2:172.21.15.189:6800/30168 conn(0x7f22f40377d0 0x7f22f4039c90 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f2314070f10 tx=0x7f230c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:01.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.559+0000 7f230bfff640 1 --2- 172.21.15.40:0/2355624116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2314122870 0x7f2314122c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:01.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.559+0000 7f230bfff640 10 monclient: get_auth_request con 0x7f2314122870 auth_method 0 2024-04-22T08:21:01.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.559+0000 7f230bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:01.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.559+0000 7f230bfff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:01.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.559+0000 7f230bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:21:01.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.559+0000 7f230bfff640 1 -- 172.21.15.40:0/2355624116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2314122870 msgr2=0x7f2314122c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-22T08:21:01.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.559+0000 7f230bfff640 1 -- 172.21.15.40:0/2355624116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2314122870 msgr2=0x7f2314122c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:21:01.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.559+0000 7f230bfff640 1 --2- 172.21.15.40:0/2355624116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2314122870 0x7f2314122c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:21:01.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.559+0000 7f230bfff640 1 --2- 172.21.15.40:0/2355624116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2314122870 0x7f2314122c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:21:01.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.960+0000 7f230bfff640 1 --2- 172.21.15.40:0/2355624116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2314122870 0x7f2314122c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:01.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.960+0000 7f230bfff640 10 monclient: get_auth_request con 0x7f2314122870 auth_method 0 2024-04-22T08:21:01.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.960+0000 7f230bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:01.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.960+0000 7f230bfff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:01.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.960+0000 7f230bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:21:01.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.960+0000 7f230bfff640 1 -- 172.21.15.40:0/2355624116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2314122870 msgr2=0x7f2314122c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-22T08:21:01.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.960+0000 7f230bfff640 1 -- 172.21.15.40:0/2355624116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2314122870 msgr2=0x7f2314122c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:21:01.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.960+0000 7f230bfff640 1 --2- 172.21.15.40:0/2355624116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2314122870 0x7f2314122c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:21:01.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:01.960+0000 7f230bfff640 1 --2- 172.21.15.40:0/2355624116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2314122870 0x7f2314122c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:21:02.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.761+0000 7f230bfff640 1 --2- 172.21.15.40:0/2355624116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2314122870 0x7f2314122c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:02.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.761+0000 7f230bfff640 10 monclient: get_auth_request con 0x7f2314122870 auth_method 0 2024-04-22T08:21:02.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.761+0000 7f230bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:02.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.761+0000 7f230bfff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:02.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.761+0000 7f230bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:21:02.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.761+0000 7f230bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:02.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.761+0000 7f230bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:02.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.761+0000 7f230bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:02.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.761+0000 7f230bfff640 10 monclient(hunting): handle_auth_done global_id 104099 payload 824 2024-04-22T08:21:02.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.761+0000 7f230bfff640 1 -- 172.21.15.40:0/2355624116 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f23141cb820 con 0x7f2314122870 2024-04-22T08:21:02.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.764+0000 7f230bfff640 1 -- 172.21.15.40:0/2355624116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2314122870 msgr2=0x7f2314122c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:02.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.764+0000 7f230bfff640 1 --2- 172.21.15.40:0/2355624116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2314122870 0x7f2314122c50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f22fc009100 tx=0x7f22fc016da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:02.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.764+0000 7f230b7fe640 1 -- 172.21.15.40:0/2355624116 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22fc004020 con 0x7f2314122870 2024-04-22T08:21:02.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.764+0000 7f230b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:02.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.764+0000 7f230b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:21:02.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.764+0000 7f230b7fe640 10 monclient: dump: 2024-04-22T08:21:02.764 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:02.764 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:02.764 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:02.764 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:02.764 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:02.764 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:02.764 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:02.764 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:02.764 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:02.764 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:02.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.764+0000 7f230b7fe640 1 -- 172.21.15.40:0/2355624116 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f22fc00c040 con 0x7f2314122870 2024-04-22T08:21:02.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.764+0000 7f230b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:02.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.764+0000 7f230b7fe640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:21:02.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.764+0000 7f230b7fe640 1 -- 172.21.15.40:0/2355624116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2314122870 msgr2=0x7f2314122c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:02.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.764+0000 7f230b7fe640 1 --2- 172.21.15.40:0/2355624116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2314122870 0x7f2314122c50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f22fc009100 tx=0x7f22fc016da0 comp rx=0 tx=0).stop 2024-04-22T08:21:02.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.764+0000 7f230b7fe640 1 -- 172.21.15.40:0/2355624116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2314122870 msgr2=0x7f2314122c50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:21:02.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.764+0000 7f230b7fe640 1 --2- 172.21.15.40:0/2355624116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2314122870 0x7f2314122c50 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:02.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.771+0000 7f23196af640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:21:02.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.771+0000 7f23196af640 1 --2- 172.21.15.40:0/2355624116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2314123190 0x7f231411e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:02.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.771+0000 7f23196af640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:02.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.771+0000 7f23196af640 1 -- 172.21.15.40:0/2355624116 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2314136780 con 0x7f2314123190 2024-04-22T08:21:02.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.771+0000 7f230bfff640 1 --2- 172.21.15.40:0/2355624116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2314123190 0x7f231411e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:02.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.772+0000 7f230bfff640 10 monclient: get_auth_request con 0x7f2314123190 auth_method 0 2024-04-22T08:21:02.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.772+0000 7f230bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:02.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.772+0000 7f230bfff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:02.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.772+0000 7f230bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:02.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.772+0000 7f230bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:02.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.772+0000 7f230bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:02.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.772+0000 7f230bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:02.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.772+0000 7f230bfff640 10 monclient(hunting): handle_auth_done global_id 104102 payload 824 2024-04-22T08:21:02.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.772+0000 7f230bfff640 1 -- 172.21.15.40:0/2355624116 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f231404e9b0 con 0x7f2314123190 2024-04-22T08:21:02.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.774+0000 7f230bfff640 1 -- 172.21.15.40:0/2355624116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2314123190 msgr2=0x7f231411e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:02.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.774+0000 7f230bfff640 1 --2- 172.21.15.40:0/2355624116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2314123190 0x7f231411e960 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f22fc01f000 tx=0x7f22fc004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:02.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.774+0000 7f230b7fe640 1 -- 172.21.15.40:0/2355624116 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22fc018a70 con 0x7f2314123190 2024-04-22T08:21:02.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.774+0000 7f230b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:02.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.774+0000 7f230b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:21:02.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.774+0000 7f230b7fe640 10 monclient: dump: 2024-04-22T08:21:02.775 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:02.775 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:02.775 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:02.775 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:02.775 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:02.775 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:02.775 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:02.775 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:02.775 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:02.775 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:02.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.775+0000 7f230b7fe640 1 -- 172.21.15.40:0/2355624116 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4461 (secure 0 0 0) 0x7f22fc003e80 con 0x7f2314123190 2024-04-22T08:21:02.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.775+0000 7f230b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:21:02.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.775+0000 7f230b7fe640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:21:02.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.775+0000 7f230b7fe640 1 -- 172.21.15.40:0/2355624116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2314123190 msgr2=0x7f231411e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:02.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.775+0000 7f230b7fe640 1 --2- 172.21.15.40:0/2355624116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2314123190 0x7f231411e960 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f22fc01f000 tx=0x7f22fc004730 comp rx=0 tx=0).stop 2024-04-22T08:21:02.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.775+0000 7f230b7fe640 1 -- 172.21.15.40:0/2355624116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2314123190 msgr2=0x7f231411e960 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:21:02.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.775+0000 7f230b7fe640 1 --2- 172.21.15.40:0/2355624116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2314123190 0x7f231411e960 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f22fc01f000 tx=0x7f22fc004730 comp rx=0 tx=0).stop 2024-04-22T08:21:02.776 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:21:02.776 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:21:02.776 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:21:02.776 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:21:02.776 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 82, 2024-04-22T08:21:02.776 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:21:02.776 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:21:02.777 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:21:02.777 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:21:02.777 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:02.777 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 4, 2024-04-22T08:21:02.777 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:21:02.777 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:21:02.777 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:21:02.777 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:21:02.777 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:21:02.777 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:21:02.777 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:21:02.777 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:21:02.777 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:21:02.778 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:21:02.778 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:21:02.778 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:21:02.778 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:21:02.778 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:02.778 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:21:02.778 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:21:02.778 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:21:02.778 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:21:02.778 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:21:02.778 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:21:02.778 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:21:02.778 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:21:02.778 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:21:02.779 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:21:02.779 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:21:02.779 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:21:02.779 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:02.779 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:02.779 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:21:02.779 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:21:02.779 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:21:02.779 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:21:02.779 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:21:02.779 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:21:02.780 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:21:02.780 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:21:02.780 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:21:02.780 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:21:02.780 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:21:02.780 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:21:02.780 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:21:02.780 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:21:02.780 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:21:02.780 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:21:02.780 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:21:02.780 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:21:02.780 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:21:02.781 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:21:02.781 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:21:02.781 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:21:02.781 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:21:02.781 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:21:02.781 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:21:02.781 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:21:02.781 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:21:02.781 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:02.781 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:21:02.781 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:02.781 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:21:02.781 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:02.781 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:21:02.782 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:21:02.782 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:21:02.782 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:21:02.782 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:02.782 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:21:02.782 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:21:02.782 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:02.782 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:02.782 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:02.782 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:21:02.782 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:21:02.782 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:02.782 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:02.782 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:02.783 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:02.783 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:21:02.783 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:21:02.783 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:21:02.783 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:21:02.783 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:21:02.783 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:02.783 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:02.783 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:21:02.783 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:21:02.783 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:21:02.783 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:21:02.783 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:02.783 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:21:02.783 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:21:02.784 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:02.784 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:02.784 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:02.784 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:21:02.784 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:21:02.784 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:02.784 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:02.784 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:02.784 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:02.784 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:21:02.784 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:21:02.784 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:21:02.784 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:21:02.784 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:21:02.785 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:02.785 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:02.785 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:21:02.785 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:21:02.785 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:21:02.785 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:21:02.785 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:02.785 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:21:02.785 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:21:02.785 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:02.785 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:02.785 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:02.785 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:21:02.785 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:21:02.786 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:02.786 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:02.786 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:02.786 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:02.786 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:21:02.786 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:21:02.786 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:21:02.786 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:21:02.786 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:21:02.786 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:02.786 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:02.786 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:02.786 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:21:02.786 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:21:02.787 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:02.787 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:21:02.787 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:21:02.787 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:21:02.787 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:02.787 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:02.787 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:21:02.787 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:02.787 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:21:02.787 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:21:02.787 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:21:02.787 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:02.787 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:02.787 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:02.788 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:21:02.788 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:21:02.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.775+0000 7f23196af640 1 -- 172.21.15.40:0/2355624116 >> v2:172.21.15.189:6800/30168 conn(0x7f22f40377d0 msgr2=0x7f22f4039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:02.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.775+0000 7f23196af640 1 --2- 172.21.15.40:0/2355624116 >> v2:172.21.15.189:6800/30168 conn(0x7f22f40377d0 0x7f22f4039c90 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f2314070f10 tx=0x7f230c009040 comp rx=0 tx=0).stop 2024-04-22T08:21:02.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.775+0000 7f23196af640 10 monclient: shutdown 2024-04-22T08:21:02.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.775+0000 7f23196af640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:02.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.775+0000 7f23196af640 1 -- 172.21.15.40:0/2355624116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23140a1c60 msgr2=0x7f23141cac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:02.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.775+0000 7f23196af640 1 --2- 172.21.15.40:0/2355624116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23140a1c60 0x7f23141cac10 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f23040098c0 tx=0x7f23040046c0 comp rx=0 tx=0).stop 2024-04-22T08:21:02.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.775+0000 7f23137fe640 1 -- 172.21.15.40:0/2355624116 reap_dead start 2024-04-22T08:21:02.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.775+0000 7f23137fe640 1 -- 172.21.15.40:0/2355624116 reap_dead start 2024-04-22T08:21:02.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.775+0000 7f23196af640 1 -- 172.21.15.40:0/2355624116 shutdown_connections 2024-04-22T08:21:02.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.775+0000 7f23196af640 1 -- 172.21.15.40:0/2355624116 >> 172.21.15.40:0/2355624116 conn(0x7f23140a89a0 msgr2=0x7f23140a0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:02.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.775+0000 7f23196af640 1 -- 172.21.15.40:0/2355624116 shutdown_connections 2024-04-22T08:21:02.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.775+0000 7f23196af640 1 -- 172.21.15.40:0/2355624116 wait complete. 2024-04-22T08:21:02.790 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:21:02.790 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-22T08:21:02.790 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-22T08:21:02.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.899+0000 7f18a6537640 10 monclient: get_monmap_and_config 2024-04-22T08:21:02.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.899+0000 7f18a6537640 10 monclient: build_initial_monmap 2024-04-22T08:21:02.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.899+0000 7f18a6537640 10 monclient: monmap: 2024-04-22T08:21:02.899 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:02.899 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:02.899 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:21:02.900262+0000 2024-04-22T08:21:02.899 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:21:02.900262+0000 2024-04-22T08:21:02.899 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:02.899 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:02.899 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:21:02.899 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:21:02.899 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:21:02.899 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:21:02.900 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:21:02.900 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:02.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.900+0000 7f18a6537640 1 Processor -- start 2024-04-22T08:21:02.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.900+0000 7f18a6537640 1 -- start start 2024-04-22T08:21:02.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.900+0000 7f18a6537640 10 monclient: init 2024-04-22T08:21:02.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.900+0000 7f18a6537640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:02.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.900+0000 7f18a6537640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-04-22T08:21:02.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.900+0000 7f18a6537640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f18a0136070 0x7f18a009aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:02.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.900+0000 7f18a6537640 10 monclient(hunting): picked mon.noname-a con 0x7f18a0136070 addr v2:172.21.15.40:3300/0 2024-04-22T08:21:02.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.900+0000 7f18a6537640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f18a0136450 0x7f18a00a0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:02.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.900+0000 7f18a6537640 10 monclient(hunting): picked mon.noname-c con 0x7f18a0136450 addr v2:172.21.15.189:3300/0 2024-04-22T08:21:02.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.900+0000 7f18a6537640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f18a01368b0 addr v1:172.21.15.189:6789/0 2024-04-22T08:21:02.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.900+0000 7f18a6537640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18a006bb50 con 0x7f18a01368b0 2024-04-22T08:21:02.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.900+0000 7f18a6537640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:02.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.900+0000 7f18a6537640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18a009a390 con 0x7f18a0136070 2024-04-22T08:21:02.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.900+0000 7f18a6537640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:02.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.900+0000 7f18a6537640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18a0099dd0 con 0x7f18a0136450 2024-04-22T08:21:02.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.900+0000 7f18a6537640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:02.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.900+0000 7f18a6537640 10 monclient(hunting): authenticate will time out at 1336.894555s 2024-04-22T08:21:02.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.900+0000 7f189ffff640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f18a0136070 0x7f18a009aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:02.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.900+0000 7f189ffff640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f18a0136070 0x7f18a009aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45926/0 (socket says 172.21.15.40:45926) 2024-04-22T08:21:02.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.900+0000 7f189ffff640 1 -- 172.21.15.40:0/4229087432 learned_addr learned my addr 172.21.15.40:0/4229087432 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:21:02.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.900+0000 7f189ffff640 10 monclient(hunting): get_auth_request con 0x7f18a0136070 auth_method 0 2024-04-22T08:21:02.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.900+0000 7f189ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:02.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.900+0000 7f189ffff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:02.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.900+0000 7f189ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:02.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.901+0000 7f189ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:02.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.901+0000 7f189ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:02.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.901+0000 7f189ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:02.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.901+0000 7f189ffff640 10 monclient(hunting): handle_auth_done global_id 94362 payload 274 2024-04-22T08:21:02.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.901+0000 7f189f7fe640 1 --2- 172.21.15.40:0/4229087432 >> v2:172.21.15.189:3300/0 conn(0x7f18a0136450 0x7f18a00a0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:02.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.901+0000 7f189ffff640 1 -- 172.21.15.40:0/4229087432 >> v2:172.21.15.189:3300/0 conn(0x7f18a0136450 msgr2=0x7f18a00a0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:21:02.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.901+0000 7f189ffff640 1 --2- 172.21.15.40:0/4229087432 >> v2:172.21.15.189:3300/0 conn(0x7f18a0136450 0x7f18a00a0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:02.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.901+0000 7f189ffff640 1 -- 172.21.15.40:0/4229087432 >> v1:172.21.15.189:6789/0 conn(0x7f18a01368b0 legacy=0x7f18a00a0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:21:02.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.901+0000 7f189ffff640 10 monclient: _finish_hunting 0 2024-04-22T08:21:02.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.901+0000 7f189ffff640 1 monclient: found mon.noname-a 2024-04-22T08:21:02.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.901+0000 7f189ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:02.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.901+0000 7f189ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:21:02.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.901+0000 7f189ffff640 1 -- 172.21.15.40:0/4229087432 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a009a0b0 con 0x7f18a0136070 2024-04-22T08:21:02.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.904+0000 7f189ffff640 1 -- 172.21.15.40:0/4229087432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18a0136070 msgr2=0x7f18a009aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:02.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.904+0000 7f189ffff640 1 --2- 172.21.15.40:0/4229087432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18a0136070 0x7f18a009aa90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f18a0072120 tx=0x7f189002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cef7f8eab8115812 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:02.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.904+0000 7f189effd640 1 -- 172.21.15.40:0/4229087432 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1890031a40 con 0x7f18a0136070 2024-04-22T08:21:02.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.904+0000 7f189effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:02.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.904+0000 7f189effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:21:02.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.904+0000 7f189effd640 10 monclient: dump: 2024-04-22T08:21:02.911 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:02.911 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:02.911 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:02.911 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:02.911 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:02.911 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:02.911 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:02.911 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:02.912 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:02.912 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:02.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.904+0000 7f189effd640 10 monclient: _finish_auth 0 2024-04-22T08:21:02.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.904+0000 7f189effd640 10 monclient: _check_auth_tickets 2024-04-22T08:21:02.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.904+0000 7f189effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:02.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.904+0000 7f189effd640 1 -- 172.21.15.40:0/4229087432 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1890031ba0 con 0x7f18a0136070 2024-04-22T08:21:02.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.904+0000 7f18a6537640 5 monclient: authenticate success, global_id 94362 2024-04-22T08:21:02.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.904+0000 7f18a6537640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:21:02.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.904+0000 7f189effd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:02.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.904+0000 7f18a6537640 10 monclient: get_monmap_and_config success 2024-04-22T08:21:02.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.904+0000 7f189effd640 1 -- 172.21.15.40:0/4229087432 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1890033440 con 0x7f18a0136070 2024-04-22T08:21:02.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.904+0000 7f189effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:02.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.904+0000 7f18a6537640 4 set_mon_vals no callback set 2024-04-22T08:21:02.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.904+0000 7f189effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:21:02.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.904+0000 7f189effd640 10 monclient: dump: 2024-04-22T08:21:02.913 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:02.913 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:02.913 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:02.913 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:02.913 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:02.913 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:02.913 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:02.913 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:02.913 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:02.913 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:02.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.904+0000 7f18a6537640 10 monclient: shutdown 2024-04-22T08:21:02.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.904+0000 7f18a6537640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:02.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.904+0000 7f18a6537640 1 -- 172.21.15.40:0/4229087432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18a0136070 msgr2=0x7f18a009aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:02.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.904+0000 7f18a6537640 1 --2- 172.21.15.40:0/4229087432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18a0136070 0x7f18a009aa90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f18a0072120 tx=0x7f189002b7f0 comp rx=0 tx=0).stop 2024-04-22T08:21:02.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.904+0000 7f18a6537640 1 -- 172.21.15.40:0/4229087432 shutdown_connections 2024-04-22T08:21:02.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.904+0000 7f18a6537640 1 --2- 172.21.15.40:0/4229087432 >> v2:172.21.15.189:3300/0 conn(0x7f18a0136450 0x7f18a00a0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:02.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.904+0000 7f18a6537640 1 --2- 172.21.15.40:0/4229087432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18a0136070 0x7f18a009aa90 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:02.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.904+0000 7f18a6537640 1 -- 172.21.15.40:0/4229087432 >> 172.21.15.40:0/4229087432 conn(0x7f18a00a1b60 msgr2=0x7f18a00a1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:02.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.904+0000 7f18a6537640 1 -- 172.21.15.40:0/4229087432 shutdown_connections 2024-04-22T08:21:02.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.904+0000 7f18a6537640 1 -- 172.21.15.40:0/4229087432 wait complete. 2024-04-22T08:21:02.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.905+0000 7f18a6537640 10 monclient: build_initial_monmap 2024-04-22T08:21:02.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.905+0000 7f18a6537640 10 monclient: monmap: 2024-04-22T08:21:02.914 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:02.914 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:21:02.915 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:21:02.915 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:21:02.915 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:02.915 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:02.915 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:21:02.915 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:21:02.915 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:21:02.915 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:02.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.905+0000 7f18a6537640 1 Processor -- start 2024-04-22T08:21:02.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.905+0000 7f18a6537640 1 -- start start 2024-04-22T08:21:02.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.905+0000 7f18a6537640 10 monclient: init 2024-04-22T08:21:02.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f18a6537640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:02.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f18a6537640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-22T08:21:02.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f18a6537640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18a0136070 0x7f18a01cad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:02.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f18a6537640 10 monclient(hunting): picked mon.noname-b con 0x7f18a0136070 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:21:02.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f18a6537640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18a0136450 0x7f18a01cb250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:02.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f18a6537640 10 monclient(hunting): picked mon.noname-a con 0x7f18a0136450 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:21:02.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f18a6537640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f18a0122330 0x7f18a01227c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:02.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f18a6537640 10 monclient(hunting): picked mon.noname-c con 0x7f18a0122330 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:02.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f18a6537640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:02.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f18a6537640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a0125960 con 0x7f18a0136450 2024-04-22T08:21:02.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f18a6537640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:02.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f18a6537640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a009eca0 con 0x7f18a0136070 2024-04-22T08:21:02.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f18a6537640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:02.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f18a6537640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18a0124ae0 con 0x7f18a0122330 2024-04-22T08:21:02.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f18a6537640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:02.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f18a6537640 10 monclient(hunting): authenticate will time out at 1336.900184s 2024-04-22T08:21:02.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f189f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18a0136450 0x7f18a01cb250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:02.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f189f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18a0136450 0x7f18a01cb250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45930/0 (socket says 172.21.15.40:45930) 2024-04-22T08:21:02.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f189f7fe640 1 -- 172.21.15.40:0/817138381 learned_addr learned my addr 172.21.15.40:0/817138381 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:21:02.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f189f7fe640 10 monclient(hunting): get_auth_request con 0x7f18a0136450 auth_method 0 2024-04-22T08:21:02.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f189f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:02.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f189f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:02.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f189f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:02.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f18a4ad4640 1 --2- 172.21.15.40:0/817138381 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f18a0122330 0x7f18a01227c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:02.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f18a4ad4640 10 monclient(hunting): get_auth_request con 0x7f18a0122330 auth_method 0 2024-04-22T08:21:02.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f18a4ad4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:02.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f18a4ad4640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:02.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f18a4ad4640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:02.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f189f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:02.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f189f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:02.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f189f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:02.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f18a4ad4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:02.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f189ffff640 1 --2- 172.21.15.40:0/817138381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18a0136070 0x7f18a01cad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:02.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f18a4ad4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:02.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f18a4ad4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:02.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f189ffff640 10 monclient(hunting): get_auth_request con 0x7f18a0136070 auth_method 0 2024-04-22T08:21:02.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f189ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:02.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f189ffff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:02.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f189ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:02.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f189ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:02.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f189ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:02.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f189ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:02.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f189f7fe640 10 monclient(hunting): handle_auth_done global_id 94365 payload 824 2024-04-22T08:21:02.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f189f7fe640 1 -- 172.21.15.40:0/817138381 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f18a0122330 msgr2=0x7f18a01227c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:02.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f189f7fe640 1 --2- 172.21.15.40:0/817138381 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f18a0122330 0x7f18a01227c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:02.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f189f7fe640 1 -- 172.21.15.40:0/817138381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18a0136070 msgr2=0x7f18a01cad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:02.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f189f7fe640 1 --2- 172.21.15.40:0/817138381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18a0136070 0x7f18a01cad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:02.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f189f7fe640 10 monclient: _finish_hunting 0 2024-04-22T08:21:02.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f189f7fe640 1 monclient: found mon.noname-a 2024-04-22T08:21:02.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f189f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:02.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f189f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:21:02.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f189f7fe640 1 -- 172.21.15.40:0/817138381 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18900094d0 con 0x7f18a0136450 2024-04-22T08:21:02.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.906+0000 7f18a4ad4640 1 --2- 172.21.15.40:0/817138381 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f18a0122330 0x7f18a01227c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:21:02.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.909+0000 7f189f7fe640 1 -- 172.21.15.40:0/817138381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18a0136450 msgr2=0x7f18a01cb250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:02.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.909+0000 7f189f7fe640 1 --2- 172.21.15.40:0/817138381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18a0136450 0x7f18a01cb250 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f189400c820 tx=0x7f189400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:02.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.909+0000 7f189cff9640 1 -- 172.21.15.40:0/817138381 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1894007c40 con 0x7f18a0136450 2024-04-22T08:21:02.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.909+0000 7f189cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:02.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.909+0000 7f189cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:21:02.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.909+0000 7f189cff9640 10 monclient: dump: 2024-04-22T08:21:02.920 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:02.920 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:02.920 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:02.920 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:02.920 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:02.920 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:02.920 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:02.920 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:02.920 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:02.920 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:02.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.909+0000 7f189cff9640 10 monclient: _finish_auth 0 2024-04-22T08:21:02.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.909+0000 7f189cff9640 10 monclient: _check_auth_tickets 2024-04-22T08:21:02.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.909+0000 7f189cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:02.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.909+0000 7f189cff9640 1 -- 172.21.15.40:0/817138381 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1894007dc0 con 0x7f18a0136450 2024-04-22T08:21:02.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.909+0000 7f18a6537640 5 monclient: authenticate success, global_id 94365 2024-04-22T08:21:02.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.909+0000 7f189cff9640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:02.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.909+0000 7f189cff9640 1 -- 172.21.15.40:0/817138381 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1894004260 con 0x7f18a0136450 2024-04-22T08:21:02.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.909+0000 7f18a6537640 10 monclient: _renew_subs 2024-04-22T08:21:02.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.909+0000 7f18a6537640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:21:02.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.909+0000 7f18a6537640 1 -- 172.21.15.40:0/817138381 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18a009b470 con 0x7f18a0136450 2024-04-22T08:21:02.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.909+0000 7f189cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:02.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.909+0000 7f189cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:21:02.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.909+0000 7f189cff9640 10 monclient: dump: 2024-04-22T08:21:02.921 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:02.921 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:02.922 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:02.922 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:02.922 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:02.922 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:02.922 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:02.922 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:02.922 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:02.922 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:02.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.909+0000 7f189e7fc640 4 set_mon_vals no callback set 2024-04-22T08:21:02.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.909+0000 7f18a6537640 10 monclient: _renew_subs 2024-04-22T08:21:02.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.909+0000 7f18a6537640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:21:02.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.909+0000 7f18a6537640 1 -- 172.21.15.40:0/817138381 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18a0125f20 con 0x7f18a0136450 2024-04-22T08:21:02.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.910+0000 7f18a6537640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:02.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.910+0000 7f18a6537640 1 --2- 172.21.15.40:0/817138381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18a01368b0 0x7f18a002b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:02.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.910+0000 7f189cff9640 1 -- 172.21.15.40:0/817138381 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f1894003610 con 0x7f18a0136450 2024-04-22T08:21:02.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.910+0000 7f18a6537640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:02.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.910+0000 7f18a6537640 1 -- 172.21.15.40:0/817138381 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a0125de0 con 0x7f18a01368b0 2024-04-22T08:21:02.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.910+0000 7f189cff9640 1 --2- 172.21.15.40:0/817138381 >> v2:172.21.15.189:6800/30168 conn(0x7f188c0377d0 0x7f188c039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:02.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.910+0000 7f189cff9640 1 -- 172.21.15.40:0/817138381 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(274..274 src has 239..274) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f189404a260 con 0x7f18a0136450 2024-04-22T08:21:02.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.910+0000 7f189ffff640 1 --2- 172.21.15.40:0/817138381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18a01368b0 0x7f18a002b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:02.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.910+0000 7f189ffff640 10 monclient: get_auth_request con 0x7f18a01368b0 auth_method 0 2024-04-22T08:21:02.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.910+0000 7f189ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:02.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.910+0000 7f189ffff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:02.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.910+0000 7f189ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:02.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.910+0000 7f18a4ad4640 1 --2- 172.21.15.40:0/817138381 >> v2:172.21.15.189:6800/30168 conn(0x7f188c0377d0 0x7f188c039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:02.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.910+0000 7f18a4ad4640 10 monclient: get_auth_request con 0x7f188c0377d0 auth_method 0 2024-04-22T08:21:02.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.910+0000 7f189ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:02.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.910+0000 7f189ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:02.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.910+0000 7f189ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:02.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.910+0000 7f189ffff640 10 monclient(hunting): handle_auth_done global_id 64915 payload 824 2024-04-22T08:21:02.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.910+0000 7f189ffff640 1 -- 172.21.15.40:0/817138381 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f18a006c810 con 0x7f18a01368b0 2024-04-22T08:21:02.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.913+0000 7f18a4ad4640 1 --2- 172.21.15.40:0/817138381 >> v2:172.21.15.189:6800/30168 conn(0x7f188c0377d0 0x7f188c039c90 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f189800f1d0 tx=0x7f1898003d10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:02.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.913+0000 7f189ffff640 1 -- 172.21.15.40:0/817138381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18a01368b0 msgr2=0x7f18a002b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:02.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.913+0000 7f189ffff640 1 --2- 172.21.15.40:0/817138381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18a01368b0 0x7f18a002b200 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f18a009bd30 tx=0x7f18900046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:02.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.913+0000 7f189cff9640 1 -- 172.21.15.40:0/817138381 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f189003d640 con 0x7f18a01368b0 2024-04-22T08:21:02.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.913+0000 7f189cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:02.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.913+0000 7f189cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:21:02.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.913+0000 7f189cff9640 10 monclient: dump: 2024-04-22T08:21:02.926 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:02.926 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:02.927 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:02.927 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:02.927 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:02.927 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:02.927 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:02.927 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:02.927 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:02.927 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:02.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.913+0000 7f189cff9640 1 -- 172.21.15.40:0/817138381 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f1890041070 con 0x7f18a01368b0 2024-04-22T08:21:02.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.913+0000 7f189cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:02.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.913+0000 7f189cff9640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:21:02.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.913+0000 7f189cff9640 1 -- 172.21.15.40:0/817138381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18a01368b0 msgr2=0x7f18a002b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:02.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.913+0000 7f189cff9640 1 --2- 172.21.15.40:0/817138381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18a01368b0 0x7f18a002b200 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f18a009bd30 tx=0x7f18900046c0 comp rx=0 tx=0).stop 2024-04-22T08:21:02.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.913+0000 7f189cff9640 1 -- 172.21.15.40:0/817138381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18a01368b0 msgr2=0x7f18a002b200 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:21:02.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.913+0000 7f189cff9640 1 --2- 172.21.15.40:0/817138381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18a01368b0 0x7f18a002b200 secure :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f18a009bd30 tx=0x7f18900046c0 comp rx=0 tx=0).stop 2024-04-22T08:21:02.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.920+0000 7f18a6537640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:21:02.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.920+0000 7f18a6537640 1 --2- 172.21.15.40:0/817138381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18a01234c0 0x7f18a002b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:02.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.920+0000 7f18a6537640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:02.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.920+0000 7f18a6537640 1 -- 172.21.15.40:0/817138381 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a00667a0 con 0x7f18a01234c0 2024-04-22T08:21:02.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.920+0000 7f189ffff640 1 --2- 172.21.15.40:0/817138381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18a01234c0 0x7f18a002b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:02.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.920+0000 7f189ffff640 10 monclient: get_auth_request con 0x7f18a01234c0 auth_method 0 2024-04-22T08:21:02.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.920+0000 7f189ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:02.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.920+0000 7f189ffff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:02.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.920+0000 7f189ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:02.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.920+0000 7f189ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:02.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.920+0000 7f189ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:02.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.920+0000 7f189ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:02.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.920+0000 7f189ffff640 10 monclient(hunting): handle_auth_done global_id 64918 payload 824 2024-04-22T08:21:02.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.920+0000 7f189ffff640 1 -- 172.21.15.40:0/817138381 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f18a0125aa0 con 0x7f18a01234c0 2024-04-22T08:21:02.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.923+0000 7f189ffff640 1 -- 172.21.15.40:0/817138381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18a01234c0 msgr2=0x7f18a002b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:02.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.923+0000 7f189ffff640 1 --2- 172.21.15.40:0/817138381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18a01234c0 0x7f18a002b740 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f18a009ee10 tx=0x7f1890031000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:02.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.923+0000 7f189cff9640 1 -- 172.21.15.40:0/817138381 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1890003820 con 0x7f18a01234c0 2024-04-22T08:21:02.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.923+0000 7f189cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:02.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.923+0000 7f189cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:21:02.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.923+0000 7f189cff9640 10 monclient: dump: 2024-04-22T08:21:02.929 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:02.929 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:02.929 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:02.929 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:02.929 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:02.929 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:02.930 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:02.930 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:02.930 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:02.930 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:02.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.923+0000 7f189cff9640 1 -- 172.21.15.40:0/817138381 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4801 (secure 0 0 0) 0x7f1890003980 con 0x7f18a01234c0 2024-04-22T08:21:02.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.923+0000 7f189cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:21:02.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.923+0000 7f189cff9640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:21:02.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.923+0000 7f189cff9640 1 -- 172.21.15.40:0/817138381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18a01234c0 msgr2=0x7f18a002b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:02.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.923+0000 7f189cff9640 1 --2- 172.21.15.40:0/817138381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18a01234c0 0x7f18a002b740 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f18a009ee10 tx=0x7f1890031000 comp rx=0 tx=0).stop 2024-04-22T08:21:02.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.923+0000 7f189cff9640 1 -- 172.21.15.40:0/817138381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18a01234c0 msgr2=0x7f18a002b740 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:21:02.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.923+0000 7f189cff9640 1 --2- 172.21.15.40:0/817138381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18a01234c0 0x7f18a002b740 secure :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f18a009ee10 tx=0x7f1890031000 comp rx=0 tx=0).stop 2024-04-22T08:21:02.931 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:21:02.931 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:21:02.931 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:21:02.931 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:21:02.931 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 82, 2024-04-22T08:21:02.931 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:21:02.931 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:21:02.931 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:21:02.931 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:21:02.931 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:02.931 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 4, 2024-04-22T08:21:02.932 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:21:02.932 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:21:02.932 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:21:02.932 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:21:02.932 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:21:02.932 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:21:02.932 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:21:02.932 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:21:02.932 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:21:02.932 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:21:02.932 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:21:02.932 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:21:02.932 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:21:02.932 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:02.933 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:21:02.933 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:21:02.933 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:21:02.933 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:21:02.933 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:21:02.933 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:21:02.933 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:21:02.933 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:21:02.933 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:21:02.933 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:21:02.933 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:21:02.933 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:21:02.933 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:02.934 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:02.934 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:21:02.934 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:21:02.934 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:21:02.934 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:21:02.934 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:21:02.934 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:21:02.934 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:21:02.934 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:21:02.934 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:21:02.934 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:21:02.934 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:21:02.934 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:21:02.935 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:21:02.935 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:21:02.935 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:21:02.935 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:21:02.935 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:21:02.935 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:21:02.935 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:21:02.935 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:21:02.935 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:21:02.935 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:21:02.935 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:21:02.935 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:21:02.935 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:21:02.936 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:21:02.936 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:21:02.936 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:02.936 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:21:02.936 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:02.936 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:21:02.936 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:02.936 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:21:02.936 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:21:02.936 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:21:02.936 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:21:02.936 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:02.936 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:21:02.937 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:21:02.937 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:02.937 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:02.937 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:02.937 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:21:02.937 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:21:02.937 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:02.937 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:02.937 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:02.937 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:02.937 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:21:02.937 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:21:02.937 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:21:02.937 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:21:02.938 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:21:02.938 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:02.938 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:02.938 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:21:02.938 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:21:02.938 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:21:02.938 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:21:02.938 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:02.938 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:21:02.938 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:21:02.938 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:02.938 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:02.938 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:02.939 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:21:02.939 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:21:02.939 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:02.939 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:02.939 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:02.939 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:02.939 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:21:02.939 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:21:02.939 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:21:02.939 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:21:02.939 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:21:02.939 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:02.939 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:02.939 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:21:02.940 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:21:02.940 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:21:02.940 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:21:02.940 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:02.940 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:21:02.940 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:21:02.940 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:02.940 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:02.940 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:02.940 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:21:02.940 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:21:02.940 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:02.940 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:02.940 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:02.941 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:02.941 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:21:02.941 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:21:02.941 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:21:02.941 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:21:02.941 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:21:02.941 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:02.941 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:02.941 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:02.941 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:21:02.941 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:21:02.941 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:02.941 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:21:02.941 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:21:02.942 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:21:02.942 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:02.942 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:02.942 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:21:02.942 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:02.942 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:21:02.942 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:21:02.942 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 4 2024-04-22T08:21:02.942 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:02.942 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:02.942 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:21:02.942 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:02.943 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:21:02.943 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:21:02.943 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 5 2024-04-22T08:21:02.943 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:02.943 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:02.943 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": [ 2024-04-22T08:21:02.943 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:02.943 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:21:02.943 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:21:02.943 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:21:02.943 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:02.943 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:02.943 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:02.944 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:21:02.944 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:21:02.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.924+0000 7f18a6537640 1 -- 172.21.15.40:0/817138381 >> v2:172.21.15.189:6800/30168 conn(0x7f188c0377d0 msgr2=0x7f188c039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:02.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.924+0000 7f18a6537640 1 --2- 172.21.15.40:0/817138381 >> v2:172.21.15.189:6800/30168 conn(0x7f188c0377d0 0x7f188c039c90 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f189800f1d0 tx=0x7f1898003d10 comp rx=0 tx=0).stop 2024-04-22T08:21:02.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.924+0000 7f18a6537640 10 monclient: shutdown 2024-04-22T08:21:02.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.924+0000 7f18a6537640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:02.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.924+0000 7f18a6537640 1 -- 172.21.15.40:0/817138381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18a0136450 msgr2=0x7f18a01cb250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:02.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.924+0000 7f18a6537640 1 --2- 172.21.15.40:0/817138381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18a0136450 0x7f18a01cb250 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f189400c820 tx=0x7f189400cce0 comp rx=0 tx=0).stop 2024-04-22T08:21:02.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.924+0000 7f18a4ad4640 1 -- 172.21.15.40:0/817138381 reap_dead start 2024-04-22T08:21:02.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.924+0000 7f18a4ad4640 1 -- 172.21.15.40:0/817138381 reap_dead start 2024-04-22T08:21:02.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.924+0000 7f18a6537640 1 -- 172.21.15.40:0/817138381 shutdown_connections 2024-04-22T08:21:02.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.924+0000 7f18a6537640 1 -- 172.21.15.40:0/817138381 >> 172.21.15.40:0/817138381 conn(0x7f18a00a1b60 msgr2=0x7f18a009d5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:02.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.924+0000 7f18a6537640 1 -- 172.21.15.40:0/817138381 shutdown_connections 2024-04-22T08:21:02.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:02.924+0000 7f18a6537640 1 -- 172.21.15.40:0/817138381 wait complete. 2024-04-22T08:21:02.946 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:21:02.946 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-22T08:21:02.946 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-22T08:21:03.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.045+0000 7f3568021640 10 monclient: get_monmap_and_config 2024-04-22T08:21:03.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.045+0000 7f3568021640 10 monclient: build_initial_monmap 2024-04-22T08:21:03.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.045+0000 7f3568021640 10 monclient: monmap: 2024-04-22T08:21:03.045 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:03.045 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:03.045 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:21:03.046447+0000 2024-04-22T08:21:03.045 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:21:03.046447+0000 2024-04-22T08:21:03.045 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:03.045 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:03.046 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:21:03.046 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:21:03.046 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:21:03.046 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:21:03.046 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:21:03.046 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:03.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.046+0000 7f3568021640 1 Processor -- start 2024-04-22T08:21:03.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.046+0000 7f3568021640 1 -- start start 2024-04-22T08:21:03.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.046+0000 7f3568021640 10 monclient: init 2024-04-22T08:21:03.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.046+0000 7f3568021640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:03.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.046+0000 7f3568021640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-04-22T08:21:03.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.046+0000 7f3568021640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3560135e30 0x7f3560136210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:03.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.046+0000 7f3568021640 10 monclient(hunting): picked mon.noname-c con 0x7f3560135e30 addr v2:172.21.15.189:3300/0 2024-04-22T08:21:03.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.046+0000 7f3568021640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3560071c90 addr v1:172.21.15.189:6789/0 2024-04-22T08:21:03.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.046+0000 7f3568021640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3560136750 0x7f35600a20b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:03.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.046+0000 7f3568021640 10 monclient(hunting): picked mon.noname-a con 0x7f3560136750 addr v2:172.21.15.40:3300/0 2024-04-22T08:21:03.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.046+0000 7f3568021640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f356009e310 con 0x7f3560071c90 2024-04-22T08:21:03.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.046+0000 7f3568021640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:03.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.046+0000 7f3568021640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f356009de80 con 0x7f3560136750 2024-04-22T08:21:03.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.046+0000 7f3568021640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:03.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.046+0000 7f3568021640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f356009f130 con 0x7f3560135e30 2024-04-22T08:21:03.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.046+0000 7f3568021640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:03.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.046+0000 7f3568021640 10 monclient(hunting): authenticate will time out at 1337.040684s 2024-04-22T08:21:03.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.047+0000 7f35665be640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3560136750 0x7f35600a20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:03.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.047+0000 7f35665be640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3560136750 0x7f35600a20b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45944/0 (socket says 172.21.15.40:45944) 2024-04-22T08:21:03.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.047+0000 7f35665be640 1 -- 172.21.15.40:0/943169115 learned_addr learned my addr 172.21.15.40:0/943169115 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:21:03.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.047+0000 7f35665be640 10 monclient(hunting): get_auth_request con 0x7f3560136750 auth_method 0 2024-04-22T08:21:03.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.047+0000 7f35665be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:03.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.047+0000 7f35665be640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:03.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.047+0000 7f35665be640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:03.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.047+0000 7f35665be640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:03.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.047+0000 7f35665be640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:03.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.047+0000 7f35665be640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:03.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.047+0000 7f3565dbd640 1 --2- 172.21.15.40:0/943169115 >> v2:172.21.15.189:3300/0 conn(0x7f3560135e30 0x7f3560136210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:03.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.047+0000 7f3565dbd640 10 monclient(hunting): get_auth_request con 0x7f3560135e30 auth_method 0 2024-04-22T08:21:03.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.047+0000 7f3565dbd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:03.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.047+0000 7f3565dbd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:03.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.047+0000 7f3565dbd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:03.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.047+0000 7f35665be640 10 monclient(hunting): handle_auth_done global_id 94368 payload 274 2024-04-22T08:21:03.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.047+0000 7f35665be640 1 -- 172.21.15.40:0/943169115 >> v2:172.21.15.189:3300/0 conn(0x7f3560135e30 msgr2=0x7f3560136210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:21:03.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.047+0000 7f35665be640 1 --2- 172.21.15.40:0/943169115 >> v2:172.21.15.189:3300/0 conn(0x7f3560135e30 0x7f3560136210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:03.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.047+0000 7f35665be640 1 -- 172.21.15.40:0/943169115 >> v1:172.21.15.189:6789/0 conn(0x7f3560071c90 legacy=0x7f35600a19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:21:03.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.047+0000 7f35665be640 10 monclient: _finish_hunting 0 2024-04-22T08:21:03.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.047+0000 7f35665be640 1 monclient: found mon.noname-a 2024-04-22T08:21:03.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.047+0000 7f35665be640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:03.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.047+0000 7f35665be640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:21:03.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.047+0000 7f35665be640 1 -- 172.21.15.40:0/943169115 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f356009f410 con 0x7f3560136750 2024-04-22T08:21:03.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.050+0000 7f35665be640 1 -- 172.21.15.40:0/943169115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3560136750 msgr2=0x7f35600a20b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:03.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.050+0000 7f35665be640 1 --2- 172.21.15.40:0/943169115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3560136750 0x7f35600a20b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3558009e50 tx=0x7f355802bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdd518e4eb5a37b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:03.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.050+0000 7f3564dbb640 1 -- 172.21.15.40:0/943169115 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355802cb70 con 0x7f3560136750 2024-04-22T08:21:03.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.050+0000 7f3564dbb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:03.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.050+0000 7f3564dbb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:21:03.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.050+0000 7f3564dbb640 10 monclient: dump: 2024-04-22T08:21:03.057 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:03.057 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:03.057 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:03.058 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:03.058 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:03.058 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:03.058 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:03.058 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:03.058 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:03.058 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:03.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.050+0000 7f3564dbb640 10 monclient: _finish_auth 0 2024-04-22T08:21:03.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.050+0000 7f3564dbb640 10 monclient: _check_auth_tickets 2024-04-22T08:21:03.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.050+0000 7f3564dbb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:03.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.050+0000 7f3564dbb640 1 -- 172.21.15.40:0/943169115 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f355802ccd0 con 0x7f3560136750 2024-04-22T08:21:03.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.050+0000 7f3568021640 5 monclient: authenticate success, global_id 94368 2024-04-22T08:21:03.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.050+0000 7f3568021640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:21:03.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.050+0000 7f3564dbb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:03.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.050+0000 7f3564dbb640 1 -- 172.21.15.40:0/943169115 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3558034610 con 0x7f3560136750 2024-04-22T08:21:03.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.050+0000 7f3568021640 10 monclient: get_monmap_and_config success 2024-04-22T08:21:03.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.050+0000 7f3568021640 4 set_mon_vals no callback set 2024-04-22T08:21:03.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.050+0000 7f3564dbb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:03.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.050+0000 7f3564dbb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:21:03.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.050+0000 7f3564dbb640 10 monclient: dump: 2024-04-22T08:21:03.059 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:03.059 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:03.059 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:03.059 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:03.059 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:03.059 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:03.060 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:03.060 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:03.060 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:03.060 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:03.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.050+0000 7f3568021640 10 monclient: shutdown 2024-04-22T08:21:03.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.050+0000 7f3568021640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:03.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.050+0000 7f3568021640 1 -- 172.21.15.40:0/943169115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3560136750 msgr2=0x7f35600a20b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:03.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.050+0000 7f3568021640 1 --2- 172.21.15.40:0/943169115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3560136750 0x7f35600a20b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3558009e50 tx=0x7f355802bdc0 comp rx=0 tx=0).stop 2024-04-22T08:21:03.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.050+0000 7f3568021640 1 -- 172.21.15.40:0/943169115 shutdown_connections 2024-04-22T08:21:03.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.050+0000 7f3568021640 1 --2- 172.21.15.40:0/943169115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3560136750 0x7f35600a20b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:03.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.050+0000 7f3568021640 1 --2- 172.21.15.40:0/943169115 >> v2:172.21.15.189:3300/0 conn(0x7f3560135e30 0x7f3560136210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:03.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.050+0000 7f3568021640 1 -- 172.21.15.40:0/943169115 >> 172.21.15.40:0/943169115 conn(0x7f35600a65c0 msgr2=0x7f35600a2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:03.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.051+0000 7f3568021640 1 -- 172.21.15.40:0/943169115 shutdown_connections 2024-04-22T08:21:03.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.051+0000 7f3568021640 1 -- 172.21.15.40:0/943169115 wait complete. 2024-04-22T08:21:03.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.051+0000 7f3568021640 10 monclient: build_initial_monmap 2024-04-22T08:21:03.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.051+0000 7f3568021640 10 monclient: monmap: 2024-04-22T08:21:03.061 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:03.061 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:21:03.061 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:21:03.061 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:21:03.061 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:03.061 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:03.061 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:21:03.061 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:21:03.061 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:21:03.061 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:03.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.051+0000 7f3568021640 1 Processor -- start 2024-04-22T08:21:03.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.051+0000 7f3568021640 1 -- start start 2024-04-22T08:21:03.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.051+0000 7f3568021640 10 monclient: init 2024-04-22T08:21:03.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f3568021640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:03.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f3568021640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-22T08:21:03.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f3568021640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3560071c90 0x7f35601209f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:03.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f3568021640 10 monclient(hunting): picked mon.noname-c con 0x7f3560071c90 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:03.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f3568021640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3560135e30 0x7f3560120f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:03.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f3568021640 10 monclient(hunting): picked mon.noname-b con 0x7f3560135e30 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:21:03.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f3568021640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3560136750 0x7f3560121470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:03.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f3568021640 10 monclient(hunting): picked mon.noname-a con 0x7f3560136750 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:21:03.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f3568021640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:03.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f3568021640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35601237f0 con 0x7f3560136750 2024-04-22T08:21:03.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f3568021640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:03.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f3568021640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3560122240 con 0x7f3560135e30 2024-04-22T08:21:03.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f3568021640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:03.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f3568021640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3560121ce0 con 0x7f3560071c90 2024-04-22T08:21:03.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f3568021640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:03.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f3568021640 10 monclient(hunting): authenticate will time out at 1337.046139s 2024-04-22T08:21:03.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f35665be640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3560136750 0x7f3560121470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:03.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f3565dbd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3560071c90 0x7f35601209f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:03.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f35665be640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3560136750 0x7f3560121470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45950/0 (socket says 172.21.15.40:45950) 2024-04-22T08:21:03.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f35665be640 1 -- 172.21.15.40:0/431880785 learned_addr learned my addr 172.21.15.40:0/431880785 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:21:03.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f35665be640 10 monclient(hunting): get_auth_request con 0x7f3560136750 auth_method 0 2024-04-22T08:21:03.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f35665be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:03.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f35665be640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:03.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f35665be640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:03.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f3565dbd640 10 monclient(hunting): get_auth_request con 0x7f3560071c90 auth_method 0 2024-04-22T08:21:03.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f3565dbd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:03.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f3565dbd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:03.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f3565dbd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:03.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f35665be640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:03.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f35665be640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:03.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f35665be640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:03.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f3565dbd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:03.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f3565dbd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:03.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f3565dbd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:03.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f35655bc640 1 --2- 172.21.15.40:0/431880785 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3560135e30 0x7f3560120f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:03.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f35655bc640 10 monclient(hunting): get_auth_request con 0x7f3560135e30 auth_method 0 2024-04-22T08:21:03.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f35655bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:03.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f35655bc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:03.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f35655bc640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:03.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f35665be640 10 monclient(hunting): handle_auth_done global_id 94371 payload 824 2024-04-22T08:21:03.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f35665be640 1 -- 172.21.15.40:0/431880785 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3560071c90 msgr2=0x7f35601209f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:03.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f35665be640 1 --2- 172.21.15.40:0/431880785 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3560071c90 0x7f35601209f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:03.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f35665be640 1 -- 172.21.15.40:0/431880785 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3560135e30 msgr2=0x7f3560120f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:03.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f35665be640 1 --2- 172.21.15.40:0/431880785 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3560135e30 0x7f3560120f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:03.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f35665be640 10 monclient: _finish_hunting 0 2024-04-22T08:21:03.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f35665be640 1 monclient: found mon.noname-a 2024-04-22T08:21:03.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f35665be640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:03.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f35665be640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:21:03.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f35665be640 1 -- 172.21.15.40:0/431880785 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3558009b30 con 0x7f3560136750 2024-04-22T08:21:03.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f35655bc640 1 --2- 172.21.15.40:0/431880785 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3560135e30 0x7f3560120f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T08:21:03.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.052+0000 7f3565dbd640 1 --2- 172.21.15.40:0/431880785 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3560071c90 0x7f35601209f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:21:03.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.055+0000 7f35665be640 1 -- 172.21.15.40:0/431880785 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3560136750 msgr2=0x7f3560121470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:03.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.055+0000 7f35665be640 1 --2- 172.21.15.40:0/431880785 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3560136750 0x7f3560121470 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3558009e50 tx=0x7f3558004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:03.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.055+0000 7f35567fc640 1 -- 172.21.15.40:0/431880785 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3558034da0 con 0x7f3560136750 2024-04-22T08:21:03.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.055+0000 7f35567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:03.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.055+0000 7f35567fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:21:03.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.055+0000 7f35567fc640 10 monclient: dump: 2024-04-22T08:21:03.066 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:03.066 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:03.066 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:03.066 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:03.066 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:03.066 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:03.066 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:03.067 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:03.067 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:03.067 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:03.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.055+0000 7f35567fc640 10 monclient: _finish_auth 0 2024-04-22T08:21:03.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.055+0000 7f35567fc640 10 monclient: _check_auth_tickets 2024-04-22T08:21:03.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.055+0000 7f35567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:03.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.055+0000 7f35567fc640 1 -- 172.21.15.40:0/431880785 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f355803e450 con 0x7f3560136750 2024-04-22T08:21:03.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.055+0000 7f35567fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:03.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.055+0000 7f3568021640 5 monclient: authenticate success, global_id 94371 2024-04-22T08:21:03.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.055+0000 7f3568021640 10 monclient: _renew_subs 2024-04-22T08:21:03.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.055+0000 7f35567fc640 1 -- 172.21.15.40:0/431880785 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355803e800 con 0x7f3560136750 2024-04-22T08:21:03.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.055+0000 7f3568021640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:21:03.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.055+0000 7f3568021640 1 -- 172.21.15.40:0/431880785 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35600720e0 con 0x7f3560136750 2024-04-22T08:21:03.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.055+0000 7f3557fff640 4 set_mon_vals no callback set 2024-04-22T08:21:03.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.055+0000 7f35567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:03.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.055+0000 7f35567fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:21:03.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.055+0000 7f35567fc640 10 monclient: dump: 2024-04-22T08:21:03.068 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:03.068 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:03.068 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:03.068 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:03.068 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:03.068 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:03.068 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:03.068 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:03.068 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:03.068 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:03.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.055+0000 7f3568021640 10 monclient: _renew_subs 2024-04-22T08:21:03.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.055+0000 7f3568021640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:21:03.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.055+0000 7f3568021640 1 -- 172.21.15.40:0/431880785 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3560124090 con 0x7f3560136750 2024-04-22T08:21:03.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.056+0000 7f35567fc640 1 -- 172.21.15.40:0/431880785 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f355803e9a0 con 0x7f3560136750 2024-04-22T08:21:03.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.056+0000 7f3568021640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:03.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.056+0000 7f3568021640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:03.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.056+0000 7f3568021640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:21:03.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.056+0000 7f3568021640 1 -- 172.21.15.40:0/431880785 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f356004e9b0 con 0x7f3560136750 2024-04-22T08:21:03.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.056+0000 7f35567fc640 1 --2- 172.21.15.40:0/431880785 >> v2:172.21.15.189:6800/30168 conn(0x7f3544037800 0x7f3544039cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:03.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.056+0000 7f35567fc640 1 -- 172.21.15.40:0/431880785 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(274..274 src has 239..274) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f355802ce40 con 0x7f3560136750 2024-04-22T08:21:03.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.056+0000 7f3565dbd640 1 --2- 172.21.15.40:0/431880785 >> v2:172.21.15.189:6800/30168 conn(0x7f3544037800 0x7f3544039cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:03.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.056+0000 7f3565dbd640 10 monclient: get_auth_request con 0x7f3544037800 auth_method 0 2024-04-22T08:21:03.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.058+0000 7f3565dbd640 1 --2- 172.21.15.40:0/431880785 >> v2:172.21.15.189:6800/30168 conn(0x7f3544037800 0x7f3544039cc0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f355c006fd0 tx=0x7f355c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:03.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.061+0000 7f35567fc640 1 -- 172.21.15.40:0/431880785 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f3558033d70 con 0x7f3560136750 2024-04-22T08:21:03.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.061+0000 7f35567fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:03.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.061+0000 7f35567fc640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:21:03.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.320+0000 7f3568021640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-22T08:21:03.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.321+0000 7f3568021640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-22T08:21:03.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.321+0000 7f3568021640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:21:03.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.321+0000 7f3568021640 1 -- 172.21.15.40:0/431880785 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f3560122d50 con 0x7f3560136750 2024-04-22T08:21:03.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.322+0000 7f35567fc640 1 -- 172.21.15.40:0/431880785 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f3558033b90 con 0x7f3560136750 2024-04-22T08:21:03.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.322+0000 7f35567fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-22T08:21:03.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.322+0000 7f35567fc640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:21:03.322 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.322+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:03.322 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.322+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=18,config=2,health=11,kv=8,logm=61} crc {auth=1272072322,config=1618278320,health=3888942785,kv=1244876340,logm=4237338664}) 2024-04-22T08:21:03.322 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.322+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=17,config=2,health=10,kv=9,logm=62} crc {auth=798127616,config=1618278320,health=3917380091,kv=2184066631,logm=1474970715}) 2024-04-22T08:21:03.322 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.322+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:03.322 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.322+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=18,config=2,health=11,kv=8,logm=61} crc {auth=1272072322,config=1618278320,health=3888942785,kv=1244876340,logm=4237338664}) 2024-04-22T08:21:03.322 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.322+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=17,config=2,health=12,kv=9,logm=60} crc {auth=1645622226,config=1618278320,health=1703418744,kv=1350343662,logm=2990405955}) 2024-04-22T08:21:03.323 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.323+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:03.323 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.323+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1426129998}) 2024-04-22T08:21:03.323 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.323+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4093252747}) 2024-04-22T08:21:03.323 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.323+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:03.323 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.323+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1426129998}) 2024-04-22T08:21:03.323 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.323+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4248287336}) 2024-04-22T08:21:03.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.324+0000 7f3568021640 1 -- 172.21.15.40:0/431880785 >> v2:172.21.15.189:6800/30168 conn(0x7f3544037800 msgr2=0x7f3544039cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:03.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.324+0000 7f3568021640 1 --2- 172.21.15.40:0/431880785 >> v2:172.21.15.189:6800/30168 conn(0x7f3544037800 0x7f3544039cc0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f355c006fd0 tx=0x7f355c008040 comp rx=0 tx=0).stop 2024-04-22T08:21:03.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.324+0000 7f3568021640 10 monclient: shutdown 2024-04-22T08:21:03.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.324+0000 7f3568021640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:03.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.324+0000 7f3568021640 1 -- 172.21.15.40:0/431880785 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3560136750 msgr2=0x7f3560121470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:03.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.324+0000 7f3568021640 1 --2- 172.21.15.40:0/431880785 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3560136750 0x7f3560121470 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3558009e50 tx=0x7f3558004a40 comp rx=0 tx=0).stop 2024-04-22T08:21:03.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.324+0000 7f3568021640 1 -- 172.21.15.40:0/431880785 shutdown_connections 2024-04-22T08:21:03.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.324+0000 7f3568021640 1 --2- 172.21.15.40:0/431880785 >> v2:172.21.15.189:6800/30168 conn(0x7f3544037800 0x7f3544039cc0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:03.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.324+0000 7f3568021640 1 --2- 172.21.15.40:0/431880785 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3560135e30 0x7f3560120f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:03.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.324+0000 7f3568021640 1 --2- 172.21.15.40:0/431880785 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3560136750 0x7f3560121470 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:03.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.324+0000 7f3568021640 1 --2- 172.21.15.40:0/431880785 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3560071c90 0x7f35601209f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:03.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.324+0000 7f3568021640 1 -- 172.21.15.40:0/431880785 >> 172.21.15.40:0/431880785 conn(0x7f35600a65c0 msgr2=0x7f3560134380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:03.325 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.324+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:03.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.324+0000 7f3568021640 1 -- 172.21.15.40:0/431880785 shutdown_connections 2024-04-22T08:21:03.326 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.324+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3383103418}) 2024-04-22T08:21:03.326 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.324+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=861644485}) 2024-04-22T08:21:03.326 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.324+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:03.326 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.324+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3383103418}) 2024-04-22T08:21:03.326 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.324+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=964740352}) 2024-04-22T08:21:03.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:03.324+0000 7f3568021640 1 -- 172.21.15.40:0/431880785 wait complete. 2024-04-22T08:21:03.328 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.325+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:03.328 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.325+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2053025105}) 2024-04-22T08:21:03.328 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.325+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=820800711}) 2024-04-22T08:21:03.328 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.325+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:03.328 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.326+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2053025105}) 2024-04-22T08:21:03.328 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.326+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1143613958}) 2024-04-22T08:21:03.329 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.326+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:03.329 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.326+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3702288155}) 2024-04-22T08:21:03.329 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.326+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3101759164}) 2024-04-22T08:21:03.329 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.326+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:03.329 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.326+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3702288155}) 2024-04-22T08:21:03.329 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.327+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4175617817}) 2024-04-22T08:21:03.329 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.327+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:03.329 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.327+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3687610094}) 2024-04-22T08:21:03.329 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.327+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3678573083}) 2024-04-22T08:21:03.329 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.328+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:03.329 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.328+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3687610094}) 2024-04-22T08:21:03.329 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.328+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1533072222}) 2024-04-22T08:21:03.329 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.328+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:03.329 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.328+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=830360186}) 2024-04-22T08:21:03.330 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.328+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4213422728}) 2024-04-22T08:21:03.330 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.328+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:03.330 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.328+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=830360186}) 2024-04-22T08:21:03.330 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.329+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1522911839}) 2024-04-22T08:21:03.331 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.329+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:03.331 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.329+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=946426129}) 2024-04-22T08:21:03.331 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.330+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3682760382}) 2024-04-22T08:21:03.331 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.330+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:03.331 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.330+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=946426129}) 2024-04-22T08:21:03.331 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.330+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3352968688}) 2024-04-22T08:21:03.332 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.330+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:03.332 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.330+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=405101771}) 2024-04-22T08:21:03.332 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.330+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=661285115}) 2024-04-22T08:21:03.332 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.331+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:03.332 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.331+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=405101771}) 2024-04-22T08:21:03.332 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.331+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1367682463}) 2024-04-22T08:21:03.332 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.331+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:03.332 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.331+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2132603391}) 2024-04-22T08:21:03.332 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.331+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3649687017}) 2024-04-22T08:21:03.332 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.331+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:03.332 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.331+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2132603391}) 2024-04-22T08:21:03.332 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.331+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2283706653}) 2024-04-22T08:21:03.334 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.332+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:03.334 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.332+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=44416462}) 2024-04-22T08:21:03.334 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.332+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=526678512}) 2024-04-22T08:21:03.334 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.332+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:03.334 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.332+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=44416462}) 2024-04-22T08:21:03.334 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.332+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1531706496}) 2024-04-22T08:21:03.334 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.333+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:03.334 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.333+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4278964013}) 2024-04-22T08:21:03.334 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.333+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3904113812}) 2024-04-22T08:21:03.334 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.333+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:03.334 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.333+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4278964013}) 2024-04-22T08:21:03.334 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.333+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3438846738}) 2024-04-22T08:21:03.335 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.334+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:03.335 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.334+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4264396186}) 2024-04-22T08:21:03.335 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.334+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=182126491}) 2024-04-22T08:21:03.335 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.334+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:03.335 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.334+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4264396186}) 2024-04-22T08:21:03.335 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.334+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=311365558}) 2024-04-22T08:21:03.336 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.335+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:03.336 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.335+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1708285939}) 2024-04-22T08:21:03.336 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.335+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3835327889}) 2024-04-22T08:21:03.336 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.335+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:03.336 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.335+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1708285939}) 2024-04-22T08:21:03.336 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.335+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=118897915}) 2024-04-22T08:21:03.337 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-22T08:21:03.340 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.335+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:03.340 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.336+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=836612381}) 2024-04-22T08:21:03.340 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.336+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=112137108}) 2024-04-22T08:21:03.341 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.336+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:03.341 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.336+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=836612381}) 2024-04-22T08:21:03.341 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.336+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=781201422}) 2024-04-22T08:21:03.341 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.336+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:03.341 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.336+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1096066952}) 2024-04-22T08:21:03.341 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.336+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3557106418}) 2024-04-22T08:21:03.341 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.336+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:03.341 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.336+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1096066952}) 2024-04-22T08:21:03.341 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.336+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3870058131}) 2024-04-22T08:21:03.341 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.337+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:03.341 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.337+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2417946345}) 2024-04-22T08:21:03.341 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.337+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3566605992}) 2024-04-22T08:21:03.341 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.337+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:03.341 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.337+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2417946345}) 2024-04-22T08:21:03.342 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.337+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2106315366}) 2024-04-22T08:21:03.342 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.338+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:03.342 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.338+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4190621822}) 2024-04-22T08:21:03.342 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.338+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=755493131}) 2024-04-22T08:21:03.342 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.338+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:03.342 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.338+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4190621822}) 2024-04-22T08:21:03.342 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.338+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2890257814}) 2024-04-22T08:21:03.342 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.339+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:03.342 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.339+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2269691026}) 2024-04-22T08:21:03.342 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.339+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1734696485}) 2024-04-22T08:21:03.342 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.339+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:03.342 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.339+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2269691026}) 2024-04-22T08:21:03.342 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.339+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1721778713}) 2024-04-22T08:21:03.343 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.339+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:03.343 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.340+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2398159730}) 2024-04-22T08:21:03.343 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.340+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3922969895}) 2024-04-22T08:21:03.343 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.340+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:03.343 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.340+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2398159730}) 2024-04-22T08:21:03.343 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.340+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3434534426}) 2024-04-22T08:21:03.343 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.340+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:03.343 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.340+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=34,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=8,monmap=3,osd_metadata=5,osd_pg_creating=1,osdmap=27} crc {logm=2592828104,mdsmap=1531151339,mgr=878880162,mgr_command_descs=470045235,mgr_metadata=4222303120,mgrstat=434958566,mon_config_key=4255067121,monmap=3469086636,osd_metadata=280645704,osd_pg_creating=3124688284,osdmap=3493414471}) 2024-04-22T08:21:03.343 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.340+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=34,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=8,monmap=3,osd_metadata=5,osd_pg_creating=1,osdmap=28} crc {logm=852092518,mdsmap=1531151339,mgr=878880162,mgr_command_descs=470045235,mgr_metadata=4222303120,mgrstat=5352870,mon_config_key=4255067121,monmap=3469086636,osd_metadata=434999189,osd_pg_creating=3124688284,osdmap=215908836}) 2024-04-22T08:21:03.343 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.340+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:03.343 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.340+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=34,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=8,monmap=3,osd_metadata=5,osd_pg_creating=1,osdmap=27} crc {logm=2592828104,mdsmap=1531151339,mgr=878880162,mgr_command_descs=470045235,mgr_metadata=4222303120,mgrstat=434958566,mon_config_key=4255067121,monmap=3469086636,osd_metadata=280645704,osd_pg_creating=3124688284,osdmap=3493414471}) 2024-04-22T08:21:03.343 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.341+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=33,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=7,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=29} crc {logm=318030323,mdsmap=1531151339,mgr=493696324,mgr_command_descs=470045235,mgr_metadata=4222303120,mgrstat=434958566,mon_config_key=1659486084,monmap=3469086636,osd_metadata=4151711904,osd_pg_creating=3124688284,osdmap=763366706}) 2024-04-22T08:21:03.344 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.345+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:03.345 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.345+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=45} crc {osdmap=44486446}) 2024-04-22T08:21:03.345 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.345+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=43} crc {osdmap=1188252918}) 2024-04-22T08:21:03.345 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.345+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:03.345 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.345+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=45} crc {osdmap=44486446}) 2024-04-22T08:21:03.345 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:03.345+0000 7fc179368640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=43} crc {osdmap=2287787386}) 2024-04-22T08:21:03.406 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38894]: starting. 2024-04-22T08:21:03.406 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38894]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-38893 2024-04-22T08:21:03.406 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38894]: created object 0... 2024-04-22T08:21:03.406 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38894]: created object 25... 2024-04-22T08:21:03.406 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38894]: created object 49... 2024-04-22T08:21:03.406 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38894]: finishing. 2024-04-22T08:21:03.406 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[38894]: shutting down. 2024-04-22T08:21:04.304 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[38895]: starting. 2024-04-22T08:21:04.304 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[38895]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-38893 2024-04-22T08:21:04.304 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[38895]: shutting down. 2024-04-22T08:21:04.338 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-22T08:21:04.339 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:21:04.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.445+0000 7fb4d58a1640 10 monclient: get_monmap_and_config 2024-04-22T08:21:04.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.445+0000 7fb4d58a1640 10 monclient: build_initial_monmap 2024-04-22T08:21:04.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.445+0000 7fb4d58a1640 10 monclient: monmap: 2024-04-22T08:21:04.445 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:04.445 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:04.445 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:21:04.446467+0000 2024-04-22T08:21:04.445 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:21:04.446467+0000 2024-04-22T08:21:04.445 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:04.445 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:04.445 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:21:04.446 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:21:04.446 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:21:04.446 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:21:04.446 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:21:04.446 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:04.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.446+0000 7fb4d58a1640 1 Processor -- start 2024-04-22T08:21:04.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.446+0000 7fb4d58a1640 1 -- start start 2024-04-22T08:21:04.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.446+0000 7fb4d58a1640 10 monclient: init 2024-04-22T08:21:04.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.446+0000 7fb4d58a1640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:04.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.446+0000 7fb4d58a1640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-04-22T08:21:04.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.446+0000 7fb4d58a1640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb4d0132b00 0x7fb4d009af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:04.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.446+0000 7fb4d58a1640 10 monclient(hunting): picked mon.noname-a con 0x7fb4d0132b00 addr v2:172.21.15.40:3300/0 2024-04-22T08:21:04.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.446+0000 7fb4d58a1640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb4d0132ee0 0x7fb4d00a0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:04.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.446+0000 7fb4d58a1640 10 monclient(hunting): picked mon.noname-c con 0x7fb4d0132ee0 addr v2:172.21.15.189:3300/0 2024-04-22T08:21:04.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.446+0000 7fb4d58a1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb4d0133350 addr v1:172.21.15.40:6789/0 2024-04-22T08:21:04.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.446+0000 7fb4d58a1640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4d009b4b0 con 0x7fb4d0133350 2024-04-22T08:21:04.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.446+0000 7fb4d58a1640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:04.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.446+0000 7fb4d58a1640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4d006c840 con 0x7fb4d0132b00 2024-04-22T08:21:04.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.446+0000 7fb4d58a1640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:04.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.446+0000 7fb4d58a1640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4d009a220 con 0x7fb4d0132ee0 2024-04-22T08:21:04.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.446+0000 7fb4d58a1640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:04.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.446+0000 7fb4d58a1640 10 monclient(hunting): authenticate will time out at 1338.440744s 2024-04-22T08:21:04.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.447+0000 7fb4ceffd640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb4d0132b00 0x7fb4d009af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:04.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.447+0000 7fb4cf7fe640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fb4d0133350 0x7fb4d00a0f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:36900/0 (socket says 172.21.15.40:36900) 2024-04-22T08:21:04.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.447+0000 7fb4ceffd640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb4d0132b00 0x7fb4d009af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45958/0 (socket says 172.21.15.40:45958) 2024-04-22T08:21:04.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.447+0000 7fb4cf7fe640 1 -- 172.21.15.40:0/2851947164 learned_addr learned my addr 172.21.15.40:0/2851947164 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:21:04.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.447+0000 7fb4ceffd640 10 monclient(hunting): get_auth_request con 0x7fb4d0132b00 auth_method 0 2024-04-22T08:21:04.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.447+0000 7fb4ceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:04.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.447+0000 7fb4ceffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:04.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.447+0000 7fb4ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:04.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.447+0000 7fb4ceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:04.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.447+0000 7fb4ceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:04.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.447+0000 7fb4ceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:04.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.447+0000 7fb4c7fff640 1 --2- 172.21.15.40:0/2851947164 >> v2:172.21.15.189:3300/0 conn(0x7fb4d0132ee0 0x7fb4d00a0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:04.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.447+0000 7fb4c7fff640 10 monclient(hunting): get_auth_request con 0x7fb4d0132ee0 auth_method 0 2024-04-22T08:21:04.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.447+0000 7fb4c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:04.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.447+0000 7fb4c7fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:04.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.447+0000 7fb4c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:04.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.447+0000 7fb4ceffd640 10 monclient(hunting): handle_auth_done global_id 94386 payload 274 2024-04-22T08:21:04.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.447+0000 7fb4ceffd640 1 -- 172.21.15.40:0/2851947164 >> v2:172.21.15.189:3300/0 conn(0x7fb4d0132ee0 msgr2=0x7fb4d00a0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:21:04.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.447+0000 7fb4ceffd640 1 --2- 172.21.15.40:0/2851947164 >> v2:172.21.15.189:3300/0 conn(0x7fb4d0132ee0 0x7fb4d00a0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:04.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.447+0000 7fb4ceffd640 1 -- 172.21.15.40:0/2851947164 >> v1:172.21.15.40:6789/0 conn(0x7fb4d0133350 legacy=0x7fb4d00a0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:21:04.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.447+0000 7fb4ceffd640 10 monclient: _finish_hunting 0 2024-04-22T08:21:04.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.447+0000 7fb4ceffd640 1 monclient: found mon.noname-a 2024-04-22T08:21:04.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.447+0000 7fb4ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:04.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.447+0000 7fb4ceffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:21:04.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.447+0000 7fb4ceffd640 1 -- 172.21.15.40:0/2851947164 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4d009a500 con 0x7fb4d0132b00 2024-04-22T08:21:04.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.450+0000 7fb4ceffd640 1 -- 172.21.15.40:0/2851947164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4d0132b00 msgr2=0x7fb4d009af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:04.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.450+0000 7fb4ceffd640 1 --2- 172.21.15.40:0/2851947164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4d0132b00 0x7fb4d009af70 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb4d0099f40 tx=0x7fb4c002b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94dab985f42b96f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:04.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.450+0000 7fb4ce7fc640 1 -- 172.21.15.40:0/2851947164 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c0030920 con 0x7fb4d0132b00 2024-04-22T08:21:04.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.450+0000 7fb4ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:04.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.450+0000 7fb4ce7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:21:04.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.450+0000 7fb4ce7fc640 10 monclient: dump: 2024-04-22T08:21:04.458 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:04.458 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:04.458 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:04.458 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:04.458 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:04.458 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:04.458 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:04.458 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:04.458 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:04.458 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:04.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.450+0000 7fb4ce7fc640 10 monclient: _finish_auth 0 2024-04-22T08:21:04.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.450+0000 7fb4ce7fc640 10 monclient: _check_auth_tickets 2024-04-22T08:21:04.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.450+0000 7fb4ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:04.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.450+0000 7fb4ce7fc640 1 -- 172.21.15.40:0/2851947164 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4c002e050 con 0x7fb4d0132b00 2024-04-22T08:21:04.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.450+0000 7fb4ce7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:04.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.450+0000 7fb4ce7fc640 1 -- 172.21.15.40:0/2851947164 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c0030ca0 con 0x7fb4d0132b00 2024-04-22T08:21:04.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.450+0000 7fb4d58a1640 5 monclient: authenticate success, global_id 94386 2024-04-22T08:21:04.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.450+0000 7fb4d58a1640 10 monclient: get_monmap_and_config success 2024-04-22T08:21:04.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.450+0000 7fb4d58a1640 4 set_mon_vals no callback set 2024-04-22T08:21:04.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.450+0000 7fb4ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:04.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.450+0000 7fb4ce7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:21:04.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.450+0000 7fb4ce7fc640 10 monclient: dump: 2024-04-22T08:21:04.459 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:04.459 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:04.459 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:04.460 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:04.460 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:04.460 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:04.460 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:04.460 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:04.460 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:04.460 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:04.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.451+0000 7fb4d58a1640 10 monclient: shutdown 2024-04-22T08:21:04.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.451+0000 7fb4d58a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:04.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.451+0000 7fb4d58a1640 1 -- 172.21.15.40:0/2851947164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4d0132b00 msgr2=0x7fb4d009af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:04.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.451+0000 7fb4d58a1640 1 --2- 172.21.15.40:0/2851947164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4d0132b00 0x7fb4d009af70 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb4d0099f40 tx=0x7fb4c002b730 comp rx=0 tx=0).stop 2024-04-22T08:21:04.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.451+0000 7fb4d58a1640 1 -- 172.21.15.40:0/2851947164 shutdown_connections 2024-04-22T08:21:04.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.451+0000 7fb4d58a1640 1 --2- 172.21.15.40:0/2851947164 >> v2:172.21.15.189:3300/0 conn(0x7fb4d0132ee0 0x7fb4d00a0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:04.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.451+0000 7fb4d58a1640 1 --2- 172.21.15.40:0/2851947164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4d0132b00 0x7fb4d009af70 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:04.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.451+0000 7fb4d58a1640 1 -- 172.21.15.40:0/2851947164 >> 172.21.15.40:0/2851947164 conn(0x7fb4d00a5960 msgr2=0x7fb4d00a1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:04.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.451+0000 7fb4d58a1640 1 -- 172.21.15.40:0/2851947164 shutdown_connections 2024-04-22T08:21:04.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.451+0000 7fb4d58a1640 1 -- 172.21.15.40:0/2851947164 wait complete. 2024-04-22T08:21:04.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.451+0000 7fb4d58a1640 10 monclient: build_initial_monmap 2024-04-22T08:21:04.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.451+0000 7fb4d58a1640 10 monclient: monmap: 2024-04-22T08:21:04.461 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:04.461 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:21:04.461 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:21:04.461 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:21:04.461 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:04.461 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:04.461 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:21:04.461 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:21:04.461 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:21:04.462 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:04.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.452+0000 7fb4d58a1640 1 Processor -- start 2024-04-22T08:21:04.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.452+0000 7fb4d58a1640 1 -- start start 2024-04-22T08:21:04.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.452+0000 7fb4d58a1640 10 monclient: init 2024-04-22T08:21:04.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.452+0000 7fb4d58a1640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:04.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.452+0000 7fb4d58a1640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:21:04.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.452+0000 7fb4d58a1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4d0132b00 0x7fb4d01caae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:04.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.452+0000 7fb4d58a1640 10 monclient(hunting): picked mon.noname-a con 0x7fb4d0132b00 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:21:04.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.452+0000 7fb4d58a1640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4d0132ee0 0x7fb4d01cb020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:04.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.452+0000 7fb4d58a1640 10 monclient(hunting): picked mon.noname-b con 0x7fb4d0132ee0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:21:04.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.452+0000 7fb4d58a1640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb4d0133350 0x7fb4d01220c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:04.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.452+0000 7fb4d58a1640 10 monclient(hunting): picked mon.noname-c con 0x7fb4d0133350 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:04.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.452+0000 7fb4d58a1640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:04.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.452+0000 7fb4d58a1640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d0124550 con 0x7fb4d0132b00 2024-04-22T08:21:04.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.452+0000 7fb4d58a1640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:04.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.452+0000 7fb4d58a1640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d009b8c0 con 0x7fb4d0132ee0 2024-04-22T08:21:04.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.452+0000 7fb4d58a1640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:04.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.452+0000 7fb4d58a1640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d009ed20 con 0x7fb4d0133350 2024-04-22T08:21:04.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.452+0000 7fb4d58a1640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:04.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.452+0000 7fb4d58a1640 10 monclient(hunting): authenticate will time out at 1338.446601s 2024-04-22T08:21:04.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.452+0000 7fb4ceffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4d0132b00 0x7fb4d01caae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:04.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.452+0000 7fb4ceffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4d0132b00 0x7fb4d01caae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45970/0 (socket says 172.21.15.40:45970) 2024-04-22T08:21:04.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.452+0000 7fb4ceffd640 1 -- 172.21.15.40:0/2585139398 learned_addr learned my addr 172.21.15.40:0/2585139398 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:21:04.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.452+0000 7fb4ceffd640 10 monclient(hunting): get_auth_request con 0x7fb4d0132b00 auth_method 0 2024-04-22T08:21:04.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.452+0000 7fb4ceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:04.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.452+0000 7fb4ceffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:04.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.452+0000 7fb4ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:04.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.452+0000 7fb4cf7fe640 1 --2- 172.21.15.40:0/2585139398 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb4d0133350 0x7fb4d01220c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:04.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.452+0000 7fb4cf7fe640 10 monclient(hunting): get_auth_request con 0x7fb4d0133350 auth_method 0 2024-04-22T08:21:04.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.452+0000 7fb4cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:04.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.452+0000 7fb4cf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:04.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.452+0000 7fb4cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:04.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.453+0000 7fb4ceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:04.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.453+0000 7fb4ceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:04.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.453+0000 7fb4ceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:04.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.453+0000 7fb4c7fff640 1 --2- 172.21.15.40:0/2585139398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4d0132ee0 0x7fb4d01cb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:04.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.453+0000 7fb4cf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:04.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.453+0000 7fb4cf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:04.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.453+0000 7fb4cf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:04.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.453+0000 7fb4c7fff640 10 monclient(hunting): get_auth_request con 0x7fb4d0132ee0 auth_method 0 2024-04-22T08:21:04.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.453+0000 7fb4c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:04.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.453+0000 7fb4c7fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:04.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.453+0000 7fb4c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:04.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.453+0000 7fb4c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:04.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.453+0000 7fb4c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:04.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.453+0000 7fb4c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:04.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.453+0000 7fb4ceffd640 10 monclient(hunting): handle_auth_done global_id 94389 payload 824 2024-04-22T08:21:04.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.453+0000 7fb4ceffd640 1 -- 172.21.15.40:0/2585139398 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb4d0133350 msgr2=0x7fb4d01220c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:04.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.453+0000 7fb4ceffd640 1 --2- 172.21.15.40:0/2585139398 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb4d0133350 0x7fb4d01220c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:04.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.453+0000 7fb4ceffd640 1 -- 172.21.15.40:0/2585139398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4d0132ee0 msgr2=0x7fb4d01cb020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:04.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.453+0000 7fb4ceffd640 1 --2- 172.21.15.40:0/2585139398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4d0132ee0 0x7fb4d01cb020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:04.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.453+0000 7fb4ceffd640 10 monclient: _finish_hunting 0 2024-04-22T08:21:04.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.453+0000 7fb4ceffd640 1 monclient: found mon.noname-a 2024-04-22T08:21:04.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.453+0000 7fb4ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:04.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.453+0000 7fb4ceffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:21:04.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.453+0000 7fb4ceffd640 1 -- 172.21.15.40:0/2585139398 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c0009400 con 0x7fb4d0132b00 2024-04-22T08:21:04.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.453+0000 7fb4cf7fe640 1 --2- 172.21.15.40:0/2585139398 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb4d0133350 0x7fb4d01220c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:21:04.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.455+0000 7fb4ceffd640 1 -- 172.21.15.40:0/2585139398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4d0132b00 msgr2=0x7fb4d01caae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:04.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.455+0000 7fb4ceffd640 1 --2- 172.21.15.40:0/2585139398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4d0132b00 0x7fb4d01caae0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb4c0000c00 tx=0x7fb4c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:04.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.455+0000 7fb4ccff9640 1 -- 172.21.15.40:0/2585139398 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c0032b60 con 0x7fb4d0132b00 2024-04-22T08:21:04.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.455+0000 7fb4ccff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:04.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.455+0000 7fb4ccff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:21:04.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.455+0000 7fb4ccff9640 10 monclient: dump: 2024-04-22T08:21:04.466 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:04.466 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:04.466 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:04.466 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:04.466 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:04.467 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:04.467 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:04.467 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:04.467 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:04.467 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:04.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.455+0000 7fb4ccff9640 10 monclient: _finish_auth 0 2024-04-22T08:21:04.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.455+0000 7fb4ccff9640 10 monclient: _check_auth_tickets 2024-04-22T08:21:04.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.455+0000 7fb4ccff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:04.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.455+0000 7fb4ccff9640 1 -- 172.21.15.40:0/2585139398 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4c0032ce0 con 0x7fb4d0132b00 2024-04-22T08:21:04.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.455+0000 7fb4d58a1640 5 monclient: authenticate success, global_id 94389 2024-04-22T08:21:04.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.455+0000 7fb4d58a1640 10 monclient: _renew_subs 2024-04-22T08:21:04.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.455+0000 7fb4d58a1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:21:04.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.455+0000 7fb4d58a1640 1 -- 172.21.15.40:0/2585139398 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4d0124720 con 0x7fb4d0132b00 2024-04-22T08:21:04.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.455+0000 7fb4ccff9640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:04.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.455+0000 7fb4ccff9640 1 -- 172.21.15.40:0/2585139398 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c003b640 con 0x7fb4d0132b00 2024-04-22T08:21:04.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.455+0000 7fb4ccff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:04.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.455+0000 7fb4ccff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:21:04.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.455+0000 7fb4ccff9640 10 monclient: dump: 2024-04-22T08:21:04.468 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:04.468 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:04.468 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:04.468 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:04.468 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:04.468 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:04.468 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:04.468 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:04.468 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:04.468 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:04.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.455+0000 7fb4d58a1640 10 monclient: _renew_subs 2024-04-22T08:21:04.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.455+0000 7fb4d58a1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:21:04.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.455+0000 7fb4cdffb640 4 set_mon_vals no callback set 2024-04-22T08:21:04.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.455+0000 7fb4d58a1640 1 -- 172.21.15.40:0/2585139398 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4d0125f20 con 0x7fb4d0132b00 2024-04-22T08:21:04.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.456+0000 7fb4ccff9640 1 -- 172.21.15.40:0/2585139398 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fb4c0039040 con 0x7fb4d0132b00 2024-04-22T08:21:04.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.456+0000 7fb4d58a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:04.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.456+0000 7fb4d58a1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:04.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.456+0000 7fb4d58a1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:21:04.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.456+0000 7fb4d58a1640 1 -- 172.21.15.40:0/2585139398 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4d0126890 con 0x7fb4d0132b00 2024-04-22T08:21:04.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.456+0000 7fb4ccff9640 1 --2- 172.21.15.40:0/2585139398 >> v2:172.21.15.189:6800/30168 conn(0x7fb4b0037820 0x7fb4b0039ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:04.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.456+0000 7fb4ccff9640 1 -- 172.21.15.40:0/2585139398 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(276..276 src has 239..276) v4 ==== 3506+0+0 (secure 0 0 0) 0x7fb4c006b9d0 con 0x7fb4d0132b00 2024-04-22T08:21:04.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.457+0000 7fb4c7fff640 1 --2- 172.21.15.40:0/2585139398 >> v2:172.21.15.189:6800/30168 conn(0x7fb4b0037820 0x7fb4b0039ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:04.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.457+0000 7fb4c7fff640 10 monclient: get_auth_request con 0x7fb4b0037820 auth_method 0 2024-04-22T08:21:04.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.459+0000 7fb4c7fff640 1 --2- 172.21.15.40:0/2585139398 >> v2:172.21.15.189:6800/30168 conn(0x7fb4b0037820 0x7fb4b0039ce0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fb4d009b380 tx=0x7fb4b8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:04.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.461+0000 7fb4ccff9640 1 -- 172.21.15.40:0/2585139398 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7fb4d0126890 con 0x7fb4d0132b00 2024-04-22T08:21:04.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.461+0000 7fb4ccff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:04.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.461+0000 7fb4ccff9640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:21:04.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.712+0000 7fb4d58a1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:21:04.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.712+0000 7fb4d58a1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:21:04.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.712+0000 7fb4d58a1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:21:04.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.712+0000 7fb4d58a1640 1 -- 172.21.15.40:0/2585139398 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb4d0126890 con 0x7fb4d0132b00 2024-04-22T08:21:04.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.712+0000 7fb4ccff9640 1 -- 172.21.15.40:0/2585139398 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7fb4d0126890 con 0x7fb4d0132b00 2024-04-22T08:21:04.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.712+0000 7fb4ccff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:21:04.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.712+0000 7fb4ccff9640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:21:04.713 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":82,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:21:04.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.714+0000 7fb4d58a1640 1 -- 172.21.15.40:0/2585139398 >> v2:172.21.15.189:6800/30168 conn(0x7fb4b0037820 msgr2=0x7fb4b0039ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:04.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.714+0000 7fb4d58a1640 1 --2- 172.21.15.40:0/2585139398 >> v2:172.21.15.189:6800/30168 conn(0x7fb4b0037820 0x7fb4b0039ce0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fb4d009b380 tx=0x7fb4b8008040 comp rx=0 tx=0).stop 2024-04-22T08:21:04.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.714+0000 7fb4d58a1640 10 monclient: shutdown 2024-04-22T08:21:04.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.714+0000 7fb4d58a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:04.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.714+0000 7fb4d58a1640 1 -- 172.21.15.40:0/2585139398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4d0132b00 msgr2=0x7fb4d01caae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:04.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.714+0000 7fb4d58a1640 1 --2- 172.21.15.40:0/2585139398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4d0132b00 0x7fb4d01caae0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb4c0000c00 tx=0x7fb4c00046c0 comp rx=0 tx=0).stop 2024-04-22T08:21:04.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.714+0000 7fb4d58a1640 1 -- 172.21.15.40:0/2585139398 shutdown_connections 2024-04-22T08:21:04.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.714+0000 7fb4d58a1640 1 --2- 172.21.15.40:0/2585139398 >> v2:172.21.15.189:6800/30168 conn(0x7fb4b0037820 0x7fb4b0039ce0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:04.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.714+0000 7fb4d58a1640 1 --2- 172.21.15.40:0/2585139398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4d0132ee0 0x7fb4d01cb020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:04.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.714+0000 7fb4d58a1640 1 --2- 172.21.15.40:0/2585139398 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb4d0133350 0x7fb4d01220c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:04.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.714+0000 7fb4d58a1640 1 --2- 172.21.15.40:0/2585139398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4d0132b00 0x7fb4d01caae0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:04.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.714+0000 7fb4d58a1640 1 -- 172.21.15.40:0/2585139398 >> 172.21.15.40:0/2585139398 conn(0x7fb4d00a5960 msgr2=0x7fb4d00a0880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:04.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.714+0000 7fb4d58a1640 1 -- 172.21.15.40:0/2585139398 shutdown_connections 2024-04-22T08:21:04.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.714+0000 7fb4d58a1640 1 -- 172.21.15.40:0/2585139398 wait complete. 2024-04-22T08:21:04.724 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-22T08:21:04.724 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-22T08:21:04.724 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-22T08:21:04.725 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-22T08:21:04.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.824+0000 7f4018070640 10 monclient: get_monmap_and_config 2024-04-22T08:21:04.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.824+0000 7f4018070640 10 monclient: build_initial_monmap 2024-04-22T08:21:04.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.824+0000 7f4018070640 10 monclient: monmap: 2024-04-22T08:21:04.824 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:04.824 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:04.824 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:21:04.825430+0000 2024-04-22T08:21:04.824 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:21:04.825430+0000 2024-04-22T08:21:04.824 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:04.824 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:04.824 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:21:04.824 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:21:04.825 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:21:04.825 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:21:04.825 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:21:04.825 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:04.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.825+0000 7f4018070640 1 Processor -- start 2024-04-22T08:21:04.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.825+0000 7f4018070640 1 -- start start 2024-04-22T08:21:04.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.825+0000 7f4018070640 10 monclient: init 2024-04-22T08:21:04.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.825+0000 7f4018070640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:04.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.825+0000 7f4018070640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-04-22T08:21:04.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.825+0000 7f4018070640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f401012fad0 0x7f401012feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:04.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.825+0000 7f4018070640 10 monclient(hunting): picked mon.noname-c con 0x7f401012fad0 addr v2:172.21.15.189:3300/0 2024-04-22T08:21:04.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.825+0000 7f4018070640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4010132920 0x7f4010132d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:04.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.825+0000 7f4018070640 10 monclient(hunting): picked mon.noname-a con 0x7f4010132920 addr v2:172.21.15.40:3300/0 2024-04-22T08:21:04.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.825+0000 7f4018070640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4010133290 0x7f4010139c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:04.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.825+0000 7f4018070640 10 monclient(hunting): picked mon.noname-b con 0x7f4010133290 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:04.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.825+0000 7f4018070640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:04.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.825+0000 7f4018070640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f401009ae50 con 0x7f4010132920 2024-04-22T08:21:04.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.825+0000 7f4018070640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:04.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.825+0000 7f4018070640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f401009efc0 con 0x7f401012fad0 2024-04-22T08:21:04.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.825+0000 7f4018070640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:04.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.825+0000 7f4018070640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f401009d070 con 0x7f4010133290 2024-04-22T08:21:04.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.825+0000 7f4018070640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:04.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.825+0000 7f4018070640 10 monclient(hunting): authenticate will time out at 1338.819659s 2024-04-22T08:21:04.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.826+0000 7f401560b640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4010132920 0x7f4010132d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:04.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.826+0000 7f401660d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4010133290 0x7f4010139c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:04.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.826+0000 7f401560b640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4010132920 0x7f4010132d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45974/0 (socket says 172.21.15.40:45974) 2024-04-22T08:21:04.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.826+0000 7f401560b640 1 -- 172.21.15.40:0/1785562092 learned_addr learned my addr 172.21.15.40:0/1785562092 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:21:04.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.826+0000 7f401560b640 10 monclient(hunting): get_auth_request con 0x7f4010132920 auth_method 0 2024-04-22T08:21:04.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.826+0000 7f401560b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:04.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.826+0000 7f401560b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:04.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.826+0000 7f401560b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:04.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.826+0000 7f401660d640 10 monclient(hunting): get_auth_request con 0x7f4010133290 auth_method 0 2024-04-22T08:21:04.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.826+0000 7f401660d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:04.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.826+0000 7f401660d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:04.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.826+0000 7f401660d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:04.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.826+0000 7f401560b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:04.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.826+0000 7f401560b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:04.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.826+0000 7f401560b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:04.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.826+0000 7f401660d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:04.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.826+0000 7f401660d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:04.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.826+0000 7f401660d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:04.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.826+0000 7f401560b640 10 monclient(hunting): handle_auth_done global_id 94392 payload 274 2024-04-22T08:21:04.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.826+0000 7f4015e0c640 1 --2- 172.21.15.40:0/1785562092 >> v2:172.21.15.189:3300/0 conn(0x7f401012fad0 0x7f401012feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:04.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.826+0000 7f401560b640 1 -- 172.21.15.40:0/1785562092 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4010133290 msgr2=0x7f4010139c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:21:04.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.826+0000 7f401560b640 1 --2- 172.21.15.40:0/1785562092 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4010133290 0x7f4010139c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:04.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.826+0000 7f401560b640 1 -- 172.21.15.40:0/1785562092 >> v2:172.21.15.189:3300/0 conn(0x7f401012fad0 msgr2=0x7f401012feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:21:04.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.826+0000 7f401560b640 1 --2- 172.21.15.40:0/1785562092 >> v2:172.21.15.189:3300/0 conn(0x7f401012fad0 0x7f401012feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:04.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.826+0000 7f401560b640 10 monclient: _finish_hunting 0 2024-04-22T08:21:04.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.826+0000 7f401560b640 1 monclient: found mon.noname-a 2024-04-22T08:21:04.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.826+0000 7f401560b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:04.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.826+0000 7f401560b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:21:04.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.826+0000 7f401560b640 1 -- 172.21.15.40:0/1785562092 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f401009d350 con 0x7f4010132920 2024-04-22T08:21:04.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.826+0000 7f401660d640 1 --2- 172.21.15.40:0/1785562092 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4010133290 0x7f4010139c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:21:04.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.829+0000 7f401560b640 1 -- 172.21.15.40:0/1785562092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4010132920 msgr2=0x7f4010132d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:04.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.829+0000 7f401560b640 1 --2- 172.21.15.40:0/1785562092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4010132920 0x7f4010132d50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f401006bb30 tx=0x7f400002bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b124e93398038648 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:04.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.829+0000 7f4014e0a640 1 -- 172.21.15.40:0/1785562092 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400002cd70 con 0x7f4010132920 2024-04-22T08:21:04.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.829+0000 7f4014e0a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:04.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.829+0000 7f4014e0a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:21:04.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.829+0000 7f4014e0a640 10 monclient: dump: 2024-04-22T08:21:04.838 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:04.838 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:04.838 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:04.838 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:04.838 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:04.838 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:04.838 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:04.838 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:04.838 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:04.838 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:04.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.829+0000 7f4014e0a640 10 monclient: _finish_auth 0 2024-04-22T08:21:04.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.829+0000 7f4014e0a640 10 monclient: _check_auth_tickets 2024-04-22T08:21:04.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.829+0000 7f4014e0a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:04.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.829+0000 7f4014e0a640 1 -- 172.21.15.40:0/1785562092 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4000034440 con 0x7f4010132920 2024-04-22T08:21:04.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.829+0000 7f4018070640 5 monclient: authenticate success, global_id 94392 2024-04-22T08:21:04.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.829+0000 7f4018070640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:21:04.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.829+0000 7f4014e0a640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:04.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.829+0000 7f4018070640 10 monclient: get_monmap_and_config success 2024-04-22T08:21:04.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.829+0000 7f4018070640 4 set_mon_vals no callback set 2024-04-22T08:21:04.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.829+0000 7f4014e0a640 1 -- 172.21.15.40:0/1785562092 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40000347d0 con 0x7f4010132920 2024-04-22T08:21:04.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.829+0000 7f4014e0a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:04.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.829+0000 7f4014e0a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:21:04.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.829+0000 7f4014e0a640 10 monclient: dump: 2024-04-22T08:21:04.839 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:04.839 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:04.839 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:04.840 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:04.840 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:04.840 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:04.840 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:04.840 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:04.840 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:04.840 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:04.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.829+0000 7f4018070640 10 monclient: shutdown 2024-04-22T08:21:04.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.829+0000 7f4018070640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:04.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.829+0000 7f4018070640 1 -- 172.21.15.40:0/1785562092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4010132920 msgr2=0x7f4010132d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:04.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.829+0000 7f4018070640 1 --2- 172.21.15.40:0/1785562092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4010132920 0x7f4010132d50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f401006bb30 tx=0x7f400002bdb0 comp rx=0 tx=0).stop 2024-04-22T08:21:04.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.829+0000 7f4018070640 1 -- 172.21.15.40:0/1785562092 shutdown_connections 2024-04-22T08:21:04.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.829+0000 7f4018070640 1 --2- 172.21.15.40:0/1785562092 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4010133290 0x7f4010139c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:04.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.829+0000 7f4018070640 1 --2- 172.21.15.40:0/1785562092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4010132920 0x7f4010132d50 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:04.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.829+0000 7f4018070640 1 --2- 172.21.15.40:0/1785562092 >> v2:172.21.15.189:3300/0 conn(0x7f401012fad0 0x7f401012feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:04.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.829+0000 7f4018070640 1 -- 172.21.15.40:0/1785562092 >> 172.21.15.40:0/1785562092 conn(0x7f40100a0ae0 msgr2=0x7f40100a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:04.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.830+0000 7f4018070640 1 -- 172.21.15.40:0/1785562092 shutdown_connections 2024-04-22T08:21:04.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.830+0000 7f4018070640 1 -- 172.21.15.40:0/1785562092 wait complete. 2024-04-22T08:21:04.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.830+0000 7f4018070640 10 monclient: build_initial_monmap 2024-04-22T08:21:04.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.830+0000 7f4018070640 10 monclient: monmap: 2024-04-22T08:21:04.841 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:04.841 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:21:04.841 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:21:04.841 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:21:04.841 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:04.841 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:04.841 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:21:04.841 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:21:04.841 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:21:04.842 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:04.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.830+0000 7f4018070640 1 Processor -- start 2024-04-22T08:21:04.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.830+0000 7f4018070640 1 -- start start 2024-04-22T08:21:04.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.830+0000 7f4018070640 10 monclient: init 2024-04-22T08:21:04.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f4018070640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:04.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f4018070640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-22T08:21:04.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f4018070640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f401012fad0 0x7f401011a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:04.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f4018070640 10 monclient(hunting): picked mon.noname-b con 0x7f401012fad0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:21:04.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f4018070640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4010132920 0x7f401011ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:04.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f4018070640 10 monclient(hunting): picked mon.noname-c con 0x7f4010132920 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:04.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f4018070640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4010133290 0x7f401011b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:04.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f4018070640 10 monclient(hunting): picked mon.noname-a con 0x7f4010133290 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:21:04.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f4018070640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:04.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f4018070640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401011da90 con 0x7f4010133290 2024-04-22T08:21:04.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f4018070640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:04.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f4018070640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401011dce0 con 0x7f401012fad0 2024-04-22T08:21:04.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f4018070640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:04.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f4018070640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4010071870 con 0x7f4010132920 2024-04-22T08:21:04.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f4018070640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:04.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f4018070640 10 monclient(hunting): authenticate will time out at 1338.825095s 2024-04-22T08:21:04.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f401560b640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4010132920 0x7f401011ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:04.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f401660d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4010133290 0x7f401011b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:04.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f401560b640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4010132920 0x7f401011ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:51838/0 (socket says 172.21.15.40:51838) 2024-04-22T08:21:04.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f401560b640 1 -- 172.21.15.40:0/673637537 learned_addr learned my addr 172.21.15.40:0/673637537 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:21:04.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f401660d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4010133290 0x7f401011b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45984/0 (socket says 172.21.15.40:45984) 2024-04-22T08:21:04.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f401560b640 10 monclient(hunting): get_auth_request con 0x7f4010132920 auth_method 0 2024-04-22T08:21:04.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f401560b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:04.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f401560b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:04.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f401560b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:04.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f401660d640 10 monclient(hunting): get_auth_request con 0x7f4010133290 auth_method 0 2024-04-22T08:21:04.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f401660d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:04.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f401660d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:04.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f401660d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:04.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f4015e0c640 1 --2- 172.21.15.40:0/673637537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f401012fad0 0x7f401011a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:04.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f401560b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:04.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f401560b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:04.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f401560b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:04.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f4015e0c640 10 monclient(hunting): get_auth_request con 0x7f401012fad0 auth_method 0 2024-04-22T08:21:04.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f4015e0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:04.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f4015e0c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:04.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f4015e0c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:04.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f401660d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:04.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f401660d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:04.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f401660d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:04.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f4015e0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:04.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f4015e0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:04.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f4015e0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:04.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f401560b640 10 monclient(hunting): handle_auth_done global_id 104135 payload 824 2024-04-22T08:21:04.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f401560b640 1 -- 172.21.15.40:0/673637537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f401012fad0 msgr2=0x7f401011a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:04.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f401560b640 1 --2- 172.21.15.40:0/673637537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f401012fad0 0x7f401011a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:04.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f401560b640 1 -- 172.21.15.40:0/673637537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4010133290 msgr2=0x7f401011b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:04.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f401560b640 1 --2- 172.21.15.40:0/673637537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4010133290 0x7f401011b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:04.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f401560b640 10 monclient: _finish_hunting 0 2024-04-22T08:21:04.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f401560b640 1 monclient: found mon.noname-c 2024-04-22T08:21:04.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f401560b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:04.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f401560b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:21:04.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f401560b640 1 -- 172.21.15.40:0/673637537 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f400800f050 con 0x7f4010132920 2024-04-22T08:21:04.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.831+0000 7f401660d640 1 --2- 172.21.15.40:0/673637537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4010133290 0x7f401011b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:21:04.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.834+0000 7f401560b640 1 -- 172.21.15.40:0/673637537 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4010132920 msgr2=0x7f401011ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:04.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.834+0000 7f401560b640 1 --2- 172.21.15.40:0/673637537 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4010132920 0x7f401011ab70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f400000b2c0 tx=0x7f40000046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:04.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.834+0000 7f40067fc640 1 -- 172.21.15.40:0/673637537 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4000043070 con 0x7f4010132920 2024-04-22T08:21:04.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.834+0000 7f40067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:04.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.834+0000 7f40067fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:21:04.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.834+0000 7f40067fc640 10 monclient: dump: 2024-04-22T08:21:04.846 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:04.846 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:04.846 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:04.846 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:04.846 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:04.846 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:04.847 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:04.847 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:04.847 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:04.847 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:04.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.834+0000 7f40067fc640 10 monclient: _finish_auth 0 2024-04-22T08:21:04.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.834+0000 7f40067fc640 10 monclient: _check_auth_tickets 2024-04-22T08:21:04.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.834+0000 7f40067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:04.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.834+0000 7f40067fc640 1 -- 172.21.15.40:0/673637537 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f400003f440 con 0x7f4010132920 2024-04-22T08:21:04.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.834+0000 7f4018070640 5 monclient: authenticate success, global_id 104135 2024-04-22T08:21:04.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.834+0000 7f4018070640 10 monclient: _renew_subs 2024-04-22T08:21:04.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.834+0000 7f4018070640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:21:04.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.834+0000 7f4018070640 1 -- 172.21.15.40:0/673637537 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f400000b580 con 0x7f4010132920 2024-04-22T08:21:04.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.834+0000 7f40067fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:04.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.834+0000 7f40067fc640 1 -- 172.21.15.40:0/673637537 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400003f7d0 con 0x7f4010132920 2024-04-22T08:21:04.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.834+0000 7f40067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:04.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.834+0000 7f40067fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:21:04.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.834+0000 7f40067fc640 10 monclient: dump: 2024-04-22T08:21:04.848 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:04.848 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:04.848 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:04.848 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:04.848 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:04.848 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:04.848 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:04.848 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:04.848 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:04.848 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:04.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.834+0000 7f4018070640 10 monclient: _renew_subs 2024-04-22T08:21:04.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.834+0000 7f4018070640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:21:04.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.834+0000 7f4018070640 1 -- 172.21.15.40:0/673637537 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f401011c060 con 0x7f4010132920 2024-04-22T08:21:04.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.834+0000 7f4007fff640 4 set_mon_vals no callback set 2024-04-22T08:21:04.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.835+0000 7f4018070640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:04.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.835+0000 7f4018070640 1 --2- 172.21.15.40:0/673637537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40101c70a0 0x7f401002b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:04.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.835+0000 7f4018070640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:04.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.835+0000 7f4018070640 1 -- 172.21.15.40:0/673637537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401011de70 con 0x7f40101c70a0 2024-04-22T08:21:04.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.835+0000 7f40067fc640 1 -- 172.21.15.40:0/673637537 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f400003b040 con 0x7f4010132920 2024-04-22T08:21:04.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.835+0000 7f4015e0c640 1 --2- 172.21.15.40:0/673637537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40101c70a0 0x7f401002b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:04.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.835+0000 7f4015e0c640 10 monclient: get_auth_request con 0x7f40101c70a0 auth_method 0 2024-04-22T08:21:04.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.835+0000 7f4015e0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:04.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.835+0000 7f4015e0c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:04.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.835+0000 7f4015e0c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:04.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.835+0000 7f4015e0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:04.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.835+0000 7f4015e0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:04.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.835+0000 7f4015e0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:04.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.835+0000 7f40067fc640 1 --2- 172.21.15.40:0/673637537 >> v2:172.21.15.189:6800/30168 conn(0x7f3ff40377d0 0x7f3ff4039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:04.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.835+0000 7f40067fc640 1 -- 172.21.15.40:0/673637537 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(276..276 src has 239..276) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f400006beb0 con 0x7f4010132920 2024-04-22T08:21:04.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.835+0000 7f4015e0c640 10 monclient(hunting): handle_auth_done global_id 94398 payload 824 2024-04-22T08:21:04.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.835+0000 7f4015e0c640 1 -- 172.21.15.40:0/673637537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f401009c9b0 con 0x7f40101c70a0 2024-04-22T08:21:04.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.835+0000 7f401660d640 1 --2- 172.21.15.40:0/673637537 >> v2:172.21.15.189:6800/30168 conn(0x7f3ff40377d0 0x7f3ff4039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:04.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.835+0000 7f401660d640 10 monclient: get_auth_request con 0x7f3ff40377d0 auth_method 0 2024-04-22T08:21:04.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.837+0000 7f4015e0c640 1 -- 172.21.15.40:0/673637537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40101c70a0 msgr2=0x7f401002b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:04.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.837+0000 7f4015e0c640 1 --2- 172.21.15.40:0/673637537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40101c70a0 0x7f401002b200 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f400c00f7e0 tx=0x7f400c00fca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:04.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.837+0000 7f40067fc640 1 -- 172.21.15.40:0/673637537 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400c00de30 con 0x7f40101c70a0 2024-04-22T08:21:04.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.837+0000 7f40067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:04.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.837+0000 7f40067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:21:04.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.837+0000 7f40067fc640 10 monclient: dump: 2024-04-22T08:21:04.850 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:04.851 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:04.851 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:04.851 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:04.851 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:04.851 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:04.851 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:04.851 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:04.851 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:04.851 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:04.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.838+0000 7f40067fc640 1 -- 172.21.15.40:0/673637537 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f400c00c040 con 0x7f40101c70a0 2024-04-22T08:21:04.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.838+0000 7f40067fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:04.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.838+0000 7f40067fc640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:21:04.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.838+0000 7f40067fc640 1 -- 172.21.15.40:0/673637537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40101c70a0 msgr2=0x7f401002b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:04.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.838+0000 7f40067fc640 1 --2- 172.21.15.40:0/673637537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40101c70a0 0x7f401002b200 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f400c00f7e0 tx=0x7f400c00fca0 comp rx=0 tx=0).stop 2024-04-22T08:21:04.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.838+0000 7f40067fc640 1 -- 172.21.15.40:0/673637537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40101c70a0 msgr2=0x7f401002b200 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:21:04.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.838+0000 7f40067fc640 1 --2- 172.21.15.40:0/673637537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40101c70a0 0x7f401002b200 secure :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f400c00f7e0 tx=0x7f400c00fca0 comp rx=0 tx=0).stop 2024-04-22T08:21:04.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.838+0000 7f401660d640 1 --2- 172.21.15.40:0/673637537 >> v2:172.21.15.189:6800/30168 conn(0x7f3ff40377d0 0x7f3ff4039c90 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f400800dfd0 tx=0x7f400800db60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:04.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.844+0000 7f4018070640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:21:04.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.844+0000 7f4018070640 1 --2- 172.21.15.40:0/673637537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40101c7480 0x7f401002b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:04.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.844+0000 7f4018070640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:04.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.844+0000 7f4018070640 1 -- 172.21.15.40:0/673637537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40100667a0 con 0x7f40101c7480 2024-04-22T08:21:04.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.844+0000 7f4015e0c640 1 --2- 172.21.15.40:0/673637537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40101c7480 0x7f401002b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:04.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.844+0000 7f4015e0c640 10 monclient: get_auth_request con 0x7f40101c7480 auth_method 0 2024-04-22T08:21:04.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.844+0000 7f4015e0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:04.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.844+0000 7f4015e0c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:04.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.844+0000 7f4015e0c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:04.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.844+0000 7f4015e0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:04.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.844+0000 7f4015e0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:04.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.844+0000 7f4015e0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:04.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.844+0000 7f4015e0c640 10 monclient(hunting): handle_auth_done global_id 94401 payload 824 2024-04-22T08:21:04.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.844+0000 7f4015e0c640 1 -- 172.21.15.40:0/673637537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f40100a3ad0 con 0x7f40101c7480 2024-04-22T08:21:04.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.847+0000 7f4015e0c640 1 -- 172.21.15.40:0/673637537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40101c7480 msgr2=0x7f401002b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:04.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.847+0000 7f4015e0c640 1 --2- 172.21.15.40:0/673637537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40101c7480 0x7f401002b740 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f400c00f7e0 tx=0x7f400c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:04.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.847+0000 7f40067fc640 1 -- 172.21.15.40:0/673637537 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400c013440 con 0x7f40101c7480 2024-04-22T08:21:04.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.847+0000 7f40067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:04.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.847+0000 7f40067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:21:04.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.847+0000 7f40067fc640 10 monclient: dump: 2024-04-22T08:21:04.855 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:04.855 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:04.855 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:04.855 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:04.855 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:04.855 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:04.855 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:04.855 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:04.855 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:04.855 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:04.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.847+0000 7f40067fc640 1 -- 172.21.15.40:0/673637537 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4633 (secure 0 0 0) 0x7f400c021040 con 0x7f40101c7480 2024-04-22T08:21:04.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.847+0000 7f40067fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:21:04.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.847+0000 7f40067fc640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:21:04.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.847+0000 7f40067fc640 1 -- 172.21.15.40:0/673637537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40101c7480 msgr2=0x7f401002b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:04.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.847+0000 7f40067fc640 1 --2- 172.21.15.40:0/673637537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40101c7480 0x7f401002b740 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f400c00f7e0 tx=0x7f400c0046c0 comp rx=0 tx=0).stop 2024-04-22T08:21:04.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.847+0000 7f40067fc640 1 -- 172.21.15.40:0/673637537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40101c7480 msgr2=0x7f401002b740 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:21:04.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.847+0000 7f40067fc640 1 --2- 172.21.15.40:0/673637537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40101c7480 0x7f401002b740 secure :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f400c00f7e0 tx=0x7f400c0046c0 comp rx=0 tx=0).stop 2024-04-22T08:21:04.856 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:21:04.856 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:21:04.856 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:21:04.856 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "leader", 2024-04-22T08:21:04.856 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 82, 2024-04-22T08:21:04.856 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:21:04.857 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:21:04.857 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:21:04.857 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:21:04.857 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:04.857 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 6, 2024-04-22T08:21:04.857 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:21:04.857 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:21:04.857 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:21:04.857 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:21:04.857 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:21:04.857 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:21:04.857 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:21:04.857 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:21:04.857 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:21:04.858 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:21:04.858 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:21:04.858 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:21:04.858 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:21:04.858 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:04.858 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:21:04.858 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:21:04.858 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:21:04.858 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:21:04.858 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:21:04.858 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:21:04.858 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:21:04.858 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:21:04.859 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:21:04.859 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:21:04.859 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:21:04.859 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:21:04.859 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:04.859 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:04.859 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:21:04.859 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:21:04.859 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:21:04.859 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:21:04.859 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:21:04.859 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:21:04.859 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:21:04.859 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:21:04.860 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:21:04.860 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:21:04.860 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:21:04.860 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:21:04.860 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:21:04.860 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:21:04.860 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:21:04.860 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:21:04.860 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:21:04.860 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:21:04.860 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:21:04.860 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:21:04.860 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:21:04.860 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:21:04.860 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:21:04.861 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:21:04.861 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:21:04.861 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:21:04.861 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:21:04.861 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:04.861 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:21:04.861 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:04.861 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:21:04.861 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:04.861 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:21:04.861 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:21:04.861 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:21:04.861 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:21:04.861 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:04.862 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:21:04.862 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:21:04.862 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:04.862 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:04.862 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:04.862 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:21:04.862 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:21:04.862 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:04.862 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:04.862 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:04.862 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:04.862 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:21:04.862 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:21:04.862 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:21:04.863 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:21:04.863 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:21:04.863 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:04.863 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:04.863 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:21:04.863 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:21:04.863 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:21:04.863 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:21:04.863 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:04.863 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:21:04.863 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:21:04.863 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:04.863 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:04.863 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:04.863 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:21:04.864 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:21:04.864 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:04.864 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:04.864 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:04.864 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:04.864 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:21:04.864 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:21:04.864 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:21:04.864 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:21:04.864 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:21:04.864 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:04.864 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:04.864 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:21:04.864 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:21:04.864 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:21:04.865 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:21:04.865 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:04.865 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:21:04.865 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:21:04.865 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:04.865 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:04.865 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:04.865 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:21:04.865 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:21:04.865 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:04.865 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:04.865 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:04.865 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:04.865 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:21:04.866 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:21:04.866 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:21:04.866 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:21:04.866 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:21:04.866 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:04.866 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:04.866 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:04.866 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:21:04.866 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:21:04.866 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:04.866 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:21:04.866 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:21:04.866 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:21:04.866 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:04.867 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:04.867 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:21:04.867 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:04.867 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:21:04.867 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:21:04.867 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 2 2024-04-22T08:21:04.867 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:04.867 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:04.867 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:21:04.867 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:04.867 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:21:04.867 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:21:04.867 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 2 2024-04-22T08:21:04.867 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:04.868 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:04.868 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:04.868 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:21:04.868 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:21:04.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.847+0000 7f4018070640 1 -- 172.21.15.40:0/673637537 >> v2:172.21.15.189:6800/30168 conn(0x7f3ff40377d0 msgr2=0x7f3ff4039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:04.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.847+0000 7f4018070640 1 --2- 172.21.15.40:0/673637537 >> v2:172.21.15.189:6800/30168 conn(0x7f3ff40377d0 0x7f3ff4039c90 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f400800dfd0 tx=0x7f400800db60 comp rx=0 tx=0).stop 2024-04-22T08:21:04.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.847+0000 7f4018070640 10 monclient: shutdown 2024-04-22T08:21:04.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.847+0000 7f4018070640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:04.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.847+0000 7f4018070640 1 -- 172.21.15.40:0/673637537 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4010132920 msgr2=0x7f401011ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:04.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.847+0000 7f4018070640 1 --2- 172.21.15.40:0/673637537 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4010132920 0x7f401011ab70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f400000b2c0 tx=0x7f40000046c0 comp rx=0 tx=0).stop 2024-04-22T08:21:04.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.847+0000 7f401660d640 1 -- 172.21.15.40:0/673637537 reap_dead start 2024-04-22T08:21:04.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.847+0000 7f401660d640 1 -- 172.21.15.40:0/673637537 reap_dead start 2024-04-22T08:21:04.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.847+0000 7f4018070640 1 -- 172.21.15.40:0/673637537 shutdown_connections 2024-04-22T08:21:04.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.847+0000 7f4018070640 1 -- 172.21.15.40:0/673637537 >> 172.21.15.40:0/673637537 conn(0x7f40100a0ae0 msgr2=0x7f401012eff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:04.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.847+0000 7f4018070640 1 -- 172.21.15.40:0/673637537 shutdown_connections 2024-04-22T08:21:04.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.847+0000 7f4018070640 1 -- 172.21.15.40:0/673637537 wait complete. 2024-04-22T08:21:04.870 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:21:04.870 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-22T08:21:04.870 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-22T08:21:04.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.970+0000 7fca8d5d7640 10 monclient: get_monmap_and_config 2024-04-22T08:21:04.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.970+0000 7fca8d5d7640 10 monclient: build_initial_monmap 2024-04-22T08:21:04.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.970+0000 7fca8d5d7640 10 monclient: monmap: 2024-04-22T08:21:04.970 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:04.970 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:04.970 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:21:04.971688+0000 2024-04-22T08:21:04.971 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:21:04.971688+0000 2024-04-22T08:21:04.971 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:04.971 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:04.971 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:21:04.971 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:21:04.971 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:21:04.971 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:21:04.971 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:21:04.971 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:04.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.971+0000 7fca8d5d7640 1 Processor -- start 2024-04-22T08:21:04.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.971+0000 7fca8d5d7640 1 -- start start 2024-04-22T08:21:04.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.971+0000 7fca8d5d7640 10 monclient: init 2024-04-22T08:21:04.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.972+0000 7fca8d5d7640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:04.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.972+0000 7fca8d5d7640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-04-22T08:21:04.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.972+0000 7fca8d5d7640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fca88070e90 0x7fca88071270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:04.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.972+0000 7fca8d5d7640 10 monclient(hunting): picked mon.noname-a con 0x7fca88070e90 addr v2:172.21.15.40:3300/0 2024-04-22T08:21:04.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.972+0000 7fca8d5d7640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fca88135c10 0x7fca8809de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:04.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.972+0000 7fca8d5d7640 10 monclient(hunting): picked mon.noname-c con 0x7fca88135c10 addr v2:172.21.15.189:3300/0 2024-04-22T08:21:04.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.972+0000 7fca8d5d7640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca88136020 0x7fca8809e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:04.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.972+0000 7fca8d5d7640 10 monclient(hunting): picked mon.noname-b con 0x7fca88136020 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:04.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.972+0000 7fca8d5d7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:04.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.972+0000 7fca8d5d7640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca88136940 con 0x7fca88070e90 2024-04-22T08:21:04.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.972+0000 7fca8d5d7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:04.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.972+0000 7fca8d5d7640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca8806d170 con 0x7fca88135c10 2024-04-22T08:21:04.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.972+0000 7fca8d5d7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:04.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.972+0000 7fca8d5d7640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca880717b0 con 0x7fca88136020 2024-04-22T08:21:04.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.972+0000 7fca8d5d7640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:04.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.972+0000 7fca8d5d7640 10 monclient(hunting): authenticate will time out at 1338.965949s 2024-04-22T08:21:04.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.972+0000 7fca86ffd640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fca88070e90 0x7fca88071270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:04.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.972+0000 7fca86ffd640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fca88070e90 0x7fca88071270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46010/0 (socket says 172.21.15.40:46010) 2024-04-22T08:21:04.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.972+0000 7fca877fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca88136020 0x7fca8809e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:04.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.972+0000 7fca86ffd640 1 -- 172.21.15.40:0/2707207269 learned_addr learned my addr 172.21.15.40:0/2707207269 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:21:04.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.972+0000 7fca86ffd640 10 monclient(hunting): get_auth_request con 0x7fca88070e90 auth_method 0 2024-04-22T08:21:04.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.972+0000 7fca86ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:04.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.972+0000 7fca86ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:04.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.972+0000 7fca86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:04.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.972+0000 7fca877fe640 10 monclient(hunting): get_auth_request con 0x7fca88136020 auth_method 0 2024-04-22T08:21:04.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.972+0000 7fca877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:04.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.972+0000 7fca877fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:04.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.972+0000 7fca877fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:04.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.972+0000 7fca86ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:04.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.972+0000 7fca86ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:04.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.972+0000 7fca86ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:04.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.972+0000 7fca877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:04.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.972+0000 7fca877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:04.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.972+0000 7fca877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:04.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.972+0000 7fca877fe640 10 monclient(hunting): handle_auth_done global_id 104138 payload 274 2024-04-22T08:21:04.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.972+0000 7fca867fc640 1 --2- 172.21.15.40:0/2707207269 >> v2:172.21.15.189:3300/0 conn(0x7fca88135c10 0x7fca8809de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:04.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.972+0000 7fca877fe640 1 -- 172.21.15.40:0/2707207269 >> v2:172.21.15.189:3300/0 conn(0x7fca88135c10 msgr2=0x7fca8809de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:21:04.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.972+0000 7fca877fe640 1 --2- 172.21.15.40:0/2707207269 >> v2:172.21.15.189:3300/0 conn(0x7fca88135c10 0x7fca8809de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:04.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.972+0000 7fca877fe640 1 -- 172.21.15.40:0/2707207269 >> v2:172.21.15.40:3300/0 conn(0x7fca88070e90 msgr2=0x7fca88071270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:21:04.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.972+0000 7fca877fe640 1 --2- 172.21.15.40:0/2707207269 >> v2:172.21.15.40:3300/0 conn(0x7fca88070e90 0x7fca88071270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:04.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.972+0000 7fca877fe640 10 monclient: _finish_hunting 0 2024-04-22T08:21:04.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.972+0000 7fca877fe640 1 monclient: found mon.noname-b 2024-04-22T08:21:04.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.972+0000 7fca877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:04.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.972+0000 7fca877fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:21:04.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.972+0000 7fca877fe640 1 -- 172.21.15.40:0/2707207269 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca8806d450 con 0x7fca88136020 2024-04-22T08:21:04.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.972+0000 7fca86ffd640 1 --2- 172.21.15.40:0/2707207269 >> v2:172.21.15.40:3300/0 conn(0x7fca88070e90 0x7fca88071270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:21:04.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.975+0000 7fca877fe640 1 -- 172.21.15.40:0/2707207269 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca88136020 msgr2=0x7fca8809e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:04.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.975+0000 7fca877fe640 1 --2- 172.21.15.40:0/2707207269 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca88136020 0x7fca8809e340 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fca880a2d60 tx=0x7fca8002bd00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38d7d55b60923c8e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:04.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.975+0000 7fca85ffb640 1 -- 172.21.15.40:0/2707207269 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca8002ccc0 con 0x7fca88136020 2024-04-22T08:21:04.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.975+0000 7fca85ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:04.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.975+0000 7fca85ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:21:04.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.975+0000 7fca85ffb640 10 monclient: dump: 2024-04-22T08:21:04.989 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:04.989 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:04.989 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:04.989 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:04.989 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:04.989 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:04.989 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:04.990 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:04.990 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:04.990 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:04.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.975+0000 7fca85ffb640 10 monclient: _finish_auth 0 2024-04-22T08:21:04.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.975+0000 7fca85ffb640 10 monclient: _check_auth_tickets 2024-04-22T08:21:04.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.975+0000 7fca85ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:04.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.975+0000 7fca8d5d7640 5 monclient: authenticate success, global_id 104138 2024-04-22T08:21:04.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.975+0000 7fca8d5d7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:21:04.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.975+0000 7fca85ffb640 1 -- 172.21.15.40:0/2707207269 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca8002ce20 con 0x7fca88136020 2024-04-22T08:21:04.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.975+0000 7fca85ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:04.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.975+0000 7fca8d5d7640 10 monclient: get_monmap_and_config success 2024-04-22T08:21:04.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.975+0000 7fca8d5d7640 4 set_mon_vals no callback set 2024-04-22T08:21:04.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.975+0000 7fca85ffb640 1 -- 172.21.15.40:0/2707207269 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca80034660 con 0x7fca88136020 2024-04-22T08:21:04.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.975+0000 7fca85ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:04.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.975+0000 7fca85ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:21:04.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.975+0000 7fca85ffb640 10 monclient: dump: 2024-04-22T08:21:04.991 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:04.991 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:04.991 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:04.991 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:04.991 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:04.991 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:04.991 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:04.992 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:04.992 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:04.992 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:04.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.976+0000 7fca8d5d7640 10 monclient: shutdown 2024-04-22T08:21:04.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.976+0000 7fca8d5d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:04.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.976+0000 7fca8d5d7640 1 -- 172.21.15.40:0/2707207269 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca88136020 msgr2=0x7fca8809e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:04.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.976+0000 7fca8d5d7640 1 --2- 172.21.15.40:0/2707207269 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca88136020 0x7fca8809e340 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fca880a2d60 tx=0x7fca8002bd00 comp rx=0 tx=0).stop 2024-04-22T08:21:04.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.976+0000 7fca8d5d7640 1 -- 172.21.15.40:0/2707207269 shutdown_connections 2024-04-22T08:21:04.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.976+0000 7fca8d5d7640 1 --2- 172.21.15.40:0/2707207269 >> v2:172.21.15.189:3300/0 conn(0x7fca88135c10 0x7fca8809de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:04.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.976+0000 7fca8d5d7640 1 --2- 172.21.15.40:0/2707207269 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca88136020 0x7fca8809e340 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:04.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.976+0000 7fca8d5d7640 1 --2- 172.21.15.40:0/2707207269 >> v2:172.21.15.40:3300/0 conn(0x7fca88070e90 0x7fca88071270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:04.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.976+0000 7fca8d5d7640 1 -- 172.21.15.40:0/2707207269 >> 172.21.15.40:0/2707207269 conn(0x7fca880a1780 msgr2=0x7fca880a1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:04.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.976+0000 7fca8d5d7640 1 -- 172.21.15.40:0/2707207269 shutdown_connections 2024-04-22T08:21:04.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.976+0000 7fca8d5d7640 1 -- 172.21.15.40:0/2707207269 wait complete. 2024-04-22T08:21:04.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.976+0000 7fca8d5d7640 10 monclient: build_initial_monmap 2024-04-22T08:21:04.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.976+0000 7fca8d5d7640 10 monclient: monmap: 2024-04-22T08:21:04.993 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:04.993 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:21:04.993 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:21:04.993 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:21:04.993 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:04.993 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:04.993 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:21:04.993 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:21:04.993 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:21:04.993 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:04.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.977+0000 7fca8d5d7640 1 Processor -- start 2024-04-22T08:21:04.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.977+0000 7fca8d5d7640 1 -- start start 2024-04-22T08:21:04.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.977+0000 7fca8d5d7640 10 monclient: init 2024-04-22T08:21:04.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.977+0000 7fca8d5d7640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:04.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.977+0000 7fca8d5d7640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:21:04.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.977+0000 7fca8d5d7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca88070e90 0x7fca881cab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:04.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.977+0000 7fca8d5d7640 10 monclient(hunting): picked mon.noname-a con 0x7fca88070e90 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:21:04.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.977+0000 7fca8d5d7640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca88135c10 0x7fca881cb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:04.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.977+0000 7fca8d5d7640 10 monclient(hunting): picked mon.noname-b con 0x7fca88135c10 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:21:04.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.977+0000 7fca8d5d7640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca88136020 0x7fca8811b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:04.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.977+0000 7fca8d5d7640 10 monclient(hunting): picked mon.noname-c con 0x7fca88136020 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:04.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.977+0000 7fca8d5d7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:04.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.977+0000 7fca8d5d7640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca880a40d0 con 0x7fca88070e90 2024-04-22T08:21:04.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.977+0000 7fca8d5d7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:04.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.977+0000 7fca8d5d7640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca8811d550 con 0x7fca88135c10 2024-04-22T08:21:04.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.977+0000 7fca8d5d7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:04.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.977+0000 7fca8d5d7640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca88134da0 con 0x7fca88136020 2024-04-22T08:21:04.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.977+0000 7fca8d5d7640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:04.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.977+0000 7fca8d5d7640 10 monclient(hunting): authenticate will time out at 1338.971644s 2024-04-22T08:21:04.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.977+0000 7fca86ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca88070e90 0x7fca881cab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:04.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.977+0000 7fca877fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca88136020 0x7fca8811b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:04.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.977+0000 7fca86ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca88070e90 0x7fca881cab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46014/0 (socket says 172.21.15.40:46014) 2024-04-22T08:21:04.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.978+0000 7fca86ffd640 1 -- 172.21.15.40:0/3691652597 learned_addr learned my addr 172.21.15.40:0/3691652597 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:21:04.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.978+0000 7fca86ffd640 10 monclient(hunting): get_auth_request con 0x7fca88070e90 auth_method 0 2024-04-22T08:21:04.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.978+0000 7fca86ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:04.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.978+0000 7fca86ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:04.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.978+0000 7fca86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:04.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.978+0000 7fca877fe640 10 monclient(hunting): get_auth_request con 0x7fca88136020 auth_method 0 2024-04-22T08:21:04.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.978+0000 7fca877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:04.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.978+0000 7fca877fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:04.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.978+0000 7fca877fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:04.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.978+0000 7fca86ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:04.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.978+0000 7fca86ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:04.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.978+0000 7fca86ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:04.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.978+0000 7fca877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:04.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.978+0000 7fca877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:04.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.978+0000 7fca877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:04.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.978+0000 7fca867fc640 1 --2- 172.21.15.40:0/3691652597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca88135c10 0x7fca881cb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:04.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.978+0000 7fca86ffd640 10 monclient(hunting): handle_auth_done global_id 94407 payload 824 2024-04-22T08:21:04.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.978+0000 7fca86ffd640 1 -- 172.21.15.40:0/3691652597 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca88136020 msgr2=0x7fca8811b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:04.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.978+0000 7fca86ffd640 1 --2- 172.21.15.40:0/3691652597 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca88136020 0x7fca8811b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:04.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.978+0000 7fca86ffd640 1 -- 172.21.15.40:0/3691652597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca88135c10 msgr2=0x7fca881cb0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:04.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.978+0000 7fca86ffd640 1 --2- 172.21.15.40:0/3691652597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca88135c10 0x7fca881cb0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:04.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.978+0000 7fca86ffd640 10 monclient: _finish_hunting 0 2024-04-22T08:21:04.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.978+0000 7fca86ffd640 1 monclient: found mon.noname-a 2024-04-22T08:21:04.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.978+0000 7fca86ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:04.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.978+0000 7fca86ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:21:04.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.978+0000 7fca86ffd640 1 -- 172.21.15.40:0/3691652597 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca780094d0 con 0x7fca88070e90 2024-04-22T08:21:04.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.978+0000 7fca877fe640 1 --2- 172.21.15.40:0/3691652597 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca88136020 0x7fca8811b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:21:04.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.980+0000 7fca86ffd640 1 -- 172.21.15.40:0/3691652597 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca88070e90 msgr2=0x7fca881cab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:04.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.980+0000 7fca86ffd640 1 --2- 172.21.15.40:0/3691652597 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca88070e90 0x7fca881cab90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fca7800ea00 tx=0x7fca7800eec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:04.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.980+0000 7fca6f7fe640 1 -- 172.21.15.40:0/3691652597 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca7800cc70 con 0x7fca88070e90 2024-04-22T08:21:04.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.980+0000 7fca6f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:04.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.980+0000 7fca6f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:21:04.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.980+0000 7fca6f7fe640 10 monclient: dump: 2024-04-22T08:21:04.998 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:04.998 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:04.998 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:04.998 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:04.998 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:04.998 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:04.998 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:04.998 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:04.998 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:04.998 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:04.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.980+0000 7fca6f7fe640 10 monclient: _finish_auth 0 2024-04-22T08:21:04.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.980+0000 7fca6f7fe640 10 monclient: _check_auth_tickets 2024-04-22T08:21:04.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.980+0000 7fca6f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:04.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.980+0000 7fca6f7fe640 1 -- 172.21.15.40:0/3691652597 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca7800cdd0 con 0x7fca88070e90 2024-04-22T08:21:04.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.980+0000 7fca8d5d7640 5 monclient: authenticate success, global_id 94407 2024-04-22T08:21:04.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.980+0000 7fca8d5d7640 10 monclient: _renew_subs 2024-04-22T08:21:04.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.980+0000 7fca8d5d7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:21:04.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.980+0000 7fca8d5d7640 1 -- 172.21.15.40:0/3691652597 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca80009a70 con 0x7fca88070e90 2024-04-22T08:21:04.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.980+0000 7fca6f7fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:04.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.980+0000 7fca6f7fe640 1 -- 172.21.15.40:0/3691652597 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca78013670 con 0x7fca88070e90 2024-04-22T08:21:04.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.980+0000 7fca6f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:04.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.980+0000 7fca6f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:21:04.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.980+0000 7fca6f7fe640 10 monclient: dump: 2024-04-22T08:21:04.999 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:05.000 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:05.000 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:05.000 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:05.000 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:05.000 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:05.000 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:05.000 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:05.000 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:05.000 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:05.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.980+0000 7fca857fa640 4 set_mon_vals no callback set 2024-04-22T08:21:05.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.980+0000 7fca8d5d7640 10 monclient: _renew_subs 2024-04-22T08:21:05.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.980+0000 7fca8d5d7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:21:05.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.980+0000 7fca8d5d7640 1 -- 172.21.15.40:0/3691652597 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca8811e020 con 0x7fca88070e90 2024-04-22T08:21:05.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.981+0000 7fca8d5d7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:05.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.981+0000 7fca8d5d7640 1 --2- 172.21.15.40:0/3691652597 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca8811bea0 0x7fca8811c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:05.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.981+0000 7fca8d5d7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:05.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.981+0000 7fca6f7fe640 1 -- 172.21.15.40:0/3691652597 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fca780137d0 con 0x7fca88070e90 2024-04-22T08:21:05.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.981+0000 7fca8d5d7640 1 -- 172.21.15.40:0/3691652597 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca88066900 con 0x7fca8811bea0 2024-04-22T08:21:05.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.981+0000 7fca867fc640 1 --2- 172.21.15.40:0/3691652597 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca8811bea0 0x7fca8811c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:05.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.981+0000 7fca867fc640 10 monclient: get_auth_request con 0x7fca8811bea0 auth_method 0 2024-04-22T08:21:05.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.981+0000 7fca867fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:05.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.981+0000 7fca867fc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:05.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.981+0000 7fca867fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:05.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.981+0000 7fca867fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:05.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.981+0000 7fca867fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:05.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.981+0000 7fca867fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:05.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.981+0000 7fca6f7fe640 1 --2- 172.21.15.40:0/3691652597 >> v2:172.21.15.189:6800/30168 conn(0x7fca740377d0 0x7fca74039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:05.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.981+0000 7fca6f7fe640 1 -- 172.21.15.40:0/3691652597 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(276..276 src has 239..276) v4 ==== 3506+0+0 (secure 0 0 0) 0x7fca7804a650 con 0x7fca88070e90 2024-04-22T08:21:05.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.981+0000 7fca867fc640 10 monclient(hunting): handle_auth_done global_id 104144 payload 824 2024-04-22T08:21:05.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.981+0000 7fca867fc640 1 -- 172.21.15.40:0/3691652597 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fca880a3e20 con 0x7fca8811bea0 2024-04-22T08:21:05.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.982+0000 7fca877fe640 1 --2- 172.21.15.40:0/3691652597 >> v2:172.21.15.189:6800/30168 conn(0x7fca740377d0 0x7fca74039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:05.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.982+0000 7fca877fe640 10 monclient: get_auth_request con 0x7fca740377d0 auth_method 0 2024-04-22T08:21:05.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.984+0000 7fca867fc640 1 -- 172.21.15.40:0/3691652597 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca8811bea0 msgr2=0x7fca8811c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:05.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.984+0000 7fca867fc640 1 --2- 172.21.15.40:0/3691652597 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca8811bea0 0x7fca8811c280 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fca8809e850 tx=0x7fca7c00d950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:05.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.984+0000 7fca6f7fe640 1 -- 172.21.15.40:0/3691652597 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca7c012b60 con 0x7fca8811bea0 2024-04-22T08:21:05.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.984+0000 7fca6f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:05.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.984+0000 7fca6f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:21:05.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.984+0000 7fca6f7fe640 10 monclient: dump: 2024-04-22T08:21:05.002 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:05.002 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:05.002 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:05.003 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:05.003 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:05.003 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:05.003 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:05.003 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:05.003 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:05.003 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:05.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.984+0000 7fca6f7fe640 1 -- 172.21.15.40:0/3691652597 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7fca7c00c040 con 0x7fca8811bea0 2024-04-22T08:21:05.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.984+0000 7fca6f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:05.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.984+0000 7fca6f7fe640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:21:05.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.984+0000 7fca6f7fe640 1 -- 172.21.15.40:0/3691652597 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca8811bea0 msgr2=0x7fca8811c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:05.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.984+0000 7fca6f7fe640 1 --2- 172.21.15.40:0/3691652597 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca8811bea0 0x7fca8811c280 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fca8809e850 tx=0x7fca7c00d950 comp rx=0 tx=0).stop 2024-04-22T08:21:05.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.984+0000 7fca6f7fe640 1 -- 172.21.15.40:0/3691652597 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca8811bea0 msgr2=0x7fca8811c280 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:21:05.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.984+0000 7fca6f7fe640 1 --2- 172.21.15.40:0/3691652597 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca8811bea0 0x7fca8811c280 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:05.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.984+0000 7fca877fe640 1 --2- 172.21.15.40:0/3691652597 >> v2:172.21.15.189:6800/30168 conn(0x7fca740377d0 0x7fca74039c90 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fca8809eec0 tx=0x7fca80030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:05.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.990+0000 7fca8d5d7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:21:05.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.990+0000 7fca8d5d7640 1 --2- 172.21.15.40:0/3691652597 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca8811c7c0 0x7fca881303a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:05.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.990+0000 7fca8d5d7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:05.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.990+0000 7fca8d5d7640 1 -- 172.21.15.40:0/3691652597 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca88069310 con 0x7fca8811c7c0 2024-04-22T08:21:05.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.990+0000 7fca867fc640 1 --2- 172.21.15.40:0/3691652597 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca8811c7c0 0x7fca881303a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:05.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.990+0000 7fca867fc640 10 monclient: get_auth_request con 0x7fca8811c7c0 auth_method 0 2024-04-22T08:21:05.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.990+0000 7fca867fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:05.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.990+0000 7fca867fc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:05.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.990+0000 7fca867fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:05.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.991+0000 7fca867fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:05.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.991+0000 7fca867fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:05.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.991+0000 7fca867fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:05.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.991+0000 7fca867fc640 10 monclient(hunting): handle_auth_done global_id 104147 payload 824 2024-04-22T08:21:05.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.991+0000 7fca867fc640 1 -- 172.21.15.40:0/3691652597 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fca8804e9b0 con 0x7fca8811c7c0 2024-04-22T08:21:05.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.993+0000 7fca867fc640 1 -- 172.21.15.40:0/3691652597 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca8811c7c0 msgr2=0x7fca881303a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:05.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.993+0000 7fca867fc640 1 --2- 172.21.15.40:0/3691652597 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca8811c7c0 0x7fca881303a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fca8809e850 tx=0x7fca7c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:05.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.993+0000 7fca6f7fe640 1 -- 172.21.15.40:0/3691652597 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca7c020040 con 0x7fca8811c7c0 2024-04-22T08:21:05.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.993+0000 7fca6f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:05.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.993+0000 7fca6f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:21:05.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.993+0000 7fca6f7fe640 10 monclient: dump: 2024-04-22T08:21:05.007 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:05.007 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:05.007 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:05.007 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:05.007 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:05.007 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:05.007 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:05.007 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:05.008 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:05.008 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:05.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.993+0000 7fca6f7fe640 1 -- 172.21.15.40:0/3691652597 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4461 (secure 0 0 0) 0x7fca7c0117d0 con 0x7fca8811c7c0 2024-04-22T08:21:05.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.993+0000 7fca6f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:21:05.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.993+0000 7fca6f7fe640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:21:05.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.993+0000 7fca6f7fe640 1 -- 172.21.15.40:0/3691652597 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca8811c7c0 msgr2=0x7fca881303a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:05.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.993+0000 7fca6f7fe640 1 --2- 172.21.15.40:0/3691652597 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca8811c7c0 0x7fca881303a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fca8809e850 tx=0x7fca7c004730 comp rx=0 tx=0).stop 2024-04-22T08:21:05.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.993+0000 7fca6f7fe640 1 -- 172.21.15.40:0/3691652597 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca8811c7c0 msgr2=0x7fca881303a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:21:05.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.993+0000 7fca6f7fe640 1 --2- 172.21.15.40:0/3691652597 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca8811c7c0 0x7fca881303a0 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fca8809e850 tx=0x7fca7c004730 comp rx=0 tx=0).stop 2024-04-22T08:21:05.008 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:21:05.008 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:21:05.009 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:21:05.009 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:21:05.009 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 82, 2024-04-22T08:21:05.009 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:21:05.009 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:21:05.009 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:21:05.009 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:21:05.009 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:05.009 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 6, 2024-04-22T08:21:05.009 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:21:05.009 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:21:05.009 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:21:05.009 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:21:05.009 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:21:05.010 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:21:05.010 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:21:05.010 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:21:05.010 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:21:05.010 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:21:05.010 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:21:05.010 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:21:05.010 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:21:05.010 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:05.010 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:21:05.011 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:21:05.011 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:21:05.011 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:21:05.011 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:21:05.011 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:21:05.011 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:21:05.011 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:21:05.011 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:21:05.011 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:21:05.011 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:21:05.011 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:21:05.011 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:05.011 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:05.011 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:21:05.011 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:21:05.012 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:21:05.012 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:21:05.012 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:21:05.012 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:21:05.012 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:21:05.012 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:21:05.012 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:21:05.012 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:21:05.012 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:21:05.012 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:21:05.012 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:21:05.012 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:21:05.012 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:21:05.012 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:21:05.012 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:21:05.013 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:21:05.013 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:21:05.013 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:21:05.013 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:21:05.013 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:21:05.013 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:21:05.013 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:21:05.013 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:21:05.013 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:21:05.013 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:21:05.013 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:05.013 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:21:05.013 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:05.013 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:21:05.014 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:05.014 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:21:05.014 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:21:05.014 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:21:05.014 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:21:05.014 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:05.014 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:21:05.014 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:21:05.014 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:05.014 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:05.014 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:05.014 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:21:05.014 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:21:05.015 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:05.015 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:05.015 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:05.015 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:05.015 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:21:05.015 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:21:05.015 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:21:05.015 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:21:05.015 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:21:05.015 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:05.015 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:05.015 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:21:05.015 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:21:05.015 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:21:05.015 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:21:05.016 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:05.016 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:21:05.016 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:21:05.016 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:05.016 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:05.016 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:05.016 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:21:05.016 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:21:05.016 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:05.016 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:05.016 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:05.016 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:05.016 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:21:05.016 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:21:05.016 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:21:05.017 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:21:05.017 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:21:05.017 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:05.017 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:05.017 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:21:05.017 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:21:05.017 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:21:05.017 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:21:05.017 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:05.017 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:21:05.017 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:21:05.017 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:05.017 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:05.017 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:05.017 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:21:05.018 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:21:05.018 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:05.018 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:05.018 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:05.018 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:05.018 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:21:05.018 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:21:05.018 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:21:05.018 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:21:05.018 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:21:05.018 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:05.018 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:05.018 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:05.018 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:21:05.018 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:21:05.019 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:05.019 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:21:05.019 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:21:05.019 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:21:05.019 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:05.019 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:05.019 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:21:05.019 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:05.019 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:21:05.019 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:21:05.019 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:21:05.019 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:05.019 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:05.019 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:05.019 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:21:05.020 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:21:05.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.994+0000 7fca8d5d7640 1 -- 172.21.15.40:0/3691652597 >> v2:172.21.15.189:6800/30168 conn(0x7fca740377d0 msgr2=0x7fca74039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:05.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.994+0000 7fca8d5d7640 1 --2- 172.21.15.40:0/3691652597 >> v2:172.21.15.189:6800/30168 conn(0x7fca740377d0 0x7fca74039c90 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fca8809eec0 tx=0x7fca80030000 comp rx=0 tx=0).stop 2024-04-22T08:21:05.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.994+0000 7fca8d5d7640 10 monclient: shutdown 2024-04-22T08:21:05.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.994+0000 7fca8d5d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:05.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.994+0000 7fca8d5d7640 1 -- 172.21.15.40:0/3691652597 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca88070e90 msgr2=0x7fca881cab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:05.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.994+0000 7fca8d5d7640 1 --2- 172.21.15.40:0/3691652597 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca88070e90 0x7fca881cab90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fca7800ea00 tx=0x7fca7800eec0 comp rx=0 tx=0).stop 2024-04-22T08:21:05.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.994+0000 7fca877fe640 1 -- 172.21.15.40:0/3691652597 reap_dead start 2024-04-22T08:21:05.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.994+0000 7fca8d5d7640 1 -- 172.21.15.40:0/3691652597 shutdown_connections 2024-04-22T08:21:05.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.994+0000 7fca8d5d7640 1 --2- 172.21.15.40:0/3691652597 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca88070e90 0x7fca881cab90 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:05.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.994+0000 7fca8d5d7640 1 -- 172.21.15.40:0/3691652597 >> 172.21.15.40:0/3691652597 conn(0x7fca880a1780 msgr2=0x7fca8812f870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:05.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.994+0000 7fca8d5d7640 1 -- 172.21.15.40:0/3691652597 shutdown_connections 2024-04-22T08:21:05.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:04.994+0000 7fca8d5d7640 1 -- 172.21.15.40:0/3691652597 wait complete. 2024-04-22T08:21:05.022 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:21:05.022 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-22T08:21:05.022 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-22T08:21:05.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.122+0000 7f2f1efa5640 10 monclient: get_monmap_and_config 2024-04-22T08:21:05.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.122+0000 7f2f1efa5640 10 monclient: build_initial_monmap 2024-04-22T08:21:05.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.122+0000 7f2f1efa5640 10 monclient: monmap: 2024-04-22T08:21:05.122 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:05.122 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:05.122 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:21:05.123653+0000 2024-04-22T08:21:05.122 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:21:05.123653+0000 2024-04-22T08:21:05.122 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:05.123 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:05.123 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:21:05.123 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:21:05.123 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:21:05.123 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:21:05.123 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:21:05.123 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:05.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.123+0000 7f2f1efa5640 1 Processor -- start 2024-04-22T08:21:05.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.123+0000 7f2f1efa5640 1 -- start start 2024-04-22T08:21:05.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.123+0000 7f2f1efa5640 10 monclient: init 2024-04-22T08:21:05.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.124+0000 7f2f1efa5640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:05.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.124+0000 7f2f1efa5640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-04-22T08:21:05.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.124+0000 7f2f1efa5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2f180a1a80 addr v1:172.21.15.189:6789/0 2024-04-22T08:21:05.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.124+0000 7f2f1efa5640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f180a2570 0x7f2f180a01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:05.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.124+0000 7f2f1efa5640 10 monclient(hunting): picked mon.noname-b con 0x7f2f180a2570 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:05.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.124+0000 7f2f1efa5640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2f1813b070 0x7f2f1813d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:05.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.124+0000 7f2f1efa5640 10 monclient(hunting): picked mon.noname-c con 0x7f2f1813b070 addr v2:172.21.15.189:3300/0 2024-04-22T08:21:05.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.124+0000 7f2f1efa5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f180721e0 con 0x7f2f180a1a80 2024-04-22T08:21:05.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.124+0000 7f2f1efa5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:05.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.124+0000 7f2f1efa5640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f180a2950 con 0x7f2f1813b070 2024-04-22T08:21:05.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.124+0000 7f2f1efa5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:05.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.124+0000 7f2f1efa5640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f180999f0 con 0x7f2f180a2570 2024-04-22T08:21:05.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.124+0000 7f2f1efa5640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:05.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.124+0000 7f2f1efa5640 10 monclient(hunting): authenticate will time out at 1339.118012s 2024-04-22T08:21:05.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.124+0000 7f2f17fff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f180a2570 0x7f2f180a01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:05.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.124+0000 7f2f17fff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f180a2570 0x7f2f180a01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:51888/0 (socket says 172.21.15.40:51888) 2024-04-22T08:21:05.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.124+0000 7f2f17fff640 1 -- 172.21.15.40:0/1909519453 learned_addr learned my addr 172.21.15.40:0/1909519453 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:21:05.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.124+0000 7f2f17fff640 10 monclient(hunting): get_auth_request con 0x7f2f180a2570 auth_method 0 2024-04-22T08:21:05.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.124+0000 7f2f17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:05.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.124+0000 7f2f17fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:05.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.124+0000 7f2f17fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:05.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.124+0000 7f2f17fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:05.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.124+0000 7f2f17fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:05.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.124+0000 7f2f17fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:05.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.124+0000 7f2f1d542640 1 --2- 172.21.15.40:0/1909519453 >> v2:172.21.15.189:3300/0 conn(0x7f2f1813b070 0x7f2f1813d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:05.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.124+0000 7f2f1d542640 10 monclient(hunting): get_auth_request con 0x7f2f1813b070 auth_method 0 2024-04-22T08:21:05.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.124+0000 7f2f1d542640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:05.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.124+0000 7f2f1d542640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:05.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.124+0000 7f2f1d542640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:05.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.124+0000 7f2f17fff640 10 monclient(hunting): handle_auth_done global_id 104150 payload 274 2024-04-22T08:21:05.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.124+0000 7f2f17fff640 1 -- 172.21.15.40:0/1909519453 >> v2:172.21.15.189:3300/0 conn(0x7f2f1813b070 msgr2=0x7f2f1813d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:21:05.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.124+0000 7f2f17fff640 1 --2- 172.21.15.40:0/1909519453 >> v2:172.21.15.189:3300/0 conn(0x7f2f1813b070 0x7f2f1813d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:05.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.124+0000 7f2f17fff640 1 -- 172.21.15.40:0/1909519453 >> v1:172.21.15.189:6789/0 conn(0x7f2f180a1a80 legacy=0x7f2f180a1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:21:05.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.124+0000 7f2f17fff640 10 monclient: _finish_hunting 0 2024-04-22T08:21:05.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.124+0000 7f2f17fff640 1 monclient: found mon.noname-b 2024-04-22T08:21:05.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.124+0000 7f2f17fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:05.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.124+0000 7f2f17fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:21:05.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.124+0000 7f2f17fff640 1 -- 172.21.15.40:0/1909519453 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f1809f260 con 0x7f2f180a2570 2024-04-22T08:21:05.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.127+0000 7f2f17fff640 1 -- 172.21.15.40:0/1909519453 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f180a2570 msgr2=0x7f2f180a01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:05.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.127+0000 7f2f17fff640 1 --2- 172.21.15.40:0/1909519453 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f180a2570 0x7f2f180a01d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c009870 tx=0x7f2f0c02b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=247a4fa0a21ace9c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:05.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.127+0000 7f2f177fe640 1 -- 172.21.15.40:0/1909519453 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f0c0319d0 con 0x7f2f180a2570 2024-04-22T08:21:05.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.127+0000 7f2f177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:05.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.127+0000 7f2f177fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:21:05.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.127+0000 7f2f177fe640 10 monclient: dump: 2024-04-22T08:21:05.135 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:05.135 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:05.135 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:05.136 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:05.136 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:05.136 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:05.136 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:05.136 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:05.136 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:05.136 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:05.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.127+0000 7f2f177fe640 10 monclient: _finish_auth 0 2024-04-22T08:21:05.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.127+0000 7f2f177fe640 10 monclient: _check_auth_tickets 2024-04-22T08:21:05.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.127+0000 7f2f177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:05.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.127+0000 7f2f177fe640 1 -- 172.21.15.40:0/1909519453 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f0c031b30 con 0x7f2f180a2570 2024-04-22T08:21:05.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.127+0000 7f2f1efa5640 5 monclient: authenticate success, global_id 104150 2024-04-22T08:21:05.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.127+0000 7f2f1efa5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:21:05.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.127+0000 7f2f177fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:05.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.127+0000 7f2f1efa5640 10 monclient: get_monmap_and_config success 2024-04-22T08:21:05.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.127+0000 7f2f177fe640 1 -- 172.21.15.40:0/1909519453 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f0c032450 con 0x7f2f180a2570 2024-04-22T08:21:05.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.127+0000 7f2f1efa5640 4 set_mon_vals no callback set 2024-04-22T08:21:05.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.127+0000 7f2f177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:05.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.127+0000 7f2f177fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:21:05.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.127+0000 7f2f177fe640 10 monclient: dump: 2024-04-22T08:21:05.137 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:05.137 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:05.137 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:05.137 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:05.137 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:05.137 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:05.137 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:05.137 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:05.138 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:05.138 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:05.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.128+0000 7f2f1efa5640 10 monclient: shutdown 2024-04-22T08:21:05.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.128+0000 7f2f1efa5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:05.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.128+0000 7f2f1efa5640 1 -- 172.21.15.40:0/1909519453 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f180a2570 msgr2=0x7f2f180a01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:05.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.128+0000 7f2f1efa5640 1 --2- 172.21.15.40:0/1909519453 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f180a2570 0x7f2f180a01d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c009870 tx=0x7f2f0c02b860 comp rx=0 tx=0).stop 2024-04-22T08:21:05.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.128+0000 7f2f1efa5640 1 -- 172.21.15.40:0/1909519453 shutdown_connections 2024-04-22T08:21:05.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.128+0000 7f2f1efa5640 1 --2- 172.21.15.40:0/1909519453 >> v2:172.21.15.189:3300/0 conn(0x7f2f1813b070 0x7f2f1813d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:05.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.128+0000 7f2f1efa5640 1 --2- 172.21.15.40:0/1909519453 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f180a2570 0x7f2f180a01d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:05.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.128+0000 7f2f1efa5640 1 -- 172.21.15.40:0/1909519453 >> 172.21.15.40:0/1909519453 conn(0x7f2f180a66a0 msgr2=0x7f2f180a2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:05.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.128+0000 7f2f1efa5640 1 -- 172.21.15.40:0/1909519453 shutdown_connections 2024-04-22T08:21:05.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.128+0000 7f2f1efa5640 1 -- 172.21.15.40:0/1909519453 wait complete. 2024-04-22T08:21:05.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.128+0000 7f2f1efa5640 10 monclient: build_initial_monmap 2024-04-22T08:21:05.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.128+0000 7f2f1efa5640 10 monclient: monmap: 2024-04-22T08:21:05.139 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:05.139 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:21:05.139 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:21:05.139 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:21:05.139 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:05.139 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:05.139 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:21:05.139 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:21:05.139 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:21:05.139 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:05.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.129+0000 7f2f1efa5640 1 Processor -- start 2024-04-22T08:21:05.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.129+0000 7f2f1efa5640 1 -- start start 2024-04-22T08:21:05.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.129+0000 7f2f1efa5640 10 monclient: init 2024-04-22T08:21:05.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.129+0000 7f2f1efa5640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:05.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.129+0000 7f2f1efa5640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:21:05.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.129+0000 7f2f1efa5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f180a1a80 0x7f2f181ca3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:05.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.129+0000 7f2f1efa5640 10 monclient(hunting): picked mon.noname-a con 0x7f2f180a1a80 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:21:05.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.129+0000 7f2f1efa5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f180a2570 0x7f2f18135c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:05.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.129+0000 7f2f1efa5640 10 monclient(hunting): picked mon.noname-b con 0x7f2f180a2570 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:21:05.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.129+0000 7f2f1efa5640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f1813b070 0x7f2f181361c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:05.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.129+0000 7f2f1efa5640 10 monclient(hunting): picked mon.noname-c con 0x7f2f1813b070 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:05.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.129+0000 7f2f1efa5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:05.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.129+0000 7f2f1efa5640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f181226c0 con 0x7f2f180a1a80 2024-04-22T08:21:05.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.129+0000 7f2f1efa5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:05.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.129+0000 7f2f1efa5640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1809cce0 con 0x7f2f180a2570 2024-04-22T08:21:05.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.129+0000 7f2f1efa5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:05.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.129+0000 7f2f1efa5640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f18138e00 con 0x7f2f1813b070 2024-04-22T08:21:05.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.129+0000 7f2f1efa5640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:05.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.129+0000 7f2f1efa5640 10 monclient(hunting): authenticate will time out at 1339.123587s 2024-04-22T08:21:05.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.129+0000 7f2f1d542640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f1813b070 0x7f2f181361c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:05.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.129+0000 7f2f1cd41640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f180a1a80 0x7f2f181ca3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:05.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.129+0000 7f2f1d542640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f1813b070 0x7f2f181361c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:51896/0 (socket says 172.21.15.40:51896) 2024-04-22T08:21:05.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.129+0000 7f2f1d542640 1 -- 172.21.15.40:0/2073815058 learned_addr learned my addr 172.21.15.40:0/2073815058 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:21:05.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.129+0000 7f2f1d542640 10 monclient(hunting): get_auth_request con 0x7f2f1813b070 auth_method 0 2024-04-22T08:21:05.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.129+0000 7f2f1d542640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:05.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.129+0000 7f2f1d542640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:05.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.129+0000 7f2f1d542640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:05.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.129+0000 7f2f1cd41640 10 monclient(hunting): get_auth_request con 0x7f2f180a1a80 auth_method 0 2024-04-22T08:21:05.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.129+0000 7f2f1cd41640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:05.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.129+0000 7f2f1cd41640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:05.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.129+0000 7f2f1cd41640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:05.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.130+0000 7f2f1d542640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:05.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.130+0000 7f2f1d542640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:05.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.130+0000 7f2f1d542640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:05.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.130+0000 7f2f1cd41640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:05.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.130+0000 7f2f1cd41640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:05.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.130+0000 7f2f17fff640 1 --2- 172.21.15.40:0/2073815058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f180a2570 0x7f2f18135c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:05.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.130+0000 7f2f1cd41640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:05.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.130+0000 7f2f17fff640 10 monclient(hunting): get_auth_request con 0x7f2f180a2570 auth_method 0 2024-04-22T08:21:05.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.130+0000 7f2f17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:05.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.130+0000 7f2f17fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:05.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.130+0000 7f2f17fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:05.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.130+0000 7f2f17fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:05.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.130+0000 7f2f17fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:05.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.130+0000 7f2f17fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:05.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.130+0000 7f2f1d542640 10 monclient(hunting): handle_auth_done global_id 104153 payload 824 2024-04-22T08:21:05.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.130+0000 7f2f1d542640 1 -- 172.21.15.40:0/2073815058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f180a2570 msgr2=0x7f2f18135c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:05.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.130+0000 7f2f1d542640 1 --2- 172.21.15.40:0/2073815058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f180a2570 0x7f2f18135c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:05.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.130+0000 7f2f1d542640 1 -- 172.21.15.40:0/2073815058 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f180a1a80 msgr2=0x7f2f181ca3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:05.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.130+0000 7f2f1d542640 1 --2- 172.21.15.40:0/2073815058 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f180a1a80 0x7f2f181ca3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:05.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.130+0000 7f2f1d542640 10 monclient: _finish_hunting 0 2024-04-22T08:21:05.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.130+0000 7f2f1d542640 1 monclient: found mon.noname-c 2024-04-22T08:21:05.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.130+0000 7f2f1d542640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:05.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.130+0000 7f2f1d542640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:21:05.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.130+0000 7f2f1d542640 1 -- 172.21.15.40:0/2073815058 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f0c0094d0 con 0x7f2f1813b070 2024-04-22T08:21:05.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.130+0000 7f2f1cd41640 1 --2- 172.21.15.40:0/2073815058 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f180a1a80 0x7f2f181ca3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:21:05.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.132+0000 7f2f1d542640 1 -- 172.21.15.40:0/2073815058 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f1813b070 msgr2=0x7f2f181361c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:05.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.132+0000 7f2f1d542640 1 --2- 172.21.15.40:0/2073815058 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f1813b070 0x7f2f181361c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2f0800d830 tx=0x7f2f0800dcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:05.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.132+0000 7f2f157fa640 1 -- 172.21.15.40:0/2073815058 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f0800b8d0 con 0x7f2f1813b070 2024-04-22T08:21:05.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.132+0000 7f2f157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:05.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.132+0000 7f2f157fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:21:05.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.132+0000 7f2f157fa640 10 monclient: dump: 2024-04-22T08:21:05.144 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:05.144 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:05.144 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:05.144 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:05.144 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:05.144 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:05.144 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:05.144 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:05.144 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:05.144 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:05.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.132+0000 7f2f157fa640 10 monclient: _finish_auth 0 2024-04-22T08:21:05.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.132+0000 7f2f157fa640 10 monclient: _check_auth_tickets 2024-04-22T08:21:05.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.132+0000 7f2f157fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:05.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.132+0000 7f2f157fa640 1 -- 172.21.15.40:0/2073815058 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f0800ba50 con 0x7f2f1813b070 2024-04-22T08:21:05.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.132+0000 7f2f1efa5640 5 monclient: authenticate success, global_id 104153 2024-04-22T08:21:05.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.132+0000 7f2f157fa640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:05.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.132+0000 7f2f1efa5640 10 monclient: _renew_subs 2024-04-22T08:21:05.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.132+0000 7f2f157fa640 1 -- 172.21.15.40:0/2073815058 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f0800be00 con 0x7f2f1813b070 2024-04-22T08:21:05.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.132+0000 7f2f1efa5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:21:05.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.132+0000 7f2f1efa5640 1 -- 172.21.15.40:0/2073815058 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f181ca920 con 0x7f2f1813b070 2024-04-22T08:21:05.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.132+0000 7f2f157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:05.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.132+0000 7f2f157fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:21:05.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.132+0000 7f2f157fa640 10 monclient: dump: 2024-04-22T08:21:05.145 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:05.145 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:05.145 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:05.145 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:05.145 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:05.146 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:05.146 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:05.146 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:05.146 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:05.146 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:05.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.132+0000 7f2f16ffd640 4 set_mon_vals no callback set 2024-04-22T08:21:05.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.132+0000 7f2f1efa5640 10 monclient: _renew_subs 2024-04-22T08:21:05.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.132+0000 7f2f1efa5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:21:05.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.132+0000 7f2f1efa5640 1 -- 172.21.15.40:0/2073815058 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f180714d0 con 0x7f2f1813b070 2024-04-22T08:21:05.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.133+0000 7f2f1efa5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:05.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.133+0000 7f2f1efa5640 1 --2- 172.21.15.40:0/2073815058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f1806f870 0x7f2f1806fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:05.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.133+0000 7f2f1efa5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:05.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.133+0000 7f2f1efa5640 1 -- 172.21.15.40:0/2073815058 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f18066770 con 0x7f2f1806f870 2024-04-22T08:21:05.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.133+0000 7f2f157fa640 1 -- 172.21.15.40:0/2073815058 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f2f0801e460 con 0x7f2f1813b070 2024-04-22T08:21:05.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.133+0000 7f2f1cd41640 1 --2- 172.21.15.40:0/2073815058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f1806f870 0x7f2f1806fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:05.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.133+0000 7f2f1cd41640 10 monclient: get_auth_request con 0x7f2f1806f870 auth_method 0 2024-04-22T08:21:05.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.133+0000 7f2f1cd41640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:05.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.133+0000 7f2f1cd41640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:05.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.133+0000 7f2f1cd41640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:05.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.133+0000 7f2f157fa640 1 --2- 172.21.15.40:0/2073815058 >> v2:172.21.15.189:6800/30168 conn(0x7f2f04037850 0x7f2f04039d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:05.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.133+0000 7f2f157fa640 1 -- 172.21.15.40:0/2073815058 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(276..276 src has 239..276) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f2f0804a730 con 0x7f2f1813b070 2024-04-22T08:21:05.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.134+0000 7f2f1cd41640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:05.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.134+0000 7f2f1cd41640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:05.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.134+0000 7f2f1cd41640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:05.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.134+0000 7f2f17fff640 1 --2- 172.21.15.40:0/2073815058 >> v2:172.21.15.189:6800/30168 conn(0x7f2f04037850 0x7f2f04039d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:05.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.134+0000 7f2f17fff640 10 monclient: get_auth_request con 0x7f2f04037850 auth_method 0 2024-04-22T08:21:05.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.134+0000 7f2f1cd41640 10 monclient(hunting): handle_auth_done global_id 64957 payload 824 2024-04-22T08:21:05.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.134+0000 7f2f1cd41640 1 -- 172.21.15.40:0/2073815058 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2f1809dbf0 con 0x7f2f1806f870 2024-04-22T08:21:05.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.136+0000 7f2f1cd41640 1 -- 172.21.15.40:0/2073815058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f1806f870 msgr2=0x7f2f1806fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:05.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.136+0000 7f2f1cd41640 1 --2- 172.21.15.40:0/2073815058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f1806f870 0x7f2f1806fc50 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f2f100048c0 tx=0x7f2f1000d520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:05.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.136+0000 7f2f17fff640 1 --2- 172.21.15.40:0/2073815058 >> v2:172.21.15.189:6800/30168 conn(0x7f2f04037850 0x7f2f04039d10 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f2f180a3c30 tx=0x7f2f0c03d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:05.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.136+0000 7f2f157fa640 1 -- 172.21.15.40:0/2073815058 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f10004950 con 0x7f2f1806f870 2024-04-22T08:21:05.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.136+0000 7f2f157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:05.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.136+0000 7f2f157fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:21:05.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.136+0000 7f2f157fa640 10 monclient: dump: 2024-04-22T08:21:05.150 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:05.150 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:05.150 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:05.151 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:05.151 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:05.151 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:05.151 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:05.151 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:05.151 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:05.151 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:05.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.137+0000 7f2f157fa640 1 -- 172.21.15.40:0/2073815058 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f2f10004ab0 con 0x7f2f1806f870 2024-04-22T08:21:05.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.137+0000 7f2f157fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:05.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.137+0000 7f2f157fa640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:21:05.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.137+0000 7f2f157fa640 1 -- 172.21.15.40:0/2073815058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f1806f870 msgr2=0x7f2f1806fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:05.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.137+0000 7f2f157fa640 1 --2- 172.21.15.40:0/2073815058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f1806f870 0x7f2f1806fc50 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f2f100048c0 tx=0x7f2f1000d520 comp rx=0 tx=0).stop 2024-04-22T08:21:05.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.137+0000 7f2f157fa640 1 -- 172.21.15.40:0/2073815058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f1806f870 msgr2=0x7f2f1806fc50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:21:05.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.137+0000 7f2f157fa640 1 --2- 172.21.15.40:0/2073815058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f1806f870 0x7f2f1806fc50 secure :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f2f100048c0 tx=0x7f2f1000d520 comp rx=0 tx=0).stop 2024-04-22T08:21:05.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.143+0000 7f2f1efa5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:21:05.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.143+0000 7f2f1efa5640 1 --2- 172.21.15.40:0/2073815058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f18070190 0x7f2f180a3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:05.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.143+0000 7f2f1efa5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:05.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.143+0000 7f2f1efa5640 1 -- 172.21.15.40:0/2073815058 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1804e9b0 con 0x7f2f18070190 2024-04-22T08:21:05.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.143+0000 7f2f1cd41640 1 --2- 172.21.15.40:0/2073815058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f18070190 0x7f2f180a3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:05.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.143+0000 7f2f1cd41640 10 monclient: get_auth_request con 0x7f2f18070190 auth_method 0 2024-04-22T08:21:05.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.143+0000 7f2f1cd41640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:05.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.143+0000 7f2f1cd41640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:05.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.143+0000 7f2f1cd41640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:05.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.143+0000 7f2f1cd41640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:05.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.143+0000 7f2f1cd41640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:05.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.143+0000 7f2f1cd41640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:05.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.143+0000 7f2f1cd41640 10 monclient(hunting): handle_auth_done global_id 64960 payload 824 2024-04-22T08:21:05.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.143+0000 7f2f1cd41640 1 -- 172.21.15.40:0/2073815058 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2f1806a1a0 con 0x7f2f18070190 2024-04-22T08:21:05.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.146+0000 7f2f1cd41640 1 -- 172.21.15.40:0/2073815058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f18070190 msgr2=0x7f2f180a3f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:05.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.146+0000 7f2f1cd41640 1 --2- 172.21.15.40:0/2073815058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f18070190 0x7f2f180a3f50 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f2f10018000 tx=0x7f2f10003ae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:05.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.146+0000 7f2f157fa640 1 -- 172.21.15.40:0/2073815058 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f10024070 con 0x7f2f18070190 2024-04-22T08:21:05.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.146+0000 7f2f157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:05.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.146+0000 7f2f157fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:21:05.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.146+0000 7f2f157fa640 10 monclient: dump: 2024-04-22T08:21:05.153 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:05.153 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:05.153 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:05.153 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:05.153 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:05.153 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:05.153 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:05.153 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:05.153 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:05.153 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:05.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.146+0000 7f2f157fa640 1 -- 172.21.15.40:0/2073815058 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4801 (secure 0 0 0) 0x7f2f10005240 con 0x7f2f18070190 2024-04-22T08:21:05.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.146+0000 7f2f157fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:21:05.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.146+0000 7f2f157fa640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:21:05.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.146+0000 7f2f157fa640 1 -- 172.21.15.40:0/2073815058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f18070190 msgr2=0x7f2f180a3f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:05.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.146+0000 7f2f157fa640 1 --2- 172.21.15.40:0/2073815058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f18070190 0x7f2f180a3f50 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f2f10018000 tx=0x7f2f10003ae0 comp rx=0 tx=0).stop 2024-04-22T08:21:05.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.146+0000 7f2f157fa640 1 -- 172.21.15.40:0/2073815058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f18070190 msgr2=0x7f2f180a3f50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:21:05.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.146+0000 7f2f157fa640 1 --2- 172.21.15.40:0/2073815058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f18070190 0x7f2f180a3f50 secure :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f2f10018000 tx=0x7f2f10003ae0 comp rx=0 tx=0).stop 2024-04-22T08:21:05.154 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:21:05.154 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:21:05.154 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:21:05.155 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:21:05.155 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 82, 2024-04-22T08:21:05.155 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:21:05.155 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:21:05.155 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:21:05.155 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:21:05.155 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:05.155 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 6, 2024-04-22T08:21:05.155 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:21:05.155 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:21:05.155 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:21:05.155 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:21:05.155 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:21:05.155 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:21:05.155 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:21:05.156 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:21:05.156 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:21:05.156 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:21:05.156 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:21:05.156 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:21:05.156 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:21:05.156 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:05.156 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:21:05.156 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:21:05.156 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:21:05.156 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:21:05.156 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:21:05.156 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:21:05.156 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:21:05.156 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:21:05.157 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:21:05.157 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:21:05.157 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:21:05.157 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:21:05.157 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:05.157 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:05.157 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:21:05.157 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:21:05.157 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:21:05.157 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:21:05.157 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:21:05.157 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:21:05.157 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:21:05.157 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:21:05.158 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:21:05.158 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:21:05.158 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:21:05.158 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:21:05.158 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:21:05.158 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:21:05.158 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:21:05.158 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:21:05.158 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:21:05.158 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:21:05.158 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:21:05.158 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:21:05.158 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:21:05.158 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:21:05.159 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:21:05.159 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:21:05.159 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:21:05.159 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:21:05.159 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:21:05.159 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:05.159 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:21:05.159 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:05.159 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:21:05.159 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:05.159 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:21:05.159 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:21:05.159 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:21:05.159 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:21:05.159 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:05.160 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:21:05.160 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:21:05.160 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:05.160 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:05.160 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:05.160 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:21:05.160 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:21:05.160 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:05.160 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:05.160 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:05.160 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:05.160 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:21:05.160 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:21:05.160 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:21:05.160 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:21:05.161 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:21:05.161 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:05.161 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:05.161 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:21:05.161 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:21:05.161 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:21:05.161 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:21:05.161 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:05.161 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:21:05.161 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:21:05.161 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:05.161 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:05.161 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:05.161 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:21:05.161 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:21:05.162 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:05.162 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:05.162 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:05.162 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:05.162 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:21:05.162 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:21:05.162 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:21:05.162 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:21:05.162 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:21:05.162 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:05.162 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:05.162 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:21:05.162 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:21:05.162 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:21:05.163 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:21:05.163 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:05.163 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:21:05.163 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:21:05.163 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:05.163 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:05.163 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:05.163 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:21:05.163 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:21:05.163 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:05.163 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:05.163 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:05.163 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:05.163 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:21:05.163 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:21:05.164 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:21:05.164 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:21:05.164 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:21:05.164 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:05.164 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:05.164 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:05.164 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:21:05.164 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:21:05.164 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:05.164 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:21:05.164 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:21:05.164 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:21:05.164 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:05.164 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:05.164 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:21:05.165 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:05.165 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:21:05.165 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:21:05.165 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 4 2024-04-22T08:21:05.165 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:05.165 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:05.165 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:21:05.165 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:05.165 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:21:05.165 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:21:05.165 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 7 2024-04-22T08:21:05.165 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:05.165 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:05.165 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": [ 2024-04-22T08:21:05.166 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:05.166 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:21:05.166 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:21:05.166 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:21:05.166 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:05.166 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:05.166 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:05.166 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:21:05.166 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:21:05.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.147+0000 7f2f1efa5640 1 -- 172.21.15.40:0/2073815058 >> v2:172.21.15.189:6800/30168 conn(0x7f2f04037850 msgr2=0x7f2f04039d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:05.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.147+0000 7f2f1efa5640 1 --2- 172.21.15.40:0/2073815058 >> v2:172.21.15.189:6800/30168 conn(0x7f2f04037850 0x7f2f04039d10 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f2f180a3c30 tx=0x7f2f0c03d040 comp rx=0 tx=0).stop 2024-04-22T08:21:05.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.147+0000 7f2f1efa5640 10 monclient: shutdown 2024-04-22T08:21:05.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.147+0000 7f2f1efa5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:05.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.147+0000 7f2f1efa5640 1 -- 172.21.15.40:0/2073815058 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f1813b070 msgr2=0x7f2f181361c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:05.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.147+0000 7f2f1d542640 1 -- 172.21.15.40:0/2073815058 reap_dead start 2024-04-22T08:21:05.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.147+0000 7f2f1efa5640 1 --2- 172.21.15.40:0/2073815058 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f1813b070 0x7f2f181361c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2f0800d830 tx=0x7f2f0800dcf0 comp rx=0 tx=0).stop 2024-04-22T08:21:05.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.147+0000 7f2f1efa5640 1 -- 172.21.15.40:0/2073815058 shutdown_connections 2024-04-22T08:21:05.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.147+0000 7f2f1efa5640 1 --2- 172.21.15.40:0/2073815058 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f1813b070 0x7f2f181361c0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:05.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.147+0000 7f2f1efa5640 1 -- 172.21.15.40:0/2073815058 >> 172.21.15.40:0/2073815058 conn(0x7f2f180a66a0 msgr2=0x7f2f18138540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:05.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.147+0000 7f2f1efa5640 1 -- 172.21.15.40:0/2073815058 shutdown_connections 2024-04-22T08:21:05.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.147+0000 7f2f1efa5640 1 -- 172.21.15.40:0/2073815058 wait complete. 2024-04-22T08:21:05.169 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:21:05.169 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-04-22T08:21:05.169 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-04-22T08:21:05.169 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-04-22T08:21:05.169 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-04-22T08:21:05.169 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-22T08:21:05.169 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T08:21:05.180 INFO:tasks.ceph.mon.a:Stopped 2024-04-22T08:21:05.180 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-22T08:21:05.180 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:21:05.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.319+0000 7fbe55abd640 10 monclient: get_monmap_and_config 2024-04-22T08:21:05.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.319+0000 7fbe55abd640 10 monclient: build_initial_monmap 2024-04-22T08:21:05.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.319+0000 7fbe55abd640 10 monclient: monmap: 2024-04-22T08:21:05.319 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:05.319 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:05.319 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:21:05.320596+0000 2024-04-22T08:21:05.319 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:21:05.320596+0000 2024-04-22T08:21:05.319 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:05.319 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:05.320 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:21:05.320 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:21:05.320 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:21:05.320 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:21:05.320 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:21:05.320 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:05.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.320+0000 7fbe55abd640 1 Processor -- start 2024-04-22T08:21:05.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.320+0000 7fbe55abd640 1 -- start start 2024-04-22T08:21:05.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.320+0000 7fbe55abd640 10 monclient: init 2024-04-22T08:21:05.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.320+0000 7fbe55abd640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:05.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.320+0000 7fbe55abd640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-04-22T08:21:05.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.320+0000 7fbe55abd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbe50135e30 addr v1:172.21.15.189:6789/0 2024-04-22T08:21:05.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.320+0000 7fbe55abd640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbe50071c90 0x7fbe5013aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:05.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.320+0000 7fbe55abd640 10 monclient(hunting): picked mon.noname-a con 0x7fbe50071c90 addr v2:172.21.15.40:3300/0 2024-04-22T08:21:05.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.321+0000 7fbe55abd640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbe5013b410 0x7fbe5013d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:05.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.321+0000 7fbe55abd640 10 monclient(hunting): picked mon.noname-c con 0x7fbe5013b410 addr v2:172.21.15.189:3300/0 2024-04-22T08:21:05.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.321+0000 7fbe55abd640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe5009de80 con 0x7fbe50135e30 2024-04-22T08:21:05.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.321+0000 7fbe55abd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:05.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.321+0000 7fbe55abd640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe5009e3d0 con 0x7fbe50071c90 2024-04-22T08:21:05.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.321+0000 7fbe55abd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:05.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.321+0000 7fbe55abd640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe5009f130 con 0x7fbe5013b410 2024-04-22T08:21:05.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.321+0000 7fbe55abd640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:05.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.321+0000 7fbe55abd640 10 monclient(hunting): authenticate will time out at 1339.314800s 2024-04-22T08:21:05.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.321+0000 7fbe4effd640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7fbe50071c90 msgr2=0x7fbe5013aed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:21:05.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.321+0000 7fbe4effd640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbe50071c90 0x7fbe5013aed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:21:05.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.321+0000 7fbe4ffff640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbe5013b410 0x7fbe5013d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:05.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.321+0000 7fbe4ffff640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbe5013b410 0x7fbe5013d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.40:45130/0 (socket says 172.21.15.40:45130) 2024-04-22T08:21:05.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.321+0000 7fbe4ffff640 1 -- 172.21.15.40:0/1641064135 learned_addr learned my addr 172.21.15.40:0/1641064135 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:21:05.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.321+0000 7fbe4ffff640 10 monclient(hunting): get_auth_request con 0x7fbe5013b410 auth_method 0 2024-04-22T08:21:05.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.321+0000 7fbe4f7fe640 1 --1- 172.21.15.40:0/1641064135 >> v1:172.21.15.189:6789/0 conn(0x7fbe50135e30 0x7fbe50136210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.40:38462/0 (socket says 172.21.15.40:38462) 2024-04-22T08:21:05.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.321+0000 7fbe4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:05.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.321+0000 7fbe4ffff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:05.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.321+0000 7fbe4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:05.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.321+0000 7fbe4ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:05.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.321+0000 7fbe4ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:05.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.321+0000 7fbe4ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:05.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.321+0000 7fbe4ffff640 10 monclient(hunting): handle_auth_done global_id 64963 payload 274 2024-04-22T08:21:05.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.321+0000 7fbe4ffff640 1 -- 172.21.15.40:0/1641064135 >> v2:172.21.15.40:3300/0 conn(0x7fbe50071c90 msgr2=0x7fbe5013aed0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:21:05.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.321+0000 7fbe4ffff640 1 --2- 172.21.15.40:0/1641064135 >> v2:172.21.15.40:3300/0 conn(0x7fbe50071c90 0x7fbe5013aed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:05.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.321+0000 7fbe4ffff640 1 -- 172.21.15.40:0/1641064135 >> v1:172.21.15.189:6789/0 conn(0x7fbe50135e30 legacy=0x7fbe50136210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:21:05.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.321+0000 7fbe4ffff640 10 monclient: _finish_hunting 0 2024-04-22T08:21:05.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.321+0000 7fbe4ffff640 1 monclient: found mon.noname-c 2024-04-22T08:21:05.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.321+0000 7fbe4ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:05.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.321+0000 7fbe4ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.189:3300/0 2024-04-22T08:21:05.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.321+0000 7fbe4ffff640 1 -- 172.21.15.40:0/1641064135 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe5009f410 con 0x7fbe5013b410 2024-04-22T08:21:05.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.324+0000 7fbe4ffff640 1 -- 172.21.15.40:0/1641064135 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe5013b410 msgr2=0x7fbe5013d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:05.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.324+0000 7fbe4ffff640 1 --2- 172.21.15.40:0/1641064135 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe5013b410 0x7fbe5013d800 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fbe48009e50 tx=0x7fbe4802bde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b9d18daf72168f06 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:05.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.324+0000 7fbe4e7fc640 1 -- 172.21.15.40:0/1641064135 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe4802cb90 con 0x7fbe5013b410 2024-04-22T08:21:05.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.324+0000 7fbe4e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:05.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.324+0000 7fbe4e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:21:05.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.324+0000 7fbe4e7fc640 10 monclient: dump: 2024-04-22T08:21:05.332 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:05.332 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:05.332 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:05.332 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:05.333 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:05.333 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:05.333 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:05.333 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:05.333 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:05.333 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:05.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.324+0000 7fbe4e7fc640 10 monclient: _finish_auth 0 2024-04-22T08:21:05.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.324+0000 7fbe4e7fc640 10 monclient: _check_auth_tickets 2024-04-22T08:21:05.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.324+0000 7fbe4e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:05.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.324+0000 7fbe55abd640 5 monclient: authenticate success, global_id 64963 2024-04-22T08:21:05.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.324+0000 7fbe55abd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:21:05.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.324+0000 7fbe4e7fc640 1 -- 172.21.15.40:0/1641064135 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe4802ccf0 con 0x7fbe5013b410 2024-04-22T08:21:05.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.324+0000 7fbe4e7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:05.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.324+0000 7fbe55abd640 10 monclient: get_monmap_and_config success 2024-04-22T08:21:05.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.324+0000 7fbe55abd640 4 set_mon_vals no callback set 2024-04-22T08:21:05.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.325+0000 7fbe4e7fc640 1 -- 172.21.15.40:0/1641064135 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe48034610 con 0x7fbe5013b410 2024-04-22T08:21:05.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.325+0000 7fbe4e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:05.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.325+0000 7fbe4e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:21:05.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.325+0000 7fbe4e7fc640 10 monclient: dump: 2024-04-22T08:21:05.334 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:05.334 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:05.334 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:05.334 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:05.334 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:05.334 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:05.334 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:05.334 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:05.334 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:05.334 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:05.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.325+0000 7fbe55abd640 10 monclient: shutdown 2024-04-22T08:21:05.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.325+0000 7fbe55abd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:05.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.325+0000 7fbe55abd640 1 -- 172.21.15.40:0/1641064135 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe5013b410 msgr2=0x7fbe5013d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:05.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.325+0000 7fbe55abd640 1 --2- 172.21.15.40:0/1641064135 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe5013b410 0x7fbe5013d800 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fbe48009e50 tx=0x7fbe4802bde0 comp rx=0 tx=0).stop 2024-04-22T08:21:05.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.325+0000 7fbe55abd640 1 -- 172.21.15.40:0/1641064135 shutdown_connections 2024-04-22T08:21:05.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.325+0000 7fbe55abd640 1 --2- 172.21.15.40:0/1641064135 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe5013b410 0x7fbe5013d800 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:05.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.325+0000 7fbe55abd640 1 --2- 172.21.15.40:0/1641064135 >> v2:172.21.15.40:3300/0 conn(0x7fbe50071c90 0x7fbe5013aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:05.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.325+0000 7fbe55abd640 1 -- 172.21.15.40:0/1641064135 >> 172.21.15.40:0/1641064135 conn(0x7fbe500a65c0 msgr2=0x7fbe500a2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:05.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.325+0000 7fbe55abd640 1 -- 172.21.15.40:0/1641064135 shutdown_connections 2024-04-22T08:21:05.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.325+0000 7fbe55abd640 1 -- 172.21.15.40:0/1641064135 wait complete. 2024-04-22T08:21:05.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.326+0000 7fbe55abd640 10 monclient: build_initial_monmap 2024-04-22T08:21:05.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.326+0000 7fbe55abd640 10 monclient: monmap: 2024-04-22T08:21:05.335 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:05.335 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:21:05.336 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:21:05.336 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:21:05.336 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:05.336 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:05.336 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:21:05.336 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:21:05.336 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:21:05.336 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:05.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.326+0000 7fbe55abd640 1 Processor -- start 2024-04-22T08:21:05.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.326+0000 7fbe55abd640 1 -- start start 2024-04-22T08:21:05.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.326+0000 7fbe55abd640 10 monclient: init 2024-04-22T08:21:05.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.327+0000 7fbe55abd640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:05.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.327+0000 7fbe55abd640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-22T08:21:05.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.327+0000 7fbe55abd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe50071c90 0x7fbe501ca820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:05.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.327+0000 7fbe55abd640 10 monclient(hunting): picked mon.noname-c con 0x7fbe50071c90 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:05.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.327+0000 7fbe55abd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe50135e30 0x7fbe50134c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:05.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.327+0000 7fbe55abd640 10 monclient(hunting): picked mon.noname-b con 0x7fbe50135e30 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:21:05.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.327+0000 7fbe55abd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe5013b410 0x7fbe5006e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:05.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.327+0000 7fbe55abd640 10 monclient(hunting): picked mon.noname-a con 0x7fbe5013b410 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:21:05.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.327+0000 7fbe55abd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:05.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.327+0000 7fbe55abd640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe501209f0 con 0x7fbe5013b410 2024-04-22T08:21:05.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.327+0000 7fbe55abd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:05.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.327+0000 7fbe55abd640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe501cb740 con 0x7fbe50135e30 2024-04-22T08:21:05.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.327+0000 7fbe55abd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:05.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.327+0000 7fbe55abd640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe500720e0 con 0x7fbe50071c90 2024-04-22T08:21:05.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.327+0000 7fbe55abd640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:05.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.327+0000 7fbe55abd640 10 monclient(hunting): authenticate will time out at 1339.320911s 2024-04-22T08:21:05.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.327+0000 7fbe4ffff640 1 -- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe5013b410 msgr2=0x7fbe5006e910 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:21:05.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.327+0000 7fbe4ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe5013b410 0x7fbe5006e910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:21:05.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.327+0000 7fbe4f7fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe50071c90 0x7fbe501ca820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:05.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.327+0000 7fbe4f7fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe50071c90 0x7fbe501ca820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:51934/0 (socket says 172.21.15.40:51934) 2024-04-22T08:21:05.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.327+0000 7fbe4f7fe640 1 -- 172.21.15.40:0/435021218 learned_addr learned my addr 172.21.15.40:0/435021218 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:21:05.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.327+0000 7fbe4f7fe640 10 monclient(hunting): get_auth_request con 0x7fbe50071c90 auth_method 0 2024-04-22T08:21:05.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.327+0000 7fbe4f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:05.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.327+0000 7fbe4f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:05.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.327+0000 7fbe4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:05.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.327+0000 7fbe4f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:05.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.327+0000 7fbe4f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:05.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.327+0000 7fbe4f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:05.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.327+0000 7fbe4effd640 1 --2- 172.21.15.40:0/435021218 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe50135e30 0x7fbe50134c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:05.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.327+0000 7fbe4effd640 10 monclient(hunting): get_auth_request con 0x7fbe50135e30 auth_method 0 2024-04-22T08:21:05.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.327+0000 7fbe4effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:05.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.327+0000 7fbe4effd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:05.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.327+0000 7fbe4effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:05.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.327+0000 7fbe4effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:05.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.327+0000 7fbe4effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:05.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.327+0000 7fbe4effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:05.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.327+0000 7fbe4f7fe640 10 monclient(hunting): handle_auth_done global_id 104156 payload 824 2024-04-22T08:21:05.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.327+0000 7fbe4f7fe640 1 -- 172.21.15.40:0/435021218 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe50135e30 msgr2=0x7fbe50134c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:05.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.327+0000 7fbe4f7fe640 1 --2- 172.21.15.40:0/435021218 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe50135e30 0x7fbe50134c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:05.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.327+0000 7fbe4f7fe640 1 -- 172.21.15.40:0/435021218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe5013b410 msgr2=0x7fbe5006e910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:21:05.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.327+0000 7fbe4f7fe640 1 --2- 172.21.15.40:0/435021218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe5013b410 0x7fbe5006e910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:05.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.327+0000 7fbe4f7fe640 10 monclient: _finish_hunting 0 2024-04-22T08:21:05.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.327+0000 7fbe4f7fe640 1 monclient: found mon.noname-c 2024-04-22T08:21:05.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.327+0000 7fbe4f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:05.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.327+0000 7fbe4f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:21:05.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.327+0000 7fbe4f7fe640 1 -- 172.21.15.40:0/435021218 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe48009b30 con 0x7fbe50071c90 2024-04-22T08:21:05.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.329+0000 7fbe4f7fe640 1 -- 172.21.15.40:0/435021218 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe50071c90 msgr2=0x7fbe501ca820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:05.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.329+0000 7fbe4f7fe640 1 --2- 172.21.15.40:0/435021218 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe50071c90 0x7fbe501ca820 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbe4000c480 tx=0x7fbe4000c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:05.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.329+0000 7fbe37fff640 1 -- 172.21.15.40:0/435021218 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe40007690 con 0x7fbe50071c90 2024-04-22T08:21:05.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.329+0000 7fbe37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:05.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.329+0000 7fbe37fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:21:05.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.329+0000 7fbe37fff640 10 monclient: dump: 2024-04-22T08:21:05.340 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:05.340 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:05.340 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:05.340 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:05.340 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:05.340 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:05.340 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:05.340 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:05.340 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:05.341 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:05.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.330+0000 7fbe37fff640 10 monclient: _finish_auth 0 2024-04-22T08:21:05.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.330+0000 7fbe37fff640 10 monclient: _check_auth_tickets 2024-04-22T08:21:05.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.330+0000 7fbe37fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:05.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.330+0000 7fbe55abd640 5 monclient: authenticate success, global_id 104156 2024-04-22T08:21:05.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.330+0000 7fbe37fff640 1 -- 172.21.15.40:0/435021218 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe40005810 con 0x7fbe50071c90 2024-04-22T08:21:05.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.330+0000 7fbe55abd640 10 monclient: _renew_subs 2024-04-22T08:21:05.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.330+0000 7fbe55abd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:21:05.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.330+0000 7fbe55abd640 1 -- 172.21.15.40:0/435021218 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe500a1cd0 con 0x7fbe50071c90 2024-04-22T08:21:05.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.330+0000 7fbe37fff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:05.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.330+0000 7fbe37fff640 1 -- 172.21.15.40:0/435021218 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe40005bc0 con 0x7fbe50071c90 2024-04-22T08:21:05.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.330+0000 7fbe37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:05.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.330+0000 7fbe37fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:21:05.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.330+0000 7fbe37fff640 10 monclient: dump: 2024-04-22T08:21:05.341 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:05.342 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:05.342 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:05.342 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:05.342 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:05.342 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:05.342 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:05.342 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:05.342 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:05.342 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:05.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.330+0000 7fbe4d7fa640 4 set_mon_vals no callback set 2024-04-22T08:21:05.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.330+0000 7fbe55abd640 10 monclient: _renew_subs 2024-04-22T08:21:05.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.330+0000 7fbe55abd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:21:05.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.330+0000 7fbe55abd640 1 -- 172.21.15.40:0/435021218 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe50121040 con 0x7fbe50071c90 2024-04-22T08:21:05.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.330+0000 7fbe37fff640 1 -- 172.21.15.40:0/435021218 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fbe40005d20 con 0x7fbe50071c90 2024-04-22T08:21:05.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.331+0000 7fbe37fff640 1 --2- 172.21.15.40:0/435021218 >> v2:172.21.15.189:6800/30168 conn(0x7fbe3c037780 0x7fbe3c039c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:05.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.331+0000 7fbe55abd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:05.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.331+0000 7fbe55abd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:05.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.331+0000 7fbe55abd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:21:05.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.331+0000 7fbe37fff640 1 -- 172.21.15.40:0/435021218 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(276..276 src has 239..276) v4 ==== 3506+0+0 (secure 0 0 0) 0x7fbe40049910 con 0x7fbe50071c90 2024-04-22T08:21:05.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.331+0000 7fbe55abd640 1 -- 172.21.15.40:0/435021218 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe5006b1e0 con 0x7fbe50071c90 2024-04-22T08:21:05.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.331+0000 7fbe4effd640 1 --2- 172.21.15.40:0/435021218 >> v2:172.21.15.189:6800/30168 conn(0x7fbe3c037780 0x7fbe3c039c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:05.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.331+0000 7fbe4effd640 10 monclient: get_auth_request con 0x7fbe3c037780 auth_method 0 2024-04-22T08:21:05.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.334+0000 7fbe4effd640 1 --2- 172.21.15.40:0/435021218 >> v2:172.21.15.189:6800/30168 conn(0x7fbe3c037780 0x7fbe3c039c40 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fbe500702f0 tx=0x7fbe440090e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:05.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.336+0000 7fbe37fff640 1 -- 172.21.15.40:0/435021218 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7fbe40017070 con 0x7fbe50071c90 2024-04-22T08:21:05.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.336+0000 7fbe37fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:05.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.336+0000 7fbe37fff640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:21:05.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.584+0000 7fbe55abd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:21:05.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.584+0000 7fbe55abd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:21:05.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.584+0000 7fbe55abd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:21:05.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.584+0000 7fbe55abd640 1 -- 172.21.15.40:0/435021218 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fbe5004e9b0 con 0x7fbe50071c90 2024-04-22T08:21:05.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.585+0000 7fbe37fff640 1 -- 172.21.15.40:0/435021218 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7fbe40019c50 con 0x7fbe50071c90 2024-04-22T08:21:05.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.585+0000 7fbe37fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:21:05.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.585+0000 7fbe37fff640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:21:05.585 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":82,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:21:05.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.586+0000 7fbe55abd640 1 -- 172.21.15.40:0/435021218 >> v2:172.21.15.189:6800/30168 conn(0x7fbe3c037780 msgr2=0x7fbe3c039c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:05.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.586+0000 7fbe55abd640 1 --2- 172.21.15.40:0/435021218 >> v2:172.21.15.189:6800/30168 conn(0x7fbe3c037780 0x7fbe3c039c40 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fbe500702f0 tx=0x7fbe440090e0 comp rx=0 tx=0).stop 2024-04-22T08:21:05.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.586+0000 7fbe55abd640 10 monclient: shutdown 2024-04-22T08:21:05.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.586+0000 7fbe55abd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:05.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.586+0000 7fbe55abd640 1 -- 172.21.15.40:0/435021218 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe50071c90 msgr2=0x7fbe501ca820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:05.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.586+0000 7fbe55abd640 1 --2- 172.21.15.40:0/435021218 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe50071c90 0x7fbe501ca820 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbe4000c480 tx=0x7fbe4000c940 comp rx=0 tx=0).stop 2024-04-22T08:21:05.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.586+0000 7fbe55abd640 1 -- 172.21.15.40:0/435021218 shutdown_connections 2024-04-22T08:21:05.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.586+0000 7fbe55abd640 1 --2- 172.21.15.40:0/435021218 >> v2:172.21.15.189:6800/30168 conn(0x7fbe3c037780 0x7fbe3c039c40 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:05.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.586+0000 7fbe55abd640 1 --2- 172.21.15.40:0/435021218 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe50135e30 0x7fbe50134c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:05.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.586+0000 7fbe55abd640 1 --2- 172.21.15.40:0/435021218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe5013b410 0x7fbe5006e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:05.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.586+0000 7fbe55abd640 1 --2- 172.21.15.40:0/435021218 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe50071c90 0x7fbe501ca820 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:05.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.587+0000 7fbe55abd640 1 -- 172.21.15.40:0/435021218 >> 172.21.15.40:0/435021218 conn(0x7fbe500a65c0 msgr2=0x7fbe500a0990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:05.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.587+0000 7fbe55abd640 1 -- 172.21.15.40:0/435021218 shutdown_connections 2024-04-22T08:21:05.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:05.587+0000 7fbe55abd640 1 -- 172.21.15.40:0/435021218 wait complete. 2024-04-22T08:21:08.597 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:21:08.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.708+0000 7f2f49abd640 10 monclient: get_monmap_and_config 2024-04-22T08:21:08.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.708+0000 7f2f49abd640 10 monclient: build_initial_monmap 2024-04-22T08:21:08.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.708+0000 7f2f49abd640 10 monclient: monmap: 2024-04-22T08:21:08.708 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:08.708 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:08.708 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:21:08.709828+0000 2024-04-22T08:21:08.709 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:21:08.709828+0000 2024-04-22T08:21:08.709 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:08.709 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:08.709 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:21:08.709 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:21:08.709 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:21:08.709 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:21:08.709 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:21:08.709 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:08.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.708+0000 7f2f49abd640 1 Processor -- start 2024-04-22T08:21:08.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.708+0000 7f2f49abd640 1 -- start start 2024-04-22T08:21:08.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.708+0000 7f2f49abd640 10 monclient: init 2024-04-22T08:21:08.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.709+0000 7f2f49abd640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:08.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.709+0000 7f2f49abd640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-04-22T08:21:08.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.709+0000 7f2f49abd640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2f4409ee10 0x7f2f440a32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:08.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.709+0000 7f2f49abd640 10 monclient(hunting): picked mon.noname-a con 0x7f2f4409ee10 addr v2:172.21.15.40:3300/0 2024-04-22T08:21:08.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.709+0000 7f2f49abd640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2f440a1d30 0x7f2f440a2180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:08.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.709+0000 7f2f49abd640 10 monclient(hunting): picked mon.noname-c con 0x7f2f440a1d30 addr v2:172.21.15.189:3300/0 2024-04-22T08:21:08.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.709+0000 7f2f49abd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2f44135d90 addr v1:172.21.15.189:6789/0 2024-04-22T08:21:08.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.709+0000 7f2f49abd640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f4409f1f0 con 0x7f2f44135d90 2024-04-22T08:21:08.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.709+0000 7f2f49abd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:08.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.709+0000 7f2f49abd640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f440a39c0 con 0x7f2f4409ee10 2024-04-22T08:21:08.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.709+0000 7f2f49abd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:08.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.709+0000 7f2f49abd640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f4409fd30 con 0x7f2f440a1d30 2024-04-22T08:21:08.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.709+0000 7f2f49abd640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:08.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.709+0000 7f2f49abd640 10 monclient(hunting): authenticate will time out at 1342.704135s 2024-04-22T08:21:08.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.709+0000 7f2f437fe640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2f4409ee10 msgr2=0x7f2f440a32b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:21:08.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.709+0000 7f2f437fe640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2f4409ee10 0x7f2f440a32b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:21:08.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.709+0000 7f2f42ffd640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2f440a1d30 0x7f2f440a2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:08.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.709+0000 7f2f43fff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2f44135d90 0x7f2f441361a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.40:38470/0 (socket says 172.21.15.40:38470) 2024-04-22T08:21:08.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.709+0000 7f2f43fff640 1 -- 172.21.15.40:0/3147218359 learned_addr learned my addr 172.21.15.40:0/3147218359 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:21:08.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.709+0000 7f2f42ffd640 10 monclient(hunting): get_auth_request con 0x7f2f440a1d30 auth_method 0 2024-04-22T08:21:08.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.709+0000 7f2f42ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:08.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.709+0000 7f2f42ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:08.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.709+0000 7f2f42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:08.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.709+0000 7f2f42ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:08.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.709+0000 7f2f42ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:08.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.709+0000 7f2f42ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:08.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.710+0000 7f2f42ffd640 10 monclient(hunting): handle_auth_done global_id 64969 payload 274 2024-04-22T08:21:08.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.710+0000 7f2f42ffd640 1 -- 172.21.15.40:0/3147218359 >> v2:172.21.15.40:3300/0 conn(0x7f2f4409ee10 msgr2=0x7f2f440a32b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:21:08.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.710+0000 7f2f42ffd640 1 --2- 172.21.15.40:0/3147218359 >> v2:172.21.15.40:3300/0 conn(0x7f2f4409ee10 0x7f2f440a32b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:08.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.710+0000 7f2f42ffd640 1 -- 172.21.15.40:0/3147218359 >> v1:172.21.15.189:6789/0 conn(0x7f2f44135d90 legacy=0x7f2f441361a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:21:08.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.710+0000 7f2f42ffd640 10 monclient: _finish_hunting 0 2024-04-22T08:21:08.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.710+0000 7f2f42ffd640 1 monclient: found mon.noname-c 2024-04-22T08:21:08.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.710+0000 7f2f42ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:08.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.710+0000 7f2f42ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.189:3300/0 2024-04-22T08:21:08.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.710+0000 7f2f42ffd640 1 -- 172.21.15.40:0/3147218359 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f440a02f0 con 0x7f2f440a1d30 2024-04-22T08:21:08.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.713+0000 7f2f42ffd640 1 -- 172.21.15.40:0/3147218359 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f440a1d30 msgr2=0x7f2f440a2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:08.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.713+0000 7f2f42ffd640 1 --2- 172.21.15.40:0/3147218359 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f440a1d30 0x7f2f440a2180 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f2f440a43e0 tx=0x7f2f3802b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10949e38d7f08fa server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:08.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.713+0000 7f2f427fc640 1 -- 172.21.15.40:0/3147218359 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f38030a00 con 0x7f2f440a1d30 2024-04-22T08:21:08.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.713+0000 7f2f427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:08.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.713+0000 7f2f427fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:21:08.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.713+0000 7f2f427fc640 10 monclient: dump: 2024-04-22T08:21:08.720 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:08.720 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:08.720 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:08.720 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:08.720 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:08.720 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:08.720 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:08.720 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:08.721 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:08.721 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:08.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.713+0000 7f2f427fc640 10 monclient: _finish_auth 0 2024-04-22T08:21:08.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.713+0000 7f2f427fc640 10 monclient: _check_auth_tickets 2024-04-22T08:21:08.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.713+0000 7f2f427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:08.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.713+0000 7f2f427fc640 1 -- 172.21.15.40:0/3147218359 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f3802e050 con 0x7f2f440a1d30 2024-04-22T08:21:08.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.713+0000 7f2f427fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:08.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.713+0000 7f2f427fc640 1 -- 172.21.15.40:0/3147218359 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f38030d80 con 0x7f2f440a1d30 2024-04-22T08:21:08.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.713+0000 7f2f427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:08.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.713+0000 7f2f427fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:21:08.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.713+0000 7f2f427fc640 10 monclient: dump: 2024-04-22T08:21:08.721 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:08.721 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:08.721 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:08.721 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:08.722 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:08.722 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:08.722 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:08.722 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:08.722 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:08.722 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:08.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.713+0000 7f2f49abd640 5 monclient: authenticate success, global_id 64969 2024-04-22T08:21:08.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.713+0000 7f2f49abd640 10 monclient: get_monmap_and_config success 2024-04-22T08:21:08.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.713+0000 7f2f49abd640 4 set_mon_vals no callback set 2024-04-22T08:21:08.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.714+0000 7f2f49abd640 10 monclient: shutdown 2024-04-22T08:21:08.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.714+0000 7f2f49abd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:08.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.714+0000 7f2f49abd640 1 -- 172.21.15.40:0/3147218359 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f440a1d30 msgr2=0x7f2f440a2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:08.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.714+0000 7f2f49abd640 1 --2- 172.21.15.40:0/3147218359 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f440a1d30 0x7f2f440a2180 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f2f440a43e0 tx=0x7f2f3802b710 comp rx=0 tx=0).stop 2024-04-22T08:21:08.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.714+0000 7f2f49abd640 1 -- 172.21.15.40:0/3147218359 shutdown_connections 2024-04-22T08:21:08.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.714+0000 7f2f49abd640 1 --2- 172.21.15.40:0/3147218359 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f440a1d30 0x7f2f440a2180 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:08.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.714+0000 7f2f49abd640 1 --2- 172.21.15.40:0/3147218359 >> v2:172.21.15.40:3300/0 conn(0x7f2f4409ee10 0x7f2f440a32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:08.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.714+0000 7f2f49abd640 1 -- 172.21.15.40:0/3147218359 >> 172.21.15.40:0/3147218359 conn(0x7f2f440a4ef0 msgr2=0x7f2f440a5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:08.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.714+0000 7f2f49abd640 1 -- 172.21.15.40:0/3147218359 shutdown_connections 2024-04-22T08:21:08.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.714+0000 7f2f49abd640 1 -- 172.21.15.40:0/3147218359 wait complete. 2024-04-22T08:21:08.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.715+0000 7f2f49abd640 10 monclient: build_initial_monmap 2024-04-22T08:21:08.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.715+0000 7f2f49abd640 10 monclient: monmap: 2024-04-22T08:21:08.723 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:08.723 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:21:08.723 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:21:08.723 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:21:08.723 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:08.723 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:08.724 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:21:08.724 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:21:08.724 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:21:08.724 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:08.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.715+0000 7f2f49abd640 1 Processor -- start 2024-04-22T08:21:08.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.715+0000 7f2f49abd640 1 -- start start 2024-04-22T08:21:08.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.715+0000 7f2f49abd640 10 monclient: init 2024-04-22T08:21:08.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.716+0000 7f2f49abd640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:08.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.716+0000 7f2f49abd640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-22T08:21:08.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.716+0000 7f2f49abd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f4409ee10 0x7f2f441cb160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:08.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.716+0000 7f2f49abd640 10 monclient(hunting): picked mon.noname-b con 0x7f2f4409ee10 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:21:08.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.716+0000 7f2f49abd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f440a1d30 0x7f2f441224f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:08.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.716+0000 7f2f49abd640 10 monclient(hunting): picked mon.noname-c con 0x7f2f440a1d30 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:08.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.716+0000 7f2f49abd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f44135d90 0x7f2f44122a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:08.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.716+0000 7f2f49abd640 10 monclient(hunting): picked mon.noname-a con 0x7f2f44135d90 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:21:08.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.716+0000 7f2f49abd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:08.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.716+0000 7f2f49abd640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f44125bd0 con 0x7f2f44135d90 2024-04-22T08:21:08.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.716+0000 7f2f49abd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:08.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.716+0000 7f2f49abd640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f44124ce0 con 0x7f2f4409ee10 2024-04-22T08:21:08.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.716+0000 7f2f49abd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:08.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.716+0000 7f2f49abd640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f44071860 con 0x7f2f440a1d30 2024-04-22T08:21:08.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.716+0000 7f2f49abd640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:08.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.716+0000 7f2f49abd640 10 monclient(hunting): authenticate will time out at 1342.710976s 2024-04-22T08:21:08.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.716+0000 7f2f43fff640 1 -- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f44135d90 msgr2=0x7f2f44122a30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:21:08.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.716+0000 7f2f43fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f44135d90 0x7f2f44122a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:21:08.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.716+0000 7f2f42ffd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f440a1d30 0x7f2f441224f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:08.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.716+0000 7f2f42ffd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f440a1d30 0x7f2f441224f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:51950/0 (socket says 172.21.15.40:51950) 2024-04-22T08:21:08.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.716+0000 7f2f42ffd640 1 -- 172.21.15.40:0/2320642987 learned_addr learned my addr 172.21.15.40:0/2320642987 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:21:08.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.716+0000 7f2f42ffd640 10 monclient(hunting): get_auth_request con 0x7f2f440a1d30 auth_method 0 2024-04-22T08:21:08.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.716+0000 7f2f42ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:08.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.716+0000 7f2f42ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:08.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.716+0000 7f2f42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:08.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.716+0000 7f2f437fe640 1 --2- 172.21.15.40:0/2320642987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f4409ee10 0x7f2f441cb160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:08.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.716+0000 7f2f437fe640 10 monclient(hunting): get_auth_request con 0x7f2f4409ee10 auth_method 0 2024-04-22T08:21:08.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.716+0000 7f2f437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:08.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.716+0000 7f2f437fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:08.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.716+0000 7f2f437fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:08.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.716+0000 7f2f42ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:08.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.716+0000 7f2f42ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:08.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.716+0000 7f2f42ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:08.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.716+0000 7f2f437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:08.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.716+0000 7f2f437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:08.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.716+0000 7f2f437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:08.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.716+0000 7f2f42ffd640 10 monclient(hunting): handle_auth_done global_id 104159 payload 824 2024-04-22T08:21:08.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.716+0000 7f2f42ffd640 1 -- 172.21.15.40:0/2320642987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f4409ee10 msgr2=0x7f2f441cb160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:08.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.716+0000 7f2f42ffd640 1 --2- 172.21.15.40:0/2320642987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f4409ee10 0x7f2f441cb160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:08.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.716+0000 7f2f42ffd640 1 -- 172.21.15.40:0/2320642987 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f44135d90 msgr2=0x7f2f44122a30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:21:08.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.716+0000 7f2f42ffd640 1 --2- 172.21.15.40:0/2320642987 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f44135d90 0x7f2f44122a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:08.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.716+0000 7f2f42ffd640 10 monclient: _finish_hunting 0 2024-04-22T08:21:08.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.716+0000 7f2f42ffd640 1 monclient: found mon.noname-c 2024-04-22T08:21:08.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.716+0000 7f2f42ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:08.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.716+0000 7f2f42ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:21:08.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.716+0000 7f2f42ffd640 1 -- 172.21.15.40:0/2320642987 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f38009400 con 0x7f2f440a1d30 2024-04-22T08:21:08.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.716+0000 7f2f437fe640 1 --2- 172.21.15.40:0/2320642987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f4409ee10 0x7f2f441cb160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:21:08.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.719+0000 7f2f42ffd640 1 -- 172.21.15.40:0/2320642987 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f440a1d30 msgr2=0x7f2f441224f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:08.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.719+0000 7f2f42ffd640 1 --2- 172.21.15.40:0/2320642987 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f440a1d30 0x7f2f441224f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2f3802bd20 tx=0x7f2f380046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:08.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.719+0000 7f2f2bfff640 1 -- 172.21.15.40:0/2320642987 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3803daa0 con 0x7f2f440a1d30 2024-04-22T08:21:08.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.719+0000 7f2f2bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:08.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.719+0000 7f2f2bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:21:08.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.719+0000 7f2f2bfff640 10 monclient: dump: 2024-04-22T08:21:08.728 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:08.728 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:08.728 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:08.728 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:08.728 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:08.728 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:08.728 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:08.728 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:08.728 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:08.729 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:08.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.719+0000 7f2f2bfff640 10 monclient: _finish_auth 0 2024-04-22T08:21:08.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.719+0000 7f2f2bfff640 10 monclient: _check_auth_tickets 2024-04-22T08:21:08.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.719+0000 7f2f2bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:08.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.719+0000 7f2f49abd640 5 monclient: authenticate success, global_id 104159 2024-04-22T08:21:08.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.719+0000 7f2f49abd640 10 monclient: _renew_subs 2024-04-22T08:21:08.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.719+0000 7f2f2bfff640 1 -- 172.21.15.40:0/2320642987 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f3803dc00 con 0x7f2f440a1d30 2024-04-22T08:21:08.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.719+0000 7f2f49abd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:21:08.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.719+0000 7f2f49abd640 1 -- 172.21.15.40:0/2320642987 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f440a2730 con 0x7f2f440a1d30 2024-04-22T08:21:08.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.719+0000 7f2f2bfff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:08.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.719+0000 7f2f2bfff640 1 -- 172.21.15.40:0/2320642987 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f38032480 con 0x7f2f440a1d30 2024-04-22T08:21:08.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.719+0000 7f2f2bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:08.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.719+0000 7f2f2bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:21:08.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.719+0000 7f2f2bfff640 10 monclient: dump: 2024-04-22T08:21:08.730 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:08.730 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:08.730 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:08.730 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:08.730 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:08.730 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:08.730 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:08.730 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:08.730 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:08.730 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:08.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.719+0000 7f2f41ffb640 4 set_mon_vals no callback set 2024-04-22T08:21:08.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.719+0000 7f2f49abd640 10 monclient: _renew_subs 2024-04-22T08:21:08.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.719+0000 7f2f49abd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:21:08.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.719+0000 7f2f49abd640 1 -- 172.21.15.40:0/2320642987 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f441240a0 con 0x7f2f440a1d30 2024-04-22T08:21:08.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.720+0000 7f2f2bfff640 1 -- 172.21.15.40:0/2320642987 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f2f38039040 con 0x7f2f440a1d30 2024-04-22T08:21:08.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.720+0000 7f2f49abd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:08.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.720+0000 7f2f49abd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:08.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.720+0000 7f2f49abd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:21:08.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.720+0000 7f2f49abd640 1 -- 172.21.15.40:0/2320642987 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f4404e9b0 con 0x7f2f440a1d30 2024-04-22T08:21:08.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.720+0000 7f2f2bfff640 1 --2- 172.21.15.40:0/2320642987 >> v2:172.21.15.189:6800/30168 conn(0x7f2f300377d0 0x7f2f30039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:08.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.720+0000 7f2f2bfff640 1 -- 172.21.15.40:0/2320642987 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(276..276 src has 239..276) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f2f3806be50 con 0x7f2f440a1d30 2024-04-22T08:21:08.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.720+0000 7f2f437fe640 1 --2- 172.21.15.40:0/2320642987 >> v2:172.21.15.189:6800/30168 conn(0x7f2f300377d0 0x7f2f30039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:08.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.720+0000 7f2f437fe640 10 monclient: get_auth_request con 0x7f2f300377d0 auth_method 0 2024-04-22T08:21:08.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.723+0000 7f2f437fe640 1 --2- 172.21.15.40:0/2320642987 >> v2:172.21.15.189:6800/30168 conn(0x7f2f300377d0 0x7f2f30039c90 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f2f34003980 tx=0x7f2f34009110 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:08.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.725+0000 7f2f2bfff640 1 -- 172.21.15.40:0/2320642987 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f2f38046ce0 con 0x7f2f440a1d30 2024-04-22T08:21:08.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.725+0000 7f2f2bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:08.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.725+0000 7f2f2bfff640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:21:08.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.972+0000 7f2f49abd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:21:08.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.972+0000 7f2f49abd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:21:08.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.972+0000 7f2f49abd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:21:08.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.972+0000 7f2f49abd640 1 -- 172.21.15.40:0/2320642987 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2f4406b410 con 0x7f2f440a1d30 2024-04-22T08:21:08.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.973+0000 7f2f2bfff640 1 -- 172.21.15.40:0/2320642987 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7f2f38037070 con 0x7f2f440a1d30 2024-04-22T08:21:08.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.973+0000 7f2f2bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:21:08.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.973+0000 7f2f2bfff640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:21:08.974 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":82,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:21:08.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.975+0000 7f2f49abd640 1 -- 172.21.15.40:0/2320642987 >> v2:172.21.15.189:6800/30168 conn(0x7f2f300377d0 msgr2=0x7f2f30039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:08.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.975+0000 7f2f49abd640 1 --2- 172.21.15.40:0/2320642987 >> v2:172.21.15.189:6800/30168 conn(0x7f2f300377d0 0x7f2f30039c90 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f2f34003980 tx=0x7f2f34009110 comp rx=0 tx=0).stop 2024-04-22T08:21:08.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.975+0000 7f2f49abd640 10 monclient: shutdown 2024-04-22T08:21:08.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.975+0000 7f2f49abd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:08.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.975+0000 7f2f49abd640 1 -- 172.21.15.40:0/2320642987 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f440a1d30 msgr2=0x7f2f441224f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:08.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.975+0000 7f2f49abd640 1 --2- 172.21.15.40:0/2320642987 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f440a1d30 0x7f2f441224f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2f3802bd20 tx=0x7f2f380046c0 comp rx=0 tx=0).stop 2024-04-22T08:21:08.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.975+0000 7f2f49abd640 1 -- 172.21.15.40:0/2320642987 shutdown_connections 2024-04-22T08:21:08.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.975+0000 7f2f49abd640 1 --2- 172.21.15.40:0/2320642987 >> v2:172.21.15.189:6800/30168 conn(0x7f2f300377d0 0x7f2f30039c90 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:08.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.975+0000 7f2f49abd640 1 --2- 172.21.15.40:0/2320642987 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f44135d90 0x7f2f44122a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:08.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.975+0000 7f2f49abd640 1 --2- 172.21.15.40:0/2320642987 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f440a1d30 0x7f2f441224f0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:08.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.975+0000 7f2f49abd640 1 --2- 172.21.15.40:0/2320642987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f4409ee10 0x7f2f441cb160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:08.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.975+0000 7f2f49abd640 1 -- 172.21.15.40:0/2320642987 >> 172.21.15.40:0/2320642987 conn(0x7f2f440a4ef0 msgr2=0x7f2f4406eea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:08.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.975+0000 7f2f49abd640 1 -- 172.21.15.40:0/2320642987 shutdown_connections 2024-04-22T08:21:08.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:08.975+0000 7f2f49abd640 1 -- 172.21.15.40:0/2320642987 wait complete. 2024-04-22T08:21:11.987 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:21:12.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.097+0000 7f95ea934640 10 monclient: get_monmap_and_config 2024-04-22T08:21:12.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.097+0000 7f95ea934640 10 monclient: build_initial_monmap 2024-04-22T08:21:12.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.097+0000 7f95ea934640 10 monclient: monmap: 2024-04-22T08:21:12.098 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:12.098 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:12.098 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:21:12.099117+0000 2024-04-22T08:21:12.098 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:21:12.099117+0000 2024-04-22T08:21:12.098 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:12.098 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:12.098 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:21:12.098 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:21:12.098 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:21:12.098 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:21:12.098 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:21:12.099 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:12.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.098+0000 7f95ea934640 1 Processor -- start 2024-04-22T08:21:12.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.098+0000 7f95ea934640 1 -- start start 2024-04-22T08:21:12.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.098+0000 7f95ea934640 10 monclient: init 2024-04-22T08:21:12.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.098+0000 7f95ea934640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:12.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.098+0000 7f95ea934640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-04-22T08:21:12.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.098+0000 7f95ea934640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f95e4135e30 0x7f95e4136210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:12.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.098+0000 7f95ea934640 10 monclient(hunting): picked mon.noname-b con 0x7f95e4135e30 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:12.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.098+0000 7f95ea934640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f95e4071c90 0x7f95e4136750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:12.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.098+0000 7f95ea934640 10 monclient(hunting): picked mon.noname-a con 0x7f95e4071c90 addr v2:172.21.15.40:3300/0 2024-04-22T08:21:12.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.098+0000 7f95ea934640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f95e40a19a0 addr v1:172.21.15.189:6789/0 2024-04-22T08:21:12.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.098+0000 7f95ea934640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95e409e310 con 0x7f95e40a19a0 2024-04-22T08:21:12.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.098+0000 7f95ea934640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:12.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.098+0000 7f95ea934640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95e409de80 con 0x7f95e4071c90 2024-04-22T08:21:12.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.098+0000 7f95ea934640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:12.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.098+0000 7f95ea934640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95e409f130 con 0x7f95e4135e30 2024-04-22T08:21:12.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.098+0000 7f95ea934640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:12.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.098+0000 7f95ea934640 10 monclient(hunting): authenticate will time out at 1346.093455s 2024-04-22T08:21:12.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.098+0000 7f95e37fe640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f95e4071c90 msgr2=0x7f95e4136750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:21:12.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.098+0000 7f95e37fe640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f95e4071c90 0x7f95e4136750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:21:12.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.098+0000 7f95e3fff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f95e4135e30 0x7f95e4136210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:12.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.098+0000 7f95e3fff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f95e4135e30 0x7f95e4136210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:55506/0 (socket says 172.21.15.40:55506) 2024-04-22T08:21:12.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.098+0000 7f95e3fff640 1 -- 172.21.15.40:0/1954586237 learned_addr learned my addr 172.21.15.40:0/1954586237 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:21:12.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.098+0000 7f95e3fff640 10 monclient(hunting): get_auth_request con 0x7f95e4135e30 auth_method 0 2024-04-22T08:21:12.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.098+0000 7f95e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:12.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.098+0000 7f95e3fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:12.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.098+0000 7f95e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:12.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.098+0000 7f95e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:12.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.098+0000 7f95e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:12.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.099+0000 7f95e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:12.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.099+0000 7f95e3fff640 10 monclient(hunting): handle_auth_done global_id 104162 payload 274 2024-04-22T08:21:12.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.099+0000 7f95e3fff640 1 -- 172.21.15.40:0/1954586237 >> v2:172.21.15.40:3300/0 conn(0x7f95e4071c90 msgr2=0x7f95e4136750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:21:12.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.099+0000 7f95e3fff640 1 --2- 172.21.15.40:0/1954586237 >> v2:172.21.15.40:3300/0 conn(0x7f95e4071c90 0x7f95e4136750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:12.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.099+0000 7f95e3fff640 1 -- 172.21.15.40:0/1954586237 >> v1:172.21.15.189:6789/0 conn(0x7f95e40a19a0 legacy=0x7f95e40a1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:21:12.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.099+0000 7f95e3fff640 10 monclient: _finish_hunting 0 2024-04-22T08:21:12.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.099+0000 7f95e3fff640 1 monclient: found mon.noname-b 2024-04-22T08:21:12.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.099+0000 7f95e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:12.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.099+0000 7f95e3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:21:12.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.099+0000 7f95e3fff640 1 -- 172.21.15.40:0/1954586237 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95e409f410 con 0x7f95e4135e30 2024-04-22T08:21:12.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.102+0000 7f95e3fff640 1 -- 172.21.15.40:0/1954586237 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f95e4135e30 msgr2=0x7f95e4136210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:12.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.102+0000 7f95e3fff640 1 --2- 172.21.15.40:0/1954586237 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f95e4135e30 0x7f95e4136210 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f95d4009870 tx=0x7f95d402b7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d249574450deaf7e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:12.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.102+0000 7f95e2ffd640 1 -- 172.21.15.40:0/1954586237 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d4031a40 con 0x7f95e4135e30 2024-04-22T08:21:12.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.102+0000 7f95e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:12.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.102+0000 7f95e2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:21:12.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.102+0000 7f95e2ffd640 10 monclient: dump: 2024-04-22T08:21:12.110 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:12.110 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:12.110 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:12.110 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:12.110 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:12.110 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:12.110 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:12.111 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:12.111 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:12.111 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:12.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.102+0000 7f95e2ffd640 10 monclient: _finish_auth 0 2024-04-22T08:21:12.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.102+0000 7f95e2ffd640 10 monclient: _check_auth_tickets 2024-04-22T08:21:12.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.102+0000 7f95e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:12.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.102+0000 7f95e2ffd640 1 -- 172.21.15.40:0/1954586237 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95d4031ba0 con 0x7f95e4135e30 2024-04-22T08:21:12.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.102+0000 7f95ea934640 5 monclient: authenticate success, global_id 104162 2024-04-22T08:21:12.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.102+0000 7f95ea934640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:21:12.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.102+0000 7f95e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:12.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.102+0000 7f95ea934640 10 monclient: get_monmap_and_config success 2024-04-22T08:21:12.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.102+0000 7f95e2ffd640 1 -- 172.21.15.40:0/1954586237 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d4033440 con 0x7f95e4135e30 2024-04-22T08:21:12.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.102+0000 7f95e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:12.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.102+0000 7f95ea934640 4 set_mon_vals no callback set 2024-04-22T08:21:12.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.102+0000 7f95e2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:21:12.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.102+0000 7f95e2ffd640 10 monclient: dump: 2024-04-22T08:21:12.112 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:12.112 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:12.112 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:12.112 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:12.112 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:12.112 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:12.112 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:12.112 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:12.112 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:12.112 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:12.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.102+0000 7f95ea934640 10 monclient: shutdown 2024-04-22T08:21:12.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.102+0000 7f95ea934640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:12.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.102+0000 7f95ea934640 1 -- 172.21.15.40:0/1954586237 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f95e4135e30 msgr2=0x7f95e4136210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:12.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.102+0000 7f95ea934640 1 --2- 172.21.15.40:0/1954586237 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f95e4135e30 0x7f95e4136210 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f95d4009870 tx=0x7f95d402b7f0 comp rx=0 tx=0).stop 2024-04-22T08:21:12.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.102+0000 7f95ea934640 1 -- 172.21.15.40:0/1954586237 shutdown_connections 2024-04-22T08:21:12.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.102+0000 7f95ea934640 1 --2- 172.21.15.40:0/1954586237 >> v2:172.21.15.40:3300/0 conn(0x7f95e4071c90 0x7f95e4136750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:12.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.102+0000 7f95ea934640 1 --2- 172.21.15.40:0/1954586237 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f95e4135e30 0x7f95e4136210 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:12.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.102+0000 7f95ea934640 1 -- 172.21.15.40:0/1954586237 >> 172.21.15.40:0/1954586237 conn(0x7f95e40a65c0 msgr2=0x7f95e40a2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:12.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.102+0000 7f95ea934640 1 -- 172.21.15.40:0/1954586237 shutdown_connections 2024-04-22T08:21:12.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.102+0000 7f95ea934640 1 -- 172.21.15.40:0/1954586237 wait complete. 2024-04-22T08:21:12.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.103+0000 7f95ea934640 10 monclient: build_initial_monmap 2024-04-22T08:21:12.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.103+0000 7f95ea934640 10 monclient: monmap: 2024-04-22T08:21:12.113 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:12.113 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:21:12.113 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:21:12.113 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:21:12.114 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:12.114 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:12.114 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:21:12.114 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:21:12.114 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:21:12.114 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:12.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.103+0000 7f95ea934640 1 Processor -- start 2024-04-22T08:21:12.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.103+0000 7f95ea934640 1 -- start start 2024-04-22T08:21:12.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.103+0000 7f95ea934640 10 monclient: init 2024-04-22T08:21:12.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.104+0000 7f95ea934640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:12.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.104+0000 7f95ea934640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-22T08:21:12.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.104+0000 7f95ea934640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f95e4071c90 0x7f95e41ca6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:12.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.104+0000 7f95ea934640 10 monclient(hunting): picked mon.noname-c con 0x7f95e4071c90 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:12.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.104+0000 7f95ea934640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95e40a19a0 0x7f95e41cade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:12.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.104+0000 7f95ea934640 10 monclient(hunting): picked mon.noname-a con 0x7f95e40a19a0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:21:12.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.104+0000 7f95ea934640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95e4135e30 0x7f95e406f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:12.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.104+0000 7f95ea934640 10 monclient(hunting): picked mon.noname-b con 0x7f95e4135e30 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:21:12.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.104+0000 7f95ea934640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:12.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.104+0000 7f95ea934640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95e40720e0 con 0x7f95e40a19a0 2024-04-22T08:21:12.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.104+0000 7f95ea934640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:12.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.104+0000 7f95ea934640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95e41bdf50 con 0x7f95e4135e30 2024-04-22T08:21:12.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.104+0000 7f95ea934640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:12.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.104+0000 7f95ea934640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95e4121cb0 con 0x7f95e4071c90 2024-04-22T08:21:12.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.104+0000 7f95ea934640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:12.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.104+0000 7f95ea934640 10 monclient(hunting): authenticate will time out at 1346.099193s 2024-04-22T08:21:12.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.104+0000 7f95e37fe640 1 -- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95e40a19a0 msgr2=0x7f95e41cade0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:21:12.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.104+0000 7f95e37fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95e40a19a0 0x7f95e41cade0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:21:12.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.104+0000 7f95e3fff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f95e4071c90 0x7f95e41ca6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:12.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.104+0000 7f95e3fff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f95e4071c90 0x7f95e41ca6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:55518/0 (socket says 172.21.15.40:55518) 2024-04-22T08:21:12.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.104+0000 7f95e3fff640 1 -- 172.21.15.40:0/165625374 learned_addr learned my addr 172.21.15.40:0/165625374 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:21:12.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.104+0000 7f95e3fff640 10 monclient(hunting): get_auth_request con 0x7f95e4071c90 auth_method 0 2024-04-22T08:21:12.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.104+0000 7f95e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:12.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.104+0000 7f95e3fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:12.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.104+0000 7f95e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:12.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.104+0000 7f95e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:12.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.104+0000 7f95e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:12.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.104+0000 7f95e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:12.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.104+0000 7f95e8ed1640 1 --2- 172.21.15.40:0/165625374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95e4135e30 0x7f95e406f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:12.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.104+0000 7f95e8ed1640 10 monclient(hunting): get_auth_request con 0x7f95e4135e30 auth_method 0 2024-04-22T08:21:12.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.104+0000 7f95e8ed1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:12.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.104+0000 7f95e8ed1640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:12.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.104+0000 7f95e8ed1640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:12.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.104+0000 7f95e3fff640 10 monclient(hunting): handle_auth_done global_id 104165 payload 824 2024-04-22T08:21:12.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.104+0000 7f95e3fff640 1 -- 172.21.15.40:0/165625374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95e4135e30 msgr2=0x7f95e406f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:12.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.104+0000 7f95e3fff640 1 --2- 172.21.15.40:0/165625374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95e4135e30 0x7f95e406f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:12.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.104+0000 7f95e3fff640 1 -- 172.21.15.40:0/165625374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95e40a19a0 msgr2=0x7f95e41cade0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:21:12.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.104+0000 7f95e3fff640 1 --2- 172.21.15.40:0/165625374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95e40a19a0 0x7f95e41cade0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:12.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.104+0000 7f95e3fff640 10 monclient: _finish_hunting 0 2024-04-22T08:21:12.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.104+0000 7f95e3fff640 1 monclient: found mon.noname-c 2024-04-22T08:21:12.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.104+0000 7f95e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:12.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.104+0000 7f95e3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:21:12.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.104+0000 7f95e3fff640 1 -- 172.21.15.40:0/165625374 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95d40094d0 con 0x7f95e4071c90 2024-04-22T08:21:12.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.104+0000 7f95e8ed1640 1 --2- 172.21.15.40:0/165625374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95e4135e30 0x7f95e406f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T08:21:12.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.107+0000 7f95e3fff640 1 -- 172.21.15.40:0/165625374 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f95e4071c90 msgr2=0x7f95e41ca6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:12.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.107+0000 7f95e3fff640 1 --2- 172.21.15.40:0/165625374 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f95e4071c90 0x7f95e41ca6b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f95d4009870 tx=0x7f95d40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:12.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.107+0000 7f95e0ff9640 1 -- 172.21.15.40:0/165625374 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d4031f10 con 0x7f95e4071c90 2024-04-22T08:21:12.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.107+0000 7f95e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:12.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.107+0000 7f95e0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:21:12.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.107+0000 7f95e0ff9640 10 monclient: dump: 2024-04-22T08:21:12.118 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:12.118 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:12.118 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:12.118 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:12.118 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:12.118 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:12.118 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:12.118 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:12.118 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:12.118 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:12.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.107+0000 7f95e0ff9640 10 monclient: _finish_auth 0 2024-04-22T08:21:12.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.107+0000 7f95e0ff9640 10 monclient: _check_auth_tickets 2024-04-22T08:21:12.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.107+0000 7f95e0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:12.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.107+0000 7f95ea934640 5 monclient: authenticate success, global_id 104165 2024-04-22T08:21:12.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.107+0000 7f95e0ff9640 1 -- 172.21.15.40:0/165625374 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95d403d470 con 0x7f95e4071c90 2024-04-22T08:21:12.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.107+0000 7f95ea934640 10 monclient: _renew_subs 2024-04-22T08:21:12.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.107+0000 7f95ea934640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:21:12.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.107+0000 7f95ea934640 1 -- 172.21.15.40:0/165625374 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95e4071390 con 0x7f95e4071c90 2024-04-22T08:21:12.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.107+0000 7f95e0ff9640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:12.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.107+0000 7f95e0ff9640 1 -- 172.21.15.40:0/165625374 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d403d820 con 0x7f95e4071c90 2024-04-22T08:21:12.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.107+0000 7f95e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:12.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.107+0000 7f95e0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:21:12.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.107+0000 7f95e0ff9640 10 monclient: dump: 2024-04-22T08:21:12.119 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:12.119 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:12.119 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:12.119 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:12.120 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:12.120 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:12.120 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:12.120 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:12.120 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:12.120 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:12.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.107+0000 7f95ea934640 10 monclient: _renew_subs 2024-04-22T08:21:12.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.107+0000 7f95ea934640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:21:12.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.107+0000 7f95ea934640 1 -- 172.21.15.40:0/165625374 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95e4135500 con 0x7f95e4071c90 2024-04-22T08:21:12.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.107+0000 7f95e27fc640 4 set_mon_vals no callback set 2024-04-22T08:21:12.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.108+0000 7f95e0ff9640 1 -- 172.21.15.40:0/165625374 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f95d403a040 con 0x7f95e4071c90 2024-04-22T08:21:12.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.108+0000 7f95e0ff9640 1 --2- 172.21.15.40:0/165625374 >> v2:172.21.15.189:6800/30168 conn(0x7f95d00377d0 0x7f95d0039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:12.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.108+0000 7f95ea934640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:12.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.108+0000 7f95ea934640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:12.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.108+0000 7f95ea934640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:21:12.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.108+0000 7f95ea934640 1 -- 172.21.15.40:0/165625374 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95e4121df0 con 0x7f95e4071c90 2024-04-22T08:21:12.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.108+0000 7f95e0ff9640 1 -- 172.21.15.40:0/165625374 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(276..276 src has 239..276) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f95d406c490 con 0x7f95e4071c90 2024-04-22T08:21:12.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.108+0000 7f95e37fe640 1 --2- 172.21.15.40:0/165625374 >> v2:172.21.15.189:6800/30168 conn(0x7f95d00377d0 0x7f95d0039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:12.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.108+0000 7f95e37fe640 10 monclient: get_auth_request con 0x7f95d00377d0 auth_method 0 2024-04-22T08:21:12.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.111+0000 7f95e37fe640 1 --2- 172.21.15.40:0/165625374 >> v2:172.21.15.189:6800/30168 conn(0x7f95d00377d0 0x7f95d0039c90 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f95d8006fd0 tx=0x7f95d8006d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:12.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.113+0000 7f95e0ff9640 1 -- 172.21.15.40:0/165625374 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f95e4121df0 con 0x7f95e4071c90 2024-04-22T08:21:12.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.113+0000 7f95e0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:12.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.113+0000 7f95e0ff9640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:21:12.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.363+0000 7f95ea934640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:21:12.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.363+0000 7f95ea934640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:21:12.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.363+0000 7f95ea934640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:21:12.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.363+0000 7f95ea934640 1 -- 172.21.15.40:0/165625374 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f95e4121df0 con 0x7f95e4071c90 2024-04-22T08:21:12.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.363+0000 7f95e0ff9640 1 -- 172.21.15.40:0/165625374 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7f95d4057620 con 0x7f95e4071c90 2024-04-22T08:21:12.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.363+0000 7f95e0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:21:12.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.363+0000 7f95e0ff9640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:21:12.364 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":82,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:21:12.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.365+0000 7f95ea934640 1 -- 172.21.15.40:0/165625374 >> v2:172.21.15.189:6800/30168 conn(0x7f95d00377d0 msgr2=0x7f95d0039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:12.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.365+0000 7f95ea934640 1 --2- 172.21.15.40:0/165625374 >> v2:172.21.15.189:6800/30168 conn(0x7f95d00377d0 0x7f95d0039c90 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f95d8006fd0 tx=0x7f95d8006d80 comp rx=0 tx=0).stop 2024-04-22T08:21:12.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.365+0000 7f95ea934640 10 monclient: shutdown 2024-04-22T08:21:12.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.365+0000 7f95ea934640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:12.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.365+0000 7f95ea934640 1 -- 172.21.15.40:0/165625374 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f95e4071c90 msgr2=0x7f95e41ca6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:12.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.365+0000 7f95ea934640 1 --2- 172.21.15.40:0/165625374 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f95e4071c90 0x7f95e41ca6b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f95d4009870 tx=0x7f95d40046c0 comp rx=0 tx=0).stop 2024-04-22T08:21:12.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.365+0000 7f95ea934640 1 -- 172.21.15.40:0/165625374 shutdown_connections 2024-04-22T08:21:12.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.365+0000 7f95ea934640 1 --2- 172.21.15.40:0/165625374 >> v2:172.21.15.189:6800/30168 conn(0x7f95d00377d0 0x7f95d0039c90 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:12.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.365+0000 7f95ea934640 1 --2- 172.21.15.40:0/165625374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95e4135e30 0x7f95e406f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:12.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.365+0000 7f95ea934640 1 --2- 172.21.15.40:0/165625374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95e40a19a0 0x7f95e41cade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:12.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.365+0000 7f95ea934640 1 --2- 172.21.15.40:0/165625374 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f95e4071c90 0x7f95e41ca6b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:12.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.365+0000 7f95ea934640 1 -- 172.21.15.40:0/165625374 >> 172.21.15.40:0/165625374 conn(0x7f95e40a65c0 msgr2=0x7f95e409bdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:12.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.365+0000 7f95ea934640 1 -- 172.21.15.40:0/165625374 shutdown_connections 2024-04-22T08:21:12.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:12.365+0000 7f95ea934640 1 -- 172.21.15.40:0/165625374 wait complete. 2024-04-22T08:21:15.378 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:21:15.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.489+0000 7fa503f22640 10 monclient: get_monmap_and_config 2024-04-22T08:21:15.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.489+0000 7fa503f22640 10 monclient: build_initial_monmap 2024-04-22T08:21:15.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.489+0000 7fa503f22640 10 monclient: monmap: 2024-04-22T08:21:15.490 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:15.490 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:15.490 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:21:15.490957+0000 2024-04-22T08:21:15.490 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:21:15.490957+0000 2024-04-22T08:21:15.490 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:15.490 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:15.490 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:21:15.490 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:21:15.490 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:21:15.490 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:21:15.490 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:21:15.490 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:15.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.489+0000 7fa503f22640 1 Processor -- start 2024-04-22T08:21:15.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.490+0000 7fa503f22640 1 -- start start 2024-04-22T08:21:15.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.490+0000 7fa503f22640 10 monclient: init 2024-04-22T08:21:15.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.490+0000 7fa503f22640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:15.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.490+0000 7fa503f22640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-04-22T08:21:15.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.490+0000 7fa503f22640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa4fc09ee10 0x7fa4fc0a32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:15.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.490+0000 7fa503f22640 10 monclient(hunting): picked mon.noname-c con 0x7fa4fc09ee10 addr v2:172.21.15.189:3300/0 2024-04-22T08:21:15.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.490+0000 7fa503f22640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4fc0a1d30 0x7fa4fc0a2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:15.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.490+0000 7fa503f22640 10 monclient(hunting): picked mon.noname-b con 0x7fa4fc0a1d30 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:15.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.490+0000 7fa503f22640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa4fc135d90 addr v1:172.21.15.189:6789/0 2024-04-22T08:21:15.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.490+0000 7fa503f22640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4fc09f1f0 con 0x7fa4fc135d90 2024-04-22T08:21:15.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.490+0000 7fa503f22640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:15.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.490+0000 7fa503f22640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4fc0a39c0 con 0x7fa4fc09ee10 2024-04-22T08:21:15.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.490+0000 7fa503f22640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:15.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.490+0000 7fa503f22640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4fc09fd30 con 0x7fa4fc0a1d30 2024-04-22T08:21:15.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.490+0000 7fa503f22640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:15.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.490+0000 7fa503f22640 10 monclient(hunting): authenticate will time out at 1349.485363s 2024-04-22T08:21:15.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.490+0000 7fa5014bd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4fc0a1d30 0x7fa4fc0a2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:15.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.490+0000 7fa5014bd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4fc0a1d30 0x7fa4fc0a2150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:55532/0 (socket says 172.21.15.40:55532) 2024-04-22T08:21:15.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.490+0000 7fa5014bd640 1 -- 172.21.15.40:0/282291303 learned_addr learned my addr 172.21.15.40:0/282291303 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:21:15.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.490+0000 7fa5014bd640 10 monclient(hunting): get_auth_request con 0x7fa4fc0a1d30 auth_method 0 2024-04-22T08:21:15.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.490+0000 7fa5014bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:15.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.490+0000 7fa5014bd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:15.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.490+0000 7fa5014bd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:15.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.490+0000 7fa5014bd640 1 -- 172.21.15.40:0/282291303 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4fc0a1d30 msgr2=0x7fa4fc0a2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:21:15.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.490+0000 7fa5014bd640 1 -- 172.21.15.40:0/282291303 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4fc0a1d30 msgr2=0x7fa4fc0a2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:21:15.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.490+0000 7fa5014bd640 1 --2- 172.21.15.40:0/282291303 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4fc0a1d30 0x7fa4fc0a2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:21:15.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.490+0000 7fa5014bd640 1 --2- 172.21.15.40:0/282291303 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4fc0a1d30 0x7fa4fc0a2150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:21:15.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.490+0000 7fa501cbe640 1 --2- 172.21.15.40:0/282291303 >> v2:172.21.15.189:3300/0 conn(0x7fa4fc09ee10 0x7fa4fc0a32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:15.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.491+0000 7fa501cbe640 10 monclient(hunting): get_auth_request con 0x7fa4fc09ee10 auth_method 0 2024-04-22T08:21:15.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.491+0000 7fa501cbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:15.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.491+0000 7fa501cbe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:15.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.491+0000 7fa501cbe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:15.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.491+0000 7fa501cbe640 1 -- 172.21.15.40:0/282291303 >> v2:172.21.15.189:3300/0 conn(0x7fa4fc09ee10 msgr2=0x7fa4fc0a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:21:15.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.491+0000 7fa501cbe640 1 -- 172.21.15.40:0/282291303 >> v2:172.21.15.189:3300/0 conn(0x7fa4fc09ee10 msgr2=0x7fa4fc0a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:21:15.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.491+0000 7fa501cbe640 1 --2- 172.21.15.40:0/282291303 >> v2:172.21.15.189:3300/0 conn(0x7fa4fc09ee10 0x7fa4fc0a32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:21:15.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.491+0000 7fa501cbe640 1 --2- 172.21.15.40:0/282291303 >> v2:172.21.15.189:3300/0 conn(0x7fa4fc09ee10 0x7fa4fc0a32b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:21:15.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.493+0000 7fa5024bf640 1 -- 172.21.15.40:0/282291303 >> v1:172.21.15.189:6789/0 conn(0x7fa4fc135d90 legacy=0x7fa4fc1361c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:15.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.691+0000 7fa5014bd640 1 --2- 172.21.15.40:0/282291303 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4fc0a1d30 0x7fa4fc0a2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:15.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.691+0000 7fa5014bd640 10 monclient(hunting): get_auth_request con 0x7fa4fc0a1d30 auth_method 0 2024-04-22T08:21:15.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.691+0000 7fa5014bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:15.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.691+0000 7fa5014bd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:15.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.691+0000 7fa5014bd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:21:15.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.691+0000 7fa5014bd640 1 -- 172.21.15.40:0/282291303 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4fc0a1d30 msgr2=0x7fa4fc0a2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:21:15.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.691+0000 7fa5014bd640 1 -- 172.21.15.40:0/282291303 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4fc0a1d30 msgr2=0x7fa4fc0a2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:21:15.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.691+0000 7fa5014bd640 1 --2- 172.21.15.40:0/282291303 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4fc0a1d30 0x7fa4fc0a2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:21:15.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.691+0000 7fa5014bd640 1 --2- 172.21.15.40:0/282291303 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4fc0a1d30 0x7fa4fc0a2150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:21:15.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.692+0000 7fa501cbe640 1 --2- 172.21.15.40:0/282291303 >> v2:172.21.15.189:3300/0 conn(0x7fa4fc09ee10 0x7fa4fc0a32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:15.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.692+0000 7fa501cbe640 10 monclient(hunting): get_auth_request con 0x7fa4fc09ee10 auth_method 0 2024-04-22T08:21:15.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.692+0000 7fa501cbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:15.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.692+0000 7fa501cbe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:15.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.692+0000 7fa501cbe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:21:15.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.692+0000 7fa501cbe640 1 -- 172.21.15.40:0/282291303 >> v2:172.21.15.189:3300/0 conn(0x7fa4fc09ee10 msgr2=0x7fa4fc0a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:21:15.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.692+0000 7fa501cbe640 1 -- 172.21.15.40:0/282291303 >> v2:172.21.15.189:3300/0 conn(0x7fa4fc09ee10 msgr2=0x7fa4fc0a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:21:15.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.692+0000 7fa501cbe640 1 --2- 172.21.15.40:0/282291303 >> v2:172.21.15.189:3300/0 conn(0x7fa4fc09ee10 0x7fa4fc0a32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:21:15.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:15.692+0000 7fa501cbe640 1 --2- 172.21.15.40:0/282291303 >> v2:172.21.15.189:3300/0 conn(0x7fa4fc09ee10 0x7fa4fc0a32b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:21:16.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:16.092+0000 7fa5014bd640 1 --2- 172.21.15.40:0/282291303 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4fc0a1d30 0x7fa4fc0a2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:16.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:16.092+0000 7fa5014bd640 10 monclient(hunting): get_auth_request con 0x7fa4fc0a1d30 auth_method 0 2024-04-22T08:21:16.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:16.092+0000 7fa5014bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:16.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:16.092+0000 7fa5014bd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:16.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:16.092+0000 7fa5014bd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:21:16.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:16.092+0000 7fa5014bd640 1 -- 172.21.15.40:0/282291303 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4fc0a1d30 msgr2=0x7fa4fc0a2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:21:16.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:16.092+0000 7fa5014bd640 1 -- 172.21.15.40:0/282291303 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4fc0a1d30 msgr2=0x7fa4fc0a2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:21:16.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:16.092+0000 7fa5014bd640 1 --2- 172.21.15.40:0/282291303 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4fc0a1d30 0x7fa4fc0a2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:21:16.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:16.092+0000 7fa5014bd640 1 --2- 172.21.15.40:0/282291303 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4fc0a1d30 0x7fa4fc0a2150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:21:16.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:16.093+0000 7fa501cbe640 1 --2- 172.21.15.40:0/282291303 >> v2:172.21.15.189:3300/0 conn(0x7fa4fc09ee10 0x7fa4fc0a32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:16.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:16.093+0000 7fa501cbe640 10 monclient(hunting): get_auth_request con 0x7fa4fc09ee10 auth_method 0 2024-04-22T08:21:16.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:16.093+0000 7fa501cbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:16.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:16.093+0000 7fa501cbe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:16.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:16.093+0000 7fa501cbe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:21:16.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:16.093+0000 7fa501cbe640 1 -- 172.21.15.40:0/282291303 >> v2:172.21.15.189:3300/0 conn(0x7fa4fc09ee10 msgr2=0x7fa4fc0a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:21:16.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:16.093+0000 7fa501cbe640 1 -- 172.21.15.40:0/282291303 >> v2:172.21.15.189:3300/0 conn(0x7fa4fc09ee10 msgr2=0x7fa4fc0a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:21:16.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:16.093+0000 7fa501cbe640 1 --2- 172.21.15.40:0/282291303 >> v2:172.21.15.189:3300/0 conn(0x7fa4fc09ee10 0x7fa4fc0a32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:21:16.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:16.093+0000 7fa501cbe640 1 --2- 172.21.15.40:0/282291303 >> v2:172.21.15.189:3300/0 conn(0x7fa4fc09ee10 0x7fa4fc0a32b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:21:16.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:16.894+0000 7fa5014bd640 1 --2- 172.21.15.40:0/282291303 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4fc0a1d30 0x7fa4fc0a2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:16.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:16.894+0000 7fa5014bd640 10 monclient(hunting): get_auth_request con 0x7fa4fc0a1d30 auth_method 0 2024-04-22T08:21:16.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:16.894+0000 7fa5014bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:16.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:16.894+0000 7fa5014bd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:16.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:16.894+0000 7fa5014bd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:21:16.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:16.894+0000 7fa5014bd640 1 -- 172.21.15.40:0/282291303 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4fc0a1d30 msgr2=0x7fa4fc0a2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:21:16.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:16.894+0000 7fa5014bd640 1 -- 172.21.15.40:0/282291303 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4fc0a1d30 msgr2=0x7fa4fc0a2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:21:16.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:16.894+0000 7fa5014bd640 1 --2- 172.21.15.40:0/282291303 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4fc0a1d30 0x7fa4fc0a2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:21:16.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:16.894+0000 7fa5014bd640 1 --2- 172.21.15.40:0/282291303 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4fc0a1d30 0x7fa4fc0a2150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T08:21:16.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:16.894+0000 7fa501cbe640 1 --2- 172.21.15.40:0/282291303 >> v2:172.21.15.189:3300/0 conn(0x7fa4fc09ee10 0x7fa4fc0a32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:16.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:16.894+0000 7fa501cbe640 10 monclient(hunting): get_auth_request con 0x7fa4fc09ee10 auth_method 0 2024-04-22T08:21:16.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:16.894+0000 7fa501cbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:16.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:16.894+0000 7fa501cbe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:16.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:16.894+0000 7fa501cbe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:21:16.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:16.895+0000 7fa501cbe640 1 -- 172.21.15.40:0/282291303 >> v2:172.21.15.189:3300/0 conn(0x7fa4fc09ee10 msgr2=0x7fa4fc0a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:21:16.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:16.895+0000 7fa501cbe640 1 -- 172.21.15.40:0/282291303 >> v2:172.21.15.189:3300/0 conn(0x7fa4fc09ee10 msgr2=0x7fa4fc0a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:21:16.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:16.895+0000 7fa501cbe640 1 --2- 172.21.15.40:0/282291303 >> v2:172.21.15.189:3300/0 conn(0x7fa4fc09ee10 0x7fa4fc0a32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:21:16.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:16.895+0000 7fa501cbe640 1 --2- 172.21.15.40:0/282291303 >> v2:172.21.15.189:3300/0 conn(0x7fa4fc09ee10 0x7fa4fc0a32b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T08:21:18.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:18.491+0000 7fa4f37fe640 10 monclient(hunting): tick 2024-04-22T08:21:18.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:18.491+0000 7fa4f37fe640 10 monclient(hunting): _check_auth_tickets 2024-04-22T08:21:18.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:18.491+0000 7fa4f37fe640 1 monclient(hunting): continuing hunt 2024-04-22T08:21:18.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:18.491+0000 7fa4f37fe640 10 monclient(hunting): _reopen_session rank -1 2024-04-22T08:21:18.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:18.491+0000 7fa4f37fe640 1 -- 172.21.15.40:0/282291303 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4fc0a1d30 msgr2=0x7fa4fc0a2150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:21:18.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:18.491+0000 7fa4f37fe640 1 --2- 172.21.15.40:0/282291303 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4fc0a1d30 0x7fa4fc0a2150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:18.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:18.491+0000 7fa4f37fe640 1 -- 172.21.15.40:0/282291303 >> v2:172.21.15.189:3300/0 conn(0x7fa4fc09ee10 msgr2=0x7fa4fc0a32b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:21:18.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:18.491+0000 7fa4f37fe640 1 --2- 172.21.15.40:0/282291303 >> v2:172.21.15.189:3300/0 conn(0x7fa4fc09ee10 0x7fa4fc0a32b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:18.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:18.491+0000 7fa4f37fe640 1 -- 172.21.15.40:0/282291303 >> v1:172.21.15.189:6789/0 conn(0x7fa4fc135d90 legacy=0x7fa4fc1361c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:18.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:18.491+0000 7fa4f37fe640 10 monclient: _add_conns ranks=[3,0] 2024-04-22T08:21:18.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:18.491+0000 7fa4f37fe640 1 --2- 172.21.15.40:0/282291303 >> v2:172.21.15.40:3300/0 conn(0x7fa4e0003760 0x7fa4e0005be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:18.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:18.491+0000 7fa4f37fe640 10 monclient(hunting): picked mon.noname-a con 0x7fa4e0003760 addr v2:172.21.15.40:3300/0 2024-04-22T08:21:18.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:18.491+0000 7fa4f37fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa4e00062d0 addr v1:172.21.15.40:6789/0 2024-04-22T08:21:18.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:18.491+0000 7fa4f37fe640 1 -- 172.21.15.40:0/282291303 --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4fc0a02f0 con 0x7fa4e00062d0 2024-04-22T08:21:18.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:18.491+0000 7fa4f37fe640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:18.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:18.491+0000 7fa4f37fe640 1 -- 172.21.15.40:0/282291303 --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4fc0a39c0 con 0x7fa4e0003760 2024-04-22T08:21:18.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:18.491+0000 7fa4f37fe640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:18.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:18.491+0000 7fa5014bd640 1 -- 172.21.15.40:0/282291303 >> v1:172.21.15.40:6789/0 conn(0x7fa4e00062d0 legacy=0x7fa4e0008770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-04-22T08:21:18.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:18.491+0000 7fa501cbe640 1 -- 172.21.15.40:0/282291303 >> v2:172.21.15.40:3300/0 conn(0x7fa4e0003760 msgr2=0x7fa4e0005be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:21:18.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:18.491+0000 7fa501cbe640 1 --2- 172.21.15.40:0/282291303 >> v2:172.21.15.40:3300/0 conn(0x7fa4e0003760 0x7fa4e0005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:21:18.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:18.692+0000 7fa501cbe640 1 -- 172.21.15.40:0/282291303 >> v2:172.21.15.40:3300/0 conn(0x7fa4e0003760 msgr2=0x7fa4e0005be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:21:18.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:18.692+0000 7fa5014bd640 1 -- 172.21.15.40:0/282291303 >> v1:172.21.15.40:6789/0 conn(0x7fa4e00062d0 legacy=0x7fa4e0008770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-04-22T08:21:18.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:18.692+0000 7fa501cbe640 1 --2- 172.21.15.40:0/282291303 >> v2:172.21.15.40:3300/0 conn(0x7fa4e0003760 0x7fa4e0005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:21:19.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:19.093+0000 7fa5014bd640 1 -- 172.21.15.40:0/282291303 >> v1:172.21.15.40:6789/0 conn(0x7fa4e00062d0 legacy=0x7fa4e0008770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-04-22T08:21:19.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:19.093+0000 7fa501cbe640 1 -- 172.21.15.40:0/282291303 >> v2:172.21.15.40:3300/0 conn(0x7fa4e0003760 msgr2=0x7fa4e0005be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:21:19.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:19.093+0000 7fa501cbe640 1 --2- 172.21.15.40:0/282291303 >> v2:172.21.15.40:3300/0 conn(0x7fa4e0003760 0x7fa4e0005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:21:19.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:19.894+0000 7fa501cbe640 1 -- 172.21.15.40:0/282291303 >> v2:172.21.15.40:3300/0 conn(0x7fa4e0003760 msgr2=0x7fa4e0005be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:21:19.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:19.894+0000 7fa5014bd640 1 -- 172.21.15.40:0/282291303 >> v1:172.21.15.40:6789/0 conn(0x7fa4e00062d0 legacy=0x7fa4e0008770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-04-22T08:21:19.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:19.894+0000 7fa501cbe640 1 --2- 172.21.15.40:0/282291303 >> v2:172.21.15.40:3300/0 conn(0x7fa4e0003760 0x7fa4e0005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T08:21:20.519 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:21:20.519 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38931]: starting. 2024-04-22T08:21:20.519 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38931]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-38893 2024-04-22T08:21:20.519 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38931]: created object 0... 2024-04-22T08:21:20.519 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38931]: created object 25... 2024-04-22T08:21:20.519 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38931]: created object 49... 2024-04-22T08:21:20.519 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38931]: finishing. 2024-04-22T08:21:20.519 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[38931]: shutting down. 2024-04-22T08:21:20.559 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:21:20.559 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38933]: starting. 2024-04-22T08:21:20.560 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38933]: listing objects. 2024-04-22T08:21:20.560 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38933]: listed object 0... 2024-04-22T08:21:20.560 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38933]: listed object 25... 2024-04-22T08:21:20.560 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38933]: saw 50 objects 2024-04-22T08:21:20.560 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[38933]: shutting down. 2024-04-22T08:21:21.403 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:21:21.403 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[38932]: starting. 2024-04-22T08:21:21.403 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[38932]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-38893 2024-04-22T08:21:21.403 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[38932]: shutting down. 2024-04-22T08:21:21.404 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:21:21.405 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:21:21.405 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:21:21.407 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:21:21.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.491+0000 7fa4f37fe640 10 monclient(hunting): tick 2024-04-22T08:21:21.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.491+0000 7fa4f37fe640 10 monclient(hunting): _check_auth_tickets 2024-04-22T08:21:21.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.491+0000 7fa4f37fe640 1 monclient(hunting): continuing hunt 2024-04-22T08:21:21.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.491+0000 7fa4f37fe640 10 monclient(hunting): _reopen_session rank -1 2024-04-22T08:21:21.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.491+0000 7fa4f37fe640 1 -- 172.21.15.40:0/282291303 >> v2:172.21.15.40:3300/0 conn(0x7fa4e0003760 msgr2=0x7fa4e0005be0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:21:21.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.491+0000 7fa4f37fe640 1 --2- 172.21.15.40:0/282291303 >> v2:172.21.15.40:3300/0 conn(0x7fa4e0003760 0x7fa4e0005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:21.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.491+0000 7fa4f37fe640 1 -- 172.21.15.40:0/282291303 >> v1:172.21.15.40:6789/0 conn(0x7fa4e00062d0 legacy=0x7fa4e0008770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:21:21.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.491+0000 7fa4f37fe640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-04-22T08:21:21.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.491+0000 7fa5024bf640 1 -- 172.21.15.40:0/282291303 reap_dead start 2024-04-22T08:21:21.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.491+0000 7fa4f37fe640 1 --2- 172.21.15.40:0/282291303 >> v2:172.21.15.40:3300/0 conn(0x7fa4e000a000 0x7fa4e000a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:21.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.491+0000 7fa4f37fe640 10 monclient(hunting): picked mon.noname-a con 0x7fa4e000a000 addr v2:172.21.15.40:3300/0 2024-04-22T08:21:21.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.491+0000 7fa4f37fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa4e00062d0 addr v1:172.21.15.40:6789/0 2024-04-22T08:21:21.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.491+0000 7fa4f37fe640 1 --2- 172.21.15.40:0/282291303 >> v2:172.21.15.189:3300/0 conn(0x7fa4fc09ee10 0x7fa4e000aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:21.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.491+0000 7fa4f37fe640 10 monclient(hunting): picked mon.noname-c con 0x7fa4fc09ee10 addr v2:172.21.15.189:3300/0 2024-04-22T08:21:21.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.491+0000 7fa4f37fe640 1 -- 172.21.15.40:0/282291303 --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4e000b0c0 con 0x7fa4e00062d0 2024-04-22T08:21:21.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.491+0000 7fa4f37fe640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:21.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.491+0000 7fa4f37fe640 1 -- 172.21.15.40:0/282291303 --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4fc0a39c0 con 0x7fa4e000a000 2024-04-22T08:21:21.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.491+0000 7fa4f37fe640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:21.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.491+0000 7fa4f37fe640 1 -- 172.21.15.40:0/282291303 --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4fc09fd30 con 0x7fa4fc09ee10 2024-04-22T08:21:21.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.491+0000 7fa4f37fe640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:21.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.491+0000 7fa5014bd640 1 -- 172.21.15.40:0/282291303 >> v1:172.21.15.40:6789/0 conn(0x7fa4e00062d0 legacy=0x7fa4e0008f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-04-22T08:21:21.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.491+0000 7fa501cbe640 1 -- 172.21.15.40:0/282291303 >> v2:172.21.15.40:3300/0 conn(0x7fa4e000a000 msgr2=0x7fa4e000a440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:21:21.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.491+0000 7fa501cbe640 1 --2- 172.21.15.40:0/282291303 >> v2:172.21.15.40:3300/0 conn(0x7fa4e000a000 0x7fa4e000a440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:21:21.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.491+0000 7fa5024bf640 1 --2- 172.21.15.40:0/282291303 >> v2:172.21.15.189:3300/0 conn(0x7fa4fc09ee10 0x7fa4e000aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:21.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.491+0000 7fa5024bf640 10 monclient(hunting): get_auth_request con 0x7fa4fc09ee10 auth_method 0 2024-04-22T08:21:21.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.491+0000 7fa5024bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:21.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.491+0000 7fa5024bf640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:21.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.491+0000 7fa5024bf640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:21.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.491+0000 7fa5024bf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:21.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.491+0000 7fa5024bf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:21.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.491+0000 7fa5024bf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:21.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.491+0000 7fa5024bf640 10 monclient(hunting): handle_auth_done global_id 64984 payload 274 2024-04-22T08:21:21.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.491+0000 7fa5024bf640 1 -- 172.21.15.40:0/282291303 >> v2:172.21.15.40:3300/0 conn(0x7fa4e000a000 msgr2=0x7fa4e000a440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:21:21.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.491+0000 7fa5024bf640 1 --2- 172.21.15.40:0/282291303 >> v2:172.21.15.40:3300/0 conn(0x7fa4e000a000 0x7fa4e000a440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:21.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.491+0000 7fa5024bf640 1 -- 172.21.15.40:0/282291303 >> v1:172.21.15.40:6789/0 conn(0x7fa4e00062d0 legacy=0x7fa4e0008f40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:21:21.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.491+0000 7fa5024bf640 10 monclient: _finish_hunting 0 2024-04-22T08:21:21.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.491+0000 7fa5024bf640 1 monclient: found mon.noname-c 2024-04-22T08:21:21.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.491+0000 7fa5024bf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:21.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.491+0000 7fa5024bf640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.189:3300/0 2024-04-22T08:21:21.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.491+0000 7fa5024bf640 1 -- 172.21.15.40:0/282291303 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4fc0a02f0 con 0x7fa4fc09ee10 2024-04-22T08:21:21.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.494+0000 7fa5024bf640 1 -- 172.21.15.40:0/282291303 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4fc09ee10 msgr2=0x7fa4e000aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:21.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.494+0000 7fa5024bf640 1 --2- 172.21.15.40:0/282291303 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4fc09ee10 0x7fa4e000aa50 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc0a4650 tx=0x7fa4f402b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd5521efcafc6ec6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:21.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.494+0000 7fa5024bf640 1 --2- 172.21.15.40:0/282291303 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4fc09ee10 0x7fa4e000aa50 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc0a4650 tx=0x7fa4f402b880 comp rx=0 tx=0).queue_received will delay after 1.71377e+09 on 0x7fa4f40073d0 config(0 keys) v1 2024-04-22T08:21:21.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.494+0000 7fa500cbc640 1 -- 172.21.15.40:0/282291303 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4f4007270 con 0x7fa4fc09ee10 2024-04-22T08:21:21.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.494+0000 7fa500cbc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:21.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.495+0000 7fa500cbc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:21:21.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.495+0000 7fa500cbc640 10 monclient: dump: 2024-04-22T08:21:21.497 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:21.498 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:21.498 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:21.498 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:21.498 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:21.498 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:21.498 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:21.498 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:21.498 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:21.498 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:21.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.495+0000 7fa500cbc640 10 monclient: _finish_auth 0 2024-04-22T08:21:21.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.495+0000 7fa500cbc640 10 monclient: _check_auth_tickets 2024-04-22T08:21:21.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.495+0000 7fa500cbc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:21.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.495+0000 7fa500cbc640 1 -- 172.21.15.40:0/282291303 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4f40073d0 con 0x7fa4fc09ee10 2024-04-22T08:21:21.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.495+0000 7fa500cbc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:21.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.495+0000 7fa503f22640 5 monclient: authenticate success, global_id 64984 2024-04-22T08:21:21.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.495+0000 7fa503f22640 10 monclient: get_monmap_and_config success 2024-04-22T08:21:21.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.495+0000 7fa503f22640 4 set_mon_vals no callback set 2024-04-22T08:21:21.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.495+0000 7fa503f22640 10 monclient: shutdown 2024-04-22T08:21:21.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.495+0000 7fa503f22640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:21.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.495+0000 7fa503f22640 1 -- 172.21.15.40:0/282291303 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4fc09ee10 msgr2=0x7fa4e000aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:21.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.495+0000 7fa503f22640 1 --2- 172.21.15.40:0/282291303 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4fc09ee10 0x7fa4e000aa50 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc0a4650 tx=0x7fa4f402b880 comp rx=0 tx=0).stop 2024-04-22T08:21:21.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.495+0000 7fa500cbc640 1 -- 172.21.15.40:0/282291303 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 0 0 0) 0x7fa4f4007740 con 0x7fa4fc09ee10 2024-04-22T08:21:21.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.495+0000 7fa500cbc640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-04-22T08:21:21.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.495+0000 7fa503f22640 1 -- 172.21.15.40:0/282291303 shutdown_connections 2024-04-22T08:21:21.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.495+0000 7fa503f22640 1 --2- 172.21.15.40:0/282291303 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4fc09ee10 0x7fa4e000aa50 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:21.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.495+0000 7fa503f22640 1 --2- 172.21.15.40:0/282291303 >> v2:172.21.15.40:3300/0 conn(0x7fa4e000a000 0x7fa4e000a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:21.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.495+0000 7fa503f22640 1 -- 172.21.15.40:0/282291303 >> 172.21.15.40:0/282291303 conn(0x7fa4fc0a4ef0 msgr2=0x7fa4fc0a5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:21.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.496+0000 7fa503f22640 1 -- 172.21.15.40:0/282291303 shutdown_connections 2024-04-22T08:21:21.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.496+0000 7fa503f22640 1 -- 172.21.15.40:0/282291303 wait complete. 2024-04-22T08:21:21.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.496+0000 7fa503f22640 10 monclient: build_initial_monmap 2024-04-22T08:21:21.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.496+0000 7fa503f22640 10 monclient: monmap: 2024-04-22T08:21:21.500 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:21.500 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:21:21.500 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:21:21.500 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:21:21.500 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:21.500 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:21.500 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:21:21.500 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:21:21.500 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:21:21.500 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:21.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.497+0000 7fa503f22640 1 Processor -- start 2024-04-22T08:21:21.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.497+0000 7fa503f22640 1 -- start start 2024-04-22T08:21:21.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.497+0000 7fa503f22640 10 monclient: init 2024-04-22T08:21:21.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.497+0000 7fa503f22640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:21.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.497+0000 7fa503f22640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-22T08:21:21.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.497+0000 7fa503f22640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4e00062d0 0x7fa4fc1caa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:21.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.497+0000 7fa503f22640 10 monclient(hunting): picked mon.noname-a con 0x7fa4e00062d0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:21:21.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.498+0000 7fa503f22640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4e000a000 0x7fa4fc134ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:21.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.498+0000 7fa503f22640 10 monclient(hunting): picked mon.noname-c con 0x7fa4e000a000 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:21.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.498+0000 7fa503f22640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4fc09ee10 0x7fa4fc135430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:21.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.498+0000 7fa503f22640 10 monclient(hunting): picked mon.noname-b con 0x7fa4fc09ee10 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:21:21.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.498+0000 7fa503f22640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:21.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.498+0000 7fa503f22640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4fc1215f0 con 0x7fa4e00062d0 2024-04-22T08:21:21.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.498+0000 7fa503f22640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:21.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.498+0000 7fa503f22640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4fc0a2730 con 0x7fa4fc09ee10 2024-04-22T08:21:21.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.498+0000 7fa503f22640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:21.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.498+0000 7fa503f22640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4fc123020 con 0x7fa4e000a000 2024-04-22T08:21:21.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.498+0000 7fa503f22640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:21.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.498+0000 7fa501cbe640 1 -- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4e00062d0 msgr2=0x7fa4fc1caa80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:21:21.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.498+0000 7fa503f22640 10 monclient(hunting): authenticate will time out at 1355.492857s 2024-04-22T08:21:21.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.498+0000 7fa501cbe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4e00062d0 0x7fa4fc1caa80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:21:21.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.498+0000 7fa5014bd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4e000a000 0x7fa4fc134ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:21.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.498+0000 7fa5014bd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4e000a000 0x7fa4fc134ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:42918/0 (socket says 172.21.15.40:42918) 2024-04-22T08:21:21.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.498+0000 7fa5014bd640 1 -- 172.21.15.40:0/3531563080 learned_addr learned my addr 172.21.15.40:0/3531563080 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:21:21.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.498+0000 7fa5014bd640 10 monclient(hunting): get_auth_request con 0x7fa4e000a000 auth_method 0 2024-04-22T08:21:21.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.498+0000 7fa5014bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:21.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.498+0000 7fa5014bd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:21.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.498+0000 7fa5014bd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:21.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.498+0000 7fa5014bd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:21.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.498+0000 7fa5014bd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:21.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.498+0000 7fa5024bf640 1 --2- 172.21.15.40:0/3531563080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4fc09ee10 0x7fa4fc135430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:21.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.498+0000 7fa5014bd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:21.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.498+0000 7fa5024bf640 10 monclient(hunting): get_auth_request con 0x7fa4fc09ee10 auth_method 0 2024-04-22T08:21:21.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.498+0000 7fa5024bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:21.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.498+0000 7fa5024bf640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:21.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.498+0000 7fa5024bf640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:21.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.498+0000 7fa5014bd640 10 monclient(hunting): handle_auth_done global_id 104180 payload 824 2024-04-22T08:21:21.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.498+0000 7fa5014bd640 1 -- 172.21.15.40:0/3531563080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4fc09ee10 msgr2=0x7fa4fc135430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:21.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.498+0000 7fa5014bd640 1 --2- 172.21.15.40:0/3531563080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4fc09ee10 0x7fa4fc135430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:21.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.498+0000 7fa5014bd640 1 -- 172.21.15.40:0/3531563080 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4e00062d0 msgr2=0x7fa4fc1caa80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:21:21.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.498+0000 7fa5014bd640 1 --2- 172.21.15.40:0/3531563080 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4e00062d0 0x7fa4fc1caa80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:21.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.498+0000 7fa5014bd640 10 monclient: _finish_hunting 0 2024-04-22T08:21:21.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.498+0000 7fa5014bd640 1 monclient: found mon.noname-c 2024-04-22T08:21:21.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.498+0000 7fa5014bd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:21.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.498+0000 7fa5014bd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:21:21.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.498+0000 7fa5014bd640 1 -- 172.21.15.40:0/3531563080 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4fc09f1f0 con 0x7fa4e000a000 2024-04-22T08:21:21.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.498+0000 7fa5024bf640 1 --2- 172.21.15.40:0/3531563080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4fc09ee10 0x7fa4fc135430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T08:21:21.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.500+0000 7fa5014bd640 1 -- 172.21.15.40:0/3531563080 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4e000a000 msgr2=0x7fa4fc134ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:21.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.501+0000 7fa5014bd640 1 --2- 172.21.15.40:0/3531563080 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4e000a000 0x7fa4fc134ef0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec0145f0 tx=0x7fa4ec014ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:21.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.501+0000 7fa4f27fc640 1 -- 172.21.15.40:0/3531563080 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ec012b10 con 0x7fa4e000a000 2024-04-22T08:21:21.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.501+0000 7fa4f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:21.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.501+0000 7fa4f27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:21:21.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.501+0000 7fa4f27fc640 10 monclient: dump: 2024-04-22T08:21:21.508 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:21.508 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:21.508 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:21.508 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:21.508 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:21.508 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:21.508 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:21.508 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:21.508 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:21.508 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:21.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.501+0000 7fa4f27fc640 10 monclient: _finish_auth 0 2024-04-22T08:21:21.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.501+0000 7fa4f27fc640 10 monclient: _check_auth_tickets 2024-04-22T08:21:21.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.501+0000 7fa4f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:21.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.501+0000 7fa4f27fc640 1 -- 172.21.15.40:0/3531563080 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4ec00c040 con 0x7fa4e000a000 2024-04-22T08:21:21.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.501+0000 7fa503f22640 5 monclient: authenticate success, global_id 104180 2024-04-22T08:21:21.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.501+0000 7fa503f22640 10 monclient: _renew_subs 2024-04-22T08:21:21.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.501+0000 7fa503f22640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:21:21.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.501+0000 7fa503f22640 1 -- 172.21.15.40:0/3531563080 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4fc123300 con 0x7fa4e000a000 2024-04-22T08:21:21.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.501+0000 7fa4f27fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:21.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.501+0000 7fa4f27fc640 1 -- 172.21.15.40:0/3531563080 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ec017470 con 0x7fa4e000a000 2024-04-22T08:21:21.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.501+0000 7fa4f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:21.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.501+0000 7fa4f27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:21:21.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.501+0000 7fa4f27fc640 10 monclient: dump: 2024-04-22T08:21:21.509 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:21.509 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:21.510 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:21.510 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:21.510 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:21.510 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:21.510 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:21.510 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:21.510 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:21.510 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:21.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.501+0000 7fa4f3fff640 4 set_mon_vals no callback set 2024-04-22T08:21:21.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.501+0000 7fa503f22640 10 monclient: _renew_subs 2024-04-22T08:21:21.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.501+0000 7fa503f22640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:21:21.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.501+0000 7fa503f22640 1 -- 172.21.15.40:0/3531563080 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4fc1cafc0 con 0x7fa4e000a000 2024-04-22T08:21:21.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.502+0000 7fa4f27fc640 1 -- 172.21.15.40:0/3531563080 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fa4ec017650 con 0x7fa4e000a000 2024-04-22T08:21:21.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.502+0000 7fa503f22640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:21.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.502+0000 7fa503f22640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:21.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.502+0000 7fa503f22640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:21:21.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.502+0000 7fa503f22640 1 -- 172.21.15.40:0/3531563080 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4fc123540 con 0x7fa4e000a000 2024-04-22T08:21:21.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.502+0000 7fa4f27fc640 1 --2- 172.21.15.40:0/3531563080 >> v2:172.21.15.189:6800/30168 conn(0x7fa4e0038a50 0x7fa4e003af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:21.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.502+0000 7fa4f27fc640 1 -- 172.21.15.40:0/3531563080 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(279..279 src has 239..279) v4 ==== 3506+0+0 (secure 0 0 0) 0x7fa4ec04a6d0 con 0x7fa4e000a000 2024-04-22T08:21:21.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.502+0000 7fa501cbe640 1 --2- 172.21.15.40:0/3531563080 >> v2:172.21.15.189:6800/30168 conn(0x7fa4e0038a50 0x7fa4e003af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:21.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.503+0000 7fa501cbe640 10 monclient: get_auth_request con 0x7fa4e0038a50 auth_method 0 2024-04-22T08:21:21.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.505+0000 7fa501cbe640 1 --2- 172.21.15.40:0/3531563080 >> v2:172.21.15.189:6800/30168 conn(0x7fa4e0038a50 0x7fa4e003af10 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fa4e00061e0 tx=0x7fa4f8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:21.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.507+0000 7fa4f27fc640 1 -- 172.21.15.40:0/3531563080 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7fa4fc123540 con 0x7fa4e000a000 2024-04-22T08:21:21.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.507+0000 7fa4f27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:21.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.507+0000 7fa4f27fc640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:21:21.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.760+0000 7fa503f22640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:21:21.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.760+0000 7fa503f22640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:21:21.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.760+0000 7fa503f22640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:21:21.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.760+0000 7fa503f22640 1 -- 172.21.15.40:0/3531563080 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa4fc123540 con 0x7fa4e000a000 2024-04-22T08:21:21.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.761+0000 7fa4f27fc640 1 -- 172.21.15.40:0/3531563080 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1537 (secure 0 0 0) 0x7fa4fc123540 con 0x7fa4e000a000 2024-04-22T08:21:21.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.761+0000 7fa4f27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:21:21.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.761+0000 7fa4f27fc640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:21:21.762 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":84,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:21:21.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.762+0000 7fa503f22640 1 -- 172.21.15.40:0/3531563080 >> v2:172.21.15.189:6800/30168 conn(0x7fa4e0038a50 msgr2=0x7fa4e003af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:21.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.762+0000 7fa503f22640 1 --2- 172.21.15.40:0/3531563080 >> v2:172.21.15.189:6800/30168 conn(0x7fa4e0038a50 0x7fa4e003af10 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fa4e00061e0 tx=0x7fa4f8008040 comp rx=0 tx=0).stop 2024-04-22T08:21:21.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.762+0000 7fa503f22640 10 monclient: shutdown 2024-04-22T08:21:21.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.762+0000 7fa503f22640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:21.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.762+0000 7fa503f22640 1 -- 172.21.15.40:0/3531563080 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4e000a000 msgr2=0x7fa4fc134ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:21.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.762+0000 7fa503f22640 1 --2- 172.21.15.40:0/3531563080 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4e000a000 0x7fa4fc134ef0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec0145f0 tx=0x7fa4ec014ab0 comp rx=0 tx=0).stop 2024-04-22T08:21:21.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.762+0000 7fa503f22640 1 -- 172.21.15.40:0/3531563080 shutdown_connections 2024-04-22T08:21:21.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.762+0000 7fa503f22640 1 --2- 172.21.15.40:0/3531563080 >> v2:172.21.15.189:6800/30168 conn(0x7fa4e0038a50 0x7fa4e003af10 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:21.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.762+0000 7fa503f22640 1 --2- 172.21.15.40:0/3531563080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4fc09ee10 0x7fa4fc135430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:21.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.762+0000 7fa503f22640 1 --2- 172.21.15.40:0/3531563080 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4e000a000 0x7fa4fc134ef0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:21.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.762+0000 7fa503f22640 1 --2- 172.21.15.40:0/3531563080 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4e00062d0 0x7fa4fc1caa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:21.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.762+0000 7fa503f22640 1 -- 172.21.15.40:0/3531563080 >> 172.21.15.40:0/3531563080 conn(0x7fa4fc0a4ef0 msgr2=0x7fa4fc0a61f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:21.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.763+0000 7fa503f22640 1 -- 172.21.15.40:0/3531563080 shutdown_connections 2024-04-22T08:21:21.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.763+0000 7fa503f22640 1 -- 172.21.15.40:0/3531563080 wait complete. 2024-04-22T08:21:21.774 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-22T08:21:21.774 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-04-22T08:21:21.774 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-22T08:21:21.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.873+0000 7fb8fc2e0640 10 monclient: get_monmap_and_config 2024-04-22T08:21:21.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.873+0000 7fb8fc2e0640 10 monclient: build_initial_monmap 2024-04-22T08:21:21.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.873+0000 7fb8fc2e0640 10 monclient: monmap: 2024-04-22T08:21:21.873 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:21.874 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:21.874 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:21:21.874805+0000 2024-04-22T08:21:21.874 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:21:21.874805+0000 2024-04-22T08:21:21.874 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:21.874 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:21.874 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:21:21.874 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:21:21.874 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:21:21.874 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:21:21.874 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:21:21.874 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:21.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.873+0000 7fb8fc2e0640 1 Processor -- start 2024-04-22T08:21:21.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.873+0000 7fb8fc2e0640 1 -- start start 2024-04-22T08:21:21.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.873+0000 7fb8fc2e0640 10 monclient: init 2024-04-22T08:21:21.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.874+0000 7fb8fc2e0640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:21.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.874+0000 7fb8fc2e0640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-04-22T08:21:21.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.874+0000 7fb8fc2e0640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8f40a1c60 0x7fb8f40a31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:21.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.874+0000 7fb8fc2e0640 10 monclient(hunting): picked mon.noname-b con 0x7fb8f40a1c60 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:21.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.874+0000 7fb8fc2e0640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb8f40a2040 0x7fb8f4135bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:21.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.874+0000 7fb8fc2e0640 10 monclient(hunting): picked mon.noname-a con 0x7fb8f40a2040 addr v2:172.21.15.40:3300/0 2024-04-22T08:21:21.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.874+0000 7fb8fc2e0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb8f40a24a0 addr v1:172.21.15.40:6789/0 2024-04-22T08:21:21.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.874+0000 7fb8fc2e0640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8f40a41f0 con 0x7fb8f40a24a0 2024-04-22T08:21:21.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.874+0000 7fb8fc2e0640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:21.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.874+0000 7fb8fc2e0640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8f409fe90 con 0x7fb8f40a2040 2024-04-22T08:21:21.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.874+0000 7fb8fc2e0640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:21.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.874+0000 7fb8fc2e0640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f40a0170 con 0x7fb8f40a1c60 2024-04-22T08:21:21.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.874+0000 7fb8fc2e0640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:21.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.874+0000 7fb8fc2e0640 10 monclient(hunting): authenticate will time out at 1355.869023s 2024-04-22T08:21:21.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.874+0000 7fb8fa87d640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7fb8f40a24a0 legacy=0x7fb8f41360f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-04-22T08:21:21.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.874+0000 7fb8f987b640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7fb8f40a2040 msgr2=0x7fb8f4135bb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:21:21.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.874+0000 7fb8f987b640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb8f40a2040 0x7fb8f4135bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:21:21.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.874+0000 7fb8fa07c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8f40a1c60 0x7fb8f40a31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:21.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.874+0000 7fb8fa07c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8f40a1c60 0x7fb8f40a31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:42926/0 (socket says 172.21.15.40:42926) 2024-04-22T08:21:21.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.874+0000 7fb8fa07c640 1 -- 172.21.15.40:0/513072103 learned_addr learned my addr 172.21.15.40:0/513072103 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:21:21.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.874+0000 7fb8fa07c640 10 monclient(hunting): get_auth_request con 0x7fb8f40a1c60 auth_method 0 2024-04-22T08:21:21.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.874+0000 7fb8fa07c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:21.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.874+0000 7fb8fa07c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:21.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.874+0000 7fb8fa07c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:21.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.874+0000 7fb8fa07c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:21.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.874+0000 7fb8fa07c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:21.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.874+0000 7fb8fa07c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:21.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.874+0000 7fb8fa07c640 10 monclient(hunting): handle_auth_done global_id 104183 payload 274 2024-04-22T08:21:21.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.874+0000 7fb8fa07c640 1 -- 172.21.15.40:0/513072103 >> v2:172.21.15.40:3300/0 conn(0x7fb8f40a2040 msgr2=0x7fb8f4135bb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:21:21.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.874+0000 7fb8fa07c640 1 --2- 172.21.15.40:0/513072103 >> v2:172.21.15.40:3300/0 conn(0x7fb8f40a2040 0x7fb8f4135bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:21.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.874+0000 7fb8fa07c640 1 -- 172.21.15.40:0/513072103 >> v1:172.21.15.40:6789/0 conn(0x7fb8f40a24a0 legacy=0x7fb8f41360f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:21:21.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.874+0000 7fb8fa07c640 10 monclient: _finish_hunting 0 2024-04-22T08:21:21.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.874+0000 7fb8fa07c640 1 monclient: found mon.noname-b 2024-04-22T08:21:21.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.874+0000 7fb8fa07c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:21.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.874+0000 7fb8fa07c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:21:21.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.874+0000 7fb8fa07c640 1 -- 172.21.15.40:0/513072103 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f40a0450 con 0x7fb8f40a1c60 2024-04-22T08:21:21.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.877+0000 7fb8fa07c640 1 -- 172.21.15.40:0/513072103 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8f40a1c60 msgr2=0x7fb8f40a31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:21.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.877+0000 7fb8fa07c640 1 --2- 172.21.15.40:0/513072103 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8f40a1c60 0x7fb8f40a31d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec009870 tx=0x7fb8ec02b7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d637ce107aaf47d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:21.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.877+0000 7fb8f907a640 1 -- 172.21.15.40:0/513072103 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ec031a40 con 0x7fb8f40a1c60 2024-04-22T08:21:21.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.877+0000 7fb8f907a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:21.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.877+0000 7fb8f907a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:21:21.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.877+0000 7fb8f907a640 10 monclient: dump: 2024-04-22T08:21:21.886 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:21.887 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:21.887 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:21.887 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:21.887 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:21.887 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:21.887 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:21.887 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:21.887 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:21.887 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:21.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.877+0000 7fb8f907a640 10 monclient: _finish_auth 0 2024-04-22T08:21:21.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.877+0000 7fb8f907a640 10 monclient: _check_auth_tickets 2024-04-22T08:21:21.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.877+0000 7fb8f907a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:21.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.877+0000 7fb8f907a640 1 -- 172.21.15.40:0/513072103 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8ec031ba0 con 0x7fb8f40a1c60 2024-04-22T08:21:21.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.877+0000 7fb8fc2e0640 5 monclient: authenticate success, global_id 104183 2024-04-22T08:21:21.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.877+0000 7fb8fc2e0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:21:21.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.877+0000 7fb8f907a640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:21.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.877+0000 7fb8fc2e0640 10 monclient: get_monmap_and_config success 2024-04-22T08:21:21.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.877+0000 7fb8f907a640 1 -- 172.21.15.40:0/513072103 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ec033440 con 0x7fb8f40a1c60 2024-04-22T08:21:21.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.877+0000 7fb8f907a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:21.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.877+0000 7fb8fc2e0640 4 set_mon_vals no callback set 2024-04-22T08:21:21.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.877+0000 7fb8f907a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:21:21.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.877+0000 7fb8f907a640 10 monclient: dump: 2024-04-22T08:21:21.888 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:21.888 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:21.888 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:21.888 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:21.888 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:21.888 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:21.888 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:21.888 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:21.889 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:21.889 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:21.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.878+0000 7fb8fc2e0640 10 monclient: shutdown 2024-04-22T08:21:21.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.878+0000 7fb8fc2e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:21.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.878+0000 7fb8fc2e0640 1 -- 172.21.15.40:0/513072103 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8f40a1c60 msgr2=0x7fb8f40a31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:21.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.878+0000 7fb8fc2e0640 1 --2- 172.21.15.40:0/513072103 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8f40a1c60 0x7fb8f40a31d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec009870 tx=0x7fb8ec02b7f0 comp rx=0 tx=0).stop 2024-04-22T08:21:21.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.878+0000 7fb8fc2e0640 1 -- 172.21.15.40:0/513072103 shutdown_connections 2024-04-22T08:21:21.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.878+0000 7fb8fc2e0640 1 --2- 172.21.15.40:0/513072103 >> v2:172.21.15.40:3300/0 conn(0x7fb8f40a2040 0x7fb8f4135bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:21.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.878+0000 7fb8fc2e0640 1 --2- 172.21.15.40:0/513072103 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8f40a1c60 0x7fb8f40a31d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:21.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.878+0000 7fb8fc2e0640 1 -- 172.21.15.40:0/513072103 >> 172.21.15.40:0/513072103 conn(0x7fb8f40a89a0 msgr2=0x7fb8f40a4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:21.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.878+0000 7fb8fc2e0640 1 -- 172.21.15.40:0/513072103 shutdown_connections 2024-04-22T08:21:21.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.878+0000 7fb8fc2e0640 1 -- 172.21.15.40:0/513072103 wait complete. 2024-04-22T08:21:21.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.878+0000 7fb8fc2e0640 10 monclient: build_initial_monmap 2024-04-22T08:21:21.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.878+0000 7fb8fc2e0640 10 monclient: monmap: 2024-04-22T08:21:21.889 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:21.890 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:21:21.890 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:21:21.890 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:21:21.890 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:21.890 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:21.890 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:21:21.890 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:21:21.890 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:21:21.890 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:21.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.879+0000 7fb8fc2e0640 1 Processor -- start 2024-04-22T08:21:21.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.879+0000 7fb8fc2e0640 1 -- start start 2024-04-22T08:21:21.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.879+0000 7fb8fc2e0640 10 monclient: init 2024-04-22T08:21:21.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.879+0000 7fb8fc2e0640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:21.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.879+0000 7fb8fc2e0640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-22T08:21:21.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.879+0000 7fb8fc2e0640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8f40a1c60 0x7fb8f41cad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:21.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.879+0000 7fb8fc2e0640 10 monclient(hunting): picked mon.noname-b con 0x7fb8f40a1c60 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:21:21.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.879+0000 7fb8fc2e0640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8f40a2040 0x7fb8f41cb250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:21.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.879+0000 7fb8fc2e0640 10 monclient(hunting): picked mon.noname-c con 0x7fb8f40a2040 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:21.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.879+0000 7fb8fc2e0640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8f40a24a0 0x7fb8f4122250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:21.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.879+0000 7fb8fc2e0640 10 monclient(hunting): picked mon.noname-a con 0x7fb8f40a24a0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:21:21.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.879+0000 7fb8fc2e0640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:21.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.879+0000 7fb8fc2e0640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f40a4800 con 0x7fb8f40a24a0 2024-04-22T08:21:21.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.879+0000 7fb8fc2e0640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:21.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.879+0000 7fb8fc2e0640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f41352d0 con 0x7fb8f40a1c60 2024-04-22T08:21:21.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.879+0000 7fb8fc2e0640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:21.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.879+0000 7fb8fc2e0640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f4070c90 con 0x7fb8f40a2040 2024-04-22T08:21:21.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.879+0000 7fb8fc2e0640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:21.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.879+0000 7fb8fa87d640 1 -- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8f40a24a0 msgr2=0x7fb8f4122250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:21:21.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.879+0000 7fb8fc2e0640 10 monclient(hunting): authenticate will time out at 1355.874654s 2024-04-22T08:21:21.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.879+0000 7fb8fa87d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8f40a24a0 0x7fb8f4122250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:21:21.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.879+0000 7fb8f987b640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8f40a2040 0x7fb8f41cb250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:21.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.879+0000 7fb8f987b640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8f40a2040 0x7fb8f41cb250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:42942/0 (socket says 172.21.15.40:42942) 2024-04-22T08:21:21.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.880+0000 7fb8f987b640 1 -- 172.21.15.40:0/1876318925 learned_addr learned my addr 172.21.15.40:0/1876318925 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:21:21.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.880+0000 7fb8f987b640 10 monclient(hunting): get_auth_request con 0x7fb8f40a2040 auth_method 0 2024-04-22T08:21:21.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.880+0000 7fb8f987b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:21.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.880+0000 7fb8f987b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:21.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.880+0000 7fb8f987b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:21.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.880+0000 7fb8f987b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:21.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.880+0000 7fb8f987b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:21.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.880+0000 7fb8f987b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:21.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.880+0000 7fb8fa07c640 1 --2- 172.21.15.40:0/1876318925 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8f40a1c60 0x7fb8f41cad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:21.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.880+0000 7fb8fa07c640 10 monclient(hunting): get_auth_request con 0x7fb8f40a1c60 auth_method 0 2024-04-22T08:21:21.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.880+0000 7fb8fa07c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:21.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.880+0000 7fb8fa07c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:21.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.880+0000 7fb8fa07c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:21.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.880+0000 7fb8f987b640 10 monclient(hunting): handle_auth_done global_id 104186 payload 824 2024-04-22T08:21:21.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.880+0000 7fb8f987b640 1 -- 172.21.15.40:0/1876318925 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8f40a1c60 msgr2=0x7fb8f41cad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:21.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.880+0000 7fb8f987b640 1 --2- 172.21.15.40:0/1876318925 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8f40a1c60 0x7fb8f41cad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:21.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.880+0000 7fb8f987b640 1 -- 172.21.15.40:0/1876318925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8f40a24a0 msgr2=0x7fb8f4122250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:21:21.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.880+0000 7fb8f987b640 1 --2- 172.21.15.40:0/1876318925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8f40a24a0 0x7fb8f4122250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:21.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.880+0000 7fb8f987b640 10 monclient: _finish_hunting 0 2024-04-22T08:21:21.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.880+0000 7fb8f987b640 1 monclient: found mon.noname-c 2024-04-22T08:21:21.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.880+0000 7fb8f987b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:21.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.880+0000 7fb8f987b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:21:21.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.880+0000 7fb8f987b640 1 -- 172.21.15.40:0/1876318925 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8ec0094d0 con 0x7fb8f40a2040 2024-04-22T08:21:21.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.882+0000 7fb8f987b640 1 -- 172.21.15.40:0/1876318925 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8f40a2040 msgr2=0x7fb8f41cb250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:21.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.882+0000 7fb8f987b640 1 --2- 172.21.15.40:0/1876318925 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8f40a2040 0x7fb8f41cb250 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb8e400b550 tx=0x7fb8e400ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:21.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.882+0000 7fb8eaffd640 1 -- 172.21.15.40:0/1876318925 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e4005070 con 0x7fb8f40a2040 2024-04-22T08:21:21.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.882+0000 7fb8eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:21.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.882+0000 7fb8eaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:21:21.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.882+0000 7fb8eaffd640 10 monclient: dump: 2024-04-22T08:21:21.894 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:21.894 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:21.894 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:21.894 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:21.894 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:21.894 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:21.894 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:21.894 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:21.894 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:21.895 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:21.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.882+0000 7fb8eaffd640 10 monclient: _finish_auth 0 2024-04-22T08:21:21.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.882+0000 7fb8eaffd640 10 monclient: _check_auth_tickets 2024-04-22T08:21:21.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.882+0000 7fb8eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:21.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.882+0000 7fb8eaffd640 1 -- 172.21.15.40:0/1876318925 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8e40051f0 con 0x7fb8f40a2040 2024-04-22T08:21:21.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.882+0000 7fb8fc2e0640 5 monclient: authenticate success, global_id 104186 2024-04-22T08:21:21.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.882+0000 7fb8fc2e0640 10 monclient: _renew_subs 2024-04-22T08:21:21.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.882+0000 7fb8fc2e0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:21:21.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.882+0000 7fb8fc2e0640 1 -- 172.21.15.40:0/1876318925 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8f4124900 con 0x7fb8f40a2040 2024-04-22T08:21:21.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.882+0000 7fb8eaffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:21.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.882+0000 7fb8eaffd640 1 -- 172.21.15.40:0/1876318925 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e40055a0 con 0x7fb8f40a2040 2024-04-22T08:21:21.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.882+0000 7fb8eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:21.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.882+0000 7fb8eaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:21:21.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.882+0000 7fb8eaffd640 10 monclient: dump: 2024-04-22T08:21:21.896 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:21.896 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:21.896 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:21.896 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:21.896 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:21.896 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:21.896 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:21.896 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:21.896 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:21.896 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:21.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.882+0000 7fb8f8879640 4 set_mon_vals no callback set 2024-04-22T08:21:21.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.882+0000 7fb8fc2e0640 10 monclient: _renew_subs 2024-04-22T08:21:21.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.882+0000 7fb8fc2e0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:21:21.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.882+0000 7fb8fc2e0640 1 -- 172.21.15.40:0/1876318925 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8f4125860 con 0x7fb8f40a2040 2024-04-22T08:21:21.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.883+0000 7fb8fc2e0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:21.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.883+0000 7fb8fc2e0640 1 --2- 172.21.15.40:0/1876318925 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8f4122790 0x7fb8f4122b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:21.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.883+0000 7fb8fc2e0640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:21.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.883+0000 7fb8fc2e0640 1 -- 172.21.15.40:0/1876318925 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f4066650 con 0x7fb8f4122790 2024-04-22T08:21:21.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.883+0000 7fb8eaffd640 1 -- 172.21.15.40:0/1876318925 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fb8e4010480 con 0x7fb8f40a2040 2024-04-22T08:21:21.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.883+0000 7fb8fa07c640 1 --2- 172.21.15.40:0/1876318925 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8f4122790 0x7fb8f4122b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:21.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.883+0000 7fb8fa07c640 10 monclient: get_auth_request con 0x7fb8f4122790 auth_method 0 2024-04-22T08:21:21.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.883+0000 7fb8fa07c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:21.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.883+0000 7fb8fa07c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:21.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.883+0000 7fb8fa07c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:21.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.883+0000 7fb8fa07c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:21.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.883+0000 7fb8fa07c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:21.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.883+0000 7fb8fa07c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:21.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.883+0000 7fb8eaffd640 1 --2- 172.21.15.40:0/1876318925 >> v2:172.21.15.189:6800/30168 conn(0x7fb8d80377d0 0x7fb8d8039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:21.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.883+0000 7fb8eaffd640 1 -- 172.21.15.40:0/1876318925 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(279..279 src has 239..279) v4 ==== 3506+0+0 (secure 0 0 0) 0x7fb8e404a100 con 0x7fb8f40a2040 2024-04-22T08:21:21.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.883+0000 7fb8fa07c640 10 monclient(hunting): handle_auth_done global_id 104189 payload 824 2024-04-22T08:21:21.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.883+0000 7fb8fa07c640 1 -- 172.21.15.40:0/1876318925 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb8f4136800 con 0x7fb8f4122790 2024-04-22T08:21:21.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.884+0000 7fb8fa87d640 1 --2- 172.21.15.40:0/1876318925 >> v2:172.21.15.189:6800/30168 conn(0x7fb8d80377d0 0x7fb8d8039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:21.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.884+0000 7fb8fa87d640 10 monclient: get_auth_request con 0x7fb8d80377d0 auth_method 0 2024-04-22T08:21:21.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.886+0000 7fb8fa07c640 1 -- 172.21.15.40:0/1876318925 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8f4122790 msgr2=0x7fb8f4122b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:21.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.886+0000 7fb8fa07c640 1 --2- 172.21.15.40:0/1876318925 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8f4122790 0x7fb8f4122b70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4071b30 tx=0x7fb8ec0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:21.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.886+0000 7fb8eaffd640 1 -- 172.21.15.40:0/1876318925 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ec03d640 con 0x7fb8f4122790 2024-04-22T08:21:21.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.886+0000 7fb8eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:21.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.886+0000 7fb8eaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:21:21.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.886+0000 7fb8eaffd640 10 monclient: dump: 2024-04-22T08:21:21.898 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:21.898 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:21.898 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:21.898 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:21.899 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:21.899 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:21.899 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:21.899 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:21.899 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:21.899 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:21.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.886+0000 7fb8eaffd640 1 -- 172.21.15.40:0/1876318925 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7fb8ec041070 con 0x7fb8f4122790 2024-04-22T08:21:21.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.886+0000 7fb8eaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:21.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.886+0000 7fb8eaffd640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:21:21.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.886+0000 7fb8eaffd640 1 -- 172.21.15.40:0/1876318925 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8f4122790 msgr2=0x7fb8f4122b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:21.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.886+0000 7fb8eaffd640 1 --2- 172.21.15.40:0/1876318925 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8f4122790 0x7fb8f4122b70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4071b30 tx=0x7fb8ec0046c0 comp rx=0 tx=0).stop 2024-04-22T08:21:21.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.886+0000 7fb8eaffd640 1 -- 172.21.15.40:0/1876318925 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8f4122790 msgr2=0x7fb8f4122b70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:21:21.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.886+0000 7fb8eaffd640 1 --2- 172.21.15.40:0/1876318925 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8f4122790 0x7fb8f4122b70 secure :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4071b30 tx=0x7fb8ec0046c0 comp rx=0 tx=0).stop 2024-04-22T08:21:21.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.886+0000 7fb8fa87d640 1 --2- 172.21.15.40:0/1876318925 >> v2:172.21.15.189:6800/30168 conn(0x7fb8d80377d0 0x7fb8d8039c90 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fb8f0003d00 tx=0x7fb8f0009270 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:21.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.892+0000 7fb8fc2e0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:21:21.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.892+0000 7fb8fc2e0640 1 --2- 172.21.15.40:0/1876318925 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8f41230b0 0x7fb8f411e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:21.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.892+0000 7fb8fc2e0640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:21.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.892+0000 7fb8fc2e0640 1 -- 172.21.15.40:0/1876318925 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f404e9b0 con 0x7fb8f41230b0 2024-04-22T08:21:21.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.892+0000 7fb8fa07c640 1 --2- 172.21.15.40:0/1876318925 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8f41230b0 0x7fb8f411e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:21.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.892+0000 7fb8fa07c640 10 monclient: get_auth_request con 0x7fb8f41230b0 auth_method 0 2024-04-22T08:21:21.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.892+0000 7fb8fa07c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:21.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.892+0000 7fb8fa07c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:21.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.892+0000 7fb8fa07c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:21.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.893+0000 7fb8fa07c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:21.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.893+0000 7fb8fa07c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:21.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.893+0000 7fb8fa07c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:21.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.893+0000 7fb8fa07c640 10 monclient(hunting): handle_auth_done global_id 104192 payload 824 2024-04-22T08:21:21.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.893+0000 7fb8fa07c640 1 -- 172.21.15.40:0/1876318925 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb8f41254f0 con 0x7fb8f41230b0 2024-04-22T08:21:21.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.895+0000 7fb8fa07c640 1 -- 172.21.15.40:0/1876318925 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8f41230b0 msgr2=0x7fb8f411e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:21.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.895+0000 7fb8fa07c640 1 --2- 172.21.15.40:0/1876318925 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8f41230b0 0x7fb8f411e960 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec02b830 tx=0x7fb8ec031000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:21.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.895+0000 7fb8eaffd640 1 -- 172.21.15.40:0/1876318925 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ec03de90 con 0x7fb8f41230b0 2024-04-22T08:21:21.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.895+0000 7fb8eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:21.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.895+0000 7fb8eaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:21:21.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.895+0000 7fb8eaffd640 10 monclient: dump: 2024-04-22T08:21:21.903 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:21.903 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:21.903 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:21.903 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:21.903 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:21.903 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:21.903 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:21.903 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:21.903 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:21.903 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:21.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.895+0000 7fb8eaffd640 1 -- 172.21.15.40:0/1876318925 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4620 (secure 0 0 0) 0x7fb8ec033e70 con 0x7fb8f41230b0 2024-04-22T08:21:21.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.895+0000 7fb8eaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:21:21.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.895+0000 7fb8eaffd640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:21:21.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.895+0000 7fb8eaffd640 1 -- 172.21.15.40:0/1876318925 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8f41230b0 msgr2=0x7fb8f411e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:21.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.895+0000 7fb8eaffd640 1 --2- 172.21.15.40:0/1876318925 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8f41230b0 0x7fb8f411e960 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec02b830 tx=0x7fb8ec031000 comp rx=0 tx=0).stop 2024-04-22T08:21:21.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.895+0000 7fb8eaffd640 1 -- 172.21.15.40:0/1876318925 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8f41230b0 msgr2=0x7fb8f411e960 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:21:21.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.895+0000 7fb8eaffd640 1 --2- 172.21.15.40:0/1876318925 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8f41230b0 0x7fb8f411e960 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec02b830 tx=0x7fb8ec031000 comp rx=0 tx=0).stop 2024-04-22T08:21:21.904 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:21:21.904 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:21:21.904 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:21:21.904 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:21:21.905 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 84, 2024-04-22T08:21:21.905 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:21:21.905 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:21:21.905 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:21:21.905 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:21.905 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 2, 2024-04-22T08:21:21.905 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:21:21.905 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:21:21.905 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:21:21.905 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:21:21.905 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:21:21.905 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:21:21.905 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:21:21.905 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:21:21.906 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:21:21.906 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:21:21.906 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:21:21.906 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:21:21.906 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:21:21.906 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:21.906 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:21:21.906 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:21:21.906 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:21:21.906 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:21:21.906 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:21:21.906 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:21:21.906 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:21:21.907 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:21:21.907 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:21:21.907 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:21:21.907 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:21:21.907 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:21:21.907 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:21.907 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:21.907 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:21:21.907 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:21:21.907 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:21:21.907 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:21:21.907 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:21:21.907 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:21:21.908 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:21:21.908 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:21:21.908 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:21:21.908 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:21:21.908 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:21:21.908 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:21:21.908 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:21:21.908 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:21:21.908 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:21:21.908 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:21:21.908 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:21:21.908 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:21:21.908 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:21:21.908 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:21:21.909 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:21:21.909 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:21:21.909 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:21:21.909 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:21:21.909 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:21:21.909 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:21:21.909 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:21:21.909 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:21.909 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:21:21.909 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:21.909 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:21:21.909 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:21.909 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:21:21.909 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:21:21.909 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:21:21.910 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:21:21.910 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:21.910 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:21:21.910 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:21:21.910 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:21.910 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:21.910 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:21.910 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:21:21.910 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:21:21.910 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:21.910 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:21.910 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:21.910 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:21.910 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:21:21.911 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:21:21.911 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:21:21.911 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:21:21.911 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:21:21.911 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:21.911 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:21.911 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:21:21.911 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:21:21.911 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:21:21.911 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:21:21.911 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:21.911 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:21:21.911 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:21:21.911 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:21.912 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:21.912 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:21.912 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:21:21.912 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:21:21.912 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:21.912 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:21.912 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:21.912 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:21.912 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:21:21.912 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:21:21.912 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:21:21.912 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:21:21.912 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:21:21.912 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:21.913 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:21.913 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:21:21.913 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:21:21.913 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:21:21.913 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:21:21.913 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:21.913 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:21:21.913 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:21:21.913 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:21.913 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:21.913 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:21.913 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:21:21.913 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:21:21.913 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:21.914 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:21.914 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:21.914 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:21.914 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:21:21.914 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:21:21.914 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:21:21.914 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:21:21.914 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:21:21.914 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:21.914 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:21.914 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:21.914 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:21:21.914 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:21:21.915 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:21.915 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:21:21.915 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:21:21.915 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:21:21.915 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:21.915 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:21.915 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:21:21.915 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:21.915 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:21:21.915 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:21:21.915 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:21:21.915 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:21.915 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:21.915 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:21:21.915 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:21.916 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:21:21.916 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:21:21.916 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 3 2024-04-22T08:21:21.916 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:21.916 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:21.916 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:21.916 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:21:21.916 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:21:21.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.896+0000 7fb8fc2e0640 1 -- 172.21.15.40:0/1876318925 >> v2:172.21.15.189:6800/30168 conn(0x7fb8d80377d0 msgr2=0x7fb8d8039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:21.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.896+0000 7fb8fc2e0640 1 --2- 172.21.15.40:0/1876318925 >> v2:172.21.15.189:6800/30168 conn(0x7fb8d80377d0 0x7fb8d8039c90 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fb8f0003d00 tx=0x7fb8f0009270 comp rx=0 tx=0).stop 2024-04-22T08:21:21.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.896+0000 7fb8fc2e0640 10 monclient: shutdown 2024-04-22T08:21:21.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.896+0000 7fb8fc2e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:21.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.896+0000 7fb8fc2e0640 1 -- 172.21.15.40:0/1876318925 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8f40a2040 msgr2=0x7fb8f41cb250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:21.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.896+0000 7fb8fc2e0640 1 --2- 172.21.15.40:0/1876318925 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8f40a2040 0x7fb8f41cb250 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb8e400b550 tx=0x7fb8e400ba10 comp rx=0 tx=0).stop 2024-04-22T08:21:21.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.896+0000 7fb8fa87d640 1 -- 172.21.15.40:0/1876318925 reap_dead start 2024-04-22T08:21:21.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.896+0000 7fb8fa87d640 1 -- 172.21.15.40:0/1876318925 reap_dead start 2024-04-22T08:21:21.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.896+0000 7fb8fc2e0640 1 -- 172.21.15.40:0/1876318925 shutdown_connections 2024-04-22T08:21:21.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.896+0000 7fb8fc2e0640 1 -- 172.21.15.40:0/1876318925 >> 172.21.15.40:0/1876318925 conn(0x7fb8f40a89a0 msgr2=0x7fb8f4070ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:21.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.896+0000 7fb8fc2e0640 1 -- 172.21.15.40:0/1876318925 shutdown_connections 2024-04-22T08:21:21.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:21.896+0000 7fb8fc2e0640 1 -- 172.21.15.40:0/1876318925 wait complete. 2024-04-22T08:21:21.918 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-22T08:21:21.919 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-04-22T08:21:21.919 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-22T08:21:22.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.018+0000 7f91135d0640 10 monclient: get_monmap_and_config 2024-04-22T08:21:22.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.018+0000 7f91135d0640 10 monclient: build_initial_monmap 2024-04-22T08:21:22.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.018+0000 7f91135d0640 10 monclient: monmap: 2024-04-22T08:21:22.019 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:22.019 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:22.019 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:21:22.020262+0000 2024-04-22T08:21:22.019 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:21:22.020262+0000 2024-04-22T08:21:22.019 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:22.019 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:22.019 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:21:22.019 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:21:22.020 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:21:22.020 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:21:22.020 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:21:22.020 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:22.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.019+0000 7f91135d0640 1 Processor -- start 2024-04-22T08:21:22.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.019+0000 7f91135d0640 1 -- start start 2024-04-22T08:21:22.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.019+0000 7f91135d0640 10 monclient: init 2024-04-22T08:21:22.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.019+0000 7f91135d0640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:22.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.019+0000 7f91135d0640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-04-22T08:21:22.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.019+0000 7f91135d0640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f910c135fe0 0x7f910c0a01c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:22.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.019+0000 7f91135d0640 10 monclient(hunting): picked mon.noname-a con 0x7f910c135fe0 addr v2:172.21.15.40:3300/0 2024-04-22T08:21:22.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.019+0000 7f91135d0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f910c1363c0 addr v1:172.21.15.40:6789/0 2024-04-22T08:21:22.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.019+0000 7f91135d0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f910c1367d0 addr v1:172.21.15.189:6789/0 2024-04-22T08:21:22.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.019+0000 7f91135d0640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f910c0a28a0 con 0x7f910c1363c0 2024-04-22T08:21:22.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.019+0000 7f91135d0640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f910c09d7f0 con 0x7f910c1367d0 2024-04-22T08:21:22.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.019+0000 7f911136c640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f910c135fe0 msgr2=0x7f910c0a01c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:21:22.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.019+0000 7f91135d0640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:22.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.019+0000 7f91135d0640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f910c09f330 con 0x7f910c135fe0 2024-04-22T08:21:22.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.019+0000 7f91135d0640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:22.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.019+0000 7f91135d0640 10 monclient(hunting): authenticate will time out at 1356.014590s 2024-04-22T08:21:22.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.019+0000 7f911136c640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f910c135fe0 0x7f910c0a01c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:21:22.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.019+0000 7f9110b6b640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f910c1363c0 legacy=0x7f910c0a1a80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-04-22T08:21:22.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.020+0000 7f9111b6d640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f910c1367d0 0x7f910c0a2190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.40:54454/0 (socket says 172.21.15.40:54454) 2024-04-22T08:21:22.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.020+0000 7f9111b6d640 1 -- 172.21.15.40:0/2812821341 learned_addr learned my addr 172.21.15.40:0/2812821341 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:21:22.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.022+0000 7f9111b6d640 1 -- 172.21.15.40:0/2812821341 >> v1:172.21.15.189:6789/0 conn(0x7f910c1367d0 legacy=0x7f910c0a2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:22.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.022+0000 7f910bfff640 1 -- 172.21.15.40:0/2812821341 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1271645544 0 0) 0x7f910c09d7f0 con 0x7f910c1367d0 2024-04-22T08:21:22.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.022+0000 7f910bfff640 10 monclient(hunting): _negotiate 2024-04-22T08:21:22.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.022+0000 7f910bfff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:22.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.022+0000 7f910bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:22.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.022+0000 7f910bfff640 10 monclient(hunting): my global_id is 64993 2024-04-22T08:21:22.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.022+0000 7f910bfff640 1 -- 172.21.15.40:0/2812821341 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90f4003520 con 0x7f910c1367d0 2024-04-22T08:21:22.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f910bfff640 1 -- 172.21.15.40:0/2812821341 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1840900909 0 0) 0x7f90f4003520 con 0x7f910c1367d0 2024-04-22T08:21:22.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f910bfff640 1 -- 172.21.15.40:0/2812821341 >> v2:172.21.15.40:3300/0 conn(0x7f910c135fe0 msgr2=0x7f910c0a01c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:21:22.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f910bfff640 1 --2- 172.21.15.40:0/2812821341 >> v2:172.21.15.40:3300/0 conn(0x7f910c135fe0 0x7f910c0a01c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:22.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f910bfff640 1 -- 172.21.15.40:0/2812821341 >> v1:172.21.15.40:6789/0 conn(0x7f910c1363c0 legacy=0x7f910c0a1a80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:21:22.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f910bfff640 10 monclient: _finish_hunting 0 2024-04-22T08:21:22.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f910bfff640 1 monclient: found mon.noname-c-legacy 2024-04-22T08:21:22.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f910bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:22.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f910bfff640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.189:6789/0 2024-04-22T08:21:22.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f910bfff640 1 -- 172.21.15.40:0/2812821341 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f910c0999f0 con 0x7f910c1367d0 2024-04-22T08:21:22.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f910bfff640 10 monclient: _finish_auth 0 2024-04-22T08:21:22.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f910bfff640 10 monclient: _check_auth_tickets 2024-04-22T08:21:22.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f910bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:22.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f910bfff640 1 -- 172.21.15.40:0/2812821341 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1087006497 0 0) 0x7f9104001920 con 0x7f910c1367d0 2024-04-22T08:21:22.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f91135d0640 5 monclient: authenticate success, global_id 64993 2024-04-22T08:21:22.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f91135d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:21:22.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f910bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:22.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f910bfff640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-04-22T08:21:22.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f910bfff640 10 monclient: dump: 2024-04-22T08:21:22.030 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:22.030 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:22.030 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:22.030 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:22.030 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:22.030 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:22.030 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:22.030 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:22.030 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:22.031 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:22.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f910bfff640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] but i'm connected to v1:172.21.15.189:6789/0, reconnecting 2024-04-22T08:21:22.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f910bfff640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:22.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f910bfff640 1 -- 172.21.15.40:0/2812821341 >> v1:172.21.15.189:6789/0 conn(0x7f910c1367d0 legacy=0x7f910c0a2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:22.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f910bfff640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:21:22.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f910bfff640 1 --2- 172.21.15.40:0/2812821341 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90f4003970 0x7f90f4003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:22.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f910bfff640 10 monclient(hunting): picked mon.a con 0x7f90f4003970 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:21:22.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f910bfff640 1 --2- 172.21.15.40:0/2812821341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90f40047c0 0x7f90f4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:22.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f910bfff640 10 monclient(hunting): picked mon.b con 0x7f90f40047c0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:21:22.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f910bfff640 1 --2- 172.21.15.40:0/2812821341 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f90f4005330 0x7f90f400b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:22.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f910bfff640 10 monclient(hunting): picked mon.c con 0x7f90f4005330 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:22.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f910bfff640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:22.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f910bfff640 1 -- 172.21.15.40:0/2812821341 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f910c09f330 con 0x7f90f4003970 2024-04-22T08:21:22.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f910bfff640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:22.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f910bfff640 1 -- 172.21.15.40:0/2812821341 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f400c280 con 0x7f90f40047c0 2024-04-22T08:21:22.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f910bfff640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:22.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f910bfff640 1 -- 172.21.15.40:0/2812821341 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90f400c440 con 0x7f90f4005330 2024-04-22T08:21:22.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f910bfff640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:22.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f911136c640 1 -- 172.21.15.40:0/2812821341 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90f4003970 msgr2=0x7f90f4003d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:21:22.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f910bfff640 10 monclient(hunting): _finish_auth 0 2024-04-22T08:21:22.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f91135d0640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-04-22T08:21:22.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f911136c640 1 --2- 172.21.15.40:0/2812821341 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90f4003970 0x7f90f4003d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:21:22.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f9111b6d640 1 --2- 172.21.15.40:0/2812821341 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f90f4005330 0x7f90f400b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:22.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f9111b6d640 10 monclient(hunting): get_auth_request con 0x7f90f4005330 auth_method 0 2024-04-22T08:21:22.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f9111b6d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:22.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f9111b6d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:22.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f9111b6d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:21:22.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f9110b6b640 1 --2- 172.21.15.40:0/2812821341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90f40047c0 0x7f90f4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:22.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f9110b6b640 10 monclient(hunting): get_auth_request con 0x7f90f40047c0 auth_method 0 2024-04-22T08:21:22.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f9110b6b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:22.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f9110b6b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:22.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f9110b6b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:21:22.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f9111b6d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:22.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f9111b6d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:22.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f9111b6d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-22T08:21:22.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f9110b6b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:22.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f9110b6b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:22.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f9110b6b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-22T08:21:22.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f9111b6d640 10 monclient(hunting): handle_auth_done global_id 64993 payload 293 2024-04-22T08:21:22.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f9111b6d640 1 -- 172.21.15.40:0/2812821341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90f40047c0 msgr2=0x7f90f4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:21:22.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f9111b6d640 1 --2- 172.21.15.40:0/2812821341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90f40047c0 0x7f90f4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:22.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f9111b6d640 1 -- 172.21.15.40:0/2812821341 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90f4003970 msgr2=0x7f90f4003d70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:21:22.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f9111b6d640 1 --2- 172.21.15.40:0/2812821341 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90f4003970 0x7f90f4003d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:22.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f9111b6d640 10 monclient: _finish_hunting 0 2024-04-22T08:21:22.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f9111b6d640 1 monclient: found mon.c 2024-04-22T08:21:22.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f9111b6d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:22.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f9111b6d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:21:22.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f9111b6d640 1 -- 172.21.15.40:0/2812821341 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90f4003520 con 0x7f90f4005330 2024-04-22T08:21:22.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f9111b6d640 10 monclient: _finish_auth 0 2024-04-22T08:21:22.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f9111b6d640 10 monclient: _check_auth_tickets 2024-04-22T08:21:22.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.023+0000 7f9111b6d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:22.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.024+0000 7f9111b6d640 1 -- 172.21.15.40:0/2812821341 reap_dead start 2024-04-22T08:21:22.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.026+0000 7f9111b6d640 1 -- 172.21.15.40:0/2812821341 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f90f4005330 msgr2=0x7f90f400b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:22.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.026+0000 7f9111b6d640 1 --2- 172.21.15.40:0/2812821341 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f90f4005330 0x7f90f400b9a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9104001490 tx=0x7f90f40057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3863ac54ce000cb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:22.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.026+0000 7f910bfff640 1 -- 172.21.15.40:0/2812821341 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f910402cb60 con 0x7f90f4005330 2024-04-22T08:21:22.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.026+0000 7f910bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:22.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.026+0000 7f910bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:21:22.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.026+0000 7f910bfff640 10 monclient: dump: 2024-04-22T08:21:22.034 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:22.034 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:22.035 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:22.035 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:22.035 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:22.035 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:22.035 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:22.035 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:22.035 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:22.035 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:22.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.026+0000 7f910bfff640 1 -- 172.21.15.40:0/2812821341 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f910402cd00 con 0x7f90f4005330 2024-04-22T08:21:22.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.026+0000 7f91135d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:21:22.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.026+0000 7f910bfff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:22.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.026+0000 7f910bfff640 1 -- 172.21.15.40:0/2812821341 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9104005b10 con 0x7f90f4005330 2024-04-22T08:21:22.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.026+0000 7f91135d0640 10 monclient: get_monmap_and_config success 2024-04-22T08:21:22.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.026+0000 7f91135d0640 4 set_mon_vals no callback set 2024-04-22T08:21:22.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.026+0000 7f910bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:22.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.026+0000 7f910bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:21:22.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.026+0000 7f910bfff640 10 monclient: dump: 2024-04-22T08:21:22.036 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:22.036 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:22.036 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:22.036 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:22.036 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:22.036 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:22.036 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:22.036 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:22.036 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:22.036 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:22.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.027+0000 7f91135d0640 10 monclient: shutdown 2024-04-22T08:21:22.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.027+0000 7f91135d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:22.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.027+0000 7f91135d0640 1 -- 172.21.15.40:0/2812821341 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f90f4005330 msgr2=0x7f90f400b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:22.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.027+0000 7f91135d0640 1 --2- 172.21.15.40:0/2812821341 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f90f4005330 0x7f90f400b9a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9104001490 tx=0x7f90f40057f0 comp rx=0 tx=0).stop 2024-04-22T08:21:22.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.027+0000 7f91135d0640 1 -- 172.21.15.40:0/2812821341 shutdown_connections 2024-04-22T08:21:22.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.027+0000 7f91135d0640 1 --2- 172.21.15.40:0/2812821341 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f90f4005330 0x7f90f400b9a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:22.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.027+0000 7f91135d0640 1 -- 172.21.15.40:0/2812821341 >> 172.21.15.40:0/2812821341 conn(0x7f910c0a66a0 msgr2=0x7f910c0a2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:22.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.027+0000 7f91135d0640 1 -- 172.21.15.40:0/2812821341 shutdown_connections 2024-04-22T08:21:22.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.027+0000 7f91135d0640 1 -- 172.21.15.40:0/2812821341 wait complete. 2024-04-22T08:21:22.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.027+0000 7f91135d0640 10 monclient: build_initial_monmap 2024-04-22T08:21:22.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.027+0000 7f91135d0640 10 monclient: monmap: 2024-04-22T08:21:22.037 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:22.037 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:21:22.037 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:21:22.037 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:21:22.037 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:22.037 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:22.037 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:21:22.037 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:21:22.037 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:21:22.037 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:22.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.028+0000 7f91135d0640 1 Processor -- start 2024-04-22T08:21:22.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.028+0000 7f91135d0640 1 -- start start 2024-04-22T08:21:22.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.028+0000 7f91135d0640 10 monclient: init 2024-04-22T08:21:22.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.028+0000 7f91135d0640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:22.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.028+0000 7f91135d0640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-22T08:21:22.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.028+0000 7f91135d0640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90f4005330 0x7f910c135060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:22.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.028+0000 7f91135d0640 10 monclient(hunting): picked mon.noname-a con 0x7f90f4005330 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:21:22.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.028+0000 7f91135d0640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f910c1ca700 0x7f910c1355a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:22.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.028+0000 7f91135d0640 10 monclient(hunting): picked mon.noname-c con 0x7f910c1ca700 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:22.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.028+0000 7f91135d0640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f910c120ad0 0x7f910c120f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:22.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.028+0000 7f91135d0640 10 monclient(hunting): picked mon.noname-b con 0x7f910c120ad0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:21:22.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.028+0000 7f91135d0640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:22.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.028+0000 7f91135d0640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f910c135ae0 con 0x7f90f4005330 2024-04-22T08:21:22.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.028+0000 7f91135d0640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:22.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.028+0000 7f91135d0640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f910c09cce0 con 0x7f910c120ad0 2024-04-22T08:21:22.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.028+0000 7f91135d0640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:22.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.028+0000 7f91135d0640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f910c123a50 con 0x7f910c1ca700 2024-04-22T08:21:22.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.028+0000 7f91135d0640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:22.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.028+0000 7f911136c640 1 -- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90f4005330 msgr2=0x7f910c135060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:3300/0 2024-04-22T08:21:22.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.028+0000 7f91135d0640 10 monclient(hunting): authenticate will time out at 1356.023416s 2024-04-22T08:21:22.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.028+0000 7f911136c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90f4005330 0x7f910c135060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:21:22.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.028+0000 7f9110b6b640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f910c1ca700 0x7f910c1355a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:22.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.028+0000 7f9110b6b640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f910c1ca700 0x7f910c1355a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:42996/0 (socket says 172.21.15.40:42996) 2024-04-22T08:21:22.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.028+0000 7f9110b6b640 1 -- 172.21.15.40:0/808644557 learned_addr learned my addr 172.21.15.40:0/808644557 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:21:22.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.028+0000 7f9110b6b640 10 monclient(hunting): get_auth_request con 0x7f910c1ca700 auth_method 0 2024-04-22T08:21:22.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.028+0000 7f9110b6b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:22.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.028+0000 7f9110b6b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:22.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.028+0000 7f9110b6b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:22.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.028+0000 7f9110b6b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:22.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.028+0000 7f9110b6b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:22.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.028+0000 7f9110b6b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:22.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.028+0000 7f9111b6d640 1 --2- 172.21.15.40:0/808644557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f910c120ad0 0x7f910c120f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:22.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.028+0000 7f9111b6d640 10 monclient(hunting): get_auth_request con 0x7f910c120ad0 auth_method 0 2024-04-22T08:21:22.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.028+0000 7f9111b6d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:22.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.028+0000 7f9111b6d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:22.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.028+0000 7f9111b6d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:22.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.028+0000 7f9110b6b640 10 monclient(hunting): handle_auth_done global_id 104195 payload 824 2024-04-22T08:21:22.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.029+0000 7f9110b6b640 1 -- 172.21.15.40:0/808644557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f910c120ad0 msgr2=0x7f910c120f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:22.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.029+0000 7f9110b6b640 1 --2- 172.21.15.40:0/808644557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f910c120ad0 0x7f910c120f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:22.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.029+0000 7f9110b6b640 1 -- 172.21.15.40:0/808644557 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90f4005330 msgr2=0x7f910c135060 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:21:22.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.029+0000 7f9110b6b640 1 --2- 172.21.15.40:0/808644557 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90f4005330 0x7f910c135060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:22.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.029+0000 7f9110b6b640 10 monclient: _finish_hunting 0 2024-04-22T08:21:22.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.029+0000 7f9110b6b640 1 monclient: found mon.noname-c 2024-04-22T08:21:22.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.029+0000 7f9110b6b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:22.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.029+0000 7f9110b6b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:21:22.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.029+0000 7f9110b6b640 1 -- 172.21.15.40:0/808644557 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f910c121fd0 con 0x7f910c1ca700 2024-04-22T08:21:22.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.031+0000 7f9110b6b640 1 -- 172.21.15.40:0/808644557 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f910c1ca700 msgr2=0x7f910c1355a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:22.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.031+0000 7f9110b6b640 1 --2- 172.21.15.40:0/808644557 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f910c1ca700 0x7f910c1355a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f910000e840 tx=0x7f910000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:22.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.031+0000 7f9109ffb640 1 -- 172.21.15.40:0/808644557 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9100009530 con 0x7f910c1ca700 2024-04-22T08:21:22.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.031+0000 7f9109ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:22.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.031+0000 7f9109ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:21:22.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.031+0000 7f9109ffb640 10 monclient: dump: 2024-04-22T08:21:22.047 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:22.047 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:22.047 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:22.047 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:22.047 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:22.047 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:22.047 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:22.047 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:22.047 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:22.047 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:22.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.031+0000 7f9109ffb640 10 monclient: _finish_auth 0 2024-04-22T08:21:22.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.031+0000 7f9109ffb640 10 monclient: _check_auth_tickets 2024-04-22T08:21:22.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.031+0000 7f9109ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:22.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.031+0000 7f9109ffb640 1 -- 172.21.15.40:0/808644557 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9100003dd0 con 0x7f910c1ca700 2024-04-22T08:21:22.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.031+0000 7f91135d0640 5 monclient: authenticate success, global_id 104195 2024-04-22T08:21:22.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.031+0000 7f91135d0640 10 monclient: _renew_subs 2024-04-22T08:21:22.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.031+0000 7f91135d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:21:22.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.031+0000 7f91135d0640 1 -- 172.21.15.40:0/808644557 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f910c1231b0 con 0x7f910c1ca700 2024-04-22T08:21:22.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.031+0000 7f9109ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:22.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.031+0000 7f9109ffb640 1 -- 172.21.15.40:0/808644557 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f910000c970 con 0x7f910c1ca700 2024-04-22T08:21:22.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.031+0000 7f9109ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:22.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.031+0000 7f9109ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:21:22.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.031+0000 7f9109ffb640 10 monclient: dump: 2024-04-22T08:21:22.048 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:22.048 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:22.048 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:22.048 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:22.048 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:22.048 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:22.048 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:22.049 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:22.049 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:22.049 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:22.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.031+0000 7f910b7fe640 4 set_mon_vals no callback set 2024-04-22T08:21:22.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.031+0000 7f91135d0640 10 monclient: _renew_subs 2024-04-22T08:21:22.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.031+0000 7f91135d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:21:22.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.031+0000 7f91135d0640 1 -- 172.21.15.40:0/808644557 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f910c071940 con 0x7f910c1ca700 2024-04-22T08:21:22.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.032+0000 7f91135d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:22.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.032+0000 7f9109ffb640 1 -- 172.21.15.40:0/808644557 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f910000cb50 con 0x7f910c1ca700 2024-04-22T08:21:22.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.032+0000 7f91135d0640 1 --2- 172.21.15.40:0/808644557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f910c121490 0x7f910c06cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:22.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.032+0000 7f91135d0640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:22.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.032+0000 7f91135d0640 1 -- 172.21.15.40:0/808644557 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f910c066770 con 0x7f910c121490 2024-04-22T08:21:22.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.032+0000 7f9109ffb640 1 --2- 172.21.15.40:0/808644557 >> v2:172.21.15.189:6800/30168 conn(0x7f90f8037780 0x7f90f8039c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:22.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.032+0000 7f9109ffb640 1 -- 172.21.15.40:0/808644557 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(279..279 src has 239..279) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f910004a210 con 0x7f910c1ca700 2024-04-22T08:21:22.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.032+0000 7f911136c640 1 --2- 172.21.15.40:0/808644557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f910c121490 0x7f910c06cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:22.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.032+0000 7f911136c640 10 monclient: get_auth_request con 0x7f910c121490 auth_method 0 2024-04-22T08:21:22.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.032+0000 7f9111b6d640 1 --2- 172.21.15.40:0/808644557 >> v2:172.21.15.189:6800/30168 conn(0x7f90f8037780 0x7f90f8039c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:22.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.032+0000 7f911136c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:22.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.032+0000 7f911136c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:22.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.032+0000 7f911136c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:22.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.032+0000 7f9111b6d640 10 monclient: get_auth_request con 0x7f90f8037780 auth_method 0 2024-04-22T08:21:22.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.032+0000 7f911136c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:22.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.032+0000 7f911136c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:22.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.032+0000 7f911136c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:22.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.033+0000 7f911136c640 10 monclient(hunting): handle_auth_done global_id 64999 payload 824 2024-04-22T08:21:22.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.033+0000 7f911136c640 1 -- 172.21.15.40:0/808644557 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f910c09dba0 con 0x7f910c121490 2024-04-22T08:21:22.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.035+0000 7f9111b6d640 1 --2- 172.21.15.40:0/808644557 >> v2:172.21.15.189:6800/30168 conn(0x7f90f8037780 0x7f90f8039c40 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f910c0a0730 tx=0x7f9104006040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:22.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.035+0000 7f911136c640 1 -- 172.21.15.40:0/808644557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f910c121490 msgr2=0x7f910c06cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:22.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.035+0000 7f911136c640 1 --2- 172.21.15.40:0/808644557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f910c121490 0x7f910c06cb70 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f90fc00bd00 tx=0x7f90fc009f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:22.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.035+0000 7f9109ffb640 1 -- 172.21.15.40:0/808644557 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90fc015670 con 0x7f910c121490 2024-04-22T08:21:22.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.035+0000 7f9109ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:22.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.035+0000 7f9109ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:21:22.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.035+0000 7f9109ffb640 10 monclient: dump: 2024-04-22T08:21:22.051 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:22.051 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:22.051 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:22.051 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:22.051 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:22.051 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:22.051 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:22.051 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:22.051 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:22.051 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:22.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.036+0000 7f9109ffb640 1 -- 172.21.15.40:0/808644557 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f90fc0157d0 con 0x7f910c121490 2024-04-22T08:21:22.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.036+0000 7f9109ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:22.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.036+0000 7f9109ffb640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:21:22.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.036+0000 7f9109ffb640 1 -- 172.21.15.40:0/808644557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f910c121490 msgr2=0x7f910c06cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:22.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.036+0000 7f9109ffb640 1 --2- 172.21.15.40:0/808644557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f910c121490 0x7f910c06cb70 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f90fc00bd00 tx=0x7f90fc009f70 comp rx=0 tx=0).stop 2024-04-22T08:21:22.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.036+0000 7f9109ffb640 1 -- 172.21.15.40:0/808644557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f910c121490 msgr2=0x7f910c06cb70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:21:22.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.036+0000 7f9109ffb640 1 --2- 172.21.15.40:0/808644557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f910c121490 0x7f910c06cb70 secure :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f90fc00bd00 tx=0x7f90fc009f70 comp rx=0 tx=0).stop 2024-04-22T08:21:22.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.042+0000 7f91135d0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:21:22.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.042+0000 7f91135d0640 1 --2- 172.21.15.40:0/808644557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f910c06d0b0 0x7f910c06d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:22.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.042+0000 7f91135d0640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:22.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.042+0000 7f91135d0640 1 -- 172.21.15.40:0/808644557 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f910c0a34f0 con 0x7f910c06d0b0 2024-04-22T08:21:22.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.042+0000 7f911136c640 1 --2- 172.21.15.40:0/808644557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f910c06d0b0 0x7f910c06d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:22.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.042+0000 7f911136c640 10 monclient: get_auth_request con 0x7f910c06d0b0 auth_method 0 2024-04-22T08:21:22.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.042+0000 7f911136c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:22.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.042+0000 7f911136c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:22.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.042+0000 7f911136c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:22.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.042+0000 7f911136c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:22.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.042+0000 7f911136c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:22.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.042+0000 7f911136c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:22.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.042+0000 7f911136c640 10 monclient(hunting): handle_auth_done global_id 65002 payload 824 2024-04-22T08:21:22.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.042+0000 7f911136c640 1 -- 172.21.15.40:0/808644557 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f910c09f860 con 0x7f910c06d0b0 2024-04-22T08:21:22.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.045+0000 7f911136c640 1 -- 172.21.15.40:0/808644557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f910c06d0b0 msgr2=0x7f910c06d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:22.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.045+0000 7f911136c640 1 --2- 172.21.15.40:0/808644557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f910c06d0b0 0x7f910c06d4c0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f90fc00bd00 tx=0x7f90fc005e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:22.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.045+0000 7f9109ffb640 1 -- 172.21.15.40:0/808644557 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90fc023070 con 0x7f910c06d0b0 2024-04-22T08:21:22.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.045+0000 7f9109ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:22.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.045+0000 7f9109ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:21:22.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.045+0000 7f9109ffb640 10 monclient: dump: 2024-04-22T08:21:22.055 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:22.055 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:22.055 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:22.055 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:22.055 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:22.055 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:22.055 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:22.055 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:22.055 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:22.055 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:22.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.045+0000 7f9109ffb640 1 -- 172.21.15.40:0/808644557 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4792 (secure 0 0 0) 0x7f90fc014be0 con 0x7f910c06d0b0 2024-04-22T08:21:22.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.045+0000 7f9109ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:21:22.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.045+0000 7f9109ffb640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:21:22.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.045+0000 7f9109ffb640 1 -- 172.21.15.40:0/808644557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f910c06d0b0 msgr2=0x7f910c06d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:22.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.045+0000 7f9109ffb640 1 --2- 172.21.15.40:0/808644557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f910c06d0b0 0x7f910c06d4c0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f90fc00bd00 tx=0x7f90fc005e30 comp rx=0 tx=0).stop 2024-04-22T08:21:22.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.045+0000 7f9109ffb640 1 -- 172.21.15.40:0/808644557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f910c06d0b0 msgr2=0x7f910c06d4c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:21:22.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.045+0000 7f9109ffb640 1 --2- 172.21.15.40:0/808644557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f910c06d0b0 0x7f910c06d4c0 secure :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f90fc00bd00 tx=0x7f90fc005e30 comp rx=0 tx=0).stop 2024-04-22T08:21:22.056 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:21:22.056 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:21:22.056 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:21:22.056 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "leader", 2024-04-22T08:21:22.056 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 84, 2024-04-22T08:21:22.056 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:21:22.056 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:21:22.057 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:21:22.057 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:22.057 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 2, 2024-04-22T08:21:22.057 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:21:22.057 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:21:22.057 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:21:22.057 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:21:22.057 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:21:22.057 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:21:22.057 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:21:22.057 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:21:22.057 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:21:22.057 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:21:22.057 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:21:22.057 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:21:22.058 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:21:22.058 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:22.058 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:21:22.058 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:21:22.058 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:21:22.058 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:21:22.058 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:21:22.058 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:21:22.058 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:21:22.058 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:21:22.058 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:21:22.058 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:21:22.058 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:21:22.059 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:21:22.059 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:22.059 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:22.059 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:21:22.059 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:21:22.059 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:21:22.059 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:21:22.059 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:21:22.059 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:21:22.059 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:21:22.059 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:21:22.059 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:21:22.059 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:21:22.059 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:21:22.059 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:21:22.060 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:21:22.060 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:21:22.060 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:21:22.060 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:21:22.060 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:21:22.060 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:21:22.060 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:21:22.060 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:21:22.060 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:21:22.060 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:21:22.060 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:21:22.060 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:21:22.060 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:21:22.060 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:21:22.060 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:21:22.061 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:22.061 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:21:22.061 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:22.061 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:21:22.061 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:22.061 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:21:22.061 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:21:22.061 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:21:22.061 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:21:22.061 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:22.061 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:21:22.061 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:21:22.061 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:22.061 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:22.061 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:22.062 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:21:22.062 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:21:22.062 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:22.062 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:22.062 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:22.062 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:22.062 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:21:22.062 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:21:22.062 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:21:22.062 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:21:22.062 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:21:22.062 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:22.062 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:22.063 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:21:22.063 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:21:22.063 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:21:22.063 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:21:22.063 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:22.063 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:21:22.063 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:21:22.063 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:22.063 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:22.063 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:22.063 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:21:22.063 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:21:22.063 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:22.063 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:22.063 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:22.064 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:22.064 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:21:22.064 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:21:22.064 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:21:22.064 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:21:22.064 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:21:22.064 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:22.064 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:22.064 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:21:22.064 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:21:22.064 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:21:22.064 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:21:22.064 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:22.064 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:21:22.065 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:21:22.065 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:22.065 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:22.065 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:22.065 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:21:22.065 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:21:22.065 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:22.065 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:22.065 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:22.065 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:22.065 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:21:22.065 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:21:22.065 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:21:22.065 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:21:22.066 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:21:22.066 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:22.066 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:22.066 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:22.066 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:21:22.066 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:21:22.066 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:22.066 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:21:22.066 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:21:22.066 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:21:22.066 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:22.066 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:22.066 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:21:22.066 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:22.066 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:21:22.067 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:21:22.067 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 5 2024-04-22T08:21:22.067 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:22.067 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:22.067 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:21:22.067 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:22.067 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:21:22.067 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:21:22.067 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 6 2024-04-22T08:21:22.067 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:22.067 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:22.067 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": [ 2024-04-22T08:21:22.067 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:22.067 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:21:22.067 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:21:22.068 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:21:22.068 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:22.068 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:22.068 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:22.068 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:21:22.068 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:21:22.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.045+0000 7f91135d0640 1 -- 172.21.15.40:0/808644557 >> v2:172.21.15.189:6800/30168 conn(0x7f90f8037780 msgr2=0x7f90f8039c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:22.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.045+0000 7f91135d0640 1 --2- 172.21.15.40:0/808644557 >> v2:172.21.15.189:6800/30168 conn(0x7f90f8037780 0x7f90f8039c40 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f910c0a0730 tx=0x7f9104006040 comp rx=0 tx=0).stop 2024-04-22T08:21:22.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.045+0000 7f91135d0640 10 monclient: shutdown 2024-04-22T08:21:22.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.045+0000 7f91135d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:22.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.045+0000 7f91135d0640 1 -- 172.21.15.40:0/808644557 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f910c1ca700 msgr2=0x7f910c1355a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:22.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.045+0000 7f91135d0640 1 --2- 172.21.15.40:0/808644557 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f910c1ca700 0x7f910c1355a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f910000e840 tx=0x7f910000ed00 comp rx=0 tx=0).stop 2024-04-22T08:21:22.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.046+0000 7f9111b6d640 1 -- 172.21.15.40:0/808644557 reap_dead start 2024-04-22T08:21:22.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.046+0000 7f91135d0640 1 -- 172.21.15.40:0/808644557 shutdown_connections 2024-04-22T08:21:22.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.046+0000 7f91135d0640 1 --2- 172.21.15.40:0/808644557 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f910c1ca700 0x7f910c1355a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:22.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.046+0000 7f91135d0640 1 -- 172.21.15.40:0/808644557 >> 172.21.15.40:0/808644557 conn(0x7f910c0a66a0 msgr2=0x7f910c0a3f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:22.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.046+0000 7f91135d0640 1 -- 172.21.15.40:0/808644557 shutdown_connections 2024-04-22T08:21:22.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:22.046+0000 7f91135d0640 1 -- 172.21.15.40:0/808644557 wait complete. 2024-04-22T08:21:22.070 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-22T08:21:22.070 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-04-22T08:21:23.548 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[39003]: starting. 2024-04-22T08:21:23.548 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[39003]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-39002 2024-04-22T08:21:23.549 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[39003]: created object 0... 2024-04-22T08:21:23.549 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[39003]: created object 25... 2024-04-22T08:21:23.549 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[39003]: created object 49... 2024-04-22T08:21:23.549 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[39003]: finishing. 2024-04-22T08:21:23.549 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[39003]: shutting down. 2024-04-22T08:21:24.419 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[39004]: starting. 2024-04-22T08:21:24.419 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[39004]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-39002 2024-04-22T08:21:24.419 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[39004]: shutting down. 2024-04-22T08:21:26.546 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:21:26.546 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[39040]: starting. 2024-04-22T08:21:26.546 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[39040]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-39002 2024-04-22T08:21:26.546 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[39040]: created object 0... 2024-04-22T08:21:26.546 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[39040]: created object 25... 2024-04-22T08:21:26.546 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[39040]: created object 49... 2024-04-22T08:21:26.547 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[39040]: finishing. 2024-04-22T08:21:26.547 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[39040]: shutting down. 2024-04-22T08:21:26.586 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:21:26.586 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[39042]: starting. 2024-04-22T08:21:26.586 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[39042]: listing objects. 2024-04-22T08:21:26.587 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[39042]: listed object 0... 2024-04-22T08:21:26.587 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[39042]: listed object 25... 2024-04-22T08:21:26.587 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[39042]: saw 50 objects 2024-04-22T08:21:26.587 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[39042]: shutting down. 2024-04-22T08:21:27.442 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:21:27.443 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[39041]: starting. 2024-04-22T08:21:27.443 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[39041]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-39002 2024-04-22T08:21:27.443 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[39041]: shutting down. 2024-04-22T08:21:27.444 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:21:27.444 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:21:27.444 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:21:27.446 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:21:30.083 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[39111]: starting. 2024-04-22T08:21:30.083 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[39111]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-39110 2024-04-22T08:21:30.083 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[39111]: created object 0... 2024-04-22T08:21:30.083 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[39111]: created object 25... 2024-04-22T08:21:30.083 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[39111]: created object 49... 2024-04-22T08:21:30.083 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[39111]: finishing. 2024-04-22T08:21:30.083 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[39111]: shutting down. 2024-04-22T08:21:30.979 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[39112]: starting. 2024-04-22T08:21:30.979 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[39112]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-39110 2024-04-22T08:21:30.980 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[39112]: shutting down. 2024-04-22T08:21:33.290 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:21:33.290 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[39148]: starting. 2024-04-22T08:21:33.290 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[39148]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-39110 2024-04-22T08:21:33.290 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[39148]: created object 0... 2024-04-22T08:21:33.290 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[39148]: created object 25... 2024-04-22T08:21:33.290 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[39148]: created object 49... 2024-04-22T08:21:33.291 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[39148]: finishing. 2024-04-22T08:21:33.291 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[39148]: shutting down. 2024-04-22T08:21:33.329 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:21:33.329 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[39150]: starting. 2024-04-22T08:21:33.330 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[39150]: listing objects. 2024-04-22T08:21:33.330 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[39150]: listed object 0... 2024-04-22T08:21:33.330 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[39150]: listed object 25... 2024-04-22T08:21:33.330 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[39150]: saw 50 objects 2024-04-22T08:21:33.330 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[39150]: shutting down. 2024-04-22T08:21:34.190 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:21:34.190 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[39149]: starting. 2024-04-22T08:21:34.190 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[39149]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-39110 2024-04-22T08:21:34.190 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[39149]: shutting down. 2024-04-22T08:21:34.192 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:21:34.192 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:21:34.192 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:21:34.194 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:21:36.347 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[39219]: starting. 2024-04-22T08:21:36.347 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[39219]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-39218 2024-04-22T08:21:36.347 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[39219]: created object 0... 2024-04-22T08:21:36.348 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[39219]: created object 25... 2024-04-22T08:21:36.348 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[39219]: created object 49... 2024-04-22T08:21:36.348 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[39219]: finishing. 2024-04-22T08:21:36.348 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[39219]: shutting down. 2024-04-22T08:21:37.234 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[39220]: starting. 2024-04-22T08:21:37.234 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[39220]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-39218 2024-04-22T08:21:37.234 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[39220]: shutting down. 2024-04-22T08:21:39.083 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:21:39.083 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[39255]: starting. 2024-04-22T08:21:39.083 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[39255]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-39218 2024-04-22T08:21:39.083 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[39255]: created object 0... 2024-04-22T08:21:39.084 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[39255]: created object 25... 2024-04-22T08:21:39.084 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[39255]: created object 49... 2024-04-22T08:21:39.084 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[39255]: finishing. 2024-04-22T08:21:39.084 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[39255]: shutting down. 2024-04-22T08:21:39.124 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:21:39.124 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[39257]: starting. 2024-04-22T08:21:39.124 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[39257]: listing objects. 2024-04-22T08:21:39.125 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[39257]: listed object 0... 2024-04-22T08:21:39.125 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[39257]: listed object 25... 2024-04-22T08:21:39.125 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[39257]: saw 50 objects 2024-04-22T08:21:39.125 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[39257]: shutting down. 2024-04-22T08:21:39.973 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:21:39.973 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[39256]: starting. 2024-04-22T08:21:39.973 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[39256]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-39218 2024-04-22T08:21:39.973 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[39256]: shutting down. 2024-04-22T08:21:39.975 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:21:39.975 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:21:39.975 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:21:39.978 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:21:42.071 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-04-22T08:21:42.072 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-04-22T08:21:42.072 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-22T08:21:42.072 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-22T08:21:42.075 INFO:tasks.ceph.mon.a:Started 2024-04-22T08:21:42.075 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-22T08:21:42.075 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:21:42.096 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[39326]: starting. 2024-04-22T08:21:42.096 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[39326]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-39325 2024-04-22T08:21:42.096 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[39326]: created object 0... 2024-04-22T08:21:42.096 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[39326]: created object 25... 2024-04-22T08:21:42.096 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[39326]: created object 49... 2024-04-22T08:21:42.096 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[39326]: finishing. 2024-04-22T08:21:42.096 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[39326]: shutting down. 2024-04-22T08:21:42.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.179+0000 7f83414d5640 10 monclient: get_monmap_and_config 2024-04-22T08:21:42.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.179+0000 7f83414d5640 10 monclient: build_initial_monmap 2024-04-22T08:21:42.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.179+0000 7f83414d5640 10 monclient: monmap: 2024-04-22T08:21:42.179 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:42.179 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:42.179 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:21:42.180358+0000 2024-04-22T08:21:42.179 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:21:42.180358+0000 2024-04-22T08:21:42.179 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:42.179 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:42.179 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:21:42.179 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:21:42.180 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:21:42.180 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:21:42.180 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:21:42.180 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:42.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.180+0000 7f83414d5640 1 Processor -- start 2024-04-22T08:21:42.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.180+0000 7f83414d5640 1 -- start start 2024-04-22T08:21:42.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.180+0000 7f83414d5640 10 monclient: init 2024-04-22T08:21:42.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.180+0000 7f83414d5640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:42.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.180+0000 7f83414d5640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-04-22T08:21:42.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.180+0000 7f83414d5640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f833c09ee10 0x7f833c0a32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:42.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.180+0000 7f83414d5640 10 monclient(hunting): picked mon.noname-a con 0x7f833c09ee10 addr v2:172.21.15.40:3300/0 2024-04-22T08:21:42.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.180+0000 7f83414d5640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f833c0a1d30 0x7f833c0a2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:42.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.180+0000 7f83414d5640 10 monclient(hunting): picked mon.noname-b con 0x7f833c0a1d30 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:42.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.180+0000 7f83414d5640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f833c135d90 0x7f833c1361c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:42.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.180+0000 7f83414d5640 10 monclient(hunting): picked mon.noname-c con 0x7f833c135d90 addr v2:172.21.15.189:3300/0 2024-04-22T08:21:42.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.180+0000 7f83414d5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:42.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.180+0000 7f83414d5640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f833c09f1f0 con 0x7f833c09ee10 2024-04-22T08:21:42.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.180+0000 7f83414d5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:42.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.180+0000 7f83414d5640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f833c0a39c0 con 0x7f833c135d90 2024-04-22T08:21:42.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.180+0000 7f83414d5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:42.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.180+0000 7f83414d5640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833c09fd30 con 0x7f833c0a1d30 2024-04-22T08:21:42.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.180+0000 7f83414d5640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:42.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.180+0000 7f83414d5640 10 monclient(hunting): authenticate will time out at 1376.174414s 2024-04-22T08:21:42.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.180+0000 7f833affd640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f833c09ee10 0x7f833c0a32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:42.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.180+0000 7f833a7fc640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f833c0a1d30 0x7f833c0a2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:42.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.180+0000 7f833a7fc640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f833c0a1d30 0x7f833c0a2150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:41102/0 (socket says 172.21.15.40:41102) 2024-04-22T08:21:42.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.180+0000 7f833affd640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f833c09ee10 0x7f833c0a32b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46618/0 (socket says 172.21.15.40:46618) 2024-04-22T08:21:42.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.180+0000 7f833a7fc640 1 -- 172.21.15.40:0/2670219684 learned_addr learned my addr 172.21.15.40:0/2670219684 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:21:42.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.180+0000 7f833a7fc640 10 monclient(hunting): get_auth_request con 0x7f833c0a1d30 auth_method 0 2024-04-22T08:21:42.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.180+0000 7f833a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:42.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.180+0000 7f833a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:42.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.180+0000 7f833a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:42.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.180+0000 7f833affd640 10 monclient(hunting): get_auth_request con 0x7f833c09ee10 auth_method 0 2024-04-22T08:21:42.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.180+0000 7f833affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:42.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.180+0000 7f833affd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:42.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.180+0000 7f833affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:42.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.180+0000 7f833a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:42.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.180+0000 7f833a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:42.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.180+0000 7f833b7fe640 1 --2- 172.21.15.40:0/2670219684 >> v2:172.21.15.189:3300/0 conn(0x7f833c135d90 0x7f833c1361c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:42.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.180+0000 7f833a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:42.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.180+0000 7f833b7fe640 10 monclient(hunting): get_auth_request con 0x7f833c135d90 auth_method 0 2024-04-22T08:21:42.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.180+0000 7f833b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:42.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.180+0000 7f833b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:42.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.180+0000 7f833b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:42.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.180+0000 7f833affd640 1 -- 172.21.15.40:0/2670219684 >> v2:172.21.15.40:3300/0 conn(0x7f833c09ee10 msgr2=0x7f833c0a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:21:42.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.180+0000 7f833affd640 1 -- 172.21.15.40:0/2670219684 >> v2:172.21.15.40:3300/0 conn(0x7f833c09ee10 msgr2=0x7f833c0a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:21:42.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.180+0000 7f833affd640 1 --2- 172.21.15.40:0/2670219684 >> v2:172.21.15.40:3300/0 conn(0x7f833c09ee10 0x7f833c0a32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:21:42.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.180+0000 7f833affd640 1 --2- 172.21.15.40:0/2670219684 >> v2:172.21.15.40:3300/0 conn(0x7f833c09ee10 0x7f833c0a32b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:21:42.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.181+0000 7f833b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:42.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.181+0000 7f833b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:42.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.181+0000 7f833b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:42.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.181+0000 7f833a7fc640 10 monclient(hunting): handle_auth_done global_id 104258 payload 274 2024-04-22T08:21:42.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.181+0000 7f833a7fc640 1 -- 172.21.15.40:0/2670219684 >> v2:172.21.15.189:3300/0 conn(0x7f833c135d90 msgr2=0x7f833c1361c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:21:42.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.181+0000 7f833a7fc640 1 --2- 172.21.15.40:0/2670219684 >> v2:172.21.15.189:3300/0 conn(0x7f833c135d90 0x7f833c1361c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:42.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.181+0000 7f833a7fc640 1 -- 172.21.15.40:0/2670219684 >> v2:172.21.15.40:3300/0 conn(0x7f833c09ee10 msgr2=0x7f833c0a32b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:21:42.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.181+0000 7f833a7fc640 1 --2- 172.21.15.40:0/2670219684 >> v2:172.21.15.40:3300/0 conn(0x7f833c09ee10 0x7f833c0a32b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:42.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.181+0000 7f833a7fc640 10 monclient: _finish_hunting 0 2024-04-22T08:21:42.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.181+0000 7f833a7fc640 1 monclient: found mon.noname-b 2024-04-22T08:21:42.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.181+0000 7f833a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:42.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.181+0000 7f833a7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:21:42.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.181+0000 7f833a7fc640 1 -- 172.21.15.40:0/2670219684 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f833c0a02f0 con 0x7f833c0a1d30 2024-04-22T08:21:42.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.183+0000 7f833a7fc640 1 -- 172.21.15.40:0/2670219684 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f833c0a1d30 msgr2=0x7f833c0a2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:42.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.183+0000 7f833a7fc640 1 --2- 172.21.15.40:0/2670219684 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f833c0a1d30 0x7f833c0a2150 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f833c074690 tx=0x7f833002b7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4fa4e91a6935a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:42.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.184+0000 7f8339ffb640 1 -- 172.21.15.40:0/2670219684 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8330031960 con 0x7f833c0a1d30 2024-04-22T08:21:42.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.184+0000 7f8339ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:42.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.184+0000 7f8339ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:21:42.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.184+0000 7f8339ffb640 10 monclient: dump: 2024-04-22T08:21:42.189 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:42.189 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:42.190 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:42.190 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:42.190 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:42.190 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:42.190 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:42.190 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:42.190 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:42.190 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:42.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.184+0000 7f8339ffb640 10 monclient: _finish_auth 0 2024-04-22T08:21:42.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.184+0000 7f8339ffb640 10 monclient: _check_auth_tickets 2024-04-22T08:21:42.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.184+0000 7f8339ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:42.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.184+0000 7f8339ffb640 1 -- 172.21.15.40:0/2670219684 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8330031ac0 con 0x7f833c0a1d30 2024-04-22T08:21:42.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.184+0000 7f83414d5640 5 monclient: authenticate success, global_id 104258 2024-04-22T08:21:42.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.184+0000 7f83414d5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:21:42.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.184+0000 7f8339ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:42.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.184+0000 7f83414d5640 10 monclient: get_monmap_and_config success 2024-04-22T08:21:42.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.184+0000 7f8339ffb640 1 -- 172.21.15.40:0/2670219684 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8330031e50 con 0x7f833c0a1d30 2024-04-22T08:21:42.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.184+0000 7f8339ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:42.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.184+0000 7f83414d5640 4 set_mon_vals no callback set 2024-04-22T08:21:42.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.184+0000 7f8339ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:21:42.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.184+0000 7f8339ffb640 10 monclient: dump: 2024-04-22T08:21:42.191 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:42.191 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:42.191 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:42.191 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:42.191 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:42.191 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:42.191 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:42.192 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:42.192 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:42.192 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:42.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.184+0000 7f83414d5640 10 monclient: shutdown 2024-04-22T08:21:42.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.184+0000 7f83414d5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:42.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.184+0000 7f83414d5640 1 -- 172.21.15.40:0/2670219684 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f833c0a1d30 msgr2=0x7f833c0a2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:42.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.184+0000 7f83414d5640 1 --2- 172.21.15.40:0/2670219684 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f833c0a1d30 0x7f833c0a2150 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f833c074690 tx=0x7f833002b7f0 comp rx=0 tx=0).stop 2024-04-22T08:21:42.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.184+0000 7f83414d5640 1 -- 172.21.15.40:0/2670219684 shutdown_connections 2024-04-22T08:21:42.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.184+0000 7f83414d5640 1 --2- 172.21.15.40:0/2670219684 >> v2:172.21.15.189:3300/0 conn(0x7f833c135d90 0x7f833c1361c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:42.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.184+0000 7f83414d5640 1 --2- 172.21.15.40:0/2670219684 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f833c0a1d30 0x7f833c0a2150 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:42.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.184+0000 7f83414d5640 1 --2- 172.21.15.40:0/2670219684 >> v2:172.21.15.40:3300/0 conn(0x7f833c09ee10 0x7f833c0a32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:42.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.184+0000 7f83414d5640 1 -- 172.21.15.40:0/2670219684 >> 172.21.15.40:0/2670219684 conn(0x7f833c0a4ef0 msgr2=0x7f833c0a5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:42.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.184+0000 7f83414d5640 1 -- 172.21.15.40:0/2670219684 shutdown_connections 2024-04-22T08:21:42.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.184+0000 7f83414d5640 1 -- 172.21.15.40:0/2670219684 wait complete. 2024-04-22T08:21:42.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.185+0000 7f83414d5640 10 monclient: build_initial_monmap 2024-04-22T08:21:42.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.185+0000 7f83414d5640 10 monclient: monmap: 2024-04-22T08:21:42.193 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:42.193 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:21:42.193 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:21:42.193 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:21:42.193 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:42.193 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:42.193 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:21:42.193 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:21:42.193 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:21:42.193 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:42.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.185+0000 7f83414d5640 1 Processor -- start 2024-04-22T08:21:42.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.185+0000 7f83414d5640 1 -- start start 2024-04-22T08:21:42.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.185+0000 7f83414d5640 10 monclient: init 2024-04-22T08:21:42.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f83414d5640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:42.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f83414d5640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-22T08:21:42.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f83414d5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f833c09ee10 0x7f833c11fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:42.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f83414d5640 10 monclient(hunting): picked mon.noname-a con 0x7f833c09ee10 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:21:42.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f83414d5640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f833c0a1d30 0x7f833c11ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:42.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f83414d5640 10 monclient(hunting): picked mon.noname-c con 0x7f833c0a1d30 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:42.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f83414d5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f833c135d90 0x7f833c120510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:42.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f83414d5640 10 monclient(hunting): picked mon.noname-b con 0x7f833c135d90 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:21:42.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f83414d5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:42.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f83414d5640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833c131c70 con 0x7f833c09ee10 2024-04-22T08:21:42.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f83414d5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:42.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f83414d5640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833c120b90 con 0x7f833c135d90 2024-04-22T08:21:42.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f83414d5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:42.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f83414d5640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833c121760 con 0x7f833c0a1d30 2024-04-22T08:21:42.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f83414d5640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:42.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f83414d5640 10 monclient(hunting): authenticate will time out at 1376.179978s 2024-04-22T08:21:42.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f833affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f833c09ee10 0x7f833c11fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:42.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f833a7fc640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f833c0a1d30 0x7f833c11ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:42.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f833affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f833c09ee10 0x7f833c11fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46630/0 (socket says 172.21.15.40:46630) 2024-04-22T08:21:42.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f833affd640 1 -- 172.21.15.40:0/1273620898 learned_addr learned my addr 172.21.15.40:0/1273620898 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:21:42.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f833a7fc640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f833c0a1d30 0x7f833c11ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:41110/0 (socket says 172.21.15.40:41110) 2024-04-22T08:21:42.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f833affd640 10 monclient(hunting): get_auth_request con 0x7f833c09ee10 auth_method 0 2024-04-22T08:21:42.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f833affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:42.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f833affd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:42.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f833affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:42.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f833a7fc640 10 monclient(hunting): get_auth_request con 0x7f833c0a1d30 auth_method 0 2024-04-22T08:21:42.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f833a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:42.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f833a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:42.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f833a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:42.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f833affd640 1 -- 172.21.15.40:0/1273620898 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f833c09ee10 msgr2=0x7f833c11fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-22T08:21:42.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f833a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:42.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f833affd640 1 -- 172.21.15.40:0/1273620898 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f833c09ee10 msgr2=0x7f833c11fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:21:42.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f833a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:42.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f833affd640 1 --2- 172.21.15.40:0/1273620898 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f833c09ee10 0x7f833c11fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:21:42.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f833a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:42.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f833affd640 1 --2- 172.21.15.40:0/1273620898 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f833c09ee10 0x7f833c11fa90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:21:42.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f833b7fe640 1 --2- 172.21.15.40:0/1273620898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f833c135d90 0x7f833c120510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:42.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f833b7fe640 10 monclient(hunting): get_auth_request con 0x7f833c135d90 auth_method 0 2024-04-22T08:21:42.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f833b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:42.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f833b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:42.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f833b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:42.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f833b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:42.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f833b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:42.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f833b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:42.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f833a7fc640 10 monclient(hunting): handle_auth_done global_id 104261 payload 824 2024-04-22T08:21:42.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f833a7fc640 1 -- 172.21.15.40:0/1273620898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f833c135d90 msgr2=0x7f833c120510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:42.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f833a7fc640 1 --2- 172.21.15.40:0/1273620898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f833c135d90 0x7f833c120510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:42.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f833a7fc640 1 -- 172.21.15.40:0/1273620898 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f833c09ee10 msgr2=0x7f833c11fa90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:21:42.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f833a7fc640 1 --2- 172.21.15.40:0/1273620898 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f833c09ee10 0x7f833c11fa90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:42.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f833a7fc640 10 monclient: _finish_hunting 0 2024-04-22T08:21:42.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f833a7fc640 1 monclient: found mon.noname-c 2024-04-22T08:21:42.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f833a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:42.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f833a7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:21:42.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.186+0000 7f833a7fc640 1 -- 172.21.15.40:0/1273620898 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83300094d0 con 0x7f833c0a1d30 2024-04-22T08:21:42.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.188+0000 7f833a7fc640 1 -- 172.21.15.40:0/1273620898 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f833c0a1d30 msgr2=0x7f833c11ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:42.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.188+0000 7f833a7fc640 1 --2- 172.21.15.40:0/1273620898 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f833c0a1d30 0x7f833c11ffd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8330035000 tx=0x7f83300046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:42.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f83237fe640 1 -- 172.21.15.40:0/1273620898 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8330033ba0 con 0x7f833c0a1d30 2024-04-22T08:21:42.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f83237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:42.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f83237fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:21:42.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f83237fe640 10 monclient: dump: 2024-04-22T08:21:42.206 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:42.206 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:42.206 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:42.206 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:42.206 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:42.206 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:42.206 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:42.206 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:42.207 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:42.207 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:42.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f83237fe640 10 monclient: _finish_auth 0 2024-04-22T08:21:42.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f83237fe640 10 monclient: _check_auth_tickets 2024-04-22T08:21:42.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f83237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:42.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f83414d5640 5 monclient: authenticate success, global_id 104261 2024-04-22T08:21:42.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f83237fe640 1 -- 172.21.15.40:0/1273620898 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8330033d20 con 0x7f833c0a1d30 2024-04-22T08:21:42.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f83414d5640 10 monclient: _renew_subs 2024-04-22T08:21:42.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f83414d5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:21:42.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f83414d5640 1 -- 172.21.15.40:0/1273620898 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f833c071630 con 0x7f833c0a1d30 2024-04-22T08:21:42.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f83237fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:42.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f83237fe640 1 -- 172.21.15.40:0/1273620898 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833003c640 con 0x7f833c0a1d30 2024-04-22T08:21:42.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f83237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:42.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f83237fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:21:42.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f83237fe640 10 monclient: dump: 2024-04-22T08:21:42.208 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:42.208 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:42.208 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:42.208 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:42.208 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:42.208 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:42.208 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:42.208 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:42.208 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:42.208 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:42.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f83414d5640 10 monclient: _renew_subs 2024-04-22T08:21:42.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f83397fa640 4 set_mon_vals no callback set 2024-04-22T08:21:42.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f83414d5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:21:42.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f83414d5640 1 -- 172.21.15.40:0/1273620898 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f833c122d30 con 0x7f833c0a1d30 2024-04-22T08:21:42.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f833a7fc640 1 -- 172.21.15.40:0/1273620898 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f833c0a1d30 msgr2=0x7f833c11ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-22T08:21:42.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f833a7fc640 1 -- 172.21.15.40:0/1273620898 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f833c0a1d30 msgr2=0x7f833c11ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:21:42.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f833a7fc640 1 --2- 172.21.15.40:0/1273620898 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f833c0a1d30 0x7f833c11ffd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8330035000 tx=0x7f83300046c0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:21:42.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f833a7fc640 1 --2- 172.21.15.40:0/1273620898 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f833c0a1d30 0x7f833c11ffd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8330035000 tx=0x7f83300046c0 comp rx=0 tx=0).stop 2024-04-22T08:21:42.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f83237fe640 10 monclient: ms_handle_reset current mon [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:42.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f83237fe640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:42.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f83237fe640 1 -- 172.21.15.40:0/1273620898 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f833c0a1d30 msgr2=0x7f833c11ffd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:21:42.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f83237fe640 1 --2- 172.21.15.40:0/1273620898 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f833c0a1d30 0x7f833c11ffd0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:42.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f83237fe640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-22T08:21:42.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f83237fe640 1 --2- 172.21.15.40:0/1273620898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f833c135d90 0x7f83280058c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:42.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f83237fe640 10 monclient(hunting): picked mon.b con 0x7f833c135d90 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:21:42.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f83237fe640 1 --2- 172.21.15.40:0/1273620898 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8328006250 0x7f83280086e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:42.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f83237fe640 10 monclient(hunting): picked mon.c con 0x7f8328006250 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:42.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f83237fe640 1 --2- 172.21.15.40:0/1273620898 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f833c09ee10 0x7f832800b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:42.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f83237fe640 10 monclient(hunting): picked mon.a con 0x7f833c09ee10 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:21:42.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f83237fe640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:42.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f83237fe640 1 -- 172.21.15.40:0/1273620898 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f832800bc10 con 0x7f833c09ee10 2024-04-22T08:21:42.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f83237fe640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:42.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f83237fe640 1 -- 172.21.15.40:0/1273620898 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f832800bd50 con 0x7f833c135d90 2024-04-22T08:21:42.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f83237fe640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:42.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f83237fe640 1 -- 172.21.15.40:0/1273620898 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f832800be90 con 0x7f8328006250 2024-04-22T08:21:42.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f83237fe640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:42.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f833a7fc640 1 --2- 172.21.15.40:0/1273620898 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8328006250 0x7f83280086e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:42.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f833a7fc640 10 monclient(hunting): get_auth_request con 0x7f8328006250 auth_method 0 2024-04-22T08:21:42.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f833a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:42.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f833a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:42.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f833a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:21:42.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f833a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:42.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f833a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:42.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f833b7fe640 1 --2- 172.21.15.40:0/1273620898 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f833c09ee10 0x7f832800b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:42.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f833a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-22T08:21:42.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f833b7fe640 10 monclient(hunting): get_auth_request con 0x7f833c09ee10 auth_method 0 2024-04-22T08:21:42.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f833b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:42.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f833b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:42.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f833b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:21:42.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f833affd640 1 --2- 172.21.15.40:0/1273620898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f833c135d90 0x7f83280058c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:42.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f833affd640 10 monclient(hunting): get_auth_request con 0x7f833c135d90 auth_method 0 2024-04-22T08:21:42.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f833affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:42.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f833affd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:42.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f833affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:21:42.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f833a7fc640 10 monclient(hunting): handle_auth_done global_id 104261 payload 293 2024-04-22T08:21:42.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f833a7fc640 1 -- 172.21.15.40:0/1273620898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f833c135d90 msgr2=0x7f83280058c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:42.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f833a7fc640 1 --2- 172.21.15.40:0/1273620898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f833c135d90 0x7f83280058c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:42.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f833a7fc640 1 -- 172.21.15.40:0/1273620898 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f833c09ee10 msgr2=0x7f832800b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:42.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f833a7fc640 1 --2- 172.21.15.40:0/1273620898 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f833c09ee10 0x7f832800b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:42.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f833a7fc640 10 monclient: _finish_hunting 0 2024-04-22T08:21:42.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f833a7fc640 1 monclient: found mon.c 2024-04-22T08:21:42.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f833a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:42.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f833a7fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:21:42.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f833a7fc640 1 -- 172.21.15.40:0/1273620898 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f832800c180 con 0x7f8328006250 2024-04-22T08:21:42.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f833a7fc640 10 monclient: _finish_auth 0 2024-04-22T08:21:42.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f833a7fc640 10 monclient: _check_auth_tickets 2024-04-22T08:21:42.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f833a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:42.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.189+0000 7f833b7fe640 1 --2- 172.21.15.40:0/1273620898 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f833c09ee10 0x7f832800b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T08:21:42.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.190+0000 7f83414d5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:42.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.190+0000 7f83414d5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:42.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.190+0000 7f83414d5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:21:42.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.190+0000 7f83414d5640 1 -- 172.21.15.40:0/1273620898 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f833c04e9b0 con 0x7f8328006250 2024-04-22T08:21:42.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.192+0000 7f833a7fc640 1 -- 172.21.15.40:0/1273620898 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8328006250 msgr2=0x7f83280086e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:42.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.192+0000 7f833a7fc640 1 --2- 172.21.15.40:0/1273620898 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8328006250 0x7f83280086e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8330041000 tx=0x7f8330008030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:42.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.192+0000 7f83237fe640 1 -- 172.21.15.40:0/1273620898 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8330005b90 con 0x7f8328006250 2024-04-22T08:21:42.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.192+0000 7f83237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:42.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.192+0000 7f83237fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:21:42.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.192+0000 7f83237fe640 10 monclient: dump: 2024-04-22T08:21:42.213 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:42.213 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:42.213 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:42.213 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:42.214 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:42.214 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:42.214 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:42.214 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:42.214 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:42.214 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:42.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.192+0000 7f83237fe640 1 -- 172.21.15.40:0/1273620898 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8330005cf0 con 0x7f8328006250 2024-04-22T08:21:42.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.192+0000 7f83237fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:42.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.192+0000 7f8338ff9640 4 set_mon_vals no callback set 2024-04-22T08:21:42.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.192+0000 7f83237fe640 1 -- 172.21.15.40:0/1273620898 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f833003a040 con 0x7f8328006250 2024-04-22T08:21:42.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.193+0000 7f83237fe640 1 --2- 172.21.15.40:0/1273620898 >> v2:172.21.15.189:6800/30168 conn(0x7f832803ff50 0x7f8328042410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:42.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.193+0000 7f83237fe640 1 -- 172.21.15.40:0/1273620898 <== mon.2 v2:172.21.15.40:3301/0 4 ==== osd_map(300..300 src has 239..300) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f833006d320 con 0x7f8328006250 2024-04-22T08:21:42.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.193+0000 7f833affd640 1 --2- 172.21.15.40:0/1273620898 >> v2:172.21.15.189:6800/30168 conn(0x7f832803ff50 0x7f8328042410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:42.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.193+0000 7f833affd640 10 monclient: get_auth_request con 0x7f832803ff50 auth_method 0 2024-04-22T08:21:42.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.195+0000 7f833affd640 1 --2- 172.21.15.40:0/1273620898 >> v2:172.21.15.189:6800/30168 conn(0x7f832803ff50 0x7f8328042410 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f832c006e80 tx=0x7f832c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:42.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.197+0000 7f83237fe640 1 -- 172.21.15.40:0/1273620898 <== mon.2 v2:172.21.15.40:3301/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f8330005e60 con 0x7f8328006250 2024-04-22T08:21:42.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.197+0000 7f83237fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:42.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.197+0000 7f83237fe640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:21:42.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.448+0000 7f83414d5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:21:42.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.448+0000 7f83414d5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:21:42.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.448+0000 7f83414d5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:21:42.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.448+0000 7f83414d5640 1 -- 172.21.15.40:0/1273620898 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f833c06a1e0 con 0x7f8328006250 2024-04-22T08:21:42.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.448+0000 7f83237fe640 1 -- 172.21.15.40:0/1273620898 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7f8330038080 con 0x7f8328006250 2024-04-22T08:21:42.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.448+0000 7f83237fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:21:42.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.448+0000 7f83237fe640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:21:42.448 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":84,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":23,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:21:42.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.450+0000 7f83414d5640 1 -- 172.21.15.40:0/1273620898 >> v2:172.21.15.189:6800/30168 conn(0x7f832803ff50 msgr2=0x7f8328042410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:42.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.450+0000 7f83414d5640 1 --2- 172.21.15.40:0/1273620898 >> v2:172.21.15.189:6800/30168 conn(0x7f832803ff50 0x7f8328042410 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f832c006e80 tx=0x7f832c008040 comp rx=0 tx=0).stop 2024-04-22T08:21:42.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.450+0000 7f83414d5640 10 monclient: shutdown 2024-04-22T08:21:42.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.450+0000 7f83414d5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:42.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.450+0000 7f83414d5640 1 -- 172.21.15.40:0/1273620898 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8328006250 msgr2=0x7f83280086e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:42.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.450+0000 7f83414d5640 1 --2- 172.21.15.40:0/1273620898 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8328006250 0x7f83280086e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8330041000 tx=0x7f8330008030 comp rx=0 tx=0).stop 2024-04-22T08:21:42.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.450+0000 7f83414d5640 1 -- 172.21.15.40:0/1273620898 shutdown_connections 2024-04-22T08:21:42.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.450+0000 7f83414d5640 1 --2- 172.21.15.40:0/1273620898 >> v2:172.21.15.189:6800/30168 conn(0x7f832803ff50 0x7f8328042410 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:42.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.450+0000 7f83414d5640 1 --2- 172.21.15.40:0/1273620898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f833c135d90 0x7f83280058c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:42.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.450+0000 7f83414d5640 1 --2- 172.21.15.40:0/1273620898 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f833c09ee10 0x7f832800b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:42.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.450+0000 7f83414d5640 1 --2- 172.21.15.40:0/1273620898 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8328006250 0x7f83280086e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:42.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.450+0000 7f83414d5640 1 -- 172.21.15.40:0/1273620898 >> 172.21.15.40:0/1273620898 conn(0x7f833c0a4ef0 msgr2=0x7f833c0a62f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:42.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.450+0000 7f83414d5640 1 -- 172.21.15.40:0/1273620898 shutdown_connections 2024-04-22T08:21:42.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:42.450+0000 7f83414d5640 1 -- 172.21.15.40:0/1273620898 wait complete. 2024-04-22T08:21:42.988 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[39327]: starting. 2024-04-22T08:21:42.988 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[39327]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-39325 2024-04-22T08:21:42.988 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[39327]: shutting down. 2024-04-22T08:21:45.122 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:21:45.122 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[39362]: starting. 2024-04-22T08:21:45.123 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[39362]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-39325 2024-04-22T08:21:45.123 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[39362]: created object 0... 2024-04-22T08:21:45.123 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[39362]: created object 25... 2024-04-22T08:21:45.123 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[39362]: created object 49... 2024-04-22T08:21:45.123 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[39362]: finishing. 2024-04-22T08:21:45.123 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[39362]: shutting down. 2024-04-22T08:21:45.164 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:21:45.164 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[39364]: starting. 2024-04-22T08:21:45.164 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[39364]: listing objects. 2024-04-22T08:21:45.164 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[39364]: listed object 0... 2024-04-22T08:21:45.164 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[39364]: listed object 25... 2024-04-22T08:21:45.164 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[39364]: saw 50 objects 2024-04-22T08:21:45.164 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[39364]: shutting down. 2024-04-22T08:21:45.461 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:21:45.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.571+0000 7f78d1c10640 10 monclient: get_monmap_and_config 2024-04-22T08:21:45.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.571+0000 7f78d1c10640 10 monclient: build_initial_monmap 2024-04-22T08:21:45.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.571+0000 7f78d1c10640 10 monclient: monmap: 2024-04-22T08:21:45.571 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:45.571 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:45.571 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:21:45.572530+0000 2024-04-22T08:21:45.571 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:21:45.572530+0000 2024-04-22T08:21:45.571 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:45.571 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:45.572 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:21:45.572 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:21:45.572 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:21:45.572 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:21:45.572 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:21:45.572 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:45.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.572+0000 7f78d1c10640 1 Processor -- start 2024-04-22T08:21:45.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.572+0000 7f78d1c10640 1 -- start start 2024-04-22T08:21:45.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.572+0000 7f78d1c10640 10 monclient: init 2024-04-22T08:21:45.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.572+0000 7f78d1c10640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:45.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.572+0000 7f78d1c10640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-04-22T08:21:45.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.573+0000 7f78d1c10640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f78cc133bb0 0x7f78cc09aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:45.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.573+0000 7f78d1c10640 10 monclient(hunting): picked mon.noname-b con 0x7f78cc133bb0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:45.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.573+0000 7f78d1c10640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f78cc09b420 addr v1:172.21.15.40:6789/0 2024-04-22T08:21:45.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.573+0000 7f78d1c10640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f78cc1346a0 addr v1:172.21.15.189:6789/0 2024-04-22T08:21:45.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.573+0000 7f78d1c10640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78cc09a290 con 0x7f78cc09b420 2024-04-22T08:21:45.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.573+0000 7f78d1c10640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78cc09a500 con 0x7f78cc1346a0 2024-04-22T08:21:45.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.573+0000 7f78d1c10640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:45.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.573+0000 7f78d1c10640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78cc09a7e0 con 0x7f78cc133bb0 2024-04-22T08:21:45.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.573+0000 7f78d1c10640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:45.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.573+0000 7f78d1c10640 10 monclient(hunting): authenticate will time out at 1379.566844s 2024-04-22T08:21:45.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.573+0000 7f78cb7fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f78cc133bb0 0x7f78cc09aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:45.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.573+0000 7f78cb7fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f78cc133bb0 0x7f78cc09aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:41162/0 (socket says 172.21.15.40:41162) 2024-04-22T08:21:45.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.573+0000 7f78caffd640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f78cc09b420 0x7f78cc133f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:52076/0 (socket says 172.21.15.40:52076) 2024-04-22T08:21:45.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.573+0000 7f78cb7fe640 1 -- 172.21.15.40:0/1792898358 learned_addr learned my addr 172.21.15.40:0/1792898358 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:21:45.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.573+0000 7f78cb7fe640 10 monclient(hunting): get_auth_request con 0x7f78cc133bb0 auth_method 0 2024-04-22T08:21:45.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.573+0000 7f78cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:45.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.573+0000 7f78cb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:45.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.573+0000 7f78cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:45.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.573+0000 7f78cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:45.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.573+0000 7f78cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:45.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.573+0000 7f78cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:45.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.573+0000 7f78cb7fe640 10 monclient(hunting): handle_auth_done global_id 104276 payload 274 2024-04-22T08:21:45.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.573+0000 7f78cb7fe640 1 -- 172.21.15.40:0/1792898358 >> v1:172.21.15.189:6789/0 conn(0x7f78cc1346a0 legacy=0x7f78cc13d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:21:45.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.573+0000 7f78cb7fe640 1 -- 172.21.15.40:0/1792898358 >> v1:172.21.15.40:6789/0 conn(0x7f78cc09b420 legacy=0x7f78cc133f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:21:45.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.573+0000 7f78cb7fe640 10 monclient: _finish_hunting 0 2024-04-22T08:21:45.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.573+0000 7f78cb7fe640 1 monclient: found mon.noname-b 2024-04-22T08:21:45.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.573+0000 7f78cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:45.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.573+0000 7f78cb7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:21:45.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.573+0000 7f78cb7fe640 1 -- 172.21.15.40:0/1792898358 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78cc09aac0 con 0x7f78cc133bb0 2024-04-22T08:21:45.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.576+0000 7f78cb7fe640 1 -- 172.21.15.40:0/1792898358 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f78cc133bb0 msgr2=0x7f78cc09aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:45.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.576+0000 7f78cb7fe640 1 --2- 172.21.15.40:0/1792898358 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f78cc133bb0 0x7f78cc09aee0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f78bc0098a0 tx=0x7f78bc02b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2fef4c8ccf58fc3a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:45.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.576+0000 7f78ca7fc640 1 -- 172.21.15.40:0/1792898358 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78bc031ab0 con 0x7f78cc133bb0 2024-04-22T08:21:45.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.576+0000 7f78ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:45.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.576+0000 7f78ca7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:21:45.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.576+0000 7f78ca7fc640 10 monclient: dump: 2024-04-22T08:21:45.582 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:45.582 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:45.582 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:45.582 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:45.582 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:45.582 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:45.582 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:45.582 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:45.582 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:45.582 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:45.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.576+0000 7f78ca7fc640 10 monclient: _finish_auth 0 2024-04-22T08:21:45.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.576+0000 7f78ca7fc640 10 monclient: _check_auth_tickets 2024-04-22T08:21:45.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.576+0000 7f78ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:45.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.576+0000 7f78ca7fc640 1 -- 172.21.15.40:0/1792898358 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78bc031c10 con 0x7f78cc133bb0 2024-04-22T08:21:45.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.576+0000 7f78d1c10640 5 monclient: authenticate success, global_id 104276 2024-04-22T08:21:45.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.576+0000 7f78d1c10640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:21:45.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.576+0000 7f78ca7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:45.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.576+0000 7f78d1c10640 10 monclient: get_monmap_and_config success 2024-04-22T08:21:45.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.576+0000 7f78ca7fc640 1 -- 172.21.15.40:0/1792898358 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78bc032450 con 0x7f78cc133bb0 2024-04-22T08:21:45.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.576+0000 7f78ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:45.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.576+0000 7f78d1c10640 4 set_mon_vals no callback set 2024-04-22T08:21:45.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.576+0000 7f78ca7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:21:45.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.576+0000 7f78ca7fc640 10 monclient: dump: 2024-04-22T08:21:45.583 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:45.583 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:45.584 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:45.584 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:45.584 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:45.584 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:45.584 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:45.584 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:45.584 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:45.584 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:45.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.577+0000 7f78d1c10640 10 monclient: shutdown 2024-04-22T08:21:45.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.577+0000 7f78d1c10640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:45.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.577+0000 7f78d1c10640 1 -- 172.21.15.40:0/1792898358 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f78cc133bb0 msgr2=0x7f78cc09aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:45.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.577+0000 7f78d1c10640 1 --2- 172.21.15.40:0/1792898358 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f78cc133bb0 0x7f78cc09aee0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f78bc0098a0 tx=0x7f78bc02b860 comp rx=0 tx=0).stop 2024-04-22T08:21:45.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.577+0000 7f78d1c10640 1 -- 172.21.15.40:0/1792898358 shutdown_connections 2024-04-22T08:21:45.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.577+0000 7f78d1c10640 1 --2- 172.21.15.40:0/1792898358 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f78cc133bb0 0x7f78cc09aee0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:45.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.577+0000 7f78d1c10640 1 -- 172.21.15.40:0/1792898358 >> 172.21.15.40:0/1792898358 conn(0x7f78cc0a5960 msgr2=0x7f78cc0a1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:45.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.577+0000 7f78d1c10640 1 -- 172.21.15.40:0/1792898358 shutdown_connections 2024-04-22T08:21:45.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.577+0000 7f78d1c10640 1 -- 172.21.15.40:0/1792898358 wait complete. 2024-04-22T08:21:45.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.577+0000 7f78d1c10640 10 monclient: build_initial_monmap 2024-04-22T08:21:45.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.577+0000 7f78d1c10640 10 monclient: monmap: 2024-04-22T08:21:45.585 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:45.585 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:21:45.585 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:21:45.585 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:21:45.585 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:45.585 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:45.585 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:21:45.585 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:21:45.585 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:21:45.585 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:45.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.578+0000 7f78d1c10640 1 Processor -- start 2024-04-22T08:21:45.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.578+0000 7f78d1c10640 1 -- start start 2024-04-22T08:21:45.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.578+0000 7f78d1c10640 10 monclient: init 2024-04-22T08:21:45.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.578+0000 7f78d1c10640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:45.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.578+0000 7f78d1c10640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-22T08:21:45.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.578+0000 7f78d1c10640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f78cc09b420 0x7f78cc130ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:45.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.578+0000 7f78d1c10640 10 monclient(hunting): picked mon.noname-c con 0x7f78cc09b420 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:45.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.578+0000 7f78d1c10640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78cc133bb0 0x7f78cc131400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:45.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.578+0000 7f78d1c10640 10 monclient(hunting): picked mon.noname-a con 0x7f78cc133bb0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:21:45.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.578+0000 7f78d1c10640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78cc1346a0 0x7f78cc131940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:45.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.578+0000 7f78d1c10640 10 monclient(hunting): picked mon.noname-b con 0x7f78cc1346a0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:21:45.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.578+0000 7f78d1c10640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:45.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.578+0000 7f78d1c10640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78cc13de50 con 0x7f78cc133bb0 2024-04-22T08:21:45.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.578+0000 7f78d1c10640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:45.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.578+0000 7f78d1c10640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78cc136650 con 0x7f78cc1346a0 2024-04-22T08:21:45.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.578+0000 7f78d1c10640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:45.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.578+0000 7f78d1c10640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78cc136230 con 0x7f78cc09b420 2024-04-22T08:21:45.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.578+0000 7f78d1c10640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:45.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.578+0000 7f78d1c10640 10 monclient(hunting): authenticate will time out at 1379.572628s 2024-04-22T08:21:45.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.578+0000 7f78cb7fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f78cc09b420 0x7f78cc130ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:45.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.578+0000 7f78caffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78cc133bb0 0x7f78cc131400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:45.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.578+0000 7f78cb7fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f78cc09b420 0x7f78cc130ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:41178/0 (socket says 172.21.15.40:41178) 2024-04-22T08:21:45.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.578+0000 7f78cb7fe640 1 -- 172.21.15.40:0/2416898782 learned_addr learned my addr 172.21.15.40:0/2416898782 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:21:45.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.579+0000 7f78cb7fe640 10 monclient(hunting): get_auth_request con 0x7f78cc09b420 auth_method 0 2024-04-22T08:21:45.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.579+0000 7f78cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:45.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.579+0000 7f78cb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:45.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.579+0000 7f78cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:45.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.579+0000 7f78caffd640 10 monclient(hunting): get_auth_request con 0x7f78cc133bb0 auth_method 0 2024-04-22T08:21:45.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.579+0000 7f78caffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:45.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.579+0000 7f78caffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:45.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.579+0000 7f78caffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:45.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.579+0000 7f78cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:45.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.579+0000 7f78cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:45.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.579+0000 7f78cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:45.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.579+0000 7f78caffd640 1 -- 172.21.15.40:0/2416898782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78cc133bb0 msgr2=0x7f78cc131400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-22T08:21:45.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.579+0000 7f78caffd640 1 -- 172.21.15.40:0/2416898782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78cc133bb0 msgr2=0x7f78cc131400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:21:45.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.579+0000 7f78caffd640 1 --2- 172.21.15.40:0/2416898782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78cc133bb0 0x7f78cc131400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:21:45.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.579+0000 7f78caffd640 1 --2- 172.21.15.40:0/2416898782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78cc133bb0 0x7f78cc131400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:21:45.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.579+0000 7f78cb7fe640 10 monclient(hunting): handle_auth_done global_id 104279 payload 824 2024-04-22T08:21:45.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.579+0000 7f78cb7fe640 1 -- 172.21.15.40:0/2416898782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78cc1346a0 msgr2=0x7f78cc131940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:45.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.579+0000 7f78cbfff640 1 --2- 172.21.15.40:0/2416898782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78cc1346a0 0x7f78cc131940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:45.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.579+0000 7f78cb7fe640 1 --2- 172.21.15.40:0/2416898782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78cc1346a0 0x7f78cc131940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:45.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.579+0000 7f78cb7fe640 1 -- 172.21.15.40:0/2416898782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78cc133bb0 msgr2=0x7f78cc131400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:21:45.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.579+0000 7f78cb7fe640 1 --2- 172.21.15.40:0/2416898782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78cc133bb0 0x7f78cc131400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:45.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.579+0000 7f78cb7fe640 10 monclient: _finish_hunting 0 2024-04-22T08:21:45.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.579+0000 7f78cb7fe640 1 monclient: found mon.noname-c 2024-04-22T08:21:45.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.579+0000 7f78cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:45.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.579+0000 7f78cb7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:21:45.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.579+0000 7f78cb7fe640 1 -- 172.21.15.40:0/2416898782 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78bc0094d0 con 0x7f78cc09b420 2024-04-22T08:21:45.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.579+0000 7f78cbfff640 10 monclient: get_auth_request con 0x7f78cc1346a0 auth_method 0 2024-04-22T08:21:45.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.579+0000 7f78cbfff640 1 --2- 172.21.15.40:0/2416898782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78cc1346a0 0x7f78cc131940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T08:21:45.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.581+0000 7f78cb7fe640 1 -- 172.21.15.40:0/2416898782 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f78cc09b420 msgr2=0x7f78cc130ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:45.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.581+0000 7f78cb7fe640 1 --2- 172.21.15.40:0/2416898782 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f78cc09b420 0x7f78cc130ec0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f78bc0090b0 tx=0x7f78bc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:45.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.581+0000 7f78b3fff640 1 -- 172.21.15.40:0/2416898782 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78bc032ad0 con 0x7f78cc09b420 2024-04-22T08:21:45.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.581+0000 7f78b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:45.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.581+0000 7f78b3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:21:45.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.581+0000 7f78b3fff640 10 monclient: dump: 2024-04-22T08:21:45.590 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:45.590 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:45.590 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:45.590 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:45.590 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:45.590 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:45.590 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:45.590 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:45.590 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:45.590 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:45.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.581+0000 7f78b3fff640 10 monclient: _finish_auth 0 2024-04-22T08:21:45.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.581+0000 7f78b3fff640 10 monclient: _check_auth_tickets 2024-04-22T08:21:45.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.581+0000 7f78b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:45.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.581+0000 7f78b3fff640 1 -- 172.21.15.40:0/2416898782 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78bc032c30 con 0x7f78cc09b420 2024-04-22T08:21:45.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.581+0000 7f78b3fff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:45.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.581+0000 7f78b3fff640 1 -- 172.21.15.40:0/2416898782 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78bc0453f0 con 0x7f78cc09b420 2024-04-22T08:21:45.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.581+0000 7f78d1c10640 5 monclient: authenticate success, global_id 104279 2024-04-22T08:21:45.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.581+0000 7f78b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:45.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.581+0000 7f78b3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:21:45.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.581+0000 7f78b3fff640 10 monclient: dump: 2024-04-22T08:21:45.591 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:45.591 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:45.591 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:45.591 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:45.591 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:45.592 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:45.592 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:45.592 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:45.592 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:45.592 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:45.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.581+0000 7f78c9ffb640 4 set_mon_vals no callback set 2024-04-22T08:21:45.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.581+0000 7f78d1c10640 10 monclient: _renew_subs 2024-04-22T08:21:45.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.581+0000 7f78d1c10640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:21:45.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.581+0000 7f78d1c10640 1 -- 172.21.15.40:0/2416898782 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78cc09b8c0 con 0x7f78cc09b420 2024-04-22T08:21:45.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.581+0000 7f78d1c10640 10 monclient: _renew_subs 2024-04-22T08:21:45.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.581+0000 7f78d1c10640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:21:45.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.581+0000 7f78d1c10640 1 -- 172.21.15.40:0/2416898782 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78cc133410 con 0x7f78cc09b420 2024-04-22T08:21:45.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.582+0000 7f78b3fff640 1 -- 172.21.15.40:0/2416898782 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f78bc04d3f0 con 0x7f78cc09b420 2024-04-22T08:21:45.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.583+0000 7f78d1c10640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:45.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.583+0000 7f78d1c10640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:45.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.583+0000 7f78d1c10640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:21:45.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.583+0000 7f78d1c10640 1 -- 172.21.15.40:0/2416898782 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78cc075260 con 0x7f78cc09b420 2024-04-22T08:21:45.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.583+0000 7f78b3fff640 1 --2- 172.21.15.40:0/2416898782 >> v2:172.21.15.189:6800/30168 conn(0x7f78b80377d0 0x7f78b8039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:45.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.583+0000 7f78b3fff640 1 -- 172.21.15.40:0/2416898782 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(303..303 src has 239..303) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f78bc06c330 con 0x7f78cc09b420 2024-04-22T08:21:45.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.583+0000 7f78caffd640 1 --2- 172.21.15.40:0/2416898782 >> v2:172.21.15.189:6800/30168 conn(0x7f78b80377d0 0x7f78b8039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:45.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.583+0000 7f78caffd640 10 monclient: get_auth_request con 0x7f78b80377d0 auth_method 0 2024-04-22T08:21:45.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.586+0000 7f78caffd640 1 --2- 172.21.15.40:0/2416898782 >> v2:172.21.15.189:6800/30168 conn(0x7f78b80377d0 0x7f78b8039c90 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f78cc134640 tx=0x7f78c0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:45.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.588+0000 7f78b3fff640 1 -- 172.21.15.40:0/2416898782 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f78bc030900 con 0x7f78cc09b420 2024-04-22T08:21:45.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.588+0000 7f78b3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:45.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.588+0000 7f78b3fff640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:21:45.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.841+0000 7f78d1c10640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:21:45.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.841+0000 7f78d1c10640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:21:45.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.841+0000 7f78d1c10640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:21:45.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.841+0000 7f78d1c10640 1 -- 172.21.15.40:0/2416898782 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f78cc075260 con 0x7f78cc09b420 2024-04-22T08:21:45.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.841+0000 7f78b3fff640 1 -- 172.21.15.40:0/2416898782 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7f78bc03a380 con 0x7f78cc09b420 2024-04-22T08:21:45.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.841+0000 7f78b3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:21:45.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.841+0000 7f78b3fff640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:21:45.841 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":84,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":26,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:21:45.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.843+0000 7f78d1c10640 1 -- 172.21.15.40:0/2416898782 >> v2:172.21.15.189:6800/30168 conn(0x7f78b80377d0 msgr2=0x7f78b8039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:45.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.843+0000 7f78d1c10640 1 --2- 172.21.15.40:0/2416898782 >> v2:172.21.15.189:6800/30168 conn(0x7f78b80377d0 0x7f78b8039c90 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f78cc134640 tx=0x7f78c0009040 comp rx=0 tx=0).stop 2024-04-22T08:21:45.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.843+0000 7f78d1c10640 10 monclient: shutdown 2024-04-22T08:21:45.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.843+0000 7f78d1c10640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:45.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.843+0000 7f78d1c10640 1 -- 172.21.15.40:0/2416898782 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f78cc09b420 msgr2=0x7f78cc130ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:45.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.843+0000 7f78d1c10640 1 --2- 172.21.15.40:0/2416898782 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f78cc09b420 0x7f78cc130ec0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f78bc0090b0 tx=0x7f78bc0046c0 comp rx=0 tx=0).stop 2024-04-22T08:21:45.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.843+0000 7f78d1c10640 1 -- 172.21.15.40:0/2416898782 shutdown_connections 2024-04-22T08:21:45.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.843+0000 7f78d1c10640 1 --2- 172.21.15.40:0/2416898782 >> v2:172.21.15.189:6800/30168 conn(0x7f78b80377d0 0x7f78b8039c90 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:45.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.843+0000 7f78d1c10640 1 --2- 172.21.15.40:0/2416898782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78cc1346a0 0x7f78cc131940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:45.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.843+0000 7f78d1c10640 1 --2- 172.21.15.40:0/2416898782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78cc133bb0 0x7f78cc131400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:45.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.843+0000 7f78d1c10640 1 --2- 172.21.15.40:0/2416898782 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f78cc09b420 0x7f78cc130ec0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:45.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.843+0000 7f78d1c10640 1 -- 172.21.15.40:0/2416898782 >> 172.21.15.40:0/2416898782 conn(0x7f78cc0a5960 msgr2=0x7f78cc0a2e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:45.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.843+0000 7f78d1c10640 1 -- 172.21.15.40:0/2416898782 shutdown_connections 2024-04-22T08:21:45.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:45.843+0000 7f78d1c10640 1 -- 172.21.15.40:0/2416898782 wait complete. 2024-04-22T08:21:46.003 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:21:46.003 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[39363]: starting. 2024-04-22T08:21:46.003 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[39363]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-39325 2024-04-22T08:21:46.003 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[39363]: shutting down. 2024-04-22T08:21:46.004 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:21:46.004 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:21:46.004 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:21:46.007 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:21:48.293 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[39433]: starting. 2024-04-22T08:21:48.293 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[39433]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-39432 2024-04-22T08:21:48.293 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[39433]: created object 0... 2024-04-22T08:21:48.293 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[39433]: created object 25... 2024-04-22T08:21:48.293 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[39433]: created object 49... 2024-04-22T08:21:48.293 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[39433]: finishing. 2024-04-22T08:21:48.293 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[39433]: shutting down. 2024-04-22T08:21:48.854 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:21:48.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:48.964+0000 7f9ae46db640 10 monclient: get_monmap_and_config 2024-04-22T08:21:48.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:48.964+0000 7f9ae46db640 10 monclient: build_initial_monmap 2024-04-22T08:21:48.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:48.964+0000 7f9ae46db640 10 monclient: monmap: 2024-04-22T08:21:48.964 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:48.964 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:48.964 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:21:48.965528+0000 2024-04-22T08:21:48.964 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:21:48.965528+0000 2024-04-22T08:21:48.964 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:48.964 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:48.965 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:21:48.965 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:21:48.965 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:21:48.965 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:21:48.965 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:21:48.965 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:48.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:48.965+0000 7f9ae46db640 1 Processor -- start 2024-04-22T08:21:48.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:48.965+0000 7f9ae46db640 1 -- start start 2024-04-22T08:21:48.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:48.965+0000 7f9ae46db640 10 monclient: init 2024-04-22T08:21:48.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:48.965+0000 7f9ae46db640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:48.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:48.965+0000 7f9ae46db640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-04-22T08:21:48.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:48.966+0000 7f9ae46db640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9adc135e30 addr v1:172.21.15.40:6789/0 2024-04-22T08:21:48.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:48.966+0000 7f9ae46db640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9adc071c90 0x7f9adc13aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:48.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:48.966+0000 7f9ae46db640 10 monclient(hunting): picked mon.noname-c con 0x7f9adc071c90 addr v2:172.21.15.189:3300/0 2024-04-22T08:21:48.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:48.966+0000 7f9ae46db640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9adc13b410 addr v1:172.21.15.189:6789/0 2024-04-22T08:21:48.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:48.966+0000 7f9ae46db640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9adc09de80 con 0x7f9adc135e30 2024-04-22T08:21:48.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:48.966+0000 7f9ae46db640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9adc09f130 con 0x7f9adc13b410 2024-04-22T08:21:48.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:48.966+0000 7f9ae46db640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:48.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:48.966+0000 7f9ae46db640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9adc09e3d0 con 0x7f9adc071c90 2024-04-22T08:21:48.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:48.966+0000 7f9ae46db640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:48.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:48.966+0000 7f9ae46db640 10 monclient(hunting): authenticate will time out at 1382.959835s 2024-04-22T08:21:48.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:48.966+0000 7f9ae2477640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f9adc135e30 0x7f9adc136210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:36560/0 (socket says 172.21.15.40:36560) 2024-04-22T08:21:48.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:48.966+0000 7f9ae2477640 1 -- 172.21.15.40:0/1508232717 learned_addr learned my addr 172.21.15.40:0/1508232717 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:21:48.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:48.966+0000 7f9ae1c76640 1 --2- 172.21.15.40:0/1508232717 >> v2:172.21.15.189:3300/0 conn(0x7f9adc071c90 0x7f9adc13aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:48.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:48.966+0000 7f9ae1c76640 10 monclient(hunting): get_auth_request con 0x7f9adc071c90 auth_method 0 2024-04-22T08:21:48.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:48.966+0000 7f9ae1c76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:48.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:48.966+0000 7f9ae1c76640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:48.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:48.966+0000 7f9ae1c76640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:48.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:48.966+0000 7f9ae1c76640 1 -- 172.21.15.40:0/1508232717 >> v2:172.21.15.189:3300/0 conn(0x7f9adc071c90 msgr2=0x7f9adc13aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:21:48.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:48.966+0000 7f9ae1c76640 1 -- 172.21.15.40:0/1508232717 >> v2:172.21.15.189:3300/0 conn(0x7f9adc071c90 msgr2=0x7f9adc13aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:21:48.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:48.966+0000 7f9ae1c76640 1 --2- 172.21.15.40:0/1508232717 >> v2:172.21.15.189:3300/0 conn(0x7f9adc071c90 0x7f9adc13aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:21:48.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:48.966+0000 7f9ae1c76640 1 --2- 172.21.15.40:0/1508232717 >> v2:172.21.15.189:3300/0 conn(0x7f9adc071c90 0x7f9adc13aed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:21:48.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:48.968+0000 7f9ae2477640 1 -- 172.21.15.40:0/1508232717 >> v1:172.21.15.40:6789/0 conn(0x7f9adc135e30 legacy=0x7f9adc136210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:48.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:48.968+0000 7f9ae2c78640 1 -- 172.21.15.40:0/1508232717 >> v1:172.21.15.189:6789/0 conn(0x7f9adc13b410 legacy=0x7f9adc13d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:49.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:49.167+0000 7f9ae1c76640 1 --2- 172.21.15.40:0/1508232717 >> v2:172.21.15.189:3300/0 conn(0x7f9adc071c90 0x7f9adc13aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:49.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:49.167+0000 7f9ae1c76640 10 monclient(hunting): get_auth_request con 0x7f9adc071c90 auth_method 0 2024-04-22T08:21:49.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:49.167+0000 7f9ae1c76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:49.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:49.167+0000 7f9ae1c76640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:49.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:49.167+0000 7f9ae1c76640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:21:49.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:49.167+0000 7f9ae1c76640 1 -- 172.21.15.40:0/1508232717 >> v2:172.21.15.189:3300/0 conn(0x7f9adc071c90 msgr2=0x7f9adc13aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:21:49.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:49.167+0000 7f9ae1c76640 1 -- 172.21.15.40:0/1508232717 >> v2:172.21.15.189:3300/0 conn(0x7f9adc071c90 msgr2=0x7f9adc13aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:21:49.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:49.167+0000 7f9ae1c76640 1 --2- 172.21.15.40:0/1508232717 >> v2:172.21.15.189:3300/0 conn(0x7f9adc071c90 0x7f9adc13aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:21:49.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:49.167+0000 7f9ae1c76640 1 --2- 172.21.15.40:0/1508232717 >> v2:172.21.15.189:3300/0 conn(0x7f9adc071c90 0x7f9adc13aed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:21:49.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:49.568+0000 7f9ae1c76640 1 --2- 172.21.15.40:0/1508232717 >> v2:172.21.15.189:3300/0 conn(0x7f9adc071c90 0x7f9adc13aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:49.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:49.568+0000 7f9ae1c76640 10 monclient(hunting): get_auth_request con 0x7f9adc071c90 auth_method 0 2024-04-22T08:21:49.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:49.568+0000 7f9ae1c76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:49.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:49.568+0000 7f9ae1c76640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:49.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:49.568+0000 7f9ae1c76640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:21:49.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:49.568+0000 7f9ae1c76640 1 -- 172.21.15.40:0/1508232717 >> v2:172.21.15.189:3300/0 conn(0x7f9adc071c90 msgr2=0x7f9adc13aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:21:49.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:49.568+0000 7f9ae1c76640 1 -- 172.21.15.40:0/1508232717 >> v2:172.21.15.189:3300/0 conn(0x7f9adc071c90 msgr2=0x7f9adc13aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:21:49.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:49.568+0000 7f9ae1c76640 1 --2- 172.21.15.40:0/1508232717 >> v2:172.21.15.189:3300/0 conn(0x7f9adc071c90 0x7f9adc13aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:21:49.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:49.568+0000 7f9ae1c76640 1 --2- 172.21.15.40:0/1508232717 >> v2:172.21.15.189:3300/0 conn(0x7f9adc071c90 0x7f9adc13aed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:21:50.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:50.369+0000 7f9ae1c76640 1 --2- 172.21.15.40:0/1508232717 >> v2:172.21.15.189:3300/0 conn(0x7f9adc071c90 0x7f9adc13aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:50.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:50.369+0000 7f9ae1c76640 10 monclient(hunting): get_auth_request con 0x7f9adc071c90 auth_method 0 2024-04-22T08:21:50.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:50.369+0000 7f9ae1c76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:50.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:50.369+0000 7f9ae1c76640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:50.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:50.369+0000 7f9ae1c76640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:21:50.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:50.369+0000 7f9ae1c76640 1 -- 172.21.15.40:0/1508232717 >> v2:172.21.15.189:3300/0 conn(0x7f9adc071c90 msgr2=0x7f9adc13aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:21:50.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:50.369+0000 7f9ae1c76640 1 -- 172.21.15.40:0/1508232717 >> v2:172.21.15.189:3300/0 conn(0x7f9adc071c90 msgr2=0x7f9adc13aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:21:50.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:50.369+0000 7f9ae1c76640 1 --2- 172.21.15.40:0/1508232717 >> v2:172.21.15.189:3300/0 conn(0x7f9adc071c90 0x7f9adc13aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:21:50.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:50.370+0000 7f9ae1c76640 1 --2- 172.21.15.40:0/1508232717 >> v2:172.21.15.189:3300/0 conn(0x7f9adc071c90 0x7f9adc13aed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T08:21:51.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:51.966+0000 7f9ad3fff640 10 monclient(hunting): tick 2024-04-22T08:21:51.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:51.966+0000 7f9ad3fff640 10 monclient(hunting): _check_auth_tickets 2024-04-22T08:21:51.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:51.966+0000 7f9ad3fff640 1 monclient(hunting): continuing hunt 2024-04-22T08:21:51.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:51.966+0000 7f9ad3fff640 10 monclient(hunting): _reopen_session rank -1 2024-04-22T08:21:51.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:51.966+0000 7f9ad3fff640 1 -- 172.21.15.40:0/1508232717 >> v2:172.21.15.189:3300/0 conn(0x7f9adc071c90 msgr2=0x7f9adc13aed0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:21:51.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:51.966+0000 7f9ad3fff640 1 --2- 172.21.15.40:0/1508232717 >> v2:172.21.15.189:3300/0 conn(0x7f9adc071c90 0x7f9adc13aed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:51.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:51.966+0000 7f9ad3fff640 1 -- 172.21.15.40:0/1508232717 >> v1:172.21.15.189:6789/0 conn(0x7f9adc13b410 legacy=0x7f9adc13d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:51.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:51.966+0000 7f9ad3fff640 1 -- 172.21.15.40:0/1508232717 >> v1:172.21.15.40:6789/0 conn(0x7f9adc135e30 legacy=0x7f9adc136210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:51.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:51.966+0000 7f9ad3fff640 10 monclient: _add_conns ranks=[3,2] 2024-04-22T08:21:51.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:51.966+0000 7f9ad3fff640 1 --2- 172.21.15.40:0/1508232717 >> v2:172.21.15.40:3300/0 conn(0x7f9ac0003810 0x7f9ac0005c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:51.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:51.966+0000 7f9ad3fff640 10 monclient(hunting): picked mon.noname-a con 0x7f9ac0003810 addr v2:172.21.15.40:3300/0 2024-04-22T08:21:51.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:51.966+0000 7f9ad3fff640 1 --2- 172.21.15.40:0/1508232717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ac0006350 0x7f9ac00087f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:51.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:51.966+0000 7f9ad3fff640 10 monclient(hunting): picked mon.noname-b con 0x7f9ac0006350 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:51.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:51.966+0000 7f9ad3fff640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:51.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:51.966+0000 7f9ad3fff640 1 -- 172.21.15.40:0/1508232717 --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ac0008ef0 con 0x7f9ac0003810 2024-04-22T08:21:51.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:51.966+0000 7f9ad3fff640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:51.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:51.966+0000 7f9ad3fff640 1 -- 172.21.15.40:0/1508232717 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac0009060 con 0x7f9ac0006350 2024-04-22T08:21:51.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:51.966+0000 7f9ad3fff640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:51.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:51.966+0000 7f9ae1c76640 1 --2- 172.21.15.40:0/1508232717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ac0006350 0x7f9ac00087f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:51.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:51.966+0000 7f9ae2477640 1 --2- 172.21.15.40:0/1508232717 >> v2:172.21.15.40:3300/0 conn(0x7f9ac0003810 0x7f9ac0005c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:51.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:51.966+0000 7f9ae2477640 10 monclient(hunting): get_auth_request con 0x7f9ac0003810 auth_method 0 2024-04-22T08:21:51.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:51.966+0000 7f9ae2477640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:51.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:51.966+0000 7f9ae2477640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:51.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:51.966+0000 7f9ae2477640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:51.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:51.966+0000 7f9ae1c76640 10 monclient(hunting): get_auth_request con 0x7f9ac0006350 auth_method 0 2024-04-22T08:21:51.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:51.966+0000 7f9ae1c76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:51.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:51.966+0000 7f9ae1c76640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:51.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:51.966+0000 7f9ae1c76640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:51.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:51.966+0000 7f9ae2477640 1 -- 172.21.15.40:0/1508232717 >> v2:172.21.15.40:3300/0 conn(0x7f9ac0003810 msgr2=0x7f9ac0005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:21:51.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:51.966+0000 7f9ae2477640 1 -- 172.21.15.40:0/1508232717 >> v2:172.21.15.40:3300/0 conn(0x7f9ac0003810 msgr2=0x7f9ac0005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:21:51.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:51.966+0000 7f9ae1c76640 1 -- 172.21.15.40:0/1508232717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ac0006350 msgr2=0x7f9ac00087f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:21:51.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:51.966+0000 7f9ae2477640 1 --2- 172.21.15.40:0/1508232717 >> v2:172.21.15.40:3300/0 conn(0x7f9ac0003810 0x7f9ac0005c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:21:51.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:51.966+0000 7f9ae1c76640 1 -- 172.21.15.40:0/1508232717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ac0006350 msgr2=0x7f9ac00087f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:21:51.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:51.966+0000 7f9ae1c76640 1 --2- 172.21.15.40:0/1508232717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ac0006350 0x7f9ac00087f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:21:51.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:51.966+0000 7f9ae2477640 1 --2- 172.21.15.40:0/1508232717 >> v2:172.21.15.40:3300/0 conn(0x7f9ac0003810 0x7f9ac0005c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:21:51.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:51.966+0000 7f9ae1c76640 1 --2- 172.21.15.40:0/1508232717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ac0006350 0x7f9ac00087f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:21:52.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:52.167+0000 7f9ae1c76640 1 --2- 172.21.15.40:0/1508232717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ac0006350 0x7f9ac00087f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:52.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:52.167+0000 7f9ae2477640 1 --2- 172.21.15.40:0/1508232717 >> v2:172.21.15.40:3300/0 conn(0x7f9ac0003810 0x7f9ac0005c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:52.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:52.167+0000 7f9ae2477640 10 monclient(hunting): get_auth_request con 0x7f9ac0003810 auth_method 0 2024-04-22T08:21:52.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:52.167+0000 7f9ae2477640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:52.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:52.167+0000 7f9ae2477640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:52.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:52.167+0000 7f9ae2477640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:21:52.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:52.167+0000 7f9ae1c76640 10 monclient(hunting): get_auth_request con 0x7f9ac0006350 auth_method 0 2024-04-22T08:21:52.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:52.167+0000 7f9ae1c76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:52.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:52.167+0000 7f9ae1c76640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:52.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:52.167+0000 7f9ae1c76640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:21:52.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:52.167+0000 7f9ae2477640 1 -- 172.21.15.40:0/1508232717 >> v2:172.21.15.40:3300/0 conn(0x7f9ac0003810 msgr2=0x7f9ac0005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:21:52.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:52.167+0000 7f9ae2477640 1 -- 172.21.15.40:0/1508232717 >> v2:172.21.15.40:3300/0 conn(0x7f9ac0003810 msgr2=0x7f9ac0005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:21:52.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:52.167+0000 7f9ae2477640 1 --2- 172.21.15.40:0/1508232717 >> v2:172.21.15.40:3300/0 conn(0x7f9ac0003810 0x7f9ac0005c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:21:52.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:52.167+0000 7f9ae1c76640 1 -- 172.21.15.40:0/1508232717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ac0006350 msgr2=0x7f9ac00087f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:21:52.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:52.167+0000 7f9ae1c76640 1 -- 172.21.15.40:0/1508232717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ac0006350 msgr2=0x7f9ac00087f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:21:52.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:52.167+0000 7f9ae1c76640 1 --2- 172.21.15.40:0/1508232717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ac0006350 0x7f9ac00087f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:21:52.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:52.167+0000 7f9ae2477640 1 --2- 172.21.15.40:0/1508232717 >> v2:172.21.15.40:3300/0 conn(0x7f9ac0003810 0x7f9ac0005c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:21:52.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:52.167+0000 7f9ae1c76640 1 --2- 172.21.15.40:0/1508232717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ac0006350 0x7f9ac00087f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:21:52.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:52.568+0000 7f9ae2477640 1 --2- 172.21.15.40:0/1508232717 >> v2:172.21.15.40:3300/0 conn(0x7f9ac0003810 0x7f9ac0005c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:52.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:52.568+0000 7f9ae1c76640 1 --2- 172.21.15.40:0/1508232717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ac0006350 0x7f9ac00087f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:52.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:52.568+0000 7f9ae2477640 10 monclient(hunting): get_auth_request con 0x7f9ac0003810 auth_method 0 2024-04-22T08:21:52.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:52.568+0000 7f9ae2477640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:52.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:52.568+0000 7f9ae2477640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:52.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:52.568+0000 7f9ae2477640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:21:52.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:52.568+0000 7f9ae1c76640 10 monclient(hunting): get_auth_request con 0x7f9ac0006350 auth_method 0 2024-04-22T08:21:52.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:52.568+0000 7f9ae1c76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:52.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:52.568+0000 7f9ae1c76640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:52.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:52.568+0000 7f9ae1c76640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:21:52.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:52.568+0000 7f9ae2477640 1 -- 172.21.15.40:0/1508232717 >> v2:172.21.15.40:3300/0 conn(0x7f9ac0003810 msgr2=0x7f9ac0005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:21:52.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:52.568+0000 7f9ae2477640 1 -- 172.21.15.40:0/1508232717 >> v2:172.21.15.40:3300/0 conn(0x7f9ac0003810 msgr2=0x7f9ac0005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:21:52.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:52.568+0000 7f9ae2477640 1 --2- 172.21.15.40:0/1508232717 >> v2:172.21.15.40:3300/0 conn(0x7f9ac0003810 0x7f9ac0005c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:21:52.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:52.568+0000 7f9ae1c76640 1 -- 172.21.15.40:0/1508232717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ac0006350 msgr2=0x7f9ac00087f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:21:52.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:52.568+0000 7f9ae1c76640 1 -- 172.21.15.40:0/1508232717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ac0006350 msgr2=0x7f9ac00087f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:21:52.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:52.568+0000 7f9ae1c76640 1 --2- 172.21.15.40:0/1508232717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ac0006350 0x7f9ac00087f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:21:52.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:52.568+0000 7f9ae2477640 1 --2- 172.21.15.40:0/1508232717 >> v2:172.21.15.40:3300/0 conn(0x7f9ac0003810 0x7f9ac0005c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:21:52.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:52.568+0000 7f9ae1c76640 1 --2- 172.21.15.40:0/1508232717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ac0006350 0x7f9ac00087f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:21:53.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:53.369+0000 7f9ae2477640 1 --2- 172.21.15.40:0/1508232717 >> v2:172.21.15.40:3300/0 conn(0x7f9ac0003810 0x7f9ac0005c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:53.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:53.369+0000 7f9ae1c76640 1 --2- 172.21.15.40:0/1508232717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ac0006350 0x7f9ac00087f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:53.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:53.369+0000 7f9ae2477640 10 monclient(hunting): get_auth_request con 0x7f9ac0003810 auth_method 0 2024-04-22T08:21:53.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:53.369+0000 7f9ae2477640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:53.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:53.369+0000 7f9ae2477640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:53.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:53.369+0000 7f9ae2477640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:21:53.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:53.369+0000 7f9ae1c76640 10 monclient(hunting): get_auth_request con 0x7f9ac0006350 auth_method 0 2024-04-22T08:21:53.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:53.369+0000 7f9ae1c76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:53.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:53.369+0000 7f9ae1c76640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:53.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:53.369+0000 7f9ae1c76640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:21:53.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:53.370+0000 7f9ae1c76640 1 -- 172.21.15.40:0/1508232717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ac0006350 msgr2=0x7f9ac00087f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:21:53.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:53.370+0000 7f9ae1c76640 1 -- 172.21.15.40:0/1508232717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ac0006350 msgr2=0x7f9ac00087f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:21:53.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:53.370+0000 7f9ae1c76640 1 --2- 172.21.15.40:0/1508232717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ac0006350 0x7f9ac00087f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:21:53.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:53.370+0000 7f9ae2477640 1 -- 172.21.15.40:0/1508232717 >> v2:172.21.15.40:3300/0 conn(0x7f9ac0003810 msgr2=0x7f9ac0005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:21:53.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:53.370+0000 7f9ae2477640 1 -- 172.21.15.40:0/1508232717 >> v2:172.21.15.40:3300/0 conn(0x7f9ac0003810 msgr2=0x7f9ac0005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:21:53.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:53.370+0000 7f9ae2477640 1 --2- 172.21.15.40:0/1508232717 >> v2:172.21.15.40:3300/0 conn(0x7f9ac0003810 0x7f9ac0005c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:21:53.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:53.370+0000 7f9ae1c76640 1 --2- 172.21.15.40:0/1508232717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ac0006350 0x7f9ac00087f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T08:21:53.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:53.370+0000 7f9ae2477640 1 --2- 172.21.15.40:0/1508232717 >> v2:172.21.15.40:3300/0 conn(0x7f9ac0003810 0x7f9ac0005c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T08:21:53.491 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[39434]: starting. 2024-04-22T08:21:53.491 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[39434]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-39432 2024-04-22T08:21:53.491 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[39434]: shutting down. 2024-04-22T08:21:54.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.966+0000 7f9ad3fff640 10 monclient(hunting): tick 2024-04-22T08:21:54.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.966+0000 7f9ad3fff640 10 monclient(hunting): _check_auth_tickets 2024-04-22T08:21:54.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.966+0000 7f9ad3fff640 1 monclient(hunting): continuing hunt 2024-04-22T08:21:54.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.966+0000 7f9ad3fff640 10 monclient(hunting): _reopen_session rank -1 2024-04-22T08:21:54.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.966+0000 7f9ad3fff640 1 -- 172.21.15.40:0/1508232717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ac0006350 msgr2=0x7f9ac00087f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:21:54.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.966+0000 7f9ad3fff640 1 --2- 172.21.15.40:0/1508232717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ac0006350 0x7f9ac00087f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:54.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.966+0000 7f9ad3fff640 1 -- 172.21.15.40:0/1508232717 >> v2:172.21.15.40:3300/0 conn(0x7f9ac0003810 msgr2=0x7f9ac0005c70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:21:54.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.966+0000 7f9ad3fff640 1 --2- 172.21.15.40:0/1508232717 >> v2:172.21.15.40:3300/0 conn(0x7f9ac0003810 0x7f9ac0005c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:54.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.966+0000 7f9ad3fff640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-04-22T08:21:54.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.966+0000 7f9ae2c78640 1 -- 172.21.15.40:0/1508232717 reap_dead start 2024-04-22T08:21:54.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.966+0000 7f9ad3fff640 1 --2- 172.21.15.40:0/1508232717 >> v2:172.21.15.189:3300/0 conn(0x7f9adc071c90 0x7f9ac000b490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:54.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.966+0000 7f9ad3fff640 10 monclient(hunting): picked mon.noname-c con 0x7f9adc071c90 addr v2:172.21.15.189:3300/0 2024-04-22T08:21:54.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.966+0000 7f9ad3fff640 1 --2- 172.21.15.40:0/1508232717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ac0003c60 0x7f9ac00067e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:54.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.966+0000 7f9ad3fff640 10 monclient(hunting): picked mon.noname-b con 0x7f9ac0003c60 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:54.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.966+0000 7f9ad3fff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9ac0006d20 addr v1:172.21.15.40:6789/0 2024-04-22T08:21:54.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.966+0000 7f9ad3fff640 1 -- 172.21.15.40:0/1508232717 --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9adc09f410 con 0x7f9ac0006d20 2024-04-22T08:21:54.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.966+0000 7f9ad3fff640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:54.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.966+0000 7f9ad3fff640 1 -- 172.21.15.40:0/1508232717 --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ac0008ef0 con 0x7f9adc071c90 2024-04-22T08:21:54.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.966+0000 7f9ad3fff640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:54.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.966+0000 7f9ad3fff640 1 -- 172.21.15.40:0/1508232717 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac0009060 con 0x7f9ac0003c60 2024-04-22T08:21:54.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.966+0000 7f9ad3fff640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:54.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.967+0000 7f9ae1c76640 1 --2- 172.21.15.40:0/1508232717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ac0003c60 0x7f9ac00067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:54.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.967+0000 7f9ae1c76640 10 monclient(hunting): get_auth_request con 0x7f9ac0003c60 auth_method 0 2024-04-22T08:21:54.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.967+0000 7f9ae1c76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:54.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.967+0000 7f9ae1c76640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:54.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.967+0000 7f9ae1c76640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:54.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.967+0000 7f9ae2477640 1 --2- 172.21.15.40:0/1508232717 >> v2:172.21.15.189:3300/0 conn(0x7f9adc071c90 0x7f9ac000b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:54.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.967+0000 7f9ae2477640 10 monclient(hunting): get_auth_request con 0x7f9adc071c90 auth_method 0 2024-04-22T08:21:54.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.967+0000 7f9ae2477640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:54.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.967+0000 7f9ae2477640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:54.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.967+0000 7f9ae2477640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:54.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.967+0000 7f9ae1c76640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:54.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.967+0000 7f9ae1c76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:54.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.967+0000 7f9ae1c76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:54.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.967+0000 7f9ae2477640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:54.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.967+0000 7f9ae2477640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:54.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.967+0000 7f9ae2477640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:54.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.967+0000 7f9ae1c76640 10 monclient(hunting): handle_auth_done global_id 104288 payload 274 2024-04-22T08:21:54.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.967+0000 7f9ae1c76640 1 -- 172.21.15.40:0/1508232717 >> v2:172.21.15.189:3300/0 conn(0x7f9adc071c90 msgr2=0x7f9ac000b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:21:54.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.967+0000 7f9ae1c76640 1 --2- 172.21.15.40:0/1508232717 >> v2:172.21.15.189:3300/0 conn(0x7f9adc071c90 0x7f9ac000b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:54.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.967+0000 7f9ae1c76640 1 -- 172.21.15.40:0/1508232717 >> v1:172.21.15.40:6789/0 conn(0x7f9ac0006d20 legacy=0x7f9ac0007180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:21:54.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.967+0000 7f9ae1c76640 10 monclient: _finish_hunting 0 2024-04-22T08:21:54.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.967+0000 7f9ae1c76640 1 monclient: found mon.noname-b 2024-04-22T08:21:54.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.967+0000 7f9ae1c76640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:54.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.967+0000 7f9ae1c76640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:21:54.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.967+0000 7f9ae1c76640 1 -- 172.21.15.40:0/1508232717 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac0008a00 con 0x7f9ac0003c60 2024-04-22T08:21:54.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.970+0000 7f9ae1c76640 1 -- 172.21.15.40:0/1508232717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ac0003c60 msgr2=0x7f9ac00067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:54.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.970+0000 7f9ae1c76640 1 --2- 172.21.15.40:0/1508232717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ac0003c60 0x7f9ac00067e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9acc006fd0 tx=0x7f9acc02da00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e679133893e5858 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:54.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.970+0000 7f9ae1475640 1 -- 172.21.15.40:0/1508232717 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9acc033630 con 0x7f9ac0003c60 2024-04-22T08:21:54.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.970+0000 7f9ae1475640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:54.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.970+0000 7f9ae1475640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:21:54.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.970+0000 7f9ae1475640 10 monclient: dump: 2024-04-22T08:21:54.978 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:54.978 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:54.979 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:54.979 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:54.979 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:54.979 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:54.979 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:54.979 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:54.979 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:54.979 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:54.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.970+0000 7f9ae1475640 10 monclient: _finish_auth 0 2024-04-22T08:21:54.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.970+0000 7f9ae1475640 10 monclient: _check_auth_tickets 2024-04-22T08:21:54.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.970+0000 7f9ae1475640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:54.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.970+0000 7f9ae1475640 1 -- 172.21.15.40:0/1508232717 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9acc0337b0 con 0x7f9ac0003c60 2024-04-22T08:21:54.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.970+0000 7f9ae46db640 5 monclient: authenticate success, global_id 104288 2024-04-22T08:21:54.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.970+0000 7f9ae1475640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:54.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.970+0000 7f9ae46db640 10 monclient: get_monmap_and_config success 2024-04-22T08:21:54.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.970+0000 7f9ae1475640 1 -- 172.21.15.40:0/1508232717 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9acc033b60 con 0x7f9ac0003c60 2024-04-22T08:21:54.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.970+0000 7f9ae1475640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:54.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.970+0000 7f9ae46db640 4 set_mon_vals no callback set 2024-04-22T08:21:54.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.970+0000 7f9ae1475640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:21:54.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.970+0000 7f9ae1475640 10 monclient: dump: 2024-04-22T08:21:54.980 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:54.980 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:54.980 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:54.980 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:54.980 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:54.980 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:54.980 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:54.981 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:54.981 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:54.981 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:54.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.971+0000 7f9ae46db640 10 monclient: shutdown 2024-04-22T08:21:54.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.971+0000 7f9ae46db640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:54.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.971+0000 7f9ae46db640 1 -- 172.21.15.40:0/1508232717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ac0003c60 msgr2=0x7f9ac00067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:54.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.971+0000 7f9ae46db640 1 --2- 172.21.15.40:0/1508232717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ac0003c60 0x7f9ac00067e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9acc006fd0 tx=0x7f9acc02da00 comp rx=0 tx=0).stop 2024-04-22T08:21:54.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.971+0000 7f9ae46db640 1 -- 172.21.15.40:0/1508232717 shutdown_connections 2024-04-22T08:21:54.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.971+0000 7f9ae46db640 1 --2- 172.21.15.40:0/1508232717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ac0003c60 0x7f9ac00067e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:54.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.971+0000 7f9ae46db640 1 --2- 172.21.15.40:0/1508232717 >> v2:172.21.15.189:3300/0 conn(0x7f9adc071c90 0x7f9ac000b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:54.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.971+0000 7f9ae46db640 1 -- 172.21.15.40:0/1508232717 >> 172.21.15.40:0/1508232717 conn(0x7f9adc0a65c0 msgr2=0x7f9adc0a2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:54.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.971+0000 7f9ae46db640 1 -- 172.21.15.40:0/1508232717 shutdown_connections 2024-04-22T08:21:54.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.971+0000 7f9ae46db640 1 -- 172.21.15.40:0/1508232717 wait complete. 2024-04-22T08:21:54.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.972+0000 7f9ae46db640 10 monclient: build_initial_monmap 2024-04-22T08:21:54.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.972+0000 7f9ae46db640 10 monclient: monmap: 2024-04-22T08:21:54.982 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:54.982 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:21:54.982 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:21:54.982 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:21:54.982 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:54.982 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:54.982 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:21:54.982 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:21:54.982 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:21:54.982 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:54.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.972+0000 7f9ae46db640 1 Processor -- start 2024-04-22T08:21:54.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.972+0000 7f9ae46db640 1 -- start start 2024-04-22T08:21:54.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.972+0000 7f9ae46db640 10 monclient: init 2024-04-22T08:21:54.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.973+0000 7f9ae46db640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:54.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.973+0000 7f9ae46db640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-22T08:21:54.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.973+0000 7f9ae46db640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ac0003c60 0x7f9adc134980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:54.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.973+0000 7f9ae46db640 10 monclient(hunting): picked mon.noname-b con 0x7f9ac0003c60 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:21:54.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.973+0000 7f9ae46db640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ac0006d20 0x7f9adc06e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:54.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.973+0000 7f9ae46db640 10 monclient(hunting): picked mon.noname-a con 0x7f9ac0006d20 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:21:54.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.973+0000 7f9ae46db640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9adc071c90 0x7f9adc06eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:54.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.973+0000 7f9ae46db640 10 monclient(hunting): picked mon.noname-c con 0x7f9adc071c90 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:54.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.973+0000 7f9ae46db640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:54.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.973+0000 7f9ae46db640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9adc070bb0 con 0x7f9ac0006d20 2024-04-22T08:21:54.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.973+0000 7f9ae46db640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:54.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.973+0000 7f9ae46db640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9adc120a90 con 0x7f9ac0003c60 2024-04-22T08:21:54.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.973+0000 7f9ae46db640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:54.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.973+0000 7f9ae46db640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9adc1cb3e0 con 0x7f9adc071c90 2024-04-22T08:21:54.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.973+0000 7f9ae46db640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:54.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.973+0000 7f9ae46db640 10 monclient(hunting): authenticate will time out at 1388.967212s 2024-04-22T08:21:54.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.973+0000 7f9ae1c76640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ac0006d20 0x7f9adc06e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:54.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.973+0000 7f9ae2c78640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9adc071c90 0x7f9adc06eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:54.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.973+0000 7f9ae1c76640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ac0006d20 0x7f9adc06e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56822/0 (socket says 172.21.15.40:56822) 2024-04-22T08:21:54.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.973+0000 7f9ae2c78640 10 monclient(hunting): get_auth_request con 0x7f9adc071c90 auth_method 0 2024-04-22T08:21:54.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.973+0000 7f9ae1c76640 1 -- 172.21.15.40:0/1063865245 learned_addr learned my addr 172.21.15.40:0/1063865245 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:21:54.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.973+0000 7f9ae2c78640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:54.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.973+0000 7f9ae2c78640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:54.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.973+0000 7f9ae2c78640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:54.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.973+0000 7f9ae1c76640 10 monclient(hunting): get_auth_request con 0x7f9ac0006d20 auth_method 0 2024-04-22T08:21:54.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.973+0000 7f9ae1c76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:54.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.973+0000 7f9ae1c76640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:54.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.973+0000 7f9ae1c76640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:54.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.973+0000 7f9ae2477640 1 --2- 172.21.15.40:0/1063865245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ac0003c60 0x7f9adc134980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:54.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.973+0000 7f9ae2477640 10 monclient(hunting): get_auth_request con 0x7f9ac0003c60 auth_method 0 2024-04-22T08:21:54.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.973+0000 7f9ae2477640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:54.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.973+0000 7f9ae2477640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:54.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.973+0000 7f9ae2477640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:54.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.973+0000 7f9ae2c78640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:54.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.973+0000 7f9ae2c78640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:54.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.973+0000 7f9ae2c78640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:54.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.973+0000 7f9ae1c76640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:54.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.973+0000 7f9ae1c76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:54.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.973+0000 7f9ae1c76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:54.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.973+0000 7f9ae2477640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:54.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.973+0000 7f9ae2477640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:54.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.973+0000 7f9ae2477640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:54.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.974+0000 7f9ae1c76640 10 monclient(hunting): handle_auth_done global_id 114120 payload 824 2024-04-22T08:21:54.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.974+0000 7f9ae1c76640 1 -- 172.21.15.40:0/1063865245 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9adc071c90 msgr2=0x7f9adc06eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:54.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.974+0000 7f9ae1c76640 1 --2- 172.21.15.40:0/1063865245 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9adc071c90 0x7f9adc06eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:54.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.974+0000 7f9ae1c76640 1 -- 172.21.15.40:0/1063865245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ac0003c60 msgr2=0x7f9adc134980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:54.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.974+0000 7f9ae1c76640 1 --2- 172.21.15.40:0/1063865245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ac0003c60 0x7f9adc134980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:54.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.974+0000 7f9ae1c76640 10 monclient: _finish_hunting 0 2024-04-22T08:21:54.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.974+0000 7f9ae1c76640 1 monclient: found mon.noname-a 2024-04-22T08:21:54.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.974+0000 7f9ae1c76640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:54.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.974+0000 7f9ae1c76640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:21:54.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.974+0000 7f9ae1c76640 1 -- 172.21.15.40:0/1063865245 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9acc00e040 con 0x7f9ac0006d20 2024-04-22T08:21:54.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.974+0000 7f9ae2c78640 1 --2- 172.21.15.40:0/1063865245 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9adc071c90 0x7f9adc06eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:21:54.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.976+0000 7f9ae1c76640 1 -- 172.21.15.40:0/1063865245 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ac0006d20 msgr2=0x7f9adc06e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:54.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.976+0000 7f9ae1c76640 1 --2- 172.21.15.40:0/1063865245 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ac0006d20 0x7f9adc06e650 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9acc017b90 tx=0x7f9acc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:54.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.976+0000 7f9ad2ffd640 1 -- 172.21.15.40:0/1063865245 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9acc04a070 con 0x7f9ac0006d20 2024-04-22T08:21:54.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.976+0000 7f9ad2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:54.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.976+0000 7f9ad2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:21:54.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.976+0000 7f9ad2ffd640 10 monclient: dump: 2024-04-22T08:21:54.987 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:54.987 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:54.988 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:54.988 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:54.988 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:54.988 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:54.988 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:54.988 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:54.988 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:54.988 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:54.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.976+0000 7f9ad2ffd640 10 monclient: _finish_auth 0 2024-04-22T08:21:54.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.976+0000 7f9ad2ffd640 10 monclient: _check_auth_tickets 2024-04-22T08:21:54.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.976+0000 7f9ad2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:54.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.976+0000 7f9ad2ffd640 1 -- 172.21.15.40:0/1063865245 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9acc033630 con 0x7f9ac0006d20 2024-04-22T08:21:54.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.976+0000 7f9ae46db640 5 monclient: authenticate success, global_id 114120 2024-04-22T08:21:54.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.976+0000 7f9ae46db640 10 monclient: _renew_subs 2024-04-22T08:21:54.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.976+0000 7f9ae46db640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:21:54.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.976+0000 7f9ae46db640 1 -- 172.21.15.40:0/1063865245 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9adc0720e0 con 0x7f9ac0006d20 2024-04-22T08:21:54.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.976+0000 7f9ad2ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:54.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.976+0000 7f9ad2ffd640 1 -- 172.21.15.40:0/1063865245 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9acc006bf0 con 0x7f9ac0006d20 2024-04-22T08:21:54.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.976+0000 7f9ad2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:54.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.976+0000 7f9ad2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:21:54.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.976+0000 7f9ad2ffd640 10 monclient: dump: 2024-04-22T08:21:54.989 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:54.989 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:54.989 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:54.989 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:54.989 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:54.989 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:54.989 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:54.989 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:54.989 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:54.990 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:54.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.976+0000 7f9ae46db640 10 monclient: _renew_subs 2024-04-22T08:21:54.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.976+0000 7f9ae46db640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:21:54.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.976+0000 7f9ae46db640 1 -- 172.21.15.40:0/1063865245 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9adc121330 con 0x7f9ac0006d20 2024-04-22T08:21:54.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.976+0000 7f9ae0c74640 4 set_mon_vals no callback set 2024-04-22T08:21:54.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.977+0000 7f9ad2ffd640 1 -- 172.21.15.40:0/1063865245 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f9acc02f040 con 0x7f9ac0006d20 2024-04-22T08:21:54.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.977+0000 7f9ae46db640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:54.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.977+0000 7f9ae46db640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:54.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.977+0000 7f9ae46db640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:21:54.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.977+0000 7f9ae46db640 1 -- 172.21.15.40:0/1063865245 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9adc04e9b0 con 0x7f9ac0006d20 2024-04-22T08:21:54.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.978+0000 7f9ad2ffd640 1 --2- 172.21.15.40:0/1063865245 >> v2:172.21.15.189:6800/30168 conn(0x7f9ac00394d0 0x7f9ac003b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:54.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.978+0000 7f9ad2ffd640 1 -- 172.21.15.40:0/1063865245 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(308..308 src has 239..308) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f9acc041070 con 0x7f9ac0006d20 2024-04-22T08:21:54.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.978+0000 7f9ae2477640 1 --2- 172.21.15.40:0/1063865245 >> v2:172.21.15.189:6800/30168 conn(0x7f9ac00394d0 0x7f9ac003b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:54.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.978+0000 7f9ae2477640 10 monclient: get_auth_request con 0x7f9ac00394d0 auth_method 0 2024-04-22T08:21:54.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.981+0000 7f9ae2477640 1 --2- 172.21.15.40:0/1063865245 >> v2:172.21.15.189:6800/30168 conn(0x7f9ac00394d0 0x7f9ac003b990 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9ad8007bc0 tx=0x7f9ad8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:54.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.983+0000 7f9ad2ffd640 1 -- 172.21.15.40:0/1063865245 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f9acc030390 con 0x7f9ac0006d20 2024-04-22T08:21:54.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.983+0000 7f9ad2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:54.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:54.983+0000 7f9ad2ffd640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:21:55.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.234+0000 7f9ae46db640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:21:55.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.234+0000 7f9ae46db640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:21:55.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.234+0000 7f9ae46db640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:21:55.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.234+0000 7f9ae46db640 1 -- 172.21.15.40:0/1063865245 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9adc0a0430 con 0x7f9ac0006d20 2024-04-22T08:21:55.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.234+0000 7f9ad2ffd640 1 -- 172.21.15.40:0/1063865245 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f9acc037080 con 0x7f9ac0006d20 2024-04-22T08:21:55.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.235+0000 7f9ad2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:21:55.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.235+0000 7f9ad2ffd640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:21:55.235 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":92,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:21:55.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.236+0000 7f9ae46db640 1 -- 172.21.15.40:0/1063865245 >> v2:172.21.15.189:6800/30168 conn(0x7f9ac00394d0 msgr2=0x7f9ac003b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:55.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.236+0000 7f9ae46db640 1 --2- 172.21.15.40:0/1063865245 >> v2:172.21.15.189:6800/30168 conn(0x7f9ac00394d0 0x7f9ac003b990 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9ad8007bc0 tx=0x7f9ad8009040 comp rx=0 tx=0).stop 2024-04-22T08:21:55.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.236+0000 7f9ae46db640 10 monclient: shutdown 2024-04-22T08:21:55.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.236+0000 7f9ae46db640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:55.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.236+0000 7f9ae46db640 1 -- 172.21.15.40:0/1063865245 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ac0006d20 msgr2=0x7f9adc06e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:55.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.236+0000 7f9ae46db640 1 --2- 172.21.15.40:0/1063865245 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ac0006d20 0x7f9adc06e650 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9acc017b90 tx=0x7f9acc004730 comp rx=0 tx=0).stop 2024-04-22T08:21:55.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.236+0000 7f9ae46db640 1 -- 172.21.15.40:0/1063865245 shutdown_connections 2024-04-22T08:21:55.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.236+0000 7f9ae46db640 1 --2- 172.21.15.40:0/1063865245 >> v2:172.21.15.189:6800/30168 conn(0x7f9ac00394d0 0x7f9ac003b990 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:55.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.236+0000 7f9ae46db640 1 --2- 172.21.15.40:0/1063865245 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9adc071c90 0x7f9adc06eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:55.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.236+0000 7f9ae46db640 1 --2- 172.21.15.40:0/1063865245 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ac0006d20 0x7f9adc06e650 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:55.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.236+0000 7f9ae46db640 1 --2- 172.21.15.40:0/1063865245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ac0003c60 0x7f9adc134980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:55.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.236+0000 7f9ae46db640 1 -- 172.21.15.40:0/1063865245 >> 172.21.15.40:0/1063865245 conn(0x7f9adc0a65c0 msgr2=0x7f9adc1363c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:55.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.236+0000 7f9ae46db640 1 -- 172.21.15.40:0/1063865245 shutdown_connections 2024-04-22T08:21:55.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.237+0000 7f9ae46db640 1 -- 172.21.15.40:0/1063865245 wait complete. 2024-04-22T08:21:55.246 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-22T08:21:55.247 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-22T08:21:55.247 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-22T08:21:55.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.346+0000 7f0875d47640 10 monclient: get_monmap_and_config 2024-04-22T08:21:55.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.346+0000 7f0875d47640 10 monclient: build_initial_monmap 2024-04-22T08:21:55.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.346+0000 7f0875d47640 10 monclient: monmap: 2024-04-22T08:21:55.346 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:55.347 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:55.347 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:21:55.347526+0000 2024-04-22T08:21:55.347 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:21:55.347526+0000 2024-04-22T08:21:55.347 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:55.347 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:55.347 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:21:55.347 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:21:55.347 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:21:55.348 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:21:55.348 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:21:55.348 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:55.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.347+0000 7f0875d47640 1 Processor -- start 2024-04-22T08:21:55.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.347+0000 7f0875d47640 1 -- start start 2024-04-22T08:21:55.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.347+0000 7f0875d47640 10 monclient: init 2024-04-22T08:21:55.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.347+0000 7f0875d47640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:55.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.347+0000 7f0875d47640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-04-22T08:21:55.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.347+0000 7f0875d47640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f08700736d0 0x7f0870135f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:55.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.347+0000 7f0875d47640 10 monclient(hunting): picked mon.noname-a con 0x7f08700736d0 addr v2:172.21.15.40:3300/0 2024-04-22T08:21:55.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.347+0000 7f0875d47640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0870070eb0 0x7f0870071320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:55.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.347+0000 7f0875d47640 10 monclient(hunting): picked mon.noname-c con 0x7f0870070eb0 addr v2:172.21.15.189:3300/0 2024-04-22T08:21:55.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.347+0000 7f0875d47640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f087009de00 0x7f087009e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:55.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.347+0000 7f0875d47640 10 monclient(hunting): picked mon.noname-b con 0x7f087009de00 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:55.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.347+0000 7f0875d47640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:55.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.347+0000 7f0875d47640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0870071860 con 0x7f08700736d0 2024-04-22T08:21:55.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.347+0000 7f0875d47640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:55.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.347+0000 7f0875d47640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0870073ab0 con 0x7f0870070eb0 2024-04-22T08:21:55.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.347+0000 7f0875d47640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:55.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.347+0000 7f0875d47640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0870136510 con 0x7f087009de00 2024-04-22T08:21:55.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.347+0000 7f0875d47640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:55.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.347+0000 7f0875d47640 10 monclient(hunting): authenticate will time out at 1389.341755s 2024-04-22T08:21:55.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.348+0000 7f086f7fe640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f08700736d0 0x7f0870135f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:55.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.348+0000 7f086ffff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f087009de00 0x7f087009e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:55.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.348+0000 7f086f7fe640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f08700736d0 0x7f0870135f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56826/0 (socket says 172.21.15.40:56826) 2024-04-22T08:21:55.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.348+0000 7f086f7fe640 1 -- 172.21.15.40:0/3372492390 learned_addr learned my addr 172.21.15.40:0/3372492390 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:21:55.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.348+0000 7f086f7fe640 10 monclient(hunting): get_auth_request con 0x7f08700736d0 auth_method 0 2024-04-22T08:21:55.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.348+0000 7f086f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:55.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.348+0000 7f086f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:55.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.348+0000 7f086f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:55.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.348+0000 7f086ffff640 10 monclient(hunting): get_auth_request con 0x7f087009de00 auth_method 0 2024-04-22T08:21:55.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.348+0000 7f086ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:55.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.348+0000 7f086ffff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:55.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.348+0000 7f086ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:55.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.348+0000 7f086f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:55.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.348+0000 7f086f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:55.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.348+0000 7f086f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:55.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.348+0000 7f086ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:55.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.348+0000 7f086ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:55.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.348+0000 7f086ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:55.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.348+0000 7f086effd640 1 --2- 172.21.15.40:0/3372492390 >> v2:172.21.15.189:3300/0 conn(0x7f0870070eb0 0x7f0870071320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:55.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.348+0000 7f086effd640 10 monclient(hunting): get_auth_request con 0x7f0870070eb0 auth_method 0 2024-04-22T08:21:55.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.348+0000 7f086effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:55.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.348+0000 7f086effd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:55.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.348+0000 7f086effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:55.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.348+0000 7f086f7fe640 10 monclient(hunting): handle_auth_done global_id 114123 payload 274 2024-04-22T08:21:55.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.348+0000 7f086f7fe640 1 -- 172.21.15.40:0/3372492390 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f087009de00 msgr2=0x7f087009e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:21:55.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.348+0000 7f086f7fe640 1 --2- 172.21.15.40:0/3372492390 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f087009de00 0x7f087009e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:55.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.348+0000 7f086f7fe640 1 -- 172.21.15.40:0/3372492390 >> v2:172.21.15.189:3300/0 conn(0x7f0870070eb0 msgr2=0x7f0870071320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:21:55.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.348+0000 7f086f7fe640 1 --2- 172.21.15.40:0/3372492390 >> v2:172.21.15.189:3300/0 conn(0x7f0870070eb0 0x7f0870071320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:55.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.348+0000 7f086f7fe640 10 monclient: _finish_hunting 0 2024-04-22T08:21:55.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.348+0000 7f086f7fe640 1 monclient: found mon.noname-a 2024-04-22T08:21:55.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.348+0000 7f086f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:55.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.348+0000 7f086f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:21:55.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.348+0000 7f086f7fe640 1 -- 172.21.15.40:0/3372492390 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f087006ce90 con 0x7f08700736d0 2024-04-22T08:21:55.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.348+0000 7f086ffff640 1 --2- 172.21.15.40:0/3372492390 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f087009de00 0x7f087009e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:21:55.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.351+0000 7f086f7fe640 1 -- 172.21.15.40:0/3372492390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08700736d0 msgr2=0x7f0870135f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:55.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.351+0000 7f086f7fe640 1 --2- 172.21.15.40:0/3372492390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08700736d0 0x7f0870135f10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0870073f30 tx=0x7f086402b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24e3ef79cd6f0ec2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:55.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.351+0000 7f086e7fc640 1 -- 172.21.15.40:0/3372492390 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0864031960 con 0x7f08700736d0 2024-04-22T08:21:55.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.351+0000 7f086e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:55.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.351+0000 7f086e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:21:55.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.351+0000 7f086e7fc640 10 monclient: dump: 2024-04-22T08:21:55.359 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:55.359 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:55.359 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:55.359 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:55.360 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:55.360 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:55.360 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:55.360 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:55.360 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:55.360 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:55.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.351+0000 7f086e7fc640 10 monclient: _finish_auth 0 2024-04-22T08:21:55.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.351+0000 7f086e7fc640 10 monclient: _check_auth_tickets 2024-04-22T08:21:55.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.351+0000 7f086e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:55.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.351+0000 7f086e7fc640 1 -- 172.21.15.40:0/3372492390 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0864031ac0 con 0x7f08700736d0 2024-04-22T08:21:55.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.351+0000 7f0875d47640 5 monclient: authenticate success, global_id 114123 2024-04-22T08:21:55.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.351+0000 7f0875d47640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:21:55.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.351+0000 7f086e7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:55.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.351+0000 7f0875d47640 10 monclient: get_monmap_and_config success 2024-04-22T08:21:55.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.351+0000 7f086e7fc640 1 -- 172.21.15.40:0/3372492390 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0864031e50 con 0x7f08700736d0 2024-04-22T08:21:55.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.351+0000 7f0875d47640 4 set_mon_vals no callback set 2024-04-22T08:21:55.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.351+0000 7f086e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:55.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.351+0000 7f086e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:21:55.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.351+0000 7f086e7fc640 10 monclient: dump: 2024-04-22T08:21:55.361 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:55.361 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:55.361 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:55.361 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:55.361 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:55.361 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:55.362 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:55.362 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:55.362 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:55.362 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:55.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.351+0000 7f0875d47640 10 monclient: shutdown 2024-04-22T08:21:55.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.351+0000 7f0875d47640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:55.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.351+0000 7f0875d47640 1 -- 172.21.15.40:0/3372492390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08700736d0 msgr2=0x7f0870135f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:55.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.351+0000 7f0875d47640 1 --2- 172.21.15.40:0/3372492390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08700736d0 0x7f0870135f10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0870073f30 tx=0x7f086402b7f0 comp rx=0 tx=0).stop 2024-04-22T08:21:55.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.351+0000 7f0875d47640 1 -- 172.21.15.40:0/3372492390 shutdown_connections 2024-04-22T08:21:55.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.352+0000 7f0875d47640 1 --2- 172.21.15.40:0/3372492390 >> v2:172.21.15.189:3300/0 conn(0x7f0870070eb0 0x7f0870071320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:55.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.352+0000 7f0875d47640 1 --2- 172.21.15.40:0/3372492390 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f087009de00 0x7f087009e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:55.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.352+0000 7f0875d47640 1 --2- 172.21.15.40:0/3372492390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08700736d0 0x7f0870135f10 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:55.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.352+0000 7f0875d47640 1 -- 172.21.15.40:0/3372492390 >> 172.21.15.40:0/3372492390 conn(0x7f08700a1780 msgr2=0x7f08700a1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:55.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.352+0000 7f0875d47640 1 -- 172.21.15.40:0/3372492390 shutdown_connections 2024-04-22T08:21:55.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.352+0000 7f0875d47640 1 -- 172.21.15.40:0/3372492390 wait complete. 2024-04-22T08:21:55.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.352+0000 7f0875d47640 10 monclient: build_initial_monmap 2024-04-22T08:21:55.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.352+0000 7f0875d47640 10 monclient: monmap: 2024-04-22T08:21:55.363 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:55.363 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:21:55.363 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:21:55.363 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:21:55.363 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:55.363 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:55.363 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:21:55.363 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:21:55.363 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:21:55.363 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:55.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.352+0000 7f0875d47640 1 Processor -- start 2024-04-22T08:21:55.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f0875d47640 1 -- start start 2024-04-22T08:21:55.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f0875d47640 10 monclient: init 2024-04-22T08:21:55.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f0875d47640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:55.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f0875d47640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-22T08:21:55.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f0875d47640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0870070eb0 0x7f08701caac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:55.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f0875d47640 10 monclient(hunting): picked mon.noname-a con 0x7f0870070eb0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:21:55.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f0875d47640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f08700736d0 0x7f087012efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:55.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f0875d47640 10 monclient(hunting): picked mon.noname-c con 0x7f08700736d0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:55.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f0875d47640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f087009de00 0x7f087012f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:55.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f0875d47640 10 monclient(hunting): picked mon.noname-b con 0x7f087009de00 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:21:55.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f0875d47640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:55.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f0875d47640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087011edd0 con 0x7f0870070eb0 2024-04-22T08:21:55.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f0875d47640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:55.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f0875d47640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08700a4090 con 0x7f087009de00 2024-04-22T08:21:55.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f0875d47640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:55.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f0875d47640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0870135530 con 0x7f08700736d0 2024-04-22T08:21:55.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f0875d47640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:55.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f0875d47640 10 monclient(hunting): authenticate will time out at 1389.347291s 2024-04-22T08:21:55.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f086f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0870070eb0 0x7f08701caac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:55.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f086effd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f08700736d0 0x7f087012efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:55.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f086effd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f08700736d0 0x7f087012efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:41850/0 (socket says 172.21.15.40:41850) 2024-04-22T08:21:55.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f086f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0870070eb0 0x7f08701caac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56836/0 (socket says 172.21.15.40:56836) 2024-04-22T08:21:55.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f086effd640 1 -- 172.21.15.40:0/1983902027 learned_addr learned my addr 172.21.15.40:0/1983902027 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:21:55.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f086effd640 10 monclient(hunting): get_auth_request con 0x7f08700736d0 auth_method 0 2024-04-22T08:21:55.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f086effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:55.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f086effd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:55.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f086effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:55.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f086f7fe640 10 monclient(hunting): get_auth_request con 0x7f0870070eb0 auth_method 0 2024-04-22T08:21:55.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f086f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:55.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f086f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:55.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f086f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:55.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f086effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:55.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f086effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:55.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f086effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:55.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f086f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:55.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f086f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:55.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f086f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:55.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f086f7fe640 10 monclient(hunting): handle_auth_done global_id 114126 payload 824 2024-04-22T08:21:55.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f086ffff640 1 --2- 172.21.15.40:0/1983902027 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f087009de00 0x7f087012f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:55.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f086f7fe640 1 -- 172.21.15.40:0/1983902027 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f08700736d0 msgr2=0x7f087012efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:55.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f086f7fe640 1 --2- 172.21.15.40:0/1983902027 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f08700736d0 0x7f087012efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:55.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f086f7fe640 1 -- 172.21.15.40:0/1983902027 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f087009de00 msgr2=0x7f087012f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:55.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f086f7fe640 1 --2- 172.21.15.40:0/1983902027 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f087009de00 0x7f087012f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:55.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f086f7fe640 10 monclient: _finish_hunting 0 2024-04-22T08:21:55.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f086f7fe640 1 monclient: found mon.noname-a 2024-04-22T08:21:55.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f086f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:55.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f086f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:21:55.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.353+0000 7f086f7fe640 1 -- 172.21.15.40:0/1983902027 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0868009a70 con 0x7f0870070eb0 2024-04-22T08:21:55.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.356+0000 7f086f7fe640 1 -- 172.21.15.40:0/1983902027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0870070eb0 msgr2=0x7f08701caac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:55.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.356+0000 7f086f7fe640 1 --2- 172.21.15.40:0/1983902027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0870070eb0 0x7f08701caac0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0864033000 tx=0x7f08640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:55.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.356+0000 7f0857fff640 1 -- 172.21.15.40:0/1983902027 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0864031c30 con 0x7f0870070eb0 2024-04-22T08:21:55.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.356+0000 7f0857fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:55.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.356+0000 7f0857fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:21:55.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.356+0000 7f0857fff640 10 monclient: dump: 2024-04-22T08:21:55.373 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:55.373 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:55.374 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:55.374 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:55.374 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:55.374 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:55.374 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:55.374 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:55.374 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:55.374 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:55.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.356+0000 7f0857fff640 10 monclient: _finish_auth 0 2024-04-22T08:21:55.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.356+0000 7f0857fff640 10 monclient: _check_auth_tickets 2024-04-22T08:21:55.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.356+0000 7f0857fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:55.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.356+0000 7f0875d47640 5 monclient: authenticate success, global_id 114126 2024-04-22T08:21:55.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.356+0000 7f0857fff640 1 -- 172.21.15.40:0/1983902027 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0864031db0 con 0x7f0870070eb0 2024-04-22T08:21:55.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.356+0000 7f0875d47640 10 monclient: _renew_subs 2024-04-22T08:21:55.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.356+0000 7f0875d47640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:21:55.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.356+0000 7f0875d47640 1 -- 172.21.15.40:0/1983902027 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08640094d0 con 0x7f0870070eb0 2024-04-22T08:21:55.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.356+0000 7f0857fff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:55.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.356+0000 7f0857fff640 1 -- 172.21.15.40:0/1983902027 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086403c640 con 0x7f0870070eb0 2024-04-22T08:21:55.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.356+0000 7f0857fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:55.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.356+0000 7f0857fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:21:55.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.356+0000 7f0857fff640 10 monclient: dump: 2024-04-22T08:21:55.375 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:55.375 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:55.375 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:55.375 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:55.376 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:55.376 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:55.376 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:55.376 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:55.376 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:55.376 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:55.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.356+0000 7f0875d47640 10 monclient: _renew_subs 2024-04-22T08:21:55.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.356+0000 7f0875d47640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:21:55.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.356+0000 7f086dffb640 4 set_mon_vals no callback set 2024-04-22T08:21:55.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.356+0000 7f0875d47640 1 -- 172.21.15.40:0/1983902027 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0870120140 con 0x7f0870070eb0 2024-04-22T08:21:55.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.357+0000 7f0875d47640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:55.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.357+0000 7f0875d47640 1 --2- 172.21.15.40:0/1983902027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0870120330 0x7f087011caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:55.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.357+0000 7f0875d47640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:55.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.357+0000 7f0875d47640 1 -- 172.21.15.40:0/1983902027 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0870066900 con 0x7f0870120330 2024-04-22T08:21:55.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.357+0000 7f086effd640 1 --2- 172.21.15.40:0/1983902027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0870120330 0x7f087011caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:55.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.357+0000 7f0857fff640 1 -- 172.21.15.40:0/1983902027 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f086403a040 con 0x7f0870070eb0 2024-04-22T08:21:55.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.357+0000 7f086effd640 10 monclient: get_auth_request con 0x7f0870120330 auth_method 0 2024-04-22T08:21:55.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.357+0000 7f086effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:55.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.357+0000 7f086effd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:55.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.357+0000 7f086effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:55.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.357+0000 7f086effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:55.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.357+0000 7f086effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:55.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.357+0000 7f086effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:55.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.357+0000 7f0857fff640 1 --2- 172.21.15.40:0/1983902027 >> v2:172.21.15.189:6800/30168 conn(0x7f085c0377d0 0x7f085c039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:55.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.357+0000 7f086effd640 10 monclient(hunting): handle_auth_done global_id 114129 payload 824 2024-04-22T08:21:55.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.357+0000 7f086effd640 1 -- 172.21.15.40:0/1983902027 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f08700a3980 con 0x7f0870120330 2024-04-22T08:21:55.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.357+0000 7f0857fff640 1 -- 172.21.15.40:0/1983902027 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(308..308 src has 239..308) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f086406b450 con 0x7f0870070eb0 2024-04-22T08:21:55.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.357+0000 7f086ffff640 1 --2- 172.21.15.40:0/1983902027 >> v2:172.21.15.189:6800/30168 conn(0x7f085c0377d0 0x7f085c039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:55.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.357+0000 7f086ffff640 10 monclient: get_auth_request con 0x7f085c0377d0 auth_method 0 2024-04-22T08:21:55.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.359+0000 7f086effd640 1 -- 172.21.15.40:0/1983902027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0870120330 msgr2=0x7f087011caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:55.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.359+0000 7f086effd640 1 --2- 172.21.15.40:0/1983902027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0870120330 0x7f087011caf0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f08600107f0 tx=0x7f0860010cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:55.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.359+0000 7f0857fff640 1 -- 172.21.15.40:0/1983902027 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086000ec40 con 0x7f0870120330 2024-04-22T08:21:55.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.359+0000 7f0857fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:55.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.359+0000 7f0857fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:21:55.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.359+0000 7f0857fff640 10 monclient: dump: 2024-04-22T08:21:55.378 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:55.378 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:55.379 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:55.379 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:55.379 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:55.379 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:55.379 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:55.379 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:55.379 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:55.379 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:55.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.360+0000 7f0857fff640 1 -- 172.21.15.40:0/1983902027 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f086000eda0 con 0x7f0870120330 2024-04-22T08:21:55.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.360+0000 7f0857fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:55.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.360+0000 7f0857fff640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:21:55.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.360+0000 7f0857fff640 1 -- 172.21.15.40:0/1983902027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0870120330 msgr2=0x7f087011caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:55.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.360+0000 7f0857fff640 1 --2- 172.21.15.40:0/1983902027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0870120330 0x7f087011caf0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f08600107f0 tx=0x7f0860010cb0 comp rx=0 tx=0).stop 2024-04-22T08:21:55.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.360+0000 7f0857fff640 1 -- 172.21.15.40:0/1983902027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0870120330 msgr2=0x7f087011caf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:21:55.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.360+0000 7f0857fff640 1 --2- 172.21.15.40:0/1983902027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0870120330 0x7f087011caf0 secure :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f08600107f0 tx=0x7f0860010cb0 comp rx=0 tx=0).stop 2024-04-22T08:21:55.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.360+0000 7f086ffff640 1 --2- 172.21.15.40:0/1983902027 >> v2:172.21.15.189:6800/30168 conn(0x7f085c0377d0 0x7f085c039c90 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f0868007580 tx=0x7f086800b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:55.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.366+0000 7f0875d47640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:21:55.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.366+0000 7f0875d47640 1 --2- 172.21.15.40:0/1983902027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f087011d030 0x7f087011d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:55.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.366+0000 7f0875d47640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:55.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.366+0000 7f0875d47640 1 -- 172.21.15.40:0/1983902027 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087006d450 con 0x7f087011d030 2024-04-22T08:21:55.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.366+0000 7f086effd640 1 --2- 172.21.15.40:0/1983902027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f087011d030 0x7f087011d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:55.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.366+0000 7f086effd640 10 monclient: get_auth_request con 0x7f087011d030 auth_method 0 2024-04-22T08:21:55.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.366+0000 7f086effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:55.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.366+0000 7f086effd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:55.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.366+0000 7f086effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:55.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.366+0000 7f086effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:55.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.366+0000 7f086effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:55.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.366+0000 7f086effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:55.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.366+0000 7f086effd640 10 monclient(hunting): handle_auth_done global_id 114132 payload 824 2024-04-22T08:21:55.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.366+0000 7f086effd640 1 -- 172.21.15.40:0/1983902027 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f087009e7d0 con 0x7f087011d030 2024-04-22T08:21:55.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.369+0000 7f086effd640 1 -- 172.21.15.40:0/1983902027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f087011d030 msgr2=0x7f087011d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:55.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.369+0000 7f086effd640 1 --2- 172.21.15.40:0/1983902027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f087011d030 0x7f087011d440 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f08600107f0 tx=0x7f08600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:55.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.369+0000 7f0857fff640 1 -- 172.21.15.40:0/1983902027 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086002b070 con 0x7f087011d030 2024-04-22T08:21:55.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.369+0000 7f0857fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:55.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.369+0000 7f0857fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:21:55.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.369+0000 7f0857fff640 10 monclient: dump: 2024-04-22T08:21:55.381 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:55.381 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:55.381 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:55.381 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:55.381 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:55.381 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:55.382 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:55.382 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:55.382 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:55.382 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:55.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.369+0000 7f0857fff640 1 -- 172.21.15.40:0/1983902027 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4463 (secure 0 0 0) 0x7f0860023040 con 0x7f087011d030 2024-04-22T08:21:55.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.369+0000 7f0857fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:21:55.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.369+0000 7f0857fff640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:21:55.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.369+0000 7f0857fff640 1 -- 172.21.15.40:0/1983902027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f087011d030 msgr2=0x7f087011d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:55.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.369+0000 7f0857fff640 1 --2- 172.21.15.40:0/1983902027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f087011d030 0x7f087011d440 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f08600107f0 tx=0x7f08600046c0 comp rx=0 tx=0).stop 2024-04-22T08:21:55.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.369+0000 7f0857fff640 1 -- 172.21.15.40:0/1983902027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f087011d030 msgr2=0x7f087011d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:21:55.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.369+0000 7f0857fff640 1 --2- 172.21.15.40:0/1983902027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f087011d030 0x7f087011d440 secure :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f08600107f0 tx=0x7f08600046c0 comp rx=0 tx=0).stop 2024-04-22T08:21:55.385 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:21:55.386 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:21:55.386 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:21:55.386 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "leader", 2024-04-22T08:21:55.386 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 92, 2024-04-22T08:21:55.386 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:21:55.386 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:21:55.386 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:21:55.386 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:21:55.386 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:55.386 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 0, 2024-04-22T08:21:55.386 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:21:55.386 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:21:55.387 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:21:55.387 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:21:55.387 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:21:55.387 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:21:55.387 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:21:55.387 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:21:55.387 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:21:55.387 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:21:55.387 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:21:55.387 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:21:55.387 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:21:55.387 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:55.387 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:21:55.387 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:21:55.387 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:21:55.388 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:21:55.388 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:21:55.388 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:21:55.388 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:21:55.388 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:21:55.388 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:21:55.388 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:21:55.388 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:21:55.388 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:21:55.388 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:55.388 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:55.388 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:21:55.389 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:21:55.389 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:21:55.389 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:21:55.389 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:21:55.389 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:21:55.389 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:21:55.389 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:21:55.389 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:21:55.389 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:21:55.389 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:21:55.389 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:21:55.389 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:21:55.389 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:21:55.390 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:21:55.390 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:21:55.390 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:21:55.390 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:21:55.390 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:21:55.390 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:21:55.390 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:21:55.390 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:21:55.390 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:21:55.390 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:21:55.390 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:21:55.390 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:21:55.390 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:21:55.391 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:55.391 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:21:55.391 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:55.391 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:21:55.391 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:55.391 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:21:55.391 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:21:55.391 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:21:55.391 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:21:55.391 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:55.391 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:21:55.391 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:21:55.391 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:55.391 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:55.392 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:55.392 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:21:55.392 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:21:55.392 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:55.392 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:55.392 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:55.392 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:55.392 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:21:55.392 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:21:55.392 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:21:55.392 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:21:55.392 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:21:55.392 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:55.392 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:55.392 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:21:55.393 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:21:55.393 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:21:55.393 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:21:55.393 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:55.393 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:21:55.393 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:21:55.393 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:55.393 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:55.393 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:55.393 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:21:55.393 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:21:55.393 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:55.393 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:55.393 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:55.394 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:55.394 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:21:55.394 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:21:55.394 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:21:55.394 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:21:55.394 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:21:55.394 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:55.394 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:55.394 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:21:55.394 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:21:55.394 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:21:55.394 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:21:55.394 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:55.395 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:21:55.395 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:21:55.395 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:55.395 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:55.395 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:55.395 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:21:55.395 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:21:55.395 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:55.395 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:55.395 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:55.395 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:55.395 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:21:55.395 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:21:55.395 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:21:55.395 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:21:55.396 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:21:55.396 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:55.396 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:55.396 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:55.396 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:21:55.396 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:21:55.396 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:55.396 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:21:55.396 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:21:55.396 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:21:55.396 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:55.396 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:55.396 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:21:55.396 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:55.396 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:21:55.397 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:21:55.397 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 3 2024-04-22T08:21:55.397 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:55.397 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:55.397 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:55.397 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:21:55.397 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:21:55.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.369+0000 7f0875d47640 1 -- 172.21.15.40:0/1983902027 >> v2:172.21.15.189:6800/30168 conn(0x7f085c0377d0 msgr2=0x7f085c039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:55.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.369+0000 7f0875d47640 1 --2- 172.21.15.40:0/1983902027 >> v2:172.21.15.189:6800/30168 conn(0x7f085c0377d0 0x7f085c039c90 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f0868007580 tx=0x7f086800b040 comp rx=0 tx=0).stop 2024-04-22T08:21:55.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.369+0000 7f0875d47640 10 monclient: shutdown 2024-04-22T08:21:55.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.369+0000 7f0875d47640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:55.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.369+0000 7f0875d47640 1 -- 172.21.15.40:0/1983902027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0870070eb0 msgr2=0x7f08701caac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:55.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.369+0000 7f0875d47640 1 --2- 172.21.15.40:0/1983902027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0870070eb0 0x7f08701caac0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0864033000 tx=0x7f08640046c0 comp rx=0 tx=0).stop 2024-04-22T08:21:55.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.369+0000 7f086ffff640 1 -- 172.21.15.40:0/1983902027 reap_dead start 2024-04-22T08:21:55.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.369+0000 7f086ffff640 1 -- 172.21.15.40:0/1983902027 reap_dead start 2024-04-22T08:21:55.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.369+0000 7f0875d47640 1 -- 172.21.15.40:0/1983902027 shutdown_connections 2024-04-22T08:21:55.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.369+0000 7f0875d47640 1 -- 172.21.15.40:0/1983902027 >> 172.21.15.40:0/1983902027 conn(0x7f08700a1780 msgr2=0x7f08700a0160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:55.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.369+0000 7f0875d47640 1 -- 172.21.15.40:0/1983902027 shutdown_connections 2024-04-22T08:21:55.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.369+0000 7f0875d47640 1 -- 172.21.15.40:0/1983902027 wait complete. 2024-04-22T08:21:55.400 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:21:55.400 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-22T08:21:55.400 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-22T08:21:55.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.500+0000 7f74e71f6640 10 monclient: get_monmap_and_config 2024-04-22T08:21:55.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.500+0000 7f74e71f6640 10 monclient: build_initial_monmap 2024-04-22T08:21:55.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.500+0000 7f74e71f6640 10 monclient: monmap: 2024-04-22T08:21:55.500 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:55.500 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:55.500 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:21:55.500876+0000 2024-04-22T08:21:55.500 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:21:55.500876+0000 2024-04-22T08:21:55.500 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:55.500 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:55.500 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:21:55.500 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:21:55.500 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:21:55.500 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:21:55.500 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:21:55.500 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:55.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.500+0000 7f74e71f6640 1 Processor -- start 2024-04-22T08:21:55.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.500+0000 7f74e71f6640 1 -- start start 2024-04-22T08:21:55.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.500+0000 7f74e71f6640 10 monclient: init 2024-04-22T08:21:55.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.501+0000 7f74e71f6640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:55.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.501+0000 7f74e71f6640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-04-22T08:21:55.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.501+0000 7f74e71f6640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f74e00a1c60 0x7f74e00a31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:55.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.501+0000 7f74e71f6640 10 monclient(hunting): picked mon.noname-a con 0x7f74e00a1c60 addr v2:172.21.15.40:3300/0 2024-04-22T08:21:55.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.501+0000 7f74e71f6640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f74e00a2040 0x7f74e0135bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:55.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.501+0000 7f74e71f6640 10 monclient(hunting): picked mon.noname-c con 0x7f74e00a2040 addr v2:172.21.15.189:3300/0 2024-04-22T08:21:55.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.501+0000 7f74e71f6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f74e00a24d0 addr v1:172.21.15.40:6789/0 2024-04-22T08:21:55.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.501+0000 7f74e71f6640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74e00a41f0 con 0x7f74e00a24d0 2024-04-22T08:21:55.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.501+0000 7f74e71f6640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:55.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.501+0000 7f74e71f6640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74e009fe90 con 0x7f74e00a1c60 2024-04-22T08:21:55.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.501+0000 7f74e71f6640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:55.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.501+0000 7f74e71f6640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74e00a0170 con 0x7f74e00a2040 2024-04-22T08:21:55.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.501+0000 7f74e71f6640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:55.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.501+0000 7f74e71f6640 10 monclient(hunting): authenticate will time out at 1389.495147s 2024-04-22T08:21:55.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.501+0000 7f74e4f92640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f74e00a1c60 0x7f74e00a31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:55.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.501+0000 7f74e5793640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f74e00a24d0 0x7f74e01360f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:36588/0 (socket says 172.21.15.40:36588) 2024-04-22T08:21:55.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.501+0000 7f74e5793640 1 -- 172.21.15.40:0/849162372 learned_addr learned my addr 172.21.15.40:0/849162372 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:21:55.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.501+0000 7f74e4f92640 10 monclient(hunting): get_auth_request con 0x7f74e00a1c60 auth_method 0 2024-04-22T08:21:55.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.501+0000 7f74e4f92640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:55.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.501+0000 7f74e4f92640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:55.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.501+0000 7f74e4f92640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:55.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.501+0000 7f74e4f92640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:55.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.501+0000 7f74e4f92640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:55.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.501+0000 7f74e4f92640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:55.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.501+0000 7f74dffff640 1 --2- 172.21.15.40:0/849162372 >> v2:172.21.15.189:3300/0 conn(0x7f74e00a2040 0x7f74e0135bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:55.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.501+0000 7f74e4f92640 10 monclient(hunting): handle_auth_done global_id 114135 payload 274 2024-04-22T08:21:55.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.501+0000 7f74e4f92640 1 -- 172.21.15.40:0/849162372 >> v2:172.21.15.189:3300/0 conn(0x7f74e00a2040 msgr2=0x7f74e0135bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:21:55.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.501+0000 7f74e4f92640 1 --2- 172.21.15.40:0/849162372 >> v2:172.21.15.189:3300/0 conn(0x7f74e00a2040 0x7f74e0135bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:55.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.501+0000 7f74e4f92640 1 -- 172.21.15.40:0/849162372 >> v1:172.21.15.40:6789/0 conn(0x7f74e00a24d0 legacy=0x7f74e01360f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:21:55.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.501+0000 7f74e4f92640 10 monclient: _finish_hunting 0 2024-04-22T08:21:55.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.501+0000 7f74e4f92640 1 monclient: found mon.noname-a 2024-04-22T08:21:55.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.501+0000 7f74e4f92640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:55.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.501+0000 7f74e4f92640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:21:55.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.501+0000 7f74e4f92640 1 -- 172.21.15.40:0/849162372 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74e00a0450 con 0x7f74e00a1c60 2024-04-22T08:21:55.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.501+0000 7f74dffff640 10 monclient: get_auth_request con 0x7f74e00a2040 auth_method 0 2024-04-22T08:21:55.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.501+0000 7f74dffff640 1 --2- 172.21.15.40:0/849162372 >> v2:172.21.15.189:3300/0 conn(0x7f74e00a2040 0x7f74e0135bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T08:21:55.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.504+0000 7f74e4f92640 1 -- 172.21.15.40:0/849162372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74e00a1c60 msgr2=0x7f74e00a31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:55.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.504+0000 7f74e4f92640 1 --2- 172.21.15.40:0/849162372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74e00a1c60 0x7f74e00a31d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f74e009f4b0 tx=0x7f74d802bd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a68cb966278f8183 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:55.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.505+0000 7f74df7fe640 1 -- 172.21.15.40:0/849162372 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d802cdc0 con 0x7f74e00a1c60 2024-04-22T08:21:55.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.505+0000 7f74df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:55.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.505+0000 7f74df7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:21:55.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.505+0000 7f74df7fe640 10 monclient: dump: 2024-04-22T08:21:55.513 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:55.513 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:55.513 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:55.514 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:55.514 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:55.514 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:55.514 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:55.514 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:55.514 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:55.514 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:55.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.505+0000 7f74df7fe640 10 monclient: _finish_auth 0 2024-04-22T08:21:55.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.505+0000 7f74df7fe640 10 monclient: _check_auth_tickets 2024-04-22T08:21:55.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.505+0000 7f74df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:55.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.505+0000 7f74df7fe640 1 -- 172.21.15.40:0/849162372 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74d8034440 con 0x7f74e00a1c60 2024-04-22T08:21:55.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.505+0000 7f74df7fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:55.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.505+0000 7f74df7fe640 1 -- 172.21.15.40:0/849162372 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d80347d0 con 0x7f74e00a1c60 2024-04-22T08:21:55.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.505+0000 7f74e71f6640 5 monclient: authenticate success, global_id 114135 2024-04-22T08:21:55.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.505+0000 7f74e71f6640 10 monclient: get_monmap_and_config success 2024-04-22T08:21:55.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.505+0000 7f74e71f6640 4 set_mon_vals no callback set 2024-04-22T08:21:55.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.505+0000 7f74df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:55.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.505+0000 7f74df7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:21:55.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.505+0000 7f74df7fe640 10 monclient: dump: 2024-04-22T08:21:55.515 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:55.515 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:55.515 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:55.515 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:55.515 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:55.515 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:55.515 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:55.515 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:55.515 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:55.516 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:55.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.505+0000 7f74e71f6640 10 monclient: shutdown 2024-04-22T08:21:55.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.505+0000 7f74e71f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:55.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.505+0000 7f74e71f6640 1 -- 172.21.15.40:0/849162372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74e00a1c60 msgr2=0x7f74e00a31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:55.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.505+0000 7f74e71f6640 1 --2- 172.21.15.40:0/849162372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74e00a1c60 0x7f74e00a31d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f74e009f4b0 tx=0x7f74d802bd20 comp rx=0 tx=0).stop 2024-04-22T08:21:55.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.505+0000 7f74e71f6640 1 -- 172.21.15.40:0/849162372 shutdown_connections 2024-04-22T08:21:55.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.505+0000 7f74e71f6640 1 --2- 172.21.15.40:0/849162372 >> v2:172.21.15.189:3300/0 conn(0x7f74e00a2040 0x7f74e0135bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:55.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.505+0000 7f74e71f6640 1 --2- 172.21.15.40:0/849162372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74e00a1c60 0x7f74e00a31d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:55.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.505+0000 7f74e71f6640 1 -- 172.21.15.40:0/849162372 >> 172.21.15.40:0/849162372 conn(0x7f74e00a89a0 msgr2=0x7f74e00a4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:55.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.505+0000 7f74e71f6640 1 -- 172.21.15.40:0/849162372 shutdown_connections 2024-04-22T08:21:55.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.505+0000 7f74e71f6640 1 -- 172.21.15.40:0/849162372 wait complete. 2024-04-22T08:21:55.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.506+0000 7f74e71f6640 10 monclient: build_initial_monmap 2024-04-22T08:21:55.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.506+0000 7f74e71f6640 10 monclient: monmap: 2024-04-22T08:21:55.516 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:55.517 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:21:55.517 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:21:55.517 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:21:55.517 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:55.517 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:55.517 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:21:55.517 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:21:55.517 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:21:55.517 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:55.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.506+0000 7f74e71f6640 1 Processor -- start 2024-04-22T08:21:55.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.506+0000 7f74e71f6640 1 -- start start 2024-04-22T08:21:55.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.506+0000 7f74e71f6640 10 monclient: init 2024-04-22T08:21:55.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74e71f6640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:55.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74e71f6640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-22T08:21:55.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74e71f6640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74e00a1c60 0x7f74e01cb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:55.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74e71f6640 10 monclient(hunting): picked mon.noname-c con 0x7f74e00a1c60 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:55.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74e71f6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74e00a2040 0x7f74e0121200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:55.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74e71f6640 10 monclient(hunting): picked mon.noname-a con 0x7f74e00a2040 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:21:55.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74e71f6640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74e00a24d0 0x7f74e0121740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:55.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74e71f6640 10 monclient(hunting): picked mon.noname-b con 0x7f74e00a24d0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:21:55.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74e71f6640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:55.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74e71f6640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74e0123a20 con 0x7f74e00a2040 2024-04-22T08:21:55.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74e71f6640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:55.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74e71f6640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74e00a4790 con 0x7f74e00a24d0 2024-04-22T08:21:55.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74e71f6640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:55.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74e71f6640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74e0134fd0 con 0x7f74e00a1c60 2024-04-22T08:21:55.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74e71f6640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:55.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74e71f6640 10 monclient(hunting): authenticate will time out at 1389.501014s 2024-04-22T08:21:55.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74dffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74e00a2040 0x7f74e0121200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:55.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74e4f92640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74e00a1c60 0x7f74e01cb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:55.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74e4f92640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74e00a1c60 0x7f74e01cb0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:41852/0 (socket says 172.21.15.40:41852) 2024-04-22T08:21:55.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74dffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74e00a2040 0x7f74e0121200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56870/0 (socket says 172.21.15.40:56870) 2024-04-22T08:21:55.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74e4f92640 1 -- 172.21.15.40:0/3010173806 learned_addr learned my addr 172.21.15.40:0/3010173806 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:21:55.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74e4f92640 10 monclient(hunting): get_auth_request con 0x7f74e00a1c60 auth_method 0 2024-04-22T08:21:55.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74e4f92640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:55.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74e4f92640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:55.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74e4f92640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:55.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74dffff640 10 monclient(hunting): get_auth_request con 0x7f74e00a2040 auth_method 0 2024-04-22T08:21:55.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:55.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74dffff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:55.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74dffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:55.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:55.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:55.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:55.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74e4f92640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:55.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74e4f92640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:55.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74e4f92640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:55.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74e5793640 1 --2- 172.21.15.40:0/3010173806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74e00a24d0 0x7f74e0121740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:55.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74e5793640 10 monclient(hunting): get_auth_request con 0x7f74e00a24d0 auth_method 0 2024-04-22T08:21:55.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74e5793640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:55.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74e5793640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:55.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74e5793640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:55.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74dffff640 10 monclient(hunting): handle_auth_done global_id 114138 payload 824 2024-04-22T08:21:55.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74dffff640 1 -- 172.21.15.40:0/3010173806 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74e00a1c60 msgr2=0x7f74e01cb0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:55.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74dffff640 1 --2- 172.21.15.40:0/3010173806 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74e00a1c60 0x7f74e01cb0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:55.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74dffff640 1 -- 172.21.15.40:0/3010173806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74e00a24d0 msgr2=0x7f74e0121740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:55.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74dffff640 1 --2- 172.21.15.40:0/3010173806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74e00a24d0 0x7f74e0121740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:55.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74dffff640 10 monclient: _finish_hunting 0 2024-04-22T08:21:55.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74dffff640 1 monclient: found mon.noname-a 2024-04-22T08:21:55.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:55.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74dffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:21:55.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74dffff640 1 -- 172.21.15.40:0/3010173806 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74d8009a70 con 0x7f74e00a2040 2024-04-22T08:21:55.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.507+0000 7f74e5793640 1 --2- 172.21.15.40:0/3010173806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74e00a24d0 0x7f74e0121740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T08:21:55.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.509+0000 7f74dffff640 1 -- 172.21.15.40:0/3010173806 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74e00a2040 msgr2=0x7f74e0121200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:55.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.509+0000 7f74dffff640 1 --2- 172.21.15.40:0/3010173806 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74e00a2040 0x7f74e0121200 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f74d400c7f0 tx=0x7f74d400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:55.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.510+0000 7f74dd7fa640 1 -- 172.21.15.40:0/3010173806 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d4004220 con 0x7f74e00a2040 2024-04-22T08:21:55.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.510+0000 7f74dd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:55.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.510+0000 7f74dd7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:21:55.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.510+0000 7f74dd7fa640 10 monclient: dump: 2024-04-22T08:21:55.522 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:55.522 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:55.522 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:55.522 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:55.522 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:55.522 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:55.522 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:55.522 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:55.522 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:55.522 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:55.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.510+0000 7f74dd7fa640 10 monclient: _finish_auth 0 2024-04-22T08:21:55.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.510+0000 7f74dd7fa640 10 monclient: _check_auth_tickets 2024-04-22T08:21:55.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.510+0000 7f74dd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:55.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.510+0000 7f74e71f6640 5 monclient: authenticate success, global_id 114138 2024-04-22T08:21:55.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.510+0000 7f74dd7fa640 1 -- 172.21.15.40:0/3010173806 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74d4004380 con 0x7f74e00a2040 2024-04-22T08:21:55.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.510+0000 7f74e71f6640 10 monclient: _renew_subs 2024-04-22T08:21:55.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.510+0000 7f74e71f6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:21:55.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.510+0000 7f74e71f6640 1 -- 172.21.15.40:0/3010173806 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74e01251b0 con 0x7f74e00a2040 2024-04-22T08:21:55.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.510+0000 7f74dd7fa640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:55.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.510+0000 7f74dd7fa640 1 -- 172.21.15.40:0/3010173806 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d4007a50 con 0x7f74e00a2040 2024-04-22T08:21:55.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.510+0000 7f74dd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:55.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.510+0000 7f74dd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:21:55.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.510+0000 7f74dd7fa640 10 monclient: dump: 2024-04-22T08:21:55.523 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:55.523 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:55.523 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:55.524 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:55.524 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:55.524 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:55.524 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:55.524 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:55.524 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:55.524 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:55.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.510+0000 7f74deffd640 4 set_mon_vals no callback set 2024-04-22T08:21:55.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.510+0000 7f74e71f6640 10 monclient: _renew_subs 2024-04-22T08:21:55.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.510+0000 7f74e71f6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:21:55.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.510+0000 7f74e71f6640 1 -- 172.21.15.40:0/3010173806 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74e0122e50 con 0x7f74e00a2040 2024-04-22T08:21:55.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.510+0000 7f74e71f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:55.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.510+0000 7f74dd7fa640 1 -- 172.21.15.40:0/3010173806 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f74d4007bf0 con 0x7f74e00a2040 2024-04-22T08:21:55.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.510+0000 7f74e71f6640 1 --2- 172.21.15.40:0/3010173806 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74e0125b20 0x7f74e0121c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:55.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.510+0000 7f74e71f6640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:55.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.510+0000 7f74e71f6640 1 -- 172.21.15.40:0/3010173806 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74e0066650 con 0x7f74e0125b20 2024-04-22T08:21:55.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.511+0000 7f74e4f92640 1 --2- 172.21.15.40:0/3010173806 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74e0125b20 0x7f74e0121c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:55.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.511+0000 7f74e4f92640 10 monclient: get_auth_request con 0x7f74e0125b20 auth_method 0 2024-04-22T08:21:55.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.511+0000 7f74e4f92640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:55.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.511+0000 7f74e4f92640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:55.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.511+0000 7f74e4f92640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:55.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.511+0000 7f74e4f92640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:55.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.511+0000 7f74e4f92640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:55.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.511+0000 7f74dd7fa640 1 --2- 172.21.15.40:0/3010173806 >> v2:172.21.15.189:6800/30168 conn(0x7f74cc037850 0x7f74cc039d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:55.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.511+0000 7f74e4f92640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:55.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.511+0000 7f74dd7fa640 1 -- 172.21.15.40:0/3010173806 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(308..308 src has 239..308) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f74d404a190 con 0x7f74e00a2040 2024-04-22T08:21:55.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.511+0000 7f74e4f92640 10 monclient(hunting): handle_auth_done global_id 104303 payload 824 2024-04-22T08:21:55.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.511+0000 7f74e4f92640 1 -- 172.21.15.40:0/3010173806 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f74e00a6510 con 0x7f74e0125b20 2024-04-22T08:21:55.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.511+0000 7f74e5793640 1 --2- 172.21.15.40:0/3010173806 >> v2:172.21.15.189:6800/30168 conn(0x7f74cc037850 0x7f74cc039d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:55.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.511+0000 7f74e5793640 10 monclient: get_auth_request con 0x7f74cc037850 auth_method 0 2024-04-22T08:21:55.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.513+0000 7f74e4f92640 1 -- 172.21.15.40:0/3010173806 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74e0125b20 msgr2=0x7f74e0121c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:55.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.513+0000 7f74e4f92640 1 --2- 172.21.15.40:0/3010173806 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74e0125b20 0x7f74e0121c80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f74e0123bf0 tx=0x7f74d8004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:55.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.513+0000 7f74dd7fa640 1 -- 172.21.15.40:0/3010173806 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d803f800 con 0x7f74e0125b20 2024-04-22T08:21:55.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.513+0000 7f74dd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:55.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.513+0000 7f74dd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:21:55.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.513+0000 7f74dd7fa640 10 monclient: dump: 2024-04-22T08:21:55.526 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:55.526 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:55.526 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:55.526 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:55.527 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:55.527 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:55.527 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:55.527 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:55.527 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:55.527 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:55.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.513+0000 7f74dd7fa640 1 -- 172.21.15.40:0/3010173806 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f74d803f960 con 0x7f74e0125b20 2024-04-22T08:21:55.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.513+0000 7f74dd7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:55.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.513+0000 7f74dd7fa640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:21:55.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.513+0000 7f74dd7fa640 1 -- 172.21.15.40:0/3010173806 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74e0125b20 msgr2=0x7f74e0121c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:55.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.513+0000 7f74dd7fa640 1 --2- 172.21.15.40:0/3010173806 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74e0125b20 0x7f74e0121c80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f74e0123bf0 tx=0x7f74d8004a40 comp rx=0 tx=0).stop 2024-04-22T08:21:55.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.513+0000 7f74dd7fa640 1 -- 172.21.15.40:0/3010173806 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74e0125b20 msgr2=0x7f74e0121c80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:21:55.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.513+0000 7f74dd7fa640 1 --2- 172.21.15.40:0/3010173806 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74e0125b20 0x7f74e0121c80 secure :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f74e0123bf0 tx=0x7f74d8004a40 comp rx=0 tx=0).stop 2024-04-22T08:21:55.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.514+0000 7f74e5793640 1 --2- 172.21.15.40:0/3010173806 >> v2:172.21.15.189:6800/30168 conn(0x7f74cc037850 0x7f74cc039d10 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f74d00050e0 tx=0x7f74d0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:55.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.520+0000 7f74e71f6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:21:55.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.520+0000 7f74e71f6640 1 --2- 172.21.15.40:0/3010173806 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74e01cb5e0 0x7f74e01221c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:55.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.520+0000 7f74e71f6640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:55.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.520+0000 7f74e71f6640 1 -- 172.21.15.40:0/3010173806 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74e004e9b0 con 0x7f74e01cb5e0 2024-04-22T08:21:55.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.520+0000 7f74e4f92640 1 --2- 172.21.15.40:0/3010173806 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74e01cb5e0 0x7f74e01221c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:55.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.520+0000 7f74e4f92640 10 monclient: get_auth_request con 0x7f74e01cb5e0 auth_method 0 2024-04-22T08:21:55.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.520+0000 7f74e4f92640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:55.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.520+0000 7f74e4f92640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:55.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.520+0000 7f74e4f92640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:55.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.520+0000 7f74e4f92640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:55.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.520+0000 7f74e4f92640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:55.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.520+0000 7f74e4f92640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:55.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.520+0000 7f74e4f92640 10 monclient(hunting): handle_auth_done global_id 104306 payload 824 2024-04-22T08:21:55.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.520+0000 7f74e4f92640 1 -- 172.21.15.40:0/3010173806 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f74e0124c80 con 0x7f74e01cb5e0 2024-04-22T08:21:55.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.522+0000 7f74e4f92640 1 -- 172.21.15.40:0/3010173806 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74e01cb5e0 msgr2=0x7f74e01221c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:55.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.522+0000 7f74e4f92640 1 --2- 172.21.15.40:0/3010173806 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74e01cb5e0 0x7f74e01221c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f74e0124800 tx=0x7f74d8031000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:55.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.522+0000 7f74dd7fa640 1 -- 172.21.15.40:0/3010173806 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d8004240 con 0x7f74e01cb5e0 2024-04-22T08:21:55.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.522+0000 7f74dd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:55.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.522+0000 7f74dd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:21:55.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.522+0000 7f74dd7fa640 10 monclient: dump: 2024-04-22T08:21:55.531 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:55.531 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:55.531 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:55.531 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:55.531 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:55.532 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:55.532 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:55.532 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:55.532 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:55.532 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:55.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.523+0000 7f74dd7fa640 1 -- 172.21.15.40:0/3010173806 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4631 (secure 0 0 0) 0x7f74d8003e90 con 0x7f74e01cb5e0 2024-04-22T08:21:55.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.523+0000 7f74dd7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:21:55.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.523+0000 7f74dd7fa640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:21:55.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.523+0000 7f74dd7fa640 1 -- 172.21.15.40:0/3010173806 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74e01cb5e0 msgr2=0x7f74e01221c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:55.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.523+0000 7f74dd7fa640 1 --2- 172.21.15.40:0/3010173806 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74e01cb5e0 0x7f74e01221c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f74e0124800 tx=0x7f74d8031000 comp rx=0 tx=0).stop 2024-04-22T08:21:55.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.523+0000 7f74dd7fa640 1 -- 172.21.15.40:0/3010173806 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74e01cb5e0 msgr2=0x7f74e01221c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:21:55.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.523+0000 7f74dd7fa640 1 --2- 172.21.15.40:0/3010173806 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74e01cb5e0 0x7f74e01221c0 secure :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f74e0124800 tx=0x7f74d8031000 comp rx=0 tx=0).stop 2024-04-22T08:21:55.533 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:21:55.533 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:21:55.533 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:21:55.533 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:21:55.533 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 92, 2024-04-22T08:21:55.533 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:21:55.533 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:21:55.533 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:21:55.533 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:21:55.533 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:55.534 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 0, 2024-04-22T08:21:55.534 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:21:55.534 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:21:55.534 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:21:55.534 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:21:55.534 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:21:55.534 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:21:55.534 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:21:55.534 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:21:55.534 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:21:55.534 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:21:55.534 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:21:55.534 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:21:55.535 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:21:55.535 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:55.535 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:21:55.535 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:21:55.535 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:21:55.535 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:21:55.535 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:21:55.535 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:21:55.535 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:21:55.535 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:21:55.535 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:21:55.535 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:21:55.535 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:21:55.535 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:21:55.535 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:55.536 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:55.536 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:21:55.536 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:21:55.536 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:21:55.536 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:21:55.536 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:21:55.536 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:21:55.536 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:21:55.536 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:21:55.536 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:21:55.536 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:21:55.536 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:21:55.536 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:21:55.536 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:21:55.537 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:21:55.537 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:21:55.537 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:21:55.537 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:21:55.537 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:21:55.537 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:21:55.537 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:21:55.537 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:21:55.537 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:21:55.537 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:21:55.537 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:21:55.537 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:21:55.537 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:21:55.537 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:21:55.538 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:55.538 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:21:55.538 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:55.538 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:21:55.538 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:55.538 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:21:55.538 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:21:55.538 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:21:55.538 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:21:55.538 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:55.538 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:21:55.538 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:21:55.538 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:55.538 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:55.539 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:55.539 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:21:55.539 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:21:55.539 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:55.539 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:55.539 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:55.539 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:55.539 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:21:55.539 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:21:55.539 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:21:55.539 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:21:55.539 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:21:55.539 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:55.539 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:55.539 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:21:55.539 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:21:55.540 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:21:55.540 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:21:55.540 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:55.540 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:21:55.540 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:21:55.540 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:55.540 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:55.540 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:55.540 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:21:55.540 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:21:55.540 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:55.540 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:55.540 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:55.540 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:55.541 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:21:55.541 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:21:55.541 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:21:55.541 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:21:55.541 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:21:55.541 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:55.541 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:55.541 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:21:55.541 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:21:55.541 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:21:55.541 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:21:55.541 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:55.541 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:21:55.542 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:21:55.542 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:55.542 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:55.542 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:55.542 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:21:55.542 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:21:55.542 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:55.542 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:55.542 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:55.542 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:55.542 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:21:55.542 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:21:55.543 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:21:55.543 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:21:55.543 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:21:55.543 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:55.543 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:55.543 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:55.543 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:21:55.543 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:21:55.543 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:55.543 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:21:55.543 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:21:55.543 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:21:55.543 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:55.543 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:55.543 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:21:55.543 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:55.544 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:21:55.544 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:21:55.544 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:21:55.544 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:55.544 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:55.544 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:21:55.544 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:55.544 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:21:55.544 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:21:55.544 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:21:55.544 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:55.544 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:55.544 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:55.544 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:21:55.544 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:21:55.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.523+0000 7f74e71f6640 1 -- 172.21.15.40:0/3010173806 >> v2:172.21.15.189:6800/30168 conn(0x7f74cc037850 msgr2=0x7f74cc039d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:55.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.523+0000 7f74e71f6640 1 --2- 172.21.15.40:0/3010173806 >> v2:172.21.15.189:6800/30168 conn(0x7f74cc037850 0x7f74cc039d10 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f74d00050e0 tx=0x7f74d0009040 comp rx=0 tx=0).stop 2024-04-22T08:21:55.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.523+0000 7f74e71f6640 10 monclient: shutdown 2024-04-22T08:21:55.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.523+0000 7f74e71f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:55.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.523+0000 7f74e71f6640 1 -- 172.21.15.40:0/3010173806 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74e00a2040 msgr2=0x7f74e0121200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:55.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.523+0000 7f74e71f6640 1 --2- 172.21.15.40:0/3010173806 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74e00a2040 0x7f74e0121200 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f74d400c7f0 tx=0x7f74d400ccb0 comp rx=0 tx=0).stop 2024-04-22T08:21:55.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.523+0000 7f74e5793640 1 -- 172.21.15.40:0/3010173806 reap_dead start 2024-04-22T08:21:55.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.523+0000 7f74e5793640 1 -- 172.21.15.40:0/3010173806 reap_dead start 2024-04-22T08:21:55.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.523+0000 7f74e71f6640 1 -- 172.21.15.40:0/3010173806 shutdown_connections 2024-04-22T08:21:55.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.523+0000 7f74e71f6640 1 -- 172.21.15.40:0/3010173806 >> 172.21.15.40:0/3010173806 conn(0x7f74e00a89a0 msgr2=0x7f74e00a6b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:55.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.523+0000 7f74e71f6640 1 -- 172.21.15.40:0/3010173806 shutdown_connections 2024-04-22T08:21:55.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.523+0000 7f74e71f6640 1 -- 172.21.15.40:0/3010173806 wait complete. 2024-04-22T08:21:55.547 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:21:55.547 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-22T08:21:55.547 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-22T08:21:55.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.647+0000 7f21d9ce0640 10 monclient: get_monmap_and_config 2024-04-22T08:21:55.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.647+0000 7f21d9ce0640 10 monclient: build_initial_monmap 2024-04-22T08:21:55.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.647+0000 7f21d9ce0640 10 monclient: monmap: 2024-04-22T08:21:55.647 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:55.647 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:55.647 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:21:55.648396+0000 2024-04-22T08:21:55.647 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:21:55.648396+0000 2024-04-22T08:21:55.647 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:55.648 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:55.648 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:21:55.648 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:21:55.648 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:21:55.648 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:21:55.648 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:21:55.648 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:55.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.648+0000 7f21d9ce0640 1 Processor -- start 2024-04-22T08:21:55.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.648+0000 7f21d9ce0640 1 -- start start 2024-04-22T08:21:55.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.648+0000 7f21d9ce0640 10 monclient: init 2024-04-22T08:21:55.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.648+0000 7f21d9ce0640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:55.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.648+0000 7f21d9ce0640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-04-22T08:21:55.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.648+0000 7f21d9ce0640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f21d40736d0 0x7f21d4135f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:55.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.648+0000 7f21d9ce0640 10 monclient(hunting): picked mon.noname-a con 0x7f21d40736d0 addr v2:172.21.15.40:3300/0 2024-04-22T08:21:55.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.648+0000 7f21d9ce0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f21d4070eb0 addr v1:172.21.15.40:6789/0 2024-04-22T08:21:55.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.648+0000 7f21d9ce0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f21d4071320 addr v1:172.21.15.189:6789/0 2024-04-22T08:21:55.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.648+0000 7f21d9ce0640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21d4073ab0 con 0x7f21d4070eb0 2024-04-22T08:21:55.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.648+0000 7f21d9ce0640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21d409ec20 con 0x7f21d4071320 2024-04-22T08:21:55.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.648+0000 7f21d9ce0640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:55.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.648+0000 7f21d9ce0640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21d4136510 con 0x7f21d40736d0 2024-04-22T08:21:55.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.648+0000 7f21d9ce0640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:55.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.648+0000 7f21d9ce0640 10 monclient(hunting): authenticate will time out at 1389.642687s 2024-04-22T08:21:55.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.649+0000 7f21d37fe640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f21d40736d0 0x7f21d4135f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:55.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.649+0000 7f21d2ffd640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f21d4070eb0 0x7f21d409de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:36594/0 (socket says 172.21.15.40:36594) 2024-04-22T08:21:55.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.649+0000 7f21d2ffd640 1 -- 172.21.15.40:0/3282253263 learned_addr learned my addr 172.21.15.40:0/3282253263 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:21:55.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.649+0000 7f21d37fe640 10 monclient(hunting): get_auth_request con 0x7f21d40736d0 auth_method 0 2024-04-22T08:21:55.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.649+0000 7f21d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:55.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.649+0000 7f21d37fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:55.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.649+0000 7f21d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:55.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.649+0000 7f21d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:55.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.649+0000 7f21d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:55.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.649+0000 7f21d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:55.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.649+0000 7f21d37fe640 10 monclient(hunting): handle_auth_done global_id 114141 payload 274 2024-04-22T08:21:55.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.649+0000 7f21d37fe640 1 -- 172.21.15.40:0/3282253263 >> v1:172.21.15.189:6789/0 conn(0x7f21d4071320 legacy=0x7f21d409e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:21:55.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.649+0000 7f21d37fe640 1 -- 172.21.15.40:0/3282253263 >> v1:172.21.15.40:6789/0 conn(0x7f21d4070eb0 legacy=0x7f21d409de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:21:55.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.649+0000 7f21d37fe640 10 monclient: _finish_hunting 0 2024-04-22T08:21:55.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.649+0000 7f21d37fe640 1 monclient: found mon.noname-a 2024-04-22T08:21:55.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.649+0000 7f21d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:55.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.649+0000 7f21d37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:21:55.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.649+0000 7f21d37fe640 1 -- 172.21.15.40:0/3282253263 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d4071790 con 0x7f21d40736d0 2024-04-22T08:21:55.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.652+0000 7f21d37fe640 1 -- 172.21.15.40:0/3282253263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21d40736d0 msgr2=0x7f21d4135f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:55.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.652+0000 7f21d37fe640 1 --2- 172.21.15.40:0/3282253263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21d40736d0 0x7f21d4135f10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f21c40097a0 tx=0x7f21c402b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b169a8be59eade8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:55.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.652+0000 7f21d27fc640 1 -- 172.21.15.40:0/3282253263 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c4030a00 con 0x7f21d40736d0 2024-04-22T08:21:55.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.652+0000 7f21d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:55.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.652+0000 7f21d27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:21:55.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.652+0000 7f21d27fc640 10 monclient: dump: 2024-04-22T08:21:55.660 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:55.660 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:55.660 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:55.660 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:55.660 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:55.660 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:55.660 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:55.660 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:55.660 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:55.660 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:55.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.652+0000 7f21d27fc640 10 monclient: _finish_auth 0 2024-04-22T08:21:55.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.652+0000 7f21d27fc640 10 monclient: _check_auth_tickets 2024-04-22T08:21:55.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.652+0000 7f21d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:55.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.652+0000 7f21d9ce0640 5 monclient: authenticate success, global_id 114141 2024-04-22T08:21:55.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.652+0000 7f21d9ce0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:21:55.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.652+0000 7f21d27fc640 1 -- 172.21.15.40:0/3282253263 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21c402e050 con 0x7f21d40736d0 2024-04-22T08:21:55.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.652+0000 7f21d27fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:55.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.652+0000 7f21d9ce0640 10 monclient: get_monmap_and_config success 2024-04-22T08:21:55.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.652+0000 7f21d27fc640 1 -- 172.21.15.40:0/3282253263 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c4030d80 con 0x7f21d40736d0 2024-04-22T08:21:55.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.652+0000 7f21d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:55.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.652+0000 7f21d9ce0640 4 set_mon_vals no callback set 2024-04-22T08:21:55.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.652+0000 7f21d27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:21:55.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.652+0000 7f21d27fc640 10 monclient: dump: 2024-04-22T08:21:55.661 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:55.662 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:55.662 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:55.662 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:55.662 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:55.662 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:55.662 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:55.662 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:55.662 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:55.662 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:55.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.652+0000 7f21d9ce0640 10 monclient: shutdown 2024-04-22T08:21:55.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.652+0000 7f21d9ce0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:55.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.652+0000 7f21d9ce0640 1 -- 172.21.15.40:0/3282253263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21d40736d0 msgr2=0x7f21d4135f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:55.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.652+0000 7f21d9ce0640 1 --2- 172.21.15.40:0/3282253263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21d40736d0 0x7f21d4135f10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f21c40097a0 tx=0x7f21c402b710 comp rx=0 tx=0).stop 2024-04-22T08:21:55.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.652+0000 7f21d9ce0640 1 -- 172.21.15.40:0/3282253263 shutdown_connections 2024-04-22T08:21:55.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.652+0000 7f21d9ce0640 1 --2- 172.21.15.40:0/3282253263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21d40736d0 0x7f21d4135f10 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:55.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.652+0000 7f21d9ce0640 1 -- 172.21.15.40:0/3282253263 >> 172.21.15.40:0/3282253263 conn(0x7f21d40a1780 msgr2=0x7f21d40a1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:55.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.652+0000 7f21d9ce0640 1 -- 172.21.15.40:0/3282253263 shutdown_connections 2024-04-22T08:21:55.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.652+0000 7f21d9ce0640 1 -- 172.21.15.40:0/3282253263 wait complete. 2024-04-22T08:21:55.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.653+0000 7f21d9ce0640 10 monclient: build_initial_monmap 2024-04-22T08:21:55.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.653+0000 7f21d9ce0640 10 monclient: monmap: 2024-04-22T08:21:55.663 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:55.663 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:21:55.663 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:21:55.663 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:21:55.663 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:55.663 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:55.664 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:21:55.664 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:21:55.664 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:21:55.664 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:55.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.653+0000 7f21d9ce0640 1 Processor -- start 2024-04-22T08:21:55.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d9ce0640 1 -- start start 2024-04-22T08:21:55.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d9ce0640 10 monclient: init 2024-04-22T08:21:55.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d9ce0640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:55.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d9ce0640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-22T08:21:55.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d9ce0640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f21d4070eb0 0x7f21d41cae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:55.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d9ce0640 10 monclient(hunting): picked mon.noname-c con 0x7f21d4070eb0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:55.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d9ce0640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21d4071320 0x7f21d41cb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:55.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d9ce0640 10 monclient(hunting): picked mon.noname-b con 0x7f21d4071320 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:21:55.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d9ce0640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21d40736d0 0x7f21d4138130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:55.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d9ce0640 10 monclient(hunting): picked mon.noname-a con 0x7f21d40736d0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:21:55.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d9ce0640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:55.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d9ce0640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d413bd90 con 0x7f21d40736d0 2024-04-22T08:21:55.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d9ce0640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:55.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d9ce0640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d413b6d0 con 0x7f21d4071320 2024-04-22T08:21:55.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d9ce0640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:55.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d9ce0640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21d40a4090 con 0x7f21d4070eb0 2024-04-22T08:21:55.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d9ce0640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:55.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d9ce0640 10 monclient(hunting): authenticate will time out at 1389.648289s 2024-04-22T08:21:55.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d37fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f21d4070eb0 0x7f21d41cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:55.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d37fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f21d4070eb0 0x7f21d41cae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:41890/0 (socket says 172.21.15.40:41890) 2024-04-22T08:21:55.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d37fe640 1 -- 172.21.15.40:0/54925935 learned_addr learned my addr 172.21.15.40:0/54925935 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:21:55.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d3fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21d40736d0 0x7f21d4138130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:55.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d37fe640 10 monclient(hunting): get_auth_request con 0x7f21d4070eb0 auth_method 0 2024-04-22T08:21:55.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:55.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d37fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:55.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:55.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d3fff640 10 monclient(hunting): get_auth_request con 0x7f21d40736d0 auth_method 0 2024-04-22T08:21:55.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:55.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d3fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:55.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:55.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:55.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:55.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:55.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:55.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:55.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:55.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d2ffd640 1 --2- 172.21.15.40:0/54925935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21d4071320 0x7f21d41cb3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:55.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d2ffd640 10 monclient(hunting): get_auth_request con 0x7f21d4071320 auth_method 0 2024-04-22T08:21:55.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:55.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:55.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:55.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d37fe640 10 monclient(hunting): handle_auth_done global_id 104309 payload 824 2024-04-22T08:21:55.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d37fe640 1 -- 172.21.15.40:0/54925935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21d4071320 msgr2=0x7f21d41cb3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:55.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d37fe640 1 --2- 172.21.15.40:0/54925935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21d4071320 0x7f21d41cb3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:55.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d37fe640 1 -- 172.21.15.40:0/54925935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21d40736d0 msgr2=0x7f21d4138130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:55.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d37fe640 1 --2- 172.21.15.40:0/54925935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21d40736d0 0x7f21d4138130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:55.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d37fe640 10 monclient: _finish_hunting 0 2024-04-22T08:21:55.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d37fe640 1 monclient: found mon.noname-c 2024-04-22T08:21:55.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:55.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:21:55.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d37fe640 1 -- 172.21.15.40:0/54925935 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c4009400 con 0x7f21d4070eb0 2024-04-22T08:21:55.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.654+0000 7f21d3fff640 1 --2- 172.21.15.40:0/54925935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21d40736d0 0x7f21d4138130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:21:55.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.655+0000 7f21d2ffd640 1 --2- 172.21.15.40:0/54925935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21d4071320 0x7f21d41cb3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T08:21:55.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.657+0000 7f21d37fe640 1 -- 172.21.15.40:0/54925935 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f21d4070eb0 msgr2=0x7f21d41cae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:55.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.657+0000 7f21d37fe640 1 --2- 172.21.15.40:0/54925935 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f21d4070eb0 0x7f21d41cae80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f21c40098c0 tx=0x7f21c40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:55.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.657+0000 7f21bbfff640 1 -- 172.21.15.40:0/54925935 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c4032c60 con 0x7f21d4070eb0 2024-04-22T08:21:55.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.657+0000 7f21bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:55.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.657+0000 7f21bbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:21:55.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.657+0000 7f21bbfff640 10 monclient: dump: 2024-04-22T08:21:55.669 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:55.669 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:55.669 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:55.669 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:55.669 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:55.669 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:55.669 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:55.669 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:55.669 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:55.669 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:55.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.657+0000 7f21bbfff640 10 monclient: _finish_auth 0 2024-04-22T08:21:55.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.657+0000 7f21bbfff640 10 monclient: _check_auth_tickets 2024-04-22T08:21:55.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.657+0000 7f21bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:55.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.657+0000 7f21d9ce0640 5 monclient: authenticate success, global_id 104309 2024-04-22T08:21:55.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.657+0000 7f21bbfff640 1 -- 172.21.15.40:0/54925935 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21c4032de0 con 0x7f21d4070eb0 2024-04-22T08:21:55.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.657+0000 7f21d9ce0640 10 monclient: _renew_subs 2024-04-22T08:21:55.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.657+0000 7f21d9ce0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:21:55.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.657+0000 7f21d9ce0640 1 -- 172.21.15.40:0/54925935 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21d4135350 con 0x7f21d4070eb0 2024-04-22T08:21:55.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.657+0000 7f21bbfff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:55.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.657+0000 7f21bbfff640 1 -- 172.21.15.40:0/54925935 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c403c6b0 con 0x7f21d4070eb0 2024-04-22T08:21:55.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.657+0000 7f21bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:55.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.657+0000 7f21bbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:21:55.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.657+0000 7f21bbfff640 10 monclient: dump: 2024-04-22T08:21:55.671 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:55.671 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:55.671 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:55.671 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:55.671 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:55.671 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:55.671 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:55.671 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:55.671 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:55.671 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:55.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.657+0000 7f21d9ce0640 10 monclient: _renew_subs 2024-04-22T08:21:55.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.657+0000 7f21d9ce0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:21:55.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.657+0000 7f21d9ce0640 1 -- 172.21.15.40:0/54925935 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21d413a4d0 con 0x7f21d4070eb0 2024-04-22T08:21:55.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.657+0000 7f21d1ffb640 4 set_mon_vals no callback set 2024-04-22T08:21:55.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.658+0000 7f21bbfff640 1 -- 172.21.15.40:0/54925935 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f21c4039040 con 0x7f21d4070eb0 2024-04-22T08:21:55.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.658+0000 7f21d9ce0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:55.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.658+0000 7f21d9ce0640 1 --2- 172.21.15.40:0/54925935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21d413c390 0x7f21d4138670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:55.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.658+0000 7f21d9ce0640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:55.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.658+0000 7f21d9ce0640 1 -- 172.21.15.40:0/54925935 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d4066900 con 0x7f21d413c390 2024-04-22T08:21:55.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.658+0000 7f21bbfff640 1 --2- 172.21.15.40:0/54925935 >> v2:172.21.15.189:6800/30168 conn(0x7f21c00377d0 0x7f21c0039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:55.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.658+0000 7f21bbfff640 1 -- 172.21.15.40:0/54925935 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(308..308 src has 239..308) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f21c406c1e0 con 0x7f21d4070eb0 2024-04-22T08:21:55.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.658+0000 7f21d2ffd640 1 --2- 172.21.15.40:0/54925935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21d413c390 0x7f21d4138670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:55.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.658+0000 7f21d2ffd640 10 monclient: get_auth_request con 0x7f21d413c390 auth_method 0 2024-04-22T08:21:55.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.658+0000 7f21d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:55.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.658+0000 7f21d2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:55.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.658+0000 7f21d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:55.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.658+0000 7f21d3fff640 1 --2- 172.21.15.40:0/54925935 >> v2:172.21.15.189:6800/30168 conn(0x7f21c00377d0 0x7f21c0039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:55.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.658+0000 7f21d3fff640 10 monclient: get_auth_request con 0x7f21c00377d0 auth_method 0 2024-04-22T08:21:55.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.658+0000 7f21d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:55.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.658+0000 7f21d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:55.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.658+0000 7f21d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:55.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.659+0000 7f21d2ffd640 10 monclient(hunting): handle_auth_done global_id 65149 payload 824 2024-04-22T08:21:55.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.659+0000 7f21d2ffd640 1 -- 172.21.15.40:0/54925935 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f21d406d450 con 0x7f21d413c390 2024-04-22T08:21:55.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.661+0000 7f21d3fff640 1 --2- 172.21.15.40:0/54925935 >> v2:172.21.15.189:6800/30168 conn(0x7f21c00377d0 0x7f21c0039c90 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f21d413dd50 tx=0x7f21cc003d10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:55.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.661+0000 7f21d2ffd640 1 -- 172.21.15.40:0/54925935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21d413c390 msgr2=0x7f21d4138670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:55.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.661+0000 7f21d2ffd640 1 --2- 172.21.15.40:0/54925935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21d413c390 0x7f21d4138670 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f21c800d700 tx=0x7f21c800dbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:55.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.661+0000 7f21bbfff640 1 -- 172.21.15.40:0/54925935 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c8009920 con 0x7f21d413c390 2024-04-22T08:21:55.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.661+0000 7f21bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:55.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.661+0000 7f21bbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:21:55.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.661+0000 7f21bbfff640 10 monclient: dump: 2024-04-22T08:21:55.677 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:55.677 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:55.677 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:55.677 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:55.677 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:55.677 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:55.677 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:55.677 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:55.677 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:55.677 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:55.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.661+0000 7f21bbfff640 1 -- 172.21.15.40:0/54925935 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f21c8009a80 con 0x7f21d413c390 2024-04-22T08:21:55.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.661+0000 7f21bbfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:55.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.661+0000 7f21bbfff640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:21:55.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.661+0000 7f21bbfff640 1 -- 172.21.15.40:0/54925935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21d413c390 msgr2=0x7f21d4138670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:55.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.661+0000 7f21bbfff640 1 --2- 172.21.15.40:0/54925935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21d413c390 0x7f21d4138670 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f21c800d700 tx=0x7f21c800dbc0 comp rx=0 tx=0).stop 2024-04-22T08:21:55.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.662+0000 7f21bbfff640 1 -- 172.21.15.40:0/54925935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21d413c390 msgr2=0x7f21d4138670 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:21:55.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.662+0000 7f21bbfff640 1 --2- 172.21.15.40:0/54925935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21d413c390 0x7f21d4138670 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:55.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.668+0000 7f21d9ce0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:21:55.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.668+0000 7f21d9ce0640 1 --2- 172.21.15.40:0/54925935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21d4138bb0 0x7f21d4138fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:55.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.668+0000 7f21d9ce0640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:55.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.668+0000 7f21d9ce0640 1 -- 172.21.15.40:0/54925935 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d40a3980 con 0x7f21d4138bb0 2024-04-22T08:21:55.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.668+0000 7f21d2ffd640 1 --2- 172.21.15.40:0/54925935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21d4138bb0 0x7f21d4138fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:55.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.668+0000 7f21d2ffd640 10 monclient: get_auth_request con 0x7f21d4138bb0 auth_method 0 2024-04-22T08:21:55.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.668+0000 7f21d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:55.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.668+0000 7f21d2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:55.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.668+0000 7f21d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:55.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.668+0000 7f21d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:55.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.668+0000 7f21d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:55.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.668+0000 7f21d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:55.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.668+0000 7f21d2ffd640 10 monclient(hunting): handle_auth_done global_id 65152 payload 824 2024-04-22T08:21:55.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.668+0000 7f21d2ffd640 1 -- 172.21.15.40:0/54925935 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f21d413bb40 con 0x7f21d4138bb0 2024-04-22T08:21:55.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.671+0000 7f21d2ffd640 1 -- 172.21.15.40:0/54925935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21d4138bb0 msgr2=0x7f21d4138fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:55.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.671+0000 7f21d2ffd640 1 --2- 172.21.15.40:0/54925935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21d4138bb0 0x7f21d4138fc0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f21c8019000 tx=0x7f21c80058d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:55.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.671+0000 7f21bbfff640 1 -- 172.21.15.40:0/54925935 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c801fb30 con 0x7f21d4138bb0 2024-04-22T08:21:55.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.671+0000 7f21bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:55.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.671+0000 7f21bbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:21:55.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.671+0000 7f21bbfff640 10 monclient: dump: 2024-04-22T08:21:55.680 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:55.680 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:55.680 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:55.680 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:55.680 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:55.680 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:55.680 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:55.680 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:55.680 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:55.681 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:55.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.671+0000 7f21bbfff640 1 -- 172.21.15.40:0/54925935 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4801 (secure 0 0 0) 0x7f21c8023070 con 0x7f21d4138bb0 2024-04-22T08:21:55.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.671+0000 7f21bbfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:21:55.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.671+0000 7f21bbfff640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:21:55.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.671+0000 7f21bbfff640 1 -- 172.21.15.40:0/54925935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21d4138bb0 msgr2=0x7f21d4138fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:55.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.671+0000 7f21bbfff640 1 --2- 172.21.15.40:0/54925935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21d4138bb0 0x7f21d4138fc0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f21c8019000 tx=0x7f21c80058d0 comp rx=0 tx=0).stop 2024-04-22T08:21:55.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.671+0000 7f21bbfff640 1 -- 172.21.15.40:0/54925935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21d4138bb0 msgr2=0x7f21d4138fc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:21:55.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.671+0000 7f21bbfff640 1 --2- 172.21.15.40:0/54925935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21d4138bb0 0x7f21d4138fc0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:55.681 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:21:55.682 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:21:55.682 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:21:55.682 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:21:55.682 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 92, 2024-04-22T08:21:55.682 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:21:55.682 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:21:55.682 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:21:55.682 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:21:55.682 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:55.682 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 0, 2024-04-22T08:21:55.682 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:21:55.683 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:21:55.683 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:21:55.683 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:21:55.683 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:21:55.683 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:21:55.683 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:21:55.683 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:21:55.683 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:21:55.683 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:21:55.683 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:21:55.683 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:21:55.683 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:21:55.684 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:55.684 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:21:55.684 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:21:55.684 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:21:55.684 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:21:55.684 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:21:55.684 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:21:55.684 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:21:55.684 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:21:55.684 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:21:55.684 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:21:55.684 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:21:55.684 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:21:55.685 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:55.685 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:55.685 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:21:55.685 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:21:55.685 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:21:55.685 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:21:55.685 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:21:55.685 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:21:55.685 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:21:55.685 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:21:55.685 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:21:55.685 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:21:55.685 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:21:55.685 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:21:55.686 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:21:55.686 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:21:55.686 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:21:55.686 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:21:55.686 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:21:55.686 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:21:55.686 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:21:55.686 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:21:55.686 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:21:55.686 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:21:55.686 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:21:55.686 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:21:55.687 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:21:55.687 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:21:55.687 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:21:55.687 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:55.687 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:21:55.687 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:55.687 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:21:55.687 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:55.687 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:21:55.687 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:21:55.687 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:21:55.687 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:21:55.688 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:55.688 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:21:55.688 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:21:55.688 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:55.688 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:55.688 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:55.688 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:21:55.688 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:21:55.688 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:55.688 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:55.688 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:55.688 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:55.688 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:21:55.689 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:21:55.689 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:21:55.689 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:21:55.689 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:21:55.689 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:55.689 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:55.689 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:21:55.689 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:21:55.689 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:21:55.689 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:21:55.689 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:55.689 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:21:55.689 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:21:55.689 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:55.690 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:55.690 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:55.690 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:21:55.690 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:21:55.690 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:55.690 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:55.690 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:55.690 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:55.690 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:21:55.690 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:21:55.690 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:21:55.690 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:21:55.691 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:21:55.691 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:55.691 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:55.691 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:21:55.691 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:21:55.691 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:21:55.691 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:21:55.691 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:55.691 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:21:55.691 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:21:55.691 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:55.691 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:55.691 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:55.692 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:21:55.692 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:21:55.692 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:55.692 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:55.692 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:55.692 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:55.692 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:21:55.692 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:21:55.692 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:21:55.692 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:21:55.692 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:21:55.692 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:55.692 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:55.692 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:55.693 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:21:55.693 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:21:55.693 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:55.693 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:21:55.693 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:21:55.693 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:21:55.693 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:55.693 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:55.693 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:21:55.693 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:55.693 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:21:55.693 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:21:55.694 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 5 2024-04-22T08:21:55.694 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:55.694 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:55.694 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:21:55.694 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:55.694 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:21:55.694 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:21:55.694 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 7 2024-04-22T08:21:55.694 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:55.694 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:55.694 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": [ 2024-04-22T08:21:55.694 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:55.694 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:21:55.695 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:21:55.695 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:21:55.695 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:55.695 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:55.695 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:55.695 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:21:55.695 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:21:55.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.671+0000 7f21d9ce0640 1 -- 172.21.15.40:0/54925935 >> v2:172.21.15.189:6800/30168 conn(0x7f21c00377d0 msgr2=0x7f21c0039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:55.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.671+0000 7f21d9ce0640 1 --2- 172.21.15.40:0/54925935 >> v2:172.21.15.189:6800/30168 conn(0x7f21c00377d0 0x7f21c0039c90 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f21d413dd50 tx=0x7f21cc003d10 comp rx=0 tx=0).stop 2024-04-22T08:21:55.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.671+0000 7f21d9ce0640 10 monclient: shutdown 2024-04-22T08:21:55.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.671+0000 7f21d9ce0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:55.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.671+0000 7f21d9ce0640 1 -- 172.21.15.40:0/54925935 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f21d4070eb0 msgr2=0x7f21d41cae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:55.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.671+0000 7f21d9ce0640 1 --2- 172.21.15.40:0/54925935 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f21d4070eb0 0x7f21d41cae80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f21c40098c0 tx=0x7f21c40046c0 comp rx=0 tx=0).stop 2024-04-22T08:21:55.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.671+0000 7f21d3fff640 1 -- 172.21.15.40:0/54925935 reap_dead start 2024-04-22T08:21:55.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.672+0000 7f21d3fff640 1 -- 172.21.15.40:0/54925935 reap_dead start 2024-04-22T08:21:55.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.672+0000 7f21d9ce0640 1 -- 172.21.15.40:0/54925935 shutdown_connections 2024-04-22T08:21:55.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.672+0000 7f21d9ce0640 1 -- 172.21.15.40:0/54925935 >> 172.21.15.40:0/54925935 conn(0x7f21d40a1780 msgr2=0x7f21d409f2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:55.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.672+0000 7f21d9ce0640 1 -- 172.21.15.40:0/54925935 shutdown_connections 2024-04-22T08:21:55.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.672+0000 7f21d9ce0640 1 -- 172.21.15.40:0/54925935 wait complete. 2024-04-22T08:21:55.697 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:21:55.697 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-22T08:21:55.698 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-22T08:21:55.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.797+0000 7fb842eb6640 10 monclient: get_monmap_and_config 2024-04-22T08:21:55.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.797+0000 7fb842eb6640 10 monclient: build_initial_monmap 2024-04-22T08:21:55.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.797+0000 7fb842eb6640 10 monclient: monmap: 2024-04-22T08:21:55.801 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:55.801 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:55.801 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:21:55.798483+0000 2024-04-22T08:21:55.801 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:21:55.798483+0000 2024-04-22T08:21:55.801 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:55.801 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:55.801 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:21:55.801 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:21:55.802 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:21:55.802 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:21:55.802 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:21:55.802 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:55.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.798+0000 7fb842eb6640 1 Processor -- start 2024-04-22T08:21:55.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.798+0000 7fb842eb6640 1 -- start start 2024-04-22T08:21:55.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.798+0000 7fb842eb6640 10 monclient: init 2024-04-22T08:21:55.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.798+0000 7fb842eb6640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:55.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.798+0000 7fb842eb6640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-04-22T08:21:55.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.798+0000 7fb842eb6640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb83c09ee10 0x7fb83c0a32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:55.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.798+0000 7fb842eb6640 10 monclient(hunting): picked mon.noname-b con 0x7fb83c09ee10 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:55.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.798+0000 7fb842eb6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb83c0a1d30 addr v1:172.21.15.189:6789/0 2024-04-22T08:21:55.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.798+0000 7fb842eb6640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb83c0a2150 0x7fb83c1364a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:55.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.799+0000 7fb842eb6640 10 monclient(hunting): picked mon.noname-a con 0x7fb83c0a2150 addr v2:172.21.15.40:3300/0 2024-04-22T08:21:55.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.799+0000 7fb842eb6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb83c0a2580 con 0x7fb83c0a1d30 2024-04-22T08:21:55.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.799+0000 7fb842eb6640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:55.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.799+0000 7fb842eb6640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb83c09f1f0 con 0x7fb83c0a2150 2024-04-22T08:21:55.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.799+0000 7fb842eb6640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:55.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.799+0000 7fb842eb6640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb83c0a39c0 con 0x7fb83c09ee10 2024-04-22T08:21:55.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.799+0000 7fb842eb6640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:55.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.799+0000 7fb842eb6640 10 monclient(hunting): authenticate will time out at 1389.792836s 2024-04-22T08:21:55.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.799+0000 7fb840c52640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb83c09ee10 0x7fb83c0a32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:55.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.799+0000 7fb840c52640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb83c09ee10 0x7fb83c0a32b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:41906/0 (socket says 172.21.15.40:41906) 2024-04-22T08:21:55.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.799+0000 7fb840c52640 1 -- 172.21.15.40:0/2091711311 learned_addr learned my addr 172.21.15.40:0/2091711311 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:21:55.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.799+0000 7fb840c52640 10 monclient(hunting): get_auth_request con 0x7fb83c09ee10 auth_method 0 2024-04-22T08:21:55.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.799+0000 7fb840c52640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:55.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.799+0000 7fb840c52640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:55.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.799+0000 7fb840c52640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:55.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.799+0000 7fb841453640 1 --2- 172.21.15.40:0/2091711311 >> v2:172.21.15.40:3300/0 conn(0x7fb83c0a2150 0x7fb83c1364a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:55.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.799+0000 7fb841453640 10 monclient(hunting): get_auth_request con 0x7fb83c0a2150 auth_method 0 2024-04-22T08:21:55.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.799+0000 7fb841453640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:55.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.799+0000 7fb841453640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:55.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.799+0000 7fb841453640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:55.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.799+0000 7fb840c52640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:55.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.799+0000 7fb840c52640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:55.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.799+0000 7fb840c52640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:55.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.799+0000 7fb841453640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:55.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.799+0000 7fb841453640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:55.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.799+0000 7fb841453640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:55.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.799+0000 7fb840c52640 10 monclient(hunting): handle_auth_done global_id 104312 payload 274 2024-04-22T08:21:55.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.799+0000 7fb840c52640 1 -- 172.21.15.40:0/2091711311 >> v2:172.21.15.40:3300/0 conn(0x7fb83c0a2150 msgr2=0x7fb83c1364a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:21:55.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.799+0000 7fb840c52640 1 --2- 172.21.15.40:0/2091711311 >> v2:172.21.15.40:3300/0 conn(0x7fb83c0a2150 0x7fb83c1364a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:55.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.799+0000 7fb840c52640 1 -- 172.21.15.40:0/2091711311 >> v1:172.21.15.189:6789/0 conn(0x7fb83c0a1d30 legacy=0x7fb83c135d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:21:55.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.799+0000 7fb840c52640 10 monclient: _finish_hunting 0 2024-04-22T08:21:55.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.799+0000 7fb840c52640 1 monclient: found mon.noname-b 2024-04-22T08:21:55.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.799+0000 7fb840c52640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:55.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.799+0000 7fb840c52640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:21:55.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.799+0000 7fb840c52640 1 -- 172.21.15.40:0/2091711311 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb83c09fd30 con 0x7fb83c09ee10 2024-04-22T08:21:55.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.802+0000 7fb840c52640 1 -- 172.21.15.40:0/2091711311 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb83c09ee10 msgr2=0x7fb83c0a32b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:55.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.802+0000 7fb840c52640 1 --2- 172.21.15.40:0/2091711311 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb83c09ee10 0x7fb83c0a32b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb838009870 tx=0x7fb83802b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca1dd398afe99364 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:55.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.802+0000 7fb8337fe640 1 -- 172.21.15.40:0/2091711311 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb838031950 con 0x7fb83c09ee10 2024-04-22T08:21:55.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.802+0000 7fb8337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:55.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.802+0000 7fb8337fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:21:55.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.802+0000 7fb8337fe640 10 monclient: dump: 2024-04-22T08:21:55.811 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:55.811 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:55.811 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:55.811 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:55.811 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:55.811 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:55.811 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:55.811 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:55.811 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:55.812 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:55.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.802+0000 7fb8337fe640 10 monclient: _finish_auth 0 2024-04-22T08:21:55.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.802+0000 7fb8337fe640 10 monclient: _check_auth_tickets 2024-04-22T08:21:55.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.802+0000 7fb8337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:55.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.802+0000 7fb8337fe640 1 -- 172.21.15.40:0/2091711311 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb838031ab0 con 0x7fb83c09ee10 2024-04-22T08:21:55.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.802+0000 7fb8337fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:55.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.802+0000 7fb8337fe640 1 -- 172.21.15.40:0/2091711311 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb838031e40 con 0x7fb83c09ee10 2024-04-22T08:21:55.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.802+0000 7fb842eb6640 5 monclient: authenticate success, global_id 104312 2024-04-22T08:21:55.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.802+0000 7fb842eb6640 10 monclient: get_monmap_and_config success 2024-04-22T08:21:55.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.802+0000 7fb842eb6640 4 set_mon_vals no callback set 2024-04-22T08:21:55.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.802+0000 7fb8337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:55.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.802+0000 7fb8337fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:21:55.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.802+0000 7fb8337fe640 10 monclient: dump: 2024-04-22T08:21:55.813 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:55.813 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:55.813 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:55.813 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:55.813 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:55.813 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:55.813 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:55.813 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:55.813 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:55.813 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:55.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.802+0000 7fb842eb6640 10 monclient: shutdown 2024-04-22T08:21:55.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.802+0000 7fb842eb6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:55.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.802+0000 7fb842eb6640 1 -- 172.21.15.40:0/2091711311 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb83c09ee10 msgr2=0x7fb83c0a32b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:55.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.802+0000 7fb842eb6640 1 --2- 172.21.15.40:0/2091711311 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb83c09ee10 0x7fb83c0a32b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb838009870 tx=0x7fb83802b7e0 comp rx=0 tx=0).stop 2024-04-22T08:21:55.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.802+0000 7fb842eb6640 1 -- 172.21.15.40:0/2091711311 shutdown_connections 2024-04-22T08:21:55.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.802+0000 7fb842eb6640 1 --2- 172.21.15.40:0/2091711311 >> v2:172.21.15.40:3300/0 conn(0x7fb83c0a2150 0x7fb83c1364a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:55.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.802+0000 7fb842eb6640 1 --2- 172.21.15.40:0/2091711311 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb83c09ee10 0x7fb83c0a32b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:55.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.802+0000 7fb842eb6640 1 -- 172.21.15.40:0/2091711311 >> 172.21.15.40:0/2091711311 conn(0x7fb83c0a4ef0 msgr2=0x7fb83c0a5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:55.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.803+0000 7fb842eb6640 1 -- 172.21.15.40:0/2091711311 shutdown_connections 2024-04-22T08:21:55.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.803+0000 7fb842eb6640 1 -- 172.21.15.40:0/2091711311 wait complete. 2024-04-22T08:21:55.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.803+0000 7fb842eb6640 10 monclient: build_initial_monmap 2024-04-22T08:21:55.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.803+0000 7fb842eb6640 10 monclient: monmap: 2024-04-22T08:21:55.814 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:55.814 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:21:55.814 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:21:55.814 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:21:55.814 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:55.814 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:55.814 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:21:55.814 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:21:55.815 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:21:55.815 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:55.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.803+0000 7fb842eb6640 1 Processor -- start 2024-04-22T08:21:55.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.803+0000 7fb842eb6640 1 -- start start 2024-04-22T08:21:55.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.803+0000 7fb842eb6640 10 monclient: init 2024-04-22T08:21:55.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb842eb6640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:55.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb842eb6640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-22T08:21:55.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb842eb6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb83c09ee10 0x7fb83c1cadf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:55.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb842eb6640 10 monclient(hunting): picked mon.noname-a con 0x7fb83c09ee10 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:21:55.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb842eb6640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb83c0a1d30 0x7fb83c1cb330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:55.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb842eb6640 10 monclient(hunting): picked mon.noname-c con 0x7fb83c0a1d30 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:55.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb842eb6640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb83c0a2150 0x7fb83c1220f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:55.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb842eb6640 10 monclient(hunting): picked mon.noname-b con 0x7fb83c0a2150 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:21:55.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb842eb6640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:55.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb842eb6640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb83c125e00 con 0x7fb83c09ee10 2024-04-22T08:21:55.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb842eb6640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:55.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb842eb6640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb83c126050 con 0x7fb83c0a2150 2024-04-22T08:21:55.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb842eb6640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:55.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb842eb6640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb83c1248c0 con 0x7fb83c0a1d30 2024-04-22T08:21:55.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb842eb6640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:55.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb842eb6640 10 monclient(hunting): authenticate will time out at 1389.798363s 2024-04-22T08:21:55.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb840c52640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb83c09ee10 0x7fb83c1cadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:55.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb840c52640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb83c09ee10 0x7fb83c1cadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56914/0 (socket says 172.21.15.40:56914) 2024-04-22T08:21:55.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb840c52640 1 -- 172.21.15.40:0/3909138326 learned_addr learned my addr 172.21.15.40:0/3909138326 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:21:55.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb840c52640 10 monclient(hunting): get_auth_request con 0x7fb83c09ee10 auth_method 0 2024-04-22T08:21:55.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb840c52640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:55.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb840c52640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:55.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb840c52640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:55.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb833fff640 1 --2- 172.21.15.40:0/3909138326 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb83c0a1d30 0x7fb83c1cb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:55.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb833fff640 10 monclient(hunting): get_auth_request con 0x7fb83c0a1d30 auth_method 0 2024-04-22T08:21:55.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb833fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:55.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb833fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:55.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb833fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:55.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb840c52640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:55.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb840c52640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:55.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb840c52640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:55.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb833fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:55.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb833fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:55.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb833fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:55.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb841453640 1 --2- 172.21.15.40:0/3909138326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb83c0a2150 0x7fb83c1220f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:55.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb841453640 10 monclient(hunting): get_auth_request con 0x7fb83c0a2150 auth_method 0 2024-04-22T08:21:55.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb841453640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:55.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb841453640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:55.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb841453640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:55.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb840c52640 10 monclient(hunting): handle_auth_done global_id 114150 payload 824 2024-04-22T08:21:55.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb840c52640 1 -- 172.21.15.40:0/3909138326 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb83c0a1d30 msgr2=0x7fb83c1cb330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:55.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb840c52640 1 --2- 172.21.15.40:0/3909138326 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb83c0a1d30 0x7fb83c1cb330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:55.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb840c52640 1 -- 172.21.15.40:0/3909138326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb83c0a2150 msgr2=0x7fb83c1220f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:55.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb840c52640 1 --2- 172.21.15.40:0/3909138326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb83c0a2150 0x7fb83c1220f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:55.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb840c52640 10 monclient: _finish_hunting 0 2024-04-22T08:21:55.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb840c52640 1 monclient: found mon.noname-a 2024-04-22T08:21:55.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb840c52640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:55.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb840c52640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:21:55.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.804+0000 7fb840c52640 1 -- 172.21.15.40:0/3909138326 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8380094d0 con 0x7fb83c09ee10 2024-04-22T08:21:55.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.807+0000 7fb840c52640 1 -- 172.21.15.40:0/3909138326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb83c09ee10 msgr2=0x7fb83c1cadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:55.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.807+0000 7fb840c52640 1 --2- 172.21.15.40:0/3909138326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb83c09ee10 0x7fb83c1cadf0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb8380090b0 tx=0x7fb8380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:55.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.807+0000 7fb8317fa640 1 -- 172.21.15.40:0/3909138326 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb838033b90 con 0x7fb83c09ee10 2024-04-22T08:21:55.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.807+0000 7fb8317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:55.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.807+0000 7fb8317fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:21:55.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.807+0000 7fb8317fa640 10 monclient: dump: 2024-04-22T08:21:55.819 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:55.819 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:55.819 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:55.819 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:55.819 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:55.819 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:55.819 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:55.819 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:55.819 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:55.819 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:55.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.807+0000 7fb8317fa640 10 monclient: _finish_auth 0 2024-04-22T08:21:55.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.807+0000 7fb8317fa640 10 monclient: _check_auth_tickets 2024-04-22T08:21:55.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.807+0000 7fb8317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:55.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.807+0000 7fb842eb6640 5 monclient: authenticate success, global_id 114150 2024-04-22T08:21:55.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.807+0000 7fb8317fa640 1 -- 172.21.15.40:0/3909138326 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb838033d10 con 0x7fb83c09ee10 2024-04-22T08:21:55.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.807+0000 7fb842eb6640 10 monclient: _renew_subs 2024-04-22T08:21:55.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.807+0000 7fb842eb6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:21:55.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.807+0000 7fb842eb6640 1 -- 172.21.15.40:0/3909138326 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb83c0a2730 con 0x7fb83c09ee10 2024-04-22T08:21:55.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.807+0000 7fb8317fa640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:55.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.807+0000 7fb8317fa640 1 -- 172.21.15.40:0/3909138326 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb83803c640 con 0x7fb83c09ee10 2024-04-22T08:21:55.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.807+0000 7fb8317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:55.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.807+0000 7fb8317fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:21:55.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.807+0000 7fb8317fa640 10 monclient: dump: 2024-04-22T08:21:55.820 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:55.821 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:55.821 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:55.821 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:55.821 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:55.821 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:55.821 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:55.821 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:55.821 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:55.821 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:55.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.807+0000 7fb842eb6640 10 monclient: _renew_subs 2024-04-22T08:21:55.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.807+0000 7fb842eb6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:21:55.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.807+0000 7fb842eb6640 1 -- 172.21.15.40:0/3909138326 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb83c123cf0 con 0x7fb83c09ee10 2024-04-22T08:21:55.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.807+0000 7fb832ffd640 4 set_mon_vals no callback set 2024-04-22T08:21:55.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.808+0000 7fb8317fa640 1 -- 172.21.15.40:0/3909138326 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fb83803a040 con 0x7fb83c09ee10 2024-04-22T08:21:55.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.808+0000 7fb842eb6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:55.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.808+0000 7fb842eb6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:55.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.808+0000 7fb842eb6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:21:55.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.808+0000 7fb8317fa640 1 --2- 172.21.15.40:0/3909138326 >> v2:172.21.15.189:6800/30168 conn(0x7fb8280377d0 0x7fb828039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:55.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.808+0000 7fb842eb6640 1 -- 172.21.15.40:0/3909138326 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb83c04e9b0 con 0x7fb83c09ee10 2024-04-22T08:21:55.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.808+0000 7fb8317fa640 1 -- 172.21.15.40:0/3909138326 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(308..308 src has 239..308) v4 ==== 3898+0+0 (secure 0 0 0) 0x7fb83806ba40 con 0x7fb83c09ee10 2024-04-22T08:21:55.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.808+0000 7fb833fff640 1 --2- 172.21.15.40:0/3909138326 >> v2:172.21.15.189:6800/30168 conn(0x7fb8280377d0 0x7fb828039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:55.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.808+0000 7fb833fff640 10 monclient: get_auth_request con 0x7fb8280377d0 auth_method 0 2024-04-22T08:21:55.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.811+0000 7fb833fff640 1 --2- 172.21.15.40:0/3909138326 >> v2:172.21.15.189:6800/30168 conn(0x7fb8280377d0 0x7fb828039c90 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fb83c0a7120 tx=0x7fb82c007e30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:55.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.813+0000 7fb8317fa640 1 -- 172.21.15.40:0/3909138326 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7fb83809b7f0 con 0x7fb83c09ee10 2024-04-22T08:21:55.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.813+0000 7fb8317fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:55.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:55.813+0000 7fb8317fa640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:21:56.019 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:21:56.019 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[39469]: starting. 2024-04-22T08:21:56.019 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[39469]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-39432 2024-04-22T08:21:56.019 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[39469]: created object 0... 2024-04-22T08:21:56.019 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[39469]: created object 25... 2024-04-22T08:21:56.020 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[39469]: created object 49... 2024-04-22T08:21:56.020 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[39469]: finishing. 2024-04-22T08:21:56.020 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[39469]: shutting down. 2024-04-22T08:21:56.058 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:21:56.058 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[39471]: starting. 2024-04-22T08:21:56.058 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[39471]: listing objects. 2024-04-22T08:21:56.058 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[39471]: listed object 0... 2024-04-22T08:21:56.058 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[39471]: listed object 25... 2024-04-22T08:21:56.058 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[39471]: saw 50 objects 2024-04-22T08:21:56.058 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[39471]: shutting down. 2024-04-22T08:21:56.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.085+0000 7fb842eb6640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-22T08:21:56.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.085+0000 7fb842eb6640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-22T08:21:56.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.085+0000 7fb842eb6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:21:56.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.085+0000 7fb842eb6640 1 -- 172.21.15.40:0/3909138326 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fb83c06a1e0 con 0x7fb83c09ee10 2024-04-22T08:21:56.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.086+0000 7fb840c52640 1 -- 172.21.15.40:0/3909138326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb83c09ee10 msgr2=0x7fb83c1cadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-22T08:21:56.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.086+0000 7fb840c52640 1 -- 172.21.15.40:0/3909138326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb83c09ee10 msgr2=0x7fb83c1cadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:21:56.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.086+0000 7fb840c52640 1 --2- 172.21.15.40:0/3909138326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb83c09ee10 0x7fb83c1cadf0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb8380090b0 tx=0x7fb8380046c0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:21:56.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.086+0000 7fb840c52640 1 --2- 172.21.15.40:0/3909138326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb83c09ee10 0x7fb83c1cadf0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb8380090b0 tx=0x7fb8380046c0 comp rx=0 tx=0).stop 2024-04-22T08:21:56.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.086+0000 7fb8317fa640 10 monclient: ms_handle_reset current mon [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:21:56.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.086+0000 7fb8317fa640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:56.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.086+0000 7fb8317fa640 1 -- 172.21.15.40:0/3909138326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb83c09ee10 msgr2=0x7fb83c1cadf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:21:56.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.086+0000 7fb8317fa640 1 --2- 172.21.15.40:0/3909138326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb83c09ee10 0x7fb83c1cadf0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:56.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.086+0000 7fb8317fa640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-22T08:21:56.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.086+0000 7fb8317fa640 1 --2- 172.21.15.40:0/3909138326 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb83c0a1d30 0x7fb82803fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:56.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.086+0000 7fb8317fa640 10 monclient(hunting): picked mon.c con 0x7fb83c0a1d30 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:56.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.086+0000 7fb8317fa640 1 --2- 172.21.15.40:0/3909138326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb828040530 0x7fb8280429c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:56.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.086+0000 7fb8317fa640 10 monclient(hunting): picked mon.a con 0x7fb828040530 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:21:56.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.086+0000 7fb8317fa640 1 --2- 172.21.15.40:0/3909138326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb83c0a2150 0x7fb8280454e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:56.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.086+0000 7fb8317fa640 10 monclient(hunting): picked mon.b con 0x7fb83c0a2150 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:21:56.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.086+0000 7fb8317fa640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:56.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.086+0000 7fb8317fa640 1 -- 172.21.15.40:0/3909138326 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb828045ef0 con 0x7fb828040530 2024-04-22T08:21:56.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.086+0000 7fb8317fa640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:56.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.086+0000 7fb8317fa640 1 -- 172.21.15.40:0/3909138326 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb828046030 con 0x7fb83c0a2150 2024-04-22T08:21:56.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.086+0000 7fb8317fa640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:56.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.086+0000 7fb8317fa640 1 -- 172.21.15.40:0/3909138326 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8280461f0 con 0x7fb83c0a1d30 2024-04-22T08:21:56.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.086+0000 7fb8317fa640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:56.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.086+0000 7fb841453640 1 --2- 172.21.15.40:0/3909138326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb828040530 0x7fb8280429c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:56.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.086+0000 7fb840c52640 1 --2- 172.21.15.40:0/3909138326 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb83c0a1d30 0x7fb82803fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:56.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.086+0000 7fb841453640 10 monclient(hunting): get_auth_request con 0x7fb828040530 auth_method 0 2024-04-22T08:21:56.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.086+0000 7fb841453640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:56.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.086+0000 7fb841453640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:56.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.086+0000 7fb841453640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:21:56.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.086+0000 7fb840c52640 10 monclient(hunting): get_auth_request con 0x7fb83c0a1d30 auth_method 0 2024-04-22T08:21:56.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.086+0000 7fb840c52640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:56.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.086+0000 7fb840c52640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:56.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.086+0000 7fb840c52640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:21:56.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.086+0000 7fb841453640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:56.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.086+0000 7fb841453640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:56.090 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.087+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.087+0000 7fb841453640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-22T08:21:56.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.087+0000 7fb840c52640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:56.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.087+0000 7fb840c52640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:56.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.087+0000 7fb840c52640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-22T08:21:56.091 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.087+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=18,config=2,health=10,kv=7,logm=63} crc {auth=3801261624,config=1618278320,health=2675324969,kv=190037718,logm=3462895787}) 2024-04-22T08:21:56.091 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.087+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=19,config=2,health=12,kv=9,logm=58} crc {auth=1282921447,config=1618278320,health=3150752593,kv=1350343662,logm=720498903}) 2024-04-22T08:21:56.091 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.087+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.091 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.087+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=18,config=2,health=10,kv=7,logm=63} crc {auth=3801261624,config=1618278320,health=2675324969,kv=190037718,logm=3462895787}) 2024-04-22T08:21:56.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.087+0000 7fb840c52640 1 --2- 172.21.15.40:0/3909138326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb83c0a2150 0x7fb8280454e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:56.092 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.087+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=18,config=2,health=12,kv=9,logm=59} crc {auth=3615970489,config=1618278320,health=3150752593,kv=1350343662,logm=1651105937}) 2024-04-22T08:21:56.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.087+0000 7fb840c52640 10 monclient(hunting): get_auth_request con 0x7fb83c0a2150 auth_method 0 2024-04-22T08:21:56.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.087+0000 7fb840c52640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:56.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.087+0000 7fb840c52640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:56.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.087+0000 7fb840c52640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:21:56.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.087+0000 7fb841453640 10 monclient(hunting): handle_auth_done global_id 114150 payload 293 2024-04-22T08:21:56.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.087+0000 7fb841453640 1 -- 172.21.15.40:0/3909138326 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb83c0a1d30 msgr2=0x7fb82803fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:56.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.087+0000 7fb841453640 1 --2- 172.21.15.40:0/3909138326 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb83c0a1d30 0x7fb82803fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:56.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.087+0000 7fb841453640 1 -- 172.21.15.40:0/3909138326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb83c0a2150 msgr2=0x7fb8280454e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:56.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.087+0000 7fb841453640 1 --2- 172.21.15.40:0/3909138326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb83c0a2150 0x7fb8280454e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:56.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.087+0000 7fb841453640 10 monclient: _finish_hunting 0 2024-04-22T08:21:56.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.087+0000 7fb841453640 1 monclient: found mon.a 2024-04-22T08:21:56.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.087+0000 7fb841453640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:56.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.087+0000 7fb841453640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:21:56.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.087+0000 7fb841453640 1 -- 172.21.15.40:0/3909138326 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7fb8280463b0 con 0x7fb828040530 2024-04-22T08:21:56.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.087+0000 7fb841453640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-22T08:21:56.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.087+0000 7fb841453640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:21:56.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.087+0000 7fb841453640 1 -- 172.21.15.40:0/3909138326 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fb8340073b0 con 0x7fb828040530 2024-04-22T08:21:56.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.087+0000 7fb841453640 10 monclient: _finish_auth 0 2024-04-22T08:21:56.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.087+0000 7fb841453640 10 monclient: _check_auth_tickets 2024-04-22T08:21:56.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.087+0000 7fb841453640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:56.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.087+0000 7fb840c52640 1 --2- 172.21.15.40:0/3909138326 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb83c0a1d30 0x7fb82803fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:21:56.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.089+0000 7fb841453640 1 -- 172.21.15.40:0/3909138326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb828040530 msgr2=0x7fb8280429c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:56.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.089+0000 7fb841453640 1 --2- 172.21.15.40:0/3909138326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb828040530 0x7fb8280429c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb8340075a0 tx=0x7fb838008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:56.095 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.089+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.095 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.089+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=421461551}) 2024-04-22T08:21:56.095 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.089+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3703372216}) 2024-04-22T08:21:56.096 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.089+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.096 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.089+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=421461551}) 2024-04-22T08:21:56.096 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.089+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4076294841}) 2024-04-22T08:21:56.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.090+0000 7fb8317fa640 1 -- 172.21.15.40:0/3909138326 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb83400e030 con 0x7fb828040530 2024-04-22T08:21:56.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.090+0000 7fb8317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:56.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.090+0000 7fb8317fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:21:56.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.090+0000 7fb8317fa640 10 monclient: dump: 2024-04-22T08:21:56.096 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:56.097 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:56.097 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:56.097 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:56.097 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:56.097 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:56.097 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:56.097 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:56.097 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:56.097 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:56.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.090+0000 7fb8317fa640 1 -- 172.21.15.40:0/3909138326 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb834011680 con 0x7fb828040530 2024-04-22T08:21:56.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.090+0000 7fb8317fa640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:56.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.090+0000 7fb832ffd640 4 set_mon_vals no callback set 2024-04-22T08:21:56.098 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.090+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.098 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.090+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3207509956}) 2024-04-22T08:21:56.098 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.090+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=584756828}) 2024-04-22T08:21:56.098 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.090+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.098 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.090+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3207509956}) 2024-04-22T08:21:56.098 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.090+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1918606109}) 2024-04-22T08:21:56.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.091+0000 7fb8317fa640 1 -- 172.21.15.40:0/3909138326 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fb8340117f0 con 0x7fb828040530 2024-04-22T08:21:56.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.091+0000 7fb8317fa640 1 -- 172.21.15.40:0/3909138326 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mon_command_ack([{"prefix": "mon scrub"}]=-16 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fb83401e7f0 con 0x7fb828040530 2024-04-22T08:21:56.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.091+0000 7fb8317fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-22T08:21:56.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:56.091+0000 7fb8317fa640 10 monclient: _finish_command 2 = mon:16 2024-04-22T08:21:56.099 INFO:teuthology.orchestra.run.smithi040.stderr:Error EBUSY: 2024-04-22T08:21:56.103 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.091+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.104 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.091+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=160732342}) 2024-04-22T08:21:56.104 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.091+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2873591761}) 2024-04-22T08:21:56.104 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.091+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.104 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.091+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=160732342}) 2024-04-22T08:21:56.104 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.091+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2756141077}) 2024-04-22T08:21:56.104 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.092+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.104 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.092+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1294641762}) 2024-04-22T08:21:56.104 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.092+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2753042105}) 2024-04-22T08:21:56.104 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.092+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.104 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.092+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1294641762}) 2024-04-22T08:21:56.104 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.092+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=518147667}) 2024-04-22T08:21:56.104 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.093+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.104 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.093+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=643812135}) 2024-04-22T08:21:56.105 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.093+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3899456232}) 2024-04-22T08:21:56.105 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.093+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.105 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.093+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=643812135}) 2024-04-22T08:21:56.105 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.093+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4250809597}) 2024-04-22T08:21:56.105 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.094+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.105 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.094+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2369650596}) 2024-04-22T08:21:56.105 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.094+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3985951295}) 2024-04-22T08:21:56.105 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.094+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.105 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.094+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2369650596}) 2024-04-22T08:21:56.105 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.094+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=75650992}) 2024-04-22T08:21:56.105 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.095+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.105 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.095+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1535888328}) 2024-04-22T08:21:56.105 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.095+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2101895092}) 2024-04-22T08:21:56.105 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.095+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.106 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.095+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1535888328}) 2024-04-22T08:21:56.106 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.095+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2459457810}) 2024-04-22T08:21:56.106 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.096+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.106 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.096+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2030113810}) 2024-04-22T08:21:56.106 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.096+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1868823203}) 2024-04-22T08:21:56.106 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.096+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.106 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.096+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2030113810}) 2024-04-22T08:21:56.106 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.096+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3015902462}) 2024-04-22T08:21:56.106 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.096+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.106 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.096+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2631864379}) 2024-04-22T08:21:56.106 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.096+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4009179416}) 2024-04-22T08:21:56.106 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.096+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.106 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.096+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2631864379}) 2024-04-22T08:21:56.106 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.096+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1742186400}) 2024-04-22T08:21:56.107 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.097+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.107 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.097+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1521653580}) 2024-04-22T08:21:56.107 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.097+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3727713797}) 2024-04-22T08:21:56.107 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.098+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.107 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.098+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1521653580}) 2024-04-22T08:21:56.107 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.098+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3731766561}) 2024-04-22T08:21:56.108 DEBUG:teuthology.orchestra.run:got remote process result: 16 2024-04-22T08:21:56.108 WARNING:tasks.mon_thrash:Ignoring exception while triggering scrub: Command failed on smithi040 with status 16: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub' 2024-04-22T08:21:56.109 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-22T08:21:56.115 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.098+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.115 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.098+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3055561937}) 2024-04-22T08:21:56.115 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.098+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1912503291}) 2024-04-22T08:21:56.115 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.098+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.115 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.098+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3055561937}) 2024-04-22T08:21:56.115 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.098+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1837256621}) 2024-04-22T08:21:56.115 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.099+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.115 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.099+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1098404218}) 2024-04-22T08:21:56.115 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.099+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4112719268}) 2024-04-22T08:21:56.115 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.099+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.116 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.099+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1098404218}) 2024-04-22T08:21:56.116 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.099+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=208459654}) 2024-04-22T08:21:56.116 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.100+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.116 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.100+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1603718763}) 2024-04-22T08:21:56.116 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.100+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=287865074}) 2024-04-22T08:21:56.116 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.100+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.116 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.100+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1603718763}) 2024-04-22T08:21:56.116 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.100+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1687539693}) 2024-04-22T08:21:56.116 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.101+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.116 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.101+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3986149124}) 2024-04-22T08:21:56.116 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.101+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3558159946}) 2024-04-22T08:21:56.116 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.101+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.116 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.101+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3986149124}) 2024-04-22T08:21:56.117 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.101+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2241788507}) 2024-04-22T08:21:56.117 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.102+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.117 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.102+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3824609648}) 2024-04-22T08:21:56.117 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.102+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2224386895}) 2024-04-22T08:21:56.117 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.102+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.117 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.102+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3824609648}) 2024-04-22T08:21:56.117 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.102+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1099561784}) 2024-04-22T08:21:56.117 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.102+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.117 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.102+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2998042635}) 2024-04-22T08:21:56.117 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.102+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1954099101}) 2024-04-22T08:21:56.117 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.102+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.117 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.102+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2998042635}) 2024-04-22T08:21:56.117 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.102+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2420791714}) 2024-04-22T08:21:56.118 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.103+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.118 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.103+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=219496915}) 2024-04-22T08:21:56.118 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.103+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1973958994}) 2024-04-22T08:21:56.118 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.103+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.118 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.103+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=219496915}) 2024-04-22T08:21:56.118 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.103+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1014447924}) 2024-04-22T08:21:56.118 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.104+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.118 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.104+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3828709026}) 2024-04-22T08:21:56.118 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.104+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=85299366}) 2024-04-22T08:21:56.118 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.104+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.118 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.104+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3828709026}) 2024-04-22T08:21:56.118 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.104+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3417918036}) 2024-04-22T08:21:56.118 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.105+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.118 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.105+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=561963803}) 2024-04-22T08:21:56.119 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.105+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1798951976}) 2024-04-22T08:21:56.119 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.105+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.119 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.105+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=561963803}) 2024-04-22T08:21:56.119 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.105+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3639720772}) 2024-04-22T08:21:56.119 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.106+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.119 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.106+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3150367823}) 2024-04-22T08:21:56.119 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.106+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1928433787}) 2024-04-22T08:21:56.119 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.106+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.119 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.106+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3150367823}) 2024-04-22T08:21:56.119 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.106+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3678490124}) 2024-04-22T08:21:56.119 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.107+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.119 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.107+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3852448717}) 2024-04-22T08:21:56.119 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.107+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=822666092}) 2024-04-22T08:21:56.120 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.107+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.120 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.107+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3852448717}) 2024-04-22T08:21:56.120 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.107+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3402434824}) 2024-04-22T08:21:56.120 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.108+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.120 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.108+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3960853113}) 2024-04-22T08:21:56.120 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.108+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2591049068}) 2024-04-22T08:21:56.120 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.108+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.120 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.108+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3960853113}) 2024-04-22T08:21:56.120 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.108+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=267283845}) 2024-04-22T08:21:56.120 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.109+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.120 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.109+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=17,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=7,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=43} crc {logm=868683603,mdsmap=1531151339,mgr=878880162,mgr_command_descs=470045235,mgr_metadata=4222303120,mgrstat=129510032,mon_config_key=506128298,monmap=3469086636,osd_metadata=4151711904,osd_pg_creating=1699746778,osdmap=2728688744}) 2024-04-22T08:21:56.120 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.109+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=17,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=6,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=45} crc {logm=3381877695,mdsmap=1531151339,mgr=878880162,mgr_command_descs=470045235,mgr_metadata=4222303121,mgrstat=2869569564,mon_config_key=1013444030,monmap=3469086636,osd_metadata=4151711904,osd_pg_creating=1699746778,osdmap=479260073}) 2024-04-22T08:21:56.120 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.109+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.120 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.109+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=17,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=7,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=43} crc {logm=868683603,mdsmap=1531151339,mgr=878880162,mgr_command_descs=470045235,mgr_metadata=4222303120,mgrstat=129510032,mon_config_key=506128298,monmap=3469086636,osd_metadata=4151711904,osd_pg_creating=1699746778,osdmap=2728688744}) 2024-04-22T08:21:56.121 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.109+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=16,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=7,monmap=3,osd_metadata=5,osd_pg_creating=1,osdmap=45} crc {logm=2010721493,mdsmap=2147943554,mgr=878880162,mgr_command_descs=470045235,mgr_metadata=4222303120,mgrstat=129510032,mon_config_key=1718430553,monmap=3469086636,osd_metadata=999924489,osd_pg_creating=1699746778,osdmap=370879264}) 2024-04-22T08:21:56.121 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.114+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.121 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.114+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=78} crc {osdmap=847849494}) 2024-04-22T08:21:56.121 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.114+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=77} crc {osdmap=2292911166}) 2024-04-22T08:21:56.121 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.114+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:21:56.121 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.114+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=78} crc {osdmap=847849494}) 2024-04-22T08:21:56.121 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:21:56.114+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=78} crc {osdmap=2745178124}) 2024-04-22T08:21:56.913 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:21:56.913 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[39470]: starting. 2024-04-22T08:21:56.913 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[39470]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-39432 2024-04-22T08:21:56.913 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[39470]: shutting down. 2024-04-22T08:21:56.914 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:21:56.914 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:21:56.914 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:21:56.917 DEBUG:teuthology.orchestra.run.smithi189:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-22T08:21:57.110 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-22T08:21:57.110 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:21:57.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.225+0000 7f2a0ca96640 10 monclient: get_monmap_and_config 2024-04-22T08:21:57.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.225+0000 7f2a0ca96640 10 monclient: build_initial_monmap 2024-04-22T08:21:57.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.225+0000 7f2a0ca96640 10 monclient: monmap: 2024-04-22T08:21:57.229 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:57.229 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:57.229 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:21:57.226653+0000 2024-04-22T08:21:57.229 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:21:57.226653+0000 2024-04-22T08:21:57.229 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:57.230 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:57.230 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:21:57.230 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:21:57.230 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:21:57.230 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:21:57.230 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:21:57.230 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:57.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.226+0000 7f2a0ca96640 1 Processor -- start 2024-04-22T08:21:57.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.226+0000 7f2a0ca96640 1 -- start start 2024-04-22T08:21:57.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.226+0000 7f2a0ca96640 10 monclient: init 2024-04-22T08:21:57.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a0ca96640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:57.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a0ca96640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-04-22T08:21:57.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a0ca96640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2a0812fc40 0x7f2a08130020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:57.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a0ca96640 10 monclient(hunting): picked mon.noname-c con 0x7f2a0812fc40 addr v2:172.21.15.189:3300/0 2024-04-22T08:21:57.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a0ca96640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2a0809e210 0x7f2a08132990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:57.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a0ca96640 10 monclient(hunting): picked mon.noname-a con 0x7f2a0809e210 addr v2:172.21.15.40:3300/0 2024-04-22T08:21:57.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a0ca96640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a08132ed0 0x7f2a081332e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:57.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a0ca96640 10 monclient(hunting): picked mon.noname-b con 0x7f2a08132ed0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:57.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a0ca96640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:57.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a0ca96640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a08133820 con 0x7f2a0809e210 2024-04-22T08:21:57.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a0ca96640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:57.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a0ca96640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a0809ea50 con 0x7f2a0812fc40 2024-04-22T08:21:57.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a0ca96640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:57.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a0ca96640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0809cf90 con 0x7f2a08132ed0 2024-04-22T08:21:57.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a0ca96640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:57.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a0ca96640 10 monclient(hunting): authenticate will time out at 1391.220926s 2024-04-22T08:21:57.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a05d9b640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2a0809e210 0x7f2a08132990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:57.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a06d9d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a08132ed0 0x7f2a081332e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:57.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a05d9b640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2a0809e210 0x7f2a08132990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56932/0 (socket says 172.21.15.40:56932) 2024-04-22T08:21:57.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a06d9d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a08132ed0 0x7f2a081332e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:41930/0 (socket says 172.21.15.40:41930) 2024-04-22T08:21:57.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a05d9b640 1 -- 172.21.15.40:0/10908971 learned_addr learned my addr 172.21.15.40:0/10908971 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:21:57.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a05d9b640 10 monclient(hunting): get_auth_request con 0x7f2a0809e210 auth_method 0 2024-04-22T08:21:57.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a05d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:57.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a05d9b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:57.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a05d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:57.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a06d9d640 10 monclient(hunting): get_auth_request con 0x7f2a08132ed0 auth_method 0 2024-04-22T08:21:57.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a06d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:57.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a06d9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:57.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a06d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:57.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a05d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:57.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a05d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:57.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a05d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:57.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a06d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:57.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a06d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:57.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a06d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:57.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a0659c640 1 --2- 172.21.15.40:0/10908971 >> v2:172.21.15.189:3300/0 conn(0x7f2a0812fc40 0x7f2a08130020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:57.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a0659c640 10 monclient(hunting): get_auth_request con 0x7f2a0812fc40 auth_method 0 2024-04-22T08:21:57.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a0659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:57.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a0659c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:57.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a0659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:57.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a05d9b640 10 monclient(hunting): handle_auth_done global_id 114165 payload 274 2024-04-22T08:21:57.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a05d9b640 1 -- 172.21.15.40:0/10908971 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a08132ed0 msgr2=0x7f2a081332e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:21:57.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a05d9b640 1 --2- 172.21.15.40:0/10908971 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a08132ed0 0x7f2a081332e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:57.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a05d9b640 1 -- 172.21.15.40:0/10908971 >> v2:172.21.15.189:3300/0 conn(0x7f2a0812fc40 msgr2=0x7f2a08130020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:21:57.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a05d9b640 1 --2- 172.21.15.40:0/10908971 >> v2:172.21.15.189:3300/0 conn(0x7f2a0812fc40 0x7f2a08130020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:57.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a05d9b640 10 monclient: _finish_hunting 0 2024-04-22T08:21:57.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a05d9b640 1 monclient: found mon.noname-a 2024-04-22T08:21:57.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a05d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:57.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a05d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:21:57.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a05d9b640 1 -- 172.21.15.40:0/10908971 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a0809d270 con 0x7f2a0809e210 2024-04-22T08:21:57.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.227+0000 7f2a06d9d640 1 --2- 172.21.15.40:0/10908971 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a08132ed0 0x7f2a081332e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:21:57.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.230+0000 7f2a05d9b640 1 -- 172.21.15.40:0/10908971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a0809e210 msgr2=0x7f2a08132990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:57.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.230+0000 7f2a05d9b640 1 --2- 172.21.15.40:0/10908971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a0809e210 0x7f2a08132990 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2a080a09c0 tx=0x7f29f002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c613c0415d31c92 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:57.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.230+0000 7f2a0559a640 1 -- 172.21.15.40:0/10908971 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f0031960 con 0x7f2a0809e210 2024-04-22T08:21:57.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.230+0000 7f2a0559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:57.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.230+0000 7f2a0559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:21:57.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.230+0000 7f2a0559a640 10 monclient: dump: 2024-04-22T08:21:57.241 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:57.241 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:57.241 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:57.241 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:57.241 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:57.241 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:57.241 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:57.241 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:57.242 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.230+0000 7f2a0559a640 10 monclient: _finish_auth 0 2024-04-22T08:21:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.230+0000 7f2a0559a640 10 monclient: _check_auth_tickets 2024-04-22T08:21:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.230+0000 7f2a0559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.230+0000 7f2a0559a640 1 -- 172.21.15.40:0/10908971 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29f0031ac0 con 0x7f2a0809e210 2024-04-22T08:21:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.230+0000 7f2a0ca96640 5 monclient: authenticate success, global_id 114165 2024-04-22T08:21:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.230+0000 7f2a0ca96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:21:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.230+0000 7f2a0559a640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.230+0000 7f2a0559a640 1 -- 172.21.15.40:0/10908971 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f0031e50 con 0x7f2a0809e210 2024-04-22T08:21:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.230+0000 7f2a0559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.230+0000 7f2a0559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:21:57.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.230+0000 7f2a0559a640 10 monclient: dump: 2024-04-22T08:21:57.243 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:57.243 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:57.243 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:57.243 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:57.243 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:57.243 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:57.243 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:57.243 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:57.243 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:57.243 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:57.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.230+0000 7f2a0ca96640 10 monclient: get_monmap_and_config success 2024-04-22T08:21:57.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.230+0000 7f2a0ca96640 4 set_mon_vals no callback set 2024-04-22T08:21:57.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.231+0000 7f2a0ca96640 10 monclient: shutdown 2024-04-22T08:21:57.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.231+0000 7f2a0ca96640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:57.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.231+0000 7f2a0ca96640 1 -- 172.21.15.40:0/10908971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a0809e210 msgr2=0x7f2a08132990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:57.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.231+0000 7f2a0ca96640 1 --2- 172.21.15.40:0/10908971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a0809e210 0x7f2a08132990 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2a080a09c0 tx=0x7f29f002b7f0 comp rx=0 tx=0).stop 2024-04-22T08:21:57.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.231+0000 7f2a0ca96640 1 -- 172.21.15.40:0/10908971 shutdown_connections 2024-04-22T08:21:57.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.231+0000 7f2a0ca96640 1 --2- 172.21.15.40:0/10908971 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a08132ed0 0x7f2a081332e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:57.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.231+0000 7f2a0ca96640 1 --2- 172.21.15.40:0/10908971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a0809e210 0x7f2a08132990 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:57.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.231+0000 7f2a0ca96640 1 --2- 172.21.15.40:0/10908971 >> v2:172.21.15.189:3300/0 conn(0x7f2a0812fc40 0x7f2a08130020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:57.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.231+0000 7f2a0ca96640 1 -- 172.21.15.40:0/10908971 >> 172.21.15.40:0/10908971 conn(0x7f2a080a0b50 msgr2=0x7f2a080a0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:57.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.231+0000 7f2a0ca96640 1 -- 172.21.15.40:0/10908971 shutdown_connections 2024-04-22T08:21:57.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.231+0000 7f2a0ca96640 1 -- 172.21.15.40:0/10908971 wait complete. 2024-04-22T08:21:57.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.232+0000 7f2a0ca96640 10 monclient: build_initial_monmap 2024-04-22T08:21:57.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.232+0000 7f2a0ca96640 10 monclient: monmap: 2024-04-22T08:21:57.244 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:57.244 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:21:57.245 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:21:57.245 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:21:57.245 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:57.245 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:57.245 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:21:57.245 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:21:57.245 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:21:57.245 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:57.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.232+0000 7f2a0ca96640 1 Processor -- start 2024-04-22T08:21:57.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.232+0000 7f2a0ca96640 1 -- start start 2024-04-22T08:21:57.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.232+0000 7f2a0ca96640 10 monclient: init 2024-04-22T08:21:57.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a0ca96640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:57.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a0ca96640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-22T08:21:57.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a0ca96640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a0809e210 0x7f2a081c6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:57.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a0ca96640 10 monclient(hunting): picked mon.noname-c con 0x7f2a0809e210 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:57.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a0ca96640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a0812fc40 0x7f2a081c6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:57.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a0ca96640 10 monclient(hunting): picked mon.noname-b con 0x7f2a0812fc40 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:21:57.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a0ca96640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a08132ed0 0x7f2a0806fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:57.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a0ca96640 10 monclient(hunting): picked mon.noname-a con 0x7f2a08132ed0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:21:57.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a0ca96640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:57.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a0ca96640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0812c870 con 0x7f2a08132ed0 2024-04-22T08:21:57.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a0ca96640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:57.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a0ca96640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0811ad30 con 0x7f2a0812fc40 2024-04-22T08:21:57.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a0ca96640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:57.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a0ca96640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0811cf30 con 0x7f2a0809e210 2024-04-22T08:21:57.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a0ca96640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:57.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a0ca96640 10 monclient(hunting): authenticate will time out at 1391.227020s 2024-04-22T08:21:57.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a0659c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a0809e210 0x7f2a081c6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:57.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a0659c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a0809e210 0x7f2a081c6a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:41932/0 (socket says 172.21.15.40:41932) 2024-04-22T08:21:57.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a0659c640 1 -- 172.21.15.40:0/3799423984 learned_addr learned my addr 172.21.15.40:0/3799423984 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:21:57.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a0659c640 10 monclient(hunting): get_auth_request con 0x7f2a0809e210 auth_method 0 2024-04-22T08:21:57.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a0659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:57.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a0659c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:57.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a0659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:57.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a06d9d640 1 --2- 172.21.15.40:0/3799423984 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a08132ed0 0x7f2a0806fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:57.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a06d9d640 10 monclient(hunting): get_auth_request con 0x7f2a08132ed0 auth_method 0 2024-04-22T08:21:57.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a06d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:57.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a06d9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:57.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a06d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:57.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a0659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:57.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a0659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:57.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a0659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:57.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a06d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:57.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a06d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:57.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a06d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:57.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a05d9b640 1 --2- 172.21.15.40:0/3799423984 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a0812fc40 0x7f2a081c6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:57.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a05d9b640 10 monclient(hunting): get_auth_request con 0x7f2a0812fc40 auth_method 0 2024-04-22T08:21:57.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a05d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:57.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a05d9b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:57.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a05d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:57.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a0659c640 10 monclient(hunting): handle_auth_done global_id 104327 payload 824 2024-04-22T08:21:57.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a0659c640 1 -- 172.21.15.40:0/3799423984 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a0812fc40 msgr2=0x7f2a081c6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:57.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a0659c640 1 --2- 172.21.15.40:0/3799423984 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a0812fc40 0x7f2a081c6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:57.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a0659c640 1 -- 172.21.15.40:0/3799423984 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a08132ed0 msgr2=0x7f2a0806fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:57.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a0659c640 1 --2- 172.21.15.40:0/3799423984 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a08132ed0 0x7f2a0806fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:57.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a0659c640 10 monclient: _finish_hunting 0 2024-04-22T08:21:57.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a0659c640 1 monclient: found mon.noname-c 2024-04-22T08:21:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a0659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a0659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:21:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a0659c640 1 -- 172.21.15.40:0/3799423984 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29f800bc80 con 0x7f2a0809e210 2024-04-22T08:21:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.233+0000 7f2a06d9d640 1 --2- 172.21.15.40:0/3799423984 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a08132ed0 0x7f2a0806fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:21:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.235+0000 7f2a0659c640 1 -- 172.21.15.40:0/3799423984 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a0809e210 msgr2=0x7f2a081c6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.235+0000 7f2a0659c640 1 --2- 172.21.15.40:0/3799423984 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a0809e210 0x7f2a081c6a00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2a0000d800 tx=0x7f2a0000dc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.235+0000 7f29feffd640 1 -- 172.21.15.40:0/3799423984 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a0000b8d0 con 0x7f2a0809e210 2024-04-22T08:21:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.236+0000 7f29feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.236+0000 7f29feffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:21:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.236+0000 7f29feffd640 10 monclient: dump: 2024-04-22T08:21:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:57.250 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:57.250 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:57.250 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:57.250 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:57.250 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:57.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.236+0000 7f29feffd640 10 monclient: _finish_auth 0 2024-04-22T08:21:57.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.236+0000 7f29feffd640 10 monclient: _check_auth_tickets 2024-04-22T08:21:57.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.236+0000 7f29feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:57.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.236+0000 7f29feffd640 1 -- 172.21.15.40:0/3799423984 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a0000ba30 con 0x7f2a0809e210 2024-04-22T08:21:57.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.236+0000 7f29feffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:57.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.236+0000 7f29feffd640 1 -- 172.21.15.40:0/3799423984 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a0000bdc0 con 0x7f2a0809e210 2024-04-22T08:21:57.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.236+0000 7f2a0ca96640 5 monclient: authenticate success, global_id 104327 2024-04-22T08:21:57.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.236+0000 7f29feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:57.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.236+0000 7f29feffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:21:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.236+0000 7f29feffd640 10 monclient: dump: 2024-04-22T08:21:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:57.251 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.236+0000 7f2a04d99640 4 set_mon_vals no callback set 2024-04-22T08:21:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.236+0000 7f2a0ca96640 10 monclient: _renew_subs 2024-04-22T08:21:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.236+0000 7f2a0ca96640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:21:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.236+0000 7f2a0ca96640 1 -- 172.21.15.40:0/3799423984 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29f00094d0 con 0x7f2a0809e210 2024-04-22T08:21:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.236+0000 7f2a0ca96640 10 monclient: _renew_subs 2024-04-22T08:21:57.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.236+0000 7f2a0ca96640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:21:57.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.236+0000 7f2a0ca96640 1 -- 172.21.15.40:0/3799423984 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a0811cb10 con 0x7f2a0809e210 2024-04-22T08:21:57.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.236+0000 7f29feffd640 1 -- 172.21.15.40:0/3799423984 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f2a0001e460 con 0x7f2a0809e210 2024-04-22T08:21:57.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.237+0000 7f2a0ca96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:57.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.237+0000 7f2a0ca96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:57.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.237+0000 7f2a0ca96640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:21:57.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.237+0000 7f2a0ca96640 1 -- 172.21.15.40:0/3799423984 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a0804e9b0 con 0x7f2a0809e210 2024-04-22T08:21:57.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.237+0000 7f29feffd640 1 --2- 172.21.15.40:0/3799423984 >> v2:172.21.15.189:6800/30168 conn(0x7f29e80377d0 0x7f29e8039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:57.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.237+0000 7f29feffd640 1 -- 172.21.15.40:0/3799423984 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(311..311 src has 239..311) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f2a0004a950 con 0x7f2a0809e210 2024-04-22T08:21:57.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.237+0000 7f2a05d9b640 1 --2- 172.21.15.40:0/3799423984 >> v2:172.21.15.189:6800/30168 conn(0x7f29e80377d0 0x7f29e8039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:57.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.237+0000 7f2a05d9b640 10 monclient: get_auth_request con 0x7f29e80377d0 auth_method 0 2024-04-22T08:21:57.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.240+0000 7f2a05d9b640 1 --2- 172.21.15.40:0/3799423984 >> v2:172.21.15.189:6800/30168 conn(0x7f29e80377d0 0x7f29e8039c90 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f29f0035000 tx=0x7f29f0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:57.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.242+0000 7f29feffd640 1 -- 172.21.15.40:0/3799423984 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f2a0001e710 con 0x7f2a0809e210 2024-04-22T08:21:57.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.242+0000 7f29feffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:57.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.242+0000 7f29feffd640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:21:57.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.493+0000 7f2a0ca96640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:21:57.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.493+0000 7f2a0ca96640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:21:57.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.493+0000 7f2a0ca96640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:21:57.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.493+0000 7f2a0ca96640 1 -- 172.21.15.40:0/3799423984 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2a0812d5b0 con 0x7f2a0809e210 2024-04-22T08:21:57.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.493+0000 7f29feffd640 1 -- 172.21.15.40:0/3799423984 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f2a00011340 con 0x7f2a0809e210 2024-04-22T08:21:57.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.493+0000 7f29feffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:21:57.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.493+0000 7f29feffd640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:21:57.493 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":92,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:21:57.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.495+0000 7f2a0ca96640 1 -- 172.21.15.40:0/3799423984 >> v2:172.21.15.189:6800/30168 conn(0x7f29e80377d0 msgr2=0x7f29e8039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:57.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.495+0000 7f2a0ca96640 1 --2- 172.21.15.40:0/3799423984 >> v2:172.21.15.189:6800/30168 conn(0x7f29e80377d0 0x7f29e8039c90 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f29f0035000 tx=0x7f29f0008030 comp rx=0 tx=0).stop 2024-04-22T08:21:57.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.495+0000 7f2a0ca96640 10 monclient: shutdown 2024-04-22T08:21:57.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.495+0000 7f2a0ca96640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:57.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.495+0000 7f2a0ca96640 1 -- 172.21.15.40:0/3799423984 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a0809e210 msgr2=0x7f2a081c6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:57.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.495+0000 7f2a0ca96640 1 --2- 172.21.15.40:0/3799423984 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a0809e210 0x7f2a081c6a00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2a0000d800 tx=0x7f2a0000dc00 comp rx=0 tx=0).stop 2024-04-22T08:21:57.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.495+0000 7f2a0ca96640 1 -- 172.21.15.40:0/3799423984 shutdown_connections 2024-04-22T08:21:57.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.495+0000 7f2a0ca96640 1 --2- 172.21.15.40:0/3799423984 >> v2:172.21.15.189:6800/30168 conn(0x7f29e80377d0 0x7f29e8039c90 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:57.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.495+0000 7f2a0ca96640 1 --2- 172.21.15.40:0/3799423984 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a0812fc40 0x7f2a081c6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:57.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.495+0000 7f2a0ca96640 1 --2- 172.21.15.40:0/3799423984 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a08132ed0 0x7f2a0806fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:57.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.495+0000 7f2a0ca96640 1 --2- 172.21.15.40:0/3799423984 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a0809e210 0x7f2a081c6a00 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:57.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.495+0000 7f2a0ca96640 1 -- 172.21.15.40:0/3799423984 >> 172.21.15.40:0/3799423984 conn(0x7f2a080a0b50 msgr2=0x7f2a0812f0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:57.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.495+0000 7f2a0ca96640 1 -- 172.21.15.40:0/3799423984 shutdown_connections 2024-04-22T08:21:57.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.495+0000 7f2a0ca96640 1 -- 172.21.15.40:0/3799423984 wait complete. 2024-04-22T08:21:57.505 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-22T08:21:57.505 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-22T08:21:57.505 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-22T08:21:57.506 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-22T08:21:57.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.606+0000 7f2d0264a640 10 monclient: get_monmap_and_config 2024-04-22T08:21:57.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.606+0000 7f2d0264a640 10 monclient: build_initial_monmap 2024-04-22T08:21:57.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.606+0000 7f2d0264a640 10 monclient: monmap: 2024-04-22T08:21:57.606 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:57.606 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:57.606 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:21:57.607237+0000 2024-04-22T08:21:57.607 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:21:57.607237+0000 2024-04-22T08:21:57.607 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:57.607 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:57.607 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:21:57.607 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:21:57.607 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:21:57.607 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:21:57.607 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:21:57.607 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:57.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.607+0000 7f2d0264a640 1 Processor -- start 2024-04-22T08:21:57.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.607+0000 7f2d0264a640 1 -- start start 2024-04-22T08:21:57.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.607+0000 7f2d0264a640 10 monclient: init 2024-04-22T08:21:57.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.607+0000 7f2d0264a640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:57.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.607+0000 7f2d0264a640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-04-22T08:21:57.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.607+0000 7f2d0264a640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2cfc12fad0 0x7f2cfc12feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:57.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.607+0000 7f2d0264a640 10 monclient(hunting): picked mon.noname-c con 0x7f2cfc12fad0 addr v2:172.21.15.189:3300/0 2024-04-22T08:21:57.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.607+0000 7f2d0264a640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2cfc132920 0x7f2cfc132d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:57.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.607+0000 7f2d0264a640 10 monclient(hunting): picked mon.noname-b con 0x7f2cfc132920 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:57.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.607+0000 7f2d0264a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2cfc133260 addr v1:172.21.15.189:6789/0 2024-04-22T08:21:57.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.607+0000 7f2d0264a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cfc09ae50 con 0x7f2cfc133260 2024-04-22T08:21:57.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.607+0000 7f2d0264a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:57.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.607+0000 7f2cfb7fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2cfc132920 0x7f2cfc132d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:57.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.607+0000 7f2d0264a640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cfc09efc0 con 0x7f2cfc12fad0 2024-04-22T08:21:57.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.607+0000 7f2d0264a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:57.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.607+0000 7f2d0264a640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc09d070 con 0x7f2cfc132920 2024-04-22T08:21:57.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.607+0000 7f2d0264a640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:57.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.607+0000 7f2d0264a640 10 monclient(hunting): authenticate will time out at 1391.601705s 2024-04-22T08:21:57.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.607+0000 7f2cfb7fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2cfc132920 0x7f2cfc132d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:41940/0 (socket says 172.21.15.40:41940) 2024-04-22T08:21:57.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.607+0000 7f2cfb7fe640 1 -- 172.21.15.40:0/1543456167 learned_addr learned my addr 172.21.15.40:0/1543456167 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:21:57.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.607+0000 7f2cfb7fe640 10 monclient(hunting): get_auth_request con 0x7f2cfc132920 auth_method 0 2024-04-22T08:21:57.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.607+0000 7f2cfb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:57.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.607+0000 7f2cfb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:57.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.607+0000 7f2cfb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:57.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.608+0000 7f2cfbfff640 1 --2- 172.21.15.40:0/1543456167 >> v2:172.21.15.189:3300/0 conn(0x7f2cfc12fad0 0x7f2cfc12feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:57.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.608+0000 7f2cfbfff640 10 monclient(hunting): get_auth_request con 0x7f2cfc12fad0 auth_method 0 2024-04-22T08:21:57.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.608+0000 7f2cfbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:57.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.608+0000 7f2cfbfff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:57.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.608+0000 7f2cfbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:57.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.608+0000 7f2cfb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:57.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.608+0000 7f2cfb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:57.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.608+0000 7f2cfb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:57.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.608+0000 7f2cfbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:57.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.608+0000 7f2cfbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:57.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.608+0000 7f2cfbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:57.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.608+0000 7f2cfb7fe640 10 monclient(hunting): handle_auth_done global_id 104330 payload 274 2024-04-22T08:21:57.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.608+0000 7f2cfb7fe640 1 -- 172.21.15.40:0/1543456167 >> v2:172.21.15.189:3300/0 conn(0x7f2cfc12fad0 msgr2=0x7f2cfc12feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:21:57.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.608+0000 7f2cfb7fe640 1 --2- 172.21.15.40:0/1543456167 >> v2:172.21.15.189:3300/0 conn(0x7f2cfc12fad0 0x7f2cfc12feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:57.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.608+0000 7f2cfb7fe640 1 -- 172.21.15.40:0/1543456167 >> v1:172.21.15.189:6789/0 conn(0x7f2cfc133260 legacy=0x7f2cfc139c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:21:57.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.608+0000 7f2cfb7fe640 10 monclient: _finish_hunting 0 2024-04-22T08:21:57.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.608+0000 7f2cfb7fe640 1 monclient: found mon.noname-b 2024-04-22T08:21:57.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.608+0000 7f2cfb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:57.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.608+0000 7f2cfb7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:21:57.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.608+0000 7f2cfb7fe640 1 -- 172.21.15.40:0/1543456167 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cfc09d350 con 0x7f2cfc132920 2024-04-22T08:21:57.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.611+0000 7f2cfb7fe640 1 -- 172.21.15.40:0/1543456167 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2cfc132920 msgr2=0x7f2cfc132d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:57.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.611+0000 7f2cfb7fe640 1 --2- 172.21.15.40:0/1543456167 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2cfc132920 0x7f2cfc132d20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2cf0009870 tx=0x7f2cf002b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed293253a2ad0c0a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:57.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.611+0000 7f2cfaffd640 1 -- 172.21.15.40:0/1543456167 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf0031950 con 0x7f2cfc132920 2024-04-22T08:21:57.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.611+0000 7f2cfaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:57.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.611+0000 7f2cfaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:21:57.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.611+0000 7f2cfaffd640 10 monclient: dump: 2024-04-22T08:21:57.619 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:57.620 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:57.620 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:57.620 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:57.620 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:57.620 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:57.620 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:57.620 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:57.620 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:57.620 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:57.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.611+0000 7f2cfaffd640 10 monclient: _finish_auth 0 2024-04-22T08:21:57.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.611+0000 7f2cfaffd640 10 monclient: _check_auth_tickets 2024-04-22T08:21:57.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.611+0000 7f2cfaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:57.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.611+0000 7f2d0264a640 5 monclient: authenticate success, global_id 104330 2024-04-22T08:21:57.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.611+0000 7f2cfaffd640 1 -- 172.21.15.40:0/1543456167 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cf0031ab0 con 0x7f2cfc132920 2024-04-22T08:21:57.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.611+0000 7f2d0264a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:21:57.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.611+0000 7f2cfaffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:57.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.611+0000 7f2d0264a640 10 monclient: get_monmap_and_config success 2024-04-22T08:21:57.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.611+0000 7f2d0264a640 4 set_mon_vals no callback set 2024-04-22T08:21:57.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.611+0000 7f2cfaffd640 1 -- 172.21.15.40:0/1543456167 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf0031e40 con 0x7f2cfc132920 2024-04-22T08:21:57.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.611+0000 7f2cfaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:57.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.611+0000 7f2cfaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:21:57.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.611+0000 7f2cfaffd640 10 monclient: dump: 2024-04-22T08:21:57.621 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:57.621 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:57.621 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:57.621 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:57.621 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:57.621 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:57.622 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:57.622 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:57.622 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:57.622 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:57.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.611+0000 7f2d0264a640 10 monclient: shutdown 2024-04-22T08:21:57.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.611+0000 7f2d0264a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:57.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.611+0000 7f2d0264a640 1 -- 172.21.15.40:0/1543456167 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2cfc132920 msgr2=0x7f2cfc132d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:57.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.611+0000 7f2d0264a640 1 --2- 172.21.15.40:0/1543456167 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2cfc132920 0x7f2cfc132d20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2cf0009870 tx=0x7f2cf002b7e0 comp rx=0 tx=0).stop 2024-04-22T08:21:57.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.611+0000 7f2d0264a640 1 -- 172.21.15.40:0/1543456167 shutdown_connections 2024-04-22T08:21:57.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.611+0000 7f2d0264a640 1 --2- 172.21.15.40:0/1543456167 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2cfc132920 0x7f2cfc132d20 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:57.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.611+0000 7f2d0264a640 1 --2- 172.21.15.40:0/1543456167 >> v2:172.21.15.189:3300/0 conn(0x7f2cfc12fad0 0x7f2cfc12feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:57.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.611+0000 7f2d0264a640 1 -- 172.21.15.40:0/1543456167 >> 172.21.15.40:0/1543456167 conn(0x7f2cfc0a0ae0 msgr2=0x7f2cfc0a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:57.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.611+0000 7f2d0264a640 1 -- 172.21.15.40:0/1543456167 shutdown_connections 2024-04-22T08:21:57.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.611+0000 7f2d0264a640 1 -- 172.21.15.40:0/1543456167 wait complete. 2024-04-22T08:21:57.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.612+0000 7f2d0264a640 10 monclient: build_initial_monmap 2024-04-22T08:21:57.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.612+0000 7f2d0264a640 10 monclient: monmap: 2024-04-22T08:21:57.623 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:57.623 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:21:57.623 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:21:57.623 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:21:57.623 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:57.623 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:57.623 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:21:57.623 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:21:57.623 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:21:57.623 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:57.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.612+0000 7f2d0264a640 1 Processor -- start 2024-04-22T08:21:57.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.612+0000 7f2d0264a640 1 -- start start 2024-04-22T08:21:57.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.612+0000 7f2d0264a640 10 monclient: init 2024-04-22T08:21:57.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2d0264a640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:57.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2d0264a640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-22T08:21:57.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2d0264a640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2cfc12fad0 0x7f2cfc1c6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:57.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2d0264a640 10 monclient(hunting): picked mon.noname-c con 0x7f2cfc12fad0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:57.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2d0264a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cfc132920 0x7f2cfc1c6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:57.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2d0264a640 10 monclient(hunting): picked mon.noname-a con 0x7f2cfc132920 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:21:57.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2d0264a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cfc133260 0x7f2cfc06fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:57.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2d0264a640 10 monclient(hunting): picked mon.noname-b con 0x7f2cfc133260 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:21:57.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2d0264a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:57.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2d0264a640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc11cc60 con 0x7f2cfc132920 2024-04-22T08:21:57.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2d0264a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:57.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2d0264a640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc11a890 con 0x7f2cfc133260 2024-04-22T08:21:57.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2d0264a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:57.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2d0264a640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc131fd0 con 0x7f2cfc12fad0 2024-04-22T08:21:57.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2d0264a640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:57.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2d0264a640 10 monclient(hunting): authenticate will time out at 1391.607157s 2024-04-22T08:21:57.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2cfbfff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2cfc12fad0 0x7f2cfc1c6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:57.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2cfb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cfc132920 0x7f2cfc1c6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:57.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2cfbfff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2cfc12fad0 0x7f2cfc1c6a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:41950/0 (socket says 172.21.15.40:41950) 2024-04-22T08:21:57.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2cfbfff640 1 -- 172.21.15.40:0/1822645328 learned_addr learned my addr 172.21.15.40:0/1822645328 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:21:57.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2cfbfff640 10 monclient(hunting): get_auth_request con 0x7f2cfc12fad0 auth_method 0 2024-04-22T08:21:57.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2cfbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:57.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2cfbfff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:57.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2cfbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:57.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2cfb7fe640 10 monclient(hunting): get_auth_request con 0x7f2cfc132920 auth_method 0 2024-04-22T08:21:57.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2cfb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:57.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2cfb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:57.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2cfb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:57.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2cfbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:57.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2cfbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:57.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2cfbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:57.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2cfb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:57.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2cfb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:57.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2cfb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:57.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2d00be7640 1 --2- 172.21.15.40:0/1822645328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cfc133260 0x7f2cfc06fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:57.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2d00be7640 10 monclient(hunting): get_auth_request con 0x7f2cfc133260 auth_method 0 2024-04-22T08:21:57.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2d00be7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:57.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2d00be7640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:57.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2d00be7640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:57.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2cfbfff640 10 monclient(hunting): handle_auth_done global_id 104333 payload 824 2024-04-22T08:21:57.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2cfbfff640 1 -- 172.21.15.40:0/1822645328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cfc133260 msgr2=0x7f2cfc06fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:57.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2cfbfff640 1 --2- 172.21.15.40:0/1822645328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cfc133260 0x7f2cfc06fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:57.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2cfbfff640 1 -- 172.21.15.40:0/1822645328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cfc132920 msgr2=0x7f2cfc1c6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:57.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2cfbfff640 1 --2- 172.21.15.40:0/1822645328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cfc132920 0x7f2cfc1c6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:57.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2cfbfff640 10 monclient: _finish_hunting 0 2024-04-22T08:21:57.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2cfbfff640 1 monclient: found mon.noname-c 2024-04-22T08:21:57.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2cfbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:57.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2cfbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:21:57.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.613+0000 7f2cfbfff640 1 -- 172.21.15.40:0/1822645328 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cf00094d0 con 0x7f2cfc12fad0 2024-04-22T08:21:57.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.616+0000 7f2cfbfff640 1 -- 172.21.15.40:0/1822645328 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2cfc12fad0 msgr2=0x7f2cfc1c6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:57.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.616+0000 7f2cfbfff640 1 --2- 172.21.15.40:0/1822645328 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2cfc12fad0 0x7f2cfc1c6a00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2cec00e9b0 tx=0x7f2cec00ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:57.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.616+0000 7f2cf8ff9640 1 -- 172.21.15.40:0/1822645328 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cec009670 con 0x7f2cfc12fad0 2024-04-22T08:21:57.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.616+0000 7f2cf8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:57.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.616+0000 7f2cf8ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:21:57.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.616+0000 7f2cf8ff9640 10 monclient: dump: 2024-04-22T08:21:57.628 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:57.628 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:57.628 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:57.628 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:57.628 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:57.628 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:57.628 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:57.628 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:57.628 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:57.628 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:57.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.616+0000 7f2cf8ff9640 10 monclient: _finish_auth 0 2024-04-22T08:21:57.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.616+0000 7f2cf8ff9640 10 monclient: _check_auth_tickets 2024-04-22T08:21:57.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.616+0000 7f2cf8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:57.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.616+0000 7f2cf8ff9640 1 -- 172.21.15.40:0/1822645328 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cec00cbb0 con 0x7f2cfc12fad0 2024-04-22T08:21:57.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.616+0000 7f2d0264a640 5 monclient: authenticate success, global_id 104333 2024-04-22T08:21:57.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.616+0000 7f2d0264a640 10 monclient: _renew_subs 2024-04-22T08:21:57.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.616+0000 7f2d0264a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:21:57.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.616+0000 7f2d0264a640 1 -- 172.21.15.40:0/1822645328 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cfc071870 con 0x7f2cfc12fad0 2024-04-22T08:21:57.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.616+0000 7f2cf8ff9640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:57.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.616+0000 7f2cf8ff9640 1 -- 172.21.15.40:0/1822645328 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cec013470 con 0x7f2cfc12fad0 2024-04-22T08:21:57.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.616+0000 7f2cf8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:57.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.616+0000 7f2cf8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:21:57.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.616+0000 7f2cf8ff9640 10 monclient: dump: 2024-04-22T08:21:57.629 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:57.629 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:57.629 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:57.629 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:57.629 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:57.629 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:57.629 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:57.630 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:57.630 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:57.630 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:57.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.616+0000 7f2cfa7fc640 4 set_mon_vals no callback set 2024-04-22T08:21:57.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.616+0000 7f2d0264a640 10 monclient: _renew_subs 2024-04-22T08:21:57.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.616+0000 7f2d0264a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:21:57.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.616+0000 7f2d0264a640 1 -- 172.21.15.40:0/1822645328 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cfc11c3c0 con 0x7f2cfc12fad0 2024-04-22T08:21:57.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.617+0000 7f2cf8ff9640 1 -- 172.21.15.40:0/1822645328 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f2cec0135d0 con 0x7f2cfc12fad0 2024-04-22T08:21:57.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.617+0000 7f2d0264a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:57.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.617+0000 7f2d0264a640 1 --2- 172.21.15.40:0/1822645328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cfc070360 0x7f2cfc070740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:57.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.617+0000 7f2d0264a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:57.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.617+0000 7f2d0264a640 1 -- 172.21.15.40:0/1822645328 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc0667a0 con 0x7f2cfc070360 2024-04-22T08:21:57.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.617+0000 7f2cf8ff9640 1 --2- 172.21.15.40:0/1822645328 >> v2:172.21.15.189:6800/30168 conn(0x7f2ce80377d0 0x7f2ce8039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:57.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.617+0000 7f2cf8ff9640 1 -- 172.21.15.40:0/1822645328 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(311..311 src has 239..311) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f2cec04a740 con 0x7f2cfc12fad0 2024-04-22T08:21:57.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.617+0000 7f2cfb7fe640 1 --2- 172.21.15.40:0/1822645328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cfc070360 0x7f2cfc070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:57.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.617+0000 7f2cfb7fe640 10 monclient: get_auth_request con 0x7f2cfc070360 auth_method 0 2024-04-22T08:21:57.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.617+0000 7f2cfb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:57.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.617+0000 7f2cfb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:57.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.617+0000 7f2cfb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:57.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.617+0000 7f2cfb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:57.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.617+0000 7f2cfb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:57.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.617+0000 7f2cfb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:57.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.617+0000 7f2d00be7640 1 --2- 172.21.15.40:0/1822645328 >> v2:172.21.15.189:6800/30168 conn(0x7f2ce80377d0 0x7f2ce8039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:57.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.617+0000 7f2d00be7640 10 monclient: get_auth_request con 0x7f2ce80377d0 auth_method 0 2024-04-22T08:21:57.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.617+0000 7f2cfb7fe640 10 monclient(hunting): handle_auth_done global_id 114174 payload 824 2024-04-22T08:21:57.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.617+0000 7f2cfb7fe640 1 -- 172.21.15.40:0/1822645328 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2cfc09c9b0 con 0x7f2cfc070360 2024-04-22T08:21:57.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.619+0000 7f2cfb7fe640 1 -- 172.21.15.40:0/1822645328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cfc070360 msgr2=0x7f2cfc070740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:57.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.619+0000 7f2cfb7fe640 1 --2- 172.21.15.40:0/1822645328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cfc070360 0x7f2cfc070740 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2cf0035000 tx=0x7f2cf00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:57.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.620+0000 7f2d00be7640 1 --2- 172.21.15.40:0/1822645328 >> v2:172.21.15.189:6800/30168 conn(0x7f2ce80377d0 0x7f2ce8039c90 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc11af70 tx=0x7f2cf4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:57.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.620+0000 7f2cf8ff9640 1 -- 172.21.15.40:0/1822645328 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf0033d60 con 0x7f2cfc070360 2024-04-22T08:21:57.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.620+0000 7f2cf8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:57.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.620+0000 7f2cf8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:21:57.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.620+0000 7f2cf8ff9640 10 monclient: dump: 2024-04-22T08:21:57.632 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:57.632 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:57.632 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:57.632 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:57.632 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:57.632 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:57.632 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:57.632 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:57.633 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:57.633 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:57.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.620+0000 7f2cf8ff9640 1 -- 172.21.15.40:0/1822645328 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f2cf0041070 con 0x7f2cfc070360 2024-04-22T08:21:57.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.620+0000 7f2cf8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:57.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.620+0000 7f2cf8ff9640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:21:57.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.620+0000 7f2cf8ff9640 1 -- 172.21.15.40:0/1822645328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cfc070360 msgr2=0x7f2cfc070740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:57.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.620+0000 7f2cf8ff9640 1 --2- 172.21.15.40:0/1822645328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cfc070360 0x7f2cfc070740 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2cf0035000 tx=0x7f2cf00046c0 comp rx=0 tx=0).stop 2024-04-22T08:21:57.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.620+0000 7f2cf8ff9640 1 -- 172.21.15.40:0/1822645328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cfc070360 msgr2=0x7f2cfc070740 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:21:57.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.620+0000 7f2cf8ff9640 1 --2- 172.21.15.40:0/1822645328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cfc070360 0x7f2cfc070740 secure :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2cf0035000 tx=0x7f2cf00046c0 comp rx=0 tx=0).stop 2024-04-22T08:21:57.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.626+0000 7f2d0264a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:21:57.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.626+0000 7f2d0264a640 1 --2- 172.21.15.40:0/1822645328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cfc070c80 0x7f2cfc06c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:57.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.626+0000 7f2d0264a640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:57.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.626+0000 7f2d0264a640 1 -- 172.21.15.40:0/1822645328 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc11c5b0 con 0x7f2cfc070c80 2024-04-22T08:21:57.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.626+0000 7f2cfb7fe640 1 --2- 172.21.15.40:0/1822645328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cfc070c80 0x7f2cfc06c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:57.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.626+0000 7f2cfb7fe640 10 monclient: get_auth_request con 0x7f2cfc070c80 auth_method 0 2024-04-22T08:21:57.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.626+0000 7f2cfb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:57.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.626+0000 7f2cfb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:57.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.626+0000 7f2cfb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:57.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.626+0000 7f2cfb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:57.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.626+0000 7f2cfb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:57.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.626+0000 7f2cfb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:57.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.627+0000 7f2cfb7fe640 10 monclient(hunting): handle_auth_done global_id 114177 payload 824 2024-04-22T08:21:57.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.627+0000 7f2cfb7fe640 1 -- 172.21.15.40:0/1822645328 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2cfc04e9b0 con 0x7f2cfc070c80 2024-04-22T08:21:57.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.629+0000 7f2cfb7fe640 1 -- 172.21.15.40:0/1822645328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cfc070c80 msgr2=0x7f2cfc06c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:57.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.629+0000 7f2cfb7fe640 1 --2- 172.21.15.40:0/1822645328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cfc070c80 0x7f2cfc06c720 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2cf002b820 tx=0x7f2cf0009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:57.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.629+0000 7f2cf8ff9640 1 -- 172.21.15.40:0/1822645328 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf0056070 con 0x7f2cfc070c80 2024-04-22T08:21:57.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.629+0000 7f2cf8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:57.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.629+0000 7f2cf8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:21:57.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.629+0000 7f2cf8ff9640 10 monclient: dump: 2024-04-22T08:21:57.636 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:57.636 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:57.637 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:57.637 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:57.637 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:57.637 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:57.637 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:57.637 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:57.637 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:57.637 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:57.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.629+0000 7f2cf8ff9640 1 -- 172.21.15.40:0/1822645328 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4463 (secure 0 0 0) 0x7f2cf0050840 con 0x7f2cfc070c80 2024-04-22T08:21:57.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.629+0000 7f2cf8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:21:57.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.629+0000 7f2cf8ff9640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:21:57.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.629+0000 7f2cf8ff9640 1 -- 172.21.15.40:0/1822645328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cfc070c80 msgr2=0x7f2cfc06c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:57.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.629+0000 7f2cf8ff9640 1 --2- 172.21.15.40:0/1822645328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cfc070c80 0x7f2cfc06c720 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2cf002b820 tx=0x7f2cf0009240 comp rx=0 tx=0).stop 2024-04-22T08:21:57.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.629+0000 7f2cf8ff9640 1 -- 172.21.15.40:0/1822645328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cfc070c80 msgr2=0x7f2cfc06c720 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:21:57.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.629+0000 7f2cf8ff9640 1 --2- 172.21.15.40:0/1822645328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cfc070c80 0x7f2cfc06c720 secure :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2cf002b820 tx=0x7f2cf0009240 comp rx=0 tx=0).stop 2024-04-22T08:21:57.638 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:21:57.638 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:21:57.638 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:21:57.638 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "leader", 2024-04-22T08:21:57.638 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 92, 2024-04-22T08:21:57.638 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:21:57.638 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:21:57.638 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:21:57.638 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:21:57.639 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:57.639 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 2, 2024-04-22T08:21:57.639 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:21:57.639 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:21:57.639 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:21:57.639 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:21:57.639 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:21:57.639 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:21:57.639 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:21:57.639 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:21:57.639 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:21:57.639 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:21:57.639 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:21:57.640 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:21:57.640 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:21:57.640 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:57.640 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:21:57.640 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:21:57.640 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:21:57.640 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:21:57.640 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:21:57.640 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:21:57.640 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:21:57.640 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:21:57.640 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:21:57.640 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:21:57.640 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:21:57.640 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:21:57.641 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:57.641 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:57.641 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:21:57.641 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:21:57.641 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:21:57.641 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:21:57.641 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:21:57.641 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:21:57.641 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:21:57.641 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:21:57.641 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:21:57.641 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:21:57.641 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:21:57.641 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:21:57.642 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:21:57.642 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:21:57.642 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:21:57.642 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:21:57.642 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:21:57.642 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:21:57.642 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:21:57.642 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:21:57.642 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:21:57.642 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:21:57.642 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:21:57.642 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:21:57.642 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:21:57.642 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:21:57.642 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:21:57.643 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:57.643 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:21:57.643 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:57.643 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:21:57.643 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:57.643 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:21:57.643 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:21:57.643 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:21:57.643 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:21:57.643 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:57.643 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:21:57.643 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:21:57.643 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:57.643 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:57.643 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:57.643 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:21:57.644 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:21:57.644 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:57.644 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:57.644 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:57.644 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:57.644 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:21:57.644 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:21:57.644 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:21:57.644 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:21:57.644 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:21:57.644 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:57.644 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:57.644 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:21:57.644 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:21:57.644 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:21:57.644 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:21:57.645 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:57.645 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:21:57.645 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:21:57.645 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:57.645 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:57.645 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:57.645 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:21:57.645 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:21:57.645 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:57.645 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:57.645 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:57.645 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:57.645 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:21:57.645 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:21:57.645 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:21:57.646 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:21:57.646 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:21:57.646 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:57.646 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:57.646 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:21:57.646 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:21:57.646 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:21:57.646 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:21:57.646 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:57.646 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:21:57.646 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:21:57.646 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:57.646 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:57.646 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:57.647 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:21:57.647 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:21:57.647 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:57.647 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:57.647 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:57.647 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:57.647 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:21:57.647 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:21:57.647 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:21:57.647 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:21:57.647 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:21:57.647 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:57.647 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:57.647 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:57.648 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:21:57.648 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:21:57.648 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:57.648 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:21:57.648 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:21:57.648 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:21:57.648 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:57.648 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:57.648 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:21:57.648 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:57.648 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:21:57.648 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:21:57.648 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:21:57.648 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:57.649 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:57.649 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:57.649 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:21:57.649 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:21:57.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.630+0000 7f2d0264a640 1 -- 172.21.15.40:0/1822645328 >> v2:172.21.15.189:6800/30168 conn(0x7f2ce80377d0 msgr2=0x7f2ce8039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:57.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.630+0000 7f2d0264a640 1 --2- 172.21.15.40:0/1822645328 >> v2:172.21.15.189:6800/30168 conn(0x7f2ce80377d0 0x7f2ce8039c90 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc11af70 tx=0x7f2cf4009040 comp rx=0 tx=0).stop 2024-04-22T08:21:57.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.630+0000 7f2d0264a640 10 monclient: shutdown 2024-04-22T08:21:57.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.630+0000 7f2d0264a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:57.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.630+0000 7f2d0264a640 1 -- 172.21.15.40:0/1822645328 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2cfc12fad0 msgr2=0x7f2cfc1c6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:57.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.630+0000 7f2d0264a640 1 --2- 172.21.15.40:0/1822645328 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2cfc12fad0 0x7f2cfc1c6a00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2cec00e9b0 tx=0x7f2cec00ee70 comp rx=0 tx=0).stop 2024-04-22T08:21:57.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.630+0000 7f2d00be7640 1 -- 172.21.15.40:0/1822645328 reap_dead start 2024-04-22T08:21:57.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.630+0000 7f2d00be7640 1 -- 172.21.15.40:0/1822645328 reap_dead start 2024-04-22T08:21:57.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.630+0000 7f2d0264a640 1 -- 172.21.15.40:0/1822645328 shutdown_connections 2024-04-22T08:21:57.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.630+0000 7f2d0264a640 1 -- 172.21.15.40:0/1822645328 >> 172.21.15.40:0/1822645328 conn(0x7f2cfc0a0ae0 msgr2=0x7f2cfc12f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:57.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.630+0000 7f2d0264a640 1 -- 172.21.15.40:0/1822645328 shutdown_connections 2024-04-22T08:21:57.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.630+0000 7f2d0264a640 1 -- 172.21.15.40:0/1822645328 wait complete. 2024-04-22T08:21:57.651 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:21:57.651 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-22T08:21:57.651 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-22T08:21:57.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.751+0000 7f1bb3f75640 10 monclient: get_monmap_and_config 2024-04-22T08:21:57.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.751+0000 7f1bb3f75640 10 monclient: build_initial_monmap 2024-04-22T08:21:57.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.751+0000 7f1bb3f75640 10 monclient: monmap: 2024-04-22T08:21:57.751 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:57.751 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:57.751 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:21:57.752171+0000 2024-04-22T08:21:57.751 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:21:57.752171+0000 2024-04-22T08:21:57.751 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:57.751 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:57.751 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:21:57.751 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:21:57.751 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:21:57.752 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:21:57.752 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:21:57.752 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:57.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.751+0000 7f1bb3f75640 1 Processor -- start 2024-04-22T08:21:57.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.752+0000 7f1bb3f75640 1 -- start start 2024-04-22T08:21:57.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.752+0000 7f1bb3f75640 10 monclient: init 2024-04-22T08:21:57.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.752+0000 7f1bb3f75640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:57.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.752+0000 7f1bb3f75640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-04-22T08:21:57.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.752+0000 7f1bb3f75640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1bac136070 0x7f1bac09aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:57.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.752+0000 7f1bb3f75640 10 monclient(hunting): picked mon.noname-c con 0x7f1bac136070 addr v2:172.21.15.189:3300/0 2024-04-22T08:21:57.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.752+0000 7f1bb3f75640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1bac136450 addr v1:172.21.15.189:6789/0 2024-04-22T08:21:57.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.752+0000 7f1bb3f75640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1bac1368b0 0x7f1bac0a1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:57.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.752+0000 7f1bb3f75640 10 monclient(hunting): picked mon.noname-b con 0x7f1bac1368b0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:57.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.752+0000 7f1bb3f75640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bac06bb50 con 0x7f1bac136450 2024-04-22T08:21:57.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.752+0000 7f1bb3f75640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:57.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.752+0000 7f1bb3f75640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bac09a390 con 0x7f1bac136070 2024-04-22T08:21:57.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.752+0000 7f1bb3f75640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:57.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.752+0000 7f1bb3f75640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bac099dd0 con 0x7f1bac1368b0 2024-04-22T08:21:57.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.752+0000 7f1bb3f75640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:57.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.752+0000 7f1bb3f75640 10 monclient(hunting): authenticate will time out at 1391.746416s 2024-04-22T08:21:57.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.752+0000 7f1bb2512640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1bac1368b0 0x7f1bac0a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:57.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.752+0000 7f1bb2512640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1bac1368b0 0x7f1bac0a1180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:41954/0 (socket says 172.21.15.40:41954) 2024-04-22T08:21:57.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.752+0000 7f1bb2512640 1 -- 172.21.15.40:0/260508223 learned_addr learned my addr 172.21.15.40:0/260508223 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:21:57.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.752+0000 7f1bb2512640 10 monclient(hunting): get_auth_request con 0x7f1bac1368b0 auth_method 0 2024-04-22T08:21:57.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.752+0000 7f1bb2512640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:57.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.752+0000 7f1bb2512640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:57.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.752+0000 7f1bb2512640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:57.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.752+0000 7f1bb2512640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:57.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.752+0000 7f1bb2512640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:57.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.752+0000 7f1bb2512640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:57.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.753+0000 7f1bb1d11640 1 --2- 172.21.15.40:0/260508223 >> v2:172.21.15.189:3300/0 conn(0x7f1bac136070 0x7f1bac09aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:57.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.753+0000 7f1bb1d11640 10 monclient(hunting): get_auth_request con 0x7f1bac136070 auth_method 0 2024-04-22T08:21:57.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.753+0000 7f1bb1d11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:57.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.753+0000 7f1bb1d11640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:57.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.753+0000 7f1bb1d11640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:57.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.753+0000 7f1bb2512640 10 monclient(hunting): handle_auth_done global_id 104336 payload 274 2024-04-22T08:21:57.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.753+0000 7f1bb2512640 1 -- 172.21.15.40:0/260508223 >> v2:172.21.15.189:3300/0 conn(0x7f1bac136070 msgr2=0x7f1bac09aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:21:57.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.753+0000 7f1bb2512640 1 --2- 172.21.15.40:0/260508223 >> v2:172.21.15.189:3300/0 conn(0x7f1bac136070 0x7f1bac09aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:57.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.753+0000 7f1bb2512640 1 -- 172.21.15.40:0/260508223 >> v1:172.21.15.189:6789/0 conn(0x7f1bac136450 legacy=0x7f1bac0a0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:21:57.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.753+0000 7f1bb2512640 10 monclient: _finish_hunting 0 2024-04-22T08:21:57.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.753+0000 7f1bb2512640 1 monclient: found mon.noname-b 2024-04-22T08:21:57.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.753+0000 7f1bb2512640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:57.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.753+0000 7f1bb2512640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:21:57.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.753+0000 7f1bb2512640 1 -- 172.21.15.40:0/260508223 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bac09a0b0 con 0x7f1bac1368b0 2024-04-22T08:21:57.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.756+0000 7f1bb2512640 1 -- 172.21.15.40:0/260508223 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1bac1368b0 msgr2=0x7f1bac0a1180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:57.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.756+0000 7f1bb2512640 1 --2- 172.21.15.40:0/260508223 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1bac1368b0 0x7f1bac0a1180 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1ba4009e50 tx=0x7f1ba402bdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55f60c634177de09 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:57.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.756+0000 7f1bb0d0f640 1 -- 172.21.15.40:0/260508223 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba402cb60 con 0x7f1bac1368b0 2024-04-22T08:21:57.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.756+0000 7f1bb0d0f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:57.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.756+0000 7f1bb0d0f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:21:57.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.756+0000 7f1bb0d0f640 10 monclient: dump: 2024-04-22T08:21:57.764 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:57.764 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:57.764 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:57.764 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:57.764 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:57.764 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:57.764 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:57.764 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:57.764 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:57.764 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:57.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.756+0000 7f1bb0d0f640 10 monclient: _finish_auth 0 2024-04-22T08:21:57.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.756+0000 7f1bb0d0f640 10 monclient: _check_auth_tickets 2024-04-22T08:21:57.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.756+0000 7f1bb0d0f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:57.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.756+0000 7f1bb3f75640 5 monclient: authenticate success, global_id 104336 2024-04-22T08:21:57.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.756+0000 7f1bb0d0f640 1 -- 172.21.15.40:0/260508223 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ba402ccc0 con 0x7f1bac1368b0 2024-04-22T08:21:57.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.756+0000 7f1bb3f75640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:21:57.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.756+0000 7f1bb0d0f640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:57.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.756+0000 7f1bb3f75640 10 monclient: get_monmap_and_config success 2024-04-22T08:21:57.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.756+0000 7f1bb3f75640 4 set_mon_vals no callback set 2024-04-22T08:21:57.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.756+0000 7f1bb0d0f640 1 -- 172.21.15.40:0/260508223 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba4034610 con 0x7f1bac1368b0 2024-04-22T08:21:57.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.756+0000 7f1bb0d0f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:57.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.756+0000 7f1bb0d0f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:21:57.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.756+0000 7f1bb0d0f640 10 monclient: dump: 2024-04-22T08:21:57.766 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:57.766 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:57.766 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:57.766 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:57.766 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:57.766 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:57.766 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:57.766 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:57.766 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:57.766 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:57.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.756+0000 7f1bb3f75640 10 monclient: shutdown 2024-04-22T08:21:57.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.756+0000 7f1bb3f75640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:57.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.756+0000 7f1bb3f75640 1 -- 172.21.15.40:0/260508223 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1bac1368b0 msgr2=0x7f1bac0a1180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:57.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.756+0000 7f1bb3f75640 1 --2- 172.21.15.40:0/260508223 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1bac1368b0 0x7f1bac0a1180 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1ba4009e50 tx=0x7f1ba402bdb0 comp rx=0 tx=0).stop 2024-04-22T08:21:57.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.757+0000 7f1bb3f75640 1 -- 172.21.15.40:0/260508223 shutdown_connections 2024-04-22T08:21:57.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.757+0000 7f1bb3f75640 1 --2- 172.21.15.40:0/260508223 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1bac1368b0 0x7f1bac0a1180 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:57.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.757+0000 7f1bb3f75640 1 --2- 172.21.15.40:0/260508223 >> v2:172.21.15.189:3300/0 conn(0x7f1bac136070 0x7f1bac09aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:57.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.757+0000 7f1bb3f75640 1 -- 172.21.15.40:0/260508223 >> 172.21.15.40:0/260508223 conn(0x7f1bac0a1b60 msgr2=0x7f1bac0a1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:57.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.757+0000 7f1bb3f75640 1 -- 172.21.15.40:0/260508223 shutdown_connections 2024-04-22T08:21:57.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.757+0000 7f1bb3f75640 1 -- 172.21.15.40:0/260508223 wait complete. 2024-04-22T08:21:57.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.757+0000 7f1bb3f75640 10 monclient: build_initial_monmap 2024-04-22T08:21:57.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.757+0000 7f1bb3f75640 10 monclient: monmap: 2024-04-22T08:21:57.767 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:57.767 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:21:57.767 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:21:57.767 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:21:57.767 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:57.768 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:57.768 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:21:57.768 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:21:57.768 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:21:57.768 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:57.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.757+0000 7f1bb3f75640 1 Processor -- start 2024-04-22T08:21:57.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb3f75640 1 -- start start 2024-04-22T08:21:57.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb3f75640 10 monclient: init 2024-04-22T08:21:57.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb3f75640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:57.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb3f75640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-22T08:21:57.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb3f75640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bac136070 0x7f1bac1cad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:57.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb3f75640 10 monclient(hunting): picked mon.noname-b con 0x7f1bac136070 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:21:57.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb3f75640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1bac136450 0x7f1bac1cb250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:57.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb3f75640 10 monclient(hunting): picked mon.noname-a con 0x7f1bac136450 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:21:57.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb3f75640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1bac122610 0x7f1bac122a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:57.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb3f75640 10 monclient(hunting): picked mon.noname-c con 0x7f1bac122610 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:57.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb3f75640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:57.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb3f75640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bac09ee60 con 0x7f1bac136450 2024-04-22T08:21:57.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb3f75640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:57.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb3f75640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bac124cf0 con 0x7f1bac136070 2024-04-22T08:21:57.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb3f75640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:57.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb3f75640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bac09b470 con 0x7f1bac122610 2024-04-22T08:21:57.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb3f75640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:57.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb3f75640 10 monclient(hunting): authenticate will time out at 1391.752201s 2024-04-22T08:21:57.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb2512640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1bac122610 0x7f1bac122a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:57.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb2512640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1bac122610 0x7f1bac122a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:41970/0 (socket says 172.21.15.40:41970) 2024-04-22T08:21:57.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb2512640 1 -- 172.21.15.40:0/1632458567 learned_addr learned my addr 172.21.15.40:0/1632458567 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:21:57.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb2512640 10 monclient(hunting): get_auth_request con 0x7f1bac122610 auth_method 0 2024-04-22T08:21:57.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb2512640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:57.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb2512640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:57.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb2512640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:57.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb1510640 1 --2- 172.21.15.40:0/1632458567 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1bac136450 0x7f1bac1cb250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:57.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb1510640 10 monclient(hunting): get_auth_request con 0x7f1bac136450 auth_method 0 2024-04-22T08:21:57.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb1510640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:57.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb1510640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:57.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb1510640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:57.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb2512640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:57.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb2512640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:57.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb2512640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:57.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb1d11640 1 --2- 172.21.15.40:0/1632458567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bac136070 0x7f1bac1cad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:57.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb1d11640 10 monclient(hunting): get_auth_request con 0x7f1bac136070 auth_method 0 2024-04-22T08:21:57.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb1d11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:57.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb1d11640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:57.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb1d11640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:57.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb1510640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:57.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb1510640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:57.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb1510640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:57.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb1d11640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:57.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb1d11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:57.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb1d11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:57.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb2512640 10 monclient(hunting): handle_auth_done global_id 104339 payload 824 2024-04-22T08:21:57.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb2512640 1 -- 172.21.15.40:0/1632458567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bac136070 msgr2=0x7f1bac1cad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:57.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb2512640 1 --2- 172.21.15.40:0/1632458567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bac136070 0x7f1bac1cad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:57.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb2512640 1 -- 172.21.15.40:0/1632458567 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1bac136450 msgr2=0x7f1bac1cb250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:57.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb2512640 1 --2- 172.21.15.40:0/1632458567 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1bac136450 0x7f1bac1cb250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:57.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb2512640 10 monclient: _finish_hunting 0 2024-04-22T08:21:57.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb2512640 1 monclient: found mon.noname-c 2024-04-22T08:21:57.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb2512640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:57.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb2512640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:21:57.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.758+0000 7f1bb2512640 1 -- 172.21.15.40:0/1632458567 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba4009b30 con 0x7f1bac122610 2024-04-22T08:21:57.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.761+0000 7f1bb2512640 1 -- 172.21.15.40:0/1632458567 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1bac122610 msgr2=0x7f1bac122a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:57.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.761+0000 7f1bb2512640 1 --2- 172.21.15.40:0/1632458567 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1bac122610 0x7f1bac122a90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1ba4036040 tx=0x7f1ba4004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:57.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.761+0000 7f1ba27fc640 1 -- 172.21.15.40:0/1632458567 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba403e450 con 0x7f1bac122610 2024-04-22T08:21:57.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.761+0000 7f1ba27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:57.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.761+0000 7f1ba27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:21:57.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.761+0000 7f1ba27fc640 10 monclient: dump: 2024-04-22T08:21:57.772 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:57.772 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:57.772 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:57.772 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:57.773 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:57.773 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:57.773 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:57.773 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:57.773 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:57.773 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:57.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.761+0000 7f1ba27fc640 10 monclient: _finish_auth 0 2024-04-22T08:21:57.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.761+0000 7f1ba27fc640 10 monclient: _check_auth_tickets 2024-04-22T08:21:57.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.761+0000 7f1ba27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:57.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.761+0000 7f1ba27fc640 1 -- 172.21.15.40:0/1632458567 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ba403e5d0 con 0x7f1bac122610 2024-04-22T08:21:57.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.761+0000 7f1bb3f75640 5 monclient: authenticate success, global_id 104339 2024-04-22T08:21:57.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.761+0000 7f1bb3f75640 10 monclient: _renew_subs 2024-04-22T08:21:57.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.761+0000 7f1bb3f75640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:21:57.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.761+0000 7f1bb3f75640 1 -- 172.21.15.40:0/1632458567 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bac126210 con 0x7f1bac122610 2024-04-22T08:21:57.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.761+0000 7f1ba27fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:57.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.761+0000 7f1ba27fc640 1 -- 172.21.15.40:0/1632458567 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba403e980 con 0x7f1bac122610 2024-04-22T08:21:57.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.761+0000 7f1ba27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:57.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.761+0000 7f1ba27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:21:57.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.761+0000 7f1ba27fc640 10 monclient: dump: 2024-04-22T08:21:57.774 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:57.774 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:57.774 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:57.774 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:57.774 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:57.774 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:57.774 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:57.774 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:57.774 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:57.774 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:57.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.761+0000 7f1ba3fff640 4 set_mon_vals no callback set 2024-04-22T08:21:57.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.761+0000 7f1bb3f75640 10 monclient: _renew_subs 2024-04-22T08:21:57.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.761+0000 7f1bb3f75640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:21:57.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.761+0000 7f1bb3f75640 1 -- 172.21.15.40:0/1632458567 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bac124120 con 0x7f1bac122610 2024-04-22T08:21:57.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.762+0000 7f1bb3f75640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:57.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.762+0000 7f1bb3f75640 1 --2- 172.21.15.40:0/1632458567 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1bac136890 0x7f1bac122fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:57.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.762+0000 7f1bb3f75640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:57.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.762+0000 7f1bb3f75640 1 -- 172.21.15.40:0/1632458567 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bac0667a0 con 0x7f1bac136890 2024-04-22T08:21:57.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.762+0000 7f1bb1d11640 1 --2- 172.21.15.40:0/1632458567 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1bac136890 0x7f1bac122fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:57.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.762+0000 7f1ba27fc640 1 -- 172.21.15.40:0/1632458567 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f1ba403eb20 con 0x7f1bac122610 2024-04-22T08:21:57.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.762+0000 7f1bb1d11640 10 monclient: get_auth_request con 0x7f1bac136890 auth_method 0 2024-04-22T08:21:57.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.762+0000 7f1bb1d11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:57.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.762+0000 7f1bb1d11640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:57.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.762+0000 7f1bb1d11640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:57.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.762+0000 7f1bb1d11640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:57.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.762+0000 7f1bb1d11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:57.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.762+0000 7f1bb1d11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:57.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.762+0000 7f1ba27fc640 1 --2- 172.21.15.40:0/1632458567 >> v2:172.21.15.189:6800/30168 conn(0x7f1b900377d0 0x7f1b90039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:57.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.762+0000 7f1bb1d11640 10 monclient(hunting): handle_auth_done global_id 104342 payload 824 2024-04-22T08:21:57.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.762+0000 7f1bb1d11640 1 -- 172.21.15.40:0/1632458567 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1bac09afd0 con 0x7f1bac136890 2024-04-22T08:21:57.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.762+0000 7f1ba27fc640 1 -- 172.21.15.40:0/1632458567 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(311..311 src has 239..311) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f1ba406cff0 con 0x7f1bac122610 2024-04-22T08:21:57.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.762+0000 7f1bb1510640 1 --2- 172.21.15.40:0/1632458567 >> v2:172.21.15.189:6800/30168 conn(0x7f1b900377d0 0x7f1b90039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:57.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.762+0000 7f1bb1510640 10 monclient: get_auth_request con 0x7f1b900377d0 auth_method 0 2024-04-22T08:21:57.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.764+0000 7f1bb1d11640 1 -- 172.21.15.40:0/1632458567 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1bac136890 msgr2=0x7f1bac122fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:57.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.764+0000 7f1bb1d11640 1 --2- 172.21.15.40:0/1632458567 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1bac136890 0x7f1bac122fd0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1ba800b750 tx=0x7f1ba800bc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:57.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.764+0000 7f1ba27fc640 1 -- 172.21.15.40:0/1632458567 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba800be00 con 0x7f1bac136890 2024-04-22T08:21:57.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.764+0000 7f1ba27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:57.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.764+0000 7f1ba27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:21:57.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.764+0000 7f1ba27fc640 10 monclient: dump: 2024-04-22T08:21:57.779 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:57.779 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:57.779 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:57.779 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:57.779 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:57.779 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:57.779 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:57.779 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:57.780 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:57.780 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:57.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.765+0000 7f1ba27fc640 1 -- 172.21.15.40:0/1632458567 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f1ba8010c40 con 0x7f1bac136890 2024-04-22T08:21:57.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.765+0000 7f1ba27fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:57.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.765+0000 7f1ba27fc640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:21:57.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.765+0000 7f1ba27fc640 1 -- 172.21.15.40:0/1632458567 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1bac136890 msgr2=0x7f1bac122fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:57.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.765+0000 7f1ba27fc640 1 --2- 172.21.15.40:0/1632458567 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1bac136890 0x7f1bac122fd0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1ba800b750 tx=0x7f1ba800bc10 comp rx=0 tx=0).stop 2024-04-22T08:21:57.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.765+0000 7f1ba27fc640 1 -- 172.21.15.40:0/1632458567 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1bac136890 msgr2=0x7f1bac122fd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:21:57.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.765+0000 7f1ba27fc640 1 --2- 172.21.15.40:0/1632458567 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1bac136890 0x7f1bac122fd0 secure :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1ba800b750 tx=0x7f1ba800bc10 comp rx=0 tx=0).stop 2024-04-22T08:21:57.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.765+0000 7f1bb1510640 1 --2- 172.21.15.40:0/1632458567 >> v2:172.21.15.189:6800/30168 conn(0x7f1b900377d0 0x7f1b90039c90 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f1bac09f2f0 tx=0x7f1b9c007e30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:57.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.771+0000 7f1bb3f75640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:21:57.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.771+0000 7f1bb3f75640 1 --2- 172.21.15.40:0/1632458567 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1bac123510 0x7f1bac11fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:57.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.771+0000 7f1bb3f75640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:57.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.771+0000 7f1bb3f75640 1 -- 172.21.15.40:0/1632458567 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bac0a41d0 con 0x7f1bac123510 2024-04-22T08:21:57.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.771+0000 7f1bb1d11640 1 --2- 172.21.15.40:0/1632458567 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1bac123510 0x7f1bac11fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:57.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.771+0000 7f1bb1d11640 10 monclient: get_auth_request con 0x7f1bac123510 auth_method 0 2024-04-22T08:21:57.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.771+0000 7f1bb1d11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:57.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.771+0000 7f1bb1d11640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:57.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.771+0000 7f1bb1d11640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:57.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.771+0000 7f1bb1d11640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:57.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.771+0000 7f1bb1d11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:57.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.771+0000 7f1bb1d11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:57.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.771+0000 7f1bb1d11640 10 monclient(hunting): handle_auth_done global_id 104345 payload 824 2024-04-22T08:21:57.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.771+0000 7f1bb1d11640 1 -- 172.21.15.40:0/1632458567 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1bac04e9b0 con 0x7f1bac123510 2024-04-22T08:21:57.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.774+0000 7f1bb1d11640 1 -- 172.21.15.40:0/1632458567 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1bac123510 msgr2=0x7f1bac11fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:57.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.774+0000 7f1bb1d11640 1 --2- 172.21.15.40:0/1632458567 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1bac123510 0x7f1bac11fac0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1ba800b750 tx=0x7f1ba8004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:57.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.774+0000 7f1ba27fc640 1 -- 172.21.15.40:0/1632458567 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba8022040 con 0x7f1bac123510 2024-04-22T08:21:57.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.774+0000 7f1ba27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:57.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.774+0000 7f1ba27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:21:57.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.774+0000 7f1ba27fc640 10 monclient: dump: 2024-04-22T08:21:57.782 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:57.782 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:57.782 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:57.782 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:57.782 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:57.782 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:57.782 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:57.782 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:57.782 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:57.782 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:57.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.774+0000 7f1ba27fc640 1 -- 172.21.15.40:0/1632458567 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4631 (secure 0 0 0) 0x7f1ba8014840 con 0x7f1bac123510 2024-04-22T08:21:57.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.774+0000 7f1ba27fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:21:57.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.774+0000 7f1ba27fc640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:21:57.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.774+0000 7f1ba27fc640 1 -- 172.21.15.40:0/1632458567 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1bac123510 msgr2=0x7f1bac11fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:57.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.774+0000 7f1ba27fc640 1 --2- 172.21.15.40:0/1632458567 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1bac123510 0x7f1bac11fac0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1ba800b750 tx=0x7f1ba8004730 comp rx=0 tx=0).stop 2024-04-22T08:21:57.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.774+0000 7f1ba27fc640 1 -- 172.21.15.40:0/1632458567 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1bac123510 msgr2=0x7f1bac11fac0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:21:57.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.774+0000 7f1ba27fc640 1 --2- 172.21.15.40:0/1632458567 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1bac123510 0x7f1bac11fac0 secure :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1ba800b750 tx=0x7f1ba8004730 comp rx=0 tx=0).stop 2024-04-22T08:21:57.783 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:21:57.783 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:21:57.783 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:21:57.784 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:21:57.784 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 92, 2024-04-22T08:21:57.784 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:21:57.784 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:21:57.784 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:21:57.784 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:21:57.784 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:57.784 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 2, 2024-04-22T08:21:57.784 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:21:57.784 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:21:57.784 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:21:57.784 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:21:57.784 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:21:57.784 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:21:57.784 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:21:57.785 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:21:57.785 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:21:57.785 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:21:57.785 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:21:57.785 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:21:57.785 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:21:57.785 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:57.785 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:21:57.785 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:21:57.785 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:21:57.785 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:21:57.785 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:21:57.785 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:21:57.785 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:21:57.786 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:21:57.786 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:21:57.786 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:21:57.786 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:21:57.786 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:21:57.786 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:57.786 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:57.786 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:21:57.786 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:21:57.786 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:21:57.786 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:21:57.786 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:21:57.786 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:21:57.787 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:21:57.787 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:21:57.787 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:21:57.787 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:21:57.787 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:21:57.787 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:21:57.787 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:21:57.787 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:21:57.787 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:21:57.787 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:21:57.787 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:21:57.787 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:21:57.787 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:21:57.787 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:21:57.787 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:21:57.788 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:21:57.788 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:21:57.788 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:21:57.788 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:21:57.788 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:21:57.788 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:21:57.788 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:57.788 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:21:57.788 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:57.788 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:21:57.788 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:57.788 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:21:57.788 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:21:57.788 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:21:57.789 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:21:57.789 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:57.789 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:21:57.789 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:21:57.789 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:57.789 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:57.789 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:57.789 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:21:57.789 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:21:57.789 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:57.789 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:57.789 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:57.789 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:57.789 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:21:57.789 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:21:57.789 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:21:57.790 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:21:57.790 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:21:57.790 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:57.790 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:57.790 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:21:57.790 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:21:57.790 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:21:57.790 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:21:57.790 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:57.790 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:21:57.790 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:21:57.790 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:57.790 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:57.790 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:57.791 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:21:57.791 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:21:57.791 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:57.791 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:57.791 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:57.791 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:57.791 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:21:57.791 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:21:57.791 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:21:57.791 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:21:57.791 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:21:57.791 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:57.791 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:57.792 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:21:57.792 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:21:57.792 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:21:57.792 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:21:57.792 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:57.792 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:21:57.792 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:21:57.792 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:57.792 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:57.792 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:57.792 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:21:57.792 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:21:57.792 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:57.792 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:57.793 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:57.793 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:57.793 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:21:57.793 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:21:57.793 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:21:57.793 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:21:57.793 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:21:57.793 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:57.793 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:57.793 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:57.793 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:21:57.793 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:21:57.793 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:57.793 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:21:57.794 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:21:57.794 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:21:57.794 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:57.794 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:57.794 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:21:57.794 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:57.794 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:21:57.794 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:21:57.794 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:21:57.794 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:57.794 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:57.794 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:21:57.795 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:57.795 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:21:57.795 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:21:57.795 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 2 2024-04-22T08:21:57.795 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:57.795 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:57.795 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:57.795 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:21:57.795 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:21:57.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.774+0000 7f1bb3f75640 1 -- 172.21.15.40:0/1632458567 >> v2:172.21.15.189:6800/30168 conn(0x7f1b900377d0 msgr2=0x7f1b90039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:57.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.774+0000 7f1bb3f75640 1 --2- 172.21.15.40:0/1632458567 >> v2:172.21.15.189:6800/30168 conn(0x7f1b900377d0 0x7f1b90039c90 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f1bac09f2f0 tx=0x7f1b9c007e30 comp rx=0 tx=0).stop 2024-04-22T08:21:57.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.774+0000 7f1bb3f75640 10 monclient: shutdown 2024-04-22T08:21:57.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.774+0000 7f1bb3f75640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:57.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.774+0000 7f1bb3f75640 1 -- 172.21.15.40:0/1632458567 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1bac122610 msgr2=0x7f1bac122a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:57.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.774+0000 7f1bb3f75640 1 --2- 172.21.15.40:0/1632458567 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1bac122610 0x7f1bac122a90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1ba4036040 tx=0x7f1ba4004a40 comp rx=0 tx=0).stop 2024-04-22T08:21:57.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.774+0000 7f1bb2512640 1 -- 172.21.15.40:0/1632458567 reap_dead start 2024-04-22T08:21:57.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.774+0000 7f1bb2512640 1 -- 172.21.15.40:0/1632458567 reap_dead start 2024-04-22T08:21:57.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.774+0000 7f1bb3f75640 1 -- 172.21.15.40:0/1632458567 shutdown_connections 2024-04-22T08:21:57.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.774+0000 7f1bb3f75640 1 -- 172.21.15.40:0/1632458567 >> 172.21.15.40:0/1632458567 conn(0x7f1bac0a1b60 msgr2=0x7f1bac09cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:57.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.774+0000 7f1bb3f75640 1 -- 172.21.15.40:0/1632458567 shutdown_connections 2024-04-22T08:21:57.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.774+0000 7f1bb3f75640 1 -- 172.21.15.40:0/1632458567 wait complete. 2024-04-22T08:21:57.798 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:21:57.798 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-22T08:21:57.798 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-22T08:21:57.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.898+0000 7f87cb724640 10 monclient: get_monmap_and_config 2024-04-22T08:21:57.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.898+0000 7f87cb724640 10 monclient: build_initial_monmap 2024-04-22T08:21:57.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.898+0000 7f87cb724640 10 monclient: monmap: 2024-04-22T08:21:57.897 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:57.898 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:57.898 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:21:57.898944+0000 2024-04-22T08:21:57.898 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:21:57.898944+0000 2024-04-22T08:21:57.898 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:57.898 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:57.898 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:21:57.898 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:21:57.898 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:21:57.898 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:21:57.898 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:21:57.898 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:57.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.898+0000 7f87cb724640 1 Processor -- start 2024-04-22T08:21:57.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.898+0000 7f87cb724640 1 -- start start 2024-04-22T08:21:57.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.898+0000 7f87cb724640 10 monclient: init 2024-04-22T08:21:57.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.899+0000 7f87cb724640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:57.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.899+0000 7f87cb724640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-04-22T08:21:57.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.899+0000 7f87cb724640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f87c40736d0 addr v1:172.21.15.189:6789/0 2024-04-22T08:21:57.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.899+0000 7f87cb724640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f87c4134470 addr v1:172.21.15.40:6789/0 2024-04-22T08:21:57.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.899+0000 7f87cb724640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f87c4071370 0x7f87c413c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:57.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.899+0000 7f87cb724640 10 monclient(hunting): picked mon.noname-a con 0x7f87c4071370 addr v2:172.21.15.40:3300/0 2024-04-22T08:21:57.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.899+0000 7f87cb724640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87c4073ab0 con 0x7f87c4134470 2024-04-22T08:21:57.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.899+0000 7f87cb724640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87c4133910 con 0x7f87c40736d0 2024-04-22T08:21:57.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.899+0000 7f87cb724640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:57.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.899+0000 7f87cb724640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87c4070300 con 0x7f87c4071370 2024-04-22T08:21:57.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.899+0000 7f87cb724640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:57.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.899+0000 7f87cb724640 10 monclient(hunting): authenticate will time out at 1391.893232s 2024-04-22T08:21:57.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.899+0000 7f87c9cc1640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f87c4071370 0x7f87c413c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:57.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.899+0000 7f87c8cbf640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f87c4134470 0x7f87c4138b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:36598/0 (socket says 172.21.15.40:36598) 2024-04-22T08:21:57.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.899+0000 7f87c8cbf640 1 -- 172.21.15.40:0/2618879356 learned_addr learned my addr 172.21.15.40:0/2618879356 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:21:57.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.899+0000 7f87c9cc1640 10 monclient(hunting): get_auth_request con 0x7f87c4071370 auth_method 0 2024-04-22T08:21:57.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.899+0000 7f87c9cc1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:57.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.899+0000 7f87c9cc1640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:57.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.899+0000 7f87c9cc1640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:57.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.899+0000 7f87c9cc1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:57.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.899+0000 7f87c9cc1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:57.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.899+0000 7f87c9cc1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:57.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.899+0000 7f87c9cc1640 10 monclient(hunting): handle_auth_done global_id 114183 payload 274 2024-04-22T08:21:57.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.899+0000 7f87c9cc1640 1 -- 172.21.15.40:0/2618879356 >> v1:172.21.15.189:6789/0 conn(0x7f87c40736d0 legacy=0x7f87c4133d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:21:57.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.899+0000 7f87c9cc1640 1 -- 172.21.15.40:0/2618879356 >> v1:172.21.15.40:6789/0 conn(0x7f87c4134470 legacy=0x7f87c4138b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:21:57.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.899+0000 7f87c9cc1640 10 monclient: _finish_hunting 0 2024-04-22T08:21:57.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.899+0000 7f87c9cc1640 1 monclient: found mon.noname-a 2024-04-22T08:21:57.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.899+0000 7f87c9cc1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:57.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.899+0000 7f87c9cc1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:21:57.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.899+0000 7f87c9cc1640 1 -- 172.21.15.40:0/2618879356 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c40705e0 con 0x7f87c4071370 2024-04-22T08:21:57.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.902+0000 7f87c9cc1640 1 -- 172.21.15.40:0/2618879356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87c4071370 msgr2=0x7f87c413c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:57.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.902+0000 7f87c9cc1640 1 --2- 172.21.15.40:0/2618879356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87c4071370 0x7f87c413c2b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f87c406cb60 tx=0x7f87bc02bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be09817b4c047052 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:57.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.902+0000 7f87bbfff640 1 -- 172.21.15.40:0/2618879356 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87bc02ce10 con 0x7f87c4071370 2024-04-22T08:21:57.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.902+0000 7f87bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:57.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.902+0000 7f87bbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:21:57.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.902+0000 7f87bbfff640 10 monclient: dump: 2024-04-22T08:21:57.907 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:57.907 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:57.907 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:57.907 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:57.907 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:57.907 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:57.907 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:57.907 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:57.907 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:57.908 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:57.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.902+0000 7f87bbfff640 10 monclient: _finish_auth 0 2024-04-22T08:21:57.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.902+0000 7f87bbfff640 10 monclient: _check_auth_tickets 2024-04-22T08:21:57.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.902+0000 7f87bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:57.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.902+0000 7f87bbfff640 1 -- 172.21.15.40:0/2618879356 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87bc034440 con 0x7f87c4071370 2024-04-22T08:21:57.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.902+0000 7f87bbfff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:57.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.902+0000 7f87bbfff640 1 -- 172.21.15.40:0/2618879356 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87bc0347d0 con 0x7f87c4071370 2024-04-22T08:21:57.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.902+0000 7f87cb724640 5 monclient: authenticate success, global_id 114183 2024-04-22T08:21:57.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.902+0000 7f87cb724640 10 monclient: get_monmap_and_config success 2024-04-22T08:21:57.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.902+0000 7f87cb724640 4 set_mon_vals no callback set 2024-04-22T08:21:57.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.902+0000 7f87bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:57.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.902+0000 7f87bbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:21:57.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.902+0000 7f87bbfff640 10 monclient: dump: 2024-04-22T08:21:57.908 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:57.909 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:57.909 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:57.909 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:57.909 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:57.909 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:57.909 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:57.909 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:57.909 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:57.909 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:57.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.903+0000 7f87cb724640 10 monclient: shutdown 2024-04-22T08:21:57.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.903+0000 7f87cb724640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:57.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.903+0000 7f87cb724640 1 -- 172.21.15.40:0/2618879356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87c4071370 msgr2=0x7f87c413c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:57.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.903+0000 7f87cb724640 1 --2- 172.21.15.40:0/2618879356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87c4071370 0x7f87c413c2b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f87c406cb60 tx=0x7f87bc02bd70 comp rx=0 tx=0).stop 2024-04-22T08:21:57.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.903+0000 7f87cb724640 1 -- 172.21.15.40:0/2618879356 shutdown_connections 2024-04-22T08:21:57.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.903+0000 7f87cb724640 1 --2- 172.21.15.40:0/2618879356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87c4071370 0x7f87c413c2b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:57.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.903+0000 7f87cb724640 1 -- 172.21.15.40:0/2618879356 >> 172.21.15.40:0/2618879356 conn(0x7f87c40a1780 msgr2=0x7f87c40a1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:57.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.903+0000 7f87cb724640 1 -- 172.21.15.40:0/2618879356 shutdown_connections 2024-04-22T08:21:57.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.903+0000 7f87cb724640 1 -- 172.21.15.40:0/2618879356 wait complete. 2024-04-22T08:21:57.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.904+0000 7f87cb724640 10 monclient: build_initial_monmap 2024-04-22T08:21:57.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.904+0000 7f87cb724640 10 monclient: monmap: 2024-04-22T08:21:57.910 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:57.910 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:21:57.910 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:21:57.910 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:21:57.910 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:57.910 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:57.910 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:21:57.910 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:21:57.910 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:21:57.911 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:57.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.904+0000 7f87cb724640 1 Processor -- start 2024-04-22T08:21:57.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.904+0000 7f87cb724640 1 -- start start 2024-04-22T08:21:57.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.904+0000 7f87cb724640 10 monclient: init 2024-04-22T08:21:57.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.904+0000 7f87cb724640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:57.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.904+0000 7f87cb724640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-22T08:21:57.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.904+0000 7f87cb724640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87c4071370 0x7f87c41363d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:57.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.904+0000 7f87cb724640 10 monclient(hunting): picked mon.noname-b con 0x7f87c4071370 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:21:57.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.904+0000 7f87cb724640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87c40736d0 0x7f87c4136910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:57.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.904+0000 7f87cb724640 10 monclient(hunting): picked mon.noname-a con 0x7f87c40736d0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:21:57.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.904+0000 7f87cb724640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f87c4134470 0x7f87c4136e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:57.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.904+0000 7f87cb724640 10 monclient(hunting): picked mon.noname-c con 0x7f87c4134470 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:57.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.904+0000 7f87cb724640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:57.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.904+0000 7f87cb724640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c413b400 con 0x7f87c40736d0 2024-04-22T08:21:57.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.904+0000 7f87cb724640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:57.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.904+0000 7f87cb724640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c413bca0 con 0x7f87c4071370 2024-04-22T08:21:57.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.904+0000 7f87cb724640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:57.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.904+0000 7f87cb724640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87c4139ff0 con 0x7f87c4134470 2024-04-22T08:21:57.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.904+0000 7f87cb724640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:57.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.904+0000 7f87cb724640 10 monclient(hunting): authenticate will time out at 1391.898601s 2024-04-22T08:21:57.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.904+0000 7f87c9cc1640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f87c4134470 0x7f87c4136e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:57.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.904+0000 7f87c9cc1640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f87c4134470 0x7f87c4136e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:41996/0 (socket says 172.21.15.40:41996) 2024-04-22T08:21:57.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.904+0000 7f87c9cc1640 1 -- 172.21.15.40:0/1613707749 learned_addr learned my addr 172.21.15.40:0/1613707749 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:21:57.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.904+0000 7f87c9cc1640 10 monclient(hunting): get_auth_request con 0x7f87c4134470 auth_method 0 2024-04-22T08:21:57.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.904+0000 7f87c9cc1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:57.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.904+0000 7f87c9cc1640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:57.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.904+0000 7f87c9cc1640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:57.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.904+0000 7f87c94c0640 1 --2- 172.21.15.40:0/1613707749 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87c4071370 0x7f87c41363d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:57.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.905+0000 7f87c9cc1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:57.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.905+0000 7f87c9cc1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:57.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.905+0000 7f87c9cc1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:57.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.905+0000 7f87c94c0640 10 monclient(hunting): get_auth_request con 0x7f87c4071370 auth_method 0 2024-04-22T08:21:57.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.905+0000 7f87c94c0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:57.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.905+0000 7f87c94c0640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:57.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.905+0000 7f87c94c0640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:57.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.905+0000 7f87c94c0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:57.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.905+0000 7f87c94c0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:57.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.905+0000 7f87c94c0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:57.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.905+0000 7f87c9cc1640 10 monclient(hunting): handle_auth_done global_id 104348 payload 824 2024-04-22T08:21:57.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.905+0000 7f87c9cc1640 1 -- 172.21.15.40:0/1613707749 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87c4071370 msgr2=0x7f87c41363d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:57.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.905+0000 7f87c9cc1640 1 --2- 172.21.15.40:0/1613707749 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87c4071370 0x7f87c41363d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:57.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.905+0000 7f87c9cc1640 1 -- 172.21.15.40:0/1613707749 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87c40736d0 msgr2=0x7f87c4136910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:57.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.905+0000 7f87c9cc1640 1 --2- 172.21.15.40:0/1613707749 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87c40736d0 0x7f87c4136910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:57.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.905+0000 7f87c9cc1640 10 monclient: _finish_hunting 0 2024-04-22T08:21:57.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.905+0000 7f87c9cc1640 1 monclient: found mon.noname-c 2024-04-22T08:21:57.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.905+0000 7f87c9cc1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:57.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.905+0000 7f87c9cc1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:21:57.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.905+0000 7f87c9cc1640 1 -- 172.21.15.40:0/1613707749 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87bc009a70 con 0x7f87c4134470 2024-04-22T08:21:57.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.907+0000 7f87c9cc1640 1 -- 172.21.15.40:0/1613707749 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f87c4134470 msgr2=0x7f87c4136e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:57.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.907+0000 7f87c9cc1640 1 --2- 172.21.15.40:0/1613707749 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f87c4134470 0x7f87c4136e50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f87bc02c380 tx=0x7f87bc004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:57.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.907+0000 7f87b9ffb640 1 -- 172.21.15.40:0/1613707749 DispatchQueue::entry inject delay of 0.002000 2024-04-22T08:21:57.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.909+0000 7f87b9ffb640 1 -- 172.21.15.40:0/1613707749 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87bc03f840 con 0x7f87c4134470 2024-04-22T08:21:57.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.909+0000 7f87b9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:57.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.909+0000 7f87b9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:21:57.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.909+0000 7f87b9ffb640 10 monclient: dump: 2024-04-22T08:21:57.917 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:57.917 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:57.917 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:57.917 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:57.917 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:57.917 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:57.917 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:57.918 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:57.918 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:57.918 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:57.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.909+0000 7f87b9ffb640 10 monclient: _finish_auth 0 2024-04-22T08:21:57.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.909+0000 7f87b9ffb640 10 monclient: _check_auth_tickets 2024-04-22T08:21:57.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.909+0000 7f87b9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:57.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.909+0000 7f87b9ffb640 1 -- 172.21.15.40:0/1613707749 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87bc03f9a0 con 0x7f87c4134470 2024-04-22T08:21:57.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.909+0000 7f87b9ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:57.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.909+0000 7f87b9ffb640 1 -- 172.21.15.40:0/1613707749 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87bc03fd30 con 0x7f87c4134470 2024-04-22T08:21:57.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.909+0000 7f87cb724640 5 monclient: authenticate success, global_id 104348 2024-04-22T08:21:57.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.909+0000 7f87cb724640 10 monclient: _renew_subs 2024-04-22T08:21:57.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.909+0000 7f87cb724640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:21:57.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.909+0000 7f87cb724640 1 -- 172.21.15.40:0/1613707749 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87c406fa10 con 0x7f87c4134470 2024-04-22T08:21:57.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.909+0000 7f87bb7fe640 4 set_mon_vals no callback set 2024-04-22T08:21:57.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.909+0000 7f87b9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:57.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.909+0000 7f87b9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:21:57.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.909+0000 7f87b9ffb640 10 monclient: dump: 2024-04-22T08:21:57.919 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:57.919 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:57.919 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:57.919 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:57.919 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:57.919 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:57.919 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:57.919 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:57.919 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:57.919 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:57.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.910+0000 7f87cb724640 10 monclient: _renew_subs 2024-04-22T08:21:57.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.910+0000 7f87cb724640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:21:57.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.910+0000 7f87cb724640 1 -- 172.21.15.40:0/1613707749 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87c4139d10 con 0x7f87c4134470 2024-04-22T08:21:57.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.910+0000 7f87b9ffb640 1 -- 172.21.15.40:0/1613707749 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f87bc0473c0 con 0x7f87c4134470 2024-04-22T08:21:57.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.910+0000 7f87cb724640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:57.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.910+0000 7f87cb724640 1 --2- 172.21.15.40:0/1613707749 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87c41c9150 0x7f87c402b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:57.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.910+0000 7f87cb724640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:57.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.910+0000 7f87cb724640 1 -- 172.21.15.40:0/1613707749 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c4066900 con 0x7f87c41c9150 2024-04-22T08:21:57.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.911+0000 7f87b9ffb640 1 --2- 172.21.15.40:0/1613707749 >> v2:172.21.15.189:6800/30168 conn(0x7f87a8037820 0x7f87a8039ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:57.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.911+0000 7f87b9ffb640 1 -- 172.21.15.40:0/1613707749 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(311..311 src has 239..311) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f87bc06ceb0 con 0x7f87c4134470 2024-04-22T08:21:57.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.911+0000 7f87c94c0640 1 --2- 172.21.15.40:0/1613707749 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87c41c9150 0x7f87c402b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:57.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.911+0000 7f87c94c0640 10 monclient: get_auth_request con 0x7f87c41c9150 auth_method 0 2024-04-22T08:21:57.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.911+0000 7f87c94c0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:57.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.911+0000 7f87c94c0640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:57.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.911+0000 7f87c94c0640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:57.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.911+0000 7f87c8cbf640 1 --2- 172.21.15.40:0/1613707749 >> v2:172.21.15.189:6800/30168 conn(0x7f87a8037820 0x7f87a8039ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:57.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.911+0000 7f87c94c0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:57.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.911+0000 7f87c94c0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:57.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.911+0000 7f87c94c0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:57.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.911+0000 7f87c8cbf640 10 monclient: get_auth_request con 0x7f87a8037820 auth_method 0 2024-04-22T08:21:57.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.911+0000 7f87c94c0640 10 monclient(hunting): handle_auth_done global_id 65188 payload 824 2024-04-22T08:21:57.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.911+0000 7f87c94c0640 1 -- 172.21.15.40:0/1613707749 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f87c4070ba0 con 0x7f87c41c9150 2024-04-22T08:21:57.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.913+0000 7f87c94c0640 1 -- 172.21.15.40:0/1613707749 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87c41c9150 msgr2=0x7f87c402b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:57.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.913+0000 7f87c94c0640 1 --2- 172.21.15.40:0/1613707749 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87c41c9150 0x7f87c402b200 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f87c000f7a0 tx=0x7f87c000fc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:57.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.913+0000 7f87c8cbf640 1 --2- 172.21.15.40:0/1613707749 >> v2:172.21.15.189:6800/30168 conn(0x7f87a8037820 0x7f87a8039ce0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f87b4007fd0 tx=0x7f87b4007aa0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:57.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.914+0000 7f87b9ffb640 1 -- 172.21.15.40:0/1613707749 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c000d9b0 con 0x7f87c41c9150 2024-04-22T08:21:57.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.914+0000 7f87b9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:57.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.914+0000 7f87b9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:21:57.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.914+0000 7f87b9ffb640 10 monclient: dump: 2024-04-22T08:21:57.921 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:57.922 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:57.922 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:57.922 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:57.922 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:57.922 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:57.922 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:57.922 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:57.922 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:57.922 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:57.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.914+0000 7f87b9ffb640 1 -- 172.21.15.40:0/1613707749 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f87c000db10 con 0x7f87c41c9150 2024-04-22T08:21:57.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.914+0000 7f87b9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:57.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.914+0000 7f87b9ffb640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:21:57.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.914+0000 7f87b9ffb640 1 -- 172.21.15.40:0/1613707749 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87c41c9150 msgr2=0x7f87c402b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:57.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.914+0000 7f87b9ffb640 1 --2- 172.21.15.40:0/1613707749 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87c41c9150 0x7f87c402b200 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f87c000f7a0 tx=0x7f87c000fc60 comp rx=0 tx=0).stop 2024-04-22T08:21:57.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.914+0000 7f87b9ffb640 1 -- 172.21.15.40:0/1613707749 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87c41c9150 msgr2=0x7f87c402b200 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:21:57.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.914+0000 7f87b9ffb640 1 --2- 172.21.15.40:0/1613707749 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87c41c9150 0x7f87c402b200 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:57.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.921+0000 7f87cb724640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:21:57.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.921+0000 7f87cb724640 1 --2- 172.21.15.40:0/1613707749 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87c41c9530 0x7f87c402b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:57.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.921+0000 7f87cb724640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:57.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.921+0000 7f87cb724640 1 -- 172.21.15.40:0/1613707749 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c404e9b0 con 0x7f87c41c9530 2024-04-22T08:21:57.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.921+0000 7f87c94c0640 1 --2- 172.21.15.40:0/1613707749 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87c41c9530 0x7f87c402b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:57.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.921+0000 7f87c94c0640 10 monclient: get_auth_request con 0x7f87c41c9530 auth_method 0 2024-04-22T08:21:57.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.921+0000 7f87c94c0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:57.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.921+0000 7f87c94c0640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:57.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.921+0000 7f87c94c0640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:57.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.922+0000 7f87c94c0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:57.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.922+0000 7f87c94c0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:57.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.922+0000 7f87c94c0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:57.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.922+0000 7f87c94c0640 10 monclient(hunting): handle_auth_done global_id 65191 payload 824 2024-04-22T08:21:57.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.922+0000 7f87c94c0640 1 -- 172.21.15.40:0/1613707749 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f87c409d110 con 0x7f87c41c9530 2024-04-22T08:21:57.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.924+0000 7f87c94c0640 1 -- 172.21.15.40:0/1613707749 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87c41c9530 msgr2=0x7f87c402b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:57.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.924+0000 7f87c94c0640 1 --2- 172.21.15.40:0/1613707749 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87c41c9530 0x7f87c402b740 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f87c000f7a0 tx=0x7f87c0003760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:57.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.924+0000 7f87b9ffb640 1 -- 172.21.15.40:0/1613707749 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c0023070 con 0x7f87c41c9530 2024-04-22T08:21:57.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.924+0000 7f87b9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:57.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.924+0000 7f87b9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:21:57.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.924+0000 7f87b9ffb640 10 monclient: dump: 2024-04-22T08:21:57.926 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:57.926 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:57.926 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:57.926 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:57.926 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:57.926 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:57.926 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:57.926 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:57.926 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:57.926 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:57.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.925+0000 7f87b9ffb640 1 -- 172.21.15.40:0/1613707749 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4801 (secure 0 0 0) 0x7f87c0004800 con 0x7f87c41c9530 2024-04-22T08:21:57.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.925+0000 7f87b9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:21:57.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.925+0000 7f87b9ffb640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:21:57.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.925+0000 7f87b9ffb640 1 -- 172.21.15.40:0/1613707749 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87c41c9530 msgr2=0x7f87c402b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:57.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.925+0000 7f87b9ffb640 1 --2- 172.21.15.40:0/1613707749 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87c41c9530 0x7f87c402b740 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f87c000f7a0 tx=0x7f87c0003760 comp rx=0 tx=0).stop 2024-04-22T08:21:57.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.925+0000 7f87b9ffb640 1 -- 172.21.15.40:0/1613707749 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87c41c9530 msgr2=0x7f87c402b740 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:21:57.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.925+0000 7f87b9ffb640 1 --2- 172.21.15.40:0/1613707749 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87c41c9530 0x7f87c402b740 secure :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f87c000f7a0 tx=0x7f87c0003760 comp rx=0 tx=0).stop 2024-04-22T08:21:57.927 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:21:57.927 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:21:57.927 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:21:57.927 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:21:57.928 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 92, 2024-04-22T08:21:57.928 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:21:57.928 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:21:57.928 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:21:57.928 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:21:57.928 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:57.928 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 3, 2024-04-22T08:21:57.928 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:21:57.928 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:21:57.928 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:21:57.928 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:21:57.928 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:21:57.928 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:21:57.928 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:21:57.929 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:21:57.929 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:21:57.929 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:21:57.929 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:21:57.929 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:21:57.929 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:21:57.929 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:57.929 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:21:57.929 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:21:57.929 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:21:57.929 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:21:57.929 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:21:57.929 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:21:57.929 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:21:57.930 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:21:57.930 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:21:57.930 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:21:57.930 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:21:57.930 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:21:57.930 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:57.930 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:57.930 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:21:57.930 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:21:57.930 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:21:57.930 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:21:57.930 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:21:57.930 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:21:57.930 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:21:57.931 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:21:57.931 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:21:57.931 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:21:57.931 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:21:57.931 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:21:57.931 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:21:57.931 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:21:57.931 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:21:57.931 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:21:57.931 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:21:57.931 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:21:57.931 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:21:57.931 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:21:57.931 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:21:57.931 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:21:57.932 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:21:57.932 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:21:57.932 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:21:57.932 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:21:57.932 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:21:57.932 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:57.932 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:21:57.932 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:57.932 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:21:57.932 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:57.932 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:21:57.932 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:21:57.932 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:21:57.932 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:21:57.933 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:57.933 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:21:57.933 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:21:57.933 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:57.933 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:57.933 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:57.933 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:21:57.933 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:21:57.933 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:57.933 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:57.933 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:57.933 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:57.933 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:21:57.933 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:21:57.933 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:21:57.934 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:21:57.934 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:21:57.934 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:57.934 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:57.934 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:21:57.934 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:21:57.934 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:21:57.934 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:21:57.934 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:57.934 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:21:57.934 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:21:57.934 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:57.934 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:57.934 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:57.935 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:21:57.935 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:21:57.935 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:57.935 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:57.935 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:57.935 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:57.935 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:21:57.935 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:21:57.935 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:21:57.935 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:21:57.935 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:21:57.935 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:57.935 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:57.935 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:21:57.936 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:21:57.936 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:21:57.936 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:21:57.936 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:57.936 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:21:57.936 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:21:57.936 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:57.936 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:57.936 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:57.936 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:21:57.936 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:21:57.936 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:21:57.936 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:57.936 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:57.936 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:57.937 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:21:57.937 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:21:57.937 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:21:57.937 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:21:57.937 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:21:57.937 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:57.937 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:57.937 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:57.937 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:21:57.937 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:21:57.937 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:57.937 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:21:57.937 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:21:57.937 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:21:57.938 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:57.938 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:57.938 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:21:57.938 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:57.938 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:21:57.938 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:21:57.938 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 5 2024-04-22T08:21:57.938 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:57.938 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:57.938 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:21:57.938 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:57.938 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:21:57.938 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:21:57.938 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 7 2024-04-22T08:21:57.939 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:57.939 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:21:57.939 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": [ 2024-04-22T08:21:57.939 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:21:57.939 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:21:57.939 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:21:57.939 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:21:57.939 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:21:57.939 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:21:57.939 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:21:57.939 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:21:57.939 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:21:57.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.925+0000 7f87cb724640 1 -- 172.21.15.40:0/1613707749 >> v2:172.21.15.189:6800/30168 conn(0x7f87a8037820 msgr2=0x7f87a8039ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:57.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.925+0000 7f87cb724640 1 --2- 172.21.15.40:0/1613707749 >> v2:172.21.15.189:6800/30168 conn(0x7f87a8037820 0x7f87a8039ce0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f87b4007fd0 tx=0x7f87b4007aa0 comp rx=0 tx=0).stop 2024-04-22T08:21:57.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.925+0000 7f87cb724640 10 monclient: shutdown 2024-04-22T08:21:57.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.925+0000 7f87cb724640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:57.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.925+0000 7f87cb724640 1 -- 172.21.15.40:0/1613707749 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f87c4134470 msgr2=0x7f87c4136e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:57.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.925+0000 7f87cb724640 1 --2- 172.21.15.40:0/1613707749 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f87c4134470 0x7f87c4136e50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f87bc02c380 tx=0x7f87bc004a40 comp rx=0 tx=0).stop 2024-04-22T08:21:57.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.925+0000 7f87c9cc1640 1 -- 172.21.15.40:0/1613707749 reap_dead start 2024-04-22T08:21:57.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.925+0000 7f87c9cc1640 1 -- 172.21.15.40:0/1613707749 reap_dead start 2024-04-22T08:21:57.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.925+0000 7f87cb724640 1 -- 172.21.15.40:0/1613707749 shutdown_connections 2024-04-22T08:21:57.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.925+0000 7f87cb724640 1 -- 172.21.15.40:0/1613707749 >> 172.21.15.40:0/1613707749 conn(0x7f87c40a1780 msgr2=0x7f87c4131f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:57.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.925+0000 7f87cb724640 1 -- 172.21.15.40:0/1613707749 shutdown_connections 2024-04-22T08:21:57.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:57.925+0000 7f87cb724640 1 -- 172.21.15.40:0/1613707749 wait complete. 2024-04-22T08:21:57.942 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:21:57.942 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-04-22T08:21:57.942 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-04-22T08:21:57.942 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-04-22T08:21:57.942 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-04-22T08:21:57.943 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-22T08:21:57.943 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T08:21:57.953 INFO:tasks.ceph.mon.c:Stopped 2024-04-22T08:21:57.953 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-22T08:21:57.953 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:21:58.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.091+0000 7f48b30fe640 10 monclient: get_monmap_and_config 2024-04-22T08:21:58.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.091+0000 7f48b30fe640 10 monclient: build_initial_monmap 2024-04-22T08:21:58.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.091+0000 7f48b30fe640 10 monclient: monmap: 2024-04-22T08:21:58.091 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:58.091 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:58.091 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:21:58.092187+0000 2024-04-22T08:21:58.091 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:21:58.092187+0000 2024-04-22T08:21:58.091 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:58.091 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:58.091 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:21:58.091 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:21:58.091 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:21:58.091 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:21:58.091 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:21:58.091 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:58.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.092+0000 7f48b30fe640 1 Processor -- start 2024-04-22T08:21:58.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.092+0000 7f48b30fe640 1 -- start start 2024-04-22T08:21:58.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.092+0000 7f48b30fe640 10 monclient: init 2024-04-22T08:21:58.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.092+0000 7f48b30fe640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:58.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.092+0000 7f48b30fe640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-04-22T08:21:58.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.092+0000 7f48b30fe640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f48ac070e00 0x7f48ac0711e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:58.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.092+0000 7f48b30fe640 10 monclient(hunting): picked mon.noname-a con 0x7f48ac070e00 addr v2:172.21.15.40:3300/0 2024-04-22T08:21:58.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.092+0000 7f48b30fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f48ac0a0690 addr v1:172.21.15.189:6789/0 2024-04-22T08:21:58.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.092+0000 7f48b30fe640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f48ac1360d0 0x7f48ac13dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:58.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.092+0000 7f48b30fe640 10 monclient(hunting): picked mon.noname-c con 0x7f48ac1360d0 addr v2:172.21.15.189:3300/0 2024-04-22T08:21:58.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.092+0000 7f48b30fe640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48ac09e840 con 0x7f48ac0a0690 2024-04-22T08:21:58.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.092+0000 7f48b30fe640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:58.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.092+0000 7f48b30fe640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48ac06d170 con 0x7f48ac070e00 2024-04-22T08:21:58.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.092+0000 7f48b30fe640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:58.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.092+0000 7f48b30fe640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48ac071720 con 0x7f48ac1360d0 2024-04-22T08:21:58.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.092+0000 7f48b30fe640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:58.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.092+0000 7f48b30fe640 10 monclient(hunting): authenticate will time out at 1392.086398s 2024-04-22T08:21:58.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.092+0000 7f48b0e9a640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f48ac070e00 0x7f48ac0711e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:58.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.092+0000 7f48b0e9a640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f48ac070e00 0x7f48ac0711e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56988/0 (socket says 172.21.15.40:56988) 2024-04-22T08:21:58.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.092+0000 7f48b0e9a640 1 -- 172.21.15.40:0/3837675231 learned_addr learned my addr 172.21.15.40:0/3837675231 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:21:58.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.092+0000 7f48b0e9a640 10 monclient(hunting): get_auth_request con 0x7f48ac070e00 auth_method 0 2024-04-22T08:21:58.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.092+0000 7f48b0e9a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:58.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.092+0000 7f48b0e9a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:58.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.092+0000 7f48b0e9a640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:58.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.092+0000 7f48b0e9a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:58.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.092+0000 7f48b0e9a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:58.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.092+0000 7f48b0e9a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:58.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.092+0000 7f48b169b640 1 --2- 172.21.15.40:0/3837675231 >> v2:172.21.15.189:3300/0 conn(0x7f48ac1360d0 0x7f48ac13dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:58.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.092+0000 7f48b169b640 10 monclient(hunting): get_auth_request con 0x7f48ac1360d0 auth_method 0 2024-04-22T08:21:58.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.092+0000 7f48b169b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:58.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.092+0000 7f48b169b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:58.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.092+0000 7f48b169b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:58.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.093+0000 7f48b0e9a640 10 monclient(hunting): handle_auth_done global_id 114186 payload 274 2024-04-22T08:21:58.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.093+0000 7f48b0e9a640 1 -- 172.21.15.40:0/3837675231 >> v2:172.21.15.189:3300/0 conn(0x7f48ac1360d0 msgr2=0x7f48ac13dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:21:58.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.093+0000 7f48b0e9a640 1 --2- 172.21.15.40:0/3837675231 >> v2:172.21.15.189:3300/0 conn(0x7f48ac1360d0 0x7f48ac13dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:58.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.093+0000 7f48b0e9a640 1 -- 172.21.15.40:0/3837675231 >> v1:172.21.15.189:6789/0 conn(0x7f48ac0a0690 legacy=0x7f48ac1359c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:21:58.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.093+0000 7f48b0e9a640 10 monclient: _finish_hunting 0 2024-04-22T08:21:58.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.093+0000 7f48b0e9a640 1 monclient: found mon.noname-a 2024-04-22T08:21:58.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.093+0000 7f48b0e9a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:58.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.093+0000 7f48b0e9a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:21:58.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.093+0000 7f48b0e9a640 1 -- 172.21.15.40:0/3837675231 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48ac06d3c0 con 0x7f48ac070e00 2024-04-22T08:21:58.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.095+0000 7f48b0e9a640 1 -- 172.21.15.40:0/3837675231 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48ac070e00 msgr2=0x7f48ac0711e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:58.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.095+0000 7f48b0e9a640 1 --2- 172.21.15.40:0/3837675231 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48ac070e00 0x7f48ac0711e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f48ac09df40 tx=0x7f48a002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bbd250acea1b449 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:58.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.095+0000 7f48ab7fe640 1 -- 172.21.15.40:0/3837675231 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a0031960 con 0x7f48ac070e00 2024-04-22T08:21:58.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.096+0000 7f48ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:58.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.096+0000 7f48ab7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:21:58.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.096+0000 7f48ab7fe640 10 monclient: dump: 2024-04-22T08:21:58.102 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:58.102 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:58.103 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:58.103 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:58.103 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:58.103 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:58.103 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:58.103 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:58.103 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:58.103 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:58.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.096+0000 7f48ab7fe640 10 monclient: _finish_auth 0 2024-04-22T08:21:58.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.096+0000 7f48ab7fe640 10 monclient: _check_auth_tickets 2024-04-22T08:21:58.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.096+0000 7f48ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:58.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.096+0000 7f48ab7fe640 1 -- 172.21.15.40:0/3837675231 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48a0031ac0 con 0x7f48ac070e00 2024-04-22T08:21:58.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.096+0000 7f48b30fe640 5 monclient: authenticate success, global_id 114186 2024-04-22T08:21:58.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.096+0000 7f48b30fe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:21:58.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.096+0000 7f48ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:58.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.096+0000 7f48b30fe640 10 monclient: get_monmap_and_config success 2024-04-22T08:21:58.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.096+0000 7f48ab7fe640 1 -- 172.21.15.40:0/3837675231 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a0031e50 con 0x7f48ac070e00 2024-04-22T08:21:58.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.096+0000 7f48b30fe640 4 set_mon_vals no callback set 2024-04-22T08:21:58.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.096+0000 7f48ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:58.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.096+0000 7f48ab7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:21:58.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.096+0000 7f48ab7fe640 10 monclient: dump: 2024-04-22T08:21:58.104 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:58.104 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:58.104 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:58.104 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:58.104 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:58.104 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:58.104 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:58.105 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:58.105 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:58.105 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:58.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.096+0000 7f48b30fe640 10 monclient: shutdown 2024-04-22T08:21:58.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.096+0000 7f48b30fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:58.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.096+0000 7f48b30fe640 1 -- 172.21.15.40:0/3837675231 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48ac070e00 msgr2=0x7f48ac0711e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:58.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.096+0000 7f48b30fe640 1 --2- 172.21.15.40:0/3837675231 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48ac070e00 0x7f48ac0711e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f48ac09df40 tx=0x7f48a002b7f0 comp rx=0 tx=0).stop 2024-04-22T08:21:58.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.096+0000 7f48b30fe640 1 -- 172.21.15.40:0/3837675231 shutdown_connections 2024-04-22T08:21:58.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.096+0000 7f48b30fe640 1 --2- 172.21.15.40:0/3837675231 >> v2:172.21.15.189:3300/0 conn(0x7f48ac1360d0 0x7f48ac13dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:58.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.096+0000 7f48b30fe640 1 --2- 172.21.15.40:0/3837675231 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48ac070e00 0x7f48ac0711e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:58.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.096+0000 7f48b30fe640 1 -- 172.21.15.40:0/3837675231 >> 172.21.15.40:0/3837675231 conn(0x7f48ac0a18f0 msgr2=0x7f48ac06ba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:58.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.096+0000 7f48b30fe640 1 -- 172.21.15.40:0/3837675231 shutdown_connections 2024-04-22T08:21:58.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.096+0000 7f48b30fe640 1 -- 172.21.15.40:0/3837675231 wait complete. 2024-04-22T08:21:58.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.097+0000 7f48b30fe640 10 monclient: build_initial_monmap 2024-04-22T08:21:58.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.097+0000 7f48b30fe640 10 monclient: monmap: 2024-04-22T08:21:58.106 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:21:58.106 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:21:58.106 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:21:58.106 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:21:58.106 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:21:58.106 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:58.106 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:21:58.106 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:21:58.106 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:21:58.106 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:58.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.097+0000 7f48b30fe640 1 Processor -- start 2024-04-22T08:21:58.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.097+0000 7f48b30fe640 1 -- start start 2024-04-22T08:21:58.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.097+0000 7f48b30fe640 10 monclient: init 2024-04-22T08:21:58.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.098+0000 7f48b30fe640 10 monclient: _reopen_session rank -1 2024-04-22T08:21:58.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.098+0000 7f48b30fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:21:58.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.098+0000 7f48b30fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48ac070e00 0x7f48ac1cab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:58.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.098+0000 7f48b30fe640 10 monclient(hunting): picked mon.noname-a con 0x7f48ac070e00 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:21:58.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.098+0000 7f48b30fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48ac0a0690 0x7f48ac1cb060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:58.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.098+0000 7f48b30fe640 10 monclient(hunting): picked mon.noname-b con 0x7f48ac0a0690 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:21:58.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.098+0000 7f48b30fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48ac1360d0 0x7f48ac11c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:58.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.098+0000 7f48b30fe640 10 monclient(hunting): picked mon.noname-c con 0x7f48ac1360d0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:21:58.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.098+0000 7f48b30fe640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:58.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.098+0000 7f48b30fe640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48ac11eec0 con 0x7f48ac070e00 2024-04-22T08:21:58.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.098+0000 7f48b30fe640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:58.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.098+0000 7f48b30fe640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48ac1358c0 con 0x7f48ac0a0690 2024-04-22T08:21:58.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.098+0000 7f48b30fe640 10 monclient(hunting): start opening mon connection 2024-04-22T08:21:58.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.098+0000 7f48b30fe640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48ac070bd0 con 0x7f48ac1360d0 2024-04-22T08:21:58.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.098+0000 7f48b30fe640 10 monclient(hunting): _renew_subs 2024-04-22T08:21:58.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.098+0000 7f48b30fe640 10 monclient(hunting): authenticate will time out at 1392.091905s 2024-04-22T08:21:58.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.098+0000 7f48b169b640 1 -- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48ac1360d0 msgr2=0x7f48ac11c6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:21:58.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.098+0000 7f48b0e9a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48ac070e00 0x7f48ac1cab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:58.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.098+0000 7f48b169b640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48ac1360d0 0x7f48ac11c6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:21:58.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.098+0000 7f48b0e9a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48ac070e00 0x7f48ac1cab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57000/0 (socket says 172.21.15.40:57000) 2024-04-22T08:21:58.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.098+0000 7f48b0e9a640 1 -- 172.21.15.40:0/4109145104 learned_addr learned my addr 172.21.15.40:0/4109145104 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:21:58.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.098+0000 7f48b0e9a640 10 monclient(hunting): get_auth_request con 0x7f48ac070e00 auth_method 0 2024-04-22T08:21:58.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.098+0000 7f48b0e9a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:58.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.098+0000 7f48b0e9a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:58.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.098+0000 7f48b0e9a640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:58.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.098+0000 7f48b0e9a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:21:58.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.098+0000 7f48b0e9a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:21:58.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.098+0000 7f48b0e9a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:21:58.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.098+0000 7f48abfff640 1 --2- 172.21.15.40:0/4109145104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48ac0a0690 0x7f48ac1cb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:58.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.098+0000 7f48abfff640 10 monclient(hunting): get_auth_request con 0x7f48ac0a0690 auth_method 0 2024-04-22T08:21:58.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.098+0000 7f48abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:21:58.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.098+0000 7f48abfff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:21:58.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.098+0000 7f48abfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:21:58.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.098+0000 7f48b0e9a640 10 monclient(hunting): handle_auth_done global_id 114189 payload 824 2024-04-22T08:21:58.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.098+0000 7f48b0e9a640 1 -- 172.21.15.40:0/4109145104 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48ac1360d0 msgr2=0x7f48ac11c6f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:21:58.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.098+0000 7f48b0e9a640 1 --2- 172.21.15.40:0/4109145104 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48ac1360d0 0x7f48ac11c6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:58.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.098+0000 7f48b0e9a640 1 -- 172.21.15.40:0/4109145104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48ac0a0690 msgr2=0x7f48ac1cb060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:58.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.098+0000 7f48b0e9a640 1 --2- 172.21.15.40:0/4109145104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48ac0a0690 0x7f48ac1cb060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:58.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.098+0000 7f48b0e9a640 10 monclient: _finish_hunting 0 2024-04-22T08:21:58.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.098+0000 7f48b0e9a640 1 monclient: found mon.noname-a 2024-04-22T08:21:58.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.098+0000 7f48b0e9a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:21:58.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.098+0000 7f48b0e9a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:21:58.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.098+0000 7f48b0e9a640 1 -- 172.21.15.40:0/4109145104 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a00094d0 con 0x7f48ac070e00 2024-04-22T08:21:58.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.100+0000 7f48b0e9a640 1 -- 172.21.15.40:0/4109145104 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48ac070e00 msgr2=0x7f48ac1cab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:21:58.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.100+0000 7f48b0e9a640 1 --2- 172.21.15.40:0/4109145104 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48ac070e00 0x7f48ac1cab20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f48a002b830 tx=0x7f48a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:58.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.100+0000 7f48a97fa640 1 -- 172.21.15.40:0/4109145104 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a0031e30 con 0x7f48ac070e00 2024-04-22T08:21:58.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.100+0000 7f48a97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:58.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.100+0000 7f48a97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:21:58.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.100+0000 7f48a97fa640 10 monclient: dump: 2024-04-22T08:21:58.110 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:58.110 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:58.110 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:58.110 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:58.110 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:58.111 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:58.111 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:58.111 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:58.111 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:58.111 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:58.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.100+0000 7f48a97fa640 10 monclient: _finish_auth 0 2024-04-22T08:21:58.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.100+0000 7f48a97fa640 10 monclient: _check_auth_tickets 2024-04-22T08:21:58.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.100+0000 7f48a97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:21:58.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.100+0000 7f48a97fa640 1 -- 172.21.15.40:0/4109145104 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48a003c400 con 0x7f48ac070e00 2024-04-22T08:21:58.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.100+0000 7f48b30fe640 5 monclient: authenticate success, global_id 114189 2024-04-22T08:21:58.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.100+0000 7f48b30fe640 10 monclient: _renew_subs 2024-04-22T08:21:58.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.100+0000 7f48b30fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:21:58.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.100+0000 7f48b30fe640 1 -- 172.21.15.40:0/4109145104 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48ac11e2f0 con 0x7f48ac070e00 2024-04-22T08:21:58.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.100+0000 7f48a97fa640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:21:58.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.100+0000 7f48a97fa640 1 -- 172.21.15.40:0/4109145104 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a003c790 con 0x7f48ac070e00 2024-04-22T08:21:58.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.100+0000 7f48a97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:21:58.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.100+0000 7f48a97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:21:58.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.100+0000 7f48a97fa640 10 monclient: dump: 2024-04-22T08:21:58.112 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:21:58.112 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:21:58.112 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:58.112 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:21:58.112 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:21:58.112 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:21:58.112 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:21:58.112 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:21:58.112 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:21:58.112 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:21:58.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.100+0000 7f48aaffd640 4 set_mon_vals no callback set 2024-04-22T08:21:58.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.100+0000 7f48b30fe640 10 monclient: _renew_subs 2024-04-22T08:21:58.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.100+0000 7f48b30fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:21:58.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.100+0000 7f48b30fe640 1 -- 172.21.15.40:0/4109145104 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48ac120650 con 0x7f48ac070e00 2024-04-22T08:21:58.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.102+0000 7f48a97fa640 1 -- 172.21.15.40:0/4109145104 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f48a003a040 con 0x7f48ac070e00 2024-04-22T08:21:58.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.102+0000 7f48b30fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:58.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.102+0000 7f48b30fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:58.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.102+0000 7f48b30fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:21:58.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.102+0000 7f48b30fe640 1 -- 172.21.15.40:0/4109145104 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48ac120120 con 0x7f48ac070e00 2024-04-22T08:21:58.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.102+0000 7f48a97fa640 1 --2- 172.21.15.40:0/4109145104 >> v2:172.21.15.189:6800/30168 conn(0x7f4890037780 0x7f4890039c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:21:58.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.102+0000 7f48a97fa640 1 -- 172.21.15.40:0/4109145104 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(311..311 src has 239..311) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f48a006bba0 con 0x7f48ac070e00 2024-04-22T08:21:58.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.102+0000 7f48abfff640 1 --2- 172.21.15.40:0/4109145104 >> v2:172.21.15.189:6800/30168 conn(0x7f4890037780 0x7f4890039c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:21:58.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.102+0000 7f48abfff640 10 monclient: get_auth_request con 0x7f4890037780 auth_method 0 2024-04-22T08:21:58.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.105+0000 7f48abfff640 1 --2- 172.21.15.40:0/4109145104 >> v2:172.21.15.189:6800/30168 conn(0x7f4890037780 0x7f4890039c40 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f48ac11def0 tx=0x7f4898009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:21:58.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.107+0000 7f48a97fa640 1 -- 172.21.15.40:0/4109145104 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f48a003a2f0 con 0x7f48ac070e00 2024-04-22T08:21:58.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.107+0000 7f48a97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:21:58.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.107+0000 7f48a97fa640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:21:58.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.362+0000 7f48b30fe640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:21:58.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.363+0000 7f48b30fe640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:21:58.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.363+0000 7f48b30fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:21:58.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.363+0000 7f48b30fe640 1 -- 172.21.15.40:0/4109145104 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f48ac09e840 con 0x7f48ac070e00 2024-04-22T08:21:58.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.363+0000 7f48a97fa640 1 -- 172.21.15.40:0/4109145104 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f48ac09e840 con 0x7f48ac070e00 2024-04-22T08:21:58.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.363+0000 7f48a97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:21:58.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.363+0000 7f48a97fa640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:21:58.363 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":92,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:21:58.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.365+0000 7f48b30fe640 1 -- 172.21.15.40:0/4109145104 >> v2:172.21.15.189:6800/30168 conn(0x7f4890037780 msgr2=0x7f4890039c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:58.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.365+0000 7f48b30fe640 1 --2- 172.21.15.40:0/4109145104 >> v2:172.21.15.189:6800/30168 conn(0x7f4890037780 0x7f4890039c40 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f48ac11def0 tx=0x7f4898009040 comp rx=0 tx=0).stop 2024-04-22T08:21:58.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.365+0000 7f48b30fe640 10 monclient: shutdown 2024-04-22T08:21:58.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.365+0000 7f48b30fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:21:58.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.365+0000 7f48b30fe640 1 -- 172.21.15.40:0/4109145104 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48ac070e00 msgr2=0x7f48ac1cab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:21:58.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.365+0000 7f48b30fe640 1 --2- 172.21.15.40:0/4109145104 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48ac070e00 0x7f48ac1cab20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f48a002b830 tx=0x7f48a00046c0 comp rx=0 tx=0).stop 2024-04-22T08:21:58.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.365+0000 7f48b30fe640 1 -- 172.21.15.40:0/4109145104 shutdown_connections 2024-04-22T08:21:58.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.365+0000 7f48b30fe640 1 --2- 172.21.15.40:0/4109145104 >> v2:172.21.15.189:6800/30168 conn(0x7f4890037780 0x7f4890039c40 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:58.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.365+0000 7f48b30fe640 1 --2- 172.21.15.40:0/4109145104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48ac0a0690 0x7f48ac1cb060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:58.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.365+0000 7f48b30fe640 1 --2- 172.21.15.40:0/4109145104 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48ac1360d0 0x7f48ac11c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:58.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.365+0000 7f48b30fe640 1 --2- 172.21.15.40:0/4109145104 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48ac070e00 0x7f48ac1cab20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:21:58.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.365+0000 7f48b30fe640 1 -- 172.21.15.40:0/4109145104 >> 172.21.15.40:0/4109145104 conn(0x7f48ac0a18f0 msgr2=0x7f48ac0a3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:21:58.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.365+0000 7f48b30fe640 1 -- 172.21.15.40:0/4109145104 shutdown_connections 2024-04-22T08:21:58.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:21:58.365+0000 7f48b30fe640 1 -- 172.21.15.40:0/4109145104 wait complete. 2024-04-22T08:22:01.376 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:22:01.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.487+0000 7f9dcccd5640 10 monclient: get_monmap_and_config 2024-04-22T08:22:01.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.487+0000 7f9dcccd5640 10 monclient: build_initial_monmap 2024-04-22T08:22:01.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.487+0000 7f9dcccd5640 10 monclient: monmap: 2024-04-22T08:22:01.487 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:01.487 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:01.487 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:22:01.488337+0000 2024-04-22T08:22:01.487 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:22:01.488337+0000 2024-04-22T08:22:01.487 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:01.487 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:01.487 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:22:01.487 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:22:01.487 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:22:01.487 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:22:01.488 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:22:01.488 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:01.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.488+0000 7f9dcccd5640 1 Processor -- start 2024-04-22T08:22:01.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.488+0000 7f9dcccd5640 1 -- start start 2024-04-22T08:22:01.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.488+0000 7f9dcccd5640 10 monclient: init 2024-04-22T08:22:01.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.488+0000 7f9dcccd5640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:01.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.488+0000 7f9dcccd5640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-04-22T08:22:01.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.488+0000 7f9dcccd5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9dc809f2e0 addr v1:172.21.15.40:6789/0 2024-04-22T08:22:01.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.488+0000 7f9dcccd5640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9dc809b760 0x7f9dc8137850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:01.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.488+0000 7f9dcccd5640 10 monclient(hunting): picked mon.noname-a con 0x7f9dc809b760 addr v2:172.21.15.40:3300/0 2024-04-22T08:22:01.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.488+0000 7f9dcccd5640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9dc809fdd0 0x7f9dc8139da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:01.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.488+0000 7f9dcccd5640 10 monclient(hunting): picked mon.noname-c con 0x7f9dc809fdd0 addr v2:172.21.15.189:3300/0 2024-04-22T08:22:01.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.488+0000 7f9dcccd5640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dc809cf70 con 0x7f9dc809f2e0 2024-04-22T08:22:01.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.488+0000 7f9dcccd5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:01.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.488+0000 7f9dcccd5640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dc809c6d0 con 0x7f9dc809b760 2024-04-22T08:22:01.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.488+0000 7f9dcccd5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:01.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.488+0000 7f9dcccd5640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dc809c9b0 con 0x7f9dc809fdd0 2024-04-22T08:22:01.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.488+0000 7f9dcccd5640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:01.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.488+0000 7f9dcccd5640 10 monclient(hunting): authenticate will time out at 1395.482463s 2024-04-22T08:22:01.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.488+0000 7f9dc659c640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f9dc809f2e0 0x7f9dc809f6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53262/0 (socket says 172.21.15.40:53262) 2024-04-22T08:22:01.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.488+0000 7f9dbdd9b640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9dc809b760 0x7f9dc8137850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:01.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.488+0000 7f9dc659c640 1 -- 172.21.15.40:0/686602472 learned_addr learned my addr 172.21.15.40:0/686602472 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:22:01.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.488+0000 7f9dbdd9b640 10 monclient(hunting): get_auth_request con 0x7f9dc809b760 auth_method 0 2024-04-22T08:22:01.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.488+0000 7f9dbdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:01.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.488+0000 7f9dbdd9b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:01.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.488+0000 7f9dbdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:01.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.488+0000 7f9dbdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:01.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.488+0000 7f9dbdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:01.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.489+0000 7f9dbdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:01.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.489+0000 7f9dc6d9d640 1 --2- 172.21.15.40:0/686602472 >> v2:172.21.15.189:3300/0 conn(0x7f9dc809fdd0 0x7f9dc8139da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:01.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.489+0000 7f9dc6d9d640 10 monclient(hunting): get_auth_request con 0x7f9dc809fdd0 auth_method 0 2024-04-22T08:22:01.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.489+0000 7f9dc6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:01.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.489+0000 7f9dc6d9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:01.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.489+0000 7f9dc6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:01.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.489+0000 7f9dbdd9b640 10 monclient(hunting): handle_auth_done global_id 114192 payload 274 2024-04-22T08:22:01.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.489+0000 7f9dbdd9b640 1 -- 172.21.15.40:0/686602472 >> v2:172.21.15.189:3300/0 conn(0x7f9dc809fdd0 msgr2=0x7f9dc8139da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:01.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.489+0000 7f9dbdd9b640 1 --2- 172.21.15.40:0/686602472 >> v2:172.21.15.189:3300/0 conn(0x7f9dc809fdd0 0x7f9dc8139da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:01.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.489+0000 7f9dbdd9b640 1 -- 172.21.15.40:0/686602472 >> v1:172.21.15.40:6789/0 conn(0x7f9dc809f2e0 legacy=0x7f9dc809f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:01.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.489+0000 7f9dbdd9b640 10 monclient: _finish_hunting 0 2024-04-22T08:22:01.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.489+0000 7f9dbdd9b640 1 monclient: found mon.noname-a 2024-04-22T08:22:01.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.489+0000 7f9dbdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:01.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.489+0000 7f9dbdd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:22:01.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.489+0000 7f9dbdd9b640 1 -- 172.21.15.40:0/686602472 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dc809cc90 con 0x7f9dc809b760 2024-04-22T08:22:01.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.492+0000 7f9dbdd9b640 1 -- 172.21.15.40:0/686602472 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9dc809b760 msgr2=0x7f9dc8137850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:01.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.492+0000 7f9dbdd9b640 1 --2- 172.21.15.40:0/686602472 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9dc809b760 0x7f9dc8137850 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9dc809d390 tx=0x7f9db002b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5df5f76406eb809 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:01.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.492+0000 7f9dc5d9b640 1 -- 172.21.15.40:0/686602472 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db0030920 con 0x7f9dc809b760 2024-04-22T08:22:01.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.492+0000 7f9dc5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:01.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.492+0000 7f9dc5d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:22:01.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.492+0000 7f9dc5d9b640 10 monclient: dump: 2024-04-22T08:22:01.499 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:01.499 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:01.499 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:01.499 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:01.499 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:01.499 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:01.499 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:01.499 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:01.499 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:01.499 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:01.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.492+0000 7f9dc5d9b640 10 monclient: _finish_auth 0 2024-04-22T08:22:01.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.492+0000 7f9dc5d9b640 10 monclient: _check_auth_tickets 2024-04-22T08:22:01.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.492+0000 7f9dc5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:01.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.492+0000 7f9dc5d9b640 1 -- 172.21.15.40:0/686602472 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9db002e050 con 0x7f9dc809b760 2024-04-22T08:22:01.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.492+0000 7f9dc5d9b640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:01.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.492+0000 7f9dc5d9b640 1 -- 172.21.15.40:0/686602472 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db0030ca0 con 0x7f9dc809b760 2024-04-22T08:22:01.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.492+0000 7f9dcccd5640 5 monclient: authenticate success, global_id 114192 2024-04-22T08:22:01.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.492+0000 7f9dcccd5640 10 monclient: get_monmap_and_config success 2024-04-22T08:22:01.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.492+0000 7f9dcccd5640 4 set_mon_vals no callback set 2024-04-22T08:22:01.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.492+0000 7f9dc5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:01.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.492+0000 7f9dc5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:22:01.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.492+0000 7f9dc5d9b640 10 monclient: dump: 2024-04-22T08:22:01.500 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:01.500 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:01.500 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:01.500 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:01.500 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:01.501 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:01.501 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:01.501 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:01.501 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:01.501 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:01.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.492+0000 7f9dcccd5640 10 monclient: shutdown 2024-04-22T08:22:01.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.492+0000 7f9dcccd5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:01.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.492+0000 7f9dcccd5640 1 -- 172.21.15.40:0/686602472 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9dc809b760 msgr2=0x7f9dc8137850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:01.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.492+0000 7f9dcccd5640 1 --2- 172.21.15.40:0/686602472 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9dc809b760 0x7f9dc8137850 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9dc809d390 tx=0x7f9db002b730 comp rx=0 tx=0).stop 2024-04-22T08:22:01.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.492+0000 7f9dcccd5640 1 -- 172.21.15.40:0/686602472 shutdown_connections 2024-04-22T08:22:01.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.492+0000 7f9dcccd5640 1 --2- 172.21.15.40:0/686602472 >> v2:172.21.15.189:3300/0 conn(0x7f9dc809fdd0 0x7f9dc8139da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:01.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.492+0000 7f9dcccd5640 1 --2- 172.21.15.40:0/686602472 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9dc809b760 0x7f9dc8137850 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:01.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.492+0000 7f9dcccd5640 1 -- 172.21.15.40:0/686602472 >> 172.21.15.40:0/686602472 conn(0x7f9dc80a0b50 msgr2=0x7f9dc80a0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:01.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.492+0000 7f9dcccd5640 1 -- 172.21.15.40:0/686602472 shutdown_connections 2024-04-22T08:22:01.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.492+0000 7f9dcccd5640 1 -- 172.21.15.40:0/686602472 wait complete. 2024-04-22T08:22:01.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.493+0000 7f9dcccd5640 10 monclient: build_initial_monmap 2024-04-22T08:22:01.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.493+0000 7f9dcccd5640 10 monclient: monmap: 2024-04-22T08:22:01.502 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:01.502 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:22:01.502 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:22:01.502 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:22:01.502 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:01.502 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:01.502 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:22:01.502 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:22:01.502 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:22:01.502 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:01.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.494+0000 7f9dcccd5640 1 Processor -- start 2024-04-22T08:22:01.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.494+0000 7f9dcccd5640 1 -- start start 2024-04-22T08:22:01.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.494+0000 7f9dcccd5640 10 monclient: init 2024-04-22T08:22:01.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.494+0000 7f9dcccd5640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:01.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.494+0000 7f9dcccd5640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-22T08:22:01.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.494+0000 7f9dcccd5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9dc809f2e0 0x7f9dc8135400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:01.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.494+0000 7f9dcccd5640 10 monclient(hunting): picked mon.noname-b con 0x7f9dc809f2e0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:22:01.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.494+0000 7f9dcccd5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9dc809fdd0 0x7f9dc81caa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:01.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.494+0000 7f9dcccd5640 10 monclient(hunting): picked mon.noname-a con 0x7f9dc809fdd0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:22:01.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.494+0000 7f9dcccd5640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9dc8135940 0x7f9dc81cafa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:01.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.494+0000 7f9dcccd5640 10 monclient(hunting): picked mon.noname-c con 0x7f9dc8135940 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:22:01.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.494+0000 7f9dcccd5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:01.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.494+0000 7f9dcccd5640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc8139410 con 0x7f9dc809fdd0 2024-04-22T08:22:01.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.494+0000 7f9dcccd5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:01.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.494+0000 7f9dcccd5640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc81cb4e0 con 0x7f9dc809f2e0 2024-04-22T08:22:01.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.494+0000 7f9dcccd5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:01.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.494+0000 7f9dcccd5640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc811c2a0 con 0x7f9dc8135940 2024-04-22T08:22:01.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.494+0000 7f9dcccd5640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:01.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.494+0000 7f9dcccd5640 10 monclient(hunting): authenticate will time out at 1395.488388s 2024-04-22T08:22:01.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.494+0000 7f9dc6d9d640 1 -- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9dc8135940 msgr2=0x7f9dc81cafa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:22:01.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.494+0000 7f9dc6d9d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9dc8135940 0x7f9dc81cafa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:22:01.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.494+0000 7f9dbdd9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9dc809fdd0 0x7f9dc81caa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:01.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.494+0000 7f9dbdd9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9dc809fdd0 0x7f9dc81caa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43840/0 (socket says 172.21.15.40:43840) 2024-04-22T08:22:01.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.494+0000 7f9dbdd9b640 1 -- 172.21.15.40:0/3756088384 learned_addr learned my addr 172.21.15.40:0/3756088384 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:22:01.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.494+0000 7f9dbdd9b640 10 monclient(hunting): get_auth_request con 0x7f9dc809fdd0 auth_method 0 2024-04-22T08:22:01.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.494+0000 7f9dbdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:01.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.494+0000 7f9dbdd9b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:01.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.494+0000 7f9dbdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:01.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.494+0000 7f9dbdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:01.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.494+0000 7f9dbdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:01.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.494+0000 7f9dbdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:01.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.494+0000 7f9dc659c640 1 --2- 172.21.15.40:0/3756088384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9dc809f2e0 0x7f9dc8135400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:01.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.494+0000 7f9dc659c640 10 monclient(hunting): get_auth_request con 0x7f9dc809f2e0 auth_method 0 2024-04-22T08:22:01.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.494+0000 7f9dc659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:01.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.494+0000 7f9dc659c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:01.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.494+0000 7f9dc659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:01.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.495+0000 7f9dbdd9b640 10 monclient(hunting): handle_auth_done global_id 114195 payload 824 2024-04-22T08:22:01.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.495+0000 7f9dbdd9b640 1 -- 172.21.15.40:0/3756088384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9dc8135940 msgr2=0x7f9dc81cafa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:22:01.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.495+0000 7f9dbdd9b640 1 --2- 172.21.15.40:0/3756088384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9dc8135940 0x7f9dc81cafa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:01.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.495+0000 7f9dbdd9b640 1 -- 172.21.15.40:0/3756088384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9dc809f2e0 msgr2=0x7f9dc8135400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:01.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.495+0000 7f9dbdd9b640 1 --2- 172.21.15.40:0/3756088384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9dc809f2e0 0x7f9dc8135400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:01.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.495+0000 7f9dbdd9b640 10 monclient: _finish_hunting 0 2024-04-22T08:22:01.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.495+0000 7f9dbdd9b640 1 monclient: found mon.noname-a 2024-04-22T08:22:01.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.495+0000 7f9dbdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:01.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.495+0000 7f9dbdd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:22:01.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.495+0000 7f9dbdd9b640 1 -- 172.21.15.40:0/3756088384 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9db0009400 con 0x7f9dc809fdd0 2024-04-22T08:22:01.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.495+0000 7f9dc659c640 1 --2- 172.21.15.40:0/3756088384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9dc809f2e0 0x7f9dc8135400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T08:22:01.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.497+0000 7f9dbdd9b640 1 -- 172.21.15.40:0/3756088384 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9dc809fdd0 msgr2=0x7f9dc81caa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:01.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.497+0000 7f9dbdd9b640 1 --2- 172.21.15.40:0/3756088384 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9dc809fdd0 0x7f9dc81caa60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9db0009890 tx=0x7f9db00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:01.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.497+0000 7f9dbf7fe640 1 -- 172.21.15.40:0/3756088384 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db003bae0 con 0x7f9dc809fdd0 2024-04-22T08:22:01.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.497+0000 7f9dbf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:01.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.497+0000 7f9dbf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:22:01.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.497+0000 7f9dbf7fe640 10 monclient: dump: 2024-04-22T08:22:01.506 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:01.506 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:01.506 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:01.506 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:01.506 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:01.506 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:01.506 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:01.507 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:01.507 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:01.507 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:01.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.497+0000 7f9dbf7fe640 10 monclient: _finish_auth 0 2024-04-22T08:22:01.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.497+0000 7f9dbf7fe640 10 monclient: _check_auth_tickets 2024-04-22T08:22:01.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.497+0000 7f9dbf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:01.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.497+0000 7f9dcccd5640 5 monclient: authenticate success, global_id 114195 2024-04-22T08:22:01.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.497+0000 7f9dbf7fe640 1 -- 172.21.15.40:0/3756088384 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9db003bc40 con 0x7f9dc809fdd0 2024-04-22T08:22:01.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.497+0000 7f9dbf7fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:01.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.497+0000 7f9dbf7fe640 1 -- 172.21.15.40:0/3756088384 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db0032400 con 0x7f9dc809fdd0 2024-04-22T08:22:01.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.497+0000 7f9dbf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:01.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.497+0000 7f9dbf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:22:01.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.497+0000 7f9dbf7fe640 10 monclient: dump: 2024-04-22T08:22:01.507 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:01.507 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:01.507 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:01.508 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:01.508 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:01.508 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:01.508 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:01.508 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:01.508 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:01.508 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:01.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.497+0000 7f9dcccd5640 10 monclient: _renew_subs 2024-04-22T08:22:01.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.497+0000 7f9dc559a640 4 set_mon_vals no callback set 2024-04-22T08:22:01.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.497+0000 7f9dcccd5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:01.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.497+0000 7f9dcccd5640 1 -- 172.21.15.40:0/3756088384 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dc811caa0 con 0x7f9dc809fdd0 2024-04-22T08:22:01.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.497+0000 7f9dcccd5640 10 monclient: _renew_subs 2024-04-22T08:22:01.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.497+0000 7f9dcccd5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:01.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.497+0000 7f9dcccd5640 1 -- 172.21.15.40:0/3756088384 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dc811eac0 con 0x7f9dc809fdd0 2024-04-22T08:22:01.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.498+0000 7f9dbf7fe640 1 -- 172.21.15.40:0/3756088384 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f9db0039040 con 0x7f9dc809fdd0 2024-04-22T08:22:01.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.498+0000 7f9dcccd5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:01.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.498+0000 7f9dcccd5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:01.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.498+0000 7f9dcccd5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:01.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.498+0000 7f9dcccd5640 1 -- 172.21.15.40:0/3756088384 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dc811ecb0 con 0x7f9dc809fdd0 2024-04-22T08:22:01.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.498+0000 7f9dbf7fe640 1 --2- 172.21.15.40:0/3756088384 >> v2:172.21.15.189:6800/30168 conn(0x7f9da8037780 0x7f9da8039c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:01.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.498+0000 7f9dbf7fe640 1 -- 172.21.15.40:0/3756088384 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(311..311 src has 239..311) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f9db006bb90 con 0x7f9dc809fdd0 2024-04-22T08:22:01.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.499+0000 7f9dc659c640 1 --2- 172.21.15.40:0/3756088384 >> v2:172.21.15.189:6800/30168 conn(0x7f9da8037780 0x7f9da8039c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:01.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.499+0000 7f9dc659c640 10 monclient: get_auth_request con 0x7f9da8037780 auth_method 0 2024-04-22T08:22:01.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.501+0000 7f9dc659c640 1 --2- 172.21.15.40:0/3756088384 >> v2:172.21.15.189:6800/30168 conn(0x7f9da8037780 0x7f9da8039c40 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f9db8005160 tx=0x7f9db8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:01.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.503+0000 7f9dbf7fe640 1 -- 172.21.15.40:0/3756088384 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f9db004dc80 con 0x7f9dc809fdd0 2024-04-22T08:22:01.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.503+0000 7f9dbf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:01.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.503+0000 7f9dbf7fe640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:22:01.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.751+0000 7f9dcccd5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:22:01.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.751+0000 7f9dcccd5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:22:01.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.751+0000 7f9dcccd5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:01.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.751+0000 7f9dcccd5640 1 -- 172.21.15.40:0/3756088384 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9dc804e9b0 con 0x7f9dc809fdd0 2024-04-22T08:22:01.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.752+0000 7f9dbf7fe640 1 -- 172.21.15.40:0/3756088384 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f9db0037070 con 0x7f9dc809fdd0 2024-04-22T08:22:01.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.752+0000 7f9dbf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:22:01.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.752+0000 7f9dbf7fe640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:22:01.752 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":92,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:22:01.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.754+0000 7f9dcccd5640 1 -- 172.21.15.40:0/3756088384 >> v2:172.21.15.189:6800/30168 conn(0x7f9da8037780 msgr2=0x7f9da8039c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:01.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.754+0000 7f9dcccd5640 1 --2- 172.21.15.40:0/3756088384 >> v2:172.21.15.189:6800/30168 conn(0x7f9da8037780 0x7f9da8039c40 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f9db8005160 tx=0x7f9db8009040 comp rx=0 tx=0).stop 2024-04-22T08:22:01.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.754+0000 7f9dcccd5640 10 monclient: shutdown 2024-04-22T08:22:01.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.754+0000 7f9dcccd5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:01.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.754+0000 7f9dcccd5640 1 -- 172.21.15.40:0/3756088384 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9dc809fdd0 msgr2=0x7f9dc81caa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:01.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.754+0000 7f9dcccd5640 1 --2- 172.21.15.40:0/3756088384 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9dc809fdd0 0x7f9dc81caa60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9db0009890 tx=0x7f9db00046c0 comp rx=0 tx=0).stop 2024-04-22T08:22:01.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.754+0000 7f9dcccd5640 1 -- 172.21.15.40:0/3756088384 shutdown_connections 2024-04-22T08:22:01.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.754+0000 7f9dcccd5640 1 --2- 172.21.15.40:0/3756088384 >> v2:172.21.15.189:6800/30168 conn(0x7f9da8037780 0x7f9da8039c40 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:01.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.754+0000 7f9dcccd5640 1 --2- 172.21.15.40:0/3756088384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9dc8135940 0x7f9dc81cafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:01.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.754+0000 7f9dcccd5640 1 --2- 172.21.15.40:0/3756088384 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9dc809fdd0 0x7f9dc81caa60 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:01.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.754+0000 7f9dcccd5640 1 --2- 172.21.15.40:0/3756088384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9dc809f2e0 0x7f9dc8135400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:01.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.754+0000 7f9dcccd5640 1 -- 172.21.15.40:0/3756088384 >> 172.21.15.40:0/3756088384 conn(0x7f9dc80a0b50 msgr2=0x7f9dc813dd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:01.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.754+0000 7f9dcccd5640 1 -- 172.21.15.40:0/3756088384 shutdown_connections 2024-04-22T08:22:01.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:01.754+0000 7f9dcccd5640 1 -- 172.21.15.40:0/3756088384 wait complete. 2024-04-22T08:22:04.764 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:22:04.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.875+0000 7f0248533640 10 monclient: get_monmap_and_config 2024-04-22T08:22:04.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.875+0000 7f0248533640 10 monclient: build_initial_monmap 2024-04-22T08:22:04.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.875+0000 7f0248533640 10 monclient: monmap: 2024-04-22T08:22:04.875 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:04.875 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:04.875 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:22:04.876446+0000 2024-04-22T08:22:04.875 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:22:04.876446+0000 2024-04-22T08:22:04.875 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:04.875 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:04.875 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:22:04.876 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:22:04.876 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:22:04.876 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:22:04.876 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:22:04.876 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:04.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.876+0000 7f0248533640 1 Processor -- start 2024-04-22T08:22:04.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.876+0000 7f0248533640 1 -- start start 2024-04-22T08:22:04.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.876+0000 7f0248533640 10 monclient: init 2024-04-22T08:22:04.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.876+0000 7f0248533640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:04.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.876+0000 7f0248533640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-04-22T08:22:04.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.877+0000 7f0248533640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f024009f2e0 addr v1:172.21.15.189:6789/0 2024-04-22T08:22:04.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.877+0000 7f0248533640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f024009b760 addr v1:172.21.15.40:6789/0 2024-04-22T08:22:04.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.877+0000 7f0248533640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f024009fdd0 0x7f024013af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:04.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.877+0000 7f0248533640 10 monclient(hunting): picked mon.noname-a con 0x7f024009fdd0 addr v2:172.21.15.40:3300/0 2024-04-22T08:22:04.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.877+0000 7f0248533640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f024009cf70 con 0x7f024009b760 2024-04-22T08:22:04.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.877+0000 7f0248533640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f024009c6d0 con 0x7f024009f2e0 2024-04-22T08:22:04.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.877+0000 7f0248533640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:04.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.877+0000 7f0248533640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f024009c9b0 con 0x7f024009fdd0 2024-04-22T08:22:04.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.877+0000 7f0248533640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:04.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.877+0000 7f0248533640 10 monclient(hunting): authenticate will time out at 1398.870891s 2024-04-22T08:22:04.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.877+0000 7f0246ad0640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f024009fdd0 0x7f024013af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:04.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.877+0000 7f0245ace640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f024009b760 0x7f0240137850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53270/0 (socket says 172.21.15.40:53270) 2024-04-22T08:22:04.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.877+0000 7f0246ad0640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f024009fdd0 0x7f024013af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43846/0 (socket says 172.21.15.40:43846) 2024-04-22T08:22:04.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.877+0000 7f0245ace640 1 -- 172.21.15.40:0/389377897 learned_addr learned my addr 172.21.15.40:0/389377897 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:22:04.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.877+0000 7f0246ad0640 10 monclient(hunting): get_auth_request con 0x7f024009fdd0 auth_method 0 2024-04-22T08:22:04.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.877+0000 7f0246ad0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:04.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.877+0000 7f0246ad0640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:04.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.877+0000 7f0246ad0640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:04.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.877+0000 7f0246ad0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:04.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.877+0000 7f0246ad0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:04.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.877+0000 7f0246ad0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:04.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.877+0000 7f0246ad0640 10 monclient(hunting): handle_auth_done global_id 114198 payload 274 2024-04-22T08:22:04.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.877+0000 7f0246ad0640 1 -- 172.21.15.40:0/389377897 >> v1:172.21.15.189:6789/0 conn(0x7f024009f2e0 legacy=0x7f024009f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:04.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.877+0000 7f0246ad0640 1 -- 172.21.15.40:0/389377897 >> v1:172.21.15.40:6789/0 conn(0x7f024009b760 legacy=0x7f0240137850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:04.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.877+0000 7f0246ad0640 10 monclient: _finish_hunting 0 2024-04-22T08:22:04.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.877+0000 7f0246ad0640 1 monclient: found mon.noname-a 2024-04-22T08:22:04.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.877+0000 7f0246ad0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:04.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.877+0000 7f0246ad0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:22:04.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.877+0000 7f0246ad0640 1 -- 172.21.15.40:0/389377897 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f024009cc90 con 0x7f024009fdd0 2024-04-22T08:22:04.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.880+0000 7f0246ad0640 1 -- 172.21.15.40:0/389377897 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f024009fdd0 msgr2=0x7f024013af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:04.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.880+0000 7f0246ad0640 1 --2- 172.21.15.40:0/389377897 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f024009fdd0 0x7f024013af80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f023800aea0 tx=0x7f023802ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd16d4a9176fa8c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:04.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.880+0000 7f02452cd640 1 -- 172.21.15.40:0/389377897 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f023802dbb0 con 0x7f024009fdd0 2024-04-22T08:22:04.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.880+0000 7f02452cd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:04.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.880+0000 7f02452cd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:22:04.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.880+0000 7f02452cd640 10 monclient: dump: 2024-04-22T08:22:04.887 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:04.887 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:04.887 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:04.887 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:04.887 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:04.887 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:04.887 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:04.887 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:04.887 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:04.888 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:04.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.880+0000 7f02452cd640 10 monclient: _finish_auth 0 2024-04-22T08:22:04.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.880+0000 7f02452cd640 10 monclient: _check_auth_tickets 2024-04-22T08:22:04.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.880+0000 7f02452cd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:04.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.880+0000 7f0248533640 5 monclient: authenticate success, global_id 114198 2024-04-22T08:22:04.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.880+0000 7f02452cd640 1 -- 172.21.15.40:0/389377897 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f023802dd10 con 0x7f024009fdd0 2024-04-22T08:22:04.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.880+0000 7f0248533640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:22:04.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.880+0000 7f02452cd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:04.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.880+0000 7f0248533640 10 monclient: get_monmap_and_config success 2024-04-22T08:22:04.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.880+0000 7f0248533640 4 set_mon_vals no callback set 2024-04-22T08:22:04.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.880+0000 7f02452cd640 1 -- 172.21.15.40:0/389377897 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0238008660 con 0x7f024009fdd0 2024-04-22T08:22:04.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.880+0000 7f02452cd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:04.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.880+0000 7f02452cd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:22:04.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.880+0000 7f02452cd640 10 monclient: dump: 2024-04-22T08:22:04.888 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:04.889 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:04.889 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:04.889 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:04.889 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:04.889 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:04.889 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:04.889 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:04.889 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:04.889 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:04.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.881+0000 7f0248533640 10 monclient: shutdown 2024-04-22T08:22:04.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.881+0000 7f0248533640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:04.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.881+0000 7f0248533640 1 -- 172.21.15.40:0/389377897 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f024009fdd0 msgr2=0x7f024013af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:04.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.881+0000 7f0248533640 1 --2- 172.21.15.40:0/389377897 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f024009fdd0 0x7f024013af80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f023800aea0 tx=0x7f023802ce00 comp rx=0 tx=0).stop 2024-04-22T08:22:04.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.881+0000 7f0248533640 1 -- 172.21.15.40:0/389377897 shutdown_connections 2024-04-22T08:22:04.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.881+0000 7f0248533640 1 --2- 172.21.15.40:0/389377897 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f024009fdd0 0x7f024013af80 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:04.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.881+0000 7f0248533640 1 -- 172.21.15.40:0/389377897 >> 172.21.15.40:0/389377897 conn(0x7f02400a0b50 msgr2=0x7f02400a0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:04.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.881+0000 7f0248533640 1 -- 172.21.15.40:0/389377897 shutdown_connections 2024-04-22T08:22:04.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.881+0000 7f0248533640 1 -- 172.21.15.40:0/389377897 wait complete. 2024-04-22T08:22:04.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.881+0000 7f0248533640 10 monclient: build_initial_monmap 2024-04-22T08:22:04.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.881+0000 7f0248533640 10 monclient: monmap: 2024-04-22T08:22:04.890 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:04.890 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:22:04.890 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:22:04.890 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:22:04.890 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:04.890 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:04.890 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:22:04.891 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:22:04.891 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:22:04.891 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:04.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.882+0000 7f0248533640 1 Processor -- start 2024-04-22T08:22:04.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.882+0000 7f0248533640 1 -- start start 2024-04-22T08:22:04.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.882+0000 7f0248533640 10 monclient: init 2024-04-22T08:22:04.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.882+0000 7f0248533640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:04.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.882+0000 7f0248533640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:22:04.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.882+0000 7f0248533640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f024009f2e0 0x7f02401c25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:04.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.882+0000 7f0248533640 10 monclient(hunting): picked mon.noname-a con 0x7f024009f2e0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:22:04.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.882+0000 7f0248533640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f024009fdd0 0x7f02401c8730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:04.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.882+0000 7f0248533640 10 monclient(hunting): picked mon.noname-b con 0x7f024009fdd0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:22:04.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.882+0000 7f0248533640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f02401c8c70 0x7f02401c51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:04.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.882+0000 7f0248533640 10 monclient(hunting): picked mon.noname-c con 0x7f02401c8c70 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:22:04.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.882+0000 7f0248533640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:04.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.882+0000 7f0248533640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02400711e0 con 0x7f024009f2e0 2024-04-22T08:22:04.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.882+0000 7f0248533640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:04.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.882+0000 7f0248533640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02401a7c20 con 0x7f024009fdd0 2024-04-22T08:22:04.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.882+0000 7f0248533640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:04.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.882+0000 7f0248533640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02401a9680 con 0x7f02401c8c70 2024-04-22T08:22:04.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.882+0000 7f0248533640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:04.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.882+0000 7f0248533640 10 monclient(hunting): authenticate will time out at 1398.876542s 2024-04-22T08:22:04.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.882+0000 7f0246ad0640 1 -- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f02401c8c70 msgr2=0x7f02401c51b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:22:04.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.882+0000 7f0246ad0640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f02401c8c70 0x7f02401c51b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:22:04.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.882+0000 7f02462cf640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f024009f2e0 0x7f02401c25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:04.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.882+0000 7f02462cf640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f024009f2e0 0x7f02401c25c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43860/0 (socket says 172.21.15.40:43860) 2024-04-22T08:22:04.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.882+0000 7f02462cf640 1 -- 172.21.15.40:0/1112099238 learned_addr learned my addr 172.21.15.40:0/1112099238 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:22:04.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.882+0000 7f02462cf640 10 monclient(hunting): get_auth_request con 0x7f024009f2e0 auth_method 0 2024-04-22T08:22:04.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.882+0000 7f02462cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:04.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.882+0000 7f02462cf640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:04.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.882+0000 7f02462cf640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:04.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.882+0000 7f02462cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:04.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.882+0000 7f02462cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:04.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.882+0000 7f02462cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:04.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.883+0000 7f0245ace640 1 --2- 172.21.15.40:0/1112099238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f024009fdd0 0x7f02401c8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:04.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.883+0000 7f02462cf640 10 monclient(hunting): handle_auth_done global_id 114201 payload 824 2024-04-22T08:22:04.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.883+0000 7f02462cf640 1 -- 172.21.15.40:0/1112099238 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f02401c8c70 msgr2=0x7f02401c51b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:22:04.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.883+0000 7f02462cf640 1 --2- 172.21.15.40:0/1112099238 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f02401c8c70 0x7f02401c51b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:04.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.883+0000 7f02462cf640 1 -- 172.21.15.40:0/1112099238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f024009fdd0 msgr2=0x7f02401c8730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:04.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.883+0000 7f02462cf640 1 --2- 172.21.15.40:0/1112099238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f024009fdd0 0x7f02401c8730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:04.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.883+0000 7f02462cf640 10 monclient: _finish_hunting 0 2024-04-22T08:22:04.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.883+0000 7f02462cf640 1 monclient: found mon.noname-a 2024-04-22T08:22:04.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.883+0000 7f02462cf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:04.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.883+0000 7f02462cf640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:22:04.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.883+0000 7f02462cf640 1 -- 172.21.15.40:0/1112099238 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f023800ab80 con 0x7f024009f2e0 2024-04-22T08:22:04.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.883+0000 7f0245ace640 10 monclient: get_auth_request con 0x7f024009fdd0 auth_method 0 2024-04-22T08:22:04.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.883+0000 7f0245ace640 1 --2- 172.21.15.40:0/1112099238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f024009fdd0 0x7f02401c8730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T08:22:04.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.885+0000 7f02462cf640 1 -- 172.21.15.40:0/1112099238 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f024009f2e0 msgr2=0x7f02401c25c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:04.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.885+0000 7f02462cf640 1 --2- 172.21.15.40:0/1112099238 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f024009f2e0 0x7f02401c25c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f023c00c430 tx=0x7f023c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:04.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.885+0000 7f0236ffd640 1 -- 172.21.15.40:0/1112099238 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f023c0119a0 con 0x7f024009f2e0 2024-04-22T08:22:04.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.885+0000 7f0236ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:04.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.885+0000 7f0236ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:22:04.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.885+0000 7f0236ffd640 10 monclient: dump: 2024-04-22T08:22:04.895 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:04.895 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:04.895 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:04.895 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:04.895 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:04.895 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:04.895 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:04.895 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:04.895 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:04.895 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:04.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.885+0000 7f0236ffd640 10 monclient: _finish_auth 0 2024-04-22T08:22:04.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.885+0000 7f0236ffd640 10 monclient: _check_auth_tickets 2024-04-22T08:22:04.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.885+0000 7f0236ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:04.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.885+0000 7f0236ffd640 1 -- 172.21.15.40:0/1112099238 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f023c011b00 con 0x7f024009f2e0 2024-04-22T08:22:04.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.885+0000 7f0236ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:04.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.885+0000 7f0236ffd640 1 -- 172.21.15.40:0/1112099238 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f023c012420 con 0x7f024009f2e0 2024-04-22T08:22:04.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.885+0000 7f0236ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:04.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.885+0000 7f0236ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:22:04.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.885+0000 7f0236ffd640 10 monclient: dump: 2024-04-22T08:22:04.896 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:04.896 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:04.896 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:04.896 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:04.896 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:04.896 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:04.896 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:04.896 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:04.896 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:04.897 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:04.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.885+0000 7f0248533640 5 monclient: authenticate success, global_id 114201 2024-04-22T08:22:04.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.885+0000 7f0248533640 10 monclient: _renew_subs 2024-04-22T08:22:04.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.885+0000 7f0248533640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:04.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.885+0000 7f0248533640 1 -- 172.21.15.40:0/1112099238 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02401a9960 con 0x7f024009f2e0 2024-04-22T08:22:04.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.885+0000 7f0244acc640 4 set_mon_vals no callback set 2024-04-22T08:22:04.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.885+0000 7f0248533640 10 monclient: _renew_subs 2024-04-22T08:22:04.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.885+0000 7f0248533640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:04.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.885+0000 7f0248533640 1 -- 172.21.15.40:0/1112099238 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02401c9120 con 0x7f024009f2e0 2024-04-22T08:22:04.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.886+0000 7f0236ffd640 1 -- 172.21.15.40:0/1112099238 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f023c012600 con 0x7f024009f2e0 2024-04-22T08:22:04.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.886+0000 7f0248533640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:04.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.886+0000 7f0248533640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:04.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.886+0000 7f0248533640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:04.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.886+0000 7f0248533640 1 -- 172.21.15.40:0/1112099238 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f024004e9b0 con 0x7f024009f2e0 2024-04-22T08:22:04.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.887+0000 7f0236ffd640 1 --2- 172.21.15.40:0/1112099238 >> v2:172.21.15.189:6800/30168 conn(0x7f02240377b0 0x7f0224039c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:04.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.887+0000 7f0236ffd640 1 -- 172.21.15.40:0/1112099238 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(311..311 src has 239..311) v4 ==== 3898+0+0 (secure 0 0 0) 0x7f023c04a480 con 0x7f024009f2e0 2024-04-22T08:22:04.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.887+0000 7f0245ace640 1 --2- 172.21.15.40:0/1112099238 >> v2:172.21.15.189:6800/30168 conn(0x7f02240377b0 0x7f0224039c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:04.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.887+0000 7f0245ace640 10 monclient: get_auth_request con 0x7f02240377b0 auth_method 0 2024-04-22T08:22:04.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.889+0000 7f0245ace640 1 --2- 172.21.15.40:0/1112099238 >> v2:172.21.15.189:6800/30168 conn(0x7f02240377b0 0x7f0224039c70 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f02400753c0 tx=0x7f0230008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:04.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.891+0000 7f0236ffd640 1 -- 172.21.15.40:0/1112099238 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f023c0106d0 con 0x7f024009f2e0 2024-04-22T08:22:04.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.891+0000 7f0236ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:04.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:04.891+0000 7f0236ffd640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:22:05.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:05.142+0000 7f0248533640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:22:05.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:05.142+0000 7f0248533640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:22:05.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:05.142+0000 7f0248533640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:05.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:05.142+0000 7f0248533640 1 -- 172.21.15.40:0/1112099238 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f02401a9fb0 con 0x7f024009f2e0 2024-04-22T08:22:05.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:05.142+0000 7f0236ffd640 1 -- 172.21.15.40:0/1112099238 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7f023c0102e0 con 0x7f024009f2e0 2024-04-22T08:22:05.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:05.142+0000 7f0236ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:22:05.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:05.142+0000 7f0236ffd640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:22:05.142 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":92,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:22:05.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:05.144+0000 7f0248533640 1 -- 172.21.15.40:0/1112099238 >> v2:172.21.15.189:6800/30168 conn(0x7f02240377b0 msgr2=0x7f0224039c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:05.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:05.144+0000 7f0248533640 1 --2- 172.21.15.40:0/1112099238 >> v2:172.21.15.189:6800/30168 conn(0x7f02240377b0 0x7f0224039c70 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f02400753c0 tx=0x7f0230008040 comp rx=0 tx=0).stop 2024-04-22T08:22:05.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:05.144+0000 7f0248533640 10 monclient: shutdown 2024-04-22T08:22:05.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:05.144+0000 7f0248533640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:05.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:05.144+0000 7f0248533640 1 -- 172.21.15.40:0/1112099238 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f024009f2e0 msgr2=0x7f02401c25c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:05.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:05.144+0000 7f0248533640 1 --2- 172.21.15.40:0/1112099238 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f024009f2e0 0x7f02401c25c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f023c00c430 tx=0x7f023c00c8f0 comp rx=0 tx=0).stop 2024-04-22T08:22:05.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:05.144+0000 7f0248533640 1 -- 172.21.15.40:0/1112099238 shutdown_connections 2024-04-22T08:22:05.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:05.144+0000 7f0248533640 1 --2- 172.21.15.40:0/1112099238 >> v2:172.21.15.189:6800/30168 conn(0x7f02240377b0 0x7f0224039c70 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:05.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:05.144+0000 7f0248533640 1 --2- 172.21.15.40:0/1112099238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f024009fdd0 0x7f02401c8730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:05.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:05.144+0000 7f0248533640 1 --2- 172.21.15.40:0/1112099238 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f02401c8c70 0x7f02401c51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:05.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:05.144+0000 7f0248533640 1 --2- 172.21.15.40:0/1112099238 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f024009f2e0 0x7f02401c25c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:05.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:05.144+0000 7f0248533640 1 -- 172.21.15.40:0/1112099238 >> 172.21.15.40:0/1112099238 conn(0x7f02400a0b50 msgr2=0x7f024006f300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:05.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:05.144+0000 7f0248533640 1 -- 172.21.15.40:0/1112099238 shutdown_connections 2024-04-22T08:22:05.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:05.144+0000 7f0248533640 1 -- 172.21.15.40:0/1112099238 wait complete. 2024-04-22T08:22:08.155 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:22:08.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.265+0000 7f7db6ebd640 10 monclient: get_monmap_and_config 2024-04-22T08:22:08.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.265+0000 7f7db6ebd640 10 monclient: build_initial_monmap 2024-04-22T08:22:08.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.265+0000 7f7db6ebd640 10 monclient: monmap: 2024-04-22T08:22:08.265 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:08.265 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:08.265 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:22:08.266668+0000 2024-04-22T08:22:08.265 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:22:08.266668+0000 2024-04-22T08:22:08.266 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:08.266 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:08.266 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:22:08.266 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:22:08.266 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:22:08.266 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:22:08.266 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:22:08.266 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:08.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.266+0000 7f7db6ebd640 1 Processor -- start 2024-04-22T08:22:08.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.266+0000 7f7db6ebd640 1 -- start start 2024-04-22T08:22:08.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.266+0000 7f7db6ebd640 10 monclient: init 2024-04-22T08:22:08.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.267+0000 7f7db6ebd640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:08.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.267+0000 7f7db6ebd640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-04-22T08:22:08.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.267+0000 7f7db6ebd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7db00a2560 addr v1:172.21.15.40:6789/0 2024-04-22T08:22:08.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.267+0000 7f7db6ebd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7db00a1110 addr v1:172.21.15.189:6789/0 2024-04-22T08:22:08.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.267+0000 7f7db6ebd640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7db00a15c0 0x7f7db013c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:08.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.267+0000 7f7db6ebd640 10 monclient(hunting): picked mon.noname-c con 0x7f7db00a15c0 addr v2:172.21.15.189:3300/0 2024-04-22T08:22:08.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.267+0000 7f7db6ebd640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7db00a2940 con 0x7f7db00a2560 2024-04-22T08:22:08.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.267+0000 7f7db6ebd640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7db00719f0 con 0x7f7db00a1110 2024-04-22T08:22:08.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.267+0000 7f7db6ebd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:08.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.267+0000 7f7db6ebd640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7db009ce90 con 0x7f7db00a15c0 2024-04-22T08:22:08.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.267+0000 7f7db6ebd640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:08.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.267+0000 7f7db6ebd640 10 monclient(hunting): authenticate will time out at 1402.261061s 2024-04-22T08:22:08.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.267+0000 7f7db4c59640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f7db00a2560 0x7f7db00a0a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53276/0 (socket says 172.21.15.40:53276) 2024-04-22T08:22:08.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.267+0000 7f7db4c59640 1 -- 172.21.15.40:0/1580972925 learned_addr learned my addr 172.21.15.40:0/1580972925 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:22:08.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.267+0000 7f7db545a640 1 --2- 172.21.15.40:0/1580972925 >> v2:172.21.15.189:3300/0 conn(0x7f7db00a15c0 0x7f7db013c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:08.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.267+0000 7f7db545a640 10 monclient(hunting): get_auth_request con 0x7f7db00a15c0 auth_method 0 2024-04-22T08:22:08.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.267+0000 7f7db545a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:08.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.267+0000 7f7db545a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:08.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.267+0000 7f7db545a640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:08.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.267+0000 7f7db545a640 1 -- 172.21.15.40:0/1580972925 >> v2:172.21.15.189:3300/0 conn(0x7f7db00a15c0 msgr2=0x7f7db013c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-22T08:22:08.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.267+0000 7f7db545a640 1 -- 172.21.15.40:0/1580972925 >> v2:172.21.15.189:3300/0 conn(0x7f7db00a15c0 msgr2=0x7f7db013c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:22:08.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.267+0000 7f7db545a640 1 --2- 172.21.15.40:0/1580972925 >> v2:172.21.15.189:3300/0 conn(0x7f7db00a15c0 0x7f7db013c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:22:08.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.267+0000 7f7db545a640 1 --2- 172.21.15.40:0/1580972925 >> v2:172.21.15.189:3300/0 conn(0x7f7db00a15c0 0x7f7db013c490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:22:08.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.269+0000 7f7db4c59640 1 -- 172.21.15.40:0/1580972925 >> v1:172.21.15.40:6789/0 conn(0x7f7db00a2560 legacy=0x7f7db00a0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:08.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.269+0000 7f7daffff640 1 -- 172.21.15.40:0/1580972925 >> v1:172.21.15.189:6789/0 conn(0x7f7db00a1110 legacy=0x7f7db0138d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:08.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.468+0000 7f7db545a640 1 --2- 172.21.15.40:0/1580972925 >> v2:172.21.15.189:3300/0 conn(0x7f7db00a15c0 0x7f7db013c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:08.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.468+0000 7f7db545a640 10 monclient(hunting): get_auth_request con 0x7f7db00a15c0 auth_method 0 2024-04-22T08:22:08.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.468+0000 7f7db545a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:08.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.468+0000 7f7db545a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:08.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.468+0000 7f7db545a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:22:08.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.468+0000 7f7db545a640 1 -- 172.21.15.40:0/1580972925 >> v2:172.21.15.189:3300/0 conn(0x7f7db00a15c0 msgr2=0x7f7db013c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-22T08:22:08.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.468+0000 7f7db545a640 1 -- 172.21.15.40:0/1580972925 >> v2:172.21.15.189:3300/0 conn(0x7f7db00a15c0 msgr2=0x7f7db013c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:22:08.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.468+0000 7f7db545a640 1 --2- 172.21.15.40:0/1580972925 >> v2:172.21.15.189:3300/0 conn(0x7f7db00a15c0 0x7f7db013c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:22:08.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.468+0000 7f7db545a640 1 --2- 172.21.15.40:0/1580972925 >> v2:172.21.15.189:3300/0 conn(0x7f7db00a15c0 0x7f7db013c490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:22:08.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.869+0000 7f7db545a640 1 --2- 172.21.15.40:0/1580972925 >> v2:172.21.15.189:3300/0 conn(0x7f7db00a15c0 0x7f7db013c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:08.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.869+0000 7f7db545a640 10 monclient(hunting): get_auth_request con 0x7f7db00a15c0 auth_method 0 2024-04-22T08:22:08.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.869+0000 7f7db545a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:08.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.869+0000 7f7db545a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:08.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.869+0000 7f7db545a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:22:08.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.869+0000 7f7db545a640 1 -- 172.21.15.40:0/1580972925 >> v2:172.21.15.189:3300/0 conn(0x7f7db00a15c0 msgr2=0x7f7db013c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-22T08:22:08.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.869+0000 7f7db545a640 1 -- 172.21.15.40:0/1580972925 >> v2:172.21.15.189:3300/0 conn(0x7f7db00a15c0 msgr2=0x7f7db013c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:22:08.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.869+0000 7f7db545a640 1 --2- 172.21.15.40:0/1580972925 >> v2:172.21.15.189:3300/0 conn(0x7f7db00a15c0 0x7f7db013c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:22:08.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:08.869+0000 7f7db545a640 1 --2- 172.21.15.40:0/1580972925 >> v2:172.21.15.189:3300/0 conn(0x7f7db00a15c0 0x7f7db013c490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:22:09.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:09.671+0000 7f7db545a640 1 --2- 172.21.15.40:0/1580972925 >> v2:172.21.15.189:3300/0 conn(0x7f7db00a15c0 0x7f7db013c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:09.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:09.671+0000 7f7db545a640 10 monclient(hunting): get_auth_request con 0x7f7db00a15c0 auth_method 0 2024-04-22T08:22:09.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:09.671+0000 7f7db545a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:09.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:09.671+0000 7f7db545a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:09.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:09.671+0000 7f7db545a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:22:09.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:09.671+0000 7f7db545a640 1 -- 172.21.15.40:0/1580972925 >> v2:172.21.15.189:3300/0 conn(0x7f7db00a15c0 msgr2=0x7f7db013c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-22T08:22:09.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:09.671+0000 7f7db545a640 1 -- 172.21.15.40:0/1580972925 >> v2:172.21.15.189:3300/0 conn(0x7f7db00a15c0 msgr2=0x7f7db013c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:22:09.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:09.671+0000 7f7db545a640 1 --2- 172.21.15.40:0/1580972925 >> v2:172.21.15.189:3300/0 conn(0x7f7db00a15c0 0x7f7db013c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:22:09.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:09.671+0000 7f7db545a640 1 --2- 172.21.15.40:0/1580972925 >> v2:172.21.15.189:3300/0 conn(0x7f7db00a15c0 0x7f7db013c490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T08:22:11.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.267+0000 7f7dae7fc640 10 monclient(hunting): tick 2024-04-22T08:22:11.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.267+0000 7f7dae7fc640 10 monclient(hunting): _check_auth_tickets 2024-04-22T08:22:11.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.267+0000 7f7dae7fc640 1 monclient(hunting): continuing hunt 2024-04-22T08:22:11.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.267+0000 7f7dae7fc640 10 monclient(hunting): _reopen_session rank -1 2024-04-22T08:22:11.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.267+0000 7f7dae7fc640 1 -- 172.21.15.40:0/1580972925 >> v2:172.21.15.189:3300/0 conn(0x7f7db00a15c0 msgr2=0x7f7db013c490 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:22:11.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.267+0000 7f7dae7fc640 1 --2- 172.21.15.40:0/1580972925 >> v2:172.21.15.189:3300/0 conn(0x7f7db00a15c0 0x7f7db013c490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:11.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.267+0000 7f7dae7fc640 1 -- 172.21.15.40:0/1580972925 >> v1:172.21.15.189:6789/0 conn(0x7f7db00a1110 legacy=0x7f7db0138d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:11.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.267+0000 7f7dae7fc640 1 -- 172.21.15.40:0/1580972925 >> v1:172.21.15.40:6789/0 conn(0x7f7db00a2560 legacy=0x7f7db00a0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:11.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.267+0000 7f7dae7fc640 10 monclient: _add_conns ranks=[2,3] 2024-04-22T08:22:11.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.267+0000 7f7dae7fc640 1 --2- 172.21.15.40:0/1580972925 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d94003810 0x7f7d94005c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:11.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.267+0000 7f7dae7fc640 10 monclient(hunting): picked mon.noname-b con 0x7f7d94003810 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:22:11.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.267+0000 7f7dae7fc640 1 --2- 172.21.15.40:0/1580972925 >> v2:172.21.15.40:3300/0 conn(0x7f7d94006390 0x7f7d94008850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:11.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.267+0000 7f7dae7fc640 10 monclient(hunting): picked mon.noname-a con 0x7f7d94006390 addr v2:172.21.15.40:3300/0 2024-04-22T08:22:11.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.267+0000 7f7dae7fc640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:11.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.267+0000 7f7dae7fc640 1 -- 172.21.15.40:0/1580972925 --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7db009ce90 con 0x7f7d94006390 2024-04-22T08:22:11.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.267+0000 7f7dae7fc640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:11.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.267+0000 7f7dae7fc640 1 -- 172.21.15.40:0/1580972925 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d94008f10 con 0x7f7d94003810 2024-04-22T08:22:11.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.267+0000 7f7dae7fc640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:11.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.267+0000 7f7db4c59640 1 -- 172.21.15.40:0/1580972925 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d94003810 msgr2=0x7f7d94005c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:22:11.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.267+0000 7f7db4c59640 1 --2- 172.21.15.40:0/1580972925 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d94003810 0x7f7d94005c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:22:11.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.267+0000 7f7daffff640 1 --2- 172.21.15.40:0/1580972925 >> v2:172.21.15.40:3300/0 conn(0x7f7d94006390 0x7f7d94008850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:11.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.267+0000 7f7daffff640 10 monclient(hunting): get_auth_request con 0x7f7d94006390 auth_method 0 2024-04-22T08:22:11.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.267+0000 7f7daffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:11.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.267+0000 7f7daffff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:11.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.267+0000 7f7daffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:11.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.267+0000 7f7daffff640 1 -- 172.21.15.40:0/1580972925 >> v2:172.21.15.40:3300/0 conn(0x7f7d94006390 msgr2=0x7f7d94008850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:22:11.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.267+0000 7f7daffff640 1 -- 172.21.15.40:0/1580972925 >> v2:172.21.15.40:3300/0 conn(0x7f7d94006390 msgr2=0x7f7d94008850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:22:11.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.267+0000 7f7daffff640 1 --2- 172.21.15.40:0/1580972925 >> v2:172.21.15.40:3300/0 conn(0x7f7d94006390 0x7f7d94008850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:22:11.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.267+0000 7f7daffff640 1 --2- 172.21.15.40:0/1580972925 >> v2:172.21.15.40:3300/0 conn(0x7f7d94006390 0x7f7d94008850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:22:11.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.468+0000 7f7db4c59640 1 -- 172.21.15.40:0/1580972925 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d94003810 msgr2=0x7f7d94005c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:22:11.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.468+0000 7f7db4c59640 1 --2- 172.21.15.40:0/1580972925 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d94003810 0x7f7d94005c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:22:11.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.468+0000 7f7daffff640 1 --2- 172.21.15.40:0/1580972925 >> v2:172.21.15.40:3300/0 conn(0x7f7d94006390 0x7f7d94008850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:11.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.468+0000 7f7daffff640 10 monclient(hunting): get_auth_request con 0x7f7d94006390 auth_method 0 2024-04-22T08:22:11.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.468+0000 7f7daffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:11.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.468+0000 7f7daffff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:11.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.468+0000 7f7daffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:22:11.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.468+0000 7f7daffff640 1 -- 172.21.15.40:0/1580972925 >> v2:172.21.15.40:3300/0 conn(0x7f7d94006390 msgr2=0x7f7d94008850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:22:11.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.468+0000 7f7daffff640 1 -- 172.21.15.40:0/1580972925 >> v2:172.21.15.40:3300/0 conn(0x7f7d94006390 msgr2=0x7f7d94008850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:22:11.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.468+0000 7f7daffff640 1 --2- 172.21.15.40:0/1580972925 >> v2:172.21.15.40:3300/0 conn(0x7f7d94006390 0x7f7d94008850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:22:11.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.468+0000 7f7daffff640 1 --2- 172.21.15.40:0/1580972925 >> v2:172.21.15.40:3300/0 conn(0x7f7d94006390 0x7f7d94008850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:22:11.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.869+0000 7f7db4c59640 1 -- 172.21.15.40:0/1580972925 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d94003810 msgr2=0x7f7d94005c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:22:11.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.869+0000 7f7db4c59640 1 --2- 172.21.15.40:0/1580972925 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d94003810 0x7f7d94005c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:22:11.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.869+0000 7f7daffff640 1 --2- 172.21.15.40:0/1580972925 >> v2:172.21.15.40:3300/0 conn(0x7f7d94006390 0x7f7d94008850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:11.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.869+0000 7f7daffff640 10 monclient(hunting): get_auth_request con 0x7f7d94006390 auth_method 0 2024-04-22T08:22:11.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.869+0000 7f7daffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:11.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.869+0000 7f7daffff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:11.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.869+0000 7f7daffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:22:11.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.869+0000 7f7daffff640 1 -- 172.21.15.40:0/1580972925 >> v2:172.21.15.40:3300/0 conn(0x7f7d94006390 msgr2=0x7f7d94008850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:22:11.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.869+0000 7f7daffff640 1 -- 172.21.15.40:0/1580972925 >> v2:172.21.15.40:3300/0 conn(0x7f7d94006390 msgr2=0x7f7d94008850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:22:11.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.869+0000 7f7daffff640 1 --2- 172.21.15.40:0/1580972925 >> v2:172.21.15.40:3300/0 conn(0x7f7d94006390 0x7f7d94008850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:22:11.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:11.869+0000 7f7daffff640 1 --2- 172.21.15.40:0/1580972925 >> v2:172.21.15.40:3300/0 conn(0x7f7d94006390 0x7f7d94008850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:22:12.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:12.670+0000 7f7db4c59640 1 -- 172.21.15.40:0/1580972925 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d94003810 msgr2=0x7f7d94005c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:22:12.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:12.670+0000 7f7db4c59640 1 --2- 172.21.15.40:0/1580972925 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d94003810 0x7f7d94005c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T08:22:12.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:12.670+0000 7f7daffff640 1 --2- 172.21.15.40:0/1580972925 >> v2:172.21.15.40:3300/0 conn(0x7f7d94006390 0x7f7d94008850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:12.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:12.670+0000 7f7daffff640 10 monclient(hunting): get_auth_request con 0x7f7d94006390 auth_method 0 2024-04-22T08:22:12.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:12.670+0000 7f7daffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:12.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:12.670+0000 7f7daffff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:12.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:12.670+0000 7f7daffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:22:12.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:12.670+0000 7f7daffff640 1 -- 172.21.15.40:0/1580972925 >> v2:172.21.15.40:3300/0 conn(0x7f7d94006390 msgr2=0x7f7d94008850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:22:12.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:12.670+0000 7f7daffff640 1 -- 172.21.15.40:0/1580972925 >> v2:172.21.15.40:3300/0 conn(0x7f7d94006390 msgr2=0x7f7d94008850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:22:12.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:12.670+0000 7f7daffff640 1 --2- 172.21.15.40:0/1580972925 >> v2:172.21.15.40:3300/0 conn(0x7f7d94006390 0x7f7d94008850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:22:12.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:12.670+0000 7f7daffff640 1 --2- 172.21.15.40:0/1580972925 >> v2:172.21.15.40:3300/0 conn(0x7f7d94006390 0x7f7d94008850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T08:22:13.289 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[39538]: starting. 2024-04-22T08:22:13.289 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[39538]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-39537 2024-04-22T08:22:13.289 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[39538]: created object 0... 2024-04-22T08:22:13.289 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[39538]: created object 25... 2024-04-22T08:22:13.289 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[39538]: created object 49... 2024-04-22T08:22:13.290 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[39538]: finishing. 2024-04-22T08:22:13.290 INFO:teuthology.orchestra.run.smithi189.stdout:process_1_[39538]: shutting down. 2024-04-22T08:22:13.930 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[39539]: starting. 2024-04-22T08:22:13.931 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[39539]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-39537 2024-04-22T08:22:13.931 INFO:teuthology.orchestra.run.smithi189.stdout:process_2_[39539]: shutting down. 2024-04-22T08:22:14.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.268+0000 7f7dae7fc640 10 monclient(hunting): tick 2024-04-22T08:22:14.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.268+0000 7f7dae7fc640 10 monclient(hunting): _check_auth_tickets 2024-04-22T08:22:14.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.268+0000 7f7dae7fc640 1 monclient(hunting): continuing hunt 2024-04-22T08:22:14.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.268+0000 7f7dae7fc640 10 monclient(hunting): _reopen_session rank -1 2024-04-22T08:22:14.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.268+0000 7f7dae7fc640 1 -- 172.21.15.40:0/1580972925 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d94003810 msgr2=0x7f7d94005c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:22:14.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.268+0000 7f7dae7fc640 1 --2- 172.21.15.40:0/1580972925 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d94003810 0x7f7d94005c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:14.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.268+0000 7f7dae7fc640 1 -- 172.21.15.40:0/1580972925 >> v2:172.21.15.40:3300/0 conn(0x7f7d94006390 msgr2=0x7f7d94008850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:22:14.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.268+0000 7f7dae7fc640 1 --2- 172.21.15.40:0/1580972925 >> v2:172.21.15.40:3300/0 conn(0x7f7d94006390 0x7f7d94008850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:14.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.268+0000 7f7dae7fc640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-04-22T08:22:14.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.268+0000 7f7db545a640 1 -- 172.21.15.40:0/1580972925 reap_dead start 2024-04-22T08:22:14.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.268+0000 7f7dae7fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7db00a1110 addr v1:172.21.15.189:6789/0 2024-04-22T08:22:14.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.268+0000 7f7dae7fc640 1 --2- 172.21.15.40:0/1580972925 >> v2:172.21.15.40:3300/0 conn(0x7f7d94003c50 0x7f7d94006840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:14.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.268+0000 7f7dae7fc640 10 monclient(hunting): picked mon.noname-a con 0x7f7d94003c50 addr v2:172.21.15.40:3300/0 2024-04-22T08:22:14.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.268+0000 7f7dae7fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7d94006d80 addr v1:172.21.15.40:6789/0 2024-04-22T08:22:14.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.268+0000 7f7dae7fc640 1 -- 172.21.15.40:0/1580972925 --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7db009d170 con 0x7f7d94006d80 2024-04-22T08:22:14.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.268+0000 7f7dae7fc640 1 -- 172.21.15.40:0/1580972925 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d94008a10 con 0x7f7db00a1110 2024-04-22T08:22:14.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.268+0000 7f7dae7fc640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:14.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.268+0000 7f7dae7fc640 1 -- 172.21.15.40:0/1580972925 --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7db009ce90 con 0x7f7d94003c50 2024-04-22T08:22:14.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.268+0000 7f7dae7fc640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:14.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.268+0000 7f7daffff640 1 --2- 172.21.15.40:0/1580972925 >> v2:172.21.15.40:3300/0 conn(0x7f7d94003c50 0x7f7d94006840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:14.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.268+0000 7f7daffff640 10 monclient(hunting): get_auth_request con 0x7f7d94003c50 auth_method 0 2024-04-22T08:22:14.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.268+0000 7f7daffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:14.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.268+0000 7f7daffff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:14.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.268+0000 7f7daffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:14.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.268+0000 7f7daffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:14.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.268+0000 7f7daffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:14.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.268+0000 7f7daffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:14.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.268+0000 7f7daffff640 10 monclient(hunting): handle_auth_done global_id 114216 payload 274 2024-04-22T08:22:14.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.268+0000 7f7daffff640 1 -- 172.21.15.40:0/1580972925 >> v1:172.21.15.189:6789/0 conn(0x7f7db00a1110 legacy=0x7f7d9400b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:14.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.268+0000 7f7daffff640 1 -- 172.21.15.40:0/1580972925 >> v1:172.21.15.40:6789/0 conn(0x7f7d94006d80 legacy=0x7f7d94007230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:14.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.268+0000 7f7daffff640 10 monclient: _finish_hunting 0 2024-04-22T08:22:14.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.268+0000 7f7daffff640 1 monclient: found mon.noname-a 2024-04-22T08:22:14.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.268+0000 7f7daffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:14.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.268+0000 7f7daffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:22:14.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.268+0000 7f7daffff640 1 -- 172.21.15.40:0/1580972925 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d94008be0 con 0x7f7d94003c50 2024-04-22T08:22:14.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.271+0000 7f7daffff640 1 -- 172.21.15.40:0/1580972925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d94003c50 msgr2=0x7f7d94006840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:14.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.271+0000 7f7daffff640 1 --2- 172.21.15.40:0/1580972925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d94003c50 0x7f7d94006840 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c007ab0 tx=0x7f7d9c02c1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=918ef84e3c66530b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:14.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.271+0000 7f7daf7fe640 1 -- 172.21.15.40:0/1580972925 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d9c02ce70 con 0x7f7d94003c50 2024-04-22T08:22:14.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.271+0000 7f7daf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:14.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.271+0000 7f7daf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:22:14.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.271+0000 7f7daf7fe640 10 monclient: dump: 2024-04-22T08:22:14.272 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:14.272 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:14.272 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:14.272 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:14.273 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:14.273 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:14.273 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:14.273 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:14.273 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:14.273 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:14.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.271+0000 7f7daf7fe640 10 monclient: _finish_auth 0 2024-04-22T08:22:14.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.271+0000 7f7daf7fe640 10 monclient: _check_auth_tickets 2024-04-22T08:22:14.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.271+0000 7f7daf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:14.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.271+0000 7f7daf7fe640 1 -- 172.21.15.40:0/1580972925 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d9c030070 con 0x7f7d94003c50 2024-04-22T08:22:14.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.271+0000 7f7daf7fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:14.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.271+0000 7f7daf7fe640 1 -- 172.21.15.40:0/1580972925 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d9c036660 con 0x7f7d94003c50 2024-04-22T08:22:14.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.271+0000 7f7db6ebd640 5 monclient: authenticate success, global_id 114216 2024-04-22T08:22:14.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.271+0000 7f7db6ebd640 10 monclient: get_monmap_and_config success 2024-04-22T08:22:14.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.271+0000 7f7db6ebd640 4 set_mon_vals no callback set 2024-04-22T08:22:14.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.271+0000 7f7daf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:14.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.271+0000 7f7daf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:22:14.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.271+0000 7f7daf7fe640 10 monclient: dump: 2024-04-22T08:22:14.274 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:14.274 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:14.274 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:14.274 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:14.274 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:14.274 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:14.274 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:14.274 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:14.274 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:14.274 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:14.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.272+0000 7f7db6ebd640 10 monclient: shutdown 2024-04-22T08:22:14.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.272+0000 7f7db6ebd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:14.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.272+0000 7f7db6ebd640 1 -- 172.21.15.40:0/1580972925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d94003c50 msgr2=0x7f7d94006840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:14.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.272+0000 7f7db6ebd640 1 --2- 172.21.15.40:0/1580972925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d94003c50 0x7f7d94006840 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c007ab0 tx=0x7f7d9c02c1e0 comp rx=0 tx=0).stop 2024-04-22T08:22:14.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.272+0000 7f7db6ebd640 1 -- 172.21.15.40:0/1580972925 shutdown_connections 2024-04-22T08:22:14.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.272+0000 7f7db6ebd640 1 --2- 172.21.15.40:0/1580972925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d94003c50 0x7f7d94006840 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:14.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.272+0000 7f7db6ebd640 1 -- 172.21.15.40:0/1580972925 >> 172.21.15.40:0/1580972925 conn(0x7f7db00a5960 msgr2=0x7f7db00a1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:14.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.272+0000 7f7db6ebd640 1 -- 172.21.15.40:0/1580972925 shutdown_connections 2024-04-22T08:22:14.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.272+0000 7f7db6ebd640 1 -- 172.21.15.40:0/1580972925 wait complete. 2024-04-22T08:22:14.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.273+0000 7f7db6ebd640 10 monclient: build_initial_monmap 2024-04-22T08:22:14.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.273+0000 7f7db6ebd640 10 monclient: monmap: 2024-04-22T08:22:14.279 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:14.279 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:22:14.279 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:22:14.279 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:22:14.279 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:14.279 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:14.279 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:22:14.279 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:22:14.280 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:22:14.280 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:14.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.273+0000 7f7db6ebd640 1 Processor -- start 2024-04-22T08:22:14.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.273+0000 7f7db6ebd640 1 -- start start 2024-04-22T08:22:14.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.274+0000 7f7db6ebd640 10 monclient: init 2024-04-22T08:22:14.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.274+0000 7f7db6ebd640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:14.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.274+0000 7f7db6ebd640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:22:14.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.274+0000 7f7db6ebd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d94003c50 0x7f7db01c8c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:14.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.274+0000 7f7db6ebd640 10 monclient(hunting): picked mon.noname-a con 0x7f7d94003c50 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:22:14.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.274+0000 7f7db6ebd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d94006d80 0x7f7db01c9170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:14.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.274+0000 7f7db6ebd640 10 monclient(hunting): picked mon.noname-b con 0x7f7d94006d80 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:22:14.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.274+0000 7f7db6ebd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7db00a1110 0x7f7db006fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:14.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.274+0000 7f7db6ebd640 10 monclient(hunting): picked mon.noname-c con 0x7f7db00a1110 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:22:14.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.274+0000 7f7db6ebd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:14.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.274+0000 7f7db6ebd640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7db011d980 con 0x7f7d94003c50 2024-04-22T08:22:14.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.274+0000 7f7db6ebd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:14.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.274+0000 7f7db6ebd640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7db011dbd0 con 0x7f7d94006d80 2024-04-22T08:22:14.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.274+0000 7f7db6ebd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:14.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.274+0000 7f7db6ebd640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7db011c3d0 con 0x7f7db00a1110 2024-04-22T08:22:14.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.274+0000 7f7db6ebd640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:14.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.274+0000 7f7db6ebd640 10 monclient(hunting): authenticate will time out at 1408.268555s 2024-04-22T08:22:14.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.274+0000 7f7db545a640 1 -- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7db00a1110 msgr2=0x7f7db006fee0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:22:14.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.274+0000 7f7db545a640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7db00a1110 0x7f7db006fee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:22:14.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.274+0000 7f7db4c59640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d94003c50 0x7f7db01c8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:14.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.274+0000 7f7db4c59640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d94003c50 0x7f7db01c8c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58156/0 (socket says 172.21.15.40:58156) 2024-04-22T08:22:14.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.275+0000 7f7db4c59640 1 -- 172.21.15.40:0/138707168 learned_addr learned my addr 172.21.15.40:0/138707168 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:22:14.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.275+0000 7f7db4c59640 10 monclient(hunting): get_auth_request con 0x7f7d94003c50 auth_method 0 2024-04-22T08:22:14.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.275+0000 7f7db4c59640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:14.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.275+0000 7f7db4c59640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:14.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.275+0000 7f7db4c59640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:14.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.275+0000 7f7db4c59640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:14.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.275+0000 7f7db4c59640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:14.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.275+0000 7f7db4c59640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:14.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.275+0000 7f7daffff640 1 --2- 172.21.15.40:0/138707168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d94006d80 0x7f7db01c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:14.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.275+0000 7f7daffff640 10 monclient(hunting): get_auth_request con 0x7f7d94006d80 auth_method 0 2024-04-22T08:22:14.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.275+0000 7f7daffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:14.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.275+0000 7f7daffff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:14.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.275+0000 7f7daffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:14.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.275+0000 7f7daffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:14.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.275+0000 7f7daffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:14.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.275+0000 7f7daffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:14.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.275+0000 7f7db4c59640 10 monclient(hunting): handle_auth_done global_id 114219 payload 824 2024-04-22T08:22:14.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.275+0000 7f7db4c59640 1 -- 172.21.15.40:0/138707168 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7db00a1110 msgr2=0x7f7db006fee0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:22:14.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.275+0000 7f7db4c59640 1 --2- 172.21.15.40:0/138707168 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7db00a1110 0x7f7db006fee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:14.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.275+0000 7f7db4c59640 1 -- 172.21.15.40:0/138707168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d94006d80 msgr2=0x7f7db01c9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:14.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.275+0000 7f7db4c59640 1 --2- 172.21.15.40:0/138707168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d94006d80 0x7f7db01c9170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:14.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.275+0000 7f7db4c59640 10 monclient: _finish_hunting 0 2024-04-22T08:22:14.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.275+0000 7f7db4c59640 1 monclient: found mon.noname-a 2024-04-22T08:22:14.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.275+0000 7f7db4c59640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:14.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.275+0000 7f7db4c59640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:22:14.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.275+0000 7f7db4c59640 1 -- 172.21.15.40:0/138707168 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d9c00f040 con 0x7f7d94003c50 2024-04-22T08:22:14.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.277+0000 7f7db4c59640 1 -- 172.21.15.40:0/138707168 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d94003c50 msgr2=0x7f7db01c8c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:14.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.277+0000 7f7db4c59640 1 --2- 172.21.15.40:0/138707168 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d94003c50 0x7f7db01c8c30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7da400c750 tx=0x7f7da400cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:14.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.278+0000 7f7dad7fa640 1 -- 172.21.15.40:0/138707168 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da4005ae0 con 0x7f7d94003c50 2024-04-22T08:22:14.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.278+0000 7f7dad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:14.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.278+0000 7f7dad7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:22:14.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.278+0000 7f7dad7fa640 10 monclient: dump: 2024-04-22T08:22:14.284 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:14.284 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:14.284 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:14.284 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:14.284 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:14.284 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:14.284 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:14.284 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:14.284 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:14.284 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:14.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.278+0000 7f7dad7fa640 10 monclient: _finish_auth 0 2024-04-22T08:22:14.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.278+0000 7f7dad7fa640 10 monclient: _check_auth_tickets 2024-04-22T08:22:14.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.278+0000 7f7dad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:14.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.278+0000 7f7dad7fa640 1 -- 172.21.15.40:0/138707168 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7da4005c40 con 0x7f7d94003c50 2024-04-22T08:22:14.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.278+0000 7f7db6ebd640 5 monclient: authenticate success, global_id 114219 2024-04-22T08:22:14.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.278+0000 7f7db6ebd640 10 monclient: _renew_subs 2024-04-22T08:22:14.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.278+0000 7f7db6ebd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:14.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.278+0000 7f7db6ebd640 1 -- 172.21.15.40:0/138707168 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7db0130a90 con 0x7f7d94003c50 2024-04-22T08:22:14.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.278+0000 7f7dad7fa640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:14.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.278+0000 7f7dad7fa640 1 -- 172.21.15.40:0/138707168 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da4007070 con 0x7f7d94003c50 2024-04-22T08:22:14.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.278+0000 7f7dad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:14.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.278+0000 7f7dad7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:22:14.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.278+0000 7f7dad7fa640 10 monclient: dump: 2024-04-22T08:22:14.285 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:14.285 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:14.285 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:14.286 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:14.286 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:14.286 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:14.286 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:14.286 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:14.286 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:14.286 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:14.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.278+0000 7f7db6ebd640 10 monclient: _renew_subs 2024-04-22T08:22:14.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.278+0000 7f7dae7fc640 4 set_mon_vals no callback set 2024-04-22T08:22:14.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.278+0000 7f7db6ebd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:14.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.278+0000 7f7db6ebd640 1 -- 172.21.15.40:0/138707168 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7db011b800 con 0x7f7d94003c50 2024-04-22T08:22:14.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.279+0000 7f7dad7fa640 1 -- 172.21.15.40:0/138707168 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f7da40071d0 con 0x7f7d94003c50 2024-04-22T08:22:14.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.279+0000 7f7dad7fa640 1 --2- 172.21.15.40:0/138707168 >> v2:172.21.15.189:6800/30168 conn(0x7f7d94039310 0x7f7d9403b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:14.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.279+0000 7f7db6ebd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:14.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.279+0000 7f7dad7fa640 1 -- 172.21.15.40:0/138707168 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(313..313 src has 239..313) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f7da40496b0 con 0x7f7d94003c50 2024-04-22T08:22:14.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.279+0000 7f7db6ebd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:14.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.279+0000 7f7db6ebd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:14.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.279+0000 7f7db6ebd640 1 -- 172.21.15.40:0/138707168 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7db011d730 con 0x7f7d94003c50 2024-04-22T08:22:14.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.279+0000 7f7daffff640 1 --2- 172.21.15.40:0/138707168 >> v2:172.21.15.189:6800/30168 conn(0x7f7d94039310 0x7f7d9403b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:14.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.280+0000 7f7daffff640 10 monclient: get_auth_request con 0x7f7d94039310 auth_method 0 2024-04-22T08:22:14.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.282+0000 7f7daffff640 1 --2- 172.21.15.40:0/138707168 >> v2:172.21.15.189:6800/30168 conn(0x7f7d94039310 0x7f7d9403b7d0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c02cfd0 tx=0x7f7d9c032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:14.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.284+0000 7f7dad7fa640 1 -- 172.21.15.40:0/138707168 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f7da4017070 con 0x7f7d94003c50 2024-04-22T08:22:14.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.284+0000 7f7dad7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:14.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.284+0000 7f7dad7fa640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:22:14.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.541+0000 7f7db6ebd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:22:14.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.541+0000 7f7db6ebd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:22:14.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.541+0000 7f7db6ebd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:14.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.541+0000 7f7db6ebd640 1 -- 172.21.15.40:0/138707168 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7db004e9b0 con 0x7f7d94003c50 2024-04-22T08:22:14.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.541+0000 7f7dad7fa640 1 -- 172.21.15.40:0/138707168 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1537 (secure 0 0 0) 0x7f7da403b620 con 0x7f7d94003c50 2024-04-22T08:22:14.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.541+0000 7f7dad7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:22:14.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.541+0000 7f7dad7fa640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:22:14.542 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":96,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:22:14.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.543+0000 7f7db6ebd640 1 -- 172.21.15.40:0/138707168 >> v2:172.21.15.189:6800/30168 conn(0x7f7d94039310 msgr2=0x7f7d9403b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:14.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.543+0000 7f7db6ebd640 1 --2- 172.21.15.40:0/138707168 >> v2:172.21.15.189:6800/30168 conn(0x7f7d94039310 0x7f7d9403b7d0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c02cfd0 tx=0x7f7d9c032000 comp rx=0 tx=0).stop 2024-04-22T08:22:14.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.543+0000 7f7db6ebd640 10 monclient: shutdown 2024-04-22T08:22:14.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.543+0000 7f7db6ebd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:14.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.543+0000 7f7db6ebd640 1 -- 172.21.15.40:0/138707168 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d94003c50 msgr2=0x7f7db01c8c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:14.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.543+0000 7f7db6ebd640 1 --2- 172.21.15.40:0/138707168 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d94003c50 0x7f7db01c8c30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7da400c750 tx=0x7f7da400cc10 comp rx=0 tx=0).stop 2024-04-22T08:22:14.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.543+0000 7f7db6ebd640 1 -- 172.21.15.40:0/138707168 shutdown_connections 2024-04-22T08:22:14.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.543+0000 7f7db6ebd640 1 --2- 172.21.15.40:0/138707168 >> v2:172.21.15.189:6800/30168 conn(0x7f7d94039310 0x7f7d9403b7d0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:14.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.543+0000 7f7db6ebd640 1 --2- 172.21.15.40:0/138707168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d94006d80 0x7f7db01c9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:14.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.543+0000 7f7db6ebd640 1 --2- 172.21.15.40:0/138707168 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7db00a1110 0x7f7db006fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:14.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.543+0000 7f7db6ebd640 1 --2- 172.21.15.40:0/138707168 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d94003c50 0x7f7db01c8c30 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:14.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.543+0000 7f7db6ebd640 1 -- 172.21.15.40:0/138707168 >> 172.21.15.40:0/138707168 conn(0x7f7db00a5960 msgr2=0x7f7db00a2df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:14.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.543+0000 7f7db6ebd640 1 -- 172.21.15.40:0/138707168 shutdown_connections 2024-04-22T08:22:14.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.543+0000 7f7db6ebd640 1 -- 172.21.15.40:0/138707168 wait complete. 2024-04-22T08:22:14.553 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-22T08:22:14.554 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-04-22T08:22:14.554 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-22T08:22:14.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.653+0000 7f3b83963640 10 monclient: get_monmap_and_config 2024-04-22T08:22:14.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.653+0000 7f3b83963640 10 monclient: build_initial_monmap 2024-04-22T08:22:14.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.653+0000 7f3b83963640 10 monclient: monmap: 2024-04-22T08:22:14.653 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:14.653 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:14.653 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:22:14.654586+0000 2024-04-22T08:22:14.653 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:22:14.654586+0000 2024-04-22T08:22:14.653 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:14.654 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:14.654 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:22:14.654 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:22:14.654 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:22:14.654 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:22:14.654 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:22:14.654 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:14.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.654+0000 7f3b83963640 1 Processor -- start 2024-04-22T08:22:14.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.654+0000 7f3b83963640 1 -- start start 2024-04-22T08:22:14.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.654+0000 7f3b83963640 10 monclient: init 2024-04-22T08:22:14.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.655+0000 7f3b83963640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:14.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.655+0000 7f3b83963640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-04-22T08:22:14.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.655+0000 7f3b83963640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3b7c070e90 addr v1:172.21.15.40:6789/0 2024-04-22T08:22:14.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.655+0000 7f3b83963640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3b7c071270 0x7f3b7c13ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:14.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.655+0000 7f3b83963640 10 monclient(hunting): picked mon.noname-b con 0x7f3b7c071270 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:22:14.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.655+0000 7f3b83963640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3b7c13b2d0 0x7f3b7c13d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:14.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.655+0000 7f3b83963640 10 monclient(hunting): picked mon.noname-a con 0x7f3b7c13b2d0 addr v2:172.21.15.40:3300/0 2024-04-22T08:22:14.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.655+0000 7f3b83963640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b7c136940 con 0x7f3b7c070e90 2024-04-22T08:22:14.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.655+0000 7f3b83963640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:14.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.655+0000 7f3b83963640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b7c06d170 con 0x7f3b7c13b2d0 2024-04-22T08:22:14.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.655+0000 7f3b83963640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:14.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.655+0000 7f3b83963640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b7c136320 con 0x7f3b7c071270 2024-04-22T08:22:14.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.655+0000 7f3b83963640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:14.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.655+0000 7f3b83963640 10 monclient(hunting): authenticate will time out at 1408.648977s 2024-04-22T08:22:14.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.655+0000 7f3b80efe640 1 -- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3b7c071270 msgr2=0x7f3b7c13ad90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:22:14.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.655+0000 7f3b80efe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3b7c071270 0x7f3b7c13ad90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:22:14.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.655+0000 7f3b816ff640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f3b7c070e90 0x7f3b7c135c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41934/0 (socket says 172.21.15.40:41934) 2024-04-22T08:22:14.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.655+0000 7f3b816ff640 1 -- 172.21.15.40:0/3098206783 learned_addr learned my addr 172.21.15.40:0/3098206783 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:22:14.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.655+0000 7f3b81f00640 1 --2- 172.21.15.40:0/3098206783 >> v2:172.21.15.40:3300/0 conn(0x7f3b7c13b2d0 0x7f3b7c13d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:14.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.655+0000 7f3b81f00640 10 monclient(hunting): get_auth_request con 0x7f3b7c13b2d0 auth_method 0 2024-04-22T08:22:14.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.655+0000 7f3b81f00640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:14.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.655+0000 7f3b81f00640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:14.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.655+0000 7f3b81f00640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:14.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.655+0000 7f3b81f00640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:14.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.655+0000 7f3b81f00640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:14.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.655+0000 7f3b81f00640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:14.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.655+0000 7f3b81f00640 10 monclient(hunting): handle_auth_done global_id 114222 payload 274 2024-04-22T08:22:14.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.655+0000 7f3b81f00640 1 -- 172.21.15.40:0/3098206783 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3b7c071270 msgr2=0x7f3b7c13ad90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:22:14.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.655+0000 7f3b81f00640 1 --2- 172.21.15.40:0/3098206783 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3b7c071270 0x7f3b7c13ad90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:14.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.655+0000 7f3b81f00640 1 -- 172.21.15.40:0/3098206783 >> v1:172.21.15.40:6789/0 conn(0x7f3b7c070e90 legacy=0x7f3b7c135c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:14.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.655+0000 7f3b81f00640 10 monclient: _finish_hunting 0 2024-04-22T08:22:14.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.655+0000 7f3b81f00640 1 monclient: found mon.noname-a 2024-04-22T08:22:14.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.655+0000 7f3b81f00640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:14.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.655+0000 7f3b81f00640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:22:14.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.655+0000 7f3b81f00640 1 -- 172.21.15.40:0/3098206783 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b7c06d450 con 0x7f3b7c13b2d0 2024-04-22T08:22:14.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.658+0000 7f3b81f00640 1 -- 172.21.15.40:0/3098206783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b7c13b2d0 msgr2=0x7f3b7c13d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:14.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.658+0000 7f3b81f00640 1 --2- 172.21.15.40:0/3098206783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b7c13b2d0 0x7f3b7c13d6c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3b74009d90 tx=0x7f3b7402bd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ae84a572531a7a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:14.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.658+0000 7f3b73fff640 1 -- 172.21.15.40:0/3098206783 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b7402cdc0 con 0x7f3b7c13b2d0 2024-04-22T08:22:14.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.658+0000 7f3b73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:14.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.658+0000 7f3b73fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:22:14.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.658+0000 7f3b73fff640 10 monclient: dump: 2024-04-22T08:22:14.666 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:14.666 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:14.666 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:14.666 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:14.666 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:14.666 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:14.666 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:14.666 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:14.666 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:14.667 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:14.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.658+0000 7f3b73fff640 10 monclient: _finish_auth 0 2024-04-22T08:22:14.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.658+0000 7f3b73fff640 10 monclient: _check_auth_tickets 2024-04-22T08:22:14.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.658+0000 7f3b73fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:14.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.658+0000 7f3b73fff640 1 -- 172.21.15.40:0/3098206783 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b74034440 con 0x7f3b7c13b2d0 2024-04-22T08:22:14.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.658+0000 7f3b83963640 5 monclient: authenticate success, global_id 114222 2024-04-22T08:22:14.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.658+0000 7f3b83963640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:22:14.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.658+0000 7f3b73fff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:14.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.658+0000 7f3b83963640 10 monclient: get_monmap_and_config success 2024-04-22T08:22:14.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.658+0000 7f3b83963640 4 set_mon_vals no callback set 2024-04-22T08:22:14.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.658+0000 7f3b73fff640 1 -- 172.21.15.40:0/3098206783 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b740347d0 con 0x7f3b7c13b2d0 2024-04-22T08:22:14.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.658+0000 7f3b73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:14.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.658+0000 7f3b73fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:22:14.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.658+0000 7f3b73fff640 10 monclient: dump: 2024-04-22T08:22:14.668 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:14.668 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:14.668 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:14.668 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:14.668 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:14.668 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:14.668 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:14.668 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:14.668 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:14.668 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:14.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.659+0000 7f3b83963640 10 monclient: shutdown 2024-04-22T08:22:14.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.659+0000 7f3b83963640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:14.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.659+0000 7f3b83963640 1 -- 172.21.15.40:0/3098206783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b7c13b2d0 msgr2=0x7f3b7c13d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:14.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.659+0000 7f3b83963640 1 --2- 172.21.15.40:0/3098206783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b7c13b2d0 0x7f3b7c13d6c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3b74009d90 tx=0x7f3b7402bd20 comp rx=0 tx=0).stop 2024-04-22T08:22:14.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.659+0000 7f3b83963640 1 -- 172.21.15.40:0/3098206783 shutdown_connections 2024-04-22T08:22:14.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.659+0000 7f3b83963640 1 --2- 172.21.15.40:0/3098206783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b7c13b2d0 0x7f3b7c13d6c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:14.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.659+0000 7f3b83963640 1 --2- 172.21.15.40:0/3098206783 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3b7c071270 0x7f3b7c13ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:14.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.659+0000 7f3b83963640 1 -- 172.21.15.40:0/3098206783 >> 172.21.15.40:0/3098206783 conn(0x7f3b7c0a1780 msgr2=0x7f3b7c0a1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:14.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.659+0000 7f3b83963640 1 -- 172.21.15.40:0/3098206783 shutdown_connections 2024-04-22T08:22:14.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.659+0000 7f3b83963640 1 -- 172.21.15.40:0/3098206783 wait complete. 2024-04-22T08:22:14.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.659+0000 7f3b83963640 10 monclient: build_initial_monmap 2024-04-22T08:22:14.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.659+0000 7f3b83963640 10 monclient: monmap: 2024-04-22T08:22:14.669 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:14.669 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:22:14.669 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:22:14.669 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:22:14.669 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:14.670 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:14.670 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:22:14.670 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:22:14.670 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:22:14.670 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:14.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.660+0000 7f3b83963640 1 Processor -- start 2024-04-22T08:22:14.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.660+0000 7f3b83963640 1 -- start start 2024-04-22T08:22:14.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.660+0000 7f3b83963640 10 monclient: init 2024-04-22T08:22:14.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.660+0000 7f3b83963640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:14.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.660+0000 7f3b83963640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-22T08:22:14.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.660+0000 7f3b83963640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3b7c070e90 0x7f3b7c1ca850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:14.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.660+0000 7f3b83963640 10 monclient(hunting): picked mon.noname-c con 0x7f3b7c070e90 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:22:14.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.660+0000 7f3b83963640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b7c071270 0x7f3b7c12e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:14.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.660+0000 7f3b83963640 10 monclient(hunting): picked mon.noname-a con 0x7f3b7c071270 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:22:14.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.660+0000 7f3b83963640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b7c13b2d0 0x7f3b7c12e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:14.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.660+0000 7f3b83963640 10 monclient(hunting): picked mon.noname-b con 0x7f3b7c13b2d0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:22:14.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.660+0000 7f3b816ff640 1 -- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3b7c070e90 msgr2=0x7f3b7c1ca850 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:22:14.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.660+0000 7f3b83963640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:14.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.660+0000 7f3b83963640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b7c138f80 con 0x7f3b7c071270 2024-04-22T08:22:14.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.660+0000 7f3b83963640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:14.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.660+0000 7f3b83963640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b7c0a40d0 con 0x7f3b7c13b2d0 2024-04-22T08:22:14.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.660+0000 7f3b83963640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:14.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.660+0000 7f3b83963640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b7c0a08e0 con 0x7f3b7c070e90 2024-04-22T08:22:14.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.660+0000 7f3b83963640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:14.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.660+0000 7f3b816ff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3b7c070e90 0x7f3b7c1ca850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:22:14.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.660+0000 7f3b83963640 10 monclient(hunting): authenticate will time out at 1408.654432s 2024-04-22T08:22:14.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.660+0000 7f3b80efe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b7c071270 0x7f3b7c12e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:14.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.660+0000 7f3b80efe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b7c071270 0x7f3b7c12e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58174/0 (socket says 172.21.15.40:58174) 2024-04-22T08:22:14.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.660+0000 7f3b80efe640 1 -- 172.21.15.40:0/4116893417 learned_addr learned my addr 172.21.15.40:0/4116893417 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:22:14.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.660+0000 7f3b80efe640 10 monclient(hunting): get_auth_request con 0x7f3b7c071270 auth_method 0 2024-04-22T08:22:14.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.660+0000 7f3b80efe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:14.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.660+0000 7f3b80efe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:14.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.660+0000 7f3b80efe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:14.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.660+0000 7f3b80efe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:14.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.660+0000 7f3b80efe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:14.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.660+0000 7f3b80efe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:14.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.660+0000 7f3b80efe640 10 monclient(hunting): handle_auth_done global_id 114225 payload 824 2024-04-22T08:22:14.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.661+0000 7f3b80efe640 1 -- 172.21.15.40:0/4116893417 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3b7c070e90 msgr2=0x7f3b7c1ca850 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:22:14.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.661+0000 7f3b80efe640 1 --2- 172.21.15.40:0/4116893417 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3b7c070e90 0x7f3b7c1ca850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:14.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.661+0000 7f3b80efe640 1 -- 172.21.15.40:0/4116893417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b7c13b2d0 msgr2=0x7f3b7c12e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:14.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.661+0000 7f3b80efe640 1 --2- 172.21.15.40:0/4116893417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b7c13b2d0 0x7f3b7c12e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:14.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.661+0000 7f3b80efe640 10 monclient: _finish_hunting 0 2024-04-22T08:22:14.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.661+0000 7f3b80efe640 1 monclient: found mon.noname-a 2024-04-22T08:22:14.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.661+0000 7f3b80efe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:14.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.661+0000 7f3b80efe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:22:14.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.661+0000 7f3b80efe640 1 -- 172.21.15.40:0/4116893417 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b74009a70 con 0x7f3b7c071270 2024-04-22T08:22:14.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.663+0000 7f3b80efe640 1 -- 172.21.15.40:0/4116893417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b7c071270 msgr2=0x7f3b7c12e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:14.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.663+0000 7f3b80efe640 1 --2- 172.21.15.40:0/4116893417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b7c071270 0x7f3b7c12e350 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c00b550 tx=0x7f3b6c00ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:14.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.663+0000 7f3b80efe640 1 --2- 172.21.15.40:0/4116893417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b7c071270 0x7f3b7c12e350 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c00b550 tx=0x7f3b6c00ba10 comp rx=0 tx=0).queue_received will delay after 1.71377e+09 on 0x7f3b6c0055a0 mon_map magic: 0 v1 2024-04-22T08:22:14.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.663+0000 7f3b71ffb640 1 -- 172.21.15.40:0/4116893417 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b6c005070 con 0x7f3b7c071270 2024-04-22T08:22:14.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.663+0000 7f3b71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:14.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.663+0000 7f3b71ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:22:14.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.663+0000 7f3b71ffb640 10 monclient: dump: 2024-04-22T08:22:14.673 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:14.674 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:14.674 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:14.674 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:14.674 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:14.674 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:14.674 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:14.674 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:14.674 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:14.674 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:14.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.663+0000 7f3b71ffb640 10 monclient: _finish_auth 0 2024-04-22T08:22:14.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.663+0000 7f3b71ffb640 10 monclient: _check_auth_tickets 2024-04-22T08:22:14.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.663+0000 7f3b71ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:14.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.663+0000 7f3b83963640 5 monclient: authenticate success, global_id 114225 2024-04-22T08:22:14.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.663+0000 7f3b71ffb640 1 -- 172.21.15.40:0/4116893417 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b6c0051f0 con 0x7f3b7c071270 2024-04-22T08:22:14.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.663+0000 7f3b83963640 10 monclient: _renew_subs 2024-04-22T08:22:14.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.663+0000 7f3b83963640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:14.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.663+0000 7f3b83963640 1 -- 172.21.15.40:0/4116893417 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b7c1385f0 con 0x7f3b7c071270 2024-04-22T08:22:14.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.663+0000 7f3b71ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:14.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.663+0000 7f3b737fe640 4 set_mon_vals no callback set 2024-04-22T08:22:14.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.663+0000 7f3b83963640 10 monclient: _renew_subs 2024-04-22T08:22:14.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.663+0000 7f3b83963640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:14.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.663+0000 7f3b83963640 1 -- 172.21.15.40:0/4116893417 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b7c136b20 con 0x7f3b7c071270 2024-04-22T08:22:14.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.664+0000 7f3b71ffb640 1 -- 172.21.15.40:0/4116893417 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b6c0055a0 con 0x7f3b7c071270 2024-04-22T08:22:14.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.664+0000 7f3b71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:14.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.664+0000 7f3b71ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:22:14.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.664+0000 7f3b71ffb640 10 monclient: dump: 2024-04-22T08:22:14.675 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:14.676 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:14.676 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:14.676 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:14.676 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:14.676 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:14.676 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:14.676 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:14.676 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:14.676 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:14.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.664+0000 7f3b71ffb640 1 -- 172.21.15.40:0/4116893417 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f3b6c005700 con 0x7f3b7c071270 2024-04-22T08:22:14.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.664+0000 7f3b83963640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:14.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.664+0000 7f3b83963640 1 --2- 172.21.15.40:0/4116893417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b7c06dc30 0x7f3b7c06e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:14.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.664+0000 7f3b83963640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:14.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.664+0000 7f3b83963640 1 -- 172.21.15.40:0/4116893417 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b7c066900 con 0x7f3b7c06dc30 2024-04-22T08:22:14.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.664+0000 7f3b71ffb640 1 --2- 172.21.15.40:0/4116893417 >> v2:172.21.15.189:6800/30168 conn(0x7f3b60037780 0x7f3b60039c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:14.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.664+0000 7f3b816ff640 1 --2- 172.21.15.40:0/4116893417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b7c06dc30 0x7f3b7c06e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:14.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.664+0000 7f3b816ff640 10 monclient: get_auth_request con 0x7f3b7c06dc30 auth_method 0 2024-04-22T08:22:14.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.664+0000 7f3b816ff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:14.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.664+0000 7f3b816ff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:14.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.664+0000 7f3b816ff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:14.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.664+0000 7f3b816ff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:14.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.664+0000 7f3b816ff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:14.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.664+0000 7f3b816ff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:14.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.664+0000 7f3b81f00640 1 --2- 172.21.15.40:0/4116893417 >> v2:172.21.15.189:6800/30168 conn(0x7f3b60037780 0x7f3b60039c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:14.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.664+0000 7f3b816ff640 10 monclient(hunting): handle_auth_done global_id 114228 payload 824 2024-04-22T08:22:14.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.664+0000 7f3b816ff640 1 -- 172.21.15.40:0/4116893417 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3b7c0a3e60 con 0x7f3b7c06dc30 2024-04-22T08:22:14.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.664+0000 7f3b81f00640 10 monclient: get_auth_request con 0x7f3b60037780 auth_method 0 2024-04-22T08:22:14.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.667+0000 7f3b816ff640 1 -- 172.21.15.40:0/4116893417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b7c06dc30 msgr2=0x7f3b7c06e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:14.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.667+0000 7f3b816ff640 1 --2- 172.21.15.40:0/4116893417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b7c06dc30 0x7f3b7c06e010 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3b78004810 tx=0x7f3b7800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:14.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.667+0000 7f3b71ffb640 1 -- 172.21.15.40:0/4116893417 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b78007070 con 0x7f3b7c06dc30 2024-04-22T08:22:14.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.667+0000 7f3b71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:14.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.667+0000 7f3b71ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:22:14.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.667+0000 7f3b71ffb640 10 monclient: dump: 2024-04-22T08:22:14.680 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:14.681 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:14.681 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:14.681 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:14.681 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:14.681 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:14.681 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:14.681 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:14.681 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:14.681 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:14.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.667+0000 7f3b81f00640 1 --2- 172.21.15.40:0/4116893417 >> v2:172.21.15.189:6800/30168 conn(0x7f3b60037780 0x7f3b60039c40 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c09eec0 tx=0x7f3b7402e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:14.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.667+0000 7f3b71ffb640 1 -- 172.21.15.40:0/4116893417 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f3b780071d0 con 0x7f3b7c06dc30 2024-04-22T08:22:14.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.667+0000 7f3b71ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:14.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.667+0000 7f3b71ffb640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:22:14.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.667+0000 7f3b71ffb640 1 -- 172.21.15.40:0/4116893417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b7c06dc30 msgr2=0x7f3b7c06e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:14.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.667+0000 7f3b71ffb640 1 --2- 172.21.15.40:0/4116893417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b7c06dc30 0x7f3b7c06e010 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3b78004810 tx=0x7f3b7800c8e0 comp rx=0 tx=0).stop 2024-04-22T08:22:14.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.667+0000 7f3b71ffb640 1 -- 172.21.15.40:0/4116893417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b7c06dc30 msgr2=0x7f3b7c06e010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:22:14.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.667+0000 7f3b71ffb640 1 --2- 172.21.15.40:0/4116893417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b7c06dc30 0x7f3b7c06e010 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:14.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.673+0000 7f3b83963640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:22:14.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.673+0000 7f3b83963640 1 --2- 172.21.15.40:0/4116893417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b7c06e550 0x7f3b7c136d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:14.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.673+0000 7f3b83963640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:14.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.673+0000 7f3b83963640 1 -- 172.21.15.40:0/4116893417 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b7c13a170 con 0x7f3b7c06e550 2024-04-22T08:22:14.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.673+0000 7f3b816ff640 1 --2- 172.21.15.40:0/4116893417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b7c06e550 0x7f3b7c136d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:14.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.673+0000 7f3b816ff640 10 monclient: get_auth_request con 0x7f3b7c06e550 auth_method 0 2024-04-22T08:22:14.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.673+0000 7f3b816ff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:14.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.673+0000 7f3b816ff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:14.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.673+0000 7f3b816ff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:14.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.673+0000 7f3b816ff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:14.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.673+0000 7f3b816ff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:14.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.673+0000 7f3b816ff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:14.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.674+0000 7f3b816ff640 10 monclient(hunting): handle_auth_done global_id 114231 payload 824 2024-04-22T08:22:14.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.674+0000 7f3b816ff640 1 -- 172.21.15.40:0/4116893417 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3b7c073940 con 0x7f3b7c06e550 2024-04-22T08:22:14.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.676+0000 7f3b816ff640 1 -- 172.21.15.40:0/4116893417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b7c06e550 msgr2=0x7f3b7c136d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:14.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.676+0000 7f3b816ff640 1 --2- 172.21.15.40:0/4116893417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b7c06e550 0x7f3b7c136d30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3b78007fd0 tx=0x7f3b78004a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:14.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.676+0000 7f3b71ffb640 1 -- 172.21.15.40:0/4116893417 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b7801b630 con 0x7f3b7c06e550 2024-04-22T08:22:14.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.676+0000 7f3b71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:14.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.676+0000 7f3b71ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:22:14.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.676+0000 7f3b71ffb640 10 monclient: dump: 2024-04-22T08:22:14.683 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:14.683 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:14.683 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:14.683 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:14.683 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:14.684 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:14.684 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:14.684 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:14.684 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:14.684 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:14.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.676+0000 7f3b71ffb640 1 -- 172.21.15.40:0/4116893417 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4622 (secure 0 0 0) 0x7f3b7801b790 con 0x7f3b7c06e550 2024-04-22T08:22:14.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.676+0000 7f3b71ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:22:14.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.676+0000 7f3b71ffb640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:22:14.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.676+0000 7f3b71ffb640 1 -- 172.21.15.40:0/4116893417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b7c06e550 msgr2=0x7f3b7c136d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:14.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.676+0000 7f3b71ffb640 1 --2- 172.21.15.40:0/4116893417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b7c06e550 0x7f3b7c136d30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3b78007fd0 tx=0x7f3b78004a10 comp rx=0 tx=0).stop 2024-04-22T08:22:14.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.676+0000 7f3b71ffb640 1 -- 172.21.15.40:0/4116893417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b7c06e550 msgr2=0x7f3b7c136d30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:22:14.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.676+0000 7f3b71ffb640 1 --2- 172.21.15.40:0/4116893417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b7c06e550 0x7f3b7c136d30 secure :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3b78007fd0 tx=0x7f3b78004a10 comp rx=0 tx=0).stop 2024-04-22T08:22:14.685 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:22:14.685 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:22:14.685 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:22:14.685 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "leader", 2024-04-22T08:22:14.685 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 96, 2024-04-22T08:22:14.685 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:22:14.685 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:22:14.685 INFO:teuthology.orchestra.run.smithi040.stdout: 1 2024-04-22T08:22:14.685 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:22:14.685 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 1, 2024-04-22T08:22:14.685 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:22:14.685 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:22:14.686 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:22:14.686 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:22:14.686 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:22:14.686 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:22:14.686 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:22:14.686 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:22:14.686 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:22:14.686 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:22:14.686 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:22:14.686 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:22:14.686 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:22:14.686 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:22:14.686 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:22:14.687 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:22:14.687 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:22:14.687 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:22:14.687 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:22:14.687 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:22:14.687 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:22:14.687 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:22:14.687 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:22:14.687 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:22:14.687 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:22:14.687 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:22:14.687 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:22:14.687 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:14.687 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:22:14.688 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:22:14.688 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:22:14.688 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:22:14.688 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:22:14.688 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:22:14.688 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:22:14.688 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:22:14.688 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:22:14.688 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:22:14.688 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:22:14.688 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:22:14.688 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:22:14.688 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:22:14.688 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:22:14.689 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:22:14.689 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:22:14.689 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:22:14.689 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:22:14.689 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:22:14.689 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:22:14.689 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:22:14.689 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:22:14.689 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:22:14.689 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:22:14.689 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:22:14.689 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:22:14.689 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:22:14.689 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:22:14.690 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:14.690 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:22:14.690 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:14.690 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:22:14.690 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:22:14.690 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:22:14.690 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:22:14.690 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:14.690 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:22:14.690 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:22:14.690 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:22:14.690 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:14.690 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:14.690 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:22:14.691 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:22:14.691 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:22:14.691 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:22:14.691 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:22:14.691 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:14.691 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:22:14.691 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:22:14.691 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:22:14.691 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:22:14.691 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:22:14.691 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:14.691 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:14.691 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:22:14.691 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:22:14.692 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:22:14.692 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:22:14.692 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:14.692 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:22:14.692 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:22:14.692 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:22:14.692 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:14.692 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:14.692 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:22:14.692 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:22:14.692 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:22:14.692 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:22:14.692 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:22:14.692 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:14.692 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:22:14.693 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:22:14.693 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:22:14.693 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:22:14.693 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:22:14.693 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:14.693 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:14.693 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:22:14.693 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:22:14.693 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:22:14.693 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:22:14.693 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:14.693 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:22:14.693 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:22:14.693 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:22:14.693 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:14.694 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:14.694 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:22:14.694 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:22:14.694 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:22:14.694 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:22:14.694 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:22:14.694 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:14.694 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:22:14.694 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:22:14.694 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:22:14.694 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:22:14.694 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:22:14.694 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:22:14.694 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:22:14.695 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:14.695 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:22:14.695 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:22:14.695 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:14.695 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:22:14.695 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:22:14.695 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:22:14.695 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:22:14.695 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:22:14.695 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:22:14.695 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:14.695 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:22:14.695 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:22:14.695 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:22:14.695 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:22:14.696 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:22:14.696 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:22:14.696 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:14.696 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:22:14.696 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:22:14.696 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 3 2024-04-22T08:22:14.696 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:22:14.696 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:22:14.696 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:14.696 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:22:14.696 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:22:14.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.677+0000 7f3b83963640 1 -- 172.21.15.40:0/4116893417 >> v2:172.21.15.189:6800/30168 conn(0x7f3b60037780 msgr2=0x7f3b60039c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:14.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.677+0000 7f3b83963640 1 --2- 172.21.15.40:0/4116893417 >> v2:172.21.15.189:6800/30168 conn(0x7f3b60037780 0x7f3b60039c40 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c09eec0 tx=0x7f3b7402e000 comp rx=0 tx=0).stop 2024-04-22T08:22:14.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.677+0000 7f3b83963640 10 monclient: shutdown 2024-04-22T08:22:14.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.677+0000 7f3b83963640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:14.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.677+0000 7f3b83963640 1 -- 172.21.15.40:0/4116893417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b7c071270 msgr2=0x7f3b7c12e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:14.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.677+0000 7f3b83963640 1 --2- 172.21.15.40:0/4116893417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b7c071270 0x7f3b7c12e350 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c00b550 tx=0x7f3b6c00ba10 comp rx=0 tx=0).stop 2024-04-22T08:22:14.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.677+0000 7f3b81f00640 1 -- 172.21.15.40:0/4116893417 reap_dead start 2024-04-22T08:22:14.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.677+0000 7f3b71ffb640 1 -- 172.21.15.40:0/4116893417 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(313..313 src has 239..313) v4 ==== 3506+0+0 (unknown 0 0 0) 0x7f3b6c04a270 con 0x7f3b7c071270 2024-04-22T08:22:14.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.677+0000 7f3b83963640 1 -- 172.21.15.40:0/4116893417 shutdown_connections 2024-04-22T08:22:14.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.677+0000 7f3b83963640 1 --2- 172.21.15.40:0/4116893417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b7c071270 0x7f3b7c12e350 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:14.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.677+0000 7f3b83963640 1 -- 172.21.15.40:0/4116893417 >> 172.21.15.40:0/4116893417 conn(0x7f3b7c0a1780 msgr2=0x7f3b7c12f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:14.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.677+0000 7f3b83963640 1 -- 172.21.15.40:0/4116893417 shutdown_connections 2024-04-22T08:22:14.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.677+0000 7f3b83963640 1 -- 172.21.15.40:0/4116893417 wait complete. 2024-04-22T08:22:14.699 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-22T08:22:14.699 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-04-22T08:22:14.699 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-22T08:22:14.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.799+0000 7ff6b068f640 10 monclient: get_monmap_and_config 2024-04-22T08:22:14.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.799+0000 7ff6b068f640 10 monclient: build_initial_monmap 2024-04-22T08:22:14.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.799+0000 7ff6b068f640 10 monclient: monmap: 2024-04-22T08:22:14.800 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:14.800 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:14.800 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:22:14.800504+0000 2024-04-22T08:22:14.800 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:22:14.800504+0000 2024-04-22T08:22:14.800 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:14.800 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:14.800 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:22:14.800 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:22:14.800 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:22:14.800 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:22:14.800 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:22:14.800 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:14.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.800+0000 7ff6b068f640 1 Processor -- start 2024-04-22T08:22:14.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.800+0000 7ff6b068f640 1 -- start start 2024-04-22T08:22:14.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.800+0000 7ff6b068f640 10 monclient: init 2024-04-22T08:22:14.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.800+0000 7ff6b068f640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:14.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.800+0000 7ff6b068f640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-04-22T08:22:14.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.800+0000 7ff6b068f640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff6a812fad0 0x7ff6a812feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:14.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.800+0000 7ff6b068f640 10 monclient(hunting): picked mon.noname-a con 0x7ff6a812fad0 addr v2:172.21.15.40:3300/0 2024-04-22T08:22:14.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.801+0000 7ff6b068f640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff6a8132920 0x7ff6a8132d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:14.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.801+0000 7ff6b068f640 10 monclient(hunting): picked mon.noname-c con 0x7ff6a8132920 addr v2:172.21.15.189:3300/0 2024-04-22T08:22:14.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.801+0000 7ff6b068f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff6a8133290 addr v1:172.21.15.189:6789/0 2024-04-22T08:22:14.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.801+0000 7ff6b068f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6a809ae50 con 0x7ff6a8133290 2024-04-22T08:22:14.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.801+0000 7ff6b068f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:14.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.801+0000 7ff6b068f640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6a809efc0 con 0x7ff6a812fad0 2024-04-22T08:22:14.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.801+0000 7ff6b068f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:14.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.801+0000 7ff6b068f640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6a809d070 con 0x7ff6a8132920 2024-04-22T08:22:14.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.801+0000 7ff6b068f640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:14.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.801+0000 7ff6b068f640 10 monclient(hunting): authenticate will time out at 1408.794849s 2024-04-22T08:22:14.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.801+0000 7ff6ae42b640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff6a812fad0 0x7ff6a812feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:14.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.801+0000 7ff6ae42b640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff6a812fad0 0x7ff6a812feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58206/0 (socket says 172.21.15.40:58206) 2024-04-22T08:22:14.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.801+0000 7ff6ae42b640 1 -- 172.21.15.40:0/3152297679 learned_addr learned my addr 172.21.15.40:0/3152297679 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:22:14.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.801+0000 7ff6ae42b640 10 monclient(hunting): get_auth_request con 0x7ff6a812fad0 auth_method 0 2024-04-22T08:22:14.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.801+0000 7ff6ae42b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:14.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.801+0000 7ff6ae42b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:14.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.801+0000 7ff6ae42b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:14.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.801+0000 7ff6ae42b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:14.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.801+0000 7ff6ae42b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:14.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.801+0000 7ff6ae42b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:14.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.801+0000 7ff6adc2a640 1 --2- 172.21.15.40:0/3152297679 >> v2:172.21.15.189:3300/0 conn(0x7ff6a8132920 0x7ff6a8132d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:14.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.801+0000 7ff6ae42b640 10 monclient(hunting): handle_auth_done global_id 114234 payload 274 2024-04-22T08:22:14.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.801+0000 7ff6ae42b640 1 -- 172.21.15.40:0/3152297679 >> v2:172.21.15.189:3300/0 conn(0x7ff6a8132920 msgr2=0x7ff6a8132d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:14.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.801+0000 7ff6ae42b640 1 --2- 172.21.15.40:0/3152297679 >> v2:172.21.15.189:3300/0 conn(0x7ff6a8132920 0x7ff6a8132d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:14.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.801+0000 7ff6ae42b640 1 -- 172.21.15.40:0/3152297679 >> v1:172.21.15.189:6789/0 conn(0x7ff6a8133290 legacy=0x7ff6a8139c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:14.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.801+0000 7ff6ae42b640 10 monclient: _finish_hunting 0 2024-04-22T08:22:14.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.801+0000 7ff6ae42b640 1 monclient: found mon.noname-a 2024-04-22T08:22:14.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.801+0000 7ff6ae42b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:14.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.801+0000 7ff6ae42b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:22:14.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.801+0000 7ff6ae42b640 1 -- 172.21.15.40:0/3152297679 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6a809d350 con 0x7ff6a812fad0 2024-04-22T08:22:14.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.801+0000 7ff6adc2a640 10 monclient: get_auth_request con 0x7ff6a8132920 auth_method 0 2024-04-22T08:22:14.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.801+0000 7ff6adc2a640 1 --2- 172.21.15.40:0/3152297679 >> v2:172.21.15.189:3300/0 conn(0x7ff6a8132920 0x7ff6a8132d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T08:22:14.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.804+0000 7ff6ae42b640 1 -- 172.21.15.40:0/3152297679 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6a812fad0 msgr2=0x7ff6a812feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:14.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.804+0000 7ff6ae42b640 1 --2- 172.21.15.40:0/3152297679 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6a812fad0 0x7ff6a812feb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff6a80725d0 tx=0x7ff6a402dfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e1691aadf135c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:14.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.804+0000 7ff6ad429640 1 -- 172.21.15.40:0/3152297679 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a4036640 con 0x7ff6a812fad0 2024-04-22T08:22:14.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.804+0000 7ff6ad429640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:14.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.804+0000 7ff6ad429640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:22:14.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.804+0000 7ff6ad429640 10 monclient: dump: 2024-04-22T08:22:14.808 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:14.808 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:14.808 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:14.808 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:14.808 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:14.809 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:14.809 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:14.809 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:14.809 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:14.809 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:14.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.804+0000 7ff6ad429640 10 monclient: _finish_auth 0 2024-04-22T08:22:14.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.804+0000 7ff6ad429640 10 monclient: _check_auth_tickets 2024-04-22T08:22:14.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.804+0000 7ff6ad429640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:14.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.804+0000 7ff6ad429640 1 -- 172.21.15.40:0/3152297679 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6a40367a0 con 0x7ff6a812fad0 2024-04-22T08:22:14.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.804+0000 7ff6ad429640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:14.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.804+0000 7ff6ad429640 1 -- 172.21.15.40:0/3152297679 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a4036b30 con 0x7ff6a812fad0 2024-04-22T08:22:14.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.804+0000 7ff6b068f640 5 monclient: authenticate success, global_id 114234 2024-04-22T08:22:14.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.804+0000 7ff6b068f640 10 monclient: get_monmap_and_config success 2024-04-22T08:22:14.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.804+0000 7ff6b068f640 4 set_mon_vals no callback set 2024-04-22T08:22:14.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.804+0000 7ff6ad429640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:14.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.804+0000 7ff6ad429640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:22:14.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.804+0000 7ff6ad429640 10 monclient: dump: 2024-04-22T08:22:14.810 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:14.810 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:14.810 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:14.810 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:14.810 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:14.810 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:14.810 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:14.810 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:14.810 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:14.810 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:14.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.805+0000 7ff6b068f640 10 monclient: shutdown 2024-04-22T08:22:14.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.805+0000 7ff6b068f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:14.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.805+0000 7ff6b068f640 1 -- 172.21.15.40:0/3152297679 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6a812fad0 msgr2=0x7ff6a812feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:14.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.805+0000 7ff6b068f640 1 --2- 172.21.15.40:0/3152297679 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6a812fad0 0x7ff6a812feb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff6a80725d0 tx=0x7ff6a402dfc0 comp rx=0 tx=0).stop 2024-04-22T08:22:14.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.805+0000 7ff6b068f640 1 -- 172.21.15.40:0/3152297679 shutdown_connections 2024-04-22T08:22:14.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.805+0000 7ff6b068f640 1 --2- 172.21.15.40:0/3152297679 >> v2:172.21.15.189:3300/0 conn(0x7ff6a8132920 0x7ff6a8132d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:14.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.805+0000 7ff6b068f640 1 --2- 172.21.15.40:0/3152297679 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6a812fad0 0x7ff6a812feb0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:14.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.805+0000 7ff6b068f640 1 -- 172.21.15.40:0/3152297679 >> 172.21.15.40:0/3152297679 conn(0x7ff6a80a0ae0 msgr2=0x7ff6a80a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:14.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.805+0000 7ff6b068f640 1 -- 172.21.15.40:0/3152297679 shutdown_connections 2024-04-22T08:22:14.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.805+0000 7ff6b068f640 1 -- 172.21.15.40:0/3152297679 wait complete. 2024-04-22T08:22:14.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.805+0000 7ff6b068f640 10 monclient: build_initial_monmap 2024-04-22T08:22:14.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.805+0000 7ff6b068f640 10 monclient: monmap: 2024-04-22T08:22:14.811 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:14.811 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:22:14.811 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:22:14.811 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:22:14.812 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:14.812 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:14.812 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:22:14.812 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:22:14.812 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:22:14.812 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:14.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.806+0000 7ff6b068f640 1 Processor -- start 2024-04-22T08:22:14.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.806+0000 7ff6b068f640 1 -- start start 2024-04-22T08:22:14.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.806+0000 7ff6b068f640 10 monclient: init 2024-04-22T08:22:14.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.806+0000 7ff6b068f640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:14.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.806+0000 7ff6b068f640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-22T08:22:14.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.806+0000 7ff6b068f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6a812fad0 0x7ff6a81ca9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:14.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.806+0000 7ff6b068f640 10 monclient(hunting): picked mon.noname-a con 0x7ff6a812fad0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:22:14.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.806+0000 7ff6b068f640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff6a8132920 0x7ff6a81caf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:14.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.806+0000 7ff6b068f640 10 monclient(hunting): picked mon.noname-c con 0x7ff6a8132920 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:22:14.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.806+0000 7ff6b068f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff6a8133290 0x7ff6a806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:14.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.806+0000 7ff6b068f640 10 monclient(hunting): picked mon.noname-b con 0x7ff6a8133290 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:22:14.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.806+0000 7ff6b068f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:14.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.806+0000 7ff6b068f640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a811ef70 con 0x7ff6a812fad0 2024-04-22T08:22:14.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.806+0000 7ff6adc2a640 1 -- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff6a8132920 msgr2=0x7ff6a81caf10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:3301/0 2024-04-22T08:22:14.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.806+0000 7ff6adc2a640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff6a8132920 0x7ff6a81caf10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:22:14.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.806+0000 7ff6b068f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:14.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.806+0000 7ff6b068f640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a8071870 con 0x7ff6a8133290 2024-04-22T08:22:14.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.806+0000 7ff6b068f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:14.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.806+0000 7ff6b068f640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a8071410 con 0x7ff6a8132920 2024-04-22T08:22:14.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.806+0000 7ff6b068f640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:14.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.806+0000 7ff6ae42b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6a812fad0 0x7ff6a81ca9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:14.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.806+0000 7ff6b068f640 10 monclient(hunting): authenticate will time out at 1408.800506s 2024-04-22T08:22:14.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.806+0000 7ff6ae42b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6a812fad0 0x7ff6a81ca9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58220/0 (socket says 172.21.15.40:58220) 2024-04-22T08:22:14.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.806+0000 7ff6ae42b640 1 -- 172.21.15.40:0/3998224777 learned_addr learned my addr 172.21.15.40:0/3998224777 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:22:14.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.806+0000 7ff6ae42b640 10 monclient(hunting): get_auth_request con 0x7ff6a812fad0 auth_method 0 2024-04-22T08:22:14.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.806+0000 7ff6ae42b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:14.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.806+0000 7ff6ae42b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:14.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.806+0000 7ff6ae42b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:14.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.806+0000 7ff6ae42b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:14.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.806+0000 7ff6ae42b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:14.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.806+0000 7ff6ae42b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:14.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.806+0000 7ff6ae42b640 10 monclient(hunting): handle_auth_done global_id 114237 payload 824 2024-04-22T08:22:14.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.807+0000 7ff6ae42b640 1 -- 172.21.15.40:0/3998224777 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff6a8132920 msgr2=0x7ff6a81caf10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:22:14.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.807+0000 7ff6ae42b640 1 --2- 172.21.15.40:0/3998224777 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff6a8132920 0x7ff6a81caf10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:14.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.807+0000 7ff6ae42b640 1 -- 172.21.15.40:0/3998224777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff6a8133290 msgr2=0x7ff6a806dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:14.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.807+0000 7ff6ae42b640 1 --2- 172.21.15.40:0/3998224777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff6a8133290 0x7ff6a806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:14.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.807+0000 7ff6ae42b640 10 monclient: _finish_hunting 0 2024-04-22T08:22:14.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.807+0000 7ff6ae42b640 1 monclient: found mon.noname-a 2024-04-22T08:22:14.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.807+0000 7ff6ae42b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:14.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.807+0000 7ff6ae42b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:22:14.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.807+0000 7ff6ae42b640 1 -- 172.21.15.40:0/3998224777 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6a400b770 con 0x7ff6a812fad0 2024-04-22T08:22:14.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.809+0000 7ff6ae42b640 1 -- 172.21.15.40:0/3998224777 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6a812fad0 msgr2=0x7ff6a81ca9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:14.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.809+0000 7ff6ae42b640 1 --2- 172.21.15.40:0/3998224777 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6a812fad0 0x7ff6a81ca9d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff6a4000c00 tx=0x7ff6a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:14.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.809+0000 7ff69effd640 1 -- 172.21.15.40:0/3998224777 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a403f960 con 0x7ff6a812fad0 2024-04-22T08:22:14.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.809+0000 7ff69effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:14.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.809+0000 7ff69effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:22:14.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.809+0000 7ff69effd640 10 monclient: dump: 2024-04-22T08:22:14.820 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:14.820 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:14.820 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:14.820 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:14.820 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:14.820 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:14.820 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:14.820 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:14.820 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:14.821 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:14.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.809+0000 7ff69effd640 10 monclient: _finish_auth 0 2024-04-22T08:22:14.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.809+0000 7ff69effd640 10 monclient: _check_auth_tickets 2024-04-22T08:22:14.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.809+0000 7ff69effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:14.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.809+0000 7ff69effd640 1 -- 172.21.15.40:0/3998224777 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6a403fac0 con 0x7ff6a812fad0 2024-04-22T08:22:14.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.809+0000 7ff69effd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:14.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.809+0000 7ff69effd640 1 -- 172.21.15.40:0/3998224777 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a403fe70 con 0x7ff6a812fad0 2024-04-22T08:22:14.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.809+0000 7ff6b068f640 5 monclient: authenticate success, global_id 114237 2024-04-22T08:22:14.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.809+0000 7ff6b068f640 10 monclient: _renew_subs 2024-04-22T08:22:14.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.809+0000 7ff6b068f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:14.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.809+0000 7ff6b068f640 1 -- 172.21.15.40:0/3998224777 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6a806f950 con 0x7ff6a812fad0 2024-04-22T08:22:14.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.809+0000 7ff6acc28640 4 set_mon_vals no callback set 2024-04-22T08:22:14.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.809+0000 7ff69effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:14.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.809+0000 7ff69effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:22:14.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.809+0000 7ff69effd640 10 monclient: dump: 2024-04-22T08:22:14.822 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:14.822 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:14.822 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:14.822 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:14.822 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:14.822 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:14.822 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:14.822 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:14.822 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:14.822 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:14.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.809+0000 7ff6b068f640 10 monclient: _renew_subs 2024-04-22T08:22:14.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.809+0000 7ff6b068f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:14.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.809+0000 7ff6b068f640 1 -- 172.21.15.40:0/3998224777 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6a81cb450 con 0x7ff6a812fad0 2024-04-22T08:22:14.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.810+0000 7ff6b068f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:14.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.810+0000 7ff6b068f640 1 --2- 172.21.15.40:0/3998224777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff6a811f950 0x7ff6a806e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:14.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.810+0000 7ff6b068f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:14.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.810+0000 7ff6b068f640 1 -- 172.21.15.40:0/3998224777 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a80667a0 con 0x7ff6a811f950 2024-04-22T08:22:14.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.810+0000 7ff69effd640 1 -- 172.21.15.40:0/3998224777 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7ff6a40403e0 con 0x7ff6a812fad0 2024-04-22T08:22:14.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.810+0000 7ff69effd640 1 --2- 172.21.15.40:0/3998224777 >> v2:172.21.15.189:6800/30168 conn(0x7ff68c037780 0x7ff68c039c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:14.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.810+0000 7ff69effd640 1 -- 172.21.15.40:0/3998224777 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(313..313 src has 239..313) v4 ==== 3506+0+0 (secure 0 0 0) 0x7ff6a406c120 con 0x7ff6a812fad0 2024-04-22T08:22:14.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.810+0000 7ff6adc2a640 1 --2- 172.21.15.40:0/3998224777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff6a811f950 0x7ff6a806e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:14.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.810+0000 7ff6adc2a640 10 monclient: get_auth_request con 0x7ff6a811f950 auth_method 0 2024-04-22T08:22:14.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.810+0000 7ff6adc2a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:14.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.810+0000 7ff6adc2a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:14.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.810+0000 7ff6adc2a640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:14.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.810+0000 7ff6aec2c640 1 --2- 172.21.15.40:0/3998224777 >> v2:172.21.15.189:6800/30168 conn(0x7ff68c037780 0x7ff68c039c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:14.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.810+0000 7ff6adc2a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:14.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.810+0000 7ff6adc2a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:14.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.810+0000 7ff6adc2a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:14.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.810+0000 7ff6aec2c640 10 monclient: get_auth_request con 0x7ff68c037780 auth_method 0 2024-04-22T08:22:14.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.810+0000 7ff6adc2a640 10 monclient(hunting): handle_auth_done global_id 65224 payload 824 2024-04-22T08:22:14.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.811+0000 7ff6adc2a640 1 -- 172.21.15.40:0/3998224777 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff6a8071580 con 0x7ff6a811f950 2024-04-22T08:22:14.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.813+0000 7ff6adc2a640 1 -- 172.21.15.40:0/3998224777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff6a811f950 msgr2=0x7ff6a806e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:14.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.813+0000 7ff6adc2a640 1 --2- 172.21.15.40:0/3998224777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff6a811f950 0x7ff6a806e290 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7ff6a8071ac0 tx=0x7ff69800cb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:14.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.813+0000 7ff6aec2c640 1 --2- 172.21.15.40:0/3998224777 >> v2:172.21.15.189:6800/30168 conn(0x7ff68c037780 0x7ff68c039c40 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7ff6a806fd10 tx=0x7ff6a0007d30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:14.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.813+0000 7ff69effd640 1 -- 172.21.15.40:0/3998224777 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff698007a70 con 0x7ff6a811f950 2024-04-22T08:22:14.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.813+0000 7ff69effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:14.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.813+0000 7ff69effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:22:14.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.813+0000 7ff69effd640 10 monclient: dump: 2024-04-22T08:22:14.825 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:14.825 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:14.825 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:14.825 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:14.825 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:14.825 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:14.825 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:14.825 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:14.825 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:14.825 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:14.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.813+0000 7ff69effd640 1 -- 172.21.15.40:0/3998224777 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7ff69800cec0 con 0x7ff6a811f950 2024-04-22T08:22:14.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.813+0000 7ff69effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:14.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.813+0000 7ff69effd640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:22:14.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.813+0000 7ff69effd640 1 -- 172.21.15.40:0/3998224777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff6a811f950 msgr2=0x7ff6a806e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:14.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.813+0000 7ff69effd640 1 --2- 172.21.15.40:0/3998224777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff6a811f950 0x7ff6a806e290 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7ff6a8071ac0 tx=0x7ff69800cb80 comp rx=0 tx=0).stop 2024-04-22T08:22:14.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.813+0000 7ff69effd640 1 -- 172.21.15.40:0/3998224777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff6a811f950 msgr2=0x7ff6a806e290 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:22:14.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.813+0000 7ff69effd640 1 --2- 172.21.15.40:0/3998224777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff6a811f950 0x7ff6a806e290 secure :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7ff6a8071ac0 tx=0x7ff69800cb80 comp rx=0 tx=0).stop 2024-04-22T08:22:14.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.820+0000 7ff6b068f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:22:14.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.820+0000 7ff6b068f640 1 --2- 172.21.15.40:0/3998224777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff6a806e7d0 0x7ff6a806ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:14.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.820+0000 7ff6b068f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:14.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.820+0000 7ff6b068f640 1 -- 172.21.15.40:0/3998224777 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a809c9b0 con 0x7ff6a806e7d0 2024-04-22T08:22:14.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.820+0000 7ff6adc2a640 1 --2- 172.21.15.40:0/3998224777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff6a806e7d0 0x7ff6a806ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:14.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.820+0000 7ff6adc2a640 10 monclient: get_auth_request con 0x7ff6a806e7d0 auth_method 0 2024-04-22T08:22:14.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.820+0000 7ff6adc2a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:14.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.820+0000 7ff6adc2a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:14.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.820+0000 7ff6adc2a640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:14.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.820+0000 7ff6adc2a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:14.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.820+0000 7ff6adc2a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:14.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.820+0000 7ff6adc2a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:14.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.820+0000 7ff6adc2a640 10 monclient(hunting): handle_auth_done global_id 65227 payload 824 2024-04-22T08:22:14.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.820+0000 7ff6adc2a640 1 -- 172.21.15.40:0/3998224777 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff6a811ebf0 con 0x7ff6a806e7d0 2024-04-22T08:22:14.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.823+0000 7ff6adc2a640 1 -- 172.21.15.40:0/3998224777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff6a806e7d0 msgr2=0x7ff6a806ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:14.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.823+0000 7ff6adc2a640 1 --2- 172.21.15.40:0/3998224777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff6a806e7d0 0x7ff6a806ebe0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7ff6a8071ac0 tx=0x7ff698004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:14.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.823+0000 7ff69effd640 1 -- 172.21.15.40:0/3998224777 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff698028070 con 0x7ff6a806e7d0 2024-04-22T08:22:14.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.823+0000 7ff69effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:14.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.823+0000 7ff69effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:22:14.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.823+0000 7ff69effd640 10 monclient: dump: 2024-04-22T08:22:14.829 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:14.829 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:14.829 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:14.829 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:14.829 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:14.829 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:14.829 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:14.830 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:14.830 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:14.830 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:14.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.823+0000 7ff69effd640 1 -- 172.21.15.40:0/3998224777 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4790 (secure 0 0 0) 0x7ff698020040 con 0x7ff6a806e7d0 2024-04-22T08:22:14.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.823+0000 7ff69effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:22:14.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.823+0000 7ff69effd640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:22:14.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.823+0000 7ff69effd640 1 -- 172.21.15.40:0/3998224777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff6a806e7d0 msgr2=0x7ff6a806ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:14.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.823+0000 7ff69effd640 1 --2- 172.21.15.40:0/3998224777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff6a806e7d0 0x7ff6a806ebe0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7ff6a8071ac0 tx=0x7ff698004730 comp rx=0 tx=0).stop 2024-04-22T08:22:14.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.823+0000 7ff69effd640 1 -- 172.21.15.40:0/3998224777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff6a806e7d0 msgr2=0x7ff6a806ebe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:22:14.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.823+0000 7ff69effd640 1 --2- 172.21.15.40:0/3998224777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff6a806e7d0 0x7ff6a806ebe0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:14.831 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:22:14.831 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:22:14.831 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:22:14.831 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:22:14.831 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 96, 2024-04-22T08:22:14.831 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:22:14.831 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:22:14.831 INFO:teuthology.orchestra.run.smithi040.stdout: 1 2024-04-22T08:22:14.831 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:22:14.831 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 1, 2024-04-22T08:22:14.831 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:22:14.831 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:22:14.831 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:22:14.832 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:22:14.832 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:22:14.832 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:22:14.832 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:22:14.832 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:22:14.832 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:22:14.832 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:22:14.832 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:22:14.832 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:22:14.832 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:22:14.832 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:22:14.832 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:22:14.832 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:22:14.832 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:22:14.832 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:22:14.833 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:22:14.833 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:22:14.833 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:22:14.833 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:22:14.833 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:22:14.833 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:22:14.833 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:22:14.833 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:22:14.833 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:22:14.833 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:14.833 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:22:14.833 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:22:14.833 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:22:14.833 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:22:14.834 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:22:14.834 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:22:14.834 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:22:14.834 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:22:14.834 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:22:14.834 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:22:14.834 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:22:14.834 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:22:14.834 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:22:14.834 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:22:14.834 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:22:14.834 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:22:14.834 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:22:14.835 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:22:14.835 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:22:14.835 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:22:14.835 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:22:14.835 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:22:14.835 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:22:14.835 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:22:14.835 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:22:14.835 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:22:14.835 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:22:14.835 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:22:14.835 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:22:14.835 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:14.835 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:22:14.836 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:14.836 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:22:14.836 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:22:14.836 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:22:14.836 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:22:14.836 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:14.836 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:22:14.836 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:22:14.836 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:22:14.836 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:14.836 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:14.836 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:22:14.836 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:22:14.836 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:22:14.836 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:22:14.837 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:22:14.837 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:14.837 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:22:14.837 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:22:14.837 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:22:14.837 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:22:14.837 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:22:14.837 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:14.837 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:14.837 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:22:14.837 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:22:14.837 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:22:14.837 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:22:14.837 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:14.838 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:22:14.838 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:22:14.838 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:22:14.838 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:14.838 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:14.838 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:22:14.838 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:22:14.838 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:22:14.838 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:22:14.838 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:22:14.838 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:14.838 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:22:14.838 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:22:14.838 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:22:14.839 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:22:14.839 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:22:14.839 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:14.839 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:14.839 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:22:14.839 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:22:14.839 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:22:14.839 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:22:14.839 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:14.839 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:22:14.839 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:22:14.839 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:22:14.839 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:14.839 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:14.840 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:22:14.840 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:22:14.840 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:22:14.840 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:22:14.840 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:22:14.840 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:14.840 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:22:14.840 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:22:14.840 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:22:14.840 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:22:14.840 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:22:14.840 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:22:14.840 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:22:14.840 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:14.840 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:22:14.841 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:22:14.841 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:14.841 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:22:14.841 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:22:14.841 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:22:14.841 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:22:14.841 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:22:14.841 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:22:14.841 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:14.841 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:22:14.841 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:22:14.841 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 5 2024-04-22T08:22:14.841 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:22:14.841 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:22:14.842 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:22:14.842 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:14.842 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:22:14.842 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:22:14.842 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 7 2024-04-22T08:22:14.842 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:22:14.842 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:22:14.842 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": [ 2024-04-22T08:22:14.842 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:14.842 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:22:14.842 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:22:14.842 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:22:14.842 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:22:14.842 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:22:14.843 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:14.843 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:22:14.843 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:22:14.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.824+0000 7ff6b068f640 1 -- 172.21.15.40:0/3998224777 >> v2:172.21.15.189:6800/30168 conn(0x7ff68c037780 msgr2=0x7ff68c039c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:14.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.824+0000 7ff6b068f640 1 --2- 172.21.15.40:0/3998224777 >> v2:172.21.15.189:6800/30168 conn(0x7ff68c037780 0x7ff68c039c40 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7ff6a806fd10 tx=0x7ff6a0007d30 comp rx=0 tx=0).stop 2024-04-22T08:22:14.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.824+0000 7ff6b068f640 10 monclient: shutdown 2024-04-22T08:22:14.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.824+0000 7ff6b068f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:14.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.824+0000 7ff6b068f640 1 -- 172.21.15.40:0/3998224777 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6a812fad0 msgr2=0x7ff6a81ca9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:14.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.824+0000 7ff6b068f640 1 --2- 172.21.15.40:0/3998224777 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6a812fad0 0x7ff6a81ca9d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff6a4000c00 tx=0x7ff6a40046c0 comp rx=0 tx=0).stop 2024-04-22T08:22:14.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.824+0000 7ff6aec2c640 1 -- 172.21.15.40:0/3998224777 reap_dead start 2024-04-22T08:22:14.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.824+0000 7ff6aec2c640 1 -- 172.21.15.40:0/3998224777 reap_dead start 2024-04-22T08:22:14.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.824+0000 7ff6b068f640 1 -- 172.21.15.40:0/3998224777 shutdown_connections 2024-04-22T08:22:14.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.824+0000 7ff6b068f640 1 -- 172.21.15.40:0/3998224777 >> 172.21.15.40:0/3998224777 conn(0x7ff6a80a0ae0 msgr2=0x7ff6a812f160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:14.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.824+0000 7ff6b068f640 1 -- 172.21.15.40:0/3998224777 shutdown_connections 2024-04-22T08:22:14.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:14.824+0000 7ff6b068f640 1 -- 172.21.15.40:0/3998224777 wait complete. 2024-04-22T08:22:14.845 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-22T08:22:14.846 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-04-22T08:22:16.067 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:22:16.068 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[39572]: starting. 2024-04-22T08:22:16.068 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[39572]: creating pool ceph_test_rados_delete_pools_parallel.smithi189-39537 2024-04-22T08:22:16.068 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[39572]: created object 0... 2024-04-22T08:22:16.068 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[39572]: created object 25... 2024-04-22T08:22:16.068 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[39572]: created object 49... 2024-04-22T08:22:16.068 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[39572]: finishing. 2024-04-22T08:22:16.068 INFO:teuthology.orchestra.run.smithi189.stdout:process_3_[39572]: shutting down. 2024-04-22T08:22:16.109 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:22:16.109 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[39574]: starting. 2024-04-22T08:22:16.109 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[39574]: listing objects. 2024-04-22T08:22:16.109 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[39574]: listed object 0... 2024-04-22T08:22:16.109 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[39574]: listed object 25... 2024-04-22T08:22:16.109 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[39574]: saw 50 objects 2024-04-22T08:22:16.110 INFO:teuthology.orchestra.run.smithi189.stdout:process_5_[39574]: shutting down. 2024-04-22T08:22:16.954 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:22:16.955 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[39573]: starting. 2024-04-22T08:22:16.955 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[39573]: deleting pool ceph_test_rados_delete_pools_parallel.smithi189-39537 2024-04-22T08:22:16.955 INFO:teuthology.orchestra.run.smithi189.stdout:process_4_[39573]: shutting down. 2024-04-22T08:22:16.956 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:22:16.956 INFO:teuthology.orchestra.run.smithi189.stdout:******************************* 2024-04-22T08:22:16.956 INFO:teuthology.orchestra.run.smithi189.stdout:******* SUCCESS ********** 2024-04-22T08:22:16.958 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2024-04-22T08:22:16.968 INFO:tasks.mon_thrash:joining mon_thrasher 2024-04-22T08:22:34.846 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-04-22T08:22:34.846 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-04-22T08:22:34.847 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-22T08:22:34.847 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-22T08:22:34.850 INFO:tasks.ceph.mon.c:Started 2024-04-22T08:22:34.850 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-22T08:22:34.850 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:22:34.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.960+0000 7fb0029a9640 10 monclient: get_monmap_and_config 2024-04-22T08:22:34.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.960+0000 7fb0029a9640 10 monclient: build_initial_monmap 2024-04-22T08:22:34.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.960+0000 7fb0029a9640 10 monclient: monmap: 2024-04-22T08:22:34.961 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:34.961 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:34.961 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:22:34.962399+0000 2024-04-22T08:22:34.961 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:22:34.962399+0000 2024-04-22T08:22:34.961 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:34.961 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:34.961 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:22:34.962 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:22:34.962 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:22:34.962 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:22:34.962 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:22:34.962 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:34.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.961+0000 7fb0029a9640 1 Processor -- start 2024-04-22T08:22:34.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.961+0000 7fb0029a9640 1 -- start start 2024-04-22T08:22:34.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.961+0000 7fb0029a9640 10 monclient: init 2024-04-22T08:22:34.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.961+0000 7fb0029a9640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:34.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.961+0000 7fb0029a9640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-04-22T08:22:34.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.961+0000 7fb0029a9640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7faffc09ee10 0x7faffc0a32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:34.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.961+0000 7fb0029a9640 10 monclient(hunting): picked mon.noname-c con 0x7faffc09ee10 addr v2:172.21.15.189:3300/0 2024-04-22T08:22:34.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.961+0000 7fb0029a9640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7faffc0a1d30 0x7faffc0a2180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:34.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.961+0000 7fb0029a9640 10 monclient(hunting): picked mon.noname-a con 0x7faffc0a1d30 addr v2:172.21.15.40:3300/0 2024-04-22T08:22:34.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.961+0000 7fb0029a9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faffc135d90 addr v1:172.21.15.189:6789/0 2024-04-22T08:22:34.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.961+0000 7fb0029a9640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faffc09f1f0 con 0x7faffc135d90 2024-04-22T08:22:34.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.961+0000 7fb0029a9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:34.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.961+0000 7fb0029a9640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faffc0a39c0 con 0x7faffc0a1d30 2024-04-22T08:22:34.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.961+0000 7fb0029a9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:34.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.961+0000 7fb0029a9640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faffc09fd30 con 0x7faffc09ee10 2024-04-22T08:22:34.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.961+0000 7fb0029a9640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:34.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.961+0000 7fb0029a9640 10 monclient(hunting): authenticate will time out at 1428.956705s 2024-04-22T08:22:34.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.962+0000 7faffb7fe640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7faffc0a1d30 0x7faffc0a2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:34.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.962+0000 7faffb7fe640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7faffc0a1d30 0x7faffc0a2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50096/0 (socket says 172.21.15.40:50096) 2024-04-22T08:22:34.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.962+0000 7faffb7fe640 1 -- 172.21.15.40:0/2264971966 learned_addr learned my addr 172.21.15.40:0/2264971966 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:22:34.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.962+0000 7faffb7fe640 10 monclient(hunting): get_auth_request con 0x7faffc0a1d30 auth_method 0 2024-04-22T08:22:34.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.962+0000 7faffb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:34.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.962+0000 7faffb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:34.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.962+0000 7faffb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:34.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.962+0000 7faffb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:34.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.962+0000 7faffb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:34.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.962+0000 7faffb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:34.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.962+0000 7faffbfff640 1 --2- 172.21.15.40:0/2264971966 >> v2:172.21.15.189:3300/0 conn(0x7faffc09ee10 0x7faffc0a32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:34.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.962+0000 7faffbfff640 10 monclient(hunting): get_auth_request con 0x7faffc09ee10 auth_method 0 2024-04-22T08:22:34.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.962+0000 7faffbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:34.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.962+0000 7faffbfff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:34.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.962+0000 7faffbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:34.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.962+0000 7faffb7fe640 10 monclient(hunting): handle_auth_done global_id 114240 payload 274 2024-04-22T08:22:34.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.962+0000 7faffb7fe640 1 -- 172.21.15.40:0/2264971966 >> v2:172.21.15.189:3300/0 conn(0x7faffc09ee10 msgr2=0x7faffc0a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:34.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.962+0000 7faffb7fe640 1 --2- 172.21.15.40:0/2264971966 >> v2:172.21.15.189:3300/0 conn(0x7faffc09ee10 0x7faffc0a32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:34.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.962+0000 7faffb7fe640 1 -- 172.21.15.40:0/2264971966 >> v1:172.21.15.189:6789/0 conn(0x7faffc135d90 legacy=0x7faffc1361a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:34.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.962+0000 7faffb7fe640 10 monclient: _finish_hunting 0 2024-04-22T08:22:34.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.962+0000 7faffb7fe640 1 monclient: found mon.noname-a 2024-04-22T08:22:34.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.962+0000 7faffb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:34.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.962+0000 7faffb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:22:34.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.962+0000 7faffb7fe640 1 -- 172.21.15.40:0/2264971966 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faffc0a02f0 con 0x7faffc0a1d30 2024-04-22T08:22:34.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.965+0000 7faffb7fe640 1 -- 172.21.15.40:0/2264971966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faffc0a1d30 msgr2=0x7faffc0a2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:34.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.965+0000 7faffb7fe640 1 --2- 172.21.15.40:0/2264971966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faffc0a1d30 0x7faffc0a2180 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7faffc06c620 tx=0x7fafec02b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=564a4988751b7a11 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:34.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.965+0000 7faffaffd640 1 -- 172.21.15.40:0/2264971966 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafec031730 con 0x7faffc0a1d30 2024-04-22T08:22:34.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.965+0000 7faffaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:34.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.965+0000 7faffaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:22:34.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.965+0000 7faffaffd640 10 monclient: dump: 2024-04-22T08:22:34.973 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:34.973 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:34.973 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:34.973 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:34.974 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:34.974 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:34.974 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:34.974 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:34.974 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:34.974 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:34.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.965+0000 7faffaffd640 10 monclient: _finish_auth 0 2024-04-22T08:22:34.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.965+0000 7faffaffd640 10 monclient: _check_auth_tickets 2024-04-22T08:22:34.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.965+0000 7faffaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:34.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.965+0000 7faffaffd640 1 -- 172.21.15.40:0/2264971966 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafec031890 con 0x7faffc0a1d30 2024-04-22T08:22:34.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.965+0000 7fb0029a9640 5 monclient: authenticate success, global_id 114240 2024-04-22T08:22:34.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.965+0000 7fb0029a9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:22:34.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.965+0000 7faffaffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:34.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.965+0000 7faffaffd640 1 -- 172.21.15.40:0/2264971966 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafec031c20 con 0x7faffc0a1d30 2024-04-22T08:22:34.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.965+0000 7fb0029a9640 10 monclient: get_monmap_and_config success 2024-04-22T08:22:34.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.965+0000 7fb0029a9640 4 set_mon_vals no callback set 2024-04-22T08:22:34.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.965+0000 7faffaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:34.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.965+0000 7faffaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:22:34.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.965+0000 7faffaffd640 10 monclient: dump: 2024-04-22T08:22:34.975 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:34.975 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:34.975 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:34.975 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:34.975 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:34.975 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:34.975 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:34.975 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:34.975 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:34.976 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:34.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.965+0000 7fb0029a9640 10 monclient: shutdown 2024-04-22T08:22:34.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.965+0000 7fb0029a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:34.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.965+0000 7fb0029a9640 1 -- 172.21.15.40:0/2264971966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faffc0a1d30 msgr2=0x7faffc0a2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:34.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.965+0000 7fb0029a9640 1 --2- 172.21.15.40:0/2264971966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faffc0a1d30 0x7faffc0a2180 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7faffc06c620 tx=0x7fafec02b7c0 comp rx=0 tx=0).stop 2024-04-22T08:22:34.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.965+0000 7fb0029a9640 1 -- 172.21.15.40:0/2264971966 shutdown_connections 2024-04-22T08:22:34.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.965+0000 7fb0029a9640 1 --2- 172.21.15.40:0/2264971966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faffc0a1d30 0x7faffc0a2180 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:34.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.965+0000 7fb0029a9640 1 --2- 172.21.15.40:0/2264971966 >> v2:172.21.15.189:3300/0 conn(0x7faffc09ee10 0x7faffc0a32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:34.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.965+0000 7fb0029a9640 1 -- 172.21.15.40:0/2264971966 >> 172.21.15.40:0/2264971966 conn(0x7faffc0a4ef0 msgr2=0x7faffc0a5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:34.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.965+0000 7fb0029a9640 1 -- 172.21.15.40:0/2264971966 shutdown_connections 2024-04-22T08:22:34.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.965+0000 7fb0029a9640 1 -- 172.21.15.40:0/2264971966 wait complete. 2024-04-22T08:22:34.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.966+0000 7fb0029a9640 10 monclient: build_initial_monmap 2024-04-22T08:22:34.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.966+0000 7fb0029a9640 10 monclient: monmap: 2024-04-22T08:22:34.976 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:34.977 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:22:34.977 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:22:34.977 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:22:34.977 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:34.977 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:34.977 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:22:34.977 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:22:34.977 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:22:34.977 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:34.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.966+0000 7fb0029a9640 1 Processor -- start 2024-04-22T08:22:34.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.967+0000 7fb0029a9640 1 -- start start 2024-04-22T08:22:34.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.967+0000 7fb0029a9640 10 monclient: init 2024-04-22T08:22:34.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.967+0000 7fb0029a9640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:34.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.967+0000 7fb0029a9640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-22T08:22:34.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.967+0000 7fb0029a9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faffc09ee10 0x7faffc1cade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:34.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.967+0000 7fb0029a9640 10 monclient(hunting): picked mon.noname-b con 0x7faffc09ee10 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:22:34.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.967+0000 7fb0029a9640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faffc0a1d30 0x7faffc1cb320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:34.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.967+0000 7fb0029a9640 10 monclient(hunting): picked mon.noname-c con 0x7faffc0a1d30 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:22:34.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.967+0000 7fb0029a9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faffc135d90 0x7faffc122310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:34.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.967+0000 7fb0029a9640 10 monclient(hunting): picked mon.noname-a con 0x7faffc135d90 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:22:34.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.967+0000 7fb0029a9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:34.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.967+0000 7fb0029a9640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faffc125fa0 con 0x7faffc135d90 2024-04-22T08:22:34.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.967+0000 7fb0029a9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:34.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.967+0000 7fb0029a9640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faffc1261f0 con 0x7faffc09ee10 2024-04-22T08:22:34.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.967+0000 7fb0029a9640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:34.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.967+0000 7fb0029a9640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faffc1249f0 con 0x7faffc0a1d30 2024-04-22T08:22:34.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.967+0000 7fb0029a9640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:34.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.967+0000 7fb0029a9640 10 monclient(hunting): authenticate will time out at 1428.962326s 2024-04-22T08:22:34.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.967+0000 7faffb7fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faffc0a1d30 0x7faffc1cb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:34.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.967+0000 7fb000f46640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faffc135d90 0x7faffc122310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:34.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.967+0000 7faffb7fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faffc0a1d30 0x7faffc1cb320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:57644/0 (socket says 172.21.15.40:57644) 2024-04-22T08:22:34.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.967+0000 7faffb7fe640 1 -- 172.21.15.40:0/235635992 learned_addr learned my addr 172.21.15.40:0/235635992 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:22:34.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.967+0000 7faffb7fe640 10 monclient(hunting): get_auth_request con 0x7faffc0a1d30 auth_method 0 2024-04-22T08:22:34.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.967+0000 7fb000f46640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faffc135d90 0x7faffc122310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50112/0 (socket says 172.21.15.40:50112) 2024-04-22T08:22:34.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.967+0000 7faffb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:34.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.967+0000 7faffb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:34.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.967+0000 7faffb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:34.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.967+0000 7fb000f46640 10 monclient(hunting): get_auth_request con 0x7faffc135d90 auth_method 0 2024-04-22T08:22:34.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.967+0000 7fb000f46640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:34.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.967+0000 7fb000f46640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:34.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.967+0000 7fb000f46640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:34.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.967+0000 7fb000f46640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:34.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.967+0000 7fb000f46640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:34.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.967+0000 7faffbfff640 1 --2- 172.21.15.40:0/235635992 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faffc09ee10 0x7faffc1cade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:34.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.967+0000 7fb000f46640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:34.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.967+0000 7faffb7fe640 1 -- 172.21.15.40:0/235635992 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faffc0a1d30 msgr2=0x7faffc1cb320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-22T08:22:34.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.967+0000 7faffb7fe640 1 -- 172.21.15.40:0/235635992 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faffc0a1d30 msgr2=0x7faffc1cb320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:22:34.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.967+0000 7faffb7fe640 1 --2- 172.21.15.40:0/235635992 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faffc0a1d30 0x7faffc1cb320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:22:34.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.967+0000 7faffbfff640 10 monclient(hunting): get_auth_request con 0x7faffc09ee10 auth_method 0 2024-04-22T08:22:34.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.967+0000 7faffbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:34.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.967+0000 7faffbfff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:34.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.967+0000 7faffbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:34.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.967+0000 7faffb7fe640 1 --2- 172.21.15.40:0/235635992 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faffc0a1d30 0x7faffc1cb320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:22:34.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.967+0000 7faffbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:34.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.967+0000 7faffbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:34.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.967+0000 7faffbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:34.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.968+0000 7fb000f46640 10 monclient(hunting): handle_auth_done global_id 114243 payload 824 2024-04-22T08:22:34.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.968+0000 7fb000f46640 1 -- 172.21.15.40:0/235635992 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faffc0a1d30 msgr2=0x7faffc1cb320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:22:34.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.968+0000 7fb000f46640 1 --2- 172.21.15.40:0/235635992 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faffc0a1d30 0x7faffc1cb320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:34.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.968+0000 7fb000f46640 1 -- 172.21.15.40:0/235635992 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faffc09ee10 msgr2=0x7faffc1cade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:34.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.968+0000 7fb000f46640 1 --2- 172.21.15.40:0/235635992 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faffc09ee10 0x7faffc1cade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:34.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.968+0000 7fb000f46640 10 monclient: _finish_hunting 0 2024-04-22T08:22:34.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.968+0000 7fb000f46640 1 monclient: found mon.noname-a 2024-04-22T08:22:34.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.968+0000 7fb000f46640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:34.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.968+0000 7fb000f46640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:22:34.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.968+0000 7fb000f46640 1 -- 172.21.15.40:0/235635992 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafec0094d0 con 0x7faffc135d90 2024-04-22T08:22:34.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.970+0000 7fb000f46640 1 -- 172.21.15.40:0/235635992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faffc135d90 msgr2=0x7faffc122310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:34.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.970+0000 7fb000f46640 1 --2- 172.21.15.40:0/235635992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faffc135d90 0x7faffc122310 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7faff400c7b0 tx=0x7faff400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:34.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.970+0000 7faff8ff9640 1 -- 172.21.15.40:0/235635992 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff400ac60 con 0x7faffc135d90 2024-04-22T08:22:34.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.970+0000 7faff8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:34.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.970+0000 7faff8ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:22:34.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.970+0000 7faff8ff9640 10 monclient: dump: 2024-04-22T08:22:34.981 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:34.981 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:34.982 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:34.982 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:34.982 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:34.982 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:34.982 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:34.982 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:34.982 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:34.982 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:34.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.970+0000 7faff8ff9640 10 monclient: _finish_auth 0 2024-04-22T08:22:34.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.970+0000 7faff8ff9640 10 monclient: _check_auth_tickets 2024-04-22T08:22:34.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.970+0000 7faff8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:34.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.970+0000 7faff8ff9640 1 -- 172.21.15.40:0/235635992 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faff400adc0 con 0x7faffc135d90 2024-04-22T08:22:34.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.970+0000 7faff8ff9640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:34.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.970+0000 7faff8ff9640 1 -- 172.21.15.40:0/235635992 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff4012660 con 0x7faffc135d90 2024-04-22T08:22:34.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.970+0000 7faff8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:34.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.970+0000 7faff8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:22:34.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.970+0000 7faff8ff9640 10 monclient: dump: 2024-04-22T08:22:34.983 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:34.983 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:34.983 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:34.983 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:34.983 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:34.983 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:34.983 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:34.983 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:34.983 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:34.983 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:34.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.970+0000 7fb0029a9640 5 monclient: authenticate success, global_id 114243 2024-04-22T08:22:34.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.970+0000 7fb0029a9640 10 monclient: _renew_subs 2024-04-22T08:22:34.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.970+0000 7fb0029a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:34.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.970+0000 7faffa7fc640 4 set_mon_vals no callback set 2024-04-22T08:22:34.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.970+0000 7fb0029a9640 1 -- 172.21.15.40:0/235635992 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faffc0a2730 con 0x7faffc135d90 2024-04-22T08:22:34.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.970+0000 7fb0029a9640 10 monclient: _renew_subs 2024-04-22T08:22:34.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.970+0000 7fb0029a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:34.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.970+0000 7fb0029a9640 1 -- 172.21.15.40:0/235635992 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faffc123e20 con 0x7faffc135d90 2024-04-22T08:22:34.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.971+0000 7faff8ff9640 1 -- 172.21.15.40:0/235635992 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7faff4012800 con 0x7faffc135d90 2024-04-22T08:22:34.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.971+0000 7fb0029a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:34.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.971+0000 7fb0029a9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:34.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.971+0000 7fb0029a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:34.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.971+0000 7fb0029a9640 1 -- 172.21.15.40:0/235635992 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faffc125e00 con 0x7faffc135d90 2024-04-22T08:22:34.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.971+0000 7faff8ff9640 1 --2- 172.21.15.40:0/235635992 >> v2:172.21.15.189:6800/30168 conn(0x7fafe80377d0 0x7fafe8039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:34.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.971+0000 7faff8ff9640 1 -- 172.21.15.40:0/235635992 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(317..317 src has 239..317) v4 ==== 3506+0+0 (secure 0 0 0) 0x7faff404a2c0 con 0x7faffc135d90 2024-04-22T08:22:34.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.971+0000 7faffbfff640 1 --2- 172.21.15.40:0/235635992 >> v2:172.21.15.189:6800/30168 conn(0x7fafe80377d0 0x7fafe8039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:34.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.972+0000 7faffbfff640 10 monclient: get_auth_request con 0x7fafe80377d0 auth_method 0 2024-04-22T08:22:34.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.974+0000 7faffbfff640 1 --2- 172.21.15.40:0/235635992 >> v2:172.21.15.189:6800/30168 conn(0x7fafe80377d0 0x7fafe8039c90 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7faff00079a0 tx=0x7faff0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:34.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.976+0000 7faff8ff9640 1 -- 172.21.15.40:0/235635992 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7faffc125e00 con 0x7faffc135d90 2024-04-22T08:22:34.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.976+0000 7faff8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:34.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:34.976+0000 7faff8ff9640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:22:35.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:35.226+0000 7fb0029a9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:22:35.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:35.226+0000 7fb0029a9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:22:35.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:35.226+0000 7fb0029a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:35.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:35.226+0000 7fb0029a9640 1 -- 172.21.15.40:0/235635992 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7faffc125e00 con 0x7faffc135d90 2024-04-22T08:22:35.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:35.227+0000 7faff8ff9640 1 -- 172.21.15.40:0/235635992 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7faffc125e00 con 0x7faffc135d90 2024-04-22T08:22:35.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:35.227+0000 7faff8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:22:35.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:35.227+0000 7faff8ff9640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:22:35.228 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":96,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:22:35.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:35.229+0000 7fb0029a9640 1 -- 172.21.15.40:0/235635992 >> v2:172.21.15.189:6800/30168 conn(0x7fafe80377d0 msgr2=0x7fafe8039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:35.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:35.229+0000 7fb0029a9640 1 --2- 172.21.15.40:0/235635992 >> v2:172.21.15.189:6800/30168 conn(0x7fafe80377d0 0x7fafe8039c90 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7faff00079a0 tx=0x7faff0008040 comp rx=0 tx=0).stop 2024-04-22T08:22:35.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:35.229+0000 7fb0029a9640 10 monclient: shutdown 2024-04-22T08:22:35.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:35.229+0000 7fb0029a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:35.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:35.229+0000 7fb0029a9640 1 -- 172.21.15.40:0/235635992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faffc135d90 msgr2=0x7faffc122310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:35.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:35.229+0000 7fb0029a9640 1 --2- 172.21.15.40:0/235635992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faffc135d90 0x7faffc122310 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7faff400c7b0 tx=0x7faff400cc70 comp rx=0 tx=0).stop 2024-04-22T08:22:35.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:35.229+0000 7fb0029a9640 1 -- 172.21.15.40:0/235635992 shutdown_connections 2024-04-22T08:22:35.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:35.229+0000 7fb0029a9640 1 --2- 172.21.15.40:0/235635992 >> v2:172.21.15.189:6800/30168 conn(0x7fafe80377d0 0x7fafe8039c90 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:35.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:35.229+0000 7fb0029a9640 1 --2- 172.21.15.40:0/235635992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faffc135d90 0x7faffc122310 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:35.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:35.229+0000 7fb0029a9640 1 --2- 172.21.15.40:0/235635992 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faffc0a1d30 0x7faffc1cb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:35.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:35.229+0000 7fb0029a9640 1 --2- 172.21.15.40:0/235635992 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faffc09ee10 0x7faffc1cade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:35.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:35.229+0000 7fb0029a9640 1 -- 172.21.15.40:0/235635992 >> 172.21.15.40:0/235635992 conn(0x7faffc0a4ef0 msgr2=0x7faffc06eea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:35.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:35.229+0000 7fb0029a9640 1 -- 172.21.15.40:0/235635992 shutdown_connections 2024-04-22T08:22:35.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:35.229+0000 7fb0029a9640 1 -- 172.21.15.40:0/235635992 wait complete. 2024-04-22T08:22:38.242 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:22:38.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.351+0000 7f8f39d6e640 10 monclient: get_monmap_and_config 2024-04-22T08:22:38.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.351+0000 7f8f39d6e640 10 monclient: build_initial_monmap 2024-04-22T08:22:38.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.351+0000 7f8f39d6e640 10 monclient: monmap: 2024-04-22T08:22:38.351 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:38.352 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:38.352 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:22:38.352931+0000 2024-04-22T08:22:38.352 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:22:38.352931+0000 2024-04-22T08:22:38.352 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:38.352 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:38.352 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:22:38.352 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:22:38.352 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:22:38.352 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:22:38.352 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:22:38.352 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:38.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.351+0000 7f8f39d6e640 1 Processor -- start 2024-04-22T08:22:38.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.351+0000 7f8f39d6e640 1 -- start start 2024-04-22T08:22:38.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.351+0000 7f8f39d6e640 10 monclient: init 2024-04-22T08:22:38.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.352+0000 7f8f39d6e640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:38.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.352+0000 7f8f39d6e640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-04-22T08:22:38.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.352+0000 7f8f39d6e640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8f3409ee10 0x7f8f340a32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:38.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.352+0000 7f8f39d6e640 10 monclient(hunting): picked mon.noname-b con 0x7f8f3409ee10 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:22:38.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.352+0000 7f8f39d6e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8f340a1d30 addr v1:172.21.15.189:6789/0 2024-04-22T08:22:38.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.352+0000 7f8f39d6e640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8f340a2150 0x7f8f341364a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:38.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.352+0000 7f8f39d6e640 10 monclient(hunting): picked mon.noname-a con 0x7f8f340a2150 addr v2:172.21.15.40:3300/0 2024-04-22T08:22:38.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.352+0000 7f8f39d6e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f340a2580 con 0x7f8f340a1d30 2024-04-22T08:22:38.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.352+0000 7f8f39d6e640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:38.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.352+0000 7f8f39d6e640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f3409f1f0 con 0x7f8f340a2150 2024-04-22T08:22:38.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.352+0000 7f8f39d6e640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:38.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.352+0000 7f8f39d6e640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f340a39c0 con 0x7f8f3409ee10 2024-04-22T08:22:38.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.352+0000 7f8f39d6e640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:38.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.352+0000 7f8f39d6e640 10 monclient(hunting): authenticate will time out at 1432.347272s 2024-04-22T08:22:38.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.352+0000 7f8f337fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8f3409ee10 0x7f8f340a32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:38.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.352+0000 7f8f33fff640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8f340a2150 0x7f8f341364a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:38.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.352+0000 7f8f337fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8f3409ee10 0x7f8f340a32b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:57650/0 (socket says 172.21.15.40:57650) 2024-04-22T08:22:38.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.352+0000 7f8f337fe640 1 -- 172.21.15.40:0/882156483 learned_addr learned my addr 172.21.15.40:0/882156483 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:22:38.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.352+0000 7f8f337fe640 10 monclient(hunting): get_auth_request con 0x7f8f3409ee10 auth_method 0 2024-04-22T08:22:38.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.352+0000 7f8f337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:38.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.352+0000 7f8f337fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:38.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.352+0000 7f8f337fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:38.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.352+0000 7f8f33fff640 10 monclient(hunting): get_auth_request con 0x7f8f340a2150 auth_method 0 2024-04-22T08:22:38.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.352+0000 7f8f33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:38.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.352+0000 7f8f33fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:38.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.352+0000 7f8f33fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:38.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.352+0000 7f8f337fe640 1 -- 172.21.15.40:0/882156483 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8f3409ee10 msgr2=0x7f8f340a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T08:22:38.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.352+0000 7f8f337fe640 1 -- 172.21.15.40:0/882156483 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8f3409ee10 msgr2=0x7f8f340a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:22:38.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.352+0000 7f8f337fe640 1 --2- 172.21.15.40:0/882156483 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8f3409ee10 0x7f8f340a32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:22:38.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.352+0000 7f8f33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:38.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.352+0000 7f8f33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:38.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.352+0000 7f8f33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:38.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.352+0000 7f8f337fe640 1 --2- 172.21.15.40:0/882156483 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8f3409ee10 0x7f8f340a32b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:22:38.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.352+0000 7f8f33fff640 10 monclient(hunting): handle_auth_done global_id 114246 payload 274 2024-04-22T08:22:38.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.352+0000 7f8f33fff640 1 -- 172.21.15.40:0/882156483 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8f3409ee10 msgr2=0x7f8f340a32b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:22:38.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.352+0000 7f8f33fff640 1 --2- 172.21.15.40:0/882156483 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8f3409ee10 0x7f8f340a32b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:38.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.352+0000 7f8f33fff640 1 -- 172.21.15.40:0/882156483 >> v1:172.21.15.189:6789/0 conn(0x7f8f340a1d30 legacy=0x7f8f34135d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:38.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.352+0000 7f8f33fff640 10 monclient: _finish_hunting 0 2024-04-22T08:22:38.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.352+0000 7f8f33fff640 1 monclient: found mon.noname-a 2024-04-22T08:22:38.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.352+0000 7f8f33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:38.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.352+0000 7f8f33fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:22:38.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.352+0000 7f8f33fff640 1 -- 172.21.15.40:0/882156483 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f3409fd30 con 0x7f8f340a2150 2024-04-22T08:22:38.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.355+0000 7f8f33fff640 1 -- 172.21.15.40:0/882156483 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f340a2150 msgr2=0x7f8f341364a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:38.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.355+0000 7f8f33fff640 1 --2- 172.21.15.40:0/882156483 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f340a2150 0x7f8f341364a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c009d90 tx=0x7f8f2c02bcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e45f49c6c672838 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:38.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.356+0000 7f8f327fc640 1 -- 172.21.15.40:0/882156483 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c02ccb0 con 0x7f8f340a2150 2024-04-22T08:22:38.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.356+0000 7f8f327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:38.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.356+0000 7f8f327fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:22:38.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.356+0000 7f8f327fc640 10 monclient: dump: 2024-04-22T08:22:38.364 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:38.364 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:38.364 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:38.364 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:38.364 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:38.364 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:38.364 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:38.364 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:38.364 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:38.364 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:38.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.356+0000 7f8f327fc640 10 monclient: _finish_auth 0 2024-04-22T08:22:38.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.356+0000 7f8f327fc640 10 monclient: _check_auth_tickets 2024-04-22T08:22:38.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.356+0000 7f8f327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:38.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.356+0000 7f8f327fc640 1 -- 172.21.15.40:0/882156483 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f2c02ce10 con 0x7f8f340a2150 2024-04-22T08:22:38.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.356+0000 7f8f39d6e640 5 monclient: authenticate success, global_id 114246 2024-04-22T08:22:38.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.356+0000 7f8f39d6e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:22:38.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.356+0000 7f8f327fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:38.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.356+0000 7f8f39d6e640 10 monclient: get_monmap_and_config success 2024-04-22T08:22:38.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.356+0000 7f8f327fc640 1 -- 172.21.15.40:0/882156483 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c034660 con 0x7f8f340a2150 2024-04-22T08:22:38.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.356+0000 7f8f327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:38.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.356+0000 7f8f39d6e640 4 set_mon_vals no callback set 2024-04-22T08:22:38.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.356+0000 7f8f327fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:22:38.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.356+0000 7f8f327fc640 10 monclient: dump: 2024-04-22T08:22:38.365 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:38.365 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:38.365 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:38.365 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:38.365 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:38.366 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:38.366 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:38.366 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:38.366 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:38.366 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:38.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.356+0000 7f8f39d6e640 10 monclient: shutdown 2024-04-22T08:22:38.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.356+0000 7f8f39d6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:38.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.356+0000 7f8f39d6e640 1 -- 172.21.15.40:0/882156483 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f340a2150 msgr2=0x7f8f341364a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:38.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.356+0000 7f8f39d6e640 1 --2- 172.21.15.40:0/882156483 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f340a2150 0x7f8f341364a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c009d90 tx=0x7f8f2c02bcf0 comp rx=0 tx=0).stop 2024-04-22T08:22:38.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.356+0000 7f8f39d6e640 1 -- 172.21.15.40:0/882156483 shutdown_connections 2024-04-22T08:22:38.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.356+0000 7f8f39d6e640 1 --2- 172.21.15.40:0/882156483 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f340a2150 0x7f8f341364a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:38.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.356+0000 7f8f39d6e640 1 --2- 172.21.15.40:0/882156483 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8f3409ee10 0x7f8f340a32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:38.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.356+0000 7f8f39d6e640 1 -- 172.21.15.40:0/882156483 >> 172.21.15.40:0/882156483 conn(0x7f8f340a4ef0 msgr2=0x7f8f340a5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:38.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.356+0000 7f8f39d6e640 1 -- 172.21.15.40:0/882156483 shutdown_connections 2024-04-22T08:22:38.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.356+0000 7f8f39d6e640 1 -- 172.21.15.40:0/882156483 wait complete. 2024-04-22T08:22:38.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.357+0000 7f8f39d6e640 10 monclient: build_initial_monmap 2024-04-22T08:22:38.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.357+0000 7f8f39d6e640 10 monclient: monmap: 2024-04-22T08:22:38.367 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:38.367 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:22:38.367 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:22:38.367 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:22:38.367 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:38.367 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:38.367 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:22:38.367 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:22:38.367 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:22:38.367 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:38.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.357+0000 7f8f39d6e640 1 Processor -- start 2024-04-22T08:22:38.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.357+0000 7f8f39d6e640 1 -- start start 2024-04-22T08:22:38.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.357+0000 7f8f39d6e640 10 monclient: init 2024-04-22T08:22:38.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f39d6e640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:38.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f39d6e640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:22:38.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f39d6e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f3409ee10 0x7f8f341cadf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:38.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f39d6e640 10 monclient(hunting): picked mon.noname-a con 0x7f8f3409ee10 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:22:38.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f39d6e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f340a1d30 0x7f8f341cb330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:38.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f39d6e640 10 monclient(hunting): picked mon.noname-b con 0x7f8f340a1d30 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:22:38.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f39d6e640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8f340a2150 0x7f8f34122020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:38.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f39d6e640 10 monclient(hunting): picked mon.noname-c con 0x7f8f340a2150 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:22:38.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f39d6e640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:38.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f39d6e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f34125d80 con 0x7f8f3409ee10 2024-04-22T08:22:38.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f39d6e640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:38.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f39d6e640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f34125fd0 con 0x7f8f340a1d30 2024-04-22T08:22:38.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f39d6e640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:38.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f39d6e640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f34124840 con 0x7f8f340a2150 2024-04-22T08:22:38.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f39d6e640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:38.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f39d6e640 10 monclient(hunting): authenticate will time out at 1432.353045s 2024-04-22T08:22:38.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f337fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f3409ee10 0x7f8f341cadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:38.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f33fff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8f340a2150 0x7f8f34122020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:38.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f337fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f3409ee10 0x7f8f341cadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50178/0 (socket says 172.21.15.40:50178) 2024-04-22T08:22:38.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f337fe640 1 -- 172.21.15.40:0/1633063198 learned_addr learned my addr 172.21.15.40:0/1633063198 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:22:38.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f337fe640 10 monclient(hunting): get_auth_request con 0x7f8f3409ee10 auth_method 0 2024-04-22T08:22:38.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:38.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f337fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:38.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f337fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:38.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f33fff640 10 monclient(hunting): get_auth_request con 0x7f8f340a2150 auth_method 0 2024-04-22T08:22:38.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:38.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f33fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:38.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f33fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:38.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:38.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:38.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f33fff640 1 -- 172.21.15.40:0/1633063198 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8f340a2150 msgr2=0x7f8f34122020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-22T08:22:38.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f33fff640 1 -- 172.21.15.40:0/1633063198 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8f340a2150 msgr2=0x7f8f34122020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:22:38.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f33fff640 1 --2- 172.21.15.40:0/1633063198 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8f340a2150 0x7f8f34122020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:22:38.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f32ffd640 1 --2- 172.21.15.40:0/1633063198 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f340a1d30 0x7f8f341cb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:38.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:38.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f33fff640 1 --2- 172.21.15.40:0/1633063198 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8f340a2150 0x7f8f34122020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:22:38.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f32ffd640 10 monclient(hunting): get_auth_request con 0x7f8f340a1d30 auth_method 0 2024-04-22T08:22:38.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f32ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:38.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f32ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:38.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:38.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f32ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:38.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f32ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:38.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f32ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:38.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f337fe640 10 monclient(hunting): handle_auth_done global_id 114249 payload 824 2024-04-22T08:22:38.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f337fe640 1 -- 172.21.15.40:0/1633063198 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8f340a2150 msgr2=0x7f8f34122020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:22:38.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f337fe640 1 --2- 172.21.15.40:0/1633063198 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8f340a2150 0x7f8f34122020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:38.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f337fe640 1 -- 172.21.15.40:0/1633063198 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f340a1d30 msgr2=0x7f8f341cb330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:38.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f337fe640 1 --2- 172.21.15.40:0/1633063198 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f340a1d30 0x7f8f341cb330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:38.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f337fe640 10 monclient: _finish_hunting 0 2024-04-22T08:22:38.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f337fe640 1 monclient: found mon.noname-a 2024-04-22T08:22:38.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:38.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f337fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:22:38.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.358+0000 7f8f337fe640 1 -- 172.21.15.40:0/1633063198 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f2c009a70 con 0x7f8f3409ee10 2024-04-22T08:22:38.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.360+0000 7f8f337fe640 1 -- 172.21.15.40:0/1633063198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f3409ee10 msgr2=0x7f8f341cadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:38.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.360+0000 7f8f337fe640 1 --2- 172.21.15.40:0/1633063198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f3409ee10 0x7f8f341cadf0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8f2400e810 tx=0x7f8f2400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:38.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.362+0000 7f8f1bfff640 1 -- 172.21.15.40:0/1633063198 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f24004020 con 0x7f8f3409ee10 2024-04-22T08:22:38.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.362+0000 7f8f1bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:38.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.362+0000 7f8f1bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:22:38.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.362+0000 7f8f1bfff640 10 monclient: dump: 2024-04-22T08:22:38.372 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:38.372 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:38.372 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:38.372 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:38.372 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:38.372 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:38.372 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:38.373 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:38.373 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:38.373 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:38.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.362+0000 7f8f1bfff640 10 monclient: _finish_auth 0 2024-04-22T08:22:38.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.362+0000 7f8f1bfff640 10 monclient: _check_auth_tickets 2024-04-22T08:22:38.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.362+0000 7f8f1bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:38.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.362+0000 7f8f39d6e640 5 monclient: authenticate success, global_id 114249 2024-04-22T08:22:38.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.362+0000 7f8f1bfff640 1 -- 172.21.15.40:0/1633063198 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f24004180 con 0x7f8f3409ee10 2024-04-22T08:22:38.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.362+0000 7f8f1bfff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:38.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.362+0000 7f8f39d6e640 10 monclient: _renew_subs 2024-04-22T08:22:38.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.362+0000 7f8f1bfff640 1 -- 172.21.15.40:0/1633063198 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2400c8e0 con 0x7f8f3409ee10 2024-04-22T08:22:38.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.362+0000 7f8f39d6e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:38.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.362+0000 7f8f39d6e640 1 -- 172.21.15.40:0/1633063198 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f34136090 con 0x7f8f3409ee10 2024-04-22T08:22:38.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.362+0000 7f8f1bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:38.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.362+0000 7f8f1bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:22:38.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.362+0000 7f8f31ffb640 4 set_mon_vals no callback set 2024-04-22T08:22:38.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.362+0000 7f8f1bfff640 10 monclient: dump: 2024-04-22T08:22:38.374 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:38.374 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:38.374 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:38.374 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:38.374 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:38.374 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:38.374 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:38.374 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:38.374 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:38.374 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:38.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.362+0000 7f8f39d6e640 10 monclient: _renew_subs 2024-04-22T08:22:38.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.362+0000 7f8f39d6e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:38.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.362+0000 7f8f39d6e640 1 -- 172.21.15.40:0/1633063198 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f34123c70 con 0x7f8f3409ee10 2024-04-22T08:22:38.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.362+0000 7f8f1bfff640 1 -- 172.21.15.40:0/1633063198 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f8f2400ca40 con 0x7f8f3409ee10 2024-04-22T08:22:38.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.362+0000 7f8f39d6e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:38.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.362+0000 7f8f39d6e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:38.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.362+0000 7f8f39d6e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:38.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.362+0000 7f8f39d6e640 1 -- 172.21.15.40:0/1633063198 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f3404e9b0 con 0x7f8f3409ee10 2024-04-22T08:22:38.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.362+0000 7f8f1bfff640 1 --2- 172.21.15.40:0/1633063198 >> v2:172.21.15.189:6800/30168 conn(0x7f8f200377d0 0x7f8f20039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:38.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.362+0000 7f8f1bfff640 1 -- 172.21.15.40:0/1633063198 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(317..317 src has 239..317) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f8f24049d50 con 0x7f8f3409ee10 2024-04-22T08:22:38.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.362+0000 7f8f32ffd640 1 --2- 172.21.15.40:0/1633063198 >> v2:172.21.15.189:6800/30168 conn(0x7f8f200377d0 0x7f8f20039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:38.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.362+0000 7f8f32ffd640 10 monclient: get_auth_request con 0x7f8f200377d0 auth_method 0 2024-04-22T08:22:38.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.366+0000 7f8f32ffd640 1 --2- 172.21.15.40:0/1633063198 >> v2:172.21.15.189:6800/30168 conn(0x7f8f200377d0 0x7f8f20039c90 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f8f34135b80 tx=0x7f8f28007e30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:38.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.367+0000 7f8f1bfff640 1 -- 172.21.15.40:0/1633063198 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f8f2400ccf0 con 0x7f8f3409ee10 2024-04-22T08:22:38.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.367+0000 7f8f1bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:38.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.367+0000 7f8f1bfff640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:22:38.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.618+0000 7f8f39d6e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:22:38.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.618+0000 7f8f39d6e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:22:38.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.618+0000 7f8f39d6e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:38.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.618+0000 7f8f39d6e640 1 -- 172.21.15.40:0/1633063198 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8f3406a1e0 con 0x7f8f3409ee10 2024-04-22T08:22:38.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.618+0000 7f8f1bfff640 1 -- 172.21.15.40:0/1633063198 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f8f24022770 con 0x7f8f3409ee10 2024-04-22T08:22:38.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.618+0000 7f8f1bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:22:38.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.618+0000 7f8f1bfff640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:22:38.619 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":98,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:22:38.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.621+0000 7f8f39d6e640 1 -- 172.21.15.40:0/1633063198 >> v2:172.21.15.189:6800/30168 conn(0x7f8f200377d0 msgr2=0x7f8f20039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:38.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.621+0000 7f8f39d6e640 1 --2- 172.21.15.40:0/1633063198 >> v2:172.21.15.189:6800/30168 conn(0x7f8f200377d0 0x7f8f20039c90 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f8f34135b80 tx=0x7f8f28007e30 comp rx=0 tx=0).stop 2024-04-22T08:22:38.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.621+0000 7f8f39d6e640 10 monclient: shutdown 2024-04-22T08:22:38.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.621+0000 7f8f39d6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:38.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.621+0000 7f8f39d6e640 1 -- 172.21.15.40:0/1633063198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f3409ee10 msgr2=0x7f8f341cadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:38.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.621+0000 7f8f39d6e640 1 --2- 172.21.15.40:0/1633063198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f3409ee10 0x7f8f341cadf0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8f2400e810 tx=0x7f8f2400ecd0 comp rx=0 tx=0).stop 2024-04-22T08:22:38.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.621+0000 7f8f39d6e640 1 -- 172.21.15.40:0/1633063198 shutdown_connections 2024-04-22T08:22:38.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.621+0000 7f8f39d6e640 1 --2- 172.21.15.40:0/1633063198 >> v2:172.21.15.189:6800/30168 conn(0x7f8f200377d0 0x7f8f20039c90 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:38.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.621+0000 7f8f39d6e640 1 --2- 172.21.15.40:0/1633063198 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f340a1d30 0x7f8f341cb330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:38.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.621+0000 7f8f39d6e640 1 --2- 172.21.15.40:0/1633063198 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8f340a2150 0x7f8f34122020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:38.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.621+0000 7f8f39d6e640 1 --2- 172.21.15.40:0/1633063198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f3409ee10 0x7f8f341cadf0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:38.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.621+0000 7f8f39d6e640 1 -- 172.21.15.40:0/1633063198 >> 172.21.15.40:0/1633063198 conn(0x7f8f340a4ef0 msgr2=0x7f8f341362a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:38.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.621+0000 7f8f39d6e640 1 -- 172.21.15.40:0/1633063198 shutdown_connections 2024-04-22T08:22:38.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.621+0000 7f8f39d6e640 1 -- 172.21.15.40:0/1633063198 wait complete. 2024-04-22T08:22:38.631 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-22T08:22:38.631 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-22T08:22:38.632 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-22T08:22:38.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.732+0000 7f6407dfd640 10 monclient: get_monmap_and_config 2024-04-22T08:22:38.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.732+0000 7f6407dfd640 10 monclient: build_initial_monmap 2024-04-22T08:22:38.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.732+0000 7f6407dfd640 10 monclient: monmap: 2024-04-22T08:22:38.732 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:38.732 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:38.732 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:22:38.733019+0000 2024-04-22T08:22:38.732 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:22:38.733019+0000 2024-04-22T08:22:38.732 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:38.733 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:38.733 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:22:38.733 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:22:38.733 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:22:38.733 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:22:38.733 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:22:38.733 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:38.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.732+0000 7f6407dfd640 1 Processor -- start 2024-04-22T08:22:38.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.732+0000 7f6407dfd640 1 -- start start 2024-04-22T08:22:38.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.732+0000 7f6407dfd640 10 monclient: init 2024-04-22T08:22:38.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.733+0000 7f6407dfd640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:38.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.733+0000 7f6407dfd640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-04-22T08:22:38.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.733+0000 7f6407dfd640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f64000a1c60 0x7f64000a31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:38.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.733+0000 7f6407dfd640 10 monclient(hunting): picked mon.noname-a con 0x7f64000a1c60 addr v2:172.21.15.40:3300/0 2024-04-22T08:22:38.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.733+0000 7f6407dfd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f64000a2040 addr v1:172.21.15.40:6789/0 2024-04-22T08:22:38.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.733+0000 7f6407dfd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f64000a24d0 addr v1:172.21.15.189:6789/0 2024-04-22T08:22:38.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.733+0000 7f6407dfd640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64000a41f0 con 0x7f64000a2040 2024-04-22T08:22:38.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.733+0000 7f6407dfd640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f640009fe90 con 0x7f64000a24d0 2024-04-22T08:22:38.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.733+0000 7f6407dfd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:38.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.733+0000 7f6407dfd640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64001369d0 con 0x7f64000a1c60 2024-04-22T08:22:38.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.733+0000 7f6407dfd640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:38.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.733+0000 7f6407dfd640 10 monclient(hunting): authenticate will time out at 1432.727226s 2024-04-22T08:22:38.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.733+0000 7f6405b99640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f64000a1c60 0x7f64000a31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:38.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.733+0000 7f6405398640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f64000a2040 0x7f6400135bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:54392/0 (socket says 172.21.15.40:54392) 2024-04-22T08:22:38.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.733+0000 7f6405398640 1 -- 172.21.15.40:0/1605574483 learned_addr learned my addr 172.21.15.40:0/1605574483 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:22:38.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.733+0000 7f6405b99640 10 monclient(hunting): get_auth_request con 0x7f64000a1c60 auth_method 0 2024-04-22T08:22:38.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.733+0000 7f6405b99640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:38.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.733+0000 7f6405b99640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:38.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.733+0000 7f6405b99640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:38.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.733+0000 7f6405b99640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:38.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.733+0000 7f6405b99640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:38.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.733+0000 7f6405b99640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:38.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.733+0000 7f6405b99640 10 monclient(hunting): handle_auth_done global_id 114252 payload 274 2024-04-22T08:22:38.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.733+0000 7f6405b99640 1 -- 172.21.15.40:0/1605574483 >> v1:172.21.15.189:6789/0 conn(0x7f64000a24d0 legacy=0x7f64001362c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:38.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.733+0000 7f6405b99640 1 -- 172.21.15.40:0/1605574483 >> v1:172.21.15.40:6789/0 conn(0x7f64000a2040 legacy=0x7f6400135bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:38.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.733+0000 7f6405b99640 10 monclient: _finish_hunting 0 2024-04-22T08:22:38.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.733+0000 7f6405b99640 1 monclient: found mon.noname-a 2024-04-22T08:22:38.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.733+0000 7f6405b99640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:38.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.733+0000 7f6405b99640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:22:38.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.733+0000 7f6405b99640 1 -- 172.21.15.40:0/1605574483 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64000a0170 con 0x7f64000a1c60 2024-04-22T08:22:38.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.736+0000 7f6405b99640 1 -- 172.21.15.40:0/1605574483 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64000a1c60 msgr2=0x7f64000a31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:38.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.736+0000 7f6405b99640 1 --2- 172.21.15.40:0/1605574483 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64000a1c60 0x7f64000a31d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f63f8009d90 tx=0x7f63f802bd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f814d495ded205f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:38.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.736+0000 7f6404b97640 1 -- 172.21.15.40:0/1605574483 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f802cda0 con 0x7f64000a1c60 2024-04-22T08:22:38.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.736+0000 7f6404b97640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:38.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.736+0000 7f6404b97640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:22:38.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.736+0000 7f6404b97640 10 monclient: dump: 2024-04-22T08:22:38.746 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:38.746 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:38.746 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:38.746 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:38.746 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:38.746 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:38.746 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:38.746 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:38.747 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:38.747 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:38.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.736+0000 7f6404b97640 10 monclient: _finish_auth 0 2024-04-22T08:22:38.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.736+0000 7f6404b97640 10 monclient: _check_auth_tickets 2024-04-22T08:22:38.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.736+0000 7f6404b97640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:38.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.736+0000 7f6404b97640 1 -- 172.21.15.40:0/1605574483 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f8034440 con 0x7f64000a1c60 2024-04-22T08:22:38.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.736+0000 7f6407dfd640 5 monclient: authenticate success, global_id 114252 2024-04-22T08:22:38.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.736+0000 7f6404b97640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:38.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.736+0000 7f6407dfd640 10 monclient: get_monmap_and_config success 2024-04-22T08:22:38.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.736+0000 7f6404b97640 1 -- 172.21.15.40:0/1605574483 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f80347d0 con 0x7f64000a1c60 2024-04-22T08:22:38.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.736+0000 7f6404b97640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:38.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.736+0000 7f6407dfd640 4 set_mon_vals no callback set 2024-04-22T08:22:38.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.736+0000 7f6404b97640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:22:38.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.736+0000 7f6404b97640 10 monclient: dump: 2024-04-22T08:22:38.748 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:38.748 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:38.748 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:38.748 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:38.748 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:38.748 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:38.748 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:38.748 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:38.748 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:38.748 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:38.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.737+0000 7f6407dfd640 10 monclient: shutdown 2024-04-22T08:22:38.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.737+0000 7f6407dfd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:38.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.737+0000 7f6407dfd640 1 -- 172.21.15.40:0/1605574483 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64000a1c60 msgr2=0x7f64000a31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:38.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.737+0000 7f6407dfd640 1 --2- 172.21.15.40:0/1605574483 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64000a1c60 0x7f64000a31d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f63f8009d90 tx=0x7f63f802bd00 comp rx=0 tx=0).stop 2024-04-22T08:22:38.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.737+0000 7f6407dfd640 1 -- 172.21.15.40:0/1605574483 shutdown_connections 2024-04-22T08:22:38.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.737+0000 7f6407dfd640 1 --2- 172.21.15.40:0/1605574483 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64000a1c60 0x7f64000a31d0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:38.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.737+0000 7f6407dfd640 1 -- 172.21.15.40:0/1605574483 >> 172.21.15.40:0/1605574483 conn(0x7f64000a89a0 msgr2=0x7f64000a4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:38.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.737+0000 7f6407dfd640 1 -- 172.21.15.40:0/1605574483 shutdown_connections 2024-04-22T08:22:38.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.737+0000 7f6407dfd640 1 -- 172.21.15.40:0/1605574483 wait complete. 2024-04-22T08:22:38.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.738+0000 7f6407dfd640 10 monclient: build_initial_monmap 2024-04-22T08:22:38.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.738+0000 7f6407dfd640 10 monclient: monmap: 2024-04-22T08:22:38.749 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:38.749 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:22:38.750 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:22:38.750 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:22:38.750 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:38.750 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:38.750 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:22:38.750 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:22:38.750 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:22:38.750 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:38.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.738+0000 7f6407dfd640 1 Processor -- start 2024-04-22T08:22:38.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.738+0000 7f6407dfd640 1 -- start start 2024-04-22T08:22:38.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.738+0000 7f6407dfd640 10 monclient: init 2024-04-22T08:22:38.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.738+0000 7f6407dfd640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:38.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.738+0000 7f6407dfd640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:22:38.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.738+0000 7f6407dfd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64000a1c60 0x7f64001cace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:38.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.738+0000 7f6407dfd640 10 monclient(hunting): picked mon.noname-a con 0x7f64000a1c60 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:22:38.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.738+0000 7f6407dfd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64000a2040 0x7f64001cb220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:38.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.738+0000 7f6407dfd640 10 monclient(hunting): picked mon.noname-b con 0x7f64000a2040 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:22:38.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.738+0000 7f6407dfd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f64000a24d0 0x7f640011f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:38.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.738+0000 7f6407dfd640 10 monclient(hunting): picked mon.noname-c con 0x7f64000a24d0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:22:38.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.738+0000 7f6407dfd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:38.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.738+0000 7f6407dfd640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6400122d40 con 0x7f64000a1c60 2024-04-22T08:22:38.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.738+0000 7f6407dfd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:38.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.738+0000 7f6407dfd640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6400122f90 con 0x7f64000a2040 2024-04-22T08:22:38.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.738+0000 7f6407dfd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:38.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.738+0000 7f6407dfd640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6400121420 con 0x7f64000a24d0 2024-04-22T08:22:38.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.738+0000 7f6407dfd640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:38.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.738+0000 7f6407dfd640 10 monclient(hunting): authenticate will time out at 1432.732634s 2024-04-22T08:22:38.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.738+0000 7f6405b99640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64000a1c60 0x7f64001cace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:38.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.738+0000 7f640639a640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f64000a24d0 0x7f640011f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:38.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.738+0000 7f6405b99640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64000a1c60 0x7f64001cace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50194/0 (socket says 172.21.15.40:50194) 2024-04-22T08:22:38.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.738+0000 7f6405b99640 1 -- 172.21.15.40:0/1530017067 learned_addr learned my addr 172.21.15.40:0/1530017067 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:22:38.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.738+0000 7f6405b99640 10 monclient(hunting): get_auth_request con 0x7f64000a1c60 auth_method 0 2024-04-22T08:22:38.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.739+0000 7f6405b99640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:38.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.739+0000 7f6405b99640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:38.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.739+0000 7f6405b99640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:38.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.739+0000 7f640639a640 10 monclient(hunting): get_auth_request con 0x7f64000a24d0 auth_method 0 2024-04-22T08:22:38.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.739+0000 7f640639a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:38.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.739+0000 7f640639a640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:38.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.739+0000 7f640639a640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:38.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.739+0000 7f6405b99640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:38.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.739+0000 7f6405b99640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:38.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.739+0000 7f6405b99640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:38.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.739+0000 7f640639a640 1 -- 172.21.15.40:0/1530017067 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f64000a24d0 msgr2=0x7f640011f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-22T08:22:38.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.739+0000 7f640639a640 1 -- 172.21.15.40:0/1530017067 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f64000a24d0 msgr2=0x7f640011f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:22:38.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.739+0000 7f640639a640 1 --2- 172.21.15.40:0/1530017067 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f64000a24d0 0x7f640011f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:22:38.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.739+0000 7f640639a640 1 --2- 172.21.15.40:0/1530017067 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f64000a24d0 0x7f640011f350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:22:38.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.739+0000 7f6405398640 1 --2- 172.21.15.40:0/1530017067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64000a2040 0x7f64001cb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:38.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.739+0000 7f6405398640 10 monclient(hunting): get_auth_request con 0x7f64000a2040 auth_method 0 2024-04-22T08:22:38.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.739+0000 7f6405398640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:38.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.739+0000 7f6405398640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:38.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.739+0000 7f6405398640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:38.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.739+0000 7f6405b99640 10 monclient(hunting): handle_auth_done global_id 114255 payload 824 2024-04-22T08:22:38.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.739+0000 7f6405b99640 1 -- 172.21.15.40:0/1530017067 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f64000a24d0 msgr2=0x7f640011f350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:22:38.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.739+0000 7f6405b99640 1 --2- 172.21.15.40:0/1530017067 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f64000a24d0 0x7f640011f350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:38.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.739+0000 7f6405b99640 1 -- 172.21.15.40:0/1530017067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64000a2040 msgr2=0x7f64001cb220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:38.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.739+0000 7f6405b99640 1 --2- 172.21.15.40:0/1530017067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64000a2040 0x7f64001cb220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:38.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.739+0000 7f6405b99640 10 monclient: _finish_hunting 0 2024-04-22T08:22:38.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.739+0000 7f6405b99640 1 monclient: found mon.noname-a 2024-04-22T08:22:38.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.739+0000 7f6405b99640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:38.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.739+0000 7f6405b99640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:22:38.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.739+0000 7f6405b99640 1 -- 172.21.15.40:0/1530017067 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f8009a70 con 0x7f64000a1c60 2024-04-22T08:22:38.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.741+0000 7f6405b99640 1 -- 172.21.15.40:0/1530017067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64000a1c60 msgr2=0x7f64001cace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:38.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.741+0000 7f6405b99640 1 --2- 172.21.15.40:0/1530017067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64000a1c60 0x7f64001cace0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f63f8037040 tx=0x7f63f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:38.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.741+0000 7f63fe7fc640 1 -- 172.21.15.40:0/1530017067 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f8040650 con 0x7f64000a1c60 2024-04-22T08:22:38.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.741+0000 7f63fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:38.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.741+0000 7f63fe7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:22:38.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.741+0000 7f63fe7fc640 10 monclient: dump: 2024-04-22T08:22:38.755 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:38.755 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:38.755 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:38.756 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:38.756 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:38.756 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:38.756 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:38.756 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:38.756 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:38.756 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:38.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.741+0000 7f63fe7fc640 10 monclient: _finish_auth 0 2024-04-22T08:22:38.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.741+0000 7f63fe7fc640 10 monclient: _check_auth_tickets 2024-04-22T08:22:38.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.741+0000 7f63fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:38.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.741+0000 7f6407dfd640 5 monclient: authenticate success, global_id 114255 2024-04-22T08:22:38.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.741+0000 7f6407dfd640 10 monclient: _renew_subs 2024-04-22T08:22:38.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.741+0000 7f63fe7fc640 1 -- 172.21.15.40:0/1530017067 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f80407b0 con 0x7f64000a1c60 2024-04-22T08:22:38.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.741+0000 7f6407dfd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:38.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.741+0000 7f6407dfd640 1 -- 172.21.15.40:0/1530017067 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6400121ae0 con 0x7f64000a1c60 2024-04-22T08:22:38.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.741+0000 7f63fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:38.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.741+0000 7f63fe7fc640 1 -- 172.21.15.40:0/1530017067 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f8040b60 con 0x7f64000a1c60 2024-04-22T08:22:38.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.741+0000 7f63fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:38.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.741+0000 7f63fe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:22:38.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.741+0000 7f63fe7fc640 10 monclient: dump: 2024-04-22T08:22:38.757 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:38.757 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:38.757 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:38.758 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:38.758 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:38.758 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:38.758 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:38.758 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:38.758 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:38.758 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:38.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.741+0000 7f63fffff640 4 set_mon_vals no callback set 2024-04-22T08:22:38.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.741+0000 7f6407dfd640 10 monclient: _renew_subs 2024-04-22T08:22:38.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.741+0000 7f6407dfd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:38.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.741+0000 7f6407dfd640 1 -- 172.21.15.40:0/1530017067 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6400120f10 con 0x7f64000a1c60 2024-04-22T08:22:38.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.742+0000 7f6407dfd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:38.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.742+0000 7f6407dfd640 1 --2- 172.21.15.40:0/1530017067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6400120050 0x7f6400120430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:38.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.742+0000 7f6407dfd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:38.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.742+0000 7f6407dfd640 1 -- 172.21.15.40:0/1530017067 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6400123120 con 0x7f6400120050 2024-04-22T08:22:38.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.742+0000 7f63fe7fc640 1 -- 172.21.15.40:0/1530017067 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f63f8040cc0 con 0x7f64000a1c60 2024-04-22T08:22:38.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.742+0000 7f6405398640 1 --2- 172.21.15.40:0/1530017067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6400120050 0x7f6400120430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:38.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.742+0000 7f6405398640 10 monclient: get_auth_request con 0x7f6400120050 auth_method 0 2024-04-22T08:22:38.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.742+0000 7f6405398640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:38.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.742+0000 7f6405398640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:38.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.742+0000 7f6405398640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:38.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.742+0000 7f6405398640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:38.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.742+0000 7f6405398640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:38.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.742+0000 7f6405398640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:38.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.742+0000 7f63fe7fc640 1 --2- 172.21.15.40:0/1530017067 >> v2:172.21.15.189:6800/30168 conn(0x7f63ec037820 0x7f63ec039ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:38.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.742+0000 7f63fe7fc640 1 -- 172.21.15.40:0/1530017067 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(317..317 src has 239..317) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f63f806ce60 con 0x7f64000a1c60 2024-04-22T08:22:38.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.742+0000 7f6405398640 10 monclient(hunting): handle_auth_done global_id 114258 payload 824 2024-04-22T08:22:38.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.742+0000 7f6405398640 1 -- 172.21.15.40:0/1530017067 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f640009fac0 con 0x7f6400120050 2024-04-22T08:22:38.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.743+0000 7f640639a640 1 --2- 172.21.15.40:0/1530017067 >> v2:172.21.15.189:6800/30168 conn(0x7f63ec037820 0x7f63ec039ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:38.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.743+0000 7f640639a640 10 monclient: get_auth_request con 0x7f63ec037820 auth_method 0 2024-04-22T08:22:38.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.745+0000 7f6405398640 1 -- 172.21.15.40:0/1530017067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6400120050 msgr2=0x7f6400120430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:38.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.745+0000 7f6405398640 1 --2- 172.21.15.40:0/1530017067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6400120050 0x7f6400120430 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f64000a6600 tx=0x7f63f400e960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:38.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.745+0000 7f63fe7fc640 1 -- 172.21.15.40:0/1530017067 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4005cf0 con 0x7f6400120050 2024-04-22T08:22:38.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.745+0000 7f63fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:38.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.745+0000 7f63fe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:22:38.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.745+0000 7f63fe7fc640 10 monclient: dump: 2024-04-22T08:22:38.761 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:38.761 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:38.761 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:38.761 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:38.761 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:38.761 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:38.761 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:38.761 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:38.761 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:38.761 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:38.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.745+0000 7f640639a640 1 --2- 172.21.15.40:0/1530017067 >> v2:172.21.15.189:6800/30168 conn(0x7f63ec037820 0x7f63ec039ce0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f63f0007bd0 tx=0x7f63f0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:38.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.745+0000 7f63fe7fc640 1 -- 172.21.15.40:0/1530017067 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f63f4005e50 con 0x7f6400120050 2024-04-22T08:22:38.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.745+0000 7f63fe7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:38.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.745+0000 7f63fe7fc640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:22:38.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.745+0000 7f63fe7fc640 1 -- 172.21.15.40:0/1530017067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6400120050 msgr2=0x7f6400120430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:38.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.745+0000 7f63fe7fc640 1 --2- 172.21.15.40:0/1530017067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6400120050 0x7f6400120430 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f64000a6600 tx=0x7f63f400e960 comp rx=0 tx=0).stop 2024-04-22T08:22:38.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.745+0000 7f63fe7fc640 1 -- 172.21.15.40:0/1530017067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6400120050 msgr2=0x7f6400120430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:22:38.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.745+0000 7f63fe7fc640 1 --2- 172.21.15.40:0/1530017067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6400120050 0x7f6400120430 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:38.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.752+0000 7f6407dfd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:22:38.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.752+0000 7f6407dfd640 1 --2- 172.21.15.40:0/1530017067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f640002b200 0x7f640002b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:38.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.752+0000 7f6407dfd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:38.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.752+0000 7f6407dfd640 1 -- 172.21.15.40:0/1530017067 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6400066650 con 0x7f640002b200 2024-04-22T08:22:38.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.752+0000 7f6405398640 1 --2- 172.21.15.40:0/1530017067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f640002b200 0x7f640002b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:38.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.752+0000 7f6405398640 10 monclient: get_auth_request con 0x7f640002b200 auth_method 0 2024-04-22T08:22:38.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.752+0000 7f6405398640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:38.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.752+0000 7f6405398640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:38.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.752+0000 7f6405398640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:38.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.752+0000 7f6405398640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:38.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.752+0000 7f6405398640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:38.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.752+0000 7f6405398640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:38.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.752+0000 7f6405398640 10 monclient(hunting): handle_auth_done global_id 114261 payload 824 2024-04-22T08:22:38.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.752+0000 7f6405398640 1 -- 172.21.15.40:0/1530017067 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6400122af0 con 0x7f640002b200 2024-04-22T08:22:38.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.754+0000 7f6405398640 1 -- 172.21.15.40:0/1530017067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f640002b200 msgr2=0x7f640002b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:38.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.754+0000 7f6405398640 1 --2- 172.21.15.40:0/1530017067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f640002b200 0x7f640002b610 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f64000a6600 tx=0x7f63f4004940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:38.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.754+0000 7f63fe7fc640 1 -- 172.21.15.40:0/1530017067 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4022070 con 0x7f640002b200 2024-04-22T08:22:38.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.754+0000 7f63fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:38.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.754+0000 7f63fe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:22:38.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.754+0000 7f63fe7fc640 10 monclient: dump: 2024-04-22T08:22:38.766 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:38.766 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:38.766 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:38.766 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:38.766 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:38.766 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:38.766 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:38.766 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:38.766 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:38.766 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:38.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.754+0000 7f63fe7fc640 1 -- 172.21.15.40:0/1530017067 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4633 (secure 0 0 0) 0x7f63f401db10 con 0x7f640002b200 2024-04-22T08:22:38.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.754+0000 7f63fe7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:22:38.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.754+0000 7f63fe7fc640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:22:38.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.754+0000 7f63fe7fc640 1 -- 172.21.15.40:0/1530017067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f640002b200 msgr2=0x7f640002b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:38.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.754+0000 7f63fe7fc640 1 --2- 172.21.15.40:0/1530017067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f640002b200 0x7f640002b610 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f64000a6600 tx=0x7f63f4004940 comp rx=0 tx=0).stop 2024-04-22T08:22:38.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.755+0000 7f63fe7fc640 1 -- 172.21.15.40:0/1530017067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f640002b200 msgr2=0x7f640002b610 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:22:38.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.755+0000 7f63fe7fc640 1 --2- 172.21.15.40:0/1530017067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f640002b200 0x7f640002b610 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:38.767 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:22:38.767 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:22:38.768 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:22:38.768 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "leader", 2024-04-22T08:22:38.768 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 98, 2024-04-22T08:22:38.768 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:22:38.768 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:22:38.768 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:22:38.768 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:22:38.768 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:22:38.768 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 0, 2024-04-22T08:22:38.768 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:22:38.768 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:22:38.769 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:22:38.769 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:22:38.769 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:22:38.769 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:22:38.769 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:22:38.769 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:22:38.769 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:22:38.769 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:22:38.769 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:22:38.769 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:22:38.769 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:22:38.769 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:22:38.770 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:22:38.770 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:22:38.770 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:22:38.770 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:22:38.770 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:22:38.770 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:22:38.770 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:22:38.770 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:22:38.770 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:22:38.770 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:22:38.770 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:22:38.770 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:22:38.771 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:22:38.771 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:38.771 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:22:38.771 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:22:38.771 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:22:38.771 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:22:38.771 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:22:38.771 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:22:38.771 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:22:38.771 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:22:38.771 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:22:38.771 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:22:38.772 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:22:38.772 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:22:38.772 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:22:38.772 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:22:38.772 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:22:38.772 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:22:38.772 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:22:38.772 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:22:38.772 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:22:38.772 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:22:38.772 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:22:38.772 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:22:38.773 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:22:38.773 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:22:38.773 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:22:38.773 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:22:38.773 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:22:38.773 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:22:38.773 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:22:38.773 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:38.773 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:22:38.773 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:38.773 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:22:38.773 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:22:38.773 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:22:38.774 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:22:38.774 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:38.774 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:22:38.774 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:22:38.774 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:22:38.774 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:38.774 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:38.774 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:22:38.774 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:22:38.774 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:22:38.774 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:22:38.774 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:22:38.775 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:38.775 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:22:38.775 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:22:38.775 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:22:38.775 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:22:38.775 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:22:38.775 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:38.775 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:38.775 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:22:38.775 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:22:38.775 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:22:38.775 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:22:38.775 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:38.776 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:22:38.776 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:22:38.776 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:22:38.776 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:38.776 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:38.776 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:22:38.776 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:22:38.776 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:22:38.776 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:22:38.776 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:22:38.776 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:38.777 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:22:38.777 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:22:38.777 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:22:38.777 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:22:38.777 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:22:38.777 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:38.777 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:38.777 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:22:38.777 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:22:38.777 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:22:38.777 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:22:38.777 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:38.778 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:22:38.778 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:22:38.778 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:22:38.778 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:38.778 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:38.778 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:22:38.778 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:22:38.778 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:22:38.778 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:22:38.778 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:22:38.778 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:38.779 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:22:38.779 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:22:38.779 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:22:38.779 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:22:38.779 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:22:38.779 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:22:38.779 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:22:38.779 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:38.779 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:22:38.779 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:22:38.779 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:38.779 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:22:38.780 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:22:38.780 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:22:38.780 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:22:38.780 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:22:38.780 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:22:38.780 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:38.780 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:22:38.780 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:22:38.780 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:22:38.780 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:22:38.780 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:22:38.780 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:22:38.780 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:38.781 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:22:38.781 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:22:38.781 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 2 2024-04-22T08:22:38.781 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:22:38.781 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:22:38.781 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:38.781 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:22:38.781 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:22:38.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.755+0000 7f6407dfd640 1 -- 172.21.15.40:0/1530017067 >> v2:172.21.15.189:6800/30168 conn(0x7f63ec037820 msgr2=0x7f63ec039ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:38.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.755+0000 7f6407dfd640 1 --2- 172.21.15.40:0/1530017067 >> v2:172.21.15.189:6800/30168 conn(0x7f63ec037820 0x7f63ec039ce0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f63f0007bd0 tx=0x7f63f0009040 comp rx=0 tx=0).stop 2024-04-22T08:22:38.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.755+0000 7f6407dfd640 10 monclient: shutdown 2024-04-22T08:22:38.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.755+0000 7f6407dfd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:38.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.755+0000 7f6407dfd640 1 -- 172.21.15.40:0/1530017067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64000a1c60 msgr2=0x7f64001cace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:38.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.755+0000 7f640639a640 1 -- 172.21.15.40:0/1530017067 reap_dead start 2024-04-22T08:22:38.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.755+0000 7f6407dfd640 1 --2- 172.21.15.40:0/1530017067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64000a1c60 0x7f64001cace0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f63f8037040 tx=0x7f63f8004a40 comp rx=0 tx=0).stop 2024-04-22T08:22:38.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.755+0000 7f6407dfd640 1 -- 172.21.15.40:0/1530017067 shutdown_connections 2024-04-22T08:22:38.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.755+0000 7f6407dfd640 1 --2- 172.21.15.40:0/1530017067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64000a1c60 0x7f64001cace0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:38.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.755+0000 7f6407dfd640 1 -- 172.21.15.40:0/1530017067 >> 172.21.15.40:0/1530017067 conn(0x7f64000a89a0 msgr2=0x7f640009de90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:38.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.755+0000 7f6407dfd640 1 -- 172.21.15.40:0/1530017067 shutdown_connections 2024-04-22T08:22:38.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.755+0000 7f6407dfd640 1 -- 172.21.15.40:0/1530017067 wait complete. 2024-04-22T08:22:38.784 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:22:38.784 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-22T08:22:38.784 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-22T08:22:38.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.884+0000 7f61f5ce2640 10 monclient: get_monmap_and_config 2024-04-22T08:22:38.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.884+0000 7f61f5ce2640 10 monclient: build_initial_monmap 2024-04-22T08:22:38.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.885+0000 7f61f5ce2640 10 monclient: monmap: 2024-04-22T08:22:38.884 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:38.884 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:38.884 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:22:38.885776+0000 2024-04-22T08:22:38.884 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:22:38.885776+0000 2024-04-22T08:22:38.885 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:38.885 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:38.885 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:22:38.885 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:22:38.885 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:22:38.885 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:22:38.885 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:22:38.885 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:38.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.885+0000 7f61f5ce2640 1 Processor -- start 2024-04-22T08:22:38.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.885+0000 7f61f5ce2640 1 -- start start 2024-04-22T08:22:38.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.885+0000 7f61f5ce2640 10 monclient: init 2024-04-22T08:22:38.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.886+0000 7f61f5ce2640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:38.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.886+0000 7f61f5ce2640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-04-22T08:22:38.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.886+0000 7f61f5ce2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f61f00a5a10 addr v1:172.21.15.40:6789/0 2024-04-22T08:22:38.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.886+0000 7f61f5ce2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f61f00a32b0 addr v1:172.21.15.189:6789/0 2024-04-22T08:22:38.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.886+0000 7f61f5ce2640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f61f013c660 0x7f61f013ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:38.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.886+0000 7f61f5ce2640 10 monclient(hunting): picked mon.noname-c con 0x7f61f013c660 addr v2:172.21.15.189:3300/0 2024-04-22T08:22:38.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.886+0000 7f61f5ce2640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f009fe80 con 0x7f61f00a5a10 2024-04-22T08:22:38.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.886+0000 7f61f5ce2640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f00a25b0 con 0x7f61f00a32b0 2024-04-22T08:22:38.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.886+0000 7f61f5ce2640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:38.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.886+0000 7f61f5ce2640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61f009fba0 con 0x7f61f013c660 2024-04-22T08:22:38.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.886+0000 7f61f5ce2640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:38.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.886+0000 7f61f5ce2640 10 monclient(hunting): authenticate will time out at 1432.880066s 2024-04-22T08:22:38.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.886+0000 7f61ef7fe640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f61f00a5a10 0x7f61f00a1970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:49278/0 (socket says 172.21.15.40:49278) 2024-04-22T08:22:38.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.886+0000 7f61ef7fe640 1 -- 172.21.15.40:0/1081233383 learned_addr learned my addr 172.21.15.40:0/1081233383 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:22:38.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.886+0000 7f61effff640 1 --2- 172.21.15.40:0/1081233383 >> v2:172.21.15.189:3300/0 conn(0x7f61f013c660 0x7f61f013ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:38.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.886+0000 7f61effff640 10 monclient(hunting): get_auth_request con 0x7f61f013c660 auth_method 0 2024-04-22T08:22:38.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.886+0000 7f61effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:38.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.886+0000 7f61effff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:38.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.886+0000 7f61effff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:38.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.886+0000 7f61effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:38.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.886+0000 7f61effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:38.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.886+0000 7f61effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:38.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.886+0000 7f61effff640 10 monclient(hunting): handle_auth_done global_id 65242 payload 274 2024-04-22T08:22:38.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.886+0000 7f61effff640 1 -- 172.21.15.40:0/1081233383 >> v1:172.21.15.189:6789/0 conn(0x7f61f00a32b0 legacy=0x7f61f013af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:38.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.886+0000 7f61effff640 1 -- 172.21.15.40:0/1081233383 >> v1:172.21.15.40:6789/0 conn(0x7f61f00a5a10 legacy=0x7f61f00a1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:38.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.886+0000 7f61effff640 10 monclient: _finish_hunting 0 2024-04-22T08:22:38.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.886+0000 7f61effff640 1 monclient: found mon.noname-c 2024-04-22T08:22:38.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.886+0000 7f61effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:38.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.886+0000 7f61effff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.189:3300/0 2024-04-22T08:22:38.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.886+0000 7f61effff640 1 -- 172.21.15.40:0/1081233383 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f00a0160 con 0x7f61f013c660 2024-04-22T08:22:38.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.889+0000 7f61effff640 1 -- 172.21.15.40:0/1081233383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61f013c660 msgr2=0x7f61f013ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:38.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.889+0000 7f61effff640 1 --2- 172.21.15.40:0/1081233383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61f013c660 0x7f61f013ea50 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f61e8009d90 tx=0x7f61e802bd00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44ce5503f926ec55 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:38.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.890+0000 7f61ee7fc640 1 -- 172.21.15.40:0/1081233383 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61e802cda0 con 0x7f61f013c660 2024-04-22T08:22:38.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.890+0000 7f61ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:38.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.890+0000 7f61ee7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:22:38.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.890+0000 7f61ee7fc640 10 monclient: dump: 2024-04-22T08:22:38.897 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:38.897 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:38.897 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:38.897 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:38.898 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:38.898 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:38.898 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:38.898 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:38.898 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:38.898 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:38.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.890+0000 7f61ee7fc640 10 monclient: _finish_auth 0 2024-04-22T08:22:38.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.890+0000 7f61ee7fc640 10 monclient: _check_auth_tickets 2024-04-22T08:22:38.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.890+0000 7f61ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:38.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.890+0000 7f61f5ce2640 5 monclient: authenticate success, global_id 65242 2024-04-22T08:22:38.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.890+0000 7f61ee7fc640 1 -- 172.21.15.40:0/1081233383 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61e8034440 con 0x7f61f013c660 2024-04-22T08:22:38.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.890+0000 7f61f5ce2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:22:38.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.890+0000 7f61ee7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:38.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.890+0000 7f61f5ce2640 10 monclient: get_monmap_and_config success 2024-04-22T08:22:38.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.890+0000 7f61f5ce2640 4 set_mon_vals no callback set 2024-04-22T08:22:38.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.890+0000 7f61ee7fc640 1 -- 172.21.15.40:0/1081233383 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61e80347d0 con 0x7f61f013c660 2024-04-22T08:22:38.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.890+0000 7f61ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:38.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.890+0000 7f61ee7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:22:38.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.890+0000 7f61ee7fc640 10 monclient: dump: 2024-04-22T08:22:38.899 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:38.899 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:38.899 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:38.899 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:38.899 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:38.900 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:38.900 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:38.900 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:38.900 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:38.900 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:38.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.890+0000 7f61f5ce2640 10 monclient: shutdown 2024-04-22T08:22:38.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.890+0000 7f61f5ce2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:38.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.890+0000 7f61f5ce2640 1 -- 172.21.15.40:0/1081233383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61f013c660 msgr2=0x7f61f013ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:38.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.890+0000 7f61f5ce2640 1 --2- 172.21.15.40:0/1081233383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61f013c660 0x7f61f013ea50 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f61e8009d90 tx=0x7f61e802bd00 comp rx=0 tx=0).stop 2024-04-22T08:22:38.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.890+0000 7f61f5ce2640 1 -- 172.21.15.40:0/1081233383 shutdown_connections 2024-04-22T08:22:38.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.890+0000 7f61f5ce2640 1 --2- 172.21.15.40:0/1081233383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61f013c660 0x7f61f013ea50 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:38.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.890+0000 7f61f5ce2640 1 -- 172.21.15.40:0/1081233383 >> 172.21.15.40:0/1081233383 conn(0x7f61f00a4ef0 msgr2=0x7f61f00a5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:38.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.890+0000 7f61f5ce2640 1 -- 172.21.15.40:0/1081233383 shutdown_connections 2024-04-22T08:22:38.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.890+0000 7f61f5ce2640 1 -- 172.21.15.40:0/1081233383 wait complete. 2024-04-22T08:22:38.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.891+0000 7f61f5ce2640 10 monclient: build_initial_monmap 2024-04-22T08:22:38.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.891+0000 7f61f5ce2640 10 monclient: monmap: 2024-04-22T08:22:38.901 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:38.901 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:22:38.901 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:22:38.901 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:22:38.901 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:38.901 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:38.901 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:22:38.901 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:22:38.902 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:22:38.902 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:38.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.891+0000 7f61f5ce2640 1 Processor -- start 2024-04-22T08:22:38.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.891+0000 7f61f5ce2640 1 -- start start 2024-04-22T08:22:38.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.891+0000 7f61f5ce2640 10 monclient: init 2024-04-22T08:22:38.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61f5ce2640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:38.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61f5ce2640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-22T08:22:38.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61f5ce2640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61f00a32b0 0x7f61f0135100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:38.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61f5ce2640 10 monclient(hunting): picked mon.noname-b con 0x7f61f00a32b0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:22:38.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61f5ce2640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61f00a5a10 0x7f61f0135640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:38.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61f5ce2640 10 monclient(hunting): picked mon.noname-a con 0x7f61f00a5a10 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:22:38.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61f5ce2640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61f013c660 0x7f61f0135b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:38.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61f5ce2640 10 monclient(hunting): picked mon.noname-c con 0x7f61f013c660 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:22:38.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61f5ce2640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:38.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61f5ce2640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61f00a27c0 con 0x7f61f00a5a10 2024-04-22T08:22:38.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61f5ce2640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:38.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61f5ce2640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61f0138770 con 0x7f61f00a32b0 2024-04-22T08:22:38.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61f5ce2640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:38.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61f5ce2640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61f0136cf0 con 0x7f61f013c660 2024-04-22T08:22:38.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61f5ce2640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:38.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61f5ce2640 10 monclient(hunting): authenticate will time out at 1432.885920s 2024-04-22T08:22:38.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61eeffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61f00a5a10 0x7f61f0135640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:38.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61effff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61f013c660 0x7f61f0135b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:38.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61eeffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61f00a5a10 0x7f61f0135640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44406/0 (socket says 172.21.15.40:44406) 2024-04-22T08:22:38.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61eeffd640 1 -- 172.21.15.40:0/2997159024 learned_addr learned my addr 172.21.15.40:0/2997159024 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:22:38.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61eeffd640 10 monclient(hunting): get_auth_request con 0x7f61f00a5a10 auth_method 0 2024-04-22T08:22:38.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61eeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:38.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61eeffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:38.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:38.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61effff640 10 monclient(hunting): get_auth_request con 0x7f61f013c660 auth_method 0 2024-04-22T08:22:38.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:38.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61effff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:38.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61effff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:38.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61eeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:38.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61eeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:38.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61ef7fe640 1 --2- 172.21.15.40:0/2997159024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61f00a32b0 0x7f61f0135100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:38.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61eeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:38.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61ef7fe640 10 monclient(hunting): get_auth_request con 0x7f61f00a32b0 auth_method 0 2024-04-22T08:22:38.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:38.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61ef7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:38.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:38.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61effff640 1 -- 172.21.15.40:0/2997159024 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61f013c660 msgr2=0x7f61f0135b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-22T08:22:38.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61effff640 1 -- 172.21.15.40:0/2997159024 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61f013c660 msgr2=0x7f61f0135b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:22:38.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61effff640 1 --2- 172.21.15.40:0/2997159024 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61f013c660 0x7f61f0135b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:22:38.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61effff640 1 --2- 172.21.15.40:0/2997159024 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61f013c660 0x7f61f0135b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:22:38.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61ef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:38.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61ef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:38.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61ef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:38.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61eeffd640 10 monclient(hunting): handle_auth_done global_id 114264 payload 824 2024-04-22T08:22:38.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61eeffd640 1 -- 172.21.15.40:0/2997159024 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61f013c660 msgr2=0x7f61f0135b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:22:38.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61eeffd640 1 --2- 172.21.15.40:0/2997159024 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61f013c660 0x7f61f0135b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:38.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61eeffd640 1 -- 172.21.15.40:0/2997159024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61f00a32b0 msgr2=0x7f61f0135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:38.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61eeffd640 1 --2- 172.21.15.40:0/2997159024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61f00a32b0 0x7f61f0135100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:38.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61eeffd640 10 monclient: _finish_hunting 0 2024-04-22T08:22:38.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61eeffd640 1 monclient: found mon.noname-a 2024-04-22T08:22:38.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:38.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61eeffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:22:38.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.892+0000 7f61eeffd640 1 -- 172.21.15.40:0/2997159024 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e8009a70 con 0x7f61f00a5a10 2024-04-22T08:22:38.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.894+0000 7f61eeffd640 1 -- 172.21.15.40:0/2997159024 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61f00a5a10 msgr2=0x7f61f0135640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:38.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.894+0000 7f61eeffd640 1 --2- 172.21.15.40:0/2997159024 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61f00a5a10 0x7f61f0135640 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f61e400c460 tx=0x7f61e400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:38.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.895+0000 7f61d7fff640 1 -- 172.21.15.40:0/2997159024 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61e4012a40 con 0x7f61f00a5a10 2024-04-22T08:22:38.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.895+0000 7f61d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:38.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.895+0000 7f61d7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:22:38.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.895+0000 7f61d7fff640 10 monclient: dump: 2024-04-22T08:22:38.907 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:38.907 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:38.907 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:38.907 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:38.907 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:38.907 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:38.908 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:38.908 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:38.908 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:38.908 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:38.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.895+0000 7f61d7fff640 10 monclient: _finish_auth 0 2024-04-22T08:22:38.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.895+0000 7f61d7fff640 10 monclient: _check_auth_tickets 2024-04-22T08:22:38.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.895+0000 7f61d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:38.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.895+0000 7f61f5ce2640 5 monclient: authenticate success, global_id 114264 2024-04-22T08:22:38.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.895+0000 7f61d7fff640 1 -- 172.21.15.40:0/2997159024 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61e4012be0 con 0x7f61f00a5a10 2024-04-22T08:22:38.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.895+0000 7f61d7fff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:38.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.895+0000 7f61d7fff640 1 -- 172.21.15.40:0/2997159024 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61e4011380 con 0x7f61f00a5a10 2024-04-22T08:22:38.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.895+0000 7f61f5ce2640 10 monclient: _renew_subs 2024-04-22T08:22:38.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.895+0000 7f61f5ce2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:38.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.895+0000 7f61f5ce2640 1 -- 172.21.15.40:0/2997159024 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61f013c270 con 0x7f61f00a5a10 2024-04-22T08:22:38.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.895+0000 7f61d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:38.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.895+0000 7f61edffb640 4 set_mon_vals no callback set 2024-04-22T08:22:38.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.895+0000 7f61d7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:22:38.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.895+0000 7f61d7fff640 10 monclient: dump: 2024-04-22T08:22:38.909 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:38.909 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:38.909 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:38.909 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:38.909 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:38.909 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:38.909 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:38.910 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:38.910 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:38.910 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:38.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.895+0000 7f61f5ce2640 10 monclient: _renew_subs 2024-04-22T08:22:38.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.895+0000 7f61f5ce2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:38.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.895+0000 7f61f5ce2640 1 -- 172.21.15.40:0/2997159024 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61f0137af0 con 0x7f61f00a5a10 2024-04-22T08:22:38.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.895+0000 7f61d7fff640 1 -- 172.21.15.40:0/2997159024 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f61e4011520 con 0x7f61f00a5a10 2024-04-22T08:22:38.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.896+0000 7f61f5ce2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:38.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.896+0000 7f61f5ce2640 1 --2- 172.21.15.40:0/2997159024 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61f013d0d0 0x7f61f013d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:38.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.896+0000 7f61f5ce2640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:38.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.896+0000 7f61f5ce2640 1 -- 172.21.15.40:0/2997159024 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61f01384b0 con 0x7f61f013d0d0 2024-04-22T08:22:38.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.896+0000 7f61d7fff640 1 --2- 172.21.15.40:0/2997159024 >> v2:172.21.15.189:6800/30168 conn(0x7f61dc0377d0 0x7f61dc039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:38.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.896+0000 7f61ef7fe640 1 --2- 172.21.15.40:0/2997159024 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61f013d0d0 0x7f61f013d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:38.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.896+0000 7f61ef7fe640 10 monclient: get_auth_request con 0x7f61f013d0d0 auth_method 0 2024-04-22T08:22:38.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.896+0000 7f61ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:38.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.896+0000 7f61d7fff640 1 -- 172.21.15.40:0/2997159024 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(317..317 src has 239..317) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f61e4049ea0 con 0x7f61f00a5a10 2024-04-22T08:22:38.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.896+0000 7f61ef7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:38.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.896+0000 7f61ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:38.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.896+0000 7f61ef7fe640 1 -- 172.21.15.40:0/2997159024 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61f013d0d0 msgr2=0x7f61f013d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-22T08:22:38.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.896+0000 7f61ef7fe640 1 -- 172.21.15.40:0/2997159024 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61f013d0d0 msgr2=0x7f61f013d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:22:38.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.896+0000 7f61ef7fe640 1 --2- 172.21.15.40:0/2997159024 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61f013d0d0 0x7f61f013d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:22:38.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.896+0000 7f61ef7fe640 1 --2- 172.21.15.40:0/2997159024 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61f013d0d0 0x7f61f013d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:22:38.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.896+0000 7f61effff640 1 --2- 172.21.15.40:0/2997159024 >> v2:172.21.15.189:6800/30168 conn(0x7f61dc0377d0 0x7f61dc039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:38.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.896+0000 7f61effff640 10 monclient: get_auth_request con 0x7f61dc0377d0 auth_method 0 2024-04-22T08:22:38.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:38.899+0000 7f61effff640 1 --2- 172.21.15.40:0/2997159024 >> v2:172.21.15.189:6800/30168 conn(0x7f61dc0377d0 0x7f61dc039c90 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f61e8033000 tx=0x7f61e8030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:39.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:39.097+0000 7f61ef7fe640 1 --2- 172.21.15.40:0/2997159024 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61f013d0d0 0x7f61f013d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:39.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:39.097+0000 7f61ef7fe640 10 monclient: get_auth_request con 0x7f61f013d0d0 auth_method 0 2024-04-22T08:22:39.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:39.097+0000 7f61ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:39.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:39.097+0000 7f61ef7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:39.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:39.097+0000 7f61ef7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:22:39.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:39.097+0000 7f61ef7fe640 1 -- 172.21.15.40:0/2997159024 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61f013d0d0 msgr2=0x7f61f013d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-22T08:22:39.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:39.097+0000 7f61ef7fe640 1 -- 172.21.15.40:0/2997159024 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61f013d0d0 msgr2=0x7f61f013d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:22:39.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:39.097+0000 7f61ef7fe640 1 --2- 172.21.15.40:0/2997159024 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61f013d0d0 0x7f61f013d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:22:39.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:39.097+0000 7f61ef7fe640 1 --2- 172.21.15.40:0/2997159024 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61f013d0d0 0x7f61f013d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T08:22:39.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:39.498+0000 7f61ef7fe640 1 --2- 172.21.15.40:0/2997159024 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61f013d0d0 0x7f61f013d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:39.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:39.498+0000 7f61ef7fe640 10 monclient: get_auth_request con 0x7f61f013d0d0 auth_method 0 2024-04-22T08:22:39.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:39.498+0000 7f61ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:39.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:39.498+0000 7f61ef7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:39.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:39.498+0000 7f61ef7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:22:39.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:39.498+0000 7f61ef7fe640 1 -- 172.21.15.40:0/2997159024 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61f013d0d0 msgr2=0x7f61f013d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-22T08:22:39.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:39.498+0000 7f61ef7fe640 1 -- 172.21.15.40:0/2997159024 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61f013d0d0 msgr2=0x7f61f013d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:22:39.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:39.498+0000 7f61ef7fe640 1 --2- 172.21.15.40:0/2997159024 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61f013d0d0 0x7f61f013d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:22:39.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:39.498+0000 7f61ef7fe640 1 --2- 172.21.15.40:0/2997159024 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61f013d0d0 0x7f61f013d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T08:22:40.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.299+0000 7f61ef7fe640 1 --2- 172.21.15.40:0/2997159024 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61f013d0d0 0x7f61f013d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:40.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.299+0000 7f61ef7fe640 10 monclient: get_auth_request con 0x7f61f013d0d0 auth_method 0 2024-04-22T08:22:40.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.299+0000 7f61ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:40.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.299+0000 7f61ef7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:40.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.299+0000 7f61ef7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:22:40.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.299+0000 7f61ef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:40.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.299+0000 7f61ef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:40.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.299+0000 7f61ef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:40.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.299+0000 7f61ef7fe640 10 monclient(hunting): handle_auth_done global_id 124100 payload 824 2024-04-22T08:22:40.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.299+0000 7f61ef7fe640 1 -- 172.21.15.40:0/2997159024 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f61f0138ae0 con 0x7f61f013d0d0 2024-04-22T08:22:40.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.302+0000 7f61ef7fe640 1 -- 172.21.15.40:0/2997159024 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61f013d0d0 msgr2=0x7f61f013d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:40.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.302+0000 7f61ef7fe640 1 --2- 172.21.15.40:0/2997159024 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61f013d0d0 0x7f61f013d4b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f61e00149c0 tx=0x7f61e0014e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:40.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.302+0000 7f61d7fff640 1 -- 172.21.15.40:0/2997159024 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61e0012d80 con 0x7f61f013d0d0 2024-04-22T08:22:40.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.302+0000 7f61d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:40.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.302+0000 7f61d7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:22:40.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.302+0000 7f61d7fff640 10 monclient: dump: 2024-04-22T08:22:40.302 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:40.302 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:40.302 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:40.302 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:40.302 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:40.302 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:40.302 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:40.302 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:40.302 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:40.302 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:40.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.302+0000 7f61d7fff640 1 -- 172.21.15.40:0/2997159024 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7f61e0017460 con 0x7f61f013d0d0 2024-04-22T08:22:40.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.302+0000 7f61d7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:40.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.302+0000 7f61d7fff640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:22:40.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.302+0000 7f61d7fff640 1 -- 172.21.15.40:0/2997159024 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61f013d0d0 msgr2=0x7f61f013d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:40.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.302+0000 7f61d7fff640 1 --2- 172.21.15.40:0/2997159024 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61f013d0d0 0x7f61f013d4b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f61e00149c0 tx=0x7f61e0014e80 comp rx=0 tx=0).stop 2024-04-22T08:22:40.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.302+0000 7f61d7fff640 1 -- 172.21.15.40:0/2997159024 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61f013d0d0 msgr2=0x7f61f013d4b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:22:40.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.302+0000 7f61d7fff640 1 --2- 172.21.15.40:0/2997159024 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61f013d0d0 0x7f61f013d4b0 secure :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f61e00149c0 tx=0x7f61e0014e80 comp rx=0 tx=0).stop 2024-04-22T08:22:40.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.310+0000 7f61f5ce2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:22:40.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.310+0000 7f61f5ce2640 1 --2- 172.21.15.40:0/2997159024 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61f013d9f0 0x7f61f01d4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:40.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.310+0000 7f61f5ce2640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:40.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.310+0000 7f61f5ce2640 1 -- 172.21.15.40:0/2997159024 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61f00667a0 con 0x7f61f013d9f0 2024-04-22T08:22:40.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.310+0000 7f61ef7fe640 1 --2- 172.21.15.40:0/2997159024 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61f013d9f0 0x7f61f01d4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:40.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.310+0000 7f61ef7fe640 10 monclient: get_auth_request con 0x7f61f013d9f0 auth_method 0 2024-04-22T08:22:40.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.310+0000 7f61ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:40.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.310+0000 7f61ef7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:40.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.310+0000 7f61ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:40.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.310+0000 7f61ef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:40.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.310+0000 7f61ef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:40.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.310+0000 7f61ef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:40.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.310+0000 7f61ef7fe640 10 monclient(hunting): handle_auth_done global_id 124103 payload 824 2024-04-22T08:22:40.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.310+0000 7f61ef7fe640 1 -- 172.21.15.40:0/2997159024 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f61f00a2080 con 0x7f61f013d9f0 2024-04-22T08:22:40.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.312+0000 7f61ef7fe640 1 -- 172.21.15.40:0/2997159024 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61f013d9f0 msgr2=0x7f61f01d4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:40.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.313+0000 7f61ef7fe640 1 --2- 172.21.15.40:0/2997159024 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61f013d9f0 0x7f61f01d4ee0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f61e001e000 tx=0x7f61e0007f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:40.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.313+0000 7f61d7fff640 1 -- 172.21.15.40:0/2997159024 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61e0027070 con 0x7f61f013d9f0 2024-04-22T08:22:40.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.313+0000 7f61d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:40.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.313+0000 7f61d7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:22:40.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.313+0000 7f61d7fff640 10 monclient: dump: 2024-04-22T08:22:40.313 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:40.313 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:40.313 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:40.313 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:40.313 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:40.313 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:40.313 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:40.313 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:40.314 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:40.314 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:40.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.313+0000 7f61d7fff640 1 -- 172.21.15.40:0/2997159024 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4461 (secure 0 0 0) 0x7f61e00218f0 con 0x7f61f013d9f0 2024-04-22T08:22:40.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.313+0000 7f61d7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:22:40.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.313+0000 7f61d7fff640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:22:40.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.313+0000 7f61d7fff640 1 -- 172.21.15.40:0/2997159024 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61f013d9f0 msgr2=0x7f61f01d4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:40.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.313+0000 7f61d7fff640 1 --2- 172.21.15.40:0/2997159024 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61f013d9f0 0x7f61f01d4ee0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f61e001e000 tx=0x7f61e0007f20 comp rx=0 tx=0).stop 2024-04-22T08:22:40.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.313+0000 7f61d7fff640 1 -- 172.21.15.40:0/2997159024 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61f013d9f0 msgr2=0x7f61f01d4ee0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:22:40.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.313+0000 7f61d7fff640 1 --2- 172.21.15.40:0/2997159024 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61f013d9f0 0x7f61f01d4ee0 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f61e001e000 tx=0x7f61e0007f20 comp rx=0 tx=0).stop 2024-04-22T08:22:40.314 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:22:40.315 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:22:40.315 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:22:40.315 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:22:40.315 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 98, 2024-04-22T08:22:40.315 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:22:40.315 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:22:40.315 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:22:40.315 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:22:40.315 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:22:40.315 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 1, 2024-04-22T08:22:40.315 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:22:40.315 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:22:40.315 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:22:40.316 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:22:40.316 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:22:40.316 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:22:40.316 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:22:40.316 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:22:40.316 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:22:40.316 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:22:40.316 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:22:40.316 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:22:40.316 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:22:40.316 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:22:40.316 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:22:40.316 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:22:40.316 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:22:40.317 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:22:40.317 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:22:40.317 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:22:40.317 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:22:40.317 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:22:40.317 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:22:40.317 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:22:40.317 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:22:40.317 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:22:40.317 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:22:40.317 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:40.317 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:22:40.317 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:22:40.317 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:22:40.318 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:22:40.318 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:22:40.318 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:22:40.318 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:22:40.318 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:22:40.318 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:22:40.318 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:22:40.318 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:22:40.318 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:22:40.318 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:22:40.318 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:22:40.318 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:22:40.318 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:22:40.318 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:22:40.319 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:22:40.319 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:22:40.319 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:22:40.319 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:22:40.319 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:22:40.319 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:22:40.319 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:22:40.319 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:22:40.319 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:22:40.319 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:22:40.319 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:22:40.319 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:22:40.319 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:40.319 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:22:40.319 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:40.320 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:22:40.320 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:22:40.320 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:22:40.320 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:22:40.320 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:40.320 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:22:40.320 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:22:40.320 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:22:40.320 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:40.320 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:40.320 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:22:40.320 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:22:40.320 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:22:40.320 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:22:40.321 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:22:40.321 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:40.321 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:22:40.321 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:22:40.321 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:22:40.321 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:22:40.321 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:22:40.321 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:40.321 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:40.321 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:22:40.321 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:22:40.321 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:22:40.321 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:22:40.321 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:40.322 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:22:40.322 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:22:40.322 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:22:40.322 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:40.322 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:40.322 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:22:40.322 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:22:40.322 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:22:40.322 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:22:40.322 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:22:40.322 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:40.322 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:22:40.322 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:22:40.323 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:22:40.323 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:22:40.323 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:22:40.323 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:40.323 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:40.323 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:22:40.323 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:22:40.323 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:22:40.323 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:22:40.323 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:40.323 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:22:40.323 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:22:40.323 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:22:40.323 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:40.323 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:40.324 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:22:40.324 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:22:40.324 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:22:40.324 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:22:40.324 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:22:40.324 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:40.324 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:22:40.324 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:22:40.324 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:22:40.324 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:22:40.324 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:22:40.324 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:22:40.324 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:22:40.324 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:40.324 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:22:40.325 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:22:40.325 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:40.325 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:22:40.325 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:22:40.325 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:22:40.325 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:22:40.325 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:22:40.325 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:22:40.325 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:40.325 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:22:40.325 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:22:40.325 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:22:40.325 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:22:40.325 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:22:40.325 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:40.326 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:22:40.326 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:22:40.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.314+0000 7f61f5ce2640 1 -- 172.21.15.40:0/2997159024 >> v2:172.21.15.189:6800/30168 conn(0x7f61dc0377d0 msgr2=0x7f61dc039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:40.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.314+0000 7f61f5ce2640 1 --2- 172.21.15.40:0/2997159024 >> v2:172.21.15.189:6800/30168 conn(0x7f61dc0377d0 0x7f61dc039c90 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f61e8033000 tx=0x7f61e8030000 comp rx=0 tx=0).stop 2024-04-22T08:22:40.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.314+0000 7f61f5ce2640 10 monclient: shutdown 2024-04-22T08:22:40.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.314+0000 7f61f5ce2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:40.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.314+0000 7f61f5ce2640 1 -- 172.21.15.40:0/2997159024 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61f00a5a10 msgr2=0x7f61f0135640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:40.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.314+0000 7f61f5ce2640 1 --2- 172.21.15.40:0/2997159024 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61f00a5a10 0x7f61f0135640 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f61e400c460 tx=0x7f61e400c920 comp rx=0 tx=0).stop 2024-04-22T08:22:40.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.314+0000 7f61effff640 1 -- 172.21.15.40:0/2997159024 reap_dead start 2024-04-22T08:22:40.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.314+0000 7f61effff640 1 -- 172.21.15.40:0/2997159024 reap_dead start 2024-04-22T08:22:40.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.314+0000 7f61f5ce2640 1 -- 172.21.15.40:0/2997159024 shutdown_connections 2024-04-22T08:22:40.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.314+0000 7f61f5ce2640 1 -- 172.21.15.40:0/2997159024 >> 172.21.15.40:0/2997159024 conn(0x7f61f00a4ef0 msgr2=0x7f61f00a1970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:40.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.314+0000 7f61f5ce2640 1 -- 172.21.15.40:0/2997159024 shutdown_connections 2024-04-22T08:22:40.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.314+0000 7f61f5ce2640 1 -- 172.21.15.40:0/2997159024 wait complete. 2024-04-22T08:22:40.328 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:22:40.328 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-22T08:22:40.328 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-22T08:22:40.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.437+0000 7fc40ccce640 10 monclient: get_monmap_and_config 2024-04-22T08:22:40.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.437+0000 7fc40ccce640 10 monclient: build_initial_monmap 2024-04-22T08:22:40.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.437+0000 7fc40ccce640 10 monclient: monmap: 2024-04-22T08:22:40.437 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:40.437 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:40.437 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:22:40.438010+0000 2024-04-22T08:22:40.437 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:22:40.438010+0000 2024-04-22T08:22:40.437 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:40.437 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:40.437 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:22:40.437 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:22:40.437 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:22:40.437 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:22:40.437 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:22:40.437 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:40.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.437+0000 7fc40ccce640 1 Processor -- start 2024-04-22T08:22:40.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.437+0000 7fc40ccce640 1 -- start start 2024-04-22T08:22:40.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.437+0000 7fc40ccce640 10 monclient: init 2024-04-22T08:22:40.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.438+0000 7fc40ccce640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:40.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.438+0000 7fc40ccce640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-04-22T08:22:40.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.438+0000 7fc40ccce640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc4080a1c60 addr v1:172.21.15.189:6789/0 2024-04-22T08:22:40.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.438+0000 7fc40ccce640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc4080a31d0 0x7fc40813aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:40.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.438+0000 7fc40ccce640 10 monclient(hunting): picked mon.noname-b con 0x7fc4080a31d0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:22:40.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.438+0000 7fc40ccce640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc40813b410 0x7fc40813d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:40.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.438+0000 7fc40ccce640 10 monclient(hunting): picked mon.noname-a con 0x7fc40813b410 addr v2:172.21.15.40:3300/0 2024-04-22T08:22:40.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.438+0000 7fc40ccce640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4080a41f0 con 0x7fc4080a1c60 2024-04-22T08:22:40.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.438+0000 7fc40ccce640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:40.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.438+0000 7fc40ccce640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4080a3630 con 0x7fc40813b410 2024-04-22T08:22:40.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.438+0000 7fc40ccce640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:40.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.438+0000 7fc40ccce640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc40809fe90 con 0x7fc4080a31d0 2024-04-22T08:22:40.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.438+0000 7fc40ccce640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:40.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.438+0000 7fc40ccce640 10 monclient(hunting): authenticate will time out at 1434.432301s 2024-04-22T08:22:40.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.438+0000 7fc406d9d640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc40813b410 0x7fc40813d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:40.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.438+0000 7fc405d9b640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc4080a31d0 0x7fc40813aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:40.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.438+0000 7fc406d9d640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc40813b410 0x7fc40813d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44422/0 (socket says 172.21.15.40:44422) 2024-04-22T08:22:40.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.438+0000 7fc406d9d640 1 -- 172.21.15.40:0/1883697354 learned_addr learned my addr 172.21.15.40:0/1883697354 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:22:40.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.438+0000 7fc405d9b640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc4080a31d0 0x7fc40813aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49712/0 (socket says 172.21.15.40:49712) 2024-04-22T08:22:40.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.438+0000 7fc406d9d640 10 monclient(hunting): get_auth_request con 0x7fc40813b410 auth_method 0 2024-04-22T08:22:40.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.438+0000 7fc406d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:40.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.438+0000 7fc406d9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:40.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.438+0000 7fc406d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:40.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.438+0000 7fc405d9b640 10 monclient(hunting): get_auth_request con 0x7fc4080a31d0 auth_method 0 2024-04-22T08:22:40.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.438+0000 7fc405d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:40.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.438+0000 7fc405d9b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:40.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.438+0000 7fc405d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:40.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.438+0000 7fc406d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:40.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.438+0000 7fc406d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:40.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.438+0000 7fc406d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:40.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.438+0000 7fc405d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:40.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.438+0000 7fc405d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:40.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.438+0000 7fc405d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:40.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.438+0000 7fc405d9b640 10 monclient(hunting): handle_auth_done global_id 124106 payload 274 2024-04-22T08:22:40.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.439+0000 7fc405d9b640 1 -- 172.21.15.40:0/1883697354 >> v2:172.21.15.40:3300/0 conn(0x7fc40813b410 msgr2=0x7fc40813d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:40.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.439+0000 7fc405d9b640 1 --2- 172.21.15.40:0/1883697354 >> v2:172.21.15.40:3300/0 conn(0x7fc40813b410 0x7fc40813d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:40.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.439+0000 7fc405d9b640 1 -- 172.21.15.40:0/1883697354 >> v1:172.21.15.189:6789/0 conn(0x7fc4080a1c60 legacy=0x7fc4080a2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:40.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.439+0000 7fc405d9b640 10 monclient: _finish_hunting 0 2024-04-22T08:22:40.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.439+0000 7fc405d9b640 1 monclient: found mon.noname-b 2024-04-22T08:22:40.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.439+0000 7fc405d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:40.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.439+0000 7fc405d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:22:40.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.439+0000 7fc405d9b640 1 -- 172.21.15.40:0/1883697354 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4080a2750 con 0x7fc4080a31d0 2024-04-22T08:22:40.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.439+0000 7fc406d9d640 1 --2- 172.21.15.40:0/1883697354 >> v2:172.21.15.40:3300/0 conn(0x7fc40813b410 0x7fc40813d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:22:40.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.441+0000 7fc405d9b640 1 -- 172.21.15.40:0/1883697354 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc4080a31d0 msgr2=0x7fc40813aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:40.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.441+0000 7fc405d9b640 1 --2- 172.21.15.40:0/1883697354 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc4080a31d0 0x7fc40813aed0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc3f00097a0 tx=0x7fc3f002b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38a8bf8d3acd84d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:40.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.442+0000 7fc40559a640 1 -- 172.21.15.40:0/1883697354 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f0030920 con 0x7fc4080a31d0 2024-04-22T08:22:40.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.442+0000 7fc40559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:40.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.442+0000 7fc40559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:22:40.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.442+0000 7fc40559a640 10 monclient: dump: 2024-04-22T08:22:40.449 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:40.449 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:40.450 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:40.450 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:40.450 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:40.450 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:40.450 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:40.450 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:40.450 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:40.450 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:40.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.442+0000 7fc40559a640 10 monclient: _finish_auth 0 2024-04-22T08:22:40.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.442+0000 7fc40559a640 10 monclient: _check_auth_tickets 2024-04-22T08:22:40.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.442+0000 7fc40559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:40.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.442+0000 7fc40559a640 1 -- 172.21.15.40:0/1883697354 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3f002e050 con 0x7fc4080a31d0 2024-04-22T08:22:40.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.442+0000 7fc40ccce640 5 monclient: authenticate success, global_id 124106 2024-04-22T08:22:40.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.442+0000 7fc40ccce640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:22:40.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.442+0000 7fc40559a640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:40.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.442+0000 7fc40559a640 1 -- 172.21.15.40:0/1883697354 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f0030ca0 con 0x7fc4080a31d0 2024-04-22T08:22:40.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.442+0000 7fc40559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:40.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.442+0000 7fc40559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:22:40.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.442+0000 7fc40559a640 10 monclient: dump: 2024-04-22T08:22:40.451 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:40.451 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:40.451 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:40.451 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:40.451 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:40.451 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:40.451 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:40.451 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:40.451 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:40.452 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:40.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.442+0000 7fc40ccce640 10 monclient: get_monmap_and_config success 2024-04-22T08:22:40.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.442+0000 7fc40ccce640 4 set_mon_vals no callback set 2024-04-22T08:22:40.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.442+0000 7fc40ccce640 10 monclient: shutdown 2024-04-22T08:22:40.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.442+0000 7fc40ccce640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:40.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.442+0000 7fc40ccce640 1 -- 172.21.15.40:0/1883697354 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc4080a31d0 msgr2=0x7fc40813aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:40.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.442+0000 7fc40ccce640 1 --2- 172.21.15.40:0/1883697354 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc4080a31d0 0x7fc40813aed0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc3f00097a0 tx=0x7fc3f002b710 comp rx=0 tx=0).stop 2024-04-22T08:22:40.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.442+0000 7fc40ccce640 1 -- 172.21.15.40:0/1883697354 shutdown_connections 2024-04-22T08:22:40.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.442+0000 7fc40ccce640 1 --2- 172.21.15.40:0/1883697354 >> v2:172.21.15.40:3300/0 conn(0x7fc40813b410 0x7fc40813d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:40.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.442+0000 7fc40ccce640 1 --2- 172.21.15.40:0/1883697354 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc4080a31d0 0x7fc40813aed0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:40.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.442+0000 7fc40ccce640 1 -- 172.21.15.40:0/1883697354 >> 172.21.15.40:0/1883697354 conn(0x7fc4080a89a0 msgr2=0x7fc4080a4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:40.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.442+0000 7fc40ccce640 1 -- 172.21.15.40:0/1883697354 shutdown_connections 2024-04-22T08:22:40.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.442+0000 7fc40ccce640 1 -- 172.21.15.40:0/1883697354 wait complete. 2024-04-22T08:22:40.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.443+0000 7fc40ccce640 10 monclient: build_initial_monmap 2024-04-22T08:22:40.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.443+0000 7fc40ccce640 10 monclient: monmap: 2024-04-22T08:22:40.453 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:40.453 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:22:40.453 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:22:40.453 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:22:40.453 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:40.453 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:40.453 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:22:40.453 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:22:40.453 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:22:40.453 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:40.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.443+0000 7fc40ccce640 1 Processor -- start 2024-04-22T08:22:40.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.443+0000 7fc40ccce640 1 -- start start 2024-04-22T08:22:40.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.443+0000 7fc40ccce640 10 monclient: init 2024-04-22T08:22:40.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc40ccce640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:40.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc40ccce640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-22T08:22:40.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc40ccce640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc4080a1c60 0x7fc40812ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:40.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc40ccce640 10 monclient(hunting): picked mon.noname-c con 0x7fc4080a1c60 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:22:40.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc40ccce640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4080a31d0 0x7fc40812bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:40.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc40ccce640 10 monclient(hunting): picked mon.noname-b con 0x7fc4080a31d0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:22:40.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc40ccce640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc40813b410 0x7fc40812c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:40.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc40ccce640 10 monclient(hunting): picked mon.noname-a con 0x7fc40813b410 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:22:40.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc40ccce640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:40.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc40ccce640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4081355d0 con 0x7fc40813b410 2024-04-22T08:22:40.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc40ccce640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:40.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc40ccce640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4080a6110 con 0x7fc4080a31d0 2024-04-22T08:22:40.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc40ccce640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:40.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc40ccce640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc408137800 con 0x7fc4080a1c60 2024-04-22T08:22:40.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc40ccce640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:40.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc40ccce640 10 monclient(hunting): authenticate will time out at 1434.438088s 2024-04-22T08:22:40.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc406d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc40813b410 0x7fc40812c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:40.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc40659c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc4080a1c60 0x7fc40812ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:40.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc406d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc40813b410 0x7fc40812c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44434/0 (socket says 172.21.15.40:44434) 2024-04-22T08:22:40.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc406d9d640 1 -- 172.21.15.40:0/1617101860 learned_addr learned my addr 172.21.15.40:0/1617101860 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:22:40.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc406d9d640 10 monclient(hunting): get_auth_request con 0x7fc40813b410 auth_method 0 2024-04-22T08:22:40.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc406d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:40.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc406d9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:40.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc406d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:40.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc40659c640 10 monclient(hunting): get_auth_request con 0x7fc4080a1c60 auth_method 0 2024-04-22T08:22:40.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc40659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:40.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc40659c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:40.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc40659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:40.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc406d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:40.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc406d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:40.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc406d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:40.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc40659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:40.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc40659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:40.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc40659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:40.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc405d9b640 1 --2- 172.21.15.40:0/1617101860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4080a31d0 0x7fc40812bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:40.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc405d9b640 10 monclient(hunting): get_auth_request con 0x7fc4080a31d0 auth_method 0 2024-04-22T08:22:40.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc405d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:40.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc405d9b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:40.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc405d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:40.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc406d9d640 10 monclient(hunting): handle_auth_done global_id 114270 payload 824 2024-04-22T08:22:40.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc406d9d640 1 -- 172.21.15.40:0/1617101860 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc4080a1c60 msgr2=0x7fc40812ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:40.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc406d9d640 1 --2- 172.21.15.40:0/1617101860 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc4080a1c60 0x7fc40812ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:40.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc406d9d640 1 -- 172.21.15.40:0/1617101860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4080a31d0 msgr2=0x7fc40812bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:40.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc406d9d640 1 --2- 172.21.15.40:0/1617101860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4080a31d0 0x7fc40812bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:40.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc406d9d640 10 monclient: _finish_hunting 0 2024-04-22T08:22:40.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc406d9d640 1 monclient: found mon.noname-a 2024-04-22T08:22:40.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc406d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:40.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc406d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:22:40.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc406d9d640 1 -- 172.21.15.40:0/1617101860 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4000094d0 con 0x7fc40813b410 2024-04-22T08:22:40.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc405d9b640 1 --2- 172.21.15.40:0/1617101860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4080a31d0 0x7fc40812bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T08:22:40.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.444+0000 7fc40659c640 1 --2- 172.21.15.40:0/1617101860 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc4080a1c60 0x7fc40812ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:22:40.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.447+0000 7fc406d9d640 1 -- 172.21.15.40:0/1617101860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc40813b410 msgr2=0x7fc40812c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:40.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.447+0000 7fc406d9d640 1 --2- 172.21.15.40:0/1617101860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc40813b410 0x7fc40812c4b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc40000e9b0 tx=0x7fc40000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:40.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.447+0000 7fc3feffd640 1 -- 172.21.15.40:0/1617101860 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc40000cbb0 con 0x7fc40813b410 2024-04-22T08:22:40.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.447+0000 7fc3feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:40.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.447+0000 7fc3feffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:22:40.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.447+0000 7fc3feffd640 10 monclient: dump: 2024-04-22T08:22:40.457 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:40.457 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:40.458 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:40.458 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:40.458 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:40.458 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:40.458 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:40.458 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:40.458 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:40.458 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:40.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.447+0000 7fc3feffd640 10 monclient: _finish_auth 0 2024-04-22T08:22:40.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.447+0000 7fc3feffd640 10 monclient: _check_auth_tickets 2024-04-22T08:22:40.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.447+0000 7fc3feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:40.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.447+0000 7fc3feffd640 1 -- 172.21.15.40:0/1617101860 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc40000cd30 con 0x7fc40813b410 2024-04-22T08:22:40.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.447+0000 7fc3feffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:40.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.447+0000 7fc3feffd640 1 -- 172.21.15.40:0/1617101860 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4000136b0 con 0x7fc40813b410 2024-04-22T08:22:40.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.447+0000 7fc40ccce640 5 monclient: authenticate success, global_id 114270 2024-04-22T08:22:40.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.447+0000 7fc3feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:40.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.447+0000 7fc3feffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:22:40.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.447+0000 7fc3feffd640 10 monclient: dump: 2024-04-22T08:22:40.459 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:40.459 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:40.459 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:40.459 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:40.459 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:40.459 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:40.459 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:40.459 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:40.459 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:40.459 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:40.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.447+0000 7fc404d99640 4 set_mon_vals no callback set 2024-04-22T08:22:40.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.447+0000 7fc40ccce640 10 monclient: _renew_subs 2024-04-22T08:22:40.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.447+0000 7fc40ccce640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:40.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.447+0000 7fc40ccce640 1 -- 172.21.15.40:0/1617101860 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3f0009400 con 0x7fc40813b410 2024-04-22T08:22:40.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.447+0000 7fc40ccce640 10 monclient: _renew_subs 2024-04-22T08:22:40.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.447+0000 7fc40ccce640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:40.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.447+0000 7fc40ccce640 1 -- 172.21.15.40:0/1617101860 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4080a0220 con 0x7fc40813b410 2024-04-22T08:22:40.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.448+0000 7fc40ccce640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:40.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.448+0000 7fc40ccce640 1 --2- 172.21.15.40:0/1617101860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc408136f60 0x7fc408130010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:40.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.448+0000 7fc40ccce640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:40.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.448+0000 7fc40ccce640 1 -- 172.21.15.40:0/1617101860 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc408066650 con 0x7fc408136f60 2024-04-22T08:22:40.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.448+0000 7fc3feffd640 1 -- 172.21.15.40:0/1617101860 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fc400013810 con 0x7fc40813b410 2024-04-22T08:22:40.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.448+0000 7fc3feffd640 1 --2- 172.21.15.40:0/1617101860 >> v2:172.21.15.189:6800/30168 conn(0x7fc3e8037820 0x7fc3e8039ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:40.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.448+0000 7fc3feffd640 1 -- 172.21.15.40:0/1617101860 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(317..317 src has 239..317) v4 ==== 3506+0+0 (secure 0 0 0) 0x7fc40004a580 con 0x7fc40813b410 2024-04-22T08:22:40.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.448+0000 7fc40659c640 1 --2- 172.21.15.40:0/1617101860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc408136f60 0x7fc408130010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:40.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.448+0000 7fc40659c640 10 monclient: get_auth_request con 0x7fc408136f60 auth_method 0 2024-04-22T08:22:40.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.448+0000 7fc40659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:40.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.448+0000 7fc40659c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:40.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.448+0000 7fc40659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:40.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.448+0000 7fc405d9b640 1 --2- 172.21.15.40:0/1617101860 >> v2:172.21.15.189:6800/30168 conn(0x7fc3e8037820 0x7fc3e8039ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:40.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.448+0000 7fc40659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:40.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.448+0000 7fc40659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:40.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.448+0000 7fc40659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:40.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.448+0000 7fc405d9b640 10 monclient: get_auth_request con 0x7fc3e8037820 auth_method 0 2024-04-22T08:22:40.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.448+0000 7fc40659c640 10 monclient(hunting): handle_auth_done global_id 65251 payload 824 2024-04-22T08:22:40.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.448+0000 7fc40659c640 1 -- 172.21.15.40:0/1617101860 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc40809fac0 con 0x7fc408136f60 2024-04-22T08:22:40.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.451+0000 7fc40659c640 1 -- 172.21.15.40:0/1617101860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc408136f60 msgr2=0x7fc408130010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:40.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.451+0000 7fc40659c640 1 --2- 172.21.15.40:0/1617101860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc408136f60 0x7fc408130010 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fc3f80048c0 tx=0x7fc3f800d520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:40.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.451+0000 7fc405d9b640 1 --2- 172.21.15.40:0/1617101860 >> v2:172.21.15.189:6800/30168 conn(0x7fc3e8037820 0x7fc3e8039ce0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fc4080a5720 tx=0x7fc3f002f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:40.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.451+0000 7fc3feffd640 1 -- 172.21.15.40:0/1617101860 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f8004950 con 0x7fc408136f60 2024-04-22T08:22:40.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.451+0000 7fc3feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:40.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.451+0000 7fc3feffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:22:40.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.451+0000 7fc3feffd640 10 monclient: dump: 2024-04-22T08:22:40.464 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:40.464 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:40.464 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:40.464 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:40.465 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:40.465 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:40.465 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:40.465 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:40.465 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:40.465 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:40.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.451+0000 7fc3feffd640 1 -- 172.21.15.40:0/1617101860 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8468 (secure 0 0 0) 0x7fc3f8004ab0 con 0x7fc408136f60 2024-04-22T08:22:40.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.451+0000 7fc3feffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:40.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.451+0000 7fc3feffd640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:22:40.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.451+0000 7fc3feffd640 1 -- 172.21.15.40:0/1617101860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc408136f60 msgr2=0x7fc408130010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:40.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.451+0000 7fc3feffd640 1 --2- 172.21.15.40:0/1617101860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc408136f60 0x7fc408130010 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fc3f80048c0 tx=0x7fc3f800d520 comp rx=0 tx=0).stop 2024-04-22T08:22:40.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.451+0000 7fc3feffd640 1 -- 172.21.15.40:0/1617101860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc408136f60 msgr2=0x7fc408130010 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:22:40.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.451+0000 7fc3feffd640 1 --2- 172.21.15.40:0/1617101860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc408136f60 0x7fc408130010 secure :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fc3f80048c0 tx=0x7fc3f800d520 comp rx=0 tx=0).stop 2024-04-22T08:22:40.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.457+0000 7fc40ccce640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-22T08:22:40.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.457+0000 7fc40ccce640 1 --2- 172.21.15.40:0/1617101860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc408130550 0x7fc408130960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:40.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.457+0000 7fc40ccce640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:40.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.457+0000 7fc40ccce640 1 -- 172.21.15.40:0/1617101860 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc408136ca0 con 0x7fc408130550 2024-04-22T08:22:40.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.458+0000 7fc40659c640 1 --2- 172.21.15.40:0/1617101860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc408130550 0x7fc408130960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:40.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.458+0000 7fc40659c640 10 monclient: get_auth_request con 0x7fc408130550 auth_method 0 2024-04-22T08:22:40.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.458+0000 7fc40659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:40.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.458+0000 7fc40659c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:40.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.458+0000 7fc40659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:40.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.458+0000 7fc40659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:40.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.458+0000 7fc40659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:40.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.458+0000 7fc40659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:40.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.458+0000 7fc40659c640 10 monclient(hunting): handle_auth_done global_id 65254 payload 824 2024-04-22T08:22:40.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.458+0000 7fc40659c640 1 -- 172.21.15.40:0/1617101860 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc408133020 con 0x7fc408130550 2024-04-22T08:22:40.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.461+0000 7fc40659c640 1 -- 172.21.15.40:0/1617101860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc408130550 msgr2=0x7fc408130960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:40.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.461+0000 7fc40659c640 1 --2- 172.21.15.40:0/1617101860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc408130550 0x7fc408130960 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fc3f8018000 tx=0x7fc3f8003ae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:40.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.461+0000 7fc3feffd640 1 -- 172.21.15.40:0/1617101860 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f8024070 con 0x7fc408130550 2024-04-22T08:22:40.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.461+0000 7fc3feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:40.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.461+0000 7fc3feffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:22:40.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.461+0000 7fc3feffd640 10 monclient: dump: 2024-04-22T08:22:40.467 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:40.467 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:40.467 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:40.467 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:40.467 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:40.467 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:40.467 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:40.467 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:40.467 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:40.467 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:40.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.461+0000 7fc3feffd640 1 -- 172.21.15.40:0/1617101860 <== mon.1 v2:172.21.15.189:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4801 (secure 0 0 0) 0x7fc3f8004e10 con 0x7fc408130550 2024-04-22T08:22:40.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.461+0000 7fc3feffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-22T08:22:40.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.461+0000 7fc3feffd640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:22:40.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.461+0000 7fc3feffd640 1 -- 172.21.15.40:0/1617101860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc408130550 msgr2=0x7fc408130960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:40.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.461+0000 7fc3feffd640 1 --2- 172.21.15.40:0/1617101860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc408130550 0x7fc408130960 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fc3f8018000 tx=0x7fc3f8003ae0 comp rx=0 tx=0).stop 2024-04-22T08:22:40.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.461+0000 7fc3feffd640 1 -- 172.21.15.40:0/1617101860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc408130550 msgr2=0x7fc408130960 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:22:40.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.461+0000 7fc3feffd640 1 --2- 172.21.15.40:0/1617101860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc408130550 0x7fc408130960 secure :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fc3f8018000 tx=0x7fc3f8003ae0 comp rx=0 tx=0).stop 2024-04-22T08:22:40.468 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:22:40.468 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:22:40.468 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:22:40.468 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "peon", 2024-04-22T08:22:40.469 INFO:teuthology.orchestra.run.smithi040.stdout: "election_epoch": 98, 2024-04-22T08:22:40.469 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum": [ 2024-04-22T08:22:40.469 INFO:teuthology.orchestra.run.smithi040.stdout: 0, 2024-04-22T08:22:40.469 INFO:teuthology.orchestra.run.smithi040.stdout: 1, 2024-04-22T08:22:40.469 INFO:teuthology.orchestra.run.smithi040.stdout: 2 2024-04-22T08:22:40.469 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:22:40.469 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_age": 1, 2024-04-22T08:22:40.469 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:22:40.469 INFO:teuthology.orchestra.run.smithi040.stdout: "required_con": "2449958758054445060", 2024-04-22T08:22:40.469 INFO:teuthology.orchestra.run.smithi040.stdout: "required_mon": [ 2024-04-22T08:22:40.469 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:22:40.469 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:22:40.469 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:22:40.469 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:22:40.470 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:22:40.470 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:22:40.470 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:22:40.470 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:22:40.470 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:22:40.470 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:22:40.470 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:22:40.470 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_con": "4540138322906710015", 2024-04-22T08:22:40.470 INFO:teuthology.orchestra.run.smithi040.stdout: "quorum_mon": [ 2024-04-22T08:22:40.470 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:22:40.470 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:22:40.470 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:22:40.470 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:22:40.470 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:22:40.471 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:22:40.471 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:22:40.471 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:22:40.471 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:22:40.471 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:22:40.471 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:22:40.471 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:40.471 INFO:teuthology.orchestra.run.smithi040.stdout: "outside_quorum": [], 2024-04-22T08:22:40.471 INFO:teuthology.orchestra.run.smithi040.stdout: "extra_probe_peers": [], 2024-04-22T08:22:40.471 INFO:teuthology.orchestra.run.smithi040.stdout: "sync_provider": [], 2024-04-22T08:22:40.471 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": { 2024-04-22T08:22:40.471 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 1, 2024-04-22T08:22:40.471 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "987d842f-7205-47b9-86e0-2e2623f3c370", 2024-04-22T08:22:40.471 INFO:teuthology.orchestra.run.smithi040.stdout: "modified": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:22:40.471 INFO:teuthology.orchestra.run.smithi040.stdout: "created": "2024-04-22T08:12:05.320036Z", 2024-04-22T08:22:40.472 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release": 18, 2024-04-22T08:22:40.472 INFO:teuthology.orchestra.run.smithi040.stdout: "min_mon_release_name": "reef", 2024-04-22T08:22:40.472 INFO:teuthology.orchestra.run.smithi040.stdout: "election_strategy": 1, 2024-04-22T08:22:40.472 INFO:teuthology.orchestra.run.smithi040.stdout: "disallowed_leaders: ": "", 2024-04-22T08:22:40.472 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false, 2024-04-22T08:22:40.472 INFO:teuthology.orchestra.run.smithi040.stdout: "tiebreaker_mon": "", 2024-04-22T08:22:40.472 INFO:teuthology.orchestra.run.smithi040.stdout: "removed_ranks: ": "", 2024-04-22T08:22:40.472 INFO:teuthology.orchestra.run.smithi040.stdout: "features": { 2024-04-22T08:22:40.472 INFO:teuthology.orchestra.run.smithi040.stdout: "persistent": [ 2024-04-22T08:22:40.472 INFO:teuthology.orchestra.run.smithi040.stdout: "kraken", 2024-04-22T08:22:40.472 INFO:teuthology.orchestra.run.smithi040.stdout: "luminous", 2024-04-22T08:22:40.472 INFO:teuthology.orchestra.run.smithi040.stdout: "mimic", 2024-04-22T08:22:40.472 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap-prune", 2024-04-22T08:22:40.472 INFO:teuthology.orchestra.run.smithi040.stdout: "nautilus", 2024-04-22T08:22:40.472 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus", 2024-04-22T08:22:40.473 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific", 2024-04-22T08:22:40.473 INFO:teuthology.orchestra.run.smithi040.stdout: "elector-pinging", 2024-04-22T08:22:40.473 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy", 2024-04-22T08:22:40.473 INFO:teuthology.orchestra.run.smithi040.stdout: "reef" 2024-04-22T08:22:40.473 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:22:40.473 INFO:teuthology.orchestra.run.smithi040.stdout: "optional": [] 2024-04-22T08:22:40.473 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:40.473 INFO:teuthology.orchestra.run.smithi040.stdout: "mons": [ 2024-04-22T08:22:40.473 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:40.473 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 0, 2024-04-22T08:22:40.473 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "a", 2024-04-22T08:22:40.473 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:22:40.473 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:22:40.473 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:40.474 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:22:40.474 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3300", 2024-04-22T08:22:40.474 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:22:40.474 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:40.474 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:40.474 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:22:40.474 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789", 2024-04-22T08:22:40.474 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:22:40.474 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:22:40.474 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:22:40.474 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:40.474 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6789/0", 2024-04-22T08:22:40.474 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6789/0", 2024-04-22T08:22:40.474 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:22:40.474 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:22:40.475 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:22:40.475 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:40.475 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:40.475 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 1, 2024-04-22T08:22:40.475 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "b", 2024-04-22T08:22:40.475 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:22:40.475 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:22:40.475 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:40.475 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:22:40.475 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:3300", 2024-04-22T08:22:40.475 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:22:40.475 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:40.475 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:40.475 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:22:40.476 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789", 2024-04-22T08:22:40.476 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:22:40.476 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:22:40.476 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:22:40.476 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:40.476 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.189:6789/0", 2024-04-22T08:22:40.476 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.189:6789/0", 2024-04-22T08:22:40.476 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:22:40.476 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:22:40.476 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:22:40.476 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:40.476 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:40.476 INFO:teuthology.orchestra.run.smithi040.stdout: "rank": 2, 2024-04-22T08:22:40.476 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "c", 2024-04-22T08:22:40.476 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrs": { 2024-04-22T08:22:40.477 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-22T08:22:40.477 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:40.477 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-22T08:22:40.477 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:3301", 2024-04-22T08:22:40.477 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:22:40.477 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:40.477 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:40.477 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-22T08:22:40.477 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790", 2024-04-22T08:22:40.477 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 0 2024-04-22T08:22:40.477 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:22:40.477 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:22:40.477 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:40.477 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6790/0", 2024-04-22T08:22:40.477 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "172.21.15.40:6790/0", 2024-04-22T08:22:40.478 INFO:teuthology.orchestra.run.smithi040.stdout: "priority": 0, 2024-04-22T08:22:40.478 INFO:teuthology.orchestra.run.smithi040.stdout: "weight": 0, 2024-04-22T08:22:40.478 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "{}" 2024-04-22T08:22:40.478 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:22:40.478 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:22:40.478 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:40.478 INFO:teuthology.orchestra.run.smithi040.stdout: "feature_map": { 2024-04-22T08:22:40.478 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": [ 2024-04-22T08:22:40.478 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:40.478 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:22:40.478 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:22:40.478 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:22:40.478 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:22:40.478 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:22:40.479 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": [ 2024-04-22T08:22:40.479 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:40.479 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:22:40.479 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:22:40.479 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 5 2024-04-22T08:22:40.479 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:22:40.479 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:22:40.479 INFO:teuthology.orchestra.run.smithi040.stdout: "client": [ 2024-04-22T08:22:40.479 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:40.479 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:22:40.479 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:22:40.479 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 5 2024-04-22T08:22:40.479 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:22:40.479 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-22T08:22:40.480 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": [ 2024-04-22T08:22:40.480 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-22T08:22:40.480 INFO:teuthology.orchestra.run.smithi040.stdout: "features": "0x3f01cfbffffdffff", 2024-04-22T08:22:40.480 INFO:teuthology.orchestra.run.smithi040.stdout: "release": "luminous", 2024-04-22T08:22:40.480 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 1 2024-04-22T08:22:40.480 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-22T08:22:40.480 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-22T08:22:40.480 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-22T08:22:40.480 INFO:teuthology.orchestra.run.smithi040.stdout: "stretch_mode": false 2024-04-22T08:22:40.480 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:22:40.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.461+0000 7fc40ccce640 1 -- 172.21.15.40:0/1617101860 >> v2:172.21.15.189:6800/30168 conn(0x7fc3e8037820 msgr2=0x7fc3e8039ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:40.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.461+0000 7fc40ccce640 1 --2- 172.21.15.40:0/1617101860 >> v2:172.21.15.189:6800/30168 conn(0x7fc3e8037820 0x7fc3e8039ce0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fc4080a5720 tx=0x7fc3f002f000 comp rx=0 tx=0).stop 2024-04-22T08:22:40.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.461+0000 7fc40ccce640 10 monclient: shutdown 2024-04-22T08:22:40.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.461+0000 7fc40ccce640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:40.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.461+0000 7fc40ccce640 1 -- 172.21.15.40:0/1617101860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc40813b410 msgr2=0x7fc40812c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:40.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.461+0000 7fc40ccce640 1 --2- 172.21.15.40:0/1617101860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc40813b410 0x7fc40812c4b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc40000e9b0 tx=0x7fc40000ee70 comp rx=0 tx=0).stop 2024-04-22T08:22:40.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.461+0000 7fc406d9d640 1 -- 172.21.15.40:0/1617101860 reap_dead start 2024-04-22T08:22:40.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.461+0000 7fc40ccce640 1 -- 172.21.15.40:0/1617101860 shutdown_connections 2024-04-22T08:22:40.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.461+0000 7fc40ccce640 1 --2- 172.21.15.40:0/1617101860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc40813b410 0x7fc40812c4b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:40.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.461+0000 7fc40ccce640 1 -- 172.21.15.40:0/1617101860 >> 172.21.15.40:0/1617101860 conn(0x7fc4080a89a0 msgr2=0x7fc408132150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:40.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.462+0000 7fc40ccce640 1 -- 172.21.15.40:0/1617101860 shutdown_connections 2024-04-22T08:22:40.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.462+0000 7fc40ccce640 1 -- 172.21.15.40:0/1617101860 wait complete. 2024-04-22T08:22:40.483 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-22T08:22:40.483 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-22T08:22:40.483 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-22T08:22:40.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.582+0000 7f48d8124640 10 monclient: get_monmap_and_config 2024-04-22T08:22:40.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.582+0000 7f48d8124640 10 monclient: build_initial_monmap 2024-04-22T08:22:40.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.582+0000 7f48d8124640 10 monclient: monmap: 2024-04-22T08:22:40.583 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:40.583 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:40.583 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:22:40.583568+0000 2024-04-22T08:22:40.583 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:22:40.583568+0000 2024-04-22T08:22:40.583 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:40.583 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:40.583 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:22:40.583 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:22:40.583 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:22:40.583 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:22:40.583 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:22:40.583 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:40.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.583+0000 7f48d8124640 1 Processor -- start 2024-04-22T08:22:40.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.583+0000 7f48d8124640 1 -- start start 2024-04-22T08:22:40.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.583+0000 7f48d8124640 10 monclient: init 2024-04-22T08:22:40.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.583+0000 7f48d8124640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:40.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.583+0000 7f48d8124640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-04-22T08:22:40.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.583+0000 7f48d8124640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f48d012fc40 addr v1:172.21.15.189:6789/0 2024-04-22T08:22:40.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.583+0000 7f48d8124640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48d009e210 0x7f48d0137cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:40.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.584+0000 7f48d8124640 10 monclient(hunting): picked mon.noname-b con 0x7f48d009e210 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:22:40.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.584+0000 7f48d8124640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f48d01381f0 0x7f48d013a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:40.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.584+0000 7f48d8124640 10 monclient(hunting): picked mon.noname-a con 0x7f48d01381f0 addr v2:172.21.15.40:3300/0 2024-04-22T08:22:40.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.584+0000 7f48d8124640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48d009ea50 con 0x7f48d012fc40 2024-04-22T08:22:40.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.584+0000 7f48d8124640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:40.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.584+0000 7f48d8124640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48d009cf90 con 0x7f48d01381f0 2024-04-22T08:22:40.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.584+0000 7f48d8124640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:40.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.584+0000 7f48d8124640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48d009d270 con 0x7f48d009e210 2024-04-22T08:22:40.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.584+0000 7f48d8124640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:40.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.584+0000 7f48d8124640 10 monclient(hunting): authenticate will time out at 1434.577823s 2024-04-22T08:22:40.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.584+0000 7f48d56bf640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48d009e210 0x7f48d0137cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:40.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.584+0000 7f48d66c1640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f48d01381f0 0x7f48d013a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:40.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.584+0000 7f48d56bf640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48d009e210 0x7f48d0137cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49728/0 (socket says 172.21.15.40:49728) 2024-04-22T08:22:40.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.584+0000 7f48d66c1640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f48d01381f0 0x7f48d013a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44438/0 (socket says 172.21.15.40:44438) 2024-04-22T08:22:40.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.584+0000 7f48d56bf640 1 -- 172.21.15.40:0/3793086509 learned_addr learned my addr 172.21.15.40:0/3793086509 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:22:40.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.584+0000 7f48d56bf640 10 monclient(hunting): get_auth_request con 0x7f48d009e210 auth_method 0 2024-04-22T08:22:40.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.584+0000 7f48d56bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:40.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.584+0000 7f48d56bf640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:40.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.584+0000 7f48d56bf640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:40.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.584+0000 7f48d66c1640 10 monclient(hunting): get_auth_request con 0x7f48d01381f0 auth_method 0 2024-04-22T08:22:40.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.584+0000 7f48d66c1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:40.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.584+0000 7f48d66c1640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:40.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.584+0000 7f48d66c1640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:40.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.584+0000 7f48d56bf640 1 -- 172.21.15.40:0/3793086509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48d009e210 msgr2=0x7f48d0137cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:22:40.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.584+0000 7f48d56bf640 1 -- 172.21.15.40:0/3793086509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48d009e210 msgr2=0x7f48d0137cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:22:40.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.584+0000 7f48d56bf640 1 --2- 172.21.15.40:0/3793086509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48d009e210 0x7f48d0137cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:22:40.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.584+0000 7f48d56bf640 1 --2- 172.21.15.40:0/3793086509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48d009e210 0x7f48d0137cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:22:40.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.584+0000 7f48d66c1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:40.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.584+0000 7f48d66c1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:40.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.584+0000 7f48d66c1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:40.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.584+0000 7f48d66c1640 10 monclient(hunting): handle_auth_done global_id 114273 payload 274 2024-04-22T08:22:40.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.584+0000 7f48d66c1640 1 -- 172.21.15.40:0/3793086509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48d009e210 msgr2=0x7f48d0137cb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:22:40.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.584+0000 7f48d66c1640 1 --2- 172.21.15.40:0/3793086509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48d009e210 0x7f48d0137cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:40.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.584+0000 7f48d66c1640 1 -- 172.21.15.40:0/3793086509 >> v1:172.21.15.189:6789/0 conn(0x7f48d012fc40 legacy=0x7f48d0130020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:40.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.584+0000 7f48d66c1640 10 monclient: _finish_hunting 0 2024-04-22T08:22:40.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.584+0000 7f48d66c1640 1 monclient: found mon.noname-a 2024-04-22T08:22:40.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.584+0000 7f48d66c1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:40.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.584+0000 7f48d66c1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:22:40.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.584+0000 7f48d66c1640 1 -- 172.21.15.40:0/3793086509 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d009d550 con 0x7f48d01381f0 2024-04-22T08:22:40.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.587+0000 7f48d66c1640 1 -- 172.21.15.40:0/3793086509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48d01381f0 msgr2=0x7f48d013a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:40.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.587+0000 7f48d66c1640 1 --2- 172.21.15.40:0/3793086509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48d01381f0 0x7f48d013a5e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f48c8009d90 tx=0x7f48c802bcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20883ebc1f7cfb03 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:40.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.587+0000 7f48d4ebe640 1 -- 172.21.15.40:0/3793086509 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48c802ccb0 con 0x7f48d01381f0 2024-04-22T08:22:40.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.587+0000 7f48d4ebe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:40.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.587+0000 7f48d4ebe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:22:40.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.587+0000 7f48d4ebe640 10 monclient: dump: 2024-04-22T08:22:40.595 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:40.595 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:40.595 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:40.595 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:40.595 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:40.595 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:40.596 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:40.596 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:40.596 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:40.596 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:40.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.587+0000 7f48d4ebe640 10 monclient: _finish_auth 0 2024-04-22T08:22:40.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.587+0000 7f48d4ebe640 10 monclient: _check_auth_tickets 2024-04-22T08:22:40.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.587+0000 7f48d4ebe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:40.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.587+0000 7f48d4ebe640 1 -- 172.21.15.40:0/3793086509 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48c802ce10 con 0x7f48d01381f0 2024-04-22T08:22:40.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.587+0000 7f48d8124640 5 monclient: authenticate success, global_id 114273 2024-04-22T08:22:40.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.587+0000 7f48d8124640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:22:40.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.587+0000 7f48d4ebe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:40.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.587+0000 7f48d8124640 10 monclient: get_monmap_and_config success 2024-04-22T08:22:40.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.587+0000 7f48d4ebe640 1 -- 172.21.15.40:0/3793086509 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48c8034660 con 0x7f48d01381f0 2024-04-22T08:22:40.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.587+0000 7f48d4ebe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:40.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.587+0000 7f48d8124640 4 set_mon_vals no callback set 2024-04-22T08:22:40.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.587+0000 7f48d4ebe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:22:40.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.587+0000 7f48d4ebe640 10 monclient: dump: 2024-04-22T08:22:40.597 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:40.597 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:40.597 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:40.597 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:40.597 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:40.597 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:40.597 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:40.597 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:40.597 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:40.597 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:40.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.588+0000 7f48d8124640 10 monclient: shutdown 2024-04-22T08:22:40.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.588+0000 7f48d8124640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:40.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.588+0000 7f48d8124640 1 -- 172.21.15.40:0/3793086509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48d01381f0 msgr2=0x7f48d013a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:40.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.588+0000 7f48d8124640 1 --2- 172.21.15.40:0/3793086509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48d01381f0 0x7f48d013a5e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f48c8009d90 tx=0x7f48c802bcf0 comp rx=0 tx=0).stop 2024-04-22T08:22:40.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.588+0000 7f48d8124640 1 -- 172.21.15.40:0/3793086509 shutdown_connections 2024-04-22T08:22:40.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.588+0000 7f48d8124640 1 --2- 172.21.15.40:0/3793086509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48d01381f0 0x7f48d013a5e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:40.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.588+0000 7f48d8124640 1 --2- 172.21.15.40:0/3793086509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48d009e210 0x7f48d0137cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:40.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.588+0000 7f48d8124640 1 -- 172.21.15.40:0/3793086509 >> 172.21.15.40:0/3793086509 conn(0x7f48d00a0b50 msgr2=0x7f48d00a0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:40.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.588+0000 7f48d8124640 1 -- 172.21.15.40:0/3793086509 shutdown_connections 2024-04-22T08:22:40.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.588+0000 7f48d8124640 1 -- 172.21.15.40:0/3793086509 wait complete. 2024-04-22T08:22:40.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.588+0000 7f48d8124640 10 monclient: build_initial_monmap 2024-04-22T08:22:40.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.588+0000 7f48d8124640 10 monclient: monmap: 2024-04-22T08:22:40.599 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:40.599 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:22:40.599 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:22:40.600 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:22:40.600 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:40.600 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:40.600 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:22:40.600 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:22:40.600 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:22:40.600 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:40.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d8124640 1 Processor -- start 2024-04-22T08:22:40.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d8124640 1 -- start start 2024-04-22T08:22:40.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d8124640 10 monclient: init 2024-04-22T08:22:40.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d8124640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:40.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d8124640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-22T08:22:40.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d8124640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48d009e210 0x7f48d013d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:40.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d8124640 10 monclient(hunting): picked mon.noname-c con 0x7f48d009e210 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:22:40.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d8124640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48d012fc40 0x7f48d0127060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:40.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d8124640 10 monclient(hunting): picked mon.noname-a con 0x7f48d012fc40 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:22:40.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d8124640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48d01381f0 0x7f48d01275a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:40.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d8124640 10 monclient(hunting): picked mon.noname-b con 0x7f48d01381f0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:22:40.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d8124640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:40.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d8124640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d0128e10 con 0x7f48d012fc40 2024-04-22T08:22:40.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d8124640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:40.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d8124640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d012f620 con 0x7f48d01381f0 2024-04-22T08:22:40.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d8124640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:40.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d8124640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48d013c630 con 0x7f48d009e210 2024-04-22T08:22:40.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d8124640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:40.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d8124640 10 monclient(hunting): authenticate will time out at 1434.583309s 2024-04-22T08:22:40.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d5ec0640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48d009e210 0x7f48d013d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:40.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d56bf640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48d012fc40 0x7f48d0127060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:40.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d56bf640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48d012fc40 0x7f48d0127060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44440/0 (socket says 172.21.15.40:44440) 2024-04-22T08:22:40.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d56bf640 1 -- 172.21.15.40:0/443579546 learned_addr learned my addr 172.21.15.40:0/443579546 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:22:40.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d5ec0640 10 monclient(hunting): get_auth_request con 0x7f48d009e210 auth_method 0 2024-04-22T08:22:40.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d5ec0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:40.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d5ec0640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:40.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d5ec0640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:40.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d56bf640 10 monclient(hunting): get_auth_request con 0x7f48d012fc40 auth_method 0 2024-04-22T08:22:40.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d56bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:40.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d56bf640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:40.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d56bf640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:40.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d5ec0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:40.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d5ec0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:40.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d5ec0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:40.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d56bf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:40.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d56bf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:40.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d56bf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:40.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d66c1640 1 --2- 172.21.15.40:0/443579546 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48d01381f0 0x7f48d01275a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:40.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d66c1640 10 monclient(hunting): get_auth_request con 0x7f48d01381f0 auth_method 0 2024-04-22T08:22:40.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d66c1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:40.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d66c1640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:40.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d66c1640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:40.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d5ec0640 10 monclient(hunting): handle_auth_done global_id 124112 payload 824 2024-04-22T08:22:40.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d5ec0640 1 -- 172.21.15.40:0/443579546 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48d01381f0 msgr2=0x7f48d01275a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:40.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d5ec0640 1 --2- 172.21.15.40:0/443579546 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48d01381f0 0x7f48d01275a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:40.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d5ec0640 1 -- 172.21.15.40:0/443579546 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48d012fc40 msgr2=0x7f48d0127060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:40.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d5ec0640 1 --2- 172.21.15.40:0/443579546 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48d012fc40 0x7f48d0127060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:40.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d5ec0640 10 monclient: _finish_hunting 0 2024-04-22T08:22:40.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d5ec0640 1 monclient: found mon.noname-c 2024-04-22T08:22:40.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d5ec0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:40.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d5ec0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:22:40.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.589+0000 7f48d5ec0640 1 -- 172.21.15.40:0/443579546 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c8009a70 con 0x7f48d009e210 2024-04-22T08:22:40.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.590+0000 7f48d56bf640 1 --2- 172.21.15.40:0/443579546 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48d012fc40 0x7f48d0127060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:22:40.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.590+0000 7f48d66c1640 1 --2- 172.21.15.40:0/443579546 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48d01381f0 0x7f48d01275a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T08:22:40.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.592+0000 7f48d5ec0640 1 -- 172.21.15.40:0/443579546 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48d009e210 msgr2=0x7f48d013d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:40.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.592+0000 7f48d5ec0640 1 --2- 172.21.15.40:0/443579546 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48d009e210 0x7f48d013d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f48cc00c410 tx=0x7f48cc00c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:40.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.592+0000 7f48c67fc640 1 -- 172.21.15.40:0/443579546 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48cc005230 con 0x7f48d009e210 2024-04-22T08:22:40.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.592+0000 7f48c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:40.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.592+0000 7f48c67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:22:40.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.592+0000 7f48c67fc640 10 monclient: dump: 2024-04-22T08:22:40.604 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:40.604 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:40.605 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:40.605 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:40.605 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:40.605 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:40.605 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:40.605 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:40.605 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:40.605 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:40.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.592+0000 7f48c67fc640 10 monclient: _finish_auth 0 2024-04-22T08:22:40.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.592+0000 7f48c67fc640 10 monclient: _check_auth_tickets 2024-04-22T08:22:40.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.592+0000 7f48c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:40.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.592+0000 7f48d8124640 5 monclient: authenticate success, global_id 124112 2024-04-22T08:22:40.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.592+0000 7f48c67fc640 1 -- 172.21.15.40:0/443579546 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48cc005390 con 0x7f48d009e210 2024-04-22T08:22:40.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.592+0000 7f48c67fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:40.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.592+0000 7f48d8124640 10 monclient: _renew_subs 2024-04-22T08:22:40.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.592+0000 7f48c67fc640 1 -- 172.21.15.40:0/443579546 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48cc0119e0 con 0x7f48d009e210 2024-04-22T08:22:40.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.592+0000 7f48d8124640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:22:40.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.592+0000 7f48d8124640 1 -- 172.21.15.40:0/443579546 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48d013c890 con 0x7f48d009e210 2024-04-22T08:22:40.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.592+0000 7f48c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:40.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.592+0000 7f48c67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:22:40.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.592+0000 7f48c7fff640 4 set_mon_vals no callback set 2024-04-22T08:22:40.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.592+0000 7f48c67fc640 10 monclient: dump: 2024-04-22T08:22:40.606 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:40.606 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:40.606 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:40.606 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:40.607 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:40.607 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:40.607 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:40.607 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:40.607 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:40.607 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:40.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.592+0000 7f48d8124640 10 monclient: _renew_subs 2024-04-22T08:22:40.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.592+0000 7f48d8124640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:22:40.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.592+0000 7f48d8124640 1 -- 172.21.15.40:0/443579546 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48d0070da0 con 0x7f48d009e210 2024-04-22T08:22:40.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.593+0000 7f48c67fc640 1 -- 172.21.15.40:0/443579546 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f48cc011bc0 con 0x7f48d009e210 2024-04-22T08:22:40.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.593+0000 7f48d8124640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:40.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.593+0000 7f48d8124640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:40.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.593+0000 7f48d8124640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:22:40.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.593+0000 7f48c67fc640 1 --2- 172.21.15.40:0/443579546 >> v2:172.21.15.189:6800/30168 conn(0x7f48b40377d0 0x7f48b4039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:40.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.593+0000 7f48d8124640 1 -- 172.21.15.40:0/443579546 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48d004e9b0 con 0x7f48d009e210 2024-04-22T08:22:40.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.593+0000 7f48c67fc640 1 -- 172.21.15.40:0/443579546 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(317..317 src has 239..317) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f48cc049700 con 0x7f48d009e210 2024-04-22T08:22:40.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.593+0000 7f48d56bf640 1 --2- 172.21.15.40:0/443579546 >> v2:172.21.15.189:6800/30168 conn(0x7f48b40377d0 0x7f48b4039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:40.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.593+0000 7f48d56bf640 10 monclient: get_auth_request con 0x7f48b40377d0 auth_method 0 2024-04-22T08:22:40.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.596+0000 7f48d56bf640 1 --2- 172.21.15.40:0/443579546 >> v2:172.21.15.189:6800/30168 conn(0x7f48b40377d0 0x7f48b4039c90 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f48d012fb70 tx=0x7f48c0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:40.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.598+0000 7f48c67fc640 1 -- 172.21.15.40:0/443579546 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f48cc01b3b0 con 0x7f48d009e210 2024-04-22T08:22:40.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.598+0000 7f48c67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:40.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.598+0000 7f48c67fc640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:22:40.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.845+0000 7f48d8124640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-22T08:22:40.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.845+0000 7f48d8124640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-22T08:22:40.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.845+0000 7f48d8124640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:22:40.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.845+0000 7f48d8124640 1 -- 172.21.15.40:0/443579546 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f48d006a290 con 0x7f48d009e210 2024-04-22T08:22:40.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.846+0000 7f48c67fc640 1 -- 172.21.15.40:0/443579546 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f48cc02e160 con 0x7f48d009e210 2024-04-22T08:22:40.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.846+0000 7f48c67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-22T08:22:40.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.846+0000 7f48c67fc640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:22:40.846 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.846+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.847 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.846+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=2,health=11,kv=9,logm=59} crc {auth=1078469201,config=1618278320,health=623906709,kv=1350343662,logm=3465076903}) 2024-04-22T08:22:40.847 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.846+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=19,config=2,health=12,kv=9,logm=58} crc {auth=2803132872,config=1618278320,health=1223181250,kv=1350343662,logm=2560744915}) 2024-04-22T08:22:40.847 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.846+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.847 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.846+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=2,health=11,kv=9,logm=59} crc {auth=1078469201,config=1618278320,health=623906709,kv=1350343662,logm=3465076903}) 2024-04-22T08:22:40.847 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.846+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=19,config=2,health=11,kv=9,logm=59} crc {auth=2933890614,config=1618278320,health=4200890072,kv=1350343662,logm=1871518835}) 2024-04-22T08:22:40.848 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.848+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.848 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.848+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2655153246}) 2024-04-22T08:22:40.848 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.848+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3983455895}) 2024-04-22T08:22:40.848 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.848+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.848 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.848+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2655153246}) 2024-04-22T08:22:40.848 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.848+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2842789969}) 2024-04-22T08:22:40.849 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.849+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.849 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.849+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3589157478}) 2024-04-22T08:22:40.849 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.849+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=744930028}) 2024-04-22T08:22:40.849 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.849+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.849 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.849+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3589157478}) 2024-04-22T08:22:40.849 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.849+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1920936339}) 2024-04-22T08:22:40.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.849+0000 7f48d8124640 1 -- 172.21.15.40:0/443579546 >> v2:172.21.15.189:6800/30168 conn(0x7f48b40377d0 msgr2=0x7f48b4039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:40.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.849+0000 7f48d8124640 1 --2- 172.21.15.40:0/443579546 >> v2:172.21.15.189:6800/30168 conn(0x7f48b40377d0 0x7f48b4039c90 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f48d012fb70 tx=0x7f48c0008040 comp rx=0 tx=0).stop 2024-04-22T08:22:40.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.849+0000 7f48d8124640 10 monclient: shutdown 2024-04-22T08:22:40.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.849+0000 7f48d8124640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:40.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.849+0000 7f48d8124640 1 -- 172.21.15.40:0/443579546 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48d009e210 msgr2=0x7f48d013d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:40.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.849+0000 7f48d8124640 1 --2- 172.21.15.40:0/443579546 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48d009e210 0x7f48d013d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f48cc00c410 tx=0x7f48cc00c8d0 comp rx=0 tx=0).stop 2024-04-22T08:22:40.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.849+0000 7f48d8124640 1 -- 172.21.15.40:0/443579546 shutdown_connections 2024-04-22T08:22:40.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.849+0000 7f48d8124640 1 --2- 172.21.15.40:0/443579546 >> v2:172.21.15.189:6800/30168 conn(0x7f48b40377d0 0x7f48b4039c90 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:40.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.849+0000 7f48d8124640 1 --2- 172.21.15.40:0/443579546 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48d01381f0 0x7f48d01275a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:40.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.849+0000 7f48d8124640 1 --2- 172.21.15.40:0/443579546 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48d012fc40 0x7f48d0127060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:40.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.849+0000 7f48d8124640 1 --2- 172.21.15.40:0/443579546 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48d009e210 0x7f48d013d270 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:40.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.849+0000 7f48d8124640 1 -- 172.21.15.40:0/443579546 >> 172.21.15.40:0/443579546 conn(0x7f48d00a0b50 msgr2=0x7f48d009b6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:40.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.849+0000 7f48d8124640 1 -- 172.21.15.40:0/443579546 shutdown_connections 2024-04-22T08:22:40.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:40.849+0000 7f48d8124640 1 -- 172.21.15.40:0/443579546 wait complete. 2024-04-22T08:22:40.855 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.850+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.855 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.850+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=459368998}) 2024-04-22T08:22:40.856 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.850+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1051222564}) 2024-04-22T08:22:40.856 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.850+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.856 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.850+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=459368998}) 2024-04-22T08:22:40.856 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.850+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1327125700}) 2024-04-22T08:22:40.856 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.850+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.856 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.850+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3040210073}) 2024-04-22T08:22:40.856 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.850+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3743154305}) 2024-04-22T08:22:40.856 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.850+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.856 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.850+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3040210073}) 2024-04-22T08:22:40.856 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.850+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2309533646}) 2024-04-22T08:22:40.856 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.851+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.856 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.851+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1733911399}) 2024-04-22T08:22:40.856 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.851+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=474017279}) 2024-04-22T08:22:40.856 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.851+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.856 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.851+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1733911399}) 2024-04-22T08:22:40.856 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.851+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2146524986}) 2024-04-22T08:22:40.857 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.852+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.857 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.852+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1161005591}) 2024-04-22T08:22:40.857 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.852+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3144272602}) 2024-04-22T08:22:40.857 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.852+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.857 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.852+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1161005591}) 2024-04-22T08:22:40.857 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.852+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=601305309}) 2024-04-22T08:22:40.857 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.853+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.857 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.853+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2026176378}) 2024-04-22T08:22:40.857 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.853+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=208611838}) 2024-04-22T08:22:40.857 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.853+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.857 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.853+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2026176378}) 2024-04-22T08:22:40.857 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.853+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3565862498}) 2024-04-22T08:22:40.857 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.854+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.857 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.854+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3159679837}) 2024-04-22T08:22:40.857 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.854+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1032893071}) 2024-04-22T08:22:40.857 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.854+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.858 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.854+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3159679837}) 2024-04-22T08:22:40.858 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.854+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=288168842}) 2024-04-22T08:22:40.858 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.855+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.858 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.855+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3494544861}) 2024-04-22T08:22:40.858 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.855+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1629686176}) 2024-04-22T08:22:40.858 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.855+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.858 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.855+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3494544861}) 2024-04-22T08:22:40.858 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.855+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1941349846}) 2024-04-22T08:22:40.858 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.856+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.858 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.856+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1910950163}) 2024-04-22T08:22:40.858 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.856+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1718160998}) 2024-04-22T08:22:40.858 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.856+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.858 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.856+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1910950163}) 2024-04-22T08:22:40.858 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.856+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=387865695}) 2024-04-22T08:22:40.860 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.857+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.860 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.857+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4174082646}) 2024-04-22T08:22:40.861 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.857+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=546368210}) 2024-04-22T08:22:40.861 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.857+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.861 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.857+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4174082646}) 2024-04-22T08:22:40.861 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.857+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3269894396}) 2024-04-22T08:22:40.861 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.858+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.861 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.858+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1280927499}) 2024-04-22T08:22:40.861 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.858+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1577120925}) 2024-04-22T08:22:40.861 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.858+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.861 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.858+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1280927499}) 2024-04-22T08:22:40.861 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.858+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3218816678}) 2024-04-22T08:22:40.861 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.859+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.861 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.859+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2314429807}) 2024-04-22T08:22:40.861 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.859+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3631493647}) 2024-04-22T08:22:40.861 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.859+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.861 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.859+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2314429807}) 2024-04-22T08:22:40.861 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.859+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1672343498}) 2024-04-22T08:22:40.862 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.859+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.862 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.859+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1565035350}) 2024-04-22T08:22:40.862 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.859+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2504173835}) 2024-04-22T08:22:40.862 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.860+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.862 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.860+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1565035350}) 2024-04-22T08:22:40.862 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.860+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4038383445}) 2024-04-22T08:22:40.863 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-22T08:22:40.866 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.860+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.866 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.860+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2512207598}) 2024-04-22T08:22:40.866 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.860+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3075910984}) 2024-04-22T08:22:40.867 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.860+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.867 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.860+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2512207598}) 2024-04-22T08:22:40.867 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.860+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3869521325}) 2024-04-22T08:22:40.867 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.861+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.867 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.861+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4142685637}) 2024-04-22T08:22:40.867 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.861+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2179679379}) 2024-04-22T08:22:40.867 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.861+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.867 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.861+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4142685637}) 2024-04-22T08:22:40.867 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.861+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=116896938}) 2024-04-22T08:22:40.867 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.862+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.867 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.862+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3723197676}) 2024-04-22T08:22:40.867 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.862+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2113611583}) 2024-04-22T08:22:40.867 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.862+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.867 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.862+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3723197676}) 2024-04-22T08:22:40.867 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.862+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1907394859}) 2024-04-22T08:22:40.867 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.862+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.868 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.862+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=987213335}) 2024-04-22T08:22:40.868 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.862+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=985268832}) 2024-04-22T08:22:40.868 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.862+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.868 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.862+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=987213335}) 2024-04-22T08:22:40.868 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.862+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2752976909}) 2024-04-22T08:22:40.868 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.863+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.868 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.863+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3381552304}) 2024-04-22T08:22:40.868 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.863+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=792232908}) 2024-04-22T08:22:40.868 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.863+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.868 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.863+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3381552304}) 2024-04-22T08:22:40.868 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.863+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4073184056}) 2024-04-22T08:22:40.868 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.864+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.868 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.864+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3950381169}) 2024-04-22T08:22:40.868 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.864+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2623741350}) 2024-04-22T08:22:40.868 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.864+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.869 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.864+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3950381169}) 2024-04-22T08:22:40.869 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.864+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4055823371}) 2024-04-22T08:22:40.869 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.865+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.869 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.865+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3824988555}) 2024-04-22T08:22:40.869 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.865+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=899547027}) 2024-04-22T08:22:40.869 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.865+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.869 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.865+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3824988555}) 2024-04-22T08:22:40.869 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.865+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=356917862}) 2024-04-22T08:22:40.869 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.865+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.869 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.865+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=416429248}) 2024-04-22T08:22:40.869 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.865+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=468440459}) 2024-04-22T08:22:40.869 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.865+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.869 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.865+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=416429248}) 2024-04-22T08:22:40.869 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.865+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=759506973}) 2024-04-22T08:22:40.869 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.866+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.869 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.866+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=7285832}) 2024-04-22T08:22:40.869 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.866+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=490398042}) 2024-04-22T08:22:40.870 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.866+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.870 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.866+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=7285832}) 2024-04-22T08:22:40.870 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.866+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1201804363}) 2024-04-22T08:22:40.870 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.867+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.870 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.867+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4143422311}) 2024-04-22T08:22:40.870 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.867+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4077185992}) 2024-04-22T08:22:40.870 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.867+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.870 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.867+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4143422311}) 2024-04-22T08:22:40.870 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.867+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=307529706}) 2024-04-22T08:22:40.871 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.868+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.871 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.868+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=51,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=8,monmap=3,osd_metadata=5,osd_pg_creating=1,osdmap=11} crc {logm=1029230047,mdsmap=1531151339,mgr=878880162,mgr_command_descs=470045235,mgr_metadata=4222303120,mgrstat=942895220,mon_config_key=4255067121,monmap=3469086636,osd_metadata=3694467291,osd_pg_creating=190880313,osdmap=3966010374}) 2024-04-22T08:22:40.871 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.868+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=51,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=8,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=11} crc {logm=3254555114,mdsmap=1531151339,mgr=878880162,mgr_command_descs=470045235,mgr_metadata=4222303120,mgrstat=1324828935,mon_config_key=4255067121,monmap=3469086636,osd_metadata=4151711904,osd_pg_creating=190880313,osdmap=2666812528}) 2024-04-22T08:22:40.871 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.868+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.872 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.868+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=51,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=8,monmap=3,osd_metadata=5,osd_pg_creating=1,osdmap=11} crc {logm=1029230047,mdsmap=1531151339,mgr=878880162,mgr_command_descs=470045235,mgr_metadata=4222303120,mgrstat=942895220,mon_config_key=4255067121,monmap=3469086636,osd_metadata=3694467291,osd_pg_creating=190880313,osdmap=3966010374}) 2024-04-22T08:22:40.872 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.868+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=50,mdsmap=3,mgr=7,mgr_command_descs=1,mgrstat=8,mon_config_key=8,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=13} crc {logm=479925389,mdsmap=1531151339,mgr=310612464,mgr_command_descs=470045235,mgrstat=942895220,mon_config_key=4255067121,monmap=3469086636,osd_metadata=4151711904,osd_pg_creating=190880313,osdmap=2375379959}) 2024-04-22T08:22:40.872 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.869+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.872 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.869+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3976976773}) 2024-04-22T08:22:40.872 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.869+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4217008055}) 2024-04-22T08:22:40.872 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.869+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.872 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.869+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3976976773}) 2024-04-22T08:22:40.872 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.869+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4077247219}) 2024-04-22T08:22:40.873 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.873+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.873 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.873+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=19} crc {osdmap=4118762048}) 2024-04-22T08:22:40.873 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.873+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=17} crc {osdmap=3640464001}) 2024-04-22T08:22:40.873 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.873+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-22T08:22:40.873 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.873+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=19} crc {osdmap=4118762048}) 2024-04-22T08:22:40.873 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-04-22T08:22:40.873+0000 7f63feab8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=20} crc {osdmap=1501087020}) 2024-04-22T08:22:41.864 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-22T08:22:41.864 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-22T08:22:41.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.975+0000 7fe8cc670640 10 monclient: get_monmap_and_config 2024-04-22T08:22:41.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.975+0000 7fe8cc670640 10 monclient: build_initial_monmap 2024-04-22T08:22:41.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.975+0000 7fe8cc670640 10 monclient: monmap: 2024-04-22T08:22:41.976 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:41.976 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:41.976 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:22:41.976097+0000 2024-04-22T08:22:41.976 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:22:41.976097+0000 2024-04-22T08:22:41.977 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:41.977 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:41.977 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:22:41.977 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:22:41.977 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:22:41.978 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:22:41.978 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:22:41.978 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:41.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.975+0000 7fe8cc670640 1 Processor -- start 2024-04-22T08:22:41.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.975+0000 7fe8cc670640 1 -- start start 2024-04-22T08:22:41.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.975+0000 7fe8cc670640 10 monclient: init 2024-04-22T08:22:41.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.976+0000 7fe8cc670640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:41.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.976+0000 7fe8cc670640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-04-22T08:22:41.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.976+0000 7fe8cc670640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe8c409f0b0 0x7fe8c409f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:41.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.976+0000 7fe8cc670640 10 monclient(hunting): picked mon.noname-a con 0x7fe8c409f0b0 addr v2:172.21.15.40:3300/0 2024-04-22T08:22:41.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.976+0000 7fe8cc670640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe8c409e2b0 addr v1:172.21.15.40:6789/0 2024-04-22T08:22:41.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.976+0000 7fe8cc670640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe8c4134230 addr v1:172.21.15.189:6789/0 2024-04-22T08:22:41.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.976+0000 7fe8cc670640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8c409ef10 con 0x7fe8c409e2b0 2024-04-22T08:22:41.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.976+0000 7fe8cc670640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8c4071670 con 0x7fe8c4134230 2024-04-22T08:22:41.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.976+0000 7fe8cc670640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:41.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.976+0000 7fe8cc670640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8c409d9e0 con 0x7fe8c409f0b0 2024-04-22T08:22:41.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.976+0000 7fe8cc670640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:41.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.976+0000 7fe8cc670640 10 monclient(hunting): authenticate will time out at 1435.970425s 2024-04-22T08:22:41.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.976+0000 7fe8ca40c640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe8c409f0b0 0x7fe8c409f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:41.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.976+0000 7fe8c9c0b640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fe8c409e2b0 0x7fe8c4133b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:49284/0 (socket says 172.21.15.40:49284) 2024-04-22T08:22:41.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.976+0000 7fe8ca40c640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe8c409f0b0 0x7fe8c409f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44450/0 (socket says 172.21.15.40:44450) 2024-04-22T08:22:41.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.976+0000 7fe8c9c0b640 1 -- 172.21.15.40:0/2134632405 learned_addr learned my addr 172.21.15.40:0/2134632405 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:22:41.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.976+0000 7fe8ca40c640 10 monclient(hunting): get_auth_request con 0x7fe8c409f0b0 auth_method 0 2024-04-22T08:22:41.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.976+0000 7fe8ca40c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:41.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.976+0000 7fe8ca40c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:41.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.976+0000 7fe8ca40c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:41.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.976+0000 7fe8ca40c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:41.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.976+0000 7fe8ca40c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:41.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.976+0000 7fe8ca40c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:41.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.977+0000 7fe8ca40c640 10 monclient(hunting): handle_auth_done global_id 114279 payload 274 2024-04-22T08:22:41.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.977+0000 7fe8ca40c640 1 -- 172.21.15.40:0/2134632405 >> v1:172.21.15.189:6789/0 conn(0x7fe8c4134230 legacy=0x7fe8c413be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:41.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.977+0000 7fe8ca40c640 1 -- 172.21.15.40:0/2134632405 >> v1:172.21.15.40:6789/0 conn(0x7fe8c409e2b0 legacy=0x7fe8c4133b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:41.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.977+0000 7fe8ca40c640 10 monclient: _finish_hunting 0 2024-04-22T08:22:41.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.977+0000 7fe8ca40c640 1 monclient: found mon.noname-a 2024-04-22T08:22:41.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.977+0000 7fe8ca40c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:41.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.977+0000 7fe8ca40c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:22:41.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.977+0000 7fe8ca40c640 1 -- 172.21.15.40:0/2134632405 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8c409d7a0 con 0x7fe8c409f0b0 2024-04-22T08:22:41.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.979+0000 7fe8ca40c640 1 -- 172.21.15.40:0/2134632405 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8c409f0b0 msgr2=0x7fe8c409f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:41.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.979+0000 7fe8ca40c640 1 --2- 172.21.15.40:0/2134632405 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8c409f0b0 0x7fe8c409f490 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe8bc009d90 tx=0x7fe8bc02bd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeb016279a3f5e4b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:41.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.980+0000 7fe8c940a640 1 -- 172.21.15.40:0/2134632405 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8bc02cda0 con 0x7fe8c409f0b0 2024-04-22T08:22:41.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.980+0000 7fe8c940a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:41.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.980+0000 7fe8c940a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:22:41.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.980+0000 7fe8c940a640 10 monclient: dump: 2024-04-22T08:22:41.993 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:41.993 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:41.993 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:41.993 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:41.993 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:41.993 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:41.993 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:41.993 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:41.993 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:41.993 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:41.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.980+0000 7fe8c940a640 10 monclient: _finish_auth 0 2024-04-22T08:22:41.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.980+0000 7fe8c940a640 10 monclient: _check_auth_tickets 2024-04-22T08:22:41.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.980+0000 7fe8c940a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:41.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.980+0000 7fe8c940a640 1 -- 172.21.15.40:0/2134632405 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8bc034440 con 0x7fe8c409f0b0 2024-04-22T08:22:41.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.980+0000 7fe8cc670640 5 monclient: authenticate success, global_id 114279 2024-04-22T08:22:41.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.980+0000 7fe8cc670640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:22:41.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.980+0000 7fe8c940a640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:41.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.980+0000 7fe8cc670640 10 monclient: get_monmap_and_config success 2024-04-22T08:22:41.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.980+0000 7fe8c940a640 1 -- 172.21.15.40:0/2134632405 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8bc0347d0 con 0x7fe8c409f0b0 2024-04-22T08:22:41.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.980+0000 7fe8c940a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:41.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.980+0000 7fe8cc670640 4 set_mon_vals no callback set 2024-04-22T08:22:41.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.980+0000 7fe8c940a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:22:41.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.980+0000 7fe8c940a640 10 monclient: dump: 2024-04-22T08:22:41.994 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:41.994 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:41.994 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:41.994 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:41.995 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:41.995 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:41.995 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:41.995 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:41.995 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:41.995 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:41.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.980+0000 7fe8cc670640 10 monclient: shutdown 2024-04-22T08:22:41.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.980+0000 7fe8cc670640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:41.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.980+0000 7fe8cc670640 1 -- 172.21.15.40:0/2134632405 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8c409f0b0 msgr2=0x7fe8c409f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:41.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.980+0000 7fe8cc670640 1 --2- 172.21.15.40:0/2134632405 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8c409f0b0 0x7fe8c409f490 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe8bc009d90 tx=0x7fe8bc02bd00 comp rx=0 tx=0).stop 2024-04-22T08:22:41.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.980+0000 7fe8cc670640 1 -- 172.21.15.40:0/2134632405 shutdown_connections 2024-04-22T08:22:41.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.980+0000 7fe8cc670640 1 --2- 172.21.15.40:0/2134632405 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8c409f0b0 0x7fe8c409f490 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:41.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.980+0000 7fe8cc670640 1 -- 172.21.15.40:0/2134632405 >> 172.21.15.40:0/2134632405 conn(0x7fe8c40a0b50 msgr2=0x7fe8c40a0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:41.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.980+0000 7fe8cc670640 1 -- 172.21.15.40:0/2134632405 shutdown_connections 2024-04-22T08:22:41.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.980+0000 7fe8cc670640 1 -- 172.21.15.40:0/2134632405 wait complete. 2024-04-22T08:22:41.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.981+0000 7fe8cc670640 10 monclient: build_initial_monmap 2024-04-22T08:22:41.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.981+0000 7fe8cc670640 10 monclient: monmap: 2024-04-22T08:22:41.996 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:41.996 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:22:41.996 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:22:41.996 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:22:41.996 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:41.996 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:41.996 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:22:41.996 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:22:41.996 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:22:41.997 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:41.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.981+0000 7fe8cc670640 1 Processor -- start 2024-04-22T08:22:41.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.981+0000 7fe8cc670640 1 -- start start 2024-04-22T08:22:41.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.981+0000 7fe8cc670640 10 monclient: init 2024-04-22T08:22:41.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8cc670640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:41.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8cc670640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-22T08:22:41.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8cc670640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8c409e2b0 0x7fe8c41caca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:41.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8cc670640 10 monclient(hunting): picked mon.noname-b con 0x7fe8c409e2b0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:22:41.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8cc670640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8c409f0b0 0x7fe8c4130c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:41.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8cc670640 10 monclient(hunting): picked mon.noname-a con 0x7fe8c409f0b0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:22:41.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8cc670640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8c4134230 0x7fe8c4131170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:41.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8cc670640 10 monclient(hunting): picked mon.noname-c con 0x7fe8c4134230 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:22:41.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8cc670640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:41.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8cc670640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c41cb1e0 con 0x7fe8c409f0b0 2024-04-22T08:22:41.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8cc670640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:41.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8cc670640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c4132020 con 0x7fe8c409e2b0 2024-04-22T08:22:41.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8cc670640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:41.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8cc670640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c4120330 con 0x7fe8c4134230 2024-04-22T08:22:41.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8cc670640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:41.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8cc670640 10 monclient(hunting): authenticate will time out at 1435.976034s 2024-04-22T08:22:41.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8c9c0b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8c409f0b0 0x7fe8c4130c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:41.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8cac0d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8c4134230 0x7fe8c4131170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:41.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8c9c0b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8c409f0b0 0x7fe8c4130c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44460/0 (socket says 172.21.15.40:44460) 2024-04-22T08:22:41.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8ca40c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8c409e2b0 0x7fe8c41caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:41.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8c9c0b640 1 -- 172.21.15.40:0/1795994509 learned_addr learned my addr 172.21.15.40:0/1795994509 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:22:41.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8c9c0b640 10 monclient(hunting): get_auth_request con 0x7fe8c409f0b0 auth_method 0 2024-04-22T08:22:41.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8c9c0b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:41.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8c9c0b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:41.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8c9c0b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:41.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8ca40c640 10 monclient(hunting): get_auth_request con 0x7fe8c409e2b0 auth_method 0 2024-04-22T08:22:41.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8ca40c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:41.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8ca40c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:41.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8ca40c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:41.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8cac0d640 10 monclient(hunting): get_auth_request con 0x7fe8c4134230 auth_method 0 2024-04-22T08:22:41.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8cac0d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:41.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8cac0d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:41.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8cac0d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:41.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8c9c0b640 1 -- 172.21.15.40:0/1795994509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8c409f0b0 msgr2=0x7fe8c4130c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-22T08:22:41.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8c9c0b640 1 -- 172.21.15.40:0/1795994509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8c409f0b0 msgr2=0x7fe8c4130c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:22:41.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8c9c0b640 1 --2- 172.21.15.40:0/1795994509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8c409f0b0 0x7fe8c4130c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:22:42.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8c9c0b640 1 --2- 172.21.15.40:0/1795994509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8c409f0b0 0x7fe8c4130c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:22:42.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8cac0d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:42.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8cac0d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:42.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8cac0d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:42.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8ca40c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:42.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8ca40c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:42.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8ca40c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:42.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8cac0d640 10 monclient(hunting): handle_auth_done global_id 124115 payload 824 2024-04-22T08:22:42.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8cac0d640 1 -- 172.21.15.40:0/1795994509 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8c409e2b0 msgr2=0x7fe8c41caca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:42.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8cac0d640 1 --2- 172.21.15.40:0/1795994509 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8c409e2b0 0x7fe8c41caca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:42.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8cac0d640 1 -- 172.21.15.40:0/1795994509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8c409f0b0 msgr2=0x7fe8c4130c30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:22:42.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8cac0d640 1 --2- 172.21.15.40:0/1795994509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8c409f0b0 0x7fe8c4130c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:42.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8cac0d640 10 monclient: _finish_hunting 0 2024-04-22T08:22:42.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8cac0d640 1 monclient: found mon.noname-c 2024-04-22T08:22:42.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8cac0d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:42.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8cac0d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:22:42.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.982+0000 7fe8cac0d640 1 -- 172.21.15.40:0/1795994509 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8bc009a70 con 0x7fe8c4134230 2024-04-22T08:22:42.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.985+0000 7fe8cac0d640 1 -- 172.21.15.40:0/1795994509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8c4134230 msgr2=0x7fe8c4131170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:42.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.985+0000 7fe8cac0d640 1 --2- 172.21.15.40:0/1795994509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8c4134230 0x7fe8c4131170 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe8c000c440 tx=0x7fe8c000c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:42.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.985+0000 7fe8baffd640 1 -- 172.21.15.40:0/1795994509 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c00119a0 con 0x7fe8c4134230 2024-04-22T08:22:42.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.985+0000 7fe8baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:42.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.985+0000 7fe8baffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:22:42.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.985+0000 7fe8baffd640 10 monclient: dump: 2024-04-22T08:22:42.001 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:42.001 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:42.001 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:42.001 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:42.001 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:42.002 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:42.002 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:42.002 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:42.002 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:42.002 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:42.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.985+0000 7fe8baffd640 10 monclient: _finish_auth 0 2024-04-22T08:22:42.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.985+0000 7fe8baffd640 10 monclient: _check_auth_tickets 2024-04-22T08:22:42.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.985+0000 7fe8baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:42.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.985+0000 7fe8cc670640 5 monclient: authenticate success, global_id 124115 2024-04-22T08:22:42.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.985+0000 7fe8baffd640 1 -- 172.21.15.40:0/1795994509 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8c0011b00 con 0x7fe8c4134230 2024-04-22T08:22:42.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.985+0000 7fe8cc670640 10 monclient: _renew_subs 2024-04-22T08:22:42.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.985+0000 7fe8cc670640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:22:42.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.985+0000 7fe8cc670640 1 -- 172.21.15.40:0/1795994509 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8c409cec0 con 0x7fe8c4134230 2024-04-22T08:22:42.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.985+0000 7fe8baffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:42.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.985+0000 7fe8baffd640 1 -- 172.21.15.40:0/1795994509 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c0012420 con 0x7fe8c4134230 2024-04-22T08:22:42.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.985+0000 7fe8baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:42.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.985+0000 7fe8baffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:22:42.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.985+0000 7fe8baffd640 10 monclient: dump: 2024-04-22T08:22:42.003 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:42.003 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:42.003 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:42.003 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:42.003 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:42.003 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:42.003 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:42.004 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:42.004 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:42.004 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:42.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.985+0000 7fe8c8c09640 4 set_mon_vals no callback set 2024-04-22T08:22:42.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.985+0000 7fe8cc670640 10 monclient: _renew_subs 2024-04-22T08:22:42.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.985+0000 7fe8cc670640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:22:42.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.985+0000 7fe8cc670640 1 -- 172.21.15.40:0/1795994509 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8c4120050 con 0x7fe8c4134230 2024-04-22T08:22:42.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.986+0000 7fe8baffd640 1 -- 172.21.15.40:0/1795994509 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fe8c00125c0 con 0x7fe8c4134230 2024-04-22T08:22:42.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.986+0000 7fe8cc670640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:42.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.986+0000 7fe8cc670640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:42.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.986+0000 7fe8cc670640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:22:42.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.986+0000 7fe8cc670640 1 -- 172.21.15.40:0/1795994509 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8c404e9b0 con 0x7fe8c4134230 2024-04-22T08:22:42.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.986+0000 7fe8baffd640 1 --2- 172.21.15.40:0/1795994509 >> v2:172.21.15.189:6800/30168 conn(0x7fe8a80377d0 0x7fe8a8039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:42.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.986+0000 7fe8baffd640 1 -- 172.21.15.40:0/1795994509 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(317..317 src has 239..317) v4 ==== 3506+0+0 (secure 0 0 0) 0x7fe8c004a100 con 0x7fe8c4134230 2024-04-22T08:22:42.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.986+0000 7fe8ca40c640 1 --2- 172.21.15.40:0/1795994509 >> v2:172.21.15.189:6800/30168 conn(0x7fe8a80377d0 0x7fe8a8039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:42.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.986+0000 7fe8ca40c640 10 monclient: get_auth_request con 0x7fe8a80377d0 auth_method 0 2024-04-22T08:22:42.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.989+0000 7fe8ca40c640 1 --2- 172.21.15.40:0/1795994509 >> v2:172.21.15.189:6800/30168 conn(0x7fe8a80377d0 0x7fe8a8039c90 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe8c406c890 tx=0x7fe8bc030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:42.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.991+0000 7fe8baffd640 1 -- 172.21.15.40:0/1795994509 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7fe8c0010d40 con 0x7fe8c4134230 2024-04-22T08:22:42.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.991+0000 7fe8baffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:42.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:41.991+0000 7fe8baffd640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:22:42.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.241+0000 7fe8cc670640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-22T08:22:42.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.241+0000 7fe8cc670640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-22T08:22:42.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.241+0000 7fe8cc670640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:22:42.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.241+0000 7fe8cc670640 1 -- 172.21.15.40:0/1795994509 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe8c411ee10 con 0x7fe8c4134230 2024-04-22T08:22:42.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.241+0000 7fe8baffd640 1 -- 172.21.15.40:0/1795994509 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7fe8c00102e0 con 0x7fe8c4134230 2024-04-22T08:22:42.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.241+0000 7fe8baffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-22T08:22:42.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.241+0000 7fe8baffd640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:22:42.242 INFO:teuthology.orchestra.run.smithi040.stdout:{"election_epoch":98,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","modified":"2024-04-22T08:12:05.320036Z","created":"2024-04-22T08:12:05.320036Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3301","nonce":0},{"type":"v1","addr":"172.21.15.40:6790","nonce":0}]},"addr":"172.21.15.40:6790/0","public_addr":"172.21.15.40:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-22T08:22:42.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.243+0000 7fe8cc670640 1 -- 172.21.15.40:0/1795994509 >> v2:172.21.15.189:6800/30168 conn(0x7fe8a80377d0 msgr2=0x7fe8a8039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:42.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.243+0000 7fe8cc670640 1 --2- 172.21.15.40:0/1795994509 >> v2:172.21.15.189:6800/30168 conn(0x7fe8a80377d0 0x7fe8a8039c90 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe8c406c890 tx=0x7fe8bc030000 comp rx=0 tx=0).stop 2024-04-22T08:22:42.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.243+0000 7fe8cc670640 10 monclient: shutdown 2024-04-22T08:22:42.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.243+0000 7fe8cc670640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:42.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.243+0000 7fe8cc670640 1 -- 172.21.15.40:0/1795994509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8c4134230 msgr2=0x7fe8c4131170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:42.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.243+0000 7fe8cc670640 1 --2- 172.21.15.40:0/1795994509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8c4134230 0x7fe8c4131170 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe8c000c440 tx=0x7fe8c000c900 comp rx=0 tx=0).stop 2024-04-22T08:22:42.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.243+0000 7fe8cc670640 1 -- 172.21.15.40:0/1795994509 shutdown_connections 2024-04-22T08:22:42.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.243+0000 7fe8cc670640 1 --2- 172.21.15.40:0/1795994509 >> v2:172.21.15.189:6800/30168 conn(0x7fe8a80377d0 0x7fe8a8039c90 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:42.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.243+0000 7fe8cc670640 1 --2- 172.21.15.40:0/1795994509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8c4134230 0x7fe8c4131170 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:42.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.243+0000 7fe8cc670640 1 --2- 172.21.15.40:0/1795994509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8c409f0b0 0x7fe8c4130c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:42.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.243+0000 7fe8cc670640 1 --2- 172.21.15.40:0/1795994509 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8c409e2b0 0x7fe8c41caca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:42.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.243+0000 7fe8cc670640 1 -- 172.21.15.40:0/1795994509 >> 172.21.15.40:0/1795994509 conn(0x7fe8c40a0b50 msgr2=0x7fe8c40a2ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:42.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.243+0000 7fe8cc670640 1 -- 172.21.15.40:0/1795994509 shutdown_connections 2024-04-22T08:22:42.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.243+0000 7fe8cc670640 1 -- 172.21.15.40:0/1795994509 wait complete. 2024-04-22T08:22:42.253 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-22T08:22:42.253 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-22T08:22:42.262 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-22T08:22:42.262 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:22:42.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.361+0000 7f8c91beb640 10 monclient: get_monmap_and_config 2024-04-22T08:22:42.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.361+0000 7f8c91beb640 10 monclient: build_initial_monmap 2024-04-22T08:22:42.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.361+0000 7f8c91beb640 10 monclient: monmap: 2024-04-22T08:22:42.361 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:42.361 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:42.361 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:22:42.361948+0000 2024-04-22T08:22:42.361 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:22:42.361948+0000 2024-04-22T08:22:42.361 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:42.361 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:42.361 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:22:42.361 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:22:42.361 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:22:42.361 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:22:42.361 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:22:42.361 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:42.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.361+0000 7f8c91beb640 1 Processor -- start 2024-04-22T08:22:42.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.361+0000 7f8c91beb640 1 -- start start 2024-04-22T08:22:42.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.361+0000 7f8c91beb640 10 monclient: init 2024-04-22T08:22:42.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c91beb640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:42.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c91beb640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-04-22T08:22:42.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c91beb640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8c8c09de00 0x7f8c8c09e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:42.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c91beb640 10 monclient(hunting): picked mon.noname-c con 0x7f8c8c09de00 addr v2:172.21.15.189:3300/0 2024-04-22T08:22:42.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c91beb640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8c8c0a4330 0x7f8c8c09e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:42.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c91beb640 10 monclient(hunting): picked mon.noname-b con 0x7f8c8c0a4330 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:22:42.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c91beb640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8c8c0a0240 0x7f8c8c135bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:42.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c91beb640 10 monclient(hunting): picked mon.noname-a con 0x7f8c8c0a0240 addr v2:172.21.15.40:3300/0 2024-04-22T08:22:42.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c91beb640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:42.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c91beb640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c8c09fb40 con 0x7f8c8c0a0240 2024-04-22T08:22:42.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c91beb640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:42.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c91beb640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c8c09ec60 con 0x7f8c8c09de00 2024-04-22T08:22:42.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c91beb640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:42.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c91beb640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8c0a3410 con 0x7f8c8c0a4330 2024-04-22T08:22:42.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c91beb640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:42.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c91beb640 10 monclient(hunting): authenticate will time out at 1436.356142s 2024-04-22T08:22:42.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c8affd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8c8c0a4330 0x7f8c8c09e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:42.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c8bfff640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8c8c0a0240 0x7f8c8c135bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:42.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c8affd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8c8c0a4330 0x7f8c8c09e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49758/0 (socket says 172.21.15.40:49758) 2024-04-22T08:22:42.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c8affd640 1 -- 172.21.15.40:0/4078461293 learned_addr learned my addr 172.21.15.40:0/4078461293 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:22:42.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c8affd640 10 monclient(hunting): get_auth_request con 0x7f8c8c0a4330 auth_method 0 2024-04-22T08:22:42.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c8affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:42.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c8affd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:42.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c8affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:42.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c8bfff640 10 monclient(hunting): get_auth_request con 0x7f8c8c0a0240 auth_method 0 2024-04-22T08:22:42.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:42.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c8bfff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:42.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:42.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c8affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:42.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c8affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:42.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c8affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:42.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c8bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:42.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c8bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:42.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c8b7fe640 1 --2- 172.21.15.40:0/4078461293 >> v2:172.21.15.189:3300/0 conn(0x7f8c8c09de00 0x7f8c8c09e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:42.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c8bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:42.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c8b7fe640 10 monclient(hunting): get_auth_request con 0x7f8c8c09de00 auth_method 0 2024-04-22T08:22:42.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:42.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c8b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:42.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:42.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c8affd640 10 monclient(hunting): handle_auth_done global_id 124118 payload 274 2024-04-22T08:22:42.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c8affd640 1 -- 172.21.15.40:0/4078461293 >> v2:172.21.15.189:3300/0 conn(0x7f8c8c09de00 msgr2=0x7f8c8c09e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:42.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c8affd640 1 --2- 172.21.15.40:0/4078461293 >> v2:172.21.15.189:3300/0 conn(0x7f8c8c09de00 0x7f8c8c09e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:42.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c8affd640 1 -- 172.21.15.40:0/4078461293 >> v2:172.21.15.40:3300/0 conn(0x7f8c8c0a0240 msgr2=0x7f8c8c135bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:42.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c8affd640 1 --2- 172.21.15.40:0/4078461293 >> v2:172.21.15.40:3300/0 conn(0x7f8c8c0a0240 0x7f8c8c135bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:42.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c8affd640 10 monclient: _finish_hunting 0 2024-04-22T08:22:42.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c8affd640 1 monclient: found mon.noname-b 2024-04-22T08:22:42.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c8affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:42.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c8affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:22:42.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c8affd640 1 -- 172.21.15.40:0/4078461293 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c8c0a4b00 con 0x7f8c8c0a4330 2024-04-22T08:22:42.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.362+0000 7f8c8bfff640 1 --2- 172.21.15.40:0/4078461293 >> v2:172.21.15.40:3300/0 conn(0x7f8c8c0a0240 0x7f8c8c135bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:22:42.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.365+0000 7f8c8affd640 1 -- 172.21.15.40:0/4078461293 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8c8c0a4330 msgr2=0x7f8c8c09e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:42.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.365+0000 7f8c8affd640 1 --2- 172.21.15.40:0/4078461293 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8c8c0a4330 0x7f8c8c09e720 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8c8c0a94b0 tx=0x7f8c8002b7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e31fb6ace1cdbbe server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:42.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.365+0000 7f8c8a7fc640 1 -- 172.21.15.40:0/4078461293 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c80031960 con 0x7f8c8c0a4330 2024-04-22T08:22:42.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.365+0000 7f8c8a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:42.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.365+0000 7f8c8a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:22:42.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.365+0000 7f8c8a7fc640 10 monclient: dump: 2024-04-22T08:22:42.374 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:42.374 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:42.374 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:42.374 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:42.374 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:42.374 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:42.374 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:42.374 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:42.374 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:42.374 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:42.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.365+0000 7f8c8a7fc640 10 monclient: _finish_auth 0 2024-04-22T08:22:42.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.365+0000 7f8c8a7fc640 10 monclient: _check_auth_tickets 2024-04-22T08:22:42.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.365+0000 7f8c8a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:42.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.365+0000 7f8c91beb640 5 monclient: authenticate success, global_id 124118 2024-04-22T08:22:42.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.365+0000 7f8c8a7fc640 1 -- 172.21.15.40:0/4078461293 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c80031ac0 con 0x7f8c8c0a4330 2024-04-22T08:22:42.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.365+0000 7f8c91beb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:22:42.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.365+0000 7f8c8a7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:42.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.365+0000 7f8c91beb640 10 monclient: get_monmap_and_config success 2024-04-22T08:22:42.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.365+0000 7f8c91beb640 4 set_mon_vals no callback set 2024-04-22T08:22:42.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.365+0000 7f8c8a7fc640 1 -- 172.21.15.40:0/4078461293 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c80031e50 con 0x7f8c8c0a4330 2024-04-22T08:22:42.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.365+0000 7f8c8a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:42.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.365+0000 7f8c8a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:22:42.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.365+0000 7f8c8a7fc640 10 monclient: dump: 2024-04-22T08:22:42.375 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:42.375 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:42.375 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:42.375 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:42.376 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:42.376 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:42.376 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:42.376 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:42.376 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:42.376 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:42.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.366+0000 7f8c91beb640 10 monclient: shutdown 2024-04-22T08:22:42.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.366+0000 7f8c91beb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:42.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.366+0000 7f8c91beb640 1 -- 172.21.15.40:0/4078461293 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8c8c0a4330 msgr2=0x7f8c8c09e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:42.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.366+0000 7f8c91beb640 1 --2- 172.21.15.40:0/4078461293 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8c8c0a4330 0x7f8c8c09e720 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8c8c0a94b0 tx=0x7f8c8002b7f0 comp rx=0 tx=0).stop 2024-04-22T08:22:42.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.366+0000 7f8c91beb640 1 -- 172.21.15.40:0/4078461293 shutdown_connections 2024-04-22T08:22:42.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.366+0000 7f8c91beb640 1 --2- 172.21.15.40:0/4078461293 >> v2:172.21.15.40:3300/0 conn(0x7f8c8c0a0240 0x7f8c8c135bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:42.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.366+0000 7f8c91beb640 1 --2- 172.21.15.40:0/4078461293 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8c8c0a4330 0x7f8c8c09e720 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:42.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.366+0000 7f8c91beb640 1 --2- 172.21.15.40:0/4078461293 >> v2:172.21.15.189:3300/0 conn(0x7f8c8c09de00 0x7f8c8c09e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:42.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.366+0000 7f8c91beb640 1 -- 172.21.15.40:0/4078461293 >> 172.21.15.40:0/4078461293 conn(0x7f8c8c0a83f0 msgr2=0x7f8c8c0a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:42.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.366+0000 7f8c91beb640 1 -- 172.21.15.40:0/4078461293 shutdown_connections 2024-04-22T08:22:42.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.366+0000 7f8c91beb640 1 -- 172.21.15.40:0/4078461293 wait complete. 2024-04-22T08:22:42.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.366+0000 7f8c91beb640 10 monclient: build_initial_monmap 2024-04-22T08:22:42.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.367+0000 7f8c91beb640 10 monclient: monmap: 2024-04-22T08:22:42.377 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:42.377 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:22:42.377 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:22:42.377 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:22:42.377 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:42.377 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:42.377 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:22:42.377 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:22:42.377 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:22:42.377 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:42.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.367+0000 7f8c91beb640 1 Processor -- start 2024-04-22T08:22:42.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.367+0000 7f8c91beb640 1 -- start start 2024-04-22T08:22:42.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.367+0000 7f8c91beb640 10 monclient: init 2024-04-22T08:22:42.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.367+0000 7f8c91beb640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:42.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.367+0000 7f8c91beb640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:22:42.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.367+0000 7f8c91beb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c8c09de00 0x7f8c8c1cad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:42.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.367+0000 7f8c91beb640 10 monclient(hunting): picked mon.noname-a con 0x7f8c8c09de00 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:22:42.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.367+0000 7f8c91beb640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c8c0a0240 0x7f8c8c1cb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:42.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.367+0000 7f8c91beb640 10 monclient(hunting): picked mon.noname-b con 0x7f8c8c0a0240 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:22:42.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.367+0000 7f8c91beb640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8c8c124550 0x7f8c8c120180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:42.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.367+0000 7f8c91beb640 10 monclient(hunting): picked mon.noname-c con 0x7f8c8c124550 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:22:42.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.367+0000 7f8c91beb640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:42.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.367+0000 7f8c91beb640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8c123890 con 0x7f8c8c09de00 2024-04-22T08:22:42.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.367+0000 7f8c91beb640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:42.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.367+0000 7f8c91beb640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8c1229a0 con 0x7f8c8c0a0240 2024-04-22T08:22:42.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.367+0000 7f8c91beb640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:42.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.367+0000 7f8c91beb640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8c0a1c30 con 0x7f8c8c124550 2024-04-22T08:22:42.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.367+0000 7f8c91beb640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:42.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.367+0000 7f8c91beb640 10 monclient(hunting): authenticate will time out at 1436.361730s 2024-04-22T08:22:42.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.368+0000 7f8c8b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c8c09de00 0x7f8c8c1cad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:42.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.368+0000 7f8c8bfff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8c8c124550 0x7f8c8c120180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:42.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.368+0000 7f8c8b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c8c09de00 0x7f8c8c1cad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44480/0 (socket says 172.21.15.40:44480) 2024-04-22T08:22:42.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.368+0000 7f8c8b7fe640 1 -- 172.21.15.40:0/574562814 learned_addr learned my addr 172.21.15.40:0/574562814 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:22:42.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.368+0000 7f8c8b7fe640 10 monclient(hunting): get_auth_request con 0x7f8c8c09de00 auth_method 0 2024-04-22T08:22:42.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.368+0000 7f8c8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:42.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.368+0000 7f8c8b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:42.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.368+0000 7f8c8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:42.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.368+0000 7f8c8bfff640 10 monclient(hunting): get_auth_request con 0x7f8c8c124550 auth_method 0 2024-04-22T08:22:42.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.368+0000 7f8c8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:42.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.368+0000 7f8c8bfff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:42.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.368+0000 7f8c8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:42.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.368+0000 7f8c8bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:42.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.368+0000 7f8c8bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:42.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.368+0000 7f8c8bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:42.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.368+0000 7f8c8b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:42.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.368+0000 7f8c8b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:42.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.368+0000 7f8c8b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:42.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.368+0000 7f8c8affd640 1 --2- 172.21.15.40:0/574562814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c8c0a0240 0x7f8c8c1cb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:42.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.368+0000 7f8c8affd640 10 monclient(hunting): get_auth_request con 0x7f8c8c0a0240 auth_method 0 2024-04-22T08:22:42.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.368+0000 7f8c8affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:42.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.368+0000 7f8c8affd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:42.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.368+0000 7f8c8affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:42.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.368+0000 7f8c8bfff640 10 monclient(hunting): handle_auth_done global_id 124121 payload 824 2024-04-22T08:22:42.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.368+0000 7f8c8bfff640 1 -- 172.21.15.40:0/574562814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c8c0a0240 msgr2=0x7f8c8c1cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:42.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.368+0000 7f8c8bfff640 1 --2- 172.21.15.40:0/574562814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c8c0a0240 0x7f8c8c1cb2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:42.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.368+0000 7f8c8bfff640 1 -- 172.21.15.40:0/574562814 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c8c09de00 msgr2=0x7f8c8c1cad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:42.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.368+0000 7f8c8bfff640 1 --2- 172.21.15.40:0/574562814 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c8c09de00 0x7f8c8c1cad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:42.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.368+0000 7f8c8bfff640 10 monclient: _finish_hunting 0 2024-04-22T08:22:42.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.368+0000 7f8c8bfff640 1 monclient: found mon.noname-c 2024-04-22T08:22:42.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.368+0000 7f8c8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:42.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.368+0000 7f8c8bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:22:42.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.368+0000 7f8c8bfff640 1 -- 172.21.15.40:0/574562814 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c84009a70 con 0x7f8c8c124550 2024-04-22T08:22:42.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.368+0000 7f8c8b7fe640 1 --2- 172.21.15.40:0/574562814 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c8c09de00 0x7f8c8c1cad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:22:42.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.370+0000 7f8c8bfff640 1 -- 172.21.15.40:0/574562814 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8c8c124550 msgr2=0x7f8c8c120180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:42.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.370+0000 7f8c8bfff640 1 --2- 172.21.15.40:0/574562814 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8c8c124550 0x7f8c8c120180 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8c8400efc0 tx=0x7f8c8400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:42.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.370+0000 7f8c73fff640 1 -- 172.21.15.40:0/574562814 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c84003990 con 0x7f8c8c124550 2024-04-22T08:22:42.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.370+0000 7f8c73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:42.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.370+0000 7f8c73fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:22:42.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.370+0000 7f8c73fff640 10 monclient: dump: 2024-04-22T08:22:42.382 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:42.382 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:42.382 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:42.383 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:42.383 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:42.383 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:42.383 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:42.383 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:42.383 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:42.383 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:42.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.370+0000 7f8c73fff640 10 monclient: _finish_auth 0 2024-04-22T08:22:42.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.370+0000 7f8c73fff640 10 monclient: _check_auth_tickets 2024-04-22T08:22:42.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.370+0000 7f8c73fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:42.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.370+0000 7f8c73fff640 1 -- 172.21.15.40:0/574562814 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c8400f040 con 0x7f8c8c124550 2024-04-22T08:22:42.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.370+0000 7f8c91beb640 5 monclient: authenticate success, global_id 124121 2024-04-22T08:22:42.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.370+0000 7f8c91beb640 10 monclient: _renew_subs 2024-04-22T08:22:42.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.370+0000 7f8c91beb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:22:42.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.370+0000 7f8c91beb640 1 -- 172.21.15.40:0/574562814 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c800094d0 con 0x7f8c8c124550 2024-04-22T08:22:42.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.370+0000 7f8c73fff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:42.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.370+0000 7f8c73fff640 1 -- 172.21.15.40:0/574562814 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c84003d50 con 0x7f8c8c124550 2024-04-22T08:22:42.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.370+0000 7f8c73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:42.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.370+0000 7f8c73fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:22:42.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.370+0000 7f8c73fff640 10 monclient: dump: 2024-04-22T08:22:42.384 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:42.384 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:42.384 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:42.384 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:42.384 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:42.384 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:42.384 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:42.384 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:42.384 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:42.385 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:42.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.370+0000 7f8c89ffb640 4 set_mon_vals no callback set 2024-04-22T08:22:42.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.370+0000 7f8c91beb640 10 monclient: _renew_subs 2024-04-22T08:22:42.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.370+0000 7f8c91beb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:22:42.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.370+0000 7f8c91beb640 1 -- 172.21.15.40:0/574562814 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c8c124130 con 0x7f8c8c124550 2024-04-22T08:22:42.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.371+0000 7f8c73fff640 1 -- 172.21.15.40:0/574562814 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f8c84003eb0 con 0x7f8c8c124550 2024-04-22T08:22:42.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.371+0000 7f8c91beb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:42.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.371+0000 7f8c91beb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:42.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.371+0000 7f8c91beb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:22:42.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.371+0000 7f8c91beb640 1 -- 172.21.15.40:0/574562814 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c8c134dd0 con 0x7f8c8c124550 2024-04-22T08:22:42.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.371+0000 7f8c73fff640 1 --2- 172.21.15.40:0/574562814 >> v2:172.21.15.189:6800/30168 conn(0x7f8c780377d0 0x7f8c78039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:42.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.371+0000 7f8c73fff640 1 -- 172.21.15.40:0/574562814 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(317..317 src has 239..317) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f8c8404ad90 con 0x7f8c8c124550 2024-04-22T08:22:42.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.372+0000 7f8c8b7fe640 1 --2- 172.21.15.40:0/574562814 >> v2:172.21.15.189:6800/30168 conn(0x7f8c780377d0 0x7f8c78039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:42.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.372+0000 7f8c8b7fe640 10 monclient: get_auth_request con 0x7f8c780377d0 auth_method 0 2024-04-22T08:22:42.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.374+0000 7f8c8b7fe640 1 --2- 172.21.15.40:0/574562814 >> v2:172.21.15.189:6800/30168 conn(0x7f8c780377d0 0x7f8c78039c90 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c006fd0 tx=0x7f8c7c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:42.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.376+0000 7f8c73fff640 1 -- 172.21.15.40:0/574562814 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f8c8404f070 con 0x7f8c8c124550 2024-04-22T08:22:42.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.376+0000 7f8c73fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:42.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.376+0000 7f8c73fff640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:22:42.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.552+0000 7f8c91beb640 1 -- 172.21.15.40:0/574562814 --> v2:172.21.15.189:6800/30168 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8c8c1360f0 con 0x7f8c780377d0 2024-04-22T08:22:42.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.554+0000 7f8c73fff640 1 -- 172.21.15.40:0/574562814 <== mgr.4102 v2:172.21.15.189:6800/30168 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+53849 (secure 0 0 0) 0x7f8c7c010e20 con 0x7f8c780377d0 2024-04-22T08:22:42.554 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:22:42.555 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-22T08:22:42.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.556+0000 7f8c91beb640 1 -- 172.21.15.40:0/574562814 >> v2:172.21.15.189:6800/30168 conn(0x7f8c780377d0 msgr2=0x7f8c78039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:42.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.556+0000 7f8c91beb640 1 --2- 172.21.15.40:0/574562814 >> v2:172.21.15.189:6800/30168 conn(0x7f8c780377d0 0x7f8c78039c90 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c006fd0 tx=0x7f8c7c008040 comp rx=0 tx=0).stop 2024-04-22T08:22:42.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.556+0000 7f8c91beb640 10 monclient: shutdown 2024-04-22T08:22:42.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.556+0000 7f8c91beb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:42.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.556+0000 7f8c91beb640 1 -- 172.21.15.40:0/574562814 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8c8c124550 msgr2=0x7f8c8c120180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:42.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.556+0000 7f8c91beb640 1 --2- 172.21.15.40:0/574562814 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8c8c124550 0x7f8c8c120180 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8c8400efc0 tx=0x7f8c8400c490 comp rx=0 tx=0).stop 2024-04-22T08:22:42.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.556+0000 7f8c91beb640 1 -- 172.21.15.40:0/574562814 shutdown_connections 2024-04-22T08:22:42.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.556+0000 7f8c91beb640 1 --2- 172.21.15.40:0/574562814 >> v2:172.21.15.189:6800/30168 conn(0x7f8c780377d0 0x7f8c78039c90 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:42.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.556+0000 7f8c91beb640 1 --2- 172.21.15.40:0/574562814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c8c0a0240 0x7f8c8c1cb2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:42.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.556+0000 7f8c91beb640 1 --2- 172.21.15.40:0/574562814 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8c8c124550 0x7f8c8c120180 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:42.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.556+0000 7f8c91beb640 1 --2- 172.21.15.40:0/574562814 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c8c09de00 0x7f8c8c1cad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:42.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.556+0000 7f8c91beb640 1 -- 172.21.15.40:0/574562814 >> 172.21.15.40:0/574562814 conn(0x7f8c8c0a83f0 msgr2=0x7f8c8c0a6ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:42.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.556+0000 7f8c91beb640 1 -- 172.21.15.40:0/574562814 shutdown_connections 2024-04-22T08:22:42.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.556+0000 7f8c91beb640 1 -- 172.21.15.40:0/574562814 wait complete. 2024-04-22T08:22:42.566 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":612,"stamp":"2024-04-22T08:22:42.206057+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":185760,"kb_used_data":7232,"kb_used_omap":9,"kb_used_meta":178422,"kb_avail":566045280,"statfs":{"total":579820584960,"available":579630366720,"internally_reserved":0,"allocated":7405568,"data_stored":3236636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9560,"internal_metadata":182704808},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002606"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":304,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-04-22T08:22:17.159438+0000","last_change":"2024-04-22T08:12:51.443086+0000","last_active":"2024-04-22T08:22:17.159438+0000","last_peered":"2024-04-22T08:22:17.159438+0000","last_clean":"2024-04-22T08:22:17.159438+0000","last_became_active":"2024-04-22T08:12:51.442730+0000","last_became_peered":"2024-04-22T08:12:51.442730+0000","last_unstale":"2024-04-22T08:22:17.159438+0000","last_undegraded":"2024-04-22T08:22:17.159438+0000","last_fullsized":"2024-04-22T08:22:17.159438+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:06:13.481863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":293,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-04-22T08:22:16.957788+0000","last_change":"2024-04-22T08:12:50.841276+0000","last_active":"2024-04-22T08:22:16.957788+0000","last_peered":"2024-04-22T08:22:16.957788+0000","last_clean":"2024-04-22T08:22:16.957788+0000","last_became_active":"2024-04-22T08:12:50.841008+0000","last_became_peered":"2024-04-22T08:12:50.841008+0000","last_unstale":"2024-04-22T08:22:16.957788+0000","last_undegraded":"2024-04-22T08:22:16.957788+0000","last_fullsized":"2024-04-22T08:22:16.957788+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:47:00.224321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":304,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-04-22T08:22:17.160920+0000","last_change":"2024-04-22T08:12:51.403419+0000","last_active":"2024-04-22T08:22:17.160920+0000","last_peered":"2024-04-22T08:22:17.160920+0000","last_clean":"2024-04-22T08:22:17.160920+0000","last_became_active":"2024-04-22T08:12:51.403182+0000","last_became_peered":"2024-04-22T08:12:51.403182+0000","last_unstale":"2024-04-22T08:22:17.160920+0000","last_undegraded":"2024-04-22T08:22:17.160920+0000","last_fullsized":"2024-04-22T08:22:17.160920+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:52:39.730597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":293,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-04-22T08:22:16.957078+0000","last_change":"2024-04-22T08:12:51.403260+0000","last_active":"2024-04-22T08:22:16.957078+0000","last_peered":"2024-04-22T08:22:16.957078+0000","last_clean":"2024-04-22T08:22:16.957078+0000","last_became_active":"2024-04-22T08:12:51.402980+0000","last_became_peered":"2024-04-22T08:12:51.402980+0000","last_unstale":"2024-04-22T08:22:16.957078+0000","last_undegraded":"2024-04-22T08:22:16.957078+0000","last_fullsized":"2024-04-22T08:22:16.957078+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:08:04.625870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":295,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-04-22T08:22:17.162182+0000","last_change":"2024-04-22T08:12:50.836566+0000","last_active":"2024-04-22T08:22:17.162182+0000","last_peered":"2024-04-22T08:22:17.162182+0000","last_clean":"2024-04-22T08:22:17.162182+0000","last_became_active":"2024-04-22T08:12:50.836250+0000","last_became_peered":"2024-04-22T08:12:50.836250+0000","last_unstale":"2024-04-22T08:22:17.162182+0000","last_undegraded":"2024-04-22T08:22:17.162182+0000","last_fullsized":"2024-04-22T08:22:17.162182+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:42:13.693209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":302,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-04-22T08:22:16.950081+0000","last_change":"2024-04-22T08:12:50.836508+0000","last_active":"2024-04-22T08:22:16.950081+0000","last_peered":"2024-04-22T08:22:16.950081+0000","last_clean":"2024-04-22T08:22:16.950081+0000","last_became_active":"2024-04-22T08:12:50.836210+0000","last_became_peered":"2024-04-22T08:12:50.836210+0000","last_unstale":"2024-04-22T08:22:16.950081+0000","last_undegraded":"2024-04-22T08:22:16.950081+0000","last_fullsized":"2024-04-22T08:22:16.950081+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:24:53.345661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":304,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-04-22T08:22:17.158781+0000","last_change":"2024-04-22T08:12:50.842633+0000","last_active":"2024-04-22T08:22:17.158781+0000","last_peered":"2024-04-22T08:22:17.158781+0000","last_clean":"2024-04-22T08:22:17.158781+0000","last_became_active":"2024-04-22T08:12:50.842392+0000","last_became_peered":"2024-04-22T08:12:50.842392+0000","last_unstale":"2024-04-22T08:22:17.158781+0000","last_undegraded":"2024-04-22T08:22:17.158781+0000","last_fullsized":"2024-04-22T08:22:17.158781+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:53:14.717539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":295,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-04-22T08:22:17.162502+0000","last_change":"2024-04-22T08:12:50.835820+0000","last_active":"2024-04-22T08:22:17.162502+0000","last_peered":"2024-04-22T08:22:17.162502+0000","last_clean":"2024-04-22T08:22:17.162502+0000","last_became_active":"2024-04-22T08:12:50.835541+0000","last_became_peered":"2024-04-22T08:12:50.835541+0000","last_unstale":"2024-04-22T08:22:17.162502+0000","last_undegraded":"2024-04-22T08:22:17.162502+0000","last_fullsized":"2024-04-22T08:22:17.162502+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:52:21.798509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":454,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-04-22T08:22:17.158877+0000","last_change":"2024-04-22T08:12:51.404305+0000","last_active":"2024-04-22T08:22:17.158877+0000","last_peered":"2024-04-22T08:22:17.158877+0000","last_clean":"2024-04-22T08:22:17.158877+0000","last_became_active":"2024-04-22T08:12:51.403208+0000","last_became_peered":"2024-04-22T08:12:51.403208+0000","last_unstale":"2024-04-22T08:22:17.158877+0000","last_undegraded":"2024-04-22T08:22:17.158877+0000","last_fullsized":"2024-04-22T08:22:17.158877+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:04:43.253802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640379,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28056,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343784,"statfs":{"total":96636764160,"available":96608034816,"internally_reserved":0,"allocated":1056768,"data_stored":363130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.35199999999999998,"15min":0.371},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":0.39800000000000002,"5min":2.5920000000000001,"15min":2.5920000000000001},"last":0.255},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.40899999999999997,"15min":0.38700000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.21299999999999999},"max":{"1min":0.51400000000000001,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.30599999999999999}]},{"osd":1,"last update":"Mon Apr 22 08:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.34300000000000003,"15min":0.34399999999999997},"min":{"1min":0.249,"5min":0.23400000000000001,"15min":0.17100000000000001},"max":{"1min":0.443,"5min":0.499,"15min":0.55700000000000005},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.38100000000000001,"15min":0.34499999999999997},"min":{"1min":0.254,"5min":0.24299999999999999,"15min":0.20699999999999999},"max":{"1min":0.48499999999999999,"5min":0.55600000000000005,"15min":0.70099999999999996},"last":0.33300000000000002}]},{"osd":3,"last update":"Mon Apr 22 08:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.59699999999999998,"15min":0.56499999999999995},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.22500000000000001},"max":{"1min":0.82999999999999996,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.58799999999999997,"15min":0.60199999999999998},"min":{"1min":0.33400000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.749,"5min":0.84799999999999998,"15min":1.8440000000000001},"last":0.45600000000000002}]},{"osd":4,"last update":"Mon Apr 22 08:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58399999999999996,"15min":0.68600000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.30499999999999999},"max":{"1min":0.78800000000000003,"5min":2.7879999999999998,"15min":6.2949999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.55100000000000005,"15min":0.68400000000000005},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.69499999999999995,"5min":0.91200000000000003,"15min":6.2149999999999999},"last":0.50900000000000001}]},{"osd":5,"last update":"Mon Apr 22 08:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.56899999999999995,"15min":0.53400000000000003},"min":{"1min":0.40000000000000002,"5min":0.35799999999999998,"15min":0.27800000000000002},"max":{"1min":0.72499999999999998,"5min":0.84299999999999997,"15min":0.85199999999999998},"last":0.441},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56399999999999995,"15min":0.53500000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.29699999999999999},"max":{"1min":0.72999999999999998,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.46999999999999997}]}]},{"osd":0,"up_from":11,"seq":47244640378,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28580,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343260,"statfs":{"total":96636764160,"available":96607498240,"internally_reserved":0,"allocated":1589248,"data_stored":892058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.32000000000000001,"15min":0.34599999999999997},"min":{"1min":0.23699999999999999,"5min":0.189,"15min":0.18099999999999999},"max":{"1min":0.46500000000000002,"5min":0.47599999999999998,"15min":1.7190000000000001},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.32600000000000001,"15min":0.33000000000000002},"min":{"1min":0.22,"5min":0.21199999999999999,"15min":0.19800000000000001},"max":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.56999999999999995},"last":0.28799999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.373,"15min":0.36099999999999999},"min":{"1min":0.29099999999999998,"5min":0.26300000000000001,"15min":0.222},"max":{"1min":0.499,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.313},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.38,"15min":0.39200000000000002},"min":{"1min":0.24299999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.51400000000000001,"5min":0.60399999999999998,"15min":0.626},"last":0.35999999999999999}]},{"osd":3,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55300000000000005,"15min":0.54900000000000004},"min":{"1min":0.37,"5min":0.35699999999999998,"15min":0.32200000000000001},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.54500000000000004,"15min":0.53300000000000003},"min":{"1min":0.42999999999999999,"5min":0.32300000000000001,"15min":0.29499999999999998},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.92000000000000004},"last":0.5}]},{"osd":4,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.55000000000000004,"15min":0.55100000000000005},"min":{"1min":0.42599999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.72099999999999997,"5min":0.81799999999999995,"15min":0.85999999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.54700000000000004,"15min":0.54200000000000004},"min":{"1min":0.39700000000000002,"5min":0.36299999999999999,"15min":0.33900000000000002},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.52300000000000002}]},{"osd":5,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55000000000000004,"15min":0.54600000000000004},"min":{"1min":0.41399999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.71999999999999997,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54900000000000004,"15min":0.55500000000000005},"min":{"1min":0.38500000000000001,"5min":0.36799999999999999,"15min":0.35899999999999999},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.48699999999999999}]}]},{"osd":5,"up_from":11,"seq":47244640379,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44440,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":43390,"kb_avail":94327400,"statfs":{"total":96636764160,"available":96591257600,"internally_reserved":0,"allocated":1056768,"data_stored":363130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":44431814},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:22:04 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.49099999999999999,"15min":0.49199999999999999},"min":{"1min":0.314,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.60699999999999998,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.496,"5min":0.49399999999999999,"15min":0.51700000000000002},"min":{"1min":0.32700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":1.484},"last":0.46200000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:22:04 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.49399999999999999,"15min":0.504},"min":{"1min":0.36099999999999999,"5min":0.247,"15min":0.247},"max":{"1min":0.64200000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.498},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52600000000000002,"15min":0.52700000000000002},"min":{"1min":0.36899999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.65500000000000003,"5min":0.75700000000000001,"15min":0.81499999999999995},"last":0.439}]},{"osd":2,"last update":"Mon Apr 22 08:22:04 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.5,"15min":0.51300000000000001},"min":{"1min":0.40500000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.67500000000000004,"5min":0.68600000000000005,"15min":0.76800000000000002},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.499,"5min":0.504,"15min":0.58099999999999996},"min":{"1min":0.38400000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.57599999999999996,"5min":0.74099999999999999,"15min":3.6970000000000001},"last":0.52200000000000002}]},{"osd":3,"last update":"Mon Apr 22 08:22:04 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.39900000000000002,"15min":0.39400000000000002},"min":{"1min":0.28699999999999998,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.52000000000000002,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41599999999999998,"15min":0.40899999999999997},"min":{"1min":0.29299999999999998,"5min":0.25900000000000001,"15min":0.255},"max":{"1min":0.60099999999999998,"5min":0.62,"15min":0.621},"last":0.38100000000000001}]},{"osd":4,"last update":"Mon Apr 22 08:22:04 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.38100000000000001,"15min":0.41699999999999998},"min":{"1min":0.252,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.47199999999999998,"5min":1.0820000000000001,"15min":1.528},"last":0.307},{"interface":"front","average":{"1min":0.379,"5min":0.40400000000000003,"15min":0.42999999999999999},"min":{"1min":0.27800000000000002,"5min":0.24099999999999999,"15min":0.23599999999999999},"max":{"1min":0.61299999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.33100000000000002}]}]},{"osd":4,"up_from":11,"seq":47244640378,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28060,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343780,"statfs":{"total":96636764160,"available":96608030720,"internally_reserved":0,"allocated":1056768,"data_stored":363130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.48099999999999998,"15min":0.47899999999999998},"min":{"1min":0.317,"5min":0.215,"15min":0.215},"max":{"1min":0.66000000000000003,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.47999999999999998,"15min":0.48399999999999999},"min":{"1min":0.29399999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.66000000000000003,"5min":0.67300000000000004,"15min":0.69299999999999995},"last":0.60099999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.501,"15min":0.49399999999999999},"min":{"1min":0.36899999999999999,"5min":0.27600000000000002,"15min":0.26900000000000002},"max":{"1min":0.71499999999999997,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.495},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.497,"15min":0.5},"min":{"1min":0.28399999999999997,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.68899999999999995,"5min":0.71299999999999997,"15min":0.78800000000000003},"last":0.56399999999999995}]},{"osd":2,"last update":"Mon Apr 22 08:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.52900000000000003,"15min":0.55700000000000005},"min":{"1min":0.35699999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.54100000000000004,"15min":0.53500000000000003},"min":{"1min":0.35199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.63600000000000001}]},{"osd":3,"last update":"Mon Apr 22 08:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.373,"15min":0.39000000000000001},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.57799999999999996,"5min":1.204,"15min":1.204},"last":0.436},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.39100000000000001,"15min":0.40000000000000002},"min":{"1min":0.24199999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.54700000000000004,"5min":0.62,"15min":0.89100000000000001},"last":0.47199999999999998}]},{"osd":5,"last update":"Mon Apr 22 08:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.38600000000000001,"15min":0.38200000000000001},"min":{"1min":0.26200000000000001,"5min":0.251,"15min":0.23100000000000001},"max":{"1min":0.57499999999999996,"5min":0.625,"15min":0.65800000000000003},"last":0.505},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45300000000000001,"15min":0.432},"min":{"1min":0.30299999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.70499999999999996,"5min":2.2360000000000002,"15min":2.2360000000000002},"last":0.52500000000000002}]}]},{"osd":3,"up_from":11,"seq":47244640377,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28568,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343272,"statfs":{"total":96636764160,"available":96607510528,"internally_reserved":0,"allocated":1589248,"data_stored":892058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:22:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.53000000000000003,"15min":0.50800000000000001},"min":{"1min":0.34200000000000003,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.64400000000000002,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.52600000000000002,"15min":0.51000000000000001},"min":{"1min":0.38,"5min":0.32000000000000001,"15min":0.309},"max":{"1min":0.63400000000000001,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.52200000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:22:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.505,"15min":0.51700000000000002},"min":{"1min":0.39100000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.505,"5min":0.502,"15min":0.502},"min":{"1min":0.34999999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.84599999999999997},"last":0.57499999999999996}]},{"osd":2,"last update":"Mon Apr 22 08:22:03 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.496,"15min":0.52700000000000002},"min":{"1min":0.32200000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.86899999999999999},"last":0.501},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.50700000000000001,"15min":0.53000000000000003},"min":{"1min":0.36299999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.63400000000000001,"5min":0.69099999999999995,"15min":1.0289999999999999},"last":0.55000000000000004}]},{"osd":4,"last update":"Mon Apr 22 08:22:03 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.35799999999999998,"15min":0.38500000000000001},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.53400000000000003,"5min":0.70499999999999996,"15min":0.877},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.379,"15min":0.40500000000000003},"min":{"1min":0.27500000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":2.7000000000000002},"last":0.34200000000000003}]},{"osd":5,"last update":"Mon Apr 22 08:22:03 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.39100000000000001,"15min":0.40600000000000003},"min":{"1min":0.23899999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.98399999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39900000000000002,"15min":0.436},"min":{"1min":0.26200000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.61699999999999999,"5min":0.63300000000000001,"15min":0.81399999999999995},"last":0.375}]}]},{"osd":1,"up_from":11,"seq":47244640377,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28056,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343784,"statfs":{"total":96636764160,"available":96608034816,"internally_reserved":0,"allocated":1056768,"data_stored":363130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.35599999999999998,"15min":0.33900000000000002},"min":{"1min":0.24399999999999999,"5min":0.182,"15min":0.182},"max":{"1min":0.47099999999999997,"5min":1.629,"15min":2.1850000000000001},"last":0.247},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.36899999999999999,"15min":0.36299999999999999},"min":{"1min":0.23899999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.52300000000000002,"5min":1.8220000000000001,"15min":3.1320000000000001},"last":0.26500000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39800000000000002,"15min":0.38100000000000001},"min":{"1min":0.252,"5min":0.22900000000000001,"15min":0.22},"max":{"1min":0.48899999999999999,"5min":2.0219999999999998,"15min":2.0219999999999998},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.436,"15min":0.42099999999999999},"min":{"1min":0.309,"5min":0.27400000000000002,"15min":0.255},"max":{"1min":0.66300000000000003,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":0.307}]},{"osd":3,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.55600000000000005,"15min":0.58199999999999996},"min":{"1min":0.45300000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.77800000000000002,"5min":0.80500000000000005,"15min":3.504},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56299999999999994,"15min":0.56699999999999995},"min":{"1min":0.41099999999999998,"5min":0.36599999999999999,"15min":0.35799999999999998},"max":{"1min":0.68500000000000005,"5min":1.774,"15min":1.774},"last":0.36299999999999999}]},{"osd":4,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.59399999999999997,"15min":0.59699999999999998},"min":{"1min":0.40799999999999997,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.72499999999999998,"5min":3.5579999999999998,"15min":3.5579999999999998},"last":0.44},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.56899999999999995,"15min":0.58299999999999996},"min":{"1min":0.46700000000000003,"5min":0.34599999999999997,"15min":0.316},"max":{"1min":0.73199999999999998,"5min":0.85299999999999998,"15min":3.085},"last":0.44600000000000001}]},{"osd":5,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.57399999999999995,"15min":0.57899999999999996},"min":{"1min":0.439,"5min":0.372,"15min":0.372},"max":{"1min":0.76000000000000001,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.56799999999999995,"15min":0.59499999999999997},"min":{"1min":0.42399999999999999,"5min":0.38900000000000001,"15min":0.35999999999999999},"max":{"1min":0.747,"5min":0.80200000000000005,"15min":2.6520000000000001},"last":0.42599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:22:42.567 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:22:42.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.666+0000 7fbd5b5c5640 10 monclient: get_monmap_and_config 2024-04-22T08:22:42.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.666+0000 7fbd5b5c5640 10 monclient: build_initial_monmap 2024-04-22T08:22:42.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.666+0000 7fbd5b5c5640 10 monclient: monmap: 2024-04-22T08:22:42.666 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:42.666 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:42.666 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:22:42.667419+0000 2024-04-22T08:22:42.666 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:22:42.667419+0000 2024-04-22T08:22:42.666 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:42.666 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:42.667 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:22:42.667 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:22:42.667 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:22:42.667 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:22:42.667 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:22:42.667 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:42.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.667+0000 7fbd5b5c5640 1 Processor -- start 2024-04-22T08:22:42.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.667+0000 7fbd5b5c5640 1 -- start start 2024-04-22T08:22:42.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.667+0000 7fbd5b5c5640 10 monclient: init 2024-04-22T08:22:42.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.667+0000 7fbd5b5c5640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:42.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.667+0000 7fbd5b5c5640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-04-22T08:22:42.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.667+0000 7fbd5b5c5640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbd5409de00 0x7fbd5409e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:42.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.667+0000 7fbd5b5c5640 10 monclient(hunting): picked mon.noname-b con 0x7fbd5409de00 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:22:42.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.667+0000 7fbd5b5c5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbd540a4330 addr v1:172.21.15.189:6789/0 2024-04-22T08:22:42.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.667+0000 7fbd5b5c5640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbd540a0240 0x7fbd5409e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:42.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.667+0000 7fbd5b5c5640 10 monclient(hunting): picked mon.noname-a con 0x7fbd540a0240 addr v2:172.21.15.40:3300/0 2024-04-22T08:22:42.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.667+0000 7fbd5b5c5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd5409ec60 con 0x7fbd540a4330 2024-04-22T08:22:42.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.667+0000 7fbd5b5c5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:42.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.667+0000 7fbd5b5c5640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd5409fb40 con 0x7fbd540a0240 2024-04-22T08:22:42.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.667+0000 7fbd5b5c5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:42.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.667+0000 7fbd5b5c5640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd540a3410 con 0x7fbd5409de00 2024-04-22T08:22:42.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.667+0000 7fbd5b5c5640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:42.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.667+0000 7fbd5b5c5640 10 monclient(hunting): authenticate will time out at 1436.661723s 2024-04-22T08:22:42.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.668+0000 7fbd59361640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbd5409de00 0x7fbd5409e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:42.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.668+0000 7fbd59b62640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbd540a0240 0x7fbd5409e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:42.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.668+0000 7fbd59361640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbd5409de00 0x7fbd5409e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49776/0 (socket says 172.21.15.40:49776) 2024-04-22T08:22:42.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.668+0000 7fbd59361640 1 -- 172.21.15.40:0/2356963363 learned_addr learned my addr 172.21.15.40:0/2356963363 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:22:42.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.668+0000 7fbd59361640 10 monclient(hunting): get_auth_request con 0x7fbd5409de00 auth_method 0 2024-04-22T08:22:42.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.668+0000 7fbd59361640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:42.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.668+0000 7fbd59361640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:42.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.668+0000 7fbd59361640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:42.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.668+0000 7fbd59b62640 10 monclient(hunting): get_auth_request con 0x7fbd540a0240 auth_method 0 2024-04-22T08:22:42.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.668+0000 7fbd59b62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:42.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.668+0000 7fbd59b62640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:42.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.668+0000 7fbd59b62640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:42.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.668+0000 7fbd59361640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:42.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.668+0000 7fbd59361640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:42.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.668+0000 7fbd59361640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:42.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.668+0000 7fbd59b62640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:42.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.668+0000 7fbd59b62640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:42.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.668+0000 7fbd59b62640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:42.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.668+0000 7fbd59361640 10 monclient(hunting): handle_auth_done global_id 124124 payload 274 2024-04-22T08:22:42.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.668+0000 7fbd59361640 1 -- 172.21.15.40:0/2356963363 >> v2:172.21.15.40:3300/0 conn(0x7fbd540a0240 msgr2=0x7fbd5409e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:42.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.668+0000 7fbd59361640 1 --2- 172.21.15.40:0/2356963363 >> v2:172.21.15.40:3300/0 conn(0x7fbd540a0240 0x7fbd5409e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:42.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.668+0000 7fbd59361640 1 -- 172.21.15.40:0/2356963363 >> v1:172.21.15.189:6789/0 conn(0x7fbd540a4330 legacy=0x7fbd54135bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:42.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.668+0000 7fbd59361640 10 monclient: _finish_hunting 0 2024-04-22T08:22:42.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.668+0000 7fbd59361640 1 monclient: found mon.noname-b 2024-04-22T08:22:42.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.668+0000 7fbd59361640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:42.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.668+0000 7fbd59361640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:22:42.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.668+0000 7fbd59361640 1 -- 172.21.15.40:0/2356963363 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd540a4b00 con 0x7fbd5409de00 2024-04-22T08:22:42.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.668+0000 7fbd59b62640 1 --2- 172.21.15.40:0/2356963363 >> v2:172.21.15.40:3300/0 conn(0x7fbd540a0240 0x7fbd5409e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:22:42.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.671+0000 7fbd59361640 1 -- 172.21.15.40:0/2356963363 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbd5409de00 msgr2=0x7fbd5409e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:42.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.671+0000 7fbd59361640 1 --2- 172.21.15.40:0/2356963363 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbd5409de00 0x7fbd5409e1e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fbd50009870 tx=0x7fbd5002b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5de4ed22c986dc07 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:42.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.671+0000 7fbd4bfff640 1 -- 172.21.15.40:0/2356963363 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd50031950 con 0x7fbd5409de00 2024-04-22T08:22:42.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.671+0000 7fbd4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:42.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.671+0000 7fbd4bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:22:42.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.671+0000 7fbd4bfff640 10 monclient: dump: 2024-04-22T08:22:42.678 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:42.678 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:42.678 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:42.678 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:42.678 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:42.678 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:42.678 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:42.678 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:42.678 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:42.678 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:42.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.671+0000 7fbd4bfff640 10 monclient: _finish_auth 0 2024-04-22T08:22:42.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.671+0000 7fbd4bfff640 10 monclient: _check_auth_tickets 2024-04-22T08:22:42.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.671+0000 7fbd4bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:42.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.671+0000 7fbd4bfff640 1 -- 172.21.15.40:0/2356963363 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd50031ab0 con 0x7fbd5409de00 2024-04-22T08:22:42.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.671+0000 7fbd5b5c5640 5 monclient: authenticate success, global_id 124124 2024-04-22T08:22:42.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.671+0000 7fbd5b5c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:22:42.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.671+0000 7fbd4bfff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:42.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.671+0000 7fbd5b5c5640 10 monclient: get_monmap_and_config success 2024-04-22T08:22:42.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.671+0000 7fbd4bfff640 1 -- 172.21.15.40:0/2356963363 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd50031e40 con 0x7fbd5409de00 2024-04-22T08:22:42.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.671+0000 7fbd5b5c5640 4 set_mon_vals no callback set 2024-04-22T08:22:42.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.671+0000 7fbd4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:42.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.671+0000 7fbd4bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:22:42.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.671+0000 7fbd4bfff640 10 monclient: dump: 2024-04-22T08:22:42.679 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:42.679 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:42.679 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:42.680 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:42.680 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:42.680 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:42.680 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:42.680 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:42.680 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:42.680 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:42.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.671+0000 7fbd5b5c5640 10 monclient: shutdown 2024-04-22T08:22:42.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.671+0000 7fbd5b5c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:42.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.671+0000 7fbd5b5c5640 1 -- 172.21.15.40:0/2356963363 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbd5409de00 msgr2=0x7fbd5409e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:42.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.671+0000 7fbd5b5c5640 1 --2- 172.21.15.40:0/2356963363 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbd5409de00 0x7fbd5409e1e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fbd50009870 tx=0x7fbd5002b7e0 comp rx=0 tx=0).stop 2024-04-22T08:22:42.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.671+0000 7fbd5b5c5640 1 -- 172.21.15.40:0/2356963363 shutdown_connections 2024-04-22T08:22:42.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.671+0000 7fbd5b5c5640 1 --2- 172.21.15.40:0/2356963363 >> v2:172.21.15.40:3300/0 conn(0x7fbd540a0240 0x7fbd5409e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:42.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.671+0000 7fbd5b5c5640 1 --2- 172.21.15.40:0/2356963363 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbd5409de00 0x7fbd5409e1e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:42.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.671+0000 7fbd5b5c5640 1 -- 172.21.15.40:0/2356963363 >> 172.21.15.40:0/2356963363 conn(0x7fbd540a83f0 msgr2=0x7fbd540a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:42.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.671+0000 7fbd5b5c5640 1 -- 172.21.15.40:0/2356963363 shutdown_connections 2024-04-22T08:22:42.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.671+0000 7fbd5b5c5640 1 -- 172.21.15.40:0/2356963363 wait complete. 2024-04-22T08:22:42.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.672+0000 7fbd5b5c5640 10 monclient: build_initial_monmap 2024-04-22T08:22:42.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.672+0000 7fbd5b5c5640 10 monclient: monmap: 2024-04-22T08:22:42.681 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:42.681 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:22:42.681 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:22:42.681 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:22:42.681 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:42.681 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:42.681 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:22:42.681 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:22:42.681 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:22:42.682 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:42.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.672+0000 7fbd5b5c5640 1 Processor -- start 2024-04-22T08:22:42.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.672+0000 7fbd5b5c5640 1 -- start start 2024-04-22T08:22:42.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.672+0000 7fbd5b5c5640 10 monclient: init 2024-04-22T08:22:42.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd5b5c5640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:42.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd5b5c5640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-22T08:22:42.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd5b5c5640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbd5409de00 0x7fbd541cad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:42.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd5b5c5640 10 monclient(hunting): picked mon.noname-c con 0x7fbd5409de00 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:22:42.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd5b5c5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd540a0240 0x7fbd541cb250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:42.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd5b5c5640 10 monclient(hunting): picked mon.noname-a con 0x7fbd540a0240 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:22:42.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd5b5c5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd54125550 0x7fbd54121180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:42.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd5b5c5640 10 monclient(hunting): picked mon.noname-b con 0x7fbd54125550 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:22:42.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd5b5c5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:42.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd5b5c5640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd54124890 con 0x7fbd540a0240 2024-04-22T08:22:42.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd5b5c5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:42.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd5b5c5640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd540a6ef0 con 0x7fbd54125550 2024-04-22T08:22:42.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd5b5c5640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:42.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd5b5c5640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd541239a0 con 0x7fbd5409de00 2024-04-22T08:22:42.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd5b5c5640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:42.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd5b5c5640 10 monclient(hunting): authenticate will time out at 1436.667096s 2024-04-22T08:22:42.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd59361640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbd5409de00 0x7fbd541cad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:42.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd59361640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbd5409de00 0x7fbd541cad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49780/0 (socket says 172.21.15.40:49780) 2024-04-22T08:22:42.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd59361640 1 -- 172.21.15.40:0/2221975764 learned_addr learned my addr 172.21.15.40:0/2221975764 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:22:42.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd59361640 10 monclient(hunting): get_auth_request con 0x7fbd5409de00 auth_method 0 2024-04-22T08:22:42.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd59361640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:42.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd59361640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:42.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd59361640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:42.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd58b60640 1 --2- 172.21.15.40:0/2221975764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd540a0240 0x7fbd541cb250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:42.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd58b60640 10 monclient(hunting): get_auth_request con 0x7fbd540a0240 auth_method 0 2024-04-22T08:22:42.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd58b60640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:42.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd58b60640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:42.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd58b60640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:42.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd59361640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:42.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd59361640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:42.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd59361640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:42.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd59b62640 1 --2- 172.21.15.40:0/2221975764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd54125550 0x7fbd54121180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:42.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd58b60640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:42.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd58b60640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:42.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd58b60640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:42.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd59b62640 10 monclient(hunting): get_auth_request con 0x7fbd54125550 auth_method 0 2024-04-22T08:22:42.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd59b62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:42.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd59b62640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:42.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd59b62640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:42.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd59b62640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:42.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd59b62640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:42.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd59b62640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:42.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd59361640 10 monclient(hunting): handle_auth_done global_id 124127 payload 824 2024-04-22T08:22:42.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd59361640 1 -- 172.21.15.40:0/2221975764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd54125550 msgr2=0x7fbd54121180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:42.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd59361640 1 --2- 172.21.15.40:0/2221975764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd54125550 0x7fbd54121180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:42.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd59361640 1 -- 172.21.15.40:0/2221975764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd540a0240 msgr2=0x7fbd541cb250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:42.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd59361640 1 --2- 172.21.15.40:0/2221975764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd540a0240 0x7fbd541cb250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:42.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd59361640 10 monclient: _finish_hunting 0 2024-04-22T08:22:42.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd59361640 1 monclient: found mon.noname-c 2024-04-22T08:22:42.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd59361640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:42.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd59361640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:22:42.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd59361640 1 -- 172.21.15.40:0/2221975764 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd4c009a70 con 0x7fbd5409de00 2024-04-22T08:22:42.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.673+0000 7fbd58b60640 1 --2- 172.21.15.40:0/2221975764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd540a0240 0x7fbd541cb250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:22:42.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.675+0000 7fbd59361640 1 -- 172.21.15.40:0/2221975764 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbd5409de00 msgr2=0x7fbd541cad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:42.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.675+0000 7fbd59361640 1 --2- 172.21.15.40:0/2221975764 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbd5409de00 0x7fbd541cad10 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fbd500090b0 tx=0x7fbd500046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:42.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.676+0000 7fbd49ffb640 1 -- 172.21.15.40:0/2221975764 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd50033b70 con 0x7fbd5409de00 2024-04-22T08:22:42.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.676+0000 7fbd49ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:42.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.676+0000 7fbd49ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:22:42.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.676+0000 7fbd49ffb640 10 monclient: dump: 2024-04-22T08:22:42.686 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:42.686 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:42.686 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:42.686 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:42.686 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:42.686 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:42.687 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:42.687 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:42.687 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:42.687 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:42.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.676+0000 7fbd49ffb640 10 monclient: _finish_auth 0 2024-04-22T08:22:42.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.676+0000 7fbd49ffb640 10 monclient: _check_auth_tickets 2024-04-22T08:22:42.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.676+0000 7fbd49ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:42.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.676+0000 7fbd49ffb640 1 -- 172.21.15.40:0/2221975764 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd50033cd0 con 0x7fbd5409de00 2024-04-22T08:22:42.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.676+0000 7fbd49ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:42.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.676+0000 7fbd49ffb640 1 -- 172.21.15.40:0/2221975764 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd5003c640 con 0x7fbd5409de00 2024-04-22T08:22:42.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.676+0000 7fbd5b5c5640 5 monclient: authenticate success, global_id 124127 2024-04-22T08:22:42.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.676+0000 7fbd49ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:42.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.676+0000 7fbd49ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:22:42.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.676+0000 7fbd49ffb640 10 monclient: dump: 2024-04-22T08:22:42.687 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:42.688 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:42.688 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:42.688 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:42.688 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:42.688 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:42.688 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:42.688 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:42.688 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:42.688 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:42.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.676+0000 7fbd5b5c5640 10 monclient: _renew_subs 2024-04-22T08:22:42.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.676+0000 7fbd5b5c5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:22:42.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.676+0000 7fbd4b7fe640 4 set_mon_vals no callback set 2024-04-22T08:22:42.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.676+0000 7fbd5b5c5640 1 -- 172.21.15.40:0/2221975764 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd500094d0 con 0x7fbd5409de00 2024-04-22T08:22:42.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.676+0000 7fbd5b5c5640 10 monclient: _renew_subs 2024-04-22T08:22:42.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.676+0000 7fbd5b5c5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:22:42.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.676+0000 7fbd5b5c5640 1 -- 172.21.15.40:0/2221975764 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd54125130 con 0x7fbd5409de00 2024-04-22T08:22:42.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.677+0000 7fbd49ffb640 1 -- 172.21.15.40:0/2221975764 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fbd5003a040 con 0x7fbd5409de00 2024-04-22T08:22:42.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.677+0000 7fbd5b5c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:42.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.677+0000 7fbd5b5c5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:42.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.677+0000 7fbd5b5c5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:22:42.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.677+0000 7fbd49ffb640 1 --2- 172.21.15.40:0/2221975764 >> v2:172.21.15.189:6800/30168 conn(0x7fbd38037880 0x7fbd38039d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:42.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.677+0000 7fbd5b5c5640 1 -- 172.21.15.40:0/2221975764 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd5404e9b0 con 0x7fbd5409de00 2024-04-22T08:22:42.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.677+0000 7fbd49ffb640 1 -- 172.21.15.40:0/2221975764 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(317..317 src has 239..317) v4 ==== 3506+0+0 (secure 0 0 0) 0x7fbd5006bf80 con 0x7fbd5409de00 2024-04-22T08:22:42.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.677+0000 7fbd58b60640 1 --2- 172.21.15.40:0/2221975764 >> v2:172.21.15.189:6800/30168 conn(0x7fbd38037880 0x7fbd38039d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:42.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.677+0000 7fbd58b60640 10 monclient: get_auth_request con 0x7fbd38037880 auth_method 0 2024-04-22T08:22:42.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.680+0000 7fbd58b60640 1 --2- 172.21.15.40:0/2221975764 >> v2:172.21.15.189:6800/30168 conn(0x7fbd38037880 0x7fbd38039d40 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fbd5404e810 tx=0x7fbd44007e30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:42.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.681+0000 7fbd49ffb640 1 -- 172.21.15.40:0/2221975764 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7fbd5002f070 con 0x7fbd5409de00 2024-04-22T08:22:42.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.682+0000 7fbd49ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:42.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.682+0000 7fbd49ffb640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:22:42.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.860+0000 7fbd5b5c5640 1 -- 172.21.15.40:0/2221975764 --> v2:172.21.15.189:6800/30168 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbd541362c0 con 0x7fbd38037880 2024-04-22T08:22:42.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.862+0000 7fbd49ffb640 1 -- 172.21.15.40:0/2221975764 <== mgr.4102 v2:172.21.15.189:6800/30168 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+53849 (secure 0 0 0) 0x7fbd4400fcb0 con 0x7fbd38037880 2024-04-22T08:22:42.862 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:22:42.863 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-22T08:22:42.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.864+0000 7fbd5b5c5640 1 -- 172.21.15.40:0/2221975764 >> v2:172.21.15.189:6800/30168 conn(0x7fbd38037880 msgr2=0x7fbd38039d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:42.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.864+0000 7fbd5b5c5640 1 --2- 172.21.15.40:0/2221975764 >> v2:172.21.15.189:6800/30168 conn(0x7fbd38037880 0x7fbd38039d40 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fbd5404e810 tx=0x7fbd44007e30 comp rx=0 tx=0).stop 2024-04-22T08:22:42.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.864+0000 7fbd5b5c5640 10 monclient: shutdown 2024-04-22T08:22:42.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.864+0000 7fbd5b5c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:42.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.864+0000 7fbd5b5c5640 1 -- 172.21.15.40:0/2221975764 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbd5409de00 msgr2=0x7fbd541cad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:42.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.864+0000 7fbd5b5c5640 1 --2- 172.21.15.40:0/2221975764 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbd5409de00 0x7fbd541cad10 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fbd500090b0 tx=0x7fbd500046c0 comp rx=0 tx=0).stop 2024-04-22T08:22:42.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.864+0000 7fbd5b5c5640 1 -- 172.21.15.40:0/2221975764 shutdown_connections 2024-04-22T08:22:42.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.864+0000 7fbd5b5c5640 1 --2- 172.21.15.40:0/2221975764 >> v2:172.21.15.189:6800/30168 conn(0x7fbd38037880 0x7fbd38039d40 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:42.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.864+0000 7fbd5b5c5640 1 --2- 172.21.15.40:0/2221975764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd54125550 0x7fbd54121180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:42.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.864+0000 7fbd5b5c5640 1 --2- 172.21.15.40:0/2221975764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd540a0240 0x7fbd541cb250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:42.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.864+0000 7fbd5b5c5640 1 --2- 172.21.15.40:0/2221975764 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbd5409de00 0x7fbd541cad10 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:42.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.864+0000 7fbd5b5c5640 1 -- 172.21.15.40:0/2221975764 >> 172.21.15.40:0/2221975764 conn(0x7fbd540a83f0 msgr2=0x7fbd54135520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:42.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.864+0000 7fbd5b5c5640 1 -- 172.21.15.40:0/2221975764 shutdown_connections 2024-04-22T08:22:42.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.864+0000 7fbd5b5c5640 1 -- 172.21.15.40:0/2221975764 wait complete. 2024-04-22T08:22:42.874 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":612,"stamp":"2024-04-22T08:22:42.206057+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":185760,"kb_used_data":7232,"kb_used_omap":9,"kb_used_meta":178422,"kb_avail":566045280,"statfs":{"total":579820584960,"available":579630366720,"internally_reserved":0,"allocated":7405568,"data_stored":3236636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9560,"internal_metadata":182704808},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002606"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":304,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-04-22T08:22:17.159438+0000","last_change":"2024-04-22T08:12:51.443086+0000","last_active":"2024-04-22T08:22:17.159438+0000","last_peered":"2024-04-22T08:22:17.159438+0000","last_clean":"2024-04-22T08:22:17.159438+0000","last_became_active":"2024-04-22T08:12:51.442730+0000","last_became_peered":"2024-04-22T08:12:51.442730+0000","last_unstale":"2024-04-22T08:22:17.159438+0000","last_undegraded":"2024-04-22T08:22:17.159438+0000","last_fullsized":"2024-04-22T08:22:17.159438+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:06:13.481863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":293,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-04-22T08:22:16.957788+0000","last_change":"2024-04-22T08:12:50.841276+0000","last_active":"2024-04-22T08:22:16.957788+0000","last_peered":"2024-04-22T08:22:16.957788+0000","last_clean":"2024-04-22T08:22:16.957788+0000","last_became_active":"2024-04-22T08:12:50.841008+0000","last_became_peered":"2024-04-22T08:12:50.841008+0000","last_unstale":"2024-04-22T08:22:16.957788+0000","last_undegraded":"2024-04-22T08:22:16.957788+0000","last_fullsized":"2024-04-22T08:22:16.957788+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:47:00.224321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":304,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-04-22T08:22:17.160920+0000","last_change":"2024-04-22T08:12:51.403419+0000","last_active":"2024-04-22T08:22:17.160920+0000","last_peered":"2024-04-22T08:22:17.160920+0000","last_clean":"2024-04-22T08:22:17.160920+0000","last_became_active":"2024-04-22T08:12:51.403182+0000","last_became_peered":"2024-04-22T08:12:51.403182+0000","last_unstale":"2024-04-22T08:22:17.160920+0000","last_undegraded":"2024-04-22T08:22:17.160920+0000","last_fullsized":"2024-04-22T08:22:17.160920+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:52:39.730597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":293,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-04-22T08:22:16.957078+0000","last_change":"2024-04-22T08:12:51.403260+0000","last_active":"2024-04-22T08:22:16.957078+0000","last_peered":"2024-04-22T08:22:16.957078+0000","last_clean":"2024-04-22T08:22:16.957078+0000","last_became_active":"2024-04-22T08:12:51.402980+0000","last_became_peered":"2024-04-22T08:12:51.402980+0000","last_unstale":"2024-04-22T08:22:16.957078+0000","last_undegraded":"2024-04-22T08:22:16.957078+0000","last_fullsized":"2024-04-22T08:22:16.957078+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:08:04.625870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":295,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-04-22T08:22:17.162182+0000","last_change":"2024-04-22T08:12:50.836566+0000","last_active":"2024-04-22T08:22:17.162182+0000","last_peered":"2024-04-22T08:22:17.162182+0000","last_clean":"2024-04-22T08:22:17.162182+0000","last_became_active":"2024-04-22T08:12:50.836250+0000","last_became_peered":"2024-04-22T08:12:50.836250+0000","last_unstale":"2024-04-22T08:22:17.162182+0000","last_undegraded":"2024-04-22T08:22:17.162182+0000","last_fullsized":"2024-04-22T08:22:17.162182+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:42:13.693209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":302,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-04-22T08:22:16.950081+0000","last_change":"2024-04-22T08:12:50.836508+0000","last_active":"2024-04-22T08:22:16.950081+0000","last_peered":"2024-04-22T08:22:16.950081+0000","last_clean":"2024-04-22T08:22:16.950081+0000","last_became_active":"2024-04-22T08:12:50.836210+0000","last_became_peered":"2024-04-22T08:12:50.836210+0000","last_unstale":"2024-04-22T08:22:16.950081+0000","last_undegraded":"2024-04-22T08:22:16.950081+0000","last_fullsized":"2024-04-22T08:22:16.950081+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:24:53.345661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":304,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-04-22T08:22:17.158781+0000","last_change":"2024-04-22T08:12:50.842633+0000","last_active":"2024-04-22T08:22:17.158781+0000","last_peered":"2024-04-22T08:22:17.158781+0000","last_clean":"2024-04-22T08:22:17.158781+0000","last_became_active":"2024-04-22T08:12:50.842392+0000","last_became_peered":"2024-04-22T08:12:50.842392+0000","last_unstale":"2024-04-22T08:22:17.158781+0000","last_undegraded":"2024-04-22T08:22:17.158781+0000","last_fullsized":"2024-04-22T08:22:17.158781+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:53:14.717539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":295,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-04-22T08:22:17.162502+0000","last_change":"2024-04-22T08:12:50.835820+0000","last_active":"2024-04-22T08:22:17.162502+0000","last_peered":"2024-04-22T08:22:17.162502+0000","last_clean":"2024-04-22T08:22:17.162502+0000","last_became_active":"2024-04-22T08:12:50.835541+0000","last_became_peered":"2024-04-22T08:12:50.835541+0000","last_unstale":"2024-04-22T08:22:17.162502+0000","last_undegraded":"2024-04-22T08:22:17.162502+0000","last_fullsized":"2024-04-22T08:22:17.162502+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:52:21.798509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":454,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-04-22T08:22:17.158877+0000","last_change":"2024-04-22T08:12:51.404305+0000","last_active":"2024-04-22T08:22:17.158877+0000","last_peered":"2024-04-22T08:22:17.158877+0000","last_clean":"2024-04-22T08:22:17.158877+0000","last_became_active":"2024-04-22T08:12:51.403208+0000","last_became_peered":"2024-04-22T08:12:51.403208+0000","last_unstale":"2024-04-22T08:22:17.158877+0000","last_undegraded":"2024-04-22T08:22:17.158877+0000","last_fullsized":"2024-04-22T08:22:17.158877+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:04:43.253802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640379,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28056,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343784,"statfs":{"total":96636764160,"available":96608034816,"internally_reserved":0,"allocated":1056768,"data_stored":363130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.35199999999999998,"15min":0.371},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":0.39800000000000002,"5min":2.5920000000000001,"15min":2.5920000000000001},"last":0.255},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.40899999999999997,"15min":0.38700000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.21299999999999999},"max":{"1min":0.51400000000000001,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.30599999999999999}]},{"osd":1,"last update":"Mon Apr 22 08:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.34300000000000003,"15min":0.34399999999999997},"min":{"1min":0.249,"5min":0.23400000000000001,"15min":0.17100000000000001},"max":{"1min":0.443,"5min":0.499,"15min":0.55700000000000005},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.38100000000000001,"15min":0.34499999999999997},"min":{"1min":0.254,"5min":0.24299999999999999,"15min":0.20699999999999999},"max":{"1min":0.48499999999999999,"5min":0.55600000000000005,"15min":0.70099999999999996},"last":0.33300000000000002}]},{"osd":3,"last update":"Mon Apr 22 08:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.59699999999999998,"15min":0.56499999999999995},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.22500000000000001},"max":{"1min":0.82999999999999996,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.58799999999999997,"15min":0.60199999999999998},"min":{"1min":0.33400000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.749,"5min":0.84799999999999998,"15min":1.8440000000000001},"last":0.45600000000000002}]},{"osd":4,"last update":"Mon Apr 22 08:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58399999999999996,"15min":0.68600000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.30499999999999999},"max":{"1min":0.78800000000000003,"5min":2.7879999999999998,"15min":6.2949999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.55100000000000005,"15min":0.68400000000000005},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.69499999999999995,"5min":0.91200000000000003,"15min":6.2149999999999999},"last":0.50900000000000001}]},{"osd":5,"last update":"Mon Apr 22 08:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.56899999999999995,"15min":0.53400000000000003},"min":{"1min":0.40000000000000002,"5min":0.35799999999999998,"15min":0.27800000000000002},"max":{"1min":0.72499999999999998,"5min":0.84299999999999997,"15min":0.85199999999999998},"last":0.441},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56399999999999995,"15min":0.53500000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.29699999999999999},"max":{"1min":0.72999999999999998,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.46999999999999997}]}]},{"osd":0,"up_from":11,"seq":47244640378,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28580,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343260,"statfs":{"total":96636764160,"available":96607498240,"internally_reserved":0,"allocated":1589248,"data_stored":892058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.32000000000000001,"15min":0.34599999999999997},"min":{"1min":0.23699999999999999,"5min":0.189,"15min":0.18099999999999999},"max":{"1min":0.46500000000000002,"5min":0.47599999999999998,"15min":1.7190000000000001},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.32600000000000001,"15min":0.33000000000000002},"min":{"1min":0.22,"5min":0.21199999999999999,"15min":0.19800000000000001},"max":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.56999999999999995},"last":0.28799999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.373,"15min":0.36099999999999999},"min":{"1min":0.29099999999999998,"5min":0.26300000000000001,"15min":0.222},"max":{"1min":0.499,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.313},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.38,"15min":0.39200000000000002},"min":{"1min":0.24299999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.51400000000000001,"5min":0.60399999999999998,"15min":0.626},"last":0.35999999999999999}]},{"osd":3,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55300000000000005,"15min":0.54900000000000004},"min":{"1min":0.37,"5min":0.35699999999999998,"15min":0.32200000000000001},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.54500000000000004,"15min":0.53300000000000003},"min":{"1min":0.42999999999999999,"5min":0.32300000000000001,"15min":0.29499999999999998},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.92000000000000004},"last":0.5}]},{"osd":4,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.55000000000000004,"15min":0.55100000000000005},"min":{"1min":0.42599999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.72099999999999997,"5min":0.81799999999999995,"15min":0.85999999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.54700000000000004,"15min":0.54200000000000004},"min":{"1min":0.39700000000000002,"5min":0.36299999999999999,"15min":0.33900000000000002},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.52300000000000002}]},{"osd":5,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55000000000000004,"15min":0.54600000000000004},"min":{"1min":0.41399999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.71999999999999997,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54900000000000004,"15min":0.55500000000000005},"min":{"1min":0.38500000000000001,"5min":0.36799999999999999,"15min":0.35899999999999999},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.48699999999999999}]}]},{"osd":5,"up_from":11,"seq":47244640379,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44440,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":43390,"kb_avail":94327400,"statfs":{"total":96636764160,"available":96591257600,"internally_reserved":0,"allocated":1056768,"data_stored":363130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":44431814},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:22:04 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.49099999999999999,"15min":0.49199999999999999},"min":{"1min":0.314,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.60699999999999998,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.496,"5min":0.49399999999999999,"15min":0.51700000000000002},"min":{"1min":0.32700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":1.484},"last":0.46200000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:22:04 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.49399999999999999,"15min":0.504},"min":{"1min":0.36099999999999999,"5min":0.247,"15min":0.247},"max":{"1min":0.64200000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.498},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52600000000000002,"15min":0.52700000000000002},"min":{"1min":0.36899999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.65500000000000003,"5min":0.75700000000000001,"15min":0.81499999999999995},"last":0.439}]},{"osd":2,"last update":"Mon Apr 22 08:22:04 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.5,"15min":0.51300000000000001},"min":{"1min":0.40500000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.67500000000000004,"5min":0.68600000000000005,"15min":0.76800000000000002},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.499,"5min":0.504,"15min":0.58099999999999996},"min":{"1min":0.38400000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.57599999999999996,"5min":0.74099999999999999,"15min":3.6970000000000001},"last":0.52200000000000002}]},{"osd":3,"last update":"Mon Apr 22 08:22:04 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.39900000000000002,"15min":0.39400000000000002},"min":{"1min":0.28699999999999998,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.52000000000000002,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41599999999999998,"15min":0.40899999999999997},"min":{"1min":0.29299999999999998,"5min":0.25900000000000001,"15min":0.255},"max":{"1min":0.60099999999999998,"5min":0.62,"15min":0.621},"last":0.38100000000000001}]},{"osd":4,"last update":"Mon Apr 22 08:22:04 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.38100000000000001,"15min":0.41699999999999998},"min":{"1min":0.252,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.47199999999999998,"5min":1.0820000000000001,"15min":1.528},"last":0.307},{"interface":"front","average":{"1min":0.379,"5min":0.40400000000000003,"15min":0.42999999999999999},"min":{"1min":0.27800000000000002,"5min":0.24099999999999999,"15min":0.23599999999999999},"max":{"1min":0.61299999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.33100000000000002}]}]},{"osd":4,"up_from":11,"seq":47244640378,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28060,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343780,"statfs":{"total":96636764160,"available":96608030720,"internally_reserved":0,"allocated":1056768,"data_stored":363130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.48099999999999998,"15min":0.47899999999999998},"min":{"1min":0.317,"5min":0.215,"15min":0.215},"max":{"1min":0.66000000000000003,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.47999999999999998,"15min":0.48399999999999999},"min":{"1min":0.29399999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.66000000000000003,"5min":0.67300000000000004,"15min":0.69299999999999995},"last":0.60099999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.501,"15min":0.49399999999999999},"min":{"1min":0.36899999999999999,"5min":0.27600000000000002,"15min":0.26900000000000002},"max":{"1min":0.71499999999999997,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.495},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.497,"15min":0.5},"min":{"1min":0.28399999999999997,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.68899999999999995,"5min":0.71299999999999997,"15min":0.78800000000000003},"last":0.56399999999999995}]},{"osd":2,"last update":"Mon Apr 22 08:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.52900000000000003,"15min":0.55700000000000005},"min":{"1min":0.35699999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.54100000000000004,"15min":0.53500000000000003},"min":{"1min":0.35199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.63600000000000001}]},{"osd":3,"last update":"Mon Apr 22 08:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.373,"15min":0.39000000000000001},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.57799999999999996,"5min":1.204,"15min":1.204},"last":0.436},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.39100000000000001,"15min":0.40000000000000002},"min":{"1min":0.24199999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.54700000000000004,"5min":0.62,"15min":0.89100000000000001},"last":0.47199999999999998}]},{"osd":5,"last update":"Mon Apr 22 08:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.38600000000000001,"15min":0.38200000000000001},"min":{"1min":0.26200000000000001,"5min":0.251,"15min":0.23100000000000001},"max":{"1min":0.57499999999999996,"5min":0.625,"15min":0.65800000000000003},"last":0.505},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45300000000000001,"15min":0.432},"min":{"1min":0.30299999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.70499999999999996,"5min":2.2360000000000002,"15min":2.2360000000000002},"last":0.52500000000000002}]}]},{"osd":3,"up_from":11,"seq":47244640377,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28568,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343272,"statfs":{"total":96636764160,"available":96607510528,"internally_reserved":0,"allocated":1589248,"data_stored":892058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:22:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.53000000000000003,"15min":0.50800000000000001},"min":{"1min":0.34200000000000003,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.64400000000000002,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.52600000000000002,"15min":0.51000000000000001},"min":{"1min":0.38,"5min":0.32000000000000001,"15min":0.309},"max":{"1min":0.63400000000000001,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.52200000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:22:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.505,"15min":0.51700000000000002},"min":{"1min":0.39100000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.505,"5min":0.502,"15min":0.502},"min":{"1min":0.34999999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.84599999999999997},"last":0.57499999999999996}]},{"osd":2,"last update":"Mon Apr 22 08:22:03 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.496,"15min":0.52700000000000002},"min":{"1min":0.32200000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.86899999999999999},"last":0.501},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.50700000000000001,"15min":0.53000000000000003},"min":{"1min":0.36299999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.63400000000000001,"5min":0.69099999999999995,"15min":1.0289999999999999},"last":0.55000000000000004}]},{"osd":4,"last update":"Mon Apr 22 08:22:03 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.35799999999999998,"15min":0.38500000000000001},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.53400000000000003,"5min":0.70499999999999996,"15min":0.877},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.379,"15min":0.40500000000000003},"min":{"1min":0.27500000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":2.7000000000000002},"last":0.34200000000000003}]},{"osd":5,"last update":"Mon Apr 22 08:22:03 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.39100000000000001,"15min":0.40600000000000003},"min":{"1min":0.23899999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.98399999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39900000000000002,"15min":0.436},"min":{"1min":0.26200000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.61699999999999999,"5min":0.63300000000000001,"15min":0.81399999999999995},"last":0.375}]}]},{"osd":1,"up_from":11,"seq":47244640377,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28056,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343784,"statfs":{"total":96636764160,"available":96608034816,"internally_reserved":0,"allocated":1056768,"data_stored":363130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.35599999999999998,"15min":0.33900000000000002},"min":{"1min":0.24399999999999999,"5min":0.182,"15min":0.182},"max":{"1min":0.47099999999999997,"5min":1.629,"15min":2.1850000000000001},"last":0.247},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.36899999999999999,"15min":0.36299999999999999},"min":{"1min":0.23899999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.52300000000000002,"5min":1.8220000000000001,"15min":3.1320000000000001},"last":0.26500000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39800000000000002,"15min":0.38100000000000001},"min":{"1min":0.252,"5min":0.22900000000000001,"15min":0.22},"max":{"1min":0.48899999999999999,"5min":2.0219999999999998,"15min":2.0219999999999998},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.436,"15min":0.42099999999999999},"min":{"1min":0.309,"5min":0.27400000000000002,"15min":0.255},"max":{"1min":0.66300000000000003,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":0.307}]},{"osd":3,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.55600000000000005,"15min":0.58199999999999996},"min":{"1min":0.45300000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.77800000000000002,"5min":0.80500000000000005,"15min":3.504},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56299999999999994,"15min":0.56699999999999995},"min":{"1min":0.41099999999999998,"5min":0.36599999999999999,"15min":0.35799999999999998},"max":{"1min":0.68500000000000005,"5min":1.774,"15min":1.774},"last":0.36299999999999999}]},{"osd":4,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.59399999999999997,"15min":0.59699999999999998},"min":{"1min":0.40799999999999997,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.72499999999999998,"5min":3.5579999999999998,"15min":3.5579999999999998},"last":0.44},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.56899999999999995,"15min":0.58299999999999996},"min":{"1min":0.46700000000000003,"5min":0.34599999999999997,"15min":0.316},"max":{"1min":0.73199999999999998,"5min":0.85299999999999998,"15min":3.085},"last":0.44600000000000001}]},{"osd":5,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.57399999999999995,"15min":0.57899999999999996},"min":{"1min":0.439,"5min":0.372,"15min":0.372},"max":{"1min":0.76000000000000001,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.56799999999999995,"15min":0.59499999999999997},"min":{"1min":0.42399999999999999,"5min":0.38900000000000001,"15min":0.35999999999999999},"max":{"1min":0.747,"5min":0.80200000000000005,"15min":2.6520000000000001},"last":0.42599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:22:42.875 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-22T08:22:42.876 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:22:42.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.974+0000 7f3cffdd7640 10 monclient: get_monmap_and_config 2024-04-22T08:22:42.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.974+0000 7f3cffdd7640 10 monclient: build_initial_monmap 2024-04-22T08:22:42.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.974+0000 7f3cffdd7640 10 monclient: monmap: 2024-04-22T08:22:42.974 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:42.974 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:42.974 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:22:42.975495+0000 2024-04-22T08:22:42.974 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:22:42.975495+0000 2024-04-22T08:22:42.974 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:42.975 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:42.975 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:22:42.975 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:22:42.975 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:22:42.975 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:22:42.975 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:22:42.975 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:42.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.975+0000 7f3cffdd7640 1 Processor -- start 2024-04-22T08:22:42.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.975+0000 7f3cffdd7640 1 -- start start 2024-04-22T08:22:42.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.975+0000 7f3cffdd7640 10 monclient: init 2024-04-22T08:22:42.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.975+0000 7f3cffdd7640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:42.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.975+0000 7f3cffdd7640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-04-22T08:22:42.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.975+0000 7f3cffdd7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3cf8070d00 addr v1:172.21.15.40:6789/0 2024-04-22T08:22:42.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.975+0000 7f3cffdd7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3cf8135f40 addr v1:172.21.15.189:6789/0 2024-04-22T08:22:42.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.975+0000 7f3cffdd7640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3cf813c500 0x7f3cf813e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:42.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.975+0000 7f3cffdd7640 10 monclient(hunting): picked mon.noname-a con 0x7f3cf813c500 addr v2:172.21.15.40:3300/0 2024-04-22T08:22:42.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.975+0000 7f3cffdd7640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cf80717f0 con 0x7f3cf8070d00 2024-04-22T08:22:42.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.975+0000 7f3cffdd7640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cf806d210 con 0x7f3cf8135f40 2024-04-22T08:22:42.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.975+0000 7f3cffdd7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:42.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.975+0000 7f3cffdd7640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cf81363b0 con 0x7f3cf813c500 2024-04-22T08:22:42.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.975+0000 7f3cffdd7640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:42.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.975+0000 7f3cffdd7640 10 monclient(hunting): authenticate will time out at 1436.969733s 2024-04-22T08:22:42.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.976+0000 7f3cfe374640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3cf813c500 0x7f3cf813e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:42.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.976+0000 7f3cfdb73640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f3cf8070d00 0x7f3cf80710e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:49300/0 (socket says 172.21.15.40:49300) 2024-04-22T08:22:42.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.976+0000 7f3cfdb73640 1 -- 172.21.15.40:0/804021736 learned_addr learned my addr 172.21.15.40:0/804021736 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:22:42.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.976+0000 7f3cfe374640 10 monclient(hunting): get_auth_request con 0x7f3cf813c500 auth_method 0 2024-04-22T08:22:42.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.976+0000 7f3cfe374640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:42.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.976+0000 7f3cfe374640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:42.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.976+0000 7f3cfe374640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:42.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.976+0000 7f3cfe374640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:42.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.976+0000 7f3cfe374640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:42.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.976+0000 7f3cfe374640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:42.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.976+0000 7f3cfe374640 10 monclient(hunting): handle_auth_done global_id 114297 payload 274 2024-04-22T08:22:42.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.976+0000 7f3cfe374640 1 -- 172.21.15.40:0/804021736 >> v1:172.21.15.189:6789/0 conn(0x7f3cf8135f40 legacy=0x7f3cf813ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:42.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.976+0000 7f3cfe374640 1 -- 172.21.15.40:0/804021736 >> v1:172.21.15.40:6789/0 conn(0x7f3cf8070d00 legacy=0x7f3cf80710e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:42.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.976+0000 7f3cfe374640 10 monclient: _finish_hunting 0 2024-04-22T08:22:42.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.976+0000 7f3cfe374640 1 monclient: found mon.noname-a 2024-04-22T08:22:42.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.976+0000 7f3cfe374640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:42.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.976+0000 7f3cfe374640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:22:42.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.976+0000 7f3cfe374640 1 -- 172.21.15.40:0/804021736 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cf806d4f0 con 0x7f3cf813c500 2024-04-22T08:22:42.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.979+0000 7f3cfe374640 1 -- 172.21.15.40:0/804021736 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cf813c500 msgr2=0x7f3cf813e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:42.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.979+0000 7f3cfe374640 1 --2- 172.21.15.40:0/804021736 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cf813c500 0x7f3cf813e8f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3cf0009d90 tx=0x7f3cf002bd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93eae9eff79fd2a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:42.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.979+0000 7f3cfcb71640 1 -- 172.21.15.40:0/804021736 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf002cda0 con 0x7f3cf813c500 2024-04-22T08:22:42.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.979+0000 7f3cfcb71640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:42.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.979+0000 7f3cfcb71640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:22:42.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.979+0000 7f3cfcb71640 10 monclient: dump: 2024-04-22T08:22:42.988 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:42.988 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:42.988 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:42.988 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:42.989 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:42.989 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:42.989 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:42.989 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:42.989 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:42.989 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:42.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.979+0000 7f3cfcb71640 10 monclient: _finish_auth 0 2024-04-22T08:22:42.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.979+0000 7f3cfcb71640 10 monclient: _check_auth_tickets 2024-04-22T08:22:42.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.979+0000 7f3cfcb71640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:42.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.979+0000 7f3cfcb71640 1 -- 172.21.15.40:0/804021736 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cf0034440 con 0x7f3cf813c500 2024-04-22T08:22:42.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.979+0000 7f3cfcb71640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:42.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.979+0000 7f3cfcb71640 1 -- 172.21.15.40:0/804021736 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf00347d0 con 0x7f3cf813c500 2024-04-22T08:22:42.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.979+0000 7f3cffdd7640 5 monclient: authenticate success, global_id 114297 2024-04-22T08:22:42.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.979+0000 7f3cffdd7640 10 monclient: get_monmap_and_config success 2024-04-22T08:22:42.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.979+0000 7f3cffdd7640 4 set_mon_vals no callback set 2024-04-22T08:22:42.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.979+0000 7f3cfcb71640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:42.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.979+0000 7f3cfcb71640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:22:42.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.979+0000 7f3cfcb71640 10 monclient: dump: 2024-04-22T08:22:42.990 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:42.990 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:42.990 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:42.990 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:42.990 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:42.990 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:42.990 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:42.990 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:42.990 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:42.991 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:42.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.980+0000 7f3cffdd7640 10 monclient: shutdown 2024-04-22T08:22:42.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.980+0000 7f3cffdd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:42.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.980+0000 7f3cffdd7640 1 -- 172.21.15.40:0/804021736 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cf813c500 msgr2=0x7f3cf813e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:42.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.980+0000 7f3cffdd7640 1 --2- 172.21.15.40:0/804021736 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cf813c500 0x7f3cf813e8f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3cf0009d90 tx=0x7f3cf002bd00 comp rx=0 tx=0).stop 2024-04-22T08:22:42.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.980+0000 7f3cffdd7640 1 -- 172.21.15.40:0/804021736 shutdown_connections 2024-04-22T08:22:42.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.980+0000 7f3cffdd7640 1 --2- 172.21.15.40:0/804021736 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cf813c500 0x7f3cf813e8f0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:42.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.980+0000 7f3cffdd7640 1 -- 172.21.15.40:0/804021736 >> 172.21.15.40:0/804021736 conn(0x7f3cf80a1990 msgr2=0x7f3cf806baa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:42.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.980+0000 7f3cffdd7640 1 -- 172.21.15.40:0/804021736 shutdown_connections 2024-04-22T08:22:42.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.980+0000 7f3cffdd7640 1 -- 172.21.15.40:0/804021736 wait complete. 2024-04-22T08:22:42.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.980+0000 7f3cffdd7640 10 monclient: build_initial_monmap 2024-04-22T08:22:42.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.980+0000 7f3cffdd7640 10 monclient: monmap: 2024-04-22T08:22:42.992 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:42.992 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:22:42.992 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:22:42.992 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:22:42.992 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:42.992 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:42.992 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:22:42.992 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:22:42.992 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:22:42.992 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:42.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.981+0000 7f3cffdd7640 1 Processor -- start 2024-04-22T08:22:42.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.981+0000 7f3cffdd7640 1 -- start start 2024-04-22T08:22:42.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.981+0000 7f3cffdd7640 10 monclient: init 2024-04-22T08:22:42.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.981+0000 7f3cffdd7640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:42.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.981+0000 7f3cffdd7640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:22:42.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.981+0000 7f3cffdd7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cf8070d00 0x7f3cf81cad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:42.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.981+0000 7f3cffdd7640 10 monclient(hunting): picked mon.noname-a con 0x7f3cf8070d00 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:22:42.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.981+0000 7f3cffdd7640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cf8135f40 0x7f3cf81375f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:42.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.981+0000 7f3cffdd7640 10 monclient(hunting): picked mon.noname-b con 0x7f3cf8135f40 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:22:42.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.981+0000 7f3cffdd7640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3cf813c500 0x7f3cf8137b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:42.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.981+0000 7f3cffdd7640 10 monclient(hunting): picked mon.noname-c con 0x7f3cf813c500 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:22:42.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.981+0000 7f3cffdd7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:42.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.981+0000 7f3cffdd7640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf813b020 con 0x7f3cf8070d00 2024-04-22T08:22:42.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.981+0000 7f3cffdd7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:42.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.981+0000 7f3cffdd7640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf809ea90 con 0x7f3cf8135f40 2024-04-22T08:22:42.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.981+0000 7f3cffdd7640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:42.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.981+0000 7f3cffdd7640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf813b550 con 0x7f3cf813c500 2024-04-22T08:22:42.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.981+0000 7f3cffdd7640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:42.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.981+0000 7f3cffdd7640 10 monclient(hunting): authenticate will time out at 1436.975345s 2024-04-22T08:22:42.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.981+0000 7f3cfdb73640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cf8070d00 0x7f3cf81cad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:42.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.981+0000 7f3cfdb73640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cf8070d00 0x7f3cf81cad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44512/0 (socket says 172.21.15.40:44512) 2024-04-22T08:22:42.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.981+0000 7f3cfdb73640 1 -- 172.21.15.40:0/2895791543 learned_addr learned my addr 172.21.15.40:0/2895791543 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:22:42.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.981+0000 7f3cfdb73640 10 monclient(hunting): get_auth_request con 0x7f3cf8070d00 auth_method 0 2024-04-22T08:22:42.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.981+0000 7f3cfdb73640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:42.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.981+0000 7f3cfdb73640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:42.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.981+0000 7f3cfdb73640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:42.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.981+0000 7f3cfe374640 1 --2- 172.21.15.40:0/2895791543 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3cf813c500 0x7f3cf8137b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:42.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.981+0000 7f3cfe374640 10 monclient(hunting): get_auth_request con 0x7f3cf813c500 auth_method 0 2024-04-22T08:22:42.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.981+0000 7f3cfe374640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:42.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.981+0000 7f3cfe374640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:42.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.981+0000 7f3cfe374640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:42.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.981+0000 7f3cfdb73640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:42.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.981+0000 7f3cfdb73640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:42.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.981+0000 7f3cfdb73640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:42.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.981+0000 7f3cfe374640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:42.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.981+0000 7f3cfe374640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:42.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.981+0000 7f3cfe374640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:42.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.981+0000 7f3cfd372640 1 --2- 172.21.15.40:0/2895791543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cf8135f40 0x7f3cf81375f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:42.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.981+0000 7f3cfdb73640 10 monclient(hunting): handle_auth_done global_id 114300 payload 824 2024-04-22T08:22:42.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.981+0000 7f3cfdb73640 1 -- 172.21.15.40:0/2895791543 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3cf813c500 msgr2=0x7f3cf8137b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:42.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.981+0000 7f3cfdb73640 1 --2- 172.21.15.40:0/2895791543 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3cf813c500 0x7f3cf8137b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:42.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.981+0000 7f3cfdb73640 1 -- 172.21.15.40:0/2895791543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cf8135f40 msgr2=0x7f3cf81375f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:42.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.981+0000 7f3cfdb73640 1 --2- 172.21.15.40:0/2895791543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cf8135f40 0x7f3cf81375f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:42.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.982+0000 7f3cfdb73640 10 monclient: _finish_hunting 0 2024-04-22T08:22:42.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.982+0000 7f3cfdb73640 1 monclient: found mon.noname-a 2024-04-22T08:22:42.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.982+0000 7f3cfdb73640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:42.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.982+0000 7f3cfdb73640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:22:42.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.982+0000 7f3cfdb73640 1 -- 172.21.15.40:0/2895791543 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cf0009a70 con 0x7f3cf8070d00 2024-04-22T08:22:42.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.982+0000 7f3cfd372640 10 monclient: get_auth_request con 0x7f3cf8135f40 auth_method 0 2024-04-22T08:22:42.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.982+0000 7f3cfe374640 1 --2- 172.21.15.40:0/2895791543 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3cf813c500 0x7f3cf8137b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:22:42.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.982+0000 7f3cfd372640 1 --2- 172.21.15.40:0/2895791543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cf8135f40 0x7f3cf81375f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T08:22:42.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.984+0000 7f3cfdb73640 1 -- 172.21.15.40:0/2895791543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cf8070d00 msgr2=0x7f3cf81cad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:42.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.984+0000 7f3cfdb73640 1 --2- 172.21.15.40:0/2895791543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cf8070d00 0x7f3cf81cad80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3cf400c490 tx=0x7f3cf400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:42.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.984+0000 7f3cee7fc640 1 -- 172.21.15.40:0/2895791543 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf4007070 con 0x7f3cf8070d00 2024-04-22T08:22:42.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.984+0000 7f3cee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:42.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.984+0000 7f3cee7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:22:42.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.984+0000 7f3cee7fc640 10 monclient: dump: 2024-04-22T08:22:42.997 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:42.997 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:42.997 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:42.997 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:42.997 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:42.997 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:42.997 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:42.997 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:42.997 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:42.997 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:42.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.984+0000 7f3cee7fc640 10 monclient: _finish_auth 0 2024-04-22T08:22:42.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.984+0000 7f3cee7fc640 10 monclient: _check_auth_tickets 2024-04-22T08:22:42.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.984+0000 7f3cee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:42.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.984+0000 7f3cffdd7640 5 monclient: authenticate success, global_id 114300 2024-04-22T08:22:42.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.984+0000 7f3cffdd7640 10 monclient: _renew_subs 2024-04-22T08:22:42.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.984+0000 7f3cffdd7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:42.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.984+0000 7f3cee7fc640 1 -- 172.21.15.40:0/2895791543 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cf40071f0 con 0x7f3cf8070d00 2024-04-22T08:22:42.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.984+0000 7f3cffdd7640 1 -- 172.21.15.40:0/2895791543 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cf80a3e70 con 0x7f3cf8070d00 2024-04-22T08:22:42.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.984+0000 7f3cee7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:42.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.984+0000 7f3cee7fc640 1 -- 172.21.15.40:0/2895791543 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf40075a0 con 0x7f3cf8070d00 2024-04-22T08:22:42.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.984+0000 7f3cee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:42.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.984+0000 7f3cee7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:22:42.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.984+0000 7f3cee7fc640 10 monclient: dump: 2024-04-22T08:22:42.998 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:42.998 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:42.999 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:42.999 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:42.999 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:42.999 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:42.999 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:42.999 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:42.999 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:42.999 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:42.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.984+0000 7f3ceffff640 4 set_mon_vals no callback set 2024-04-22T08:22:42.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.984+0000 7f3cffdd7640 10 monclient: _renew_subs 2024-04-22T08:22:42.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.984+0000 7f3cffdd7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:42.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.984+0000 7f3cffdd7640 1 -- 172.21.15.40:0/2895791543 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cf81391f0 con 0x7f3cf8070d00 2024-04-22T08:22:42.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.985+0000 7f3cee7fc640 1 -- 172.21.15.40:0/2895791543 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f3cf401b400 con 0x7f3cf8070d00 2024-04-22T08:22:42.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.985+0000 7f3cffdd7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:43.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.985+0000 7f3cffdd7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:43.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.985+0000 7f3cffdd7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:43.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.985+0000 7f3cffdd7640 1 -- 172.21.15.40:0/2895791543 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cf804e9b0 con 0x7f3cf8070d00 2024-04-22T08:22:43.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.985+0000 7f3cee7fc640 1 --2- 172.21.15.40:0/2895791543 >> v2:172.21.15.189:6800/30168 conn(0x7f3cdc0377d0 0x7f3cdc039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:43.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.985+0000 7f3cee7fc640 1 -- 172.21.15.40:0/2895791543 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(317..317 src has 239..317) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f3cf4049e90 con 0x7f3cf8070d00 2024-04-22T08:22:43.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.985+0000 7f3cfd372640 1 --2- 172.21.15.40:0/2895791543 >> v2:172.21.15.189:6800/30168 conn(0x7f3cdc0377d0 0x7f3cdc039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:43.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.985+0000 7f3cfd372640 10 monclient: get_auth_request con 0x7f3cdc0377d0 auth_method 0 2024-04-22T08:22:43.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.988+0000 7f3cfd372640 1 --2- 172.21.15.40:0/2895791543 >> v2:172.21.15.189:6800/30168 conn(0x7f3cdc0377d0 0x7f3cdc039c90 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f3cf8130020 tx=0x7f3ce8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:43.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.990+0000 7f3cee7fc640 1 -- 172.21.15.40:0/2895791543 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f3cf4010710 con 0x7f3cf8070d00 2024-04-22T08:22:43.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.990+0000 7f3cee7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:43.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:42.990+0000 7f3cee7fc640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:22:43.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.166+0000 7f3cffdd7640 1 -- 172.21.15.40:0/2895791543 --> v2:172.21.15.189:6800/30168 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3cf809e740 con 0x7f3cdc0377d0 2024-04-22T08:22:43.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.168+0000 7f3cee7fc640 1 -- 172.21.15.40:0/2895791543 <== mgr.4102 v2:172.21.15.189:6800/30168 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+53849 (secure 0 0 0) 0x7f3ce800d7a0 con 0x7f3cdc0377d0 2024-04-22T08:22:43.169 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:22:43.169 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-22T08:22:43.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.170+0000 7f3cffdd7640 1 -- 172.21.15.40:0/2895791543 >> v2:172.21.15.189:6800/30168 conn(0x7f3cdc0377d0 msgr2=0x7f3cdc039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:43.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.170+0000 7f3cffdd7640 1 --2- 172.21.15.40:0/2895791543 >> v2:172.21.15.189:6800/30168 conn(0x7f3cdc0377d0 0x7f3cdc039c90 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f3cf8130020 tx=0x7f3ce8009040 comp rx=0 tx=0).stop 2024-04-22T08:22:43.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.170+0000 7f3cffdd7640 10 monclient: shutdown 2024-04-22T08:22:43.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.170+0000 7f3cffdd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:43.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.170+0000 7f3cffdd7640 1 -- 172.21.15.40:0/2895791543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cf8070d00 msgr2=0x7f3cf81cad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:43.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.170+0000 7f3cffdd7640 1 --2- 172.21.15.40:0/2895791543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cf8070d00 0x7f3cf81cad80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3cf400c490 tx=0x7f3cf400c950 comp rx=0 tx=0).stop 2024-04-22T08:22:43.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.170+0000 7f3cffdd7640 1 -- 172.21.15.40:0/2895791543 shutdown_connections 2024-04-22T08:22:43.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.170+0000 7f3cffdd7640 1 --2- 172.21.15.40:0/2895791543 >> v2:172.21.15.189:6800/30168 conn(0x7f3cdc0377d0 0x7f3cdc039c90 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:43.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.170+0000 7f3cffdd7640 1 --2- 172.21.15.40:0/2895791543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cf8135f40 0x7f3cf81375f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:43.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.170+0000 7f3cffdd7640 1 --2- 172.21.15.40:0/2895791543 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3cf813c500 0x7f3cf8137b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:43.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.170+0000 7f3cffdd7640 1 --2- 172.21.15.40:0/2895791543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cf8070d00 0x7f3cf81cad80 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:43.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.170+0000 7f3cffdd7640 1 -- 172.21.15.40:0/2895791543 >> 172.21.15.40:0/2895791543 conn(0x7f3cf80a1990 msgr2=0x7f3cf80a2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:43.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.170+0000 7f3cffdd7640 1 -- 172.21.15.40:0/2895791543 shutdown_connections 2024-04-22T08:22:43.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.170+0000 7f3cffdd7640 1 -- 172.21.15.40:0/2895791543 wait complete. 2024-04-22T08:22:43.180 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":612,"stamp":"2024-04-22T08:22:42.206057+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":185760,"kb_used_data":7232,"kb_used_omap":9,"kb_used_meta":178422,"kb_avail":566045280,"statfs":{"total":579820584960,"available":579630366720,"internally_reserved":0,"allocated":7405568,"data_stored":3236636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9560,"internal_metadata":182704808},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002606"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":304,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-04-22T08:22:17.159438+0000","last_change":"2024-04-22T08:12:51.443086+0000","last_active":"2024-04-22T08:22:17.159438+0000","last_peered":"2024-04-22T08:22:17.159438+0000","last_clean":"2024-04-22T08:22:17.159438+0000","last_became_active":"2024-04-22T08:12:51.442730+0000","last_became_peered":"2024-04-22T08:12:51.442730+0000","last_unstale":"2024-04-22T08:22:17.159438+0000","last_undegraded":"2024-04-22T08:22:17.159438+0000","last_fullsized":"2024-04-22T08:22:17.159438+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:06:13.481863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":293,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-04-22T08:22:16.957788+0000","last_change":"2024-04-22T08:12:50.841276+0000","last_active":"2024-04-22T08:22:16.957788+0000","last_peered":"2024-04-22T08:22:16.957788+0000","last_clean":"2024-04-22T08:22:16.957788+0000","last_became_active":"2024-04-22T08:12:50.841008+0000","last_became_peered":"2024-04-22T08:12:50.841008+0000","last_unstale":"2024-04-22T08:22:16.957788+0000","last_undegraded":"2024-04-22T08:22:16.957788+0000","last_fullsized":"2024-04-22T08:22:16.957788+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:47:00.224321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":304,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-04-22T08:22:17.160920+0000","last_change":"2024-04-22T08:12:51.403419+0000","last_active":"2024-04-22T08:22:17.160920+0000","last_peered":"2024-04-22T08:22:17.160920+0000","last_clean":"2024-04-22T08:22:17.160920+0000","last_became_active":"2024-04-22T08:12:51.403182+0000","last_became_peered":"2024-04-22T08:12:51.403182+0000","last_unstale":"2024-04-22T08:22:17.160920+0000","last_undegraded":"2024-04-22T08:22:17.160920+0000","last_fullsized":"2024-04-22T08:22:17.160920+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:52:39.730597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":293,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-04-22T08:22:16.957078+0000","last_change":"2024-04-22T08:12:51.403260+0000","last_active":"2024-04-22T08:22:16.957078+0000","last_peered":"2024-04-22T08:22:16.957078+0000","last_clean":"2024-04-22T08:22:16.957078+0000","last_became_active":"2024-04-22T08:12:51.402980+0000","last_became_peered":"2024-04-22T08:12:51.402980+0000","last_unstale":"2024-04-22T08:22:16.957078+0000","last_undegraded":"2024-04-22T08:22:16.957078+0000","last_fullsized":"2024-04-22T08:22:16.957078+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:08:04.625870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":295,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-04-22T08:22:17.162182+0000","last_change":"2024-04-22T08:12:50.836566+0000","last_active":"2024-04-22T08:22:17.162182+0000","last_peered":"2024-04-22T08:22:17.162182+0000","last_clean":"2024-04-22T08:22:17.162182+0000","last_became_active":"2024-04-22T08:12:50.836250+0000","last_became_peered":"2024-04-22T08:12:50.836250+0000","last_unstale":"2024-04-22T08:22:17.162182+0000","last_undegraded":"2024-04-22T08:22:17.162182+0000","last_fullsized":"2024-04-22T08:22:17.162182+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:42:13.693209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":302,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-04-22T08:22:16.950081+0000","last_change":"2024-04-22T08:12:50.836508+0000","last_active":"2024-04-22T08:22:16.950081+0000","last_peered":"2024-04-22T08:22:16.950081+0000","last_clean":"2024-04-22T08:22:16.950081+0000","last_became_active":"2024-04-22T08:12:50.836210+0000","last_became_peered":"2024-04-22T08:12:50.836210+0000","last_unstale":"2024-04-22T08:22:16.950081+0000","last_undegraded":"2024-04-22T08:22:16.950081+0000","last_fullsized":"2024-04-22T08:22:16.950081+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:24:53.345661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":304,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-04-22T08:22:17.158781+0000","last_change":"2024-04-22T08:12:50.842633+0000","last_active":"2024-04-22T08:22:17.158781+0000","last_peered":"2024-04-22T08:22:17.158781+0000","last_clean":"2024-04-22T08:22:17.158781+0000","last_became_active":"2024-04-22T08:12:50.842392+0000","last_became_peered":"2024-04-22T08:12:50.842392+0000","last_unstale":"2024-04-22T08:22:17.158781+0000","last_undegraded":"2024-04-22T08:22:17.158781+0000","last_fullsized":"2024-04-22T08:22:17.158781+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:53:14.717539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":295,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-04-22T08:22:17.162502+0000","last_change":"2024-04-22T08:12:50.835820+0000","last_active":"2024-04-22T08:22:17.162502+0000","last_peered":"2024-04-22T08:22:17.162502+0000","last_clean":"2024-04-22T08:22:17.162502+0000","last_became_active":"2024-04-22T08:12:50.835541+0000","last_became_peered":"2024-04-22T08:12:50.835541+0000","last_unstale":"2024-04-22T08:22:17.162502+0000","last_undegraded":"2024-04-22T08:22:17.162502+0000","last_fullsized":"2024-04-22T08:22:17.162502+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:52:21.798509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":454,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-04-22T08:22:17.158877+0000","last_change":"2024-04-22T08:12:51.404305+0000","last_active":"2024-04-22T08:22:17.158877+0000","last_peered":"2024-04-22T08:22:17.158877+0000","last_clean":"2024-04-22T08:22:17.158877+0000","last_became_active":"2024-04-22T08:12:51.403208+0000","last_became_peered":"2024-04-22T08:12:51.403208+0000","last_unstale":"2024-04-22T08:22:17.158877+0000","last_undegraded":"2024-04-22T08:22:17.158877+0000","last_fullsized":"2024-04-22T08:22:17.158877+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:04:43.253802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640379,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28056,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343784,"statfs":{"total":96636764160,"available":96608034816,"internally_reserved":0,"allocated":1056768,"data_stored":363130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.35199999999999998,"15min":0.371},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":0.39800000000000002,"5min":2.5920000000000001,"15min":2.5920000000000001},"last":0.255},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.40899999999999997,"15min":0.38700000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.21299999999999999},"max":{"1min":0.51400000000000001,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.30599999999999999}]},{"osd":1,"last update":"Mon Apr 22 08:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.34300000000000003,"15min":0.34399999999999997},"min":{"1min":0.249,"5min":0.23400000000000001,"15min":0.17100000000000001},"max":{"1min":0.443,"5min":0.499,"15min":0.55700000000000005},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.38100000000000001,"15min":0.34499999999999997},"min":{"1min":0.254,"5min":0.24299999999999999,"15min":0.20699999999999999},"max":{"1min":0.48499999999999999,"5min":0.55600000000000005,"15min":0.70099999999999996},"last":0.33300000000000002}]},{"osd":3,"last update":"Mon Apr 22 08:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.59699999999999998,"15min":0.56499999999999995},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.22500000000000001},"max":{"1min":0.82999999999999996,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.58799999999999997,"15min":0.60199999999999998},"min":{"1min":0.33400000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.749,"5min":0.84799999999999998,"15min":1.8440000000000001},"last":0.45600000000000002}]},{"osd":4,"last update":"Mon Apr 22 08:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58399999999999996,"15min":0.68600000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.30499999999999999},"max":{"1min":0.78800000000000003,"5min":2.7879999999999998,"15min":6.2949999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.55100000000000005,"15min":0.68400000000000005},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.69499999999999995,"5min":0.91200000000000003,"15min":6.2149999999999999},"last":0.50900000000000001}]},{"osd":5,"last update":"Mon Apr 22 08:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.56899999999999995,"15min":0.53400000000000003},"min":{"1min":0.40000000000000002,"5min":0.35799999999999998,"15min":0.27800000000000002},"max":{"1min":0.72499999999999998,"5min":0.84299999999999997,"15min":0.85199999999999998},"last":0.441},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56399999999999995,"15min":0.53500000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.29699999999999999},"max":{"1min":0.72999999999999998,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.46999999999999997}]}]},{"osd":0,"up_from":11,"seq":47244640378,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28580,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343260,"statfs":{"total":96636764160,"available":96607498240,"internally_reserved":0,"allocated":1589248,"data_stored":892058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.32000000000000001,"15min":0.34599999999999997},"min":{"1min":0.23699999999999999,"5min":0.189,"15min":0.18099999999999999},"max":{"1min":0.46500000000000002,"5min":0.47599999999999998,"15min":1.7190000000000001},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.32600000000000001,"15min":0.33000000000000002},"min":{"1min":0.22,"5min":0.21199999999999999,"15min":0.19800000000000001},"max":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.56999999999999995},"last":0.28799999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.373,"15min":0.36099999999999999},"min":{"1min":0.29099999999999998,"5min":0.26300000000000001,"15min":0.222},"max":{"1min":0.499,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.313},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.38,"15min":0.39200000000000002},"min":{"1min":0.24299999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.51400000000000001,"5min":0.60399999999999998,"15min":0.626},"last":0.35999999999999999}]},{"osd":3,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55300000000000005,"15min":0.54900000000000004},"min":{"1min":0.37,"5min":0.35699999999999998,"15min":0.32200000000000001},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.54500000000000004,"15min":0.53300000000000003},"min":{"1min":0.42999999999999999,"5min":0.32300000000000001,"15min":0.29499999999999998},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.92000000000000004},"last":0.5}]},{"osd":4,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.55000000000000004,"15min":0.55100000000000005},"min":{"1min":0.42599999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.72099999999999997,"5min":0.81799999999999995,"15min":0.85999999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.54700000000000004,"15min":0.54200000000000004},"min":{"1min":0.39700000000000002,"5min":0.36299999999999999,"15min":0.33900000000000002},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.52300000000000002}]},{"osd":5,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55000000000000004,"15min":0.54600000000000004},"min":{"1min":0.41399999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.71999999999999997,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54900000000000004,"15min":0.55500000000000005},"min":{"1min":0.38500000000000001,"5min":0.36799999999999999,"15min":0.35899999999999999},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.48699999999999999}]}]},{"osd":5,"up_from":11,"seq":47244640379,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44440,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":43390,"kb_avail":94327400,"statfs":{"total":96636764160,"available":96591257600,"internally_reserved":0,"allocated":1056768,"data_stored":363130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":44431814},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:22:04 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.49099999999999999,"15min":0.49199999999999999},"min":{"1min":0.314,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.60699999999999998,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.496,"5min":0.49399999999999999,"15min":0.51700000000000002},"min":{"1min":0.32700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":1.484},"last":0.46200000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:22:04 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.49399999999999999,"15min":0.504},"min":{"1min":0.36099999999999999,"5min":0.247,"15min":0.247},"max":{"1min":0.64200000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.498},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52600000000000002,"15min":0.52700000000000002},"min":{"1min":0.36899999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.65500000000000003,"5min":0.75700000000000001,"15min":0.81499999999999995},"last":0.439}]},{"osd":2,"last update":"Mon Apr 22 08:22:04 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.5,"15min":0.51300000000000001},"min":{"1min":0.40500000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.67500000000000004,"5min":0.68600000000000005,"15min":0.76800000000000002},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.499,"5min":0.504,"15min":0.58099999999999996},"min":{"1min":0.38400000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.57599999999999996,"5min":0.74099999999999999,"15min":3.6970000000000001},"last":0.52200000000000002}]},{"osd":3,"last update":"Mon Apr 22 08:22:04 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.39900000000000002,"15min":0.39400000000000002},"min":{"1min":0.28699999999999998,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.52000000000000002,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41599999999999998,"15min":0.40899999999999997},"min":{"1min":0.29299999999999998,"5min":0.25900000000000001,"15min":0.255},"max":{"1min":0.60099999999999998,"5min":0.62,"15min":0.621},"last":0.38100000000000001}]},{"osd":4,"last update":"Mon Apr 22 08:22:04 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.38100000000000001,"15min":0.41699999999999998},"min":{"1min":0.252,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.47199999999999998,"5min":1.0820000000000001,"15min":1.528},"last":0.307},{"interface":"front","average":{"1min":0.379,"5min":0.40400000000000003,"15min":0.42999999999999999},"min":{"1min":0.27800000000000002,"5min":0.24099999999999999,"15min":0.23599999999999999},"max":{"1min":0.61299999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.33100000000000002}]}]},{"osd":4,"up_from":11,"seq":47244640378,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28060,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343780,"statfs":{"total":96636764160,"available":96608030720,"internally_reserved":0,"allocated":1056768,"data_stored":363130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.48099999999999998,"15min":0.47899999999999998},"min":{"1min":0.317,"5min":0.215,"15min":0.215},"max":{"1min":0.66000000000000003,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.47999999999999998,"15min":0.48399999999999999},"min":{"1min":0.29399999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.66000000000000003,"5min":0.67300000000000004,"15min":0.69299999999999995},"last":0.60099999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.501,"15min":0.49399999999999999},"min":{"1min":0.36899999999999999,"5min":0.27600000000000002,"15min":0.26900000000000002},"max":{"1min":0.71499999999999997,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.495},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.497,"15min":0.5},"min":{"1min":0.28399999999999997,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.68899999999999995,"5min":0.71299999999999997,"15min":0.78800000000000003},"last":0.56399999999999995}]},{"osd":2,"last update":"Mon Apr 22 08:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.52900000000000003,"15min":0.55700000000000005},"min":{"1min":0.35699999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.54100000000000004,"15min":0.53500000000000003},"min":{"1min":0.35199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.63600000000000001}]},{"osd":3,"last update":"Mon Apr 22 08:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.373,"15min":0.39000000000000001},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.57799999999999996,"5min":1.204,"15min":1.204},"last":0.436},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.39100000000000001,"15min":0.40000000000000002},"min":{"1min":0.24199999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.54700000000000004,"5min":0.62,"15min":0.89100000000000001},"last":0.47199999999999998}]},{"osd":5,"last update":"Mon Apr 22 08:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.38600000000000001,"15min":0.38200000000000001},"min":{"1min":0.26200000000000001,"5min":0.251,"15min":0.23100000000000001},"max":{"1min":0.57499999999999996,"5min":0.625,"15min":0.65800000000000003},"last":0.505},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45300000000000001,"15min":0.432},"min":{"1min":0.30299999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.70499999999999996,"5min":2.2360000000000002,"15min":2.2360000000000002},"last":0.52500000000000002}]}]},{"osd":3,"up_from":11,"seq":47244640377,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28568,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343272,"statfs":{"total":96636764160,"available":96607510528,"internally_reserved":0,"allocated":1589248,"data_stored":892058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:22:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.53000000000000003,"15min":0.50800000000000001},"min":{"1min":0.34200000000000003,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.64400000000000002,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.52600000000000002,"15min":0.51000000000000001},"min":{"1min":0.38,"5min":0.32000000000000001,"15min":0.309},"max":{"1min":0.63400000000000001,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.52200000000000002}]},{"osd":1,"last update":"Mon Apr 22 08:22:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.505,"15min":0.51700000000000002},"min":{"1min":0.39100000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.505,"5min":0.502,"15min":0.502},"min":{"1min":0.34999999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.84599999999999997},"last":0.57499999999999996}]},{"osd":2,"last update":"Mon Apr 22 08:22:03 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.496,"15min":0.52700000000000002},"min":{"1min":0.32200000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.86899999999999999},"last":0.501},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.50700000000000001,"15min":0.53000000000000003},"min":{"1min":0.36299999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.63400000000000001,"5min":0.69099999999999995,"15min":1.0289999999999999},"last":0.55000000000000004}]},{"osd":4,"last update":"Mon Apr 22 08:22:03 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.35799999999999998,"15min":0.38500000000000001},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.53400000000000003,"5min":0.70499999999999996,"15min":0.877},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.379,"15min":0.40500000000000003},"min":{"1min":0.27500000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":2.7000000000000002},"last":0.34200000000000003}]},{"osd":5,"last update":"Mon Apr 22 08:22:03 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.39100000000000001,"15min":0.40600000000000003},"min":{"1min":0.23899999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.98399999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39900000000000002,"15min":0.436},"min":{"1min":0.26200000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.61699999999999999,"5min":0.63300000000000001,"15min":0.81399999999999995},"last":0.375}]}]},{"osd":1,"up_from":11,"seq":47244640377,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28056,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343784,"statfs":{"total":96636764160,"available":96608034816,"internally_reserved":0,"allocated":1056768,"data_stored":363130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.35599999999999998,"15min":0.33900000000000002},"min":{"1min":0.24399999999999999,"5min":0.182,"15min":0.182},"max":{"1min":0.47099999999999997,"5min":1.629,"15min":2.1850000000000001},"last":0.247},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.36899999999999999,"15min":0.36299999999999999},"min":{"1min":0.23899999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.52300000000000002,"5min":1.8220000000000001,"15min":3.1320000000000001},"last":0.26500000000000001}]},{"osd":2,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39800000000000002,"15min":0.38100000000000001},"min":{"1min":0.252,"5min":0.22900000000000001,"15min":0.22},"max":{"1min":0.48899999999999999,"5min":2.0219999999999998,"15min":2.0219999999999998},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.436,"15min":0.42099999999999999},"min":{"1min":0.309,"5min":0.27400000000000002,"15min":0.255},"max":{"1min":0.66300000000000003,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":0.307}]},{"osd":3,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.55600000000000005,"15min":0.58199999999999996},"min":{"1min":0.45300000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.77800000000000002,"5min":0.80500000000000005,"15min":3.504},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56299999999999994,"15min":0.56699999999999995},"min":{"1min":0.41099999999999998,"5min":0.36599999999999999,"15min":0.35799999999999998},"max":{"1min":0.68500000000000005,"5min":1.774,"15min":1.774},"last":0.36299999999999999}]},{"osd":4,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.59399999999999997,"15min":0.59699999999999998},"min":{"1min":0.40799999999999997,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.72499999999999998,"5min":3.5579999999999998,"15min":3.5579999999999998},"last":0.44},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.56899999999999995,"15min":0.58299999999999996},"min":{"1min":0.46700000000000003,"5min":0.34599999999999997,"15min":0.316},"max":{"1min":0.73199999999999998,"5min":0.85299999999999998,"15min":3.085},"last":0.44600000000000001}]},{"osd":5,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.57399999999999995,"15min":0.57899999999999996},"min":{"1min":0.439,"5min":0.372,"15min":0.372},"max":{"1min":0.76000000000000001,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.56799999999999995,"15min":0.59499999999999997},"min":{"1min":0.42399999999999999,"5min":0.38900000000000001,"15min":0.35999999999999999},"max":{"1min":0.747,"5min":0.80200000000000005,"15min":2.6520000000000001},"last":0.42599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:22:43.182 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T08:22:43.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.280+0000 7f04ea469640 10 monclient: get_monmap_and_config 2024-04-22T08:22:43.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.280+0000 7f04ea469640 10 monclient: build_initial_monmap 2024-04-22T08:22:43.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.280+0000 7f04ea469640 10 monclient: monmap: 2024-04-22T08:22:43.280 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:43.280 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:43.280 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:22:43.281307+0000 2024-04-22T08:22:43.280 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:22:43.281307+0000 2024-04-22T08:22:43.280 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:43.281 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:43.281 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:22:43.281 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:22:43.281 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:22:43.281 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:22:43.281 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:22:43.281 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:43.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.281+0000 7f04ea469640 1 Processor -- start 2024-04-22T08:22:43.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.281+0000 7f04ea469640 1 -- start start 2024-04-22T08:22:43.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.281+0000 7f04ea469640 10 monclient: init 2024-04-22T08:22:43.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.281+0000 7f04ea469640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:43.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.281+0000 7f04ea469640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-04-22T08:22:43.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.281+0000 7f04ea469640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f04e40a1a80 0x7f04e40a1e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:43.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.281+0000 7f04ea469640 10 monclient(hunting): picked mon.noname-b con 0x7f04e40a1a80 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:22:43.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.281+0000 7f04ea469640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f04e40a0140 addr v1:172.21.15.40:6789/0 2024-04-22T08:22:43.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.281+0000 7f04ea469640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f04e40a23a0 0x7f04e4136330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:43.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.281+0000 7f04ea469640 10 monclient(hunting): picked mon.noname-c con 0x7f04e40a23a0 addr v2:172.21.15.189:3300/0 2024-04-22T08:22:43.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.281+0000 7f04ea469640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04e409d8d0 con 0x7f04e40a0140 2024-04-22T08:22:43.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.281+0000 7f04ea469640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:43.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.281+0000 7f04ea469640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04e409f860 con 0x7f04e40a23a0 2024-04-22T08:22:43.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.281+0000 7f04ea469640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:43.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.281+0000 7f04ea469640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04e409efc0 con 0x7f04e40a1a80 2024-04-22T08:22:43.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.281+0000 7f04ea469640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:43.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.281+0000 7f04ea469640 10 monclient(hunting): authenticate will time out at 1437.275509s 2024-04-22T08:22:43.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.281+0000 7f04e3fff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f04e40a1a80 0x7f04e40a1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:43.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.281+0000 7f04e37fe640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f04e40a0140 0x7f04e4135c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:49314/0 (socket says 172.21.15.40:49314) 2024-04-22T08:22:43.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.281+0000 7f04e37fe640 1 -- 172.21.15.40:0/2768666796 learned_addr learned my addr 172.21.15.40:0/2768666796 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:22:43.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.281+0000 7f04e3fff640 10 monclient(hunting): get_auth_request con 0x7f04e40a1a80 auth_method 0 2024-04-22T08:22:43.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.281+0000 7f04e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:43.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.281+0000 7f04e3fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:43.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.281+0000 7f04e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:43.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.281+0000 7f04e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:43.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.281+0000 7f04e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:43.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.281+0000 7f04e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:43.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.282+0000 7f04e3fff640 10 monclient(hunting): handle_auth_done global_id 124133 payload 274 2024-04-22T08:22:43.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.282+0000 7f04e8a06640 1 --2- 172.21.15.40:0/2768666796 >> v2:172.21.15.189:3300/0 conn(0x7f04e40a23a0 0x7f04e4136330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:43.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.282+0000 7f04e3fff640 1 -- 172.21.15.40:0/2768666796 >> v2:172.21.15.189:3300/0 conn(0x7f04e40a23a0 msgr2=0x7f04e4136330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:43.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.282+0000 7f04e3fff640 1 --2- 172.21.15.40:0/2768666796 >> v2:172.21.15.189:3300/0 conn(0x7f04e40a23a0 0x7f04e4136330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:43.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.282+0000 7f04e3fff640 1 -- 172.21.15.40:0/2768666796 >> v1:172.21.15.40:6789/0 conn(0x7f04e40a0140 legacy=0x7f04e4135c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:43.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.282+0000 7f04e3fff640 10 monclient: _finish_hunting 0 2024-04-22T08:22:43.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.282+0000 7f04e3fff640 1 monclient: found mon.noname-b 2024-04-22T08:22:43.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.282+0000 7f04e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:43.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.282+0000 7f04e3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:22:43.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.282+0000 7f04e3fff640 1 -- 172.21.15.40:0/2768666796 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04e409f2a0 con 0x7f04e40a1a80 2024-04-22T08:22:43.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.285+0000 7f04e3fff640 1 -- 172.21.15.40:0/2768666796 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f04e40a1a80 msgr2=0x7f04e40a1e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:43.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.285+0000 7f04e3fff640 1 --2- 172.21.15.40:0/2768666796 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f04e40a1a80 0x7f04e40a1e60 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f04d40097a0 tx=0x7f04d402b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27405ea9c90cc50e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:43.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.285+0000 7f04e2ffd640 1 -- 172.21.15.40:0/2768666796 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d4030a00 con 0x7f04e40a1a80 2024-04-22T08:22:43.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.285+0000 7f04e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:43.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.285+0000 7f04e2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:22:43.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.285+0000 7f04e2ffd640 10 monclient: dump: 2024-04-22T08:22:43.292 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:43.292 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:43.293 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:43.293 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:43.293 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:43.293 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:43.293 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:43.293 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:43.293 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:43.293 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:43.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.285+0000 7f04e2ffd640 10 monclient: _finish_auth 0 2024-04-22T08:22:43.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.285+0000 7f04e2ffd640 10 monclient: _check_auth_tickets 2024-04-22T08:22:43.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.285+0000 7f04e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:43.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.285+0000 7f04e2ffd640 1 -- 172.21.15.40:0/2768666796 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04d402e050 con 0x7f04e40a1a80 2024-04-22T08:22:43.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.285+0000 7f04ea469640 5 monclient: authenticate success, global_id 124133 2024-04-22T08:22:43.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.285+0000 7f04ea469640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:22:43.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.285+0000 7f04e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:43.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.285+0000 7f04ea469640 10 monclient: get_monmap_and_config success 2024-04-22T08:22:43.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.285+0000 7f04ea469640 4 set_mon_vals no callback set 2024-04-22T08:22:43.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.285+0000 7f04e2ffd640 1 -- 172.21.15.40:0/2768666796 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d4030d80 con 0x7f04e40a1a80 2024-04-22T08:22:43.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.285+0000 7f04e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:43.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.285+0000 7f04e2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:22:43.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.285+0000 7f04e2ffd640 10 monclient: dump: 2024-04-22T08:22:43.294 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:43.294 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:43.294 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:43.294 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:43.295 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:43.295 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:43.295 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:43.295 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:43.295 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:43.295 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:43.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.285+0000 7f04ea469640 10 monclient: shutdown 2024-04-22T08:22:43.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.285+0000 7f04ea469640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:43.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.285+0000 7f04ea469640 1 -- 172.21.15.40:0/2768666796 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f04e40a1a80 msgr2=0x7f04e40a1e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:43.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.285+0000 7f04ea469640 1 --2- 172.21.15.40:0/2768666796 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f04e40a1a80 0x7f04e40a1e60 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f04d40097a0 tx=0x7f04d402b710 comp rx=0 tx=0).stop 2024-04-22T08:22:43.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.286+0000 7f04ea469640 1 -- 172.21.15.40:0/2768666796 shutdown_connections 2024-04-22T08:22:43.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.286+0000 7f04ea469640 1 --2- 172.21.15.40:0/2768666796 >> v2:172.21.15.189:3300/0 conn(0x7f04e40a23a0 0x7f04e4136330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:43.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.286+0000 7f04ea469640 1 --2- 172.21.15.40:0/2768666796 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f04e40a1a80 0x7f04e40a1e60 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:43.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.286+0000 7f04ea469640 1 -- 172.21.15.40:0/2768666796 >> 172.21.15.40:0/2768666796 conn(0x7f04e40a66a0 msgr2=0x7f04e40a2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:43.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.286+0000 7f04ea469640 1 -- 172.21.15.40:0/2768666796 shutdown_connections 2024-04-22T08:22:43.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.286+0000 7f04ea469640 1 -- 172.21.15.40:0/2768666796 wait complete. 2024-04-22T08:22:43.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.286+0000 7f04ea469640 10 monclient: build_initial_monmap 2024-04-22T08:22:43.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.286+0000 7f04ea469640 10 monclient: monmap: 2024-04-22T08:22:43.296 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:43.296 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:22:43.296 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:22:43.296 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:22:43.296 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:43.296 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:43.297 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:22:43.297 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:22:43.297 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:22:43.297 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:43.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.286+0000 7f04ea469640 1 Processor -- start 2024-04-22T08:22:43.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.287+0000 7f04ea469640 1 -- start start 2024-04-22T08:22:43.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.287+0000 7f04ea469640 10 monclient: init 2024-04-22T08:22:43.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.287+0000 7f04ea469640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:43.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.287+0000 7f04ea469640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-22T08:22:43.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.287+0000 7f04ea469640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04e40a0140 0x7f04e41ca8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:43.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.287+0000 7f04ea469640 10 monclient(hunting): picked mon.noname-b con 0x7f04e40a0140 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:22:43.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.287+0000 7f04ea469640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f04e40a1a80 0x7f04e41b9980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:43.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.287+0000 7f04ea469640 10 monclient(hunting): picked mon.noname-c con 0x7f04e40a1a80 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:22:43.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.287+0000 7f04ea469640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04e40a23a0 0x7f04e41b9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:43.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.287+0000 7f04ea469640 10 monclient(hunting): picked mon.noname-a con 0x7f04e40a23a0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:22:43.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.287+0000 7f04ea469640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:43.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.287+0000 7f04ea469640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04e41cadf0 con 0x7f04e40a23a0 2024-04-22T08:22:43.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.287+0000 7f04ea469640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:43.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.287+0000 7f04ea469640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04e409cbd0 con 0x7f04e40a0140 2024-04-22T08:22:43.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.287+0000 7f04ea469640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:43.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.287+0000 7f04ea469640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04e41a6aa0 con 0x7f04e40a1a80 2024-04-22T08:22:43.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.287+0000 7f04ea469640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:43.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.287+0000 7f04ea469640 10 monclient(hunting): authenticate will time out at 1437.281374s 2024-04-22T08:22:43.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.287+0000 7f04e37fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f04e40a1a80 0x7f04e41b9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:43.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.287+0000 7f04e37fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f04e40a1a80 0x7f04e41b9980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49810/0 (socket says 172.21.15.40:49810) 2024-04-22T08:22:43.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.287+0000 7f04e37fe640 1 -- 172.21.15.40:0/2524269038 learned_addr learned my addr 172.21.15.40:0/2524269038 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:22:43.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.287+0000 7f04e37fe640 10 monclient(hunting): get_auth_request con 0x7f04e40a1a80 auth_method 0 2024-04-22T08:22:43.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.287+0000 7f04e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:43.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.287+0000 7f04e8a06640 1 --2- 172.21.15.40:0/2524269038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04e40a23a0 0x7f04e41b9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:43.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.287+0000 7f04e37fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:43.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.287+0000 7f04e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:43.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.287+0000 7f04e8a06640 10 monclient(hunting): get_auth_request con 0x7f04e40a23a0 auth_method 0 2024-04-22T08:22:43.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.287+0000 7f04e8a06640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:43.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.287+0000 7f04e8a06640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:43.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.287+0000 7f04e8a06640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:43.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.287+0000 7f04e37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:43.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.287+0000 7f04e37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:43.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.287+0000 7f04e37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:43.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.287+0000 7f04e8a06640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:43.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.287+0000 7f04e8a06640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:43.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.287+0000 7f04e8a06640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:43.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.287+0000 7f04e3fff640 1 --2- 172.21.15.40:0/2524269038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04e40a0140 0x7f04e41ca8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:43.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.288+0000 7f04e8a06640 10 monclient(hunting): handle_auth_done global_id 114303 payload 824 2024-04-22T08:22:43.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.288+0000 7f04e8a06640 1 -- 172.21.15.40:0/2524269038 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f04e40a1a80 msgr2=0x7f04e41b9980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:43.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.288+0000 7f04e8a06640 1 --2- 172.21.15.40:0/2524269038 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f04e40a1a80 0x7f04e41b9980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:43.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.288+0000 7f04e8a06640 1 -- 172.21.15.40:0/2524269038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04e40a0140 msgr2=0x7f04e41ca8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:43.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.288+0000 7f04e8a06640 1 --2- 172.21.15.40:0/2524269038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04e40a0140 0x7f04e41ca8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:43.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.288+0000 7f04e8a06640 10 monclient: _finish_hunting 0 2024-04-22T08:22:43.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.288+0000 7f04e8a06640 1 monclient: found mon.noname-a 2024-04-22T08:22:43.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.288+0000 7f04e8a06640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:43.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.288+0000 7f04e8a06640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:22:43.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.288+0000 7f04e8a06640 1 -- 172.21.15.40:0/2524269038 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04d4009400 con 0x7f04e40a23a0 2024-04-22T08:22:43.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.288+0000 7f04e37fe640 1 --2- 172.21.15.40:0/2524269038 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f04e40a1a80 0x7f04e41b9980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:22:43.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.290+0000 7f04e8a06640 1 -- 172.21.15.40:0/2524269038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04e40a23a0 msgr2=0x7f04e41b9ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:43.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.290+0000 7f04e8a06640 1 --2- 172.21.15.40:0/2524269038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04e40a23a0 0x7f04e41b9ec0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f04dc00ccb0 tx=0x7f04dc00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:43.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.290+0000 7f04e0ff9640 1 -- 172.21.15.40:0/2524269038 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04dc003b90 con 0x7f04e40a23a0 2024-04-22T08:22:43.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.290+0000 7f04e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:43.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.290+0000 7f04e0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:22:43.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.290+0000 7f04e0ff9640 10 monclient: dump: 2024-04-22T08:22:43.301 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:43.301 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:43.301 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:43.301 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:43.302 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:43.302 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:43.302 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:43.302 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:43.302 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:43.302 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:43.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.290+0000 7f04e0ff9640 10 monclient: _finish_auth 0 2024-04-22T08:22:43.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.290+0000 7f04e0ff9640 10 monclient: _check_auth_tickets 2024-04-22T08:22:43.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.290+0000 7f04e0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:43.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.290+0000 7f04e0ff9640 1 -- 172.21.15.40:0/2524269038 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04dc003cf0 con 0x7f04e40a23a0 2024-04-22T08:22:43.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.290+0000 7f04e0ff9640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:43.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.290+0000 7f04e0ff9640 1 -- 172.21.15.40:0/2524269038 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04dc004240 con 0x7f04e40a23a0 2024-04-22T08:22:43.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.290+0000 7f04ea469640 5 monclient: authenticate success, global_id 114303 2024-04-22T08:22:43.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.290+0000 7f04e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:43.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.290+0000 7f04e0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:22:43.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.290+0000 7f04e0ff9640 10 monclient: dump: 2024-04-22T08:22:43.303 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:43.303 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:43.303 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:43.303 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:43.303 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:43.303 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:43.303 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:43.303 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:43.303 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:43.303 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:43.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.290+0000 7f04ea469640 10 monclient: _renew_subs 2024-04-22T08:22:43.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.290+0000 7f04ea469640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:43.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.290+0000 7f04ea469640 1 -- 172.21.15.40:0/2524269038 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04e41a6d80 con 0x7f04e40a23a0 2024-04-22T08:22:43.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.290+0000 7f04e27fc640 4 set_mon_vals no callback set 2024-04-22T08:22:43.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.290+0000 7f04ea469640 10 monclient: _renew_subs 2024-04-22T08:22:43.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.290+0000 7f04ea469640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:43.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.290+0000 7f04ea469640 1 -- 172.21.15.40:0/2524269038 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04e40713f0 con 0x7f04e40a23a0 2024-04-22T08:22:43.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.291+0000 7f04e0ff9640 1 -- 172.21.15.40:0/2524269038 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f04dc004420 con 0x7f04e40a23a0 2024-04-22T08:22:43.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.291+0000 7f04ea469640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:43.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.291+0000 7f04ea469640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:43.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.291+0000 7f04ea469640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:43.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.291+0000 7f04ea469640 1 -- 172.21.15.40:0/2524269038 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04e404e9b0 con 0x7f04e40a23a0 2024-04-22T08:22:43.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.291+0000 7f04e0ff9640 1 --2- 172.21.15.40:0/2524269038 >> v2:172.21.15.189:6800/30168 conn(0x7f04d00377d0 0x7f04d0039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:43.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.291+0000 7f04e0ff9640 1 -- 172.21.15.40:0/2524269038 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(317..317 src has 239..317) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f04dc012070 con 0x7f04e40a23a0 2024-04-22T08:22:43.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.291+0000 7f04e3fff640 1 --2- 172.21.15.40:0/2524269038 >> v2:172.21.15.189:6800/30168 conn(0x7f04d00377d0 0x7f04d0039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:43.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.291+0000 7f04e3fff640 10 monclient: get_auth_request con 0x7f04d00377d0 auth_method 0 2024-04-22T08:22:43.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.294+0000 7f04e3fff640 1 --2- 172.21.15.40:0/2524269038 >> v2:172.21.15.189:6800/30168 conn(0x7f04d00377d0 0x7f04d0039c90 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f04e40a7070 tx=0x7f04d402f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:43.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.296+0000 7f04e0ff9640 1 -- 172.21.15.40:0/2524269038 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f04dc003e60 con 0x7f04e40a23a0 2024-04-22T08:22:43.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.296+0000 7f04e0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:43.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.296+0000 7f04e0ff9640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:22:43.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.477+0000 7f04ea469640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-22T08:22:43.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.477+0000 7f04ea469640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-22T08:22:43.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.477+0000 7f04ea469640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:43.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.477+0000 7f04ea469640 1 -- 172.21.15.40:0/2524269038 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04e41a72a0 con 0x7f04e40a23a0 2024-04-22T08:22:43.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.478+0000 7f04e0ff9640 1 -- 172.21.15.40:0/2524269038 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v317) v1 ==== 74+0+10509 (secure 0 0 0) 0x7f04dc026920 con 0x7f04e40a23a0 2024-04-22T08:22:43.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.478+0000 7f04e0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-22T08:22:43.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.478+0000 7f04e0ff9640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:22:43.478 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:22:43.478 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":317,"fsid":"987d842f-7205-47b9-86e0-2e2623f3c370","created":"2024-04-22T08:12:36.003043+0000","modified":"2024-04-22T08:22:17.168230+0000","last_up_change":"2024-04-22T08:12:47.016043+0000","last_in_change":"2024-04-22T08:12:40.364359+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":100,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T08:12:48.855046+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-22T08:12:48.986472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b9a0210c-5881-4ca0-8318-a4a987109405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":314,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":30669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6805","nonce":30669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6807","nonce":30669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":30669}]},"public_addr":"172.21.15.40:6804/30669","cluster_addr":"172.21.15.40:6805/30669","heartbeat_back_addr":"172.21.15.40:6807/30669","heartbeat_front_addr":"172.21.15.40:6806/30669","state":["exists","up"]},{"osd":1,"uuid":"11370cc8-09a6-4c35-b92c-4d926c6c18d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":314,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":30670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6801","nonce":30670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6803","nonce":30670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":30670}]},"public_addr":"172.21.15.40:6800/30670","cluster_addr":"172.21.15.40:6801/30670","heartbeat_back_addr":"172.21.15.40:6803/30670","heartbeat_front_addr":"172.21.15.40:6802/30670","state":["exists","up"]},{"osd":2,"uuid":"d03db34e-f630-4a3e-86fb-7234bee0d0f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":314,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":30671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":30671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6811","nonce":30671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":30671}]},"public_addr":"172.21.15.40:6808/30671","cluster_addr":"172.21.15.40:6809/30671","heartbeat_back_addr":"172.21.15.40:6811/30671","heartbeat_front_addr":"172.21.15.40:6810/30671","state":["exists","up"]},{"osd":3,"uuid":"d94b2052-5847-4955-b1d4-b7e627572811","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":314,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6805","nonce":30673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6807","nonce":30673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6811","nonce":30673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6809","nonce":30673}]},"public_addr":"172.21.15.189:6805/30673","cluster_addr":"172.21.15.189:6807/30673","heartbeat_back_addr":"172.21.15.189:6811/30673","heartbeat_front_addr":"172.21.15.189:6809/30673","state":["exists","up"]},{"osd":4,"uuid":"cb626202-7411-4299-8d20-988bcc9103f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":314,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6801","nonce":30672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":30672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":30672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6803","nonce":30672}]},"public_addr":"172.21.15.189:6801/30672","cluster_addr":"172.21.15.189:6802/30672","heartbeat_back_addr":"172.21.15.189:6804/30672","heartbeat_front_addr":"172.21.15.189:6803/30672","state":["exists","up"]},{"osd":5,"uuid":"abbac246-54d6-47d7-a41c-a68df0b80ddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":314,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":30674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":30674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":30674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":30674}]},"public_addr":"172.21.15.189:6806/30674","cluster_addr":"172.21.15.189:6808/30674","heartbeat_back_addr":"172.21.15.189:6812/30674","heartbeat_front_addr":"172.21.15.189:6810/30674","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T08:12:45.488082+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T08:12:45.467994+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T08:12:45.503580+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T08:12:45.488169+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T08:12:45.465787+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T08:12:45.470729+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T08:22:43.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.480+0000 7f04ea469640 1 -- 172.21.15.40:0/2524269038 >> v2:172.21.15.189:6800/30168 conn(0x7f04d00377d0 msgr2=0x7f04d0039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:43.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.480+0000 7f04ea469640 1 --2- 172.21.15.40:0/2524269038 >> v2:172.21.15.189:6800/30168 conn(0x7f04d00377d0 0x7f04d0039c90 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f04e40a7070 tx=0x7f04d402f000 comp rx=0 tx=0).stop 2024-04-22T08:22:43.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.480+0000 7f04ea469640 10 monclient: shutdown 2024-04-22T08:22:43.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.480+0000 7f04ea469640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:43.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.480+0000 7f04ea469640 1 -- 172.21.15.40:0/2524269038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04e40a23a0 msgr2=0x7f04e41b9ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:43.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.480+0000 7f04ea469640 1 --2- 172.21.15.40:0/2524269038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04e40a23a0 0x7f04e41b9ec0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f04dc00ccb0 tx=0x7f04dc00a660 comp rx=0 tx=0).stop 2024-04-22T08:22:43.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.480+0000 7f04ea469640 1 -- 172.21.15.40:0/2524269038 shutdown_connections 2024-04-22T08:22:43.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.480+0000 7f04ea469640 1 --2- 172.21.15.40:0/2524269038 >> v2:172.21.15.189:6800/30168 conn(0x7f04d00377d0 0x7f04d0039c90 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:43.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.480+0000 7f04ea469640 1 --2- 172.21.15.40:0/2524269038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04e40a23a0 0x7f04e41b9ec0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:43.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.480+0000 7f04ea469640 1 --2- 172.21.15.40:0/2524269038 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f04e40a1a80 0x7f04e41b9980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:43.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.480+0000 7f04ea469640 1 --2- 172.21.15.40:0/2524269038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04e40a0140 0x7f04e41ca8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:43.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.480+0000 7f04ea469640 1 -- 172.21.15.40:0/2524269038 >> 172.21.15.40:0/2524269038 conn(0x7f04e40a66a0 msgr2=0x7f04e40a5690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:43.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.480+0000 7f04ea469640 1 -- 172.21.15.40:0/2524269038 shutdown_connections 2024-04-22T08:22:43.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:43.480+0000 7f04ea469640 1 -- 172.21.15.40:0/2524269038 wait complete. 2024-04-22T08:22:44.490 INFO:tasks.ceph:Scrubbing osd.0 2024-04-22T08:22:44.491 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-22T08:22:44.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.607+0000 7f16bddfd640 10 monclient: get_monmap_and_config 2024-04-22T08:22:44.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.607+0000 7f16bddfd640 10 monclient: build_initial_monmap 2024-04-22T08:22:44.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.607+0000 7f16bddfd640 10 monclient: monmap: 2024-04-22T08:22:44.607 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:44.607 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:44.607 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:22:44.608287+0000 2024-04-22T08:22:44.607 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:22:44.608287+0000 2024-04-22T08:22:44.607 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:44.607 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:44.608 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:22:44.608 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:22:44.608 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:22:44.608 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:22:44.608 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:22:44.608 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:44.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.608+0000 7f16bddfd640 1 Processor -- start 2024-04-22T08:22:44.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.608+0000 7f16bddfd640 1 -- start start 2024-04-22T08:22:44.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.608+0000 7f16bddfd640 10 monclient: init 2024-04-22T08:22:44.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.608+0000 7f16bddfd640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:44.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.608+0000 7f16bddfd640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-04-22T08:22:44.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.608+0000 7f16bddfd640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f16b80a1c20 0x7f16b80a2000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:44.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.608+0000 7f16bddfd640 10 monclient(hunting): picked mon.noname-a con 0x7f16b80a1c20 addr v2:172.21.15.40:3300/0 2024-04-22T08:22:44.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.608+0000 7f16bddfd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16b80999f0 0x7f16b80a2540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:44.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.608+0000 7f16bddfd640 10 monclient(hunting): picked mon.noname-b con 0x7f16b80999f0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:22:44.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.608+0000 7f16bddfd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f16b8135dc0 addr v1:172.21.15.189:6789/0 2024-04-22T08:22:44.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.608+0000 7f16bddfd640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b80a2a80 con 0x7f16b8135dc0 2024-04-22T08:22:44.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.608+0000 7f16bddfd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:44.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.608+0000 7f16bddfd640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16b80720e0 con 0x7f16b80a1c20 2024-04-22T08:22:44.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.608+0000 7f16bddfd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:44.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.608+0000 7f16bddfd640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16b809f310 con 0x7f16b80999f0 2024-04-22T08:22:44.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.608+0000 7f16bddfd640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:44.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.608+0000 7f16bddfd640 10 monclient(hunting): authenticate will time out at 1438.602566s 2024-04-22T08:22:44.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.608+0000 7f16b6ffd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16b80999f0 0x7f16b80a2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:44.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.608+0000 7f16b6ffd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16b80999f0 0x7f16b80a2540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49824/0 (socket says 172.21.15.40:49824) 2024-04-22T08:22:44.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.608+0000 7f16b77fe640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f16b80a1c20 0x7f16b80a2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:44.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.608+0000 7f16b6ffd640 1 -- 172.21.15.40:0/341345813 learned_addr learned my addr 172.21.15.40:0/341345813 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:22:44.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.608+0000 7f16b6ffd640 10 monclient(hunting): get_auth_request con 0x7f16b80999f0 auth_method 0 2024-04-22T08:22:44.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.608+0000 7f16b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:44.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.608+0000 7f16b6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:44.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.608+0000 7f16b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:44.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.608+0000 7f16b77fe640 10 monclient(hunting): get_auth_request con 0x7f16b80a1c20 auth_method 0 2024-04-22T08:22:44.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.608+0000 7f16b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:44.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.608+0000 7f16b77fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:44.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.608+0000 7f16b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:44.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.609+0000 7f16b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:44.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.609+0000 7f16b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:44.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.609+0000 7f16b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:44.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.609+0000 7f16b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:44.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.609+0000 7f16b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:44.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.609+0000 7f16b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:44.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.609+0000 7f16b6ffd640 10 monclient(hunting): handle_auth_done global_id 124139 payload 274 2024-04-22T08:22:44.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.609+0000 7f16b6ffd640 1 -- 172.21.15.40:0/341345813 >> v2:172.21.15.40:3300/0 conn(0x7f16b80a1c20 msgr2=0x7f16b80a2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:44.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.609+0000 7f16b6ffd640 1 --2- 172.21.15.40:0/341345813 >> v2:172.21.15.40:3300/0 conn(0x7f16b80a1c20 0x7f16b80a2000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:44.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.609+0000 7f16b6ffd640 1 -- 172.21.15.40:0/341345813 >> v1:172.21.15.189:6789/0 conn(0x7f16b8135dc0 legacy=0x7f16b8136270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:44.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.609+0000 7f16b6ffd640 10 monclient: _finish_hunting 0 2024-04-22T08:22:44.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.609+0000 7f16b6ffd640 1 monclient: found mon.noname-b 2024-04-22T08:22:44.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.609+0000 7f16b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:44.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.609+0000 7f16b6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:22:44.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.609+0000 7f16b6ffd640 1 -- 172.21.15.40:0/341345813 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b80a3690 con 0x7f16b80999f0 2024-04-22T08:22:44.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.609+0000 7f16b77fe640 1 --2- 172.21.15.40:0/341345813 >> v2:172.21.15.40:3300/0 conn(0x7f16b80a1c20 0x7f16b80a2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:22:44.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.612+0000 7f16b6ffd640 1 -- 172.21.15.40:0/341345813 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16b80999f0 msgr2=0x7f16b80a2540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:44.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.612+0000 7f16b6ffd640 1 --2- 172.21.15.40:0/341345813 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16b80999f0 0x7f16b80a2540 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f16ac009870 tx=0x7f16ac02b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cec376b34a418ef7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:44.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.612+0000 7f16b67fc640 1 -- 172.21.15.40:0/341345813 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16ac031950 con 0x7f16b80999f0 2024-04-22T08:22:44.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.612+0000 7f16b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:44.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.612+0000 7f16b67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:22:44.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.612+0000 7f16b67fc640 10 monclient: dump: 2024-04-22T08:22:44.621 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:44.621 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:44.621 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:44.621 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:44.621 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:44.621 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:44.621 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:44.622 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:44.622 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:44.622 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:44.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.612+0000 7f16b67fc640 10 monclient: _finish_auth 0 2024-04-22T08:22:44.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.612+0000 7f16b67fc640 10 monclient: _check_auth_tickets 2024-04-22T08:22:44.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.612+0000 7f16b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:44.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.612+0000 7f16bddfd640 5 monclient: authenticate success, global_id 124139 2024-04-22T08:22:44.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.612+0000 7f16b67fc640 1 -- 172.21.15.40:0/341345813 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16ac031ab0 con 0x7f16b80999f0 2024-04-22T08:22:44.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.612+0000 7f16bddfd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:22:44.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.612+0000 7f16b67fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:44.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.612+0000 7f16bddfd640 10 monclient: get_monmap_and_config success 2024-04-22T08:22:44.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.612+0000 7f16bddfd640 4 set_mon_vals no callback set 2024-04-22T08:22:44.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.612+0000 7f16b67fc640 1 -- 172.21.15.40:0/341345813 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16ac031e40 con 0x7f16b80999f0 2024-04-22T08:22:44.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.612+0000 7f16b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:44.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.612+0000 7f16b67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:22:44.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.612+0000 7f16b67fc640 10 monclient: dump: 2024-04-22T08:22:44.623 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:44.623 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:44.623 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:44.623 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:44.623 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:44.623 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:44.623 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:44.623 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:44.624 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:44.624 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:44.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.613+0000 7f16bddfd640 10 monclient: shutdown 2024-04-22T08:22:44.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.613+0000 7f16bddfd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:44.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.613+0000 7f16bddfd640 1 -- 172.21.15.40:0/341345813 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16b80999f0 msgr2=0x7f16b80a2540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:44.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.613+0000 7f16bddfd640 1 --2- 172.21.15.40:0/341345813 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16b80999f0 0x7f16b80a2540 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f16ac009870 tx=0x7f16ac02b7e0 comp rx=0 tx=0).stop 2024-04-22T08:22:44.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.613+0000 7f16bddfd640 1 -- 172.21.15.40:0/341345813 shutdown_connections 2024-04-22T08:22:44.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.613+0000 7f16bddfd640 1 --2- 172.21.15.40:0/341345813 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16b80999f0 0x7f16b80a2540 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:44.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.613+0000 7f16bddfd640 1 --2- 172.21.15.40:0/341345813 >> v2:172.21.15.40:3300/0 conn(0x7f16b80a1c20 0x7f16b80a2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:44.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.613+0000 7f16bddfd640 1 -- 172.21.15.40:0/341345813 >> 172.21.15.40:0/341345813 conn(0x7f16b80a6840 msgr2=0x7f16b80a2d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:44.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.613+0000 7f16bddfd640 1 -- 172.21.15.40:0/341345813 shutdown_connections 2024-04-22T08:22:44.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.613+0000 7f16bddfd640 1 -- 172.21.15.40:0/341345813 wait complete. 2024-04-22T08:22:44.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.613+0000 7f16bddfd640 10 monclient: build_initial_monmap 2024-04-22T08:22:44.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.613+0000 7f16bddfd640 10 monclient: monmap: 2024-04-22T08:22:44.625 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:44.625 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:22:44.625 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:22:44.625 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:22:44.625 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:44.625 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:44.625 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:22:44.625 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:22:44.625 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:22:44.625 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:44.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.614+0000 7f16bddfd640 1 Processor -- start 2024-04-22T08:22:44.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.614+0000 7f16bddfd640 1 -- start start 2024-04-22T08:22:44.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.614+0000 7f16bddfd640 10 monclient: init 2024-04-22T08:22:44.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.614+0000 7f16bddfd640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:44.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.614+0000 7f16bddfd640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-22T08:22:44.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.614+0000 7f16bddfd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16b80999f0 0x7f16b81c72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:44.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.614+0000 7f16bddfd640 10 monclient(hunting): picked mon.noname-a con 0x7f16b80999f0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:22:44.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.614+0000 7f16bddfd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16b80a1c20 0x7f16b81344b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:44.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.614+0000 7f16bddfd640 10 monclient(hunting): picked mon.noname-c con 0x7f16b80a1c20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:22:44.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.614+0000 7f16bddfd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16b8135dc0 0x7f16b81ca810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:44.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.614+0000 7f16bddfd640 10 monclient(hunting): picked mon.noname-b con 0x7f16b8135dc0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:22:44.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.614+0000 7f16bddfd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:44.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.614+0000 7f16bddfd640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b80703f0 con 0x7f16b80999f0 2024-04-22T08:22:44.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.614+0000 7f16bddfd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:44.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.614+0000 7f16bddfd640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b8121ee0 con 0x7f16b8135dc0 2024-04-22T08:22:44.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.614+0000 7f16bddfd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:44.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.614+0000 7f16bddfd640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16b81c7d50 con 0x7f16b80a1c20 2024-04-22T08:22:44.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.614+0000 7f16bddfd640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:44.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.614+0000 7f16bddfd640 10 monclient(hunting): authenticate will time out at 1438.608693s 2024-04-22T08:22:44.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.614+0000 7f16b77fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16b80999f0 0x7f16b81c72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:44.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.615+0000 7f16b77fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16b80999f0 0x7f16b81c72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44560/0 (socket says 172.21.15.40:44560) 2024-04-22T08:22:44.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.615+0000 7f16b6ffd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16b80a1c20 0x7f16b81344b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:44.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.615+0000 7f16b77fe640 1 -- 172.21.15.40:0/453854919 learned_addr learned my addr 172.21.15.40:0/453854919 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:22:44.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.615+0000 7f16b77fe640 10 monclient(hunting): get_auth_request con 0x7f16b80999f0 auth_method 0 2024-04-22T08:22:44.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.615+0000 7f16b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:44.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.615+0000 7f16b77fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:44.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.615+0000 7f16b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:44.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.615+0000 7f16b6ffd640 10 monclient(hunting): get_auth_request con 0x7f16b80a1c20 auth_method 0 2024-04-22T08:22:44.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.615+0000 7f16b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:44.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.615+0000 7f16b6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:44.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.615+0000 7f16b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:44.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.615+0000 7f16b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:44.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.615+0000 7f16b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:44.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.615+0000 7f16b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:44.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.615+0000 7f16b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:44.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.615+0000 7f16b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:44.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.615+0000 7f16b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:44.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.615+0000 7f16b6ffd640 10 monclient(hunting): handle_auth_done global_id 124142 payload 824 2024-04-22T08:22:44.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.615+0000 7f16b6ffd640 1 -- 172.21.15.40:0/453854919 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16b8135dc0 msgr2=0x7f16b81ca810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:44.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.615+0000 7f16b6ffd640 1 --2- 172.21.15.40:0/453854919 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16b8135dc0 0x7f16b81ca810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:44.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.615+0000 7f16b6ffd640 1 -- 172.21.15.40:0/453854919 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16b80999f0 msgr2=0x7f16b81c72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:44.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.615+0000 7f16b6ffd640 1 --2- 172.21.15.40:0/453854919 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16b80999f0 0x7f16b81c72f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:44.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.615+0000 7f16b6ffd640 10 monclient: _finish_hunting 0 2024-04-22T08:22:44.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.615+0000 7f16b6ffd640 1 monclient: found mon.noname-c 2024-04-22T08:22:44.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.615+0000 7f16b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:44.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.615+0000 7f16b6ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:22:44.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.615+0000 7f16b6ffd640 1 -- 172.21.15.40:0/453854919 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a8009400 con 0x7f16b80a1c20 2024-04-22T08:22:44.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.615+0000 7f16b77fe640 1 --2- 172.21.15.40:0/453854919 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16b80999f0 0x7f16b81c72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:22:44.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.617+0000 7f16b6ffd640 1 -- 172.21.15.40:0/453854919 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16b80a1c20 msgr2=0x7f16b81344b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:44.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.617+0000 7f16b6ffd640 1 --2- 172.21.15.40:0/453854919 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16b80a1c20 0x7f16b81344b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f16ac033000 tx=0x7f16ac0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:44.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.617+0000 7f169ffff640 1 -- 172.21.15.40:0/453854919 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16ac033c30 con 0x7f16b80a1c20 2024-04-22T08:22:44.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.617+0000 7f169ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:44.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.617+0000 7f169ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:22:44.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.617+0000 7f169ffff640 10 monclient: dump: 2024-04-22T08:22:44.630 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:44.630 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:44.630 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:44.630 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:44.630 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:44.630 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:44.631 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:44.631 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:44.631 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:44.631 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:44.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.617+0000 7f169ffff640 10 monclient: _finish_auth 0 2024-04-22T08:22:44.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.617+0000 7f169ffff640 10 monclient: _check_auth_tickets 2024-04-22T08:22:44.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.617+0000 7f169ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:44.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.617+0000 7f169ffff640 1 -- 172.21.15.40:0/453854919 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16ac033db0 con 0x7f16b80a1c20 2024-04-22T08:22:44.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.617+0000 7f169ffff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:44.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.617+0000 7f169ffff640 1 -- 172.21.15.40:0/453854919 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16ac03d6b0 con 0x7f16b80a1c20 2024-04-22T08:22:44.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.617+0000 7f16bddfd640 5 monclient: authenticate success, global_id 124142 2024-04-22T08:22:44.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.617+0000 7f16bddfd640 10 monclient: _renew_subs 2024-04-22T08:22:44.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.617+0000 7f16bddfd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:22:44.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.617+0000 7f16bddfd640 1 -- 172.21.15.40:0/453854919 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16ac0094d0 con 0x7f16b80a1c20 2024-04-22T08:22:44.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.617+0000 7f169ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:44.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.617+0000 7f169ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:22:44.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.617+0000 7f16b57fa640 4 set_mon_vals no callback set 2024-04-22T08:22:44.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.617+0000 7f169ffff640 10 monclient: dump: 2024-04-22T08:22:44.632 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:44.632 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:44.632 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:44.633 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:44.633 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:44.633 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:44.633 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:44.633 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:44.633 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:44.633 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:44.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.617+0000 7f16bddfd640 10 monclient: _renew_subs 2024-04-22T08:22:44.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.617+0000 7f16bddfd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:22:44.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.617+0000 7f16bddfd640 1 -- 172.21.15.40:0/453854919 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16b8071e10 con 0x7f16b80a1c20 2024-04-22T08:22:44.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.618+0000 7f169ffff640 1 -- 172.21.15.40:0/453854919 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f16ac03a040 con 0x7f16b80a1c20 2024-04-22T08:22:44.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.618+0000 7f16bddfd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:22:44.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.618+0000 7f16bddfd640 1 -- 172.21.15.40:0/453854919 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f16b8136980 con 0x7f16b80a1c20 2024-04-22T08:22:44.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.618+0000 7f169ffff640 1 --2- 172.21.15.40:0/453854919 >> v2:172.21.15.189:6800/30168 conn(0x7f16a40377d0 0x7f16a4039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:44.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.618+0000 7f169ffff640 1 -- 172.21.15.40:0/453854919 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(317..317 src has 239..317) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f16ac06b8f0 con 0x7f16b80a1c20 2024-04-22T08:22:44.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.619+0000 7f169ffff640 1 --2- 172.21.15.40:0/453854919 >> v2:172.21.15.40:6804/30669 conn(0x7f16a403d280 0x7f16a403f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:44.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.619+0000 7f169ffff640 1 -- 172.21.15.40:0/453854919 --> v2:172.21.15.40:6804/30669 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f16a403fd10 con 0x7f16a403d280 2024-04-22T08:22:44.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.619+0000 7f169ffff640 1 -- 172.21.15.40:0/453854919 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_get_version_reply(handle=1 version=317) v2 ==== 24+0+0 (secure 0 0 0) 0x7f16ac070200 con 0x7f16b80a1c20 2024-04-22T08:22:44.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.619+0000 7f169ffff640 10 monclient: handle_get_version_reply finishing 1 version 317 2024-04-22T08:22:44.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.619+0000 7f16b7fff640 1 --2- 172.21.15.40:0/453854919 >> v2:172.21.15.40:6804/30669 conn(0x7f16a403d280 0x7f16a403f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:44.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.619+0000 7f16b7fff640 10 monclient: get_auth_request con 0x7f16a403d280 auth_method 0 2024-04-22T08:22:44.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.619+0000 7f16b77fe640 1 --2- 172.21.15.40:0/453854919 >> v2:172.21.15.189:6800/30168 conn(0x7f16a40377d0 0x7f16a4039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:44.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.619+0000 7f16b77fe640 10 monclient: get_auth_request con 0x7f16a40377d0 auth_method 0 2024-04-22T08:22:44.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.622+0000 7f16b77fe640 1 --2- 172.21.15.40:0/453854919 >> v2:172.21.15.189:6800/30168 conn(0x7f16a40377d0 0x7f16a4039c90 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f16a8009fd0 tx=0x7f16a8009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:44.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.623+0000 7f16b7fff640 1 --2- 172.21.15.40:0/453854919 >> v2:172.21.15.40:6804/30669 conn(0x7f16a403d280 0x7f16a403f6a0 crc :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:44.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.624+0000 7f169ffff640 1 -- 172.21.15.40:0/453854919 <== osd.0 v2:172.21.15.40:6804/30669 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f16a403fd10 con 0x7f16a403d280 2024-04-22T08:22:44.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.637+0000 7f16bddfd640 1 -- 172.21.15.40:0/453854919 --> v2:172.21.15.40:6804/30669 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f16b8075260 con 0x7f16a403d280 2024-04-22T08:22:44.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.637+0000 7f169ffff640 1 -- 172.21.15.40:0/453854919 <== osd.0 v2:172.21.15.40:6804/30669 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f16b8075260 con 0x7f16a403d280 2024-04-22T08:22:44.636 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:22:44.637 INFO:teuthology.orchestra.run.smithi040.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-22T08:22:44.637 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:22:44.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.637+0000 7f16bddfd640 1 -- 172.21.15.40:0/453854919 >> v2:172.21.15.40:6804/30669 conn(0x7f16a403d280 msgr2=0x7f16a403f6a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:44.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.637+0000 7f16bddfd640 1 --2- 172.21.15.40:0/453854919 >> v2:172.21.15.40:6804/30669 conn(0x7f16a403d280 0x7f16a403f6a0 crc :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:44.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.637+0000 7f16bddfd640 1 -- 172.21.15.40:0/453854919 >> v2:172.21.15.189:6800/30168 conn(0x7f16a40377d0 msgr2=0x7f16a4039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:44.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.637+0000 7f16bddfd640 1 --2- 172.21.15.40:0/453854919 >> v2:172.21.15.189:6800/30168 conn(0x7f16a40377d0 0x7f16a4039c90 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f16a8009fd0 tx=0x7f16a8009190 comp rx=0 tx=0).stop 2024-04-22T08:22:44.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.637+0000 7f16bddfd640 10 monclient: shutdown 2024-04-22T08:22:44.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.637+0000 7f16bddfd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:44.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.637+0000 7f16bddfd640 1 -- 172.21.15.40:0/453854919 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16b80a1c20 msgr2=0x7f16b81344b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:44.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.637+0000 7f16bddfd640 1 --2- 172.21.15.40:0/453854919 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16b80a1c20 0x7f16b81344b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f16ac033000 tx=0x7f16ac0046c0 comp rx=0 tx=0).stop 2024-04-22T08:22:44.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.637+0000 7f16b7fff640 1 -- 172.21.15.40:0/453854919 reap_dead start 2024-04-22T08:22:44.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.637+0000 7f16bddfd640 1 -- 172.21.15.40:0/453854919 shutdown_connections 2024-04-22T08:22:44.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.637+0000 7f16bddfd640 1 -- 172.21.15.40:0/453854919 >> 172.21.15.40:0/453854919 conn(0x7f16b80a6840 msgr2=0x7f16b80a40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:44.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.637+0000 7f16bddfd640 1 -- 172.21.15.40:0/453854919 shutdown_connections 2024-04-22T08:22:44.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.637+0000 7f16bddfd640 1 -- 172.21.15.40:0/453854919 wait complete. 2024-04-22T08:22:44.647 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-22T08:22:44.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.748+0000 7f4167e21640 10 monclient: get_monmap_and_config 2024-04-22T08:22:44.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.748+0000 7f4167e21640 10 monclient: build_initial_monmap 2024-04-22T08:22:44.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.748+0000 7f4167e21640 10 monclient: monmap: 2024-04-22T08:22:44.748 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:44.748 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:44.749 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:22:44.748806+0000 2024-04-22T08:22:44.749 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:22:44.748806+0000 2024-04-22T08:22:44.749 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:44.749 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:44.749 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:22:44.749 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:22:44.749 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:22:44.749 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:22:44.749 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:22:44.749 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:44.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.748+0000 7f4167e21640 1 Processor -- start 2024-04-22T08:22:44.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.748+0000 7f4167e21640 1 -- start start 2024-04-22T08:22:44.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.748+0000 7f4167e21640 10 monclient: init 2024-04-22T08:22:44.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.749+0000 7f4167e21640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:44.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.749+0000 7f4167e21640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-04-22T08:22:44.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.749+0000 7f4167e21640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4160135e30 addr v1:172.21.15.189:6789/0 2024-04-22T08:22:44.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.749+0000 7f4167e21640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4160071c90 addr v1:172.21.15.40:6789/0 2024-04-22T08:22:44.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.749+0000 7f4167e21640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f416013c5f0 0x7f416013e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:44.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.749+0000 7f4167e21640 10 monclient(hunting): picked mon.noname-c con 0x7f416013c5f0 addr v2:172.21.15.189:3300/0 2024-04-22T08:22:44.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.749+0000 7f4167e21640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f416009de80 con 0x7f4160071c90 2024-04-22T08:22:44.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.749+0000 7f4167e21640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f416009f130 con 0x7f4160135e30 2024-04-22T08:22:44.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.749+0000 7f4167e21640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:44.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.749+0000 7f4167e21640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f416009e3d0 con 0x7f416013c5f0 2024-04-22T08:22:44.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.749+0000 7f4167e21640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:44.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.749+0000 7f4167e21640 10 monclient(hunting): authenticate will time out at 1438.743042s 2024-04-22T08:22:44.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.749+0000 7f41653bc640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f4160071c90 0x7f416013aed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:49328/0 (socket says 172.21.15.40:49328) 2024-04-22T08:22:44.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.749+0000 7f41653bc640 1 -- 172.21.15.40:0/2855424439 learned_addr learned my addr 172.21.15.40:0/2855424439 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:22:44.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.749+0000 7f41663be640 1 --2- 172.21.15.40:0/2855424439 >> v2:172.21.15.189:3300/0 conn(0x7f416013c5f0 0x7f416013e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:44.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.749+0000 7f41663be640 10 monclient(hunting): get_auth_request con 0x7f416013c5f0 auth_method 0 2024-04-22T08:22:44.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.749+0000 7f41663be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:44.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.749+0000 7f41663be640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:44.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.749+0000 7f41663be640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:44.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.749+0000 7f41663be640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:44.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.749+0000 7f41663be640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:44.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.749+0000 7f41663be640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:44.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.749+0000 7f41663be640 10 monclient(hunting): handle_auth_done global_id 65272 payload 274 2024-04-22T08:22:44.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.749+0000 7f41663be640 1 -- 172.21.15.40:0/2855424439 >> v1:172.21.15.189:6789/0 conn(0x7f4160135e30 legacy=0x7f4160136210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:44.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.749+0000 7f41663be640 1 -- 172.21.15.40:0/2855424439 >> v1:172.21.15.40:6789/0 conn(0x7f4160071c90 legacy=0x7f416013aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:44.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.749+0000 7f41663be640 10 monclient: _finish_hunting 0 2024-04-22T08:22:44.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.749+0000 7f41663be640 1 monclient: found mon.noname-c 2024-04-22T08:22:44.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.749+0000 7f41663be640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:44.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.749+0000 7f41663be640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.189:3300/0 2024-04-22T08:22:44.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.749+0000 7f41663be640 1 -- 172.21.15.40:0/2855424439 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f416009f410 con 0x7f416013c5f0 2024-04-22T08:22:44.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.753+0000 7f41663be640 1 -- 172.21.15.40:0/2855424439 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f416013c5f0 msgr2=0x7f416013e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:44.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.753+0000 7f41663be640 1 --2- 172.21.15.40:0/2855424439 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f416013c5f0 0x7f416013e9e0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f4150009dc0 tx=0x7f415002bce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=170b2ea7e21bcdda server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:44.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.753+0000 7f4164bbb640 1 -- 172.21.15.40:0/2855424439 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f415002cd80 con 0x7f416013c5f0 2024-04-22T08:22:44.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.753+0000 7f4164bbb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:44.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.753+0000 7f4164bbb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:22:44.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.753+0000 7f4164bbb640 10 monclient: dump: 2024-04-22T08:22:44.763 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:44.763 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:44.764 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:44.764 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:44.764 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:44.764 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:44.764 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:44.764 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:44.764 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:44.764 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:44.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.753+0000 7f4164bbb640 10 monclient: _finish_auth 0 2024-04-22T08:22:44.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.753+0000 7f4164bbb640 10 monclient: _check_auth_tickets 2024-04-22T08:22:44.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.753+0000 7f4164bbb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:44.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.753+0000 7f4164bbb640 1 -- 172.21.15.40:0/2855424439 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4150034440 con 0x7f416013c5f0 2024-04-22T08:22:44.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.753+0000 7f4164bbb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:44.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.753+0000 7f4164bbb640 1 -- 172.21.15.40:0/2855424439 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41500347d0 con 0x7f416013c5f0 2024-04-22T08:22:44.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.753+0000 7f4167e21640 5 monclient: authenticate success, global_id 65272 2024-04-22T08:22:44.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.753+0000 7f4167e21640 10 monclient: get_monmap_and_config success 2024-04-22T08:22:44.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.753+0000 7f4167e21640 4 set_mon_vals no callback set 2024-04-22T08:22:44.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.753+0000 7f4164bbb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:44.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.753+0000 7f4164bbb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:22:44.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.753+0000 7f4164bbb640 10 monclient: dump: 2024-04-22T08:22:44.765 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:44.765 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:44.765 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:44.765 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:44.766 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:44.766 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:44.766 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:44.766 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:44.767 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:44.767 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:44.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.753+0000 7f4167e21640 10 monclient: shutdown 2024-04-22T08:22:44.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.753+0000 7f4167e21640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:44.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.753+0000 7f4167e21640 1 -- 172.21.15.40:0/2855424439 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f416013c5f0 msgr2=0x7f416013e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:44.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.753+0000 7f4167e21640 1 --2- 172.21.15.40:0/2855424439 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f416013c5f0 0x7f416013e9e0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f4150009dc0 tx=0x7f415002bce0 comp rx=0 tx=0).stop 2024-04-22T08:22:44.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.753+0000 7f4167e21640 1 -- 172.21.15.40:0/2855424439 shutdown_connections 2024-04-22T08:22:44.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.753+0000 7f4167e21640 1 --2- 172.21.15.40:0/2855424439 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f416013c5f0 0x7f416013e9e0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:44.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.753+0000 7f4167e21640 1 -- 172.21.15.40:0/2855424439 >> 172.21.15.40:0/2855424439 conn(0x7f41600a65c0 msgr2=0x7f41600a2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:44.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.753+0000 7f4167e21640 1 -- 172.21.15.40:0/2855424439 shutdown_connections 2024-04-22T08:22:44.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.753+0000 7f4167e21640 1 -- 172.21.15.40:0/2855424439 wait complete. 2024-04-22T08:22:44.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.754+0000 7f4167e21640 10 monclient: build_initial_monmap 2024-04-22T08:22:44.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.754+0000 7f4167e21640 10 monclient: monmap: 2024-04-22T08:22:44.768 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:44.768 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:22:44.768 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:22:44.768 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:22:44.768 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:44.768 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:44.768 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:22:44.768 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:22:44.768 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:22:44.768 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:44.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.754+0000 7f4167e21640 1 Processor -- start 2024-04-22T08:22:44.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.754+0000 7f4167e21640 1 -- start start 2024-04-22T08:22:44.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.754+0000 7f4167e21640 10 monclient: init 2024-04-22T08:22:44.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f4167e21640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:44.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f4167e21640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:22:44.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f4167e21640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4160071c90 0x7f41601caef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:44.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f4167e21640 10 monclient(hunting): picked mon.noname-a con 0x7f4160071c90 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:22:44.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f4167e21640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4160135e30 0x7f41601376e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:44.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f4167e21640 10 monclient(hunting): picked mon.noname-b con 0x7f4160135e30 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:22:44.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f4167e21640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f416013c5f0 0x7f4160137c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:44.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f4167e21640 10 monclient(hunting): picked mon.noname-c con 0x7f416013c5f0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:22:44.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f4167e21640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:44.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f4167e21640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4160135160 con 0x7f4160071c90 2024-04-22T08:22:44.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f4167e21640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:44.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f4167e21640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41600720e0 con 0x7f4160135e30 2024-04-22T08:22:44.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f4167e21640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:44.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f4167e21640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f416013bd80 con 0x7f416013c5f0 2024-04-22T08:22:44.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f4167e21640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:44.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f4167e21640 10 monclient(hunting): authenticate will time out at 1438.748962s 2024-04-22T08:22:44.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f4165bbd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4160071c90 0x7f41601caef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:44.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f4165bbd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4160071c90 0x7f41601caef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44568/0 (socket says 172.21.15.40:44568) 2024-04-22T08:22:44.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f4165bbd640 1 -- 172.21.15.40:0/2505905415 learned_addr learned my addr 172.21.15.40:0/2505905415 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:22:44.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f4165bbd640 10 monclient(hunting): get_auth_request con 0x7f4160071c90 auth_method 0 2024-04-22T08:22:44.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f4165bbd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:44.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f4165bbd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:44.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f4165bbd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:44.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f41663be640 1 --2- 172.21.15.40:0/2505905415 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f416013c5f0 0x7f4160137c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:44.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f4165bbd640 1 -- 172.21.15.40:0/2505905415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4160071c90 msgr2=0x7f41601caef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send send error: (32) Broken pipe 2024-04-22T08:22:44.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f4165bbd640 1 --2- 172.21.15.40:0/2505905415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4160071c90 0x7f41601caef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).write auth request write failed r=-32 ((32) Broken pipe) 2024-04-22T08:22:44.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f41663be640 10 monclient(hunting): get_auth_request con 0x7f416013c5f0 auth_method 0 2024-04-22T08:22:44.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f41663be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:44.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f41663be640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:44.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f41663be640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:44.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f4165bbd640 1 --2- 172.21.15.40:0/2505905415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4160071c90 0x7f41601caef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:22:44.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f41653bc640 1 --2- 172.21.15.40:0/2505905415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4160135e30 0x7f41601376e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:44.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f41663be640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:44.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f41663be640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:44.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f41663be640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:44.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f41653bc640 10 monclient(hunting): get_auth_request con 0x7f4160135e30 auth_method 0 2024-04-22T08:22:44.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f41653bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:44.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f41653bc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:44.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f41653bc640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:44.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f41653bc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:44.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f41653bc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:44.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f41653bc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:44.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f41663be640 10 monclient(hunting): handle_auth_done global_id 124145 payload 824 2024-04-22T08:22:44.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f41663be640 1 -- 172.21.15.40:0/2505905415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4160135e30 msgr2=0x7f41601376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:44.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f41663be640 1 --2- 172.21.15.40:0/2505905415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4160135e30 0x7f41601376e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:44.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f41663be640 1 -- 172.21.15.40:0/2505905415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4160071c90 msgr2=0x7f41601caef0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T08:22:44.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f41663be640 1 --2- 172.21.15.40:0/2505905415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4160071c90 0x7f41601caef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:44.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f41663be640 10 monclient: _finish_hunting 0 2024-04-22T08:22:44.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f41663be640 1 monclient: found mon.noname-c 2024-04-22T08:22:44.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f41663be640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:44.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f41663be640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:22:44.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.755+0000 7f41663be640 1 -- 172.21.15.40:0/2505905415 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4150009a70 con 0x7f416013c5f0 2024-04-22T08:22:44.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.757+0000 7f41663be640 1 -- 172.21.15.40:0/2505905415 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f416013c5f0 msgr2=0x7f4160137c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:44.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.757+0000 7f41663be640 1 --2- 172.21.15.40:0/2505905415 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f416013c5f0 0x7f4160137c20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4150033000 tx=0x7f4150004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:44.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.758+0000 7f415e7fc640 1 -- 172.21.15.40:0/2505905415 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f415003e410 con 0x7f416013c5f0 2024-04-22T08:22:44.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.758+0000 7f415e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:44.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.758+0000 7f415e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:22:44.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.758+0000 7f415e7fc640 10 monclient: dump: 2024-04-22T08:22:44.773 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:44.773 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:44.773 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:44.773 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:44.773 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:44.773 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:44.773 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:44.773 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:44.773 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:44.773 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:44.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.758+0000 7f415e7fc640 10 monclient: _finish_auth 0 2024-04-22T08:22:44.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.758+0000 7f415e7fc640 10 monclient: _check_auth_tickets 2024-04-22T08:22:44.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.758+0000 7f415e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:44.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.758+0000 7f415e7fc640 1 -- 172.21.15.40:0/2505905415 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f415003e570 con 0x7f416013c5f0 2024-04-22T08:22:44.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.758+0000 7f4167e21640 5 monclient: authenticate success, global_id 124145 2024-04-22T08:22:44.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.758+0000 7f4167e21640 10 monclient: _renew_subs 2024-04-22T08:22:44.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.758+0000 7f4167e21640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:22:44.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.758+0000 7f4167e21640 1 -- 172.21.15.40:0/2505905415 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f416013b640 con 0x7f416013c5f0 2024-04-22T08:22:44.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.758+0000 7f415e7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:44.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.758+0000 7f415e7fc640 1 -- 172.21.15.40:0/2505905415 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f415003e920 con 0x7f416013c5f0 2024-04-22T08:22:44.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.758+0000 7f415e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:44.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.758+0000 7f415e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:22:44.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.758+0000 7f415e7fc640 10 monclient: dump: 2024-04-22T08:22:44.774 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:44.774 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:44.774 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:44.775 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:44.775 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:44.775 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:44.775 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:44.775 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:44.775 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:44.775 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:44.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.758+0000 7f4167e21640 10 monclient: _renew_subs 2024-04-22T08:22:44.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.758+0000 7f4167e21640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:22:44.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.758+0000 7f4167e21640 1 -- 172.21.15.40:0/2505905415 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f416013ada0 con 0x7f416013c5f0 2024-04-22T08:22:44.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.758+0000 7f415f7fe640 4 set_mon_vals no callback set 2024-04-22T08:22:44.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.758+0000 7f415e7fc640 1 -- 172.21.15.40:0/2505905415 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f415003b050 con 0x7f416013c5f0 2024-04-22T08:22:44.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.759+0000 7f4167e21640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:44.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.759+0000 7f4167e21640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:44.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.759+0000 7f4167e21640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:22:44.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.759+0000 7f4167e21640 1 -- 172.21.15.40:0/2505905415 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f416004e9b0 con 0x7f416013c5f0 2024-04-22T08:22:44.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.759+0000 7f415e7fc640 1 --2- 172.21.15.40:0/2505905415 >> v2:172.21.15.189:6800/30168 conn(0x7f414c0377d0 0x7f414c039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:44.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.759+0000 7f415e7fc640 1 -- 172.21.15.40:0/2505905415 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(317..317 src has 239..317) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f415006cc60 con 0x7f416013c5f0 2024-04-22T08:22:44.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.759+0000 7f4165bbd640 1 --2- 172.21.15.40:0/2505905415 >> v2:172.21.15.189:6800/30168 conn(0x7f414c0377d0 0x7f414c039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:44.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.759+0000 7f4165bbd640 10 monclient: get_auth_request con 0x7f414c0377d0 auth_method 0 2024-04-22T08:22:44.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.762+0000 7f4165bbd640 1 --2- 172.21.15.40:0/2505905415 >> v2:172.21.15.189:6800/30168 conn(0x7f414c0377d0 0x7f414c039c90 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f4158007c90 tx=0x7f4158009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:44.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.763+0000 7f415e7fc640 1 -- 172.21.15.40:0/2505905415 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f415003eb00 con 0x7f416013c5f0 2024-04-22T08:22:44.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.763+0000 7f415e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:44.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.763+0000 7f415e7fc640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:22:44.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.942+0000 7f4167e21640 1 -- 172.21.15.40:0/2505905415 --> v2:172.21.15.189:6800/30168 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f416004e9b0 con 0x7f414c0377d0 2024-04-22T08:22:44.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.943+0000 7f415e7fc640 1 -- 172.21.15.40:0/2505905415 <== mgr.4102 v2:172.21.15.189:6800/30168 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f416004e9b0 con 0x7f414c0377d0 2024-04-22T08:22:44.943 INFO:teuthology.orchestra.run.smithi040.stderr:instructed osd(s) 0 to deep-scrub 2024-04-22T08:22:44.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.945+0000 7f4167e21640 1 -- 172.21.15.40:0/2505905415 >> v2:172.21.15.189:6800/30168 conn(0x7f414c0377d0 msgr2=0x7f414c039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:44.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.945+0000 7f4167e21640 1 --2- 172.21.15.40:0/2505905415 >> v2:172.21.15.189:6800/30168 conn(0x7f414c0377d0 0x7f414c039c90 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f4158007c90 tx=0x7f4158009040 comp rx=0 tx=0).stop 2024-04-22T08:22:44.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.945+0000 7f4167e21640 10 monclient: shutdown 2024-04-22T08:22:44.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.945+0000 7f4167e21640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:44.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.945+0000 7f4167e21640 1 -- 172.21.15.40:0/2505905415 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f416013c5f0 msgr2=0x7f4160137c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:44.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.945+0000 7f4167e21640 1 --2- 172.21.15.40:0/2505905415 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f416013c5f0 0x7f4160137c20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4150033000 tx=0x7f4150004a40 comp rx=0 tx=0).stop 2024-04-22T08:22:44.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.945+0000 7f4167e21640 1 -- 172.21.15.40:0/2505905415 shutdown_connections 2024-04-22T08:22:44.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.945+0000 7f4167e21640 1 --2- 172.21.15.40:0/2505905415 >> v2:172.21.15.189:6800/30168 conn(0x7f414c0377d0 0x7f414c039c90 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:44.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.945+0000 7f4167e21640 1 --2- 172.21.15.40:0/2505905415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4160135e30 0x7f41601376e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:44.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.945+0000 7f4167e21640 1 --2- 172.21.15.40:0/2505905415 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f416013c5f0 0x7f4160137c20 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:44.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.945+0000 7f4167e21640 1 --2- 172.21.15.40:0/2505905415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4160071c90 0x7f41601caef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:44.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.945+0000 7f4167e21640 1 -- 172.21.15.40:0/2505905415 >> 172.21.15.40:0/2505905415 conn(0x7f41600a65c0 msgr2=0x7f416006cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:44.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.945+0000 7f4167e21640 1 -- 172.21.15.40:0/2505905415 shutdown_connections 2024-04-22T08:22:44.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:44.945+0000 7f4167e21640 1 -- 172.21.15.40:0/2505905415 wait complete. 2024-04-22T08:22:44.955 INFO:tasks.ceph:Scrubbing osd.1 2024-04-22T08:22:44.955 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-22T08:22:45.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.058+0000 7f4813f66640 10 monclient: get_monmap_and_config 2024-04-22T08:22:45.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.058+0000 7f4813f66640 10 monclient: build_initial_monmap 2024-04-22T08:22:45.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.058+0000 7f4813f66640 10 monclient: monmap: 2024-04-22T08:22:45.059 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:45.059 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:45.059 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:22:45.059745+0000 2024-04-22T08:22:45.059 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:22:45.059745+0000 2024-04-22T08:22:45.059 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:45.059 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:45.059 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:22:45.059 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:22:45.059 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:22:45.060 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:22:45.060 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:22:45.060 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:45.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.059+0000 7f4813f66640 1 Processor -- start 2024-04-22T08:22:45.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.059+0000 7f4813f66640 1 -- start start 2024-04-22T08:22:45.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.059+0000 7f4813f66640 10 monclient: init 2024-04-22T08:22:45.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.060+0000 7f4813f66640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:45.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.060+0000 7f4813f66640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-04-22T08:22:45.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.060+0000 7f4813f66640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f480c135f70 0x7f480c0a0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:45.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.060+0000 7f4813f66640 10 monclient(hunting): picked mon.noname-a con 0x7f480c135f70 addr v2:172.21.15.40:3300/0 2024-04-22T08:22:45.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.060+0000 7f4813f66640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f480c09f070 addr v1:172.21.15.40:6789/0 2024-04-22T08:22:45.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.060+0000 7f4813f66640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f480c13adf0 0x7f480c13d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:45.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.060+0000 7f4813f66640 10 monclient(hunting): picked mon.noname-b con 0x7f480c13adf0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:22:45.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.060+0000 7f4813f66640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f480c0a09b0 con 0x7f480c09f070 2024-04-22T08:22:45.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.060+0000 7f4813f66640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:45.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.060+0000 7f4813f66640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f480c09fa90 con 0x7f480c135f70 2024-04-22T08:22:45.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.060+0000 7f4813f66640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:45.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.060+0000 7f4813f66640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f480c09fd70 con 0x7f480c13adf0 2024-04-22T08:22:45.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.060+0000 7f4813f66640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:45.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.060+0000 7f4813f66640 10 monclient(hunting): authenticate will time out at 1439.054037s 2024-04-22T08:22:45.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.060+0000 7f4812503640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f480c13adf0 0x7f480c13d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:45.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.060+0000 7f4811d02640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f480c135f70 0x7f480c0a0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:45.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.060+0000 7f4811501640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f480c09f070 0x7f480c136350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:49340/0 (socket says 172.21.15.40:49340) 2024-04-22T08:22:45.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.060+0000 7f4811501640 1 -- 172.21.15.40:0/2401496486 learned_addr learned my addr 172.21.15.40:0/2401496486 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:22:45.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.060+0000 7f4812503640 10 monclient(hunting): get_auth_request con 0x7f480c13adf0 auth_method 0 2024-04-22T08:22:45.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.060+0000 7f4812503640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:45.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.060+0000 7f4812503640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:45.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.060+0000 7f4812503640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:45.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.060+0000 7f4811d02640 10 monclient(hunting): get_auth_request con 0x7f480c135f70 auth_method 0 2024-04-22T08:22:45.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.060+0000 7f4811d02640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:45.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.060+0000 7f4811d02640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:45.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.060+0000 7f4811d02640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:45.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.060+0000 7f4811d02640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:45.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.060+0000 7f4811d02640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:45.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.060+0000 7f4811d02640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:45.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.060+0000 7f4812503640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:45.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.060+0000 7f4812503640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:45.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.060+0000 7f4812503640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:45.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.060+0000 7f4811d02640 10 monclient(hunting): handle_auth_done global_id 114312 payload 274 2024-04-22T08:22:45.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.060+0000 7f4811d02640 1 -- 172.21.15.40:0/2401496486 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f480c13adf0 msgr2=0x7f480c13d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:45.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.060+0000 7f4811d02640 1 --2- 172.21.15.40:0/2401496486 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f480c13adf0 0x7f480c13d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:45.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.060+0000 7f4811d02640 1 -- 172.21.15.40:0/2401496486 >> v1:172.21.15.40:6789/0 conn(0x7f480c09f070 legacy=0x7f480c136350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:45.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.060+0000 7f4811d02640 10 monclient: _finish_hunting 0 2024-04-22T08:22:45.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.060+0000 7f4811d02640 1 monclient: found mon.noname-a 2024-04-22T08:22:45.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.060+0000 7f4811d02640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:45.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.060+0000 7f4811d02640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:22:45.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.060+0000 7f4811d02640 1 -- 172.21.15.40:0/2401496486 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f480c0a43a0 con 0x7f480c135f70 2024-04-22T08:22:45.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.060+0000 7f4812503640 1 --2- 172.21.15.40:0/2401496486 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f480c13adf0 0x7f480c13d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:22:45.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.063+0000 7f4811d02640 1 -- 172.21.15.40:0/2401496486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f480c135f70 msgr2=0x7f480c0a0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:45.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.063+0000 7f4811d02640 1 --2- 172.21.15.40:0/2401496486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f480c135f70 0x7f480c0a0470 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f48080097a0 tx=0x7f480802b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7971f306ea4e99d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:45.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.063+0000 7f4810d00640 1 -- 172.21.15.40:0/2401496486 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4808030920 con 0x7f480c135f70 2024-04-22T08:22:45.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.063+0000 7f4810d00640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:45.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.063+0000 7f4810d00640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:22:45.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.063+0000 7f4810d00640 10 monclient: dump: 2024-04-22T08:22:45.071 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:45.071 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:45.072 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:45.072 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:45.072 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:45.072 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:45.072 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:45.072 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:45.072 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:45.072 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:45.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.063+0000 7f4810d00640 10 monclient: _finish_auth 0 2024-04-22T08:22:45.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.063+0000 7f4810d00640 10 monclient: _check_auth_tickets 2024-04-22T08:22:45.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.063+0000 7f4810d00640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:45.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.063+0000 7f4810d00640 1 -- 172.21.15.40:0/2401496486 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f480802e050 con 0x7f480c135f70 2024-04-22T08:22:45.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.063+0000 7f4813f66640 5 monclient: authenticate success, global_id 114312 2024-04-22T08:22:45.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.063+0000 7f4810d00640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:45.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.063+0000 7f4813f66640 10 monclient: get_monmap_and_config success 2024-04-22T08:22:45.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.063+0000 7f4813f66640 4 set_mon_vals no callback set 2024-04-22T08:22:45.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.063+0000 7f4810d00640 1 -- 172.21.15.40:0/2401496486 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4808030ca0 con 0x7f480c135f70 2024-04-22T08:22:45.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.063+0000 7f4810d00640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:45.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.063+0000 7f4810d00640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:22:45.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.063+0000 7f4810d00640 10 monclient: dump: 2024-04-22T08:22:45.073 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:45.073 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:45.073 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:45.073 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:45.073 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:45.073 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:45.074 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:45.074 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:45.074 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:45.074 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:45.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.064+0000 7f4813f66640 10 monclient: shutdown 2024-04-22T08:22:45.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.064+0000 7f4813f66640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:45.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.064+0000 7f4813f66640 1 -- 172.21.15.40:0/2401496486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f480c135f70 msgr2=0x7f480c0a0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:45.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.064+0000 7f4813f66640 1 --2- 172.21.15.40:0/2401496486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f480c135f70 0x7f480c0a0470 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f48080097a0 tx=0x7f480802b710 comp rx=0 tx=0).stop 2024-04-22T08:22:45.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.064+0000 7f4813f66640 1 -- 172.21.15.40:0/2401496486 shutdown_connections 2024-04-22T08:22:45.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.064+0000 7f4813f66640 1 --2- 172.21.15.40:0/2401496486 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f480c13adf0 0x7f480c13d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:45.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.064+0000 7f4813f66640 1 --2- 172.21.15.40:0/2401496486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f480c135f70 0x7f480c0a0470 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:45.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.064+0000 7f4813f66640 1 -- 172.21.15.40:0/2401496486 >> 172.21.15.40:0/2401496486 conn(0x7f480c0a4ef0 msgr2=0x7f480c0a5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:45.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.064+0000 7f4813f66640 1 -- 172.21.15.40:0/2401496486 shutdown_connections 2024-04-22T08:22:45.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.064+0000 7f4813f66640 1 -- 172.21.15.40:0/2401496486 wait complete. 2024-04-22T08:22:45.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.065+0000 7f4813f66640 10 monclient: build_initial_monmap 2024-04-22T08:22:45.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.065+0000 7f4813f66640 10 monclient: monmap: 2024-04-22T08:22:45.075 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:45.075 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:22:45.075 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:22:45.075 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:22:45.075 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:45.075 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:45.075 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:22:45.076 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:22:45.076 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:22:45.076 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:45.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.065+0000 7f4813f66640 1 Processor -- start 2024-04-22T08:22:45.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.065+0000 7f4813f66640 1 -- start start 2024-04-22T08:22:45.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.065+0000 7f4813f66640 10 monclient: init 2024-04-22T08:22:45.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.065+0000 7f4813f66640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:45.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.065+0000 7f4813f66640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-22T08:22:45.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.065+0000 7f4813f66640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f480c09f070 0x7f480c1cb060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:45.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.065+0000 7f4813f66640 10 monclient(hunting): picked mon.noname-c con 0x7f480c09f070 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:22:45.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.065+0000 7f4813f66640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f480c135f70 0x7f480c1cb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:45.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.065+0000 7f4813f66640 10 monclient(hunting): picked mon.noname-b con 0x7f480c135f70 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:22:45.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.065+0000 7f4813f66640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f480c13adf0 0x7f480c06e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:45.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.065+0000 7f4813f66640 10 monclient(hunting): picked mon.noname-a con 0x7f480c13adf0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:22:45.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.065+0000 7f4813f66640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:45.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.065+0000 7f4813f66640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f480c070960 con 0x7f480c13adf0 2024-04-22T08:22:45.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.065+0000 7f4813f66640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:45.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.065+0000 7f4813f66640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f480c071020 con 0x7f480c135f70 2024-04-22T08:22:45.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.065+0000 7f4813f66640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:45.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.065+0000 7f4813f66640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f480c11eec0 con 0x7f480c09f070 2024-04-22T08:22:45.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.065+0000 7f4813f66640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:45.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.065+0000 7f4813f66640 10 monclient(hunting): authenticate will time out at 1439.059657s 2024-04-22T08:22:45.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.065+0000 7f4812503640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f480c13adf0 0x7f480c06e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:45.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.065+0000 7f4811d02640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f480c09f070 0x7f480c1cb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:45.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.066+0000 7f4812503640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f480c13adf0 0x7f480c06e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44582/0 (socket says 172.21.15.40:44582) 2024-04-22T08:22:45.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.066+0000 7f4812503640 1 -- 172.21.15.40:0/1594746183 learned_addr learned my addr 172.21.15.40:0/1594746183 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:22:45.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.066+0000 7f4811d02640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f480c09f070 0x7f480c1cb060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49868/0 (socket says 172.21.15.40:49868) 2024-04-22T08:22:45.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.066+0000 7f4812503640 10 monclient(hunting): get_auth_request con 0x7f480c13adf0 auth_method 0 2024-04-22T08:22:45.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.066+0000 7f4812503640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:45.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.066+0000 7f4812503640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:45.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.066+0000 7f4812503640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:45.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.066+0000 7f4811d02640 10 monclient(hunting): get_auth_request con 0x7f480c09f070 auth_method 0 2024-04-22T08:22:45.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.066+0000 7f4811d02640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:45.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.066+0000 7f4811d02640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:45.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.066+0000 7f4811d02640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:45.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.066+0000 7f4812503640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:45.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.066+0000 7f4812503640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:45.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.066+0000 7f4811501640 1 --2- 172.21.15.40:0/1594746183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f480c135f70 0x7f480c1cb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:45.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.066+0000 7f4812503640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:45.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.066+0000 7f4811d02640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:45.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.066+0000 7f4811d02640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:45.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.066+0000 7f4811d02640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:45.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.066+0000 7f4811501640 10 monclient(hunting): get_auth_request con 0x7f480c135f70 auth_method 0 2024-04-22T08:22:45.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.066+0000 7f4811501640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:45.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.066+0000 7f4811501640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:45.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.066+0000 7f4811501640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:45.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.066+0000 7f4812503640 10 monclient(hunting): handle_auth_done global_id 114315 payload 824 2024-04-22T08:22:45.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.066+0000 7f4812503640 1 -- 172.21.15.40:0/1594746183 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f480c09f070 msgr2=0x7f480c1cb060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:45.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.066+0000 7f4812503640 1 --2- 172.21.15.40:0/1594746183 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f480c09f070 0x7f480c1cb060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:45.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.066+0000 7f4812503640 1 -- 172.21.15.40:0/1594746183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f480c135f70 msgr2=0x7f480c1cb5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:45.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.066+0000 7f4812503640 1 --2- 172.21.15.40:0/1594746183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f480c135f70 0x7f480c1cb5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:45.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.066+0000 7f4812503640 10 monclient: _finish_hunting 0 2024-04-22T08:22:45.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.066+0000 7f4812503640 1 monclient: found mon.noname-a 2024-04-22T08:22:45.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.066+0000 7f4812503640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:45.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.066+0000 7f4812503640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:22:45.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.066+0000 7f4812503640 1 -- 172.21.15.40:0/1594746183 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4804009a70 con 0x7f480c13adf0 2024-04-22T08:22:45.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.066+0000 7f4811501640 1 --2- 172.21.15.40:0/1594746183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f480c135f70 0x7f480c1cb5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T08:22:45.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.068+0000 7f4812503640 1 -- 172.21.15.40:0/1594746183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f480c13adf0 msgr2=0x7f480c06e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:45.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.068+0000 7f4812503640 1 --2- 172.21.15.40:0/1594746183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f480c13adf0 0x7f480c06e8f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f480400efc0 tx=0x7f4804009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:45.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.068+0000 7f48027fc640 1 -- 172.21.15.40:0/1594746183 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4804009d70 con 0x7f480c13adf0 2024-04-22T08:22:45.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.068+0000 7f48027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:45.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.068+0000 7f48027fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:22:45.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.068+0000 7f48027fc640 10 monclient: dump: 2024-04-22T08:22:45.081 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:45.081 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:45.081 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:45.081 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:45.081 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:45.081 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:45.081 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:45.081 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:45.081 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:45.081 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:45.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.068+0000 7f48027fc640 10 monclient: _finish_auth 0 2024-04-22T08:22:45.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.068+0000 7f48027fc640 10 monclient: _check_auth_tickets 2024-04-22T08:22:45.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.068+0000 7f48027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:45.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.068+0000 7f48027fc640 1 -- 172.21.15.40:0/1594746183 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f480400f040 con 0x7f480c13adf0 2024-04-22T08:22:45.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.068+0000 7f4813f66640 5 monclient: authenticate success, global_id 114315 2024-04-22T08:22:45.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.068+0000 7f4813f66640 10 monclient: _renew_subs 2024-04-22T08:22:45.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.068+0000 7f4813f66640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:45.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.068+0000 7f4813f66640 1 -- 172.21.15.40:0/1594746183 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4808009400 con 0x7f480c13adf0 2024-04-22T08:22:45.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.068+0000 7f48027fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:45.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.068+0000 7f48027fc640 1 -- 172.21.15.40:0/1594746183 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4804003bb0 con 0x7f480c13adf0 2024-04-22T08:22:45.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.068+0000 7f48027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:45.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.068+0000 7f48027fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:22:45.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.068+0000 7f48027fc640 10 monclient: dump: 2024-04-22T08:22:45.082 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:45.082 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:45.082 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:45.082 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:45.083 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:45.083 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:45.083 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:45.083 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:45.083 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:45.083 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:45.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.068+0000 7f4813f66640 10 monclient: _renew_subs 2024-04-22T08:22:45.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.068+0000 7f4813f66640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:45.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.068+0000 7f4803fff640 4 set_mon_vals no callback set 2024-04-22T08:22:45.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.068+0000 7f4813f66640 1 -- 172.21.15.40:0/1594746183 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f480c070450 con 0x7f480c13adf0 2024-04-22T08:22:45.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.069+0000 7f4813f66640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:45.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.069+0000 7f4813f66640 1 -- 172.21.15.40:0/1594746183 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f480c04e9b0 con 0x7f480c13adf0 2024-04-22T08:22:45.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.069+0000 7f48027fc640 1 -- 172.21.15.40:0/1594746183 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f48040144a0 con 0x7f480c13adf0 2024-04-22T08:22:45.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.069+0000 7f48027fc640 1 --2- 172.21.15.40:0/1594746183 >> v2:172.21.15.189:6800/30168 conn(0x7f47f0037820 0x7f47f0039ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:45.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.069+0000 7f48027fc640 1 -- 172.21.15.40:0/1594746183 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(317..317 src has 239..317) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f480404ae40 con 0x7f480c13adf0 2024-04-22T08:22:45.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.070+0000 7f48027fc640 1 --2- 172.21.15.40:0/1594746183 >> v2:172.21.15.40:6800/30670 conn(0x7f47f003d350 0x7f47f003f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:45.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.070+0000 7f48027fc640 1 -- 172.21.15.40:0/1594746183 --> v2:172.21.15.40:6800/30670 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f47f003fde0 con 0x7f47f003d350 2024-04-22T08:22:45.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.070+0000 7f48027fc640 1 -- 172.21.15.40:0/1594746183 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=317) v2 ==== 24+0+0 (secure 0 0 0) 0x7f480404d5e0 con 0x7f480c13adf0 2024-04-22T08:22:45.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.070+0000 7f48027fc640 10 monclient: handle_get_version_reply finishing 1 version 317 2024-04-22T08:22:45.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.070+0000 7f4811d02640 1 --2- 172.21.15.40:0/1594746183 >> v2:172.21.15.189:6800/30168 conn(0x7f47f0037820 0x7f47f0039ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:45.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.070+0000 7f4811d02640 10 monclient: get_auth_request con 0x7f47f0037820 auth_method 0 2024-04-22T08:22:45.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.070+0000 7f4811501640 1 --2- 172.21.15.40:0/1594746183 >> v2:172.21.15.40:6800/30670 conn(0x7f47f003d350 0x7f47f003f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:45.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.070+0000 7f4811501640 10 monclient: get_auth_request con 0x7f47f003d350 auth_method 0 2024-04-22T08:22:45.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.072+0000 7f4811d02640 1 --2- 172.21.15.40:0/1594746183 >> v2:172.21.15.189:6800/30168 conn(0x7f47f0037820 0x7f47f0039ce0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f480c0725c0 tx=0x7f480802f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:45.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.074+0000 7f4811501640 1 --2- 172.21.15.40:0/1594746183 >> v2:172.21.15.40:6800/30670 conn(0x7f47f003d350 0x7f47f003f770 crc :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:45.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.075+0000 7f48027fc640 1 -- 172.21.15.40:0/1594746183 <== osd.1 v2:172.21.15.40:6800/30670 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f47f003fde0 con 0x7f47f003d350 2024-04-22T08:22:45.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.087+0000 7f4813f66640 1 -- 172.21.15.40:0/1594746183 --> v2:172.21.15.40:6800/30670 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f480c0a5750 con 0x7f47f003d350 2024-04-22T08:22:45.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.088+0000 7f48027fc640 1 -- 172.21.15.40:0/1594746183 <== osd.1 v2:172.21.15.40:6800/30670 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f480c0a5750 con 0x7f47f003d350 2024-04-22T08:22:45.087 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:22:45.087 INFO:teuthology.orchestra.run.smithi040.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-22T08:22:45.087 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:22:45.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.088+0000 7f4813f66640 1 -- 172.21.15.40:0/1594746183 >> v2:172.21.15.40:6800/30670 conn(0x7f47f003d350 msgr2=0x7f47f003f770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:45.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.088+0000 7f4813f66640 1 --2- 172.21.15.40:0/1594746183 >> v2:172.21.15.40:6800/30670 conn(0x7f47f003d350 0x7f47f003f770 crc :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:45.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.088+0000 7f4813f66640 1 -- 172.21.15.40:0/1594746183 >> v2:172.21.15.189:6800/30168 conn(0x7f47f0037820 msgr2=0x7f47f0039ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:45.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.088+0000 7f4813f66640 1 --2- 172.21.15.40:0/1594746183 >> v2:172.21.15.189:6800/30168 conn(0x7f47f0037820 0x7f47f0039ce0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f480c0725c0 tx=0x7f480802f000 comp rx=0 tx=0).stop 2024-04-22T08:22:45.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.088+0000 7f4813f66640 10 monclient: shutdown 2024-04-22T08:22:45.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.088+0000 7f4813f66640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:45.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.088+0000 7f4813f66640 1 -- 172.21.15.40:0/1594746183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f480c13adf0 msgr2=0x7f480c06e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:45.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.088+0000 7f4813f66640 1 --2- 172.21.15.40:0/1594746183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f480c13adf0 0x7f480c06e8f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f480400efc0 tx=0x7f4804009f90 comp rx=0 tx=0).stop 2024-04-22T08:22:45.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.088+0000 7f4812503640 1 -- 172.21.15.40:0/1594746183 reap_dead start 2024-04-22T08:22:45.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.088+0000 7f4813f66640 1 -- 172.21.15.40:0/1594746183 shutdown_connections 2024-04-22T08:22:45.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.088+0000 7f4813f66640 1 -- 172.21.15.40:0/1594746183 >> 172.21.15.40:0/1594746183 conn(0x7f480c0a4ef0 msgr2=0x7f480c0a2cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:45.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.088+0000 7f4813f66640 1 -- 172.21.15.40:0/1594746183 shutdown_connections 2024-04-22T08:22:45.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.088+0000 7f4813f66640 1 -- 172.21.15.40:0/1594746183 wait complete. 2024-04-22T08:22:45.097 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-22T08:22:45.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.197+0000 7ff78f423640 10 monclient: get_monmap_and_config 2024-04-22T08:22:45.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.197+0000 7ff78f423640 10 monclient: build_initial_monmap 2024-04-22T08:22:45.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.197+0000 7ff78f423640 10 monclient: monmap: 2024-04-22T08:22:45.197 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:45.197 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:45.197 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:22:45.198367+0000 2024-04-22T08:22:45.197 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:22:45.198367+0000 2024-04-22T08:22:45.198 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:45.198 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:45.198 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:22:45.198 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:22:45.198 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:22:45.198 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:22:45.198 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:22:45.198 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:45.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.198+0000 7ff78f423640 1 Processor -- start 2024-04-22T08:22:45.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.198+0000 7ff78f423640 1 -- start start 2024-04-22T08:22:45.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.198+0000 7ff78f423640 10 monclient: init 2024-04-22T08:22:45.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.198+0000 7ff78f423640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:45.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.198+0000 7ff78f423640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-04-22T08:22:45.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.198+0000 7ff78f423640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff78809f2e0 addr v1:172.21.15.189:6789/0 2024-04-22T08:22:45.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.198+0000 7ff78f423640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff78809b760 0x7ff788137850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:45.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.198+0000 7ff78f423640 10 monclient(hunting): picked mon.noname-a con 0x7ff78809b760 addr v2:172.21.15.40:3300/0 2024-04-22T08:22:45.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.198+0000 7ff78f423640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff78809fdd0 0x7ff788139da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:45.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.198+0000 7ff78f423640 10 monclient(hunting): picked mon.noname-b con 0x7ff78809fdd0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:22:45.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.198+0000 7ff78f423640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff78809cf70 con 0x7ff78809f2e0 2024-04-22T08:22:45.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.198+0000 7ff78f423640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:45.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.198+0000 7ff78f423640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff78809c6d0 con 0x7ff78809b760 2024-04-22T08:22:45.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.198+0000 7ff78f423640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:45.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.198+0000 7ff78f423640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff78809c9b0 con 0x7ff78809fdd0 2024-04-22T08:22:45.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.198+0000 7ff78f423640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:45.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.198+0000 7ff78f423640 10 monclient(hunting): authenticate will time out at 1439.192610s 2024-04-22T08:22:45.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.198+0000 7ff78c9be640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff78809b760 0x7ff788137850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:45.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.198+0000 7ff78d9c0640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff78809fdd0 0x7ff788139da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:45.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.199+0000 7ff78c9be640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7ff78809b760 msgr2=0x7ff788137850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-22T08:22:45.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.199+0000 7ff78c9be640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7ff78809b760 msgr2=0x7ff788137850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T08:22:45.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.199+0000 7ff78c9be640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff78809b760 0x7ff788137850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:22:45.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.199+0000 7ff78d9c0640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff78809fdd0 0x7ff788139da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49874/0 (socket says 172.21.15.40:49874) 2024-04-22T08:22:45.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.199+0000 7ff78d9c0640 1 -- 172.21.15.40:0/511640909 learned_addr learned my addr 172.21.15.40:0/511640909 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:22:45.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.199+0000 7ff78d9c0640 10 monclient(hunting): get_auth_request con 0x7ff78809fdd0 auth_method 0 2024-04-22T08:22:45.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.199+0000 7ff78c9be640 1 --2- 172.21.15.40:0/511640909 >> v2:172.21.15.40:3300/0 conn(0x7ff78809b760 0x7ff788137850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:22:45.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.199+0000 7ff78d9c0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:45.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.199+0000 7ff78d9c0640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:45.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.199+0000 7ff78d9c0640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:45.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.199+0000 7ff78d9c0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:45.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.199+0000 7ff78d9c0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:45.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.199+0000 7ff78d9c0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:45.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.199+0000 7ff78d9c0640 10 monclient(hunting): handle_auth_done global_id 124154 payload 274 2024-04-22T08:22:45.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.199+0000 7ff78d9c0640 1 -- 172.21.15.40:0/511640909 >> v2:172.21.15.40:3300/0 conn(0x7ff78809b760 msgr2=0x7ff788137850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T08:22:45.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.199+0000 7ff78d9c0640 1 --2- 172.21.15.40:0/511640909 >> v2:172.21.15.40:3300/0 conn(0x7ff78809b760 0x7ff788137850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:45.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.199+0000 7ff78d9c0640 1 -- 172.21.15.40:0/511640909 >> v1:172.21.15.189:6789/0 conn(0x7ff78809f2e0 legacy=0x7ff78809f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:45.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.199+0000 7ff78d9c0640 10 monclient: _finish_hunting 0 2024-04-22T08:22:45.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.199+0000 7ff78d9c0640 1 monclient: found mon.noname-b 2024-04-22T08:22:45.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.199+0000 7ff78d9c0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:45.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.199+0000 7ff78d9c0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:22:45.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.199+0000 7ff78d9c0640 1 -- 172.21.15.40:0/511640909 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff78809cc90 con 0x7ff78809fdd0 2024-04-22T08:22:45.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.202+0000 7ff78d9c0640 1 -- 172.21.15.40:0/511640909 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff78809fdd0 msgr2=0x7ff788139da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:45.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.202+0000 7ff78d9c0640 1 --2- 172.21.15.40:0/511640909 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff78809fdd0 0x7ff788139da0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff780009e50 tx=0x7ff78002bdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff8ddd4cf7443787 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:45.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.202+0000 7ff77ffff640 1 -- 172.21.15.40:0/511640909 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff78002cb60 con 0x7ff78809fdd0 2024-04-22T08:22:45.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.202+0000 7ff77ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:45.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.202+0000 7ff77ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:22:45.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.202+0000 7ff77ffff640 10 monclient: dump: 2024-04-22T08:22:45.210 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:45.210 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:45.210 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:45.210 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:45.210 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:45.210 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:45.210 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:45.210 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:45.210 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:45.211 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:45.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.202+0000 7ff77ffff640 10 monclient: _finish_auth 0 2024-04-22T08:22:45.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.202+0000 7ff77ffff640 10 monclient: _check_auth_tickets 2024-04-22T08:22:45.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.202+0000 7ff77ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:45.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.202+0000 7ff77ffff640 1 -- 172.21.15.40:0/511640909 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff78002ccc0 con 0x7ff78809fdd0 2024-04-22T08:22:45.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.202+0000 7ff78f423640 5 monclient: authenticate success, global_id 124154 2024-04-22T08:22:45.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.202+0000 7ff78f423640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:22:45.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.202+0000 7ff77ffff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:45.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.202+0000 7ff78f423640 10 monclient: get_monmap_and_config success 2024-04-22T08:22:45.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.202+0000 7ff77ffff640 1 -- 172.21.15.40:0/511640909 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff780034610 con 0x7ff78809fdd0 2024-04-22T08:22:45.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.202+0000 7ff77ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:45.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.202+0000 7ff78f423640 4 set_mon_vals no callback set 2024-04-22T08:22:45.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.202+0000 7ff77ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:22:45.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.202+0000 7ff77ffff640 10 monclient: dump: 2024-04-22T08:22:45.212 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:45.212 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:45.212 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:45.212 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:45.212 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:45.212 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:45.212 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:45.212 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:45.212 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:45.212 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:45.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.202+0000 7ff78f423640 10 monclient: shutdown 2024-04-22T08:22:45.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.202+0000 7ff78f423640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:45.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.202+0000 7ff78f423640 1 -- 172.21.15.40:0/511640909 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff78809fdd0 msgr2=0x7ff788139da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:45.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.202+0000 7ff78f423640 1 --2- 172.21.15.40:0/511640909 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff78809fdd0 0x7ff788139da0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff780009e50 tx=0x7ff78002bdb0 comp rx=0 tx=0).stop 2024-04-22T08:22:45.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.202+0000 7ff78f423640 1 -- 172.21.15.40:0/511640909 shutdown_connections 2024-04-22T08:22:45.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.202+0000 7ff78f423640 1 --2- 172.21.15.40:0/511640909 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff78809fdd0 0x7ff788139da0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:45.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.202+0000 7ff78f423640 1 --2- 172.21.15.40:0/511640909 >> v2:172.21.15.40:3300/0 conn(0x7ff78809b760 0x7ff788137850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:45.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.202+0000 7ff78f423640 1 -- 172.21.15.40:0/511640909 >> 172.21.15.40:0/511640909 conn(0x7ff7880a0b50 msgr2=0x7ff7880a0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:45.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.202+0000 7ff78f423640 1 -- 172.21.15.40:0/511640909 shutdown_connections 2024-04-22T08:22:45.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.202+0000 7ff78f423640 1 -- 172.21.15.40:0/511640909 wait complete. 2024-04-22T08:22:45.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.203+0000 7ff78f423640 10 monclient: build_initial_monmap 2024-04-22T08:22:45.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.203+0000 7ff78f423640 10 monclient: monmap: 2024-04-22T08:22:45.213 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:45.213 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:22:45.213 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:22:45.213 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:22:45.213 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:45.213 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:45.214 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:22:45.214 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:22:45.214 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:22:45.214 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:45.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.203+0000 7ff78f423640 1 Processor -- start 2024-04-22T08:22:45.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.203+0000 7ff78f423640 1 -- start start 2024-04-22T08:22:45.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.203+0000 7ff78f423640 10 monclient: init 2024-04-22T08:22:45.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78f423640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:45.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78f423640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-22T08:22:45.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78f423640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff78809f2e0 0x7ff7881ca970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:45.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78f423640 10 monclient(hunting): picked mon.noname-b con 0x7ff78809f2e0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:22:45.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78f423640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff78809fdd0 0x7ff78813ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:45.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78f423640 10 monclient(hunting): picked mon.noname-c con 0x7ff78809fdd0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:22:45.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78f423640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff78813bfd0 0x7ff7881c73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:45.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78f423640 10 monclient(hunting): picked mon.noname-a con 0x7ff78813bfd0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:22:45.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78f423640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:45.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78f423640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff788127510 con 0x7ff78813bfd0 2024-04-22T08:22:45.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78f423640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:45.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78f423640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff78809bbc0 con 0x7ff78809f2e0 2024-04-22T08:22:45.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78f423640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:45.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78f423640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff788070fa0 con 0x7ff78809fdd0 2024-04-22T08:22:45.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78f423640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:45.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78f423640 10 monclient(hunting): authenticate will time out at 1439.198002s 2024-04-22T08:22:45.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78c9be640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff78809fdd0 0x7ff78813ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:45.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78d9c0640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff78813bfd0 0x7ff7881c73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:45.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78d9c0640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff78813bfd0 0x7ff7881c73f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44592/0 (socket says 172.21.15.40:44592) 2024-04-22T08:22:45.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78c9be640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff78809fdd0 0x7ff78813ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49876/0 (socket says 172.21.15.40:49876) 2024-04-22T08:22:45.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78d9c0640 1 -- 172.21.15.40:0/2346462573 learned_addr learned my addr 172.21.15.40:0/2346462573 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:22:45.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78d9c0640 10 monclient(hunting): get_auth_request con 0x7ff78813bfd0 auth_method 0 2024-04-22T08:22:45.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78d9c0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:45.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78d9c0640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:45.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78d9c0640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:45.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78c9be640 10 monclient(hunting): get_auth_request con 0x7ff78809fdd0 auth_method 0 2024-04-22T08:22:45.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78c9be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:45.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78c9be640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:45.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78c9be640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:45.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78d1bf640 1 --2- 172.21.15.40:0/2346462573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff78809f2e0 0x7ff7881ca970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:45.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78d9c0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:45.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78d9c0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:45.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78d9c0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:45.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78d1bf640 10 monclient(hunting): get_auth_request con 0x7ff78809f2e0 auth_method 0 2024-04-22T08:22:45.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78d1bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:45.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78d1bf640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:45.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78d1bf640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:45.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78c9be640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:45.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78c9be640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:45.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78c9be640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:45.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78d1bf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:45.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78d1bf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:45.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78d1bf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:45.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78d9c0640 10 monclient(hunting): handle_auth_done global_id 114318 payload 824 2024-04-22T08:22:45.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78d9c0640 1 -- 172.21.15.40:0/2346462573 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff78809fdd0 msgr2=0x7ff78813ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:45.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78d9c0640 1 --2- 172.21.15.40:0/2346462573 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff78809fdd0 0x7ff78813ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:45.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78d9c0640 1 -- 172.21.15.40:0/2346462573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff78809f2e0 msgr2=0x7ff7881ca970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:45.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78d9c0640 1 --2- 172.21.15.40:0/2346462573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff78809f2e0 0x7ff7881ca970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:45.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78d9c0640 10 monclient: _finish_hunting 0 2024-04-22T08:22:45.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78d9c0640 1 monclient: found mon.noname-a 2024-04-22T08:22:45.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78d9c0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:45.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78d9c0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:22:45.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78d9c0640 1 -- 172.21.15.40:0/2346462573 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff780009b30 con 0x7ff78813bfd0 2024-04-22T08:22:45.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.204+0000 7ff78c9be640 1 --2- 172.21.15.40:0/2346462573 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff78809fdd0 0x7ff78813ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:22:45.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.206+0000 7ff78d9c0640 1 -- 172.21.15.40:0/2346462573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff78813bfd0 msgr2=0x7ff7881c73f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:45.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.206+0000 7ff78d9c0640 1 --2- 172.21.15.40:0/2346462573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff78813bfd0 0x7ff7881c73f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff780036040 tx=0x7ff780004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:45.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.207+0000 7ff77dffb640 1 -- 172.21.15.40:0/2346462573 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff78003e450 con 0x7ff78813bfd0 2024-04-22T08:22:45.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.207+0000 7ff77dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:45.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.207+0000 7ff77dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:22:45.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.207+0000 7ff77dffb640 10 monclient: dump: 2024-04-22T08:22:45.218 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:45.218 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:45.218 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:45.219 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:45.219 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:45.219 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:45.219 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:45.219 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:45.219 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:45.219 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:45.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.207+0000 7ff77dffb640 10 monclient: _finish_auth 0 2024-04-22T08:22:45.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.207+0000 7ff77dffb640 10 monclient: _check_auth_tickets 2024-04-22T08:22:45.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.207+0000 7ff77dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:45.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.207+0000 7ff77dffb640 1 -- 172.21.15.40:0/2346462573 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff78003e5d0 con 0x7ff78813bfd0 2024-04-22T08:22:45.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.207+0000 7ff77dffb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:45.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.207+0000 7ff77dffb640 1 -- 172.21.15.40:0/2346462573 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff78003e980 con 0x7ff78813bfd0 2024-04-22T08:22:45.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.207+0000 7ff78f423640 5 monclient: authenticate success, global_id 114318 2024-04-22T08:22:45.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.207+0000 7ff78f423640 10 monclient: _renew_subs 2024-04-22T08:22:45.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.207+0000 7ff78f423640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:45.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.207+0000 7ff78f423640 1 -- 172.21.15.40:0/2346462573 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff788126b90 con 0x7ff78813bfd0 2024-04-22T08:22:45.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.207+0000 7ff77f7fe640 4 set_mon_vals no callback set 2024-04-22T08:22:45.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.207+0000 7ff77dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:45.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.207+0000 7ff77dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:22:45.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.207+0000 7ff77dffb640 10 monclient: dump: 2024-04-22T08:22:45.220 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:45.220 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:45.220 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:45.220 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:45.220 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:45.220 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:45.220 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:45.220 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:45.221 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:45.221 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:45.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.207+0000 7ff78f423640 10 monclient: _renew_subs 2024-04-22T08:22:45.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.207+0000 7ff78f423640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:45.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.207+0000 7ff78f423640 1 -- 172.21.15.40:0/2346462573 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff788120cc0 con 0x7ff78813bfd0 2024-04-22T08:22:45.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.208+0000 7ff77dffb640 1 -- 172.21.15.40:0/2346462573 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7ff78003eb20 con 0x7ff78813bfd0 2024-04-22T08:22:45.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.208+0000 7ff77dffb640 1 --2- 172.21.15.40:0/2346462573 >> v2:172.21.15.189:6800/30168 conn(0x7ff76c037850 0x7ff76c039d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:45.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.208+0000 7ff77dffb640 1 -- 172.21.15.40:0/2346462573 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(317..317 src has 239..317) v4 ==== 3506+0+0 (secure 0 0 0) 0x7ff78006cb30 con 0x7ff78813bfd0 2024-04-22T08:22:45.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.208+0000 7ff78f423640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:45.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.208+0000 7ff78f423640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:45.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.208+0000 7ff78f423640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:45.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.208+0000 7ff78f423640 1 -- 172.21.15.40:0/2346462573 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7881cb360 con 0x7ff78813bfd0 2024-04-22T08:22:45.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.208+0000 7ff78d1bf640 1 --2- 172.21.15.40:0/2346462573 >> v2:172.21.15.189:6800/30168 conn(0x7ff76c037850 0x7ff76c039d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:45.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.208+0000 7ff78d1bf640 10 monclient: get_auth_request con 0x7ff76c037850 auth_method 0 2024-04-22T08:22:45.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.208+0000 7ff78d1bf640 1 -- 172.21.15.40:0/2346462573 >> v2:172.21.15.189:6800/30168 conn(0x7ff76c037850 msgr2=0x7ff76c039d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-22T08:22:45.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.208+0000 7ff78d1bf640 1 -- 172.21.15.40:0/2346462573 >> v2:172.21.15.189:6800/30168 conn(0x7ff76c037850 msgr2=0x7ff76c039d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:22:45.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.208+0000 7ff78d1bf640 1 --2- 172.21.15.40:0/2346462573 >> v2:172.21.15.189:6800/30168 conn(0x7ff76c037850 0x7ff76c039d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:22:45.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.208+0000 7ff78d1bf640 1 --2- 172.21.15.40:0/2346462573 >> v2:172.21.15.189:6800/30168 conn(0x7ff76c037850 0x7ff76c039d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:22:45.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.212+0000 7ff77dffb640 1 -- 172.21.15.40:0/2346462573 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7ff7881cb360 con 0x7ff78813bfd0 2024-04-22T08:22:45.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.212+0000 7ff77dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:45.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.212+0000 7ff77dffb640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:22:45.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.393+0000 7ff78f423640 1 -- 172.21.15.40:0/2346462573 --> v2:172.21.15.189:6800/30168 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7ff788072180 con 0x7ff76c037850 2024-04-22T08:22:45.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.409+0000 7ff78d1bf640 1 --2- 172.21.15.40:0/2346462573 >> v2:172.21.15.189:6800/30168 conn(0x7ff76c037850 0x7ff76c039d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:45.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.409+0000 7ff78d1bf640 10 monclient: get_auth_request con 0x7ff76c037850 auth_method 0 2024-04-22T08:22:45.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.412+0000 7ff78d1bf640 1 --2- 172.21.15.40:0/2346462573 >> v2:172.21.15.189:6800/30168 conn(0x7ff76c037850 0x7ff76c039d10 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff784004b30 tx=0x7ff784009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:45.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.412+0000 7ff77dffb640 1 -- 172.21.15.40:0/2346462573 <== mgr.4102 v2:172.21.15.189:6800/30168 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7ff784004ce0 con 0x7ff76c037850 2024-04-22T08:22:45.412 INFO:teuthology.orchestra.run.smithi040.stderr:instructed osd(s) 1 to deep-scrub 2024-04-22T08:22:45.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.415+0000 7ff78f423640 1 -- 172.21.15.40:0/2346462573 >> v2:172.21.15.189:6800/30168 conn(0x7ff76c037850 msgr2=0x7ff76c039d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:45.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.415+0000 7ff78f423640 1 --2- 172.21.15.40:0/2346462573 >> v2:172.21.15.189:6800/30168 conn(0x7ff76c037850 0x7ff76c039d10 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff784004b30 tx=0x7ff784009040 comp rx=0 tx=0).stop 2024-04-22T08:22:45.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.415+0000 7ff78f423640 10 monclient: shutdown 2024-04-22T08:22:45.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.415+0000 7ff78f423640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:45.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.415+0000 7ff78f423640 1 -- 172.21.15.40:0/2346462573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff78813bfd0 msgr2=0x7ff7881c73f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:45.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.415+0000 7ff78f423640 1 --2- 172.21.15.40:0/2346462573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff78813bfd0 0x7ff7881c73f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff780036040 tx=0x7ff780004a40 comp rx=0 tx=0).stop 2024-04-22T08:22:45.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.415+0000 7ff78f423640 1 -- 172.21.15.40:0/2346462573 shutdown_connections 2024-04-22T08:22:45.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.415+0000 7ff78f423640 1 --2- 172.21.15.40:0/2346462573 >> v2:172.21.15.189:6800/30168 conn(0x7ff76c037850 0x7ff76c039d10 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:45.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.415+0000 7ff78f423640 1 --2- 172.21.15.40:0/2346462573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff78813bfd0 0x7ff7881c73f0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:45.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.415+0000 7ff78f423640 1 --2- 172.21.15.40:0/2346462573 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff78809fdd0 0x7ff78813ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:45.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.415+0000 7ff78f423640 1 --2- 172.21.15.40:0/2346462573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff78809f2e0 0x7ff7881ca970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:45.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.415+0000 7ff78f423640 1 -- 172.21.15.40:0/2346462573 >> 172.21.15.40:0/2346462573 conn(0x7ff7880a0b50 msgr2=0x7ff78806f150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:45.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.415+0000 7ff78f423640 1 -- 172.21.15.40:0/2346462573 shutdown_connections 2024-04-22T08:22:45.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.415+0000 7ff78f423640 1 -- 172.21.15.40:0/2346462573 wait complete. 2024-04-22T08:22:45.425 INFO:tasks.ceph:Scrubbing osd.2 2024-04-22T08:22:45.425 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-22T08:22:45.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.528+0000 7f1564ab6640 10 monclient: get_monmap_and_config 2024-04-22T08:22:45.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.528+0000 7f1564ab6640 10 monclient: build_initial_monmap 2024-04-22T08:22:45.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.528+0000 7f1564ab6640 10 monclient: monmap: 2024-04-22T08:22:45.528 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:45.528 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:45.528 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:22:45.528993+0000 2024-04-22T08:22:45.528 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:22:45.528993+0000 2024-04-22T08:22:45.528 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:45.528 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:45.528 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:22:45.528 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:22:45.528 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:22:45.528 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:22:45.528 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:22:45.528 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:45.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.528+0000 7f1564ab6640 1 Processor -- start 2024-04-22T08:22:45.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.528+0000 7f1564ab6640 1 -- start start 2024-04-22T08:22:45.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.528+0000 7f1564ab6640 10 monclient: init 2024-04-22T08:22:45.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.529+0000 7f1564ab6640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:45.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.529+0000 7f1564ab6640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-04-22T08:22:45.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.529+0000 7f1564ab6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1560132900 addr v1:172.21.15.189:6789/0 2024-04-22T08:22:45.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.529+0000 7f1564ab6640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f156009c940 0x7f1560137e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:45.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.529+0000 7f1564ab6640 10 monclient(hunting): picked mon.noname-c con 0x7f156009c940 addr v2:172.21.15.189:3300/0 2024-04-22T08:22:45.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.529+0000 7f1564ab6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f15601333f0 addr v1:172.21.15.40:6789/0 2024-04-22T08:22:45.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.529+0000 7f1564ab6640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f156009eb70 con 0x7f15601333f0 2024-04-22T08:22:45.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.529+0000 7f1564ab6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f156009f050 con 0x7f1560132900 2024-04-22T08:22:45.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.529+0000 7f1564ab6640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:45.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.529+0000 7f1564ab6640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f156009d5c0 con 0x7f156009c940 2024-04-22T08:22:45.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.529+0000 7f1564ab6640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:45.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.529+0000 7f1564ab6640 10 monclient(hunting): authenticate will time out at 1439.523209s 2024-04-22T08:22:45.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.529+0000 7f155ed9d640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f15601333f0 0x7f156013a380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:49354/0 (socket says 172.21.15.40:49354) 2024-04-22T08:22:45.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.529+0000 7f155ed9d640 1 -- 172.21.15.40:0/58952372 learned_addr learned my addr 172.21.15.40:0/58952372 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:22:45.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.529+0000 7f155dd9b640 1 --2- 172.21.15.40:0/58952372 >> v2:172.21.15.189:3300/0 conn(0x7f156009c940 0x7f1560137e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:45.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.529+0000 7f155dd9b640 10 monclient(hunting): get_auth_request con 0x7f156009c940 auth_method 0 2024-04-22T08:22:45.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.529+0000 7f155dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:45.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.529+0000 7f155dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:45.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.529+0000 7f155dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:45.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.529+0000 7f155dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:45.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.529+0000 7f155dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:45.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.529+0000 7f155dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:45.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.530+0000 7f155dd9b640 10 monclient(hunting): handle_auth_done global_id 65284 payload 274 2024-04-22T08:22:45.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.530+0000 7f155dd9b640 1 -- 172.21.15.40:0/58952372 >> v1:172.21.15.189:6789/0 conn(0x7f1560132900 legacy=0x7f1560132ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:45.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.530+0000 7f155dd9b640 1 -- 172.21.15.40:0/58952372 >> v1:172.21.15.40:6789/0 conn(0x7f15601333f0 legacy=0x7f156013a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:45.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.530+0000 7f155dd9b640 10 monclient: _finish_hunting 0 2024-04-22T08:22:45.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.530+0000 7f155dd9b640 1 monclient: found mon.noname-c 2024-04-22T08:22:45.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.530+0000 7f155dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:45.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.530+0000 7f155dd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.189:3300/0 2024-04-22T08:22:45.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.530+0000 7f155dd9b640 1 -- 172.21.15.40:0/58952372 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f156009d000 con 0x7f156009c940 2024-04-22T08:22:45.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.533+0000 7f155dd9b640 1 -- 172.21.15.40:0/58952372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f156009c940 msgr2=0x7f1560137e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:45.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.533+0000 7f155dd9b640 1 --2- 172.21.15.40:0/58952372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f156009c940 0x7f1560137e30 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f15480097a0 tx=0x7f154802b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a02ec3503b8c93f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:45.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.533+0000 7f155d59a640 1 -- 172.21.15.40:0/58952372 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1548030a00 con 0x7f156009c940 2024-04-22T08:22:45.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.533+0000 7f155d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:45.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.533+0000 7f155d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:22:45.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.533+0000 7f155d59a640 10 monclient: dump: 2024-04-22T08:22:45.537 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:45.537 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:45.537 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:45.538 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:45.538 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:45.538 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:45.538 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:45.538 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:45.538 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:45.538 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:45.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.533+0000 7f155d59a640 10 monclient: _finish_auth 0 2024-04-22T08:22:45.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.533+0000 7f155d59a640 10 monclient: _check_auth_tickets 2024-04-22T08:22:45.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.533+0000 7f155d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:45.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.533+0000 7f155d59a640 1 -- 172.21.15.40:0/58952372 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f154802e050 con 0x7f156009c940 2024-04-22T08:22:45.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.533+0000 7f155d59a640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:45.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.533+0000 7f155d59a640 1 -- 172.21.15.40:0/58952372 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1548030d80 con 0x7f156009c940 2024-04-22T08:22:45.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.533+0000 7f1564ab6640 5 monclient: authenticate success, global_id 65284 2024-04-22T08:22:45.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.533+0000 7f1564ab6640 10 monclient: get_monmap_and_config success 2024-04-22T08:22:45.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.533+0000 7f155d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:45.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.533+0000 7f1564ab6640 4 set_mon_vals no callback set 2024-04-22T08:22:45.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.533+0000 7f155d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:22:45.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.533+0000 7f155d59a640 10 monclient: dump: 2024-04-22T08:22:45.539 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:45.539 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:45.539 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:45.539 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:45.539 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:45.539 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:45.539 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:45.539 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:45.539 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:45.540 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:45.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.533+0000 7f1564ab6640 10 monclient: shutdown 2024-04-22T08:22:45.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.534+0000 7f1564ab6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:45.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.534+0000 7f1564ab6640 1 -- 172.21.15.40:0/58952372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f156009c940 msgr2=0x7f1560137e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:45.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.534+0000 7f1564ab6640 1 --2- 172.21.15.40:0/58952372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f156009c940 0x7f1560137e30 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f15480097a0 tx=0x7f154802b710 comp rx=0 tx=0).stop 2024-04-22T08:22:45.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.534+0000 7f1564ab6640 1 -- 172.21.15.40:0/58952372 shutdown_connections 2024-04-22T08:22:45.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.534+0000 7f1564ab6640 1 --2- 172.21.15.40:0/58952372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f156009c940 0x7f1560137e30 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:45.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.534+0000 7f1564ab6640 1 -- 172.21.15.40:0/58952372 >> 172.21.15.40:0/58952372 conn(0x7f15600a0d30 msgr2=0x7f15600a1140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:45.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.534+0000 7f1564ab6640 1 -- 172.21.15.40:0/58952372 shutdown_connections 2024-04-22T08:22:45.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.534+0000 7f1564ab6640 1 -- 172.21.15.40:0/58952372 wait complete. 2024-04-22T08:22:45.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.534+0000 7f1564ab6640 10 monclient: build_initial_monmap 2024-04-22T08:22:45.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.534+0000 7f1564ab6640 10 monclient: monmap: 2024-04-22T08:22:45.540 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:45.540 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:22:45.541 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:22:45.541 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:22:45.541 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:45.541 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:45.541 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:22:45.541 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:22:45.541 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:22:45.541 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:45.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.535+0000 7f1564ab6640 1 Processor -- start 2024-04-22T08:22:45.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.535+0000 7f1564ab6640 1 -- start start 2024-04-22T08:22:45.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.535+0000 7f1564ab6640 10 monclient: init 2024-04-22T08:22:45.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.535+0000 7f1564ab6640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:45.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.535+0000 7f1564ab6640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-22T08:22:45.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.535+0000 7f1564ab6640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1560132900 0x7f156013a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:45.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.535+0000 7f1564ab6640 10 monclient(hunting): picked mon.noname-b con 0x7f1560132900 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:22:45.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.535+0000 7f1564ab6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15601333f0 0x7f156013aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:45.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.535+0000 7f1564ab6640 10 monclient(hunting): picked mon.noname-a con 0x7f15601333f0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:22:45.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.535+0000 7f1564ab6640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1560130b20 0x7f156012f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:45.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.535+0000 7f1564ab6640 10 monclient(hunting): picked mon.noname-c con 0x7f1560130b20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:22:45.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.535+0000 7f1564ab6640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:45.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.535+0000 7f1564ab6640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15601320d0 con 0x7f15601333f0 2024-04-22T08:22:45.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.535+0000 7f1564ab6640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:45.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.535+0000 7f1564ab6640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1560071800 con 0x7f1560132900 2024-04-22T08:22:45.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.535+0000 7f1564ab6640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:45.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.535+0000 7f1564ab6640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1560070e80 con 0x7f1560130b20 2024-04-22T08:22:45.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.535+0000 7f1564ab6640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:45.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.535+0000 7f1564ab6640 10 monclient(hunting): authenticate will time out at 1439.529606s 2024-04-22T08:22:45.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.535+0000 7f155dd9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15601333f0 0x7f156013aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:45.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.535+0000 7f155dd9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15601333f0 0x7f156013aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44594/0 (socket says 172.21.15.40:44594) 2024-04-22T08:22:45.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.535+0000 7f155dd9b640 1 -- 172.21.15.40:0/464612968 learned_addr learned my addr 172.21.15.40:0/464612968 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:22:45.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.535+0000 7f155dd9b640 10 monclient(hunting): get_auth_request con 0x7f15601333f0 auth_method 0 2024-04-22T08:22:45.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.535+0000 7f155ed9d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1560130b20 0x7f156012f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:45.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.535+0000 7f155dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:45.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.535+0000 7f155dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:45.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.535+0000 7f155dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:45.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.536+0000 7f155ed9d640 10 monclient(hunting): get_auth_request con 0x7f1560130b20 auth_method 0 2024-04-22T08:22:45.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.536+0000 7f155ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:45.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.536+0000 7f155ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:45.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.536+0000 7f155ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:45.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.536+0000 7f155dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:45.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.536+0000 7f155dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:45.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.536+0000 7f155dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:45.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.536+0000 7f155ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:45.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.536+0000 7f155ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:45.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.536+0000 7f155ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:45.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.536+0000 7f155e59c640 1 --2- 172.21.15.40:0/464612968 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1560132900 0x7f156013a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:45.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.536+0000 7f155e59c640 10 monclient(hunting): get_auth_request con 0x7f1560132900 auth_method 0 2024-04-22T08:22:45.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.536+0000 7f155e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:45.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.536+0000 7f155e59c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:45.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.536+0000 7f155e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:45.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.536+0000 7f155dd9b640 10 monclient(hunting): handle_auth_done global_id 114321 payload 824 2024-04-22T08:22:45.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.536+0000 7f155dd9b640 1 -- 172.21.15.40:0/464612968 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1560130b20 msgr2=0x7f156012f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:45.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.536+0000 7f155dd9b640 1 --2- 172.21.15.40:0/464612968 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1560130b20 0x7f156012f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:45.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.536+0000 7f155dd9b640 1 -- 172.21.15.40:0/464612968 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1560132900 msgr2=0x7f156013a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:45.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.536+0000 7f155dd9b640 1 --2- 172.21.15.40:0/464612968 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1560132900 0x7f156013a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:45.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.536+0000 7f155dd9b640 10 monclient: _finish_hunting 0 2024-04-22T08:22:45.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.536+0000 7f155dd9b640 1 monclient: found mon.noname-a 2024-04-22T08:22:45.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.536+0000 7f155dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:45.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.536+0000 7f155dd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:22:45.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.536+0000 7f155dd9b640 1 -- 172.21.15.40:0/464612968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1548009400 con 0x7f15601333f0 2024-04-22T08:22:45.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.536+0000 7f155ed9d640 1 --2- 172.21.15.40:0/464612968 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1560130b20 0x7f156012f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:22:45.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.536+0000 7f155e59c640 1 --2- 172.21.15.40:0/464612968 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1560132900 0x7f156013a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T08:22:45.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.536+0000 7f155dd9b640 1 -- 172.21.15.40:0/464612968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15601333f0 msgr2=0x7f156013aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-22T08:22:45.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.536+0000 7f155dd9b640 1 -- 172.21.15.40:0/464612968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15601333f0 msgr2=0x7f156013aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:22:45.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.536+0000 7f155dd9b640 1 --2- 172.21.15.40:0/464612968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15601333f0 0x7f156013aea0 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f15480098c0 tx=0x7f15480046c0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:22:45.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.536+0000 7f155dd9b640 1 --2- 172.21.15.40:0/464612968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15601333f0 0x7f156013aea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:22:45.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f155dd9b640 1 --2- 172.21.15.40:0/464612968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15601333f0 0x7f156013aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:45.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f155dd9b640 10 monclient: get_auth_request con 0x7f15601333f0 auth_method 0 2024-04-22T08:22:45.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f155dd9b640 0 --2- 172.21.15.40:0/464612968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15601333f0 0x7f156013aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-04-22T08:22:45.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f155dd9b640 1 --2- 172.21.15.40:0/464612968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15601333f0 0x7f156013aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:45.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f1556ffd640 10 monclient: ms_handle_reset current mon [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:22:45.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f1556ffd640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:45.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f1556ffd640 1 -- 172.21.15.40:0/464612968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15601333f0 msgr2=0x7f156013aea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:22:45.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f1556ffd640 1 --2- 172.21.15.40:0/464612968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15601333f0 0x7f156013aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:45.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f1556ffd640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-22T08:22:45.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f1556ffd640 1 --2- 172.21.15.40:0/464612968 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1560132900 0x7f15400054e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:45.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f1556ffd640 10 monclient(hunting): picked mon.noname-b con 0x7f1560132900 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:22:45.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f1556ffd640 1 --2- 172.21.15.40:0/464612968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1540005f20 0x7f15400083b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:45.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f1556ffd640 10 monclient(hunting): picked mon.noname-a con 0x7f1540005f20 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:22:45.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f1556ffd640 1 --2- 172.21.15.40:0/464612968 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1560130b20 0x7f154000af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:45.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f1556ffd640 10 monclient(hunting): picked mon.noname-c con 0x7f1560130b20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:22:45.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f1556ffd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:45.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f1556ffd640 1 -- 172.21.15.40:0/464612968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f154000b940 con 0x7f1540005f20 2024-04-22T08:22:45.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f1556ffd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:45.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f1556ffd640 1 -- 172.21.15.40:0/464612968 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f154000ba80 con 0x7f1560132900 2024-04-22T08:22:45.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f1556ffd640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:45.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f1556ffd640 1 -- 172.21.15.40:0/464612968 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f154000bc40 con 0x7f1560130b20 2024-04-22T08:22:45.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f1556ffd640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:45.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f155dd9b640 1 --2- 172.21.15.40:0/464612968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1540005f20 0x7f15400083b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:45.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f155dd9b640 10 monclient(hunting): get_auth_request con 0x7f1540005f20 auth_method 0 2024-04-22T08:22:45.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f155dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:45.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f155dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:45.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f155dd9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:22:45.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f155ed9d640 1 --2- 172.21.15.40:0/464612968 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1560130b20 0x7f154000af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:45.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f155ed9d640 10 monclient(hunting): get_auth_request con 0x7f1560130b20 auth_method 0 2024-04-22T08:22:45.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f155ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:45.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f155ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:45.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f155ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:22:45.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f155dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:45.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f155dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:45.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f155dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-22T08:22:45.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f155ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:45.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f155ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:45.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f155ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-22T08:22:45.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f155e59c640 1 --2- 172.21.15.40:0/464612968 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1560132900 0x7f15400054e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:45.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f155e59c640 10 monclient(hunting): get_auth_request con 0x7f1560132900 auth_method 0 2024-04-22T08:22:45.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f155e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:45.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f155e59c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:45.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f155e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:22:45.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f155dd9b640 10 monclient(hunting): handle_auth_done global_id 114321 payload 293 2024-04-22T08:22:45.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f155dd9b640 1 -- 172.21.15.40:0/464612968 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1560130b20 msgr2=0x7f154000af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:45.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f155dd9b640 1 --2- 172.21.15.40:0/464612968 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1560130b20 0x7f154000af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:45.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f155dd9b640 1 -- 172.21.15.40:0/464612968 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1560132900 msgr2=0x7f15400054e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:45.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f155dd9b640 1 --2- 172.21.15.40:0/464612968 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1560132900 0x7f15400054e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:45.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f155dd9b640 10 monclient: _finish_hunting 0 2024-04-22T08:22:45.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f155dd9b640 1 monclient: found mon.noname-a 2024-04-22T08:22:45.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f155dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:45.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f155dd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:22:45.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.737+0000 7f155dd9b640 1 -- 172.21.15.40:0/464612968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f154000bec0 con 0x7f1540005f20 2024-04-22T08:22:45.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.739+0000 7f155dd9b640 1 -- 172.21.15.40:0/464612968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1540005f20 msgr2=0x7f15400083b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:45.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.739+0000 7f155dd9b640 1 --2- 172.21.15.40:0/464612968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1540005f20 0x7f15400083b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f15480098c0 tx=0x7f154802f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:45.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.740+0000 7f1556ffd640 1 -- 172.21.15.40:0/464612968 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f154803dcf0 con 0x7f1540005f20 2024-04-22T08:22:45.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.740+0000 7f1556ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:45.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.740+0000 7f1556ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:22:45.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.740+0000 7f1556ffd640 10 monclient: dump: 2024-04-22T08:22:45.745 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:45.745 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:45.745 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:45.745 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:45.745 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:45.746 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:45.746 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:45.746 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:45.746 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:45.746 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:45.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.740+0000 7f1556ffd640 10 monclient: _finish_auth 0 2024-04-22T08:22:45.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.740+0000 7f1556ffd640 10 monclient: _check_auth_tickets 2024-04-22T08:22:45.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.740+0000 7f1556ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:45.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.740+0000 7f1564ab6640 5 monclient: authenticate success, global_id 114321 2024-04-22T08:22:45.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.740+0000 7f1556ffd640 1 -- 172.21.15.40:0/464612968 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f154803de70 con 0x7f1540005f20 2024-04-22T08:22:45.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.740+0000 7f1564ab6640 10 monclient: _renew_subs 2024-04-22T08:22:45.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.740+0000 7f1564ab6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:45.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.740+0000 7f1564ab6640 1 -- 172.21.15.40:0/464612968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f156012db80 con 0x7f1540005f20 2024-04-22T08:22:45.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.740+0000 7f1556ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:45.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.740+0000 7f1556ffd640 1 -- 172.21.15.40:0/464612968 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15480326a0 con 0x7f1540005f20 2024-04-22T08:22:45.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.740+0000 7f1556ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:45.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.740+0000 7f1556ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:22:45.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.740+0000 7f1556ffd640 10 monclient: dump: 2024-04-22T08:22:45.747 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:45.747 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:45.747 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:45.747 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:45.747 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:45.747 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:45.747 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:45.747 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:45.747 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:45.748 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:45.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.740+0000 7f1557fff640 4 set_mon_vals no callback set 2024-04-22T08:22:45.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.740+0000 7f1564ab6640 10 monclient: _renew_subs 2024-04-22T08:22:45.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.740+0000 7f1564ab6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:45.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.740+0000 7f1564ab6640 1 -- 172.21.15.40:0/464612968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15601c8e30 con 0x7f1540005f20 2024-04-22T08:22:45.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.740+0000 7f1564ab6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:45.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.740+0000 7f1564ab6640 1 -- 172.21.15.40:0/464612968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f156009e2a0 con 0x7f1540005f20 2024-04-22T08:22:45.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.741+0000 7f1556ffd640 1 -- 172.21.15.40:0/464612968 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f1548039040 con 0x7f1540005f20 2024-04-22T08:22:45.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.741+0000 7f1556ffd640 1 --2- 172.21.15.40:0/464612968 >> v2:172.21.15.189:6800/30168 conn(0x7f15601333f0 0x7f1540041f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:45.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.741+0000 7f1556ffd640 1 -- 172.21.15.40:0/464612968 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(317..317 src has 239..317) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f154806c260 con 0x7f1540005f20 2024-04-22T08:22:45.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.741+0000 7f1556ffd640 1 --2- 172.21.15.40:0/464612968 >> v2:172.21.15.40:6808/30671 conn(0x7f1540045470 0x7f1540047890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:45.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.741+0000 7f1556ffd640 1 -- 172.21.15.40:0/464612968 --> v2:172.21.15.40:6808/30671 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1540047f00 con 0x7f1540045470 2024-04-22T08:22:45.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.741+0000 7f1556ffd640 1 -- 172.21.15.40:0/464612968 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=317) v2 ==== 24+0+0 (secure 0 0 0) 0x7f154806e7f0 con 0x7f1540005f20 2024-04-22T08:22:45.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.741+0000 7f1556ffd640 10 monclient: handle_get_version_reply finishing 1 version 317 2024-04-22T08:22:45.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.741+0000 7f155ed9d640 1 --2- 172.21.15.40:0/464612968 >> v2:172.21.15.40:6808/30671 conn(0x7f1540045470 0x7f1540047890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:45.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.741+0000 7f155e59c640 1 --2- 172.21.15.40:0/464612968 >> v2:172.21.15.189:6800/30168 conn(0x7f15601333f0 0x7f1540041f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:45.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.741+0000 7f155ed9d640 10 monclient: get_auth_request con 0x7f1540045470 auth_method 0 2024-04-22T08:22:45.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.741+0000 7f155e59c640 10 monclient: get_auth_request con 0x7f15601333f0 auth_method 0 2024-04-22T08:22:45.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.744+0000 7f155e59c640 1 --2- 172.21.15.40:0/464612968 >> v2:172.21.15.189:6800/30168 conn(0x7f15601333f0 0x7f1540041f20 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f1558004ec0 tx=0x7f1558009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:45.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.746+0000 7f155ed9d640 1 --2- 172.21.15.40:0/464612968 >> v2:172.21.15.40:6808/30671 conn(0x7f1540045470 0x7f1540047890 crc :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:45.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.747+0000 7f1556ffd640 1 -- 172.21.15.40:0/464612968 <== osd.2 v2:172.21.15.40:6808/30671 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f1540047f00 con 0x7f1540045470 2024-04-22T08:22:45.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.759+0000 7f1564ab6640 1 -- 172.21.15.40:0/464612968 --> v2:172.21.15.40:6808/30671 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f15601c8dc0 con 0x7f1540045470 2024-04-22T08:22:45.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.760+0000 7f1556ffd640 1 -- 172.21.15.40:0/464612968 <== osd.2 v2:172.21.15.40:6808/30671 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f15601c8dc0 con 0x7f1540045470 2024-04-22T08:22:45.759 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:22:45.759 INFO:teuthology.orchestra.run.smithi040.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-22T08:22:45.760 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:22:45.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.760+0000 7f1564ab6640 1 -- 172.21.15.40:0/464612968 >> v2:172.21.15.40:6808/30671 conn(0x7f1540045470 msgr2=0x7f1540047890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:45.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.760+0000 7f1564ab6640 1 --2- 172.21.15.40:0/464612968 >> v2:172.21.15.40:6808/30671 conn(0x7f1540045470 0x7f1540047890 crc :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:45.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.760+0000 7f1564ab6640 1 -- 172.21.15.40:0/464612968 >> v2:172.21.15.189:6800/30168 conn(0x7f15601333f0 msgr2=0x7f1540041f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:45.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.760+0000 7f1564ab6640 1 --2- 172.21.15.40:0/464612968 >> v2:172.21.15.189:6800/30168 conn(0x7f15601333f0 0x7f1540041f20 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f1558004ec0 tx=0x7f1558009040 comp rx=0 tx=0).stop 2024-04-22T08:22:45.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.760+0000 7f1564ab6640 10 monclient: shutdown 2024-04-22T08:22:45.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.760+0000 7f1564ab6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:45.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.760+0000 7f1564ab6640 1 -- 172.21.15.40:0/464612968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1540005f20 msgr2=0x7f15400083b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:45.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.760+0000 7f1564ab6640 1 --2- 172.21.15.40:0/464612968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1540005f20 0x7f15400083b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f15480098c0 tx=0x7f154802f000 comp rx=0 tx=0).stop 2024-04-22T08:22:45.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.760+0000 7f155ed9d640 1 -- 172.21.15.40:0/464612968 reap_dead start 2024-04-22T08:22:45.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.760+0000 7f1564ab6640 1 -- 172.21.15.40:0/464612968 shutdown_connections 2024-04-22T08:22:45.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.760+0000 7f1564ab6640 1 -- 172.21.15.40:0/464612968 >> 172.21.15.40:0/464612968 conn(0x7f15600a0d30 msgr2=0x7f156009f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:45.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.760+0000 7f1564ab6640 1 -- 172.21.15.40:0/464612968 shutdown_connections 2024-04-22T08:22:45.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.760+0000 7f1564ab6640 1 -- 172.21.15.40:0/464612968 wait complete. 2024-04-22T08:22:45.770 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-22T08:22:45.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.870+0000 7f393111d640 10 monclient: get_monmap_and_config 2024-04-22T08:22:45.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.870+0000 7f393111d640 10 monclient: build_initial_monmap 2024-04-22T08:22:45.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.870+0000 7f393111d640 10 monclient: monmap: 2024-04-22T08:22:45.870 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:45.870 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:45.870 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:22:45.871334+0000 2024-04-22T08:22:45.870 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:22:45.871334+0000 2024-04-22T08:22:45.870 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:45.870 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:45.870 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:22:45.870 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:22:45.871 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:22:45.871 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:22:45.871 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:22:45.871 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:45.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.871+0000 7f393111d640 1 Processor -- start 2024-04-22T08:22:45.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.871+0000 7f393111d640 1 -- start start 2024-04-22T08:22:45.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.871+0000 7f393111d640 10 monclient: init 2024-04-22T08:22:45.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.871+0000 7f393111d640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:45.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.871+0000 7f393111d640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-04-22T08:22:45.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.871+0000 7f393111d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f392c070e00 addr v1:172.21.15.40:6789/0 2024-04-22T08:22:45.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.871+0000 7f393111d640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f392c0a0690 0x7f392c13ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:45.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.871+0000 7f393111d640 10 monclient(hunting): picked mon.noname-a con 0x7f392c0a0690 addr v2:172.21.15.40:3300/0 2024-04-22T08:22:45.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.871+0000 7f393111d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f392c13b260 addr v1:172.21.15.189:6789/0 2024-04-22T08:22:45.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.871+0000 7f393111d640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f392c09e840 con 0x7f392c070e00 2024-04-22T08:22:45.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.871+0000 7f393111d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f392c06d170 con 0x7f392c13b260 2024-04-22T08:22:45.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.871+0000 7f393111d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:45.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.871+0000 7f393111d640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f392c06d3c0 con 0x7f392c0a0690 2024-04-22T08:22:45.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.871+0000 7f393111d640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:45.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.871+0000 7f393111d640 10 monclient(hunting): authenticate will time out at 1439.865425s 2024-04-22T08:22:45.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.871+0000 7f392a59c640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f392c0a0690 0x7f392c13ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:45.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.871+0000 7f392ad9d640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f392c070e00 0x7f392c0711e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:49360/0 (socket says 172.21.15.40:49360) 2024-04-22T08:22:45.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.871+0000 7f392ad9d640 1 -- 172.21.15.40:0/2122602788 learned_addr learned my addr 172.21.15.40:0/2122602788 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:22:45.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.871+0000 7f392a59c640 10 monclient(hunting): get_auth_request con 0x7f392c0a0690 auth_method 0 2024-04-22T08:22:45.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.871+0000 7f392a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:45.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.871+0000 7f392a59c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:45.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.871+0000 7f392a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:45.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.871+0000 7f392a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:45.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.871+0000 7f392a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:45.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.871+0000 7f392a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:45.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.872+0000 7f392a59c640 10 monclient(hunting): handle_auth_done global_id 114324 payload 274 2024-04-22T08:22:45.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.872+0000 7f392a59c640 1 -- 172.21.15.40:0/2122602788 >> v1:172.21.15.189:6789/0 conn(0x7f392c13b260 legacy=0x7f392c13d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:45.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.872+0000 7f392a59c640 1 -- 172.21.15.40:0/2122602788 >> v1:172.21.15.40:6789/0 conn(0x7f392c070e00 legacy=0x7f392c0711e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:45.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.872+0000 7f392a59c640 10 monclient: _finish_hunting 0 2024-04-22T08:22:45.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.872+0000 7f392a59c640 1 monclient: found mon.noname-a 2024-04-22T08:22:45.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.872+0000 7f392a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:45.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.872+0000 7f392a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:22:45.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.872+0000 7f392a59c640 1 -- 172.21.15.40:0/2122602788 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f392c06d6a0 con 0x7f392c0a0690 2024-04-22T08:22:45.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.874+0000 7f392a59c640 1 -- 172.21.15.40:0/2122602788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f392c0a0690 msgr2=0x7f392c13ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:45.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.874+0000 7f392a59c640 1 --2- 172.21.15.40:0/2122602788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f392c0a0690 0x7f392c13ad20 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f39200097a0 tx=0x7f392002b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1592cee45c88015 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:45.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.875+0000 7f3929d9b640 1 -- 172.21.15.40:0/2122602788 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3920030a00 con 0x7f392c0a0690 2024-04-22T08:22:45.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.875+0000 7f3929d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:45.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.875+0000 7f3929d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:22:45.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.875+0000 7f3929d9b640 10 monclient: dump: 2024-04-22T08:22:45.879 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:45.879 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:45.879 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:45.879 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:45.879 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:45.879 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:45.880 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:45.880 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:45.880 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:45.880 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:45.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.875+0000 7f3929d9b640 10 monclient: _finish_auth 0 2024-04-22T08:22:45.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.875+0000 7f3929d9b640 10 monclient: _check_auth_tickets 2024-04-22T08:22:45.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.875+0000 7f3929d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:45.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.875+0000 7f3929d9b640 1 -- 172.21.15.40:0/2122602788 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f392002e050 con 0x7f392c0a0690 2024-04-22T08:22:45.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.875+0000 7f3929d9b640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:45.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.875+0000 7f3929d9b640 1 -- 172.21.15.40:0/2122602788 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3920030d80 con 0x7f392c0a0690 2024-04-22T08:22:45.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.875+0000 7f393111d640 5 monclient: authenticate success, global_id 114324 2024-04-22T08:22:45.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.875+0000 7f393111d640 10 monclient: get_monmap_and_config success 2024-04-22T08:22:45.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.875+0000 7f393111d640 4 set_mon_vals no callback set 2024-04-22T08:22:45.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.875+0000 7f3929d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:45.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.875+0000 7f3929d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:22:45.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.875+0000 7f3929d9b640 10 monclient: dump: 2024-04-22T08:22:45.881 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:45.881 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:45.881 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:45.881 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:45.881 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:45.881 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:45.881 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:45.881 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:45.881 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:45.881 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:45.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.875+0000 7f393111d640 10 monclient: shutdown 2024-04-22T08:22:45.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.875+0000 7f393111d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:45.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.875+0000 7f393111d640 1 -- 172.21.15.40:0/2122602788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f392c0a0690 msgr2=0x7f392c13ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:45.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.875+0000 7f393111d640 1 --2- 172.21.15.40:0/2122602788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f392c0a0690 0x7f392c13ad20 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f39200097a0 tx=0x7f392002b710 comp rx=0 tx=0).stop 2024-04-22T08:22:45.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.875+0000 7f393111d640 1 -- 172.21.15.40:0/2122602788 shutdown_connections 2024-04-22T08:22:45.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.875+0000 7f393111d640 1 --2- 172.21.15.40:0/2122602788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f392c0a0690 0x7f392c13ad20 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:45.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.875+0000 7f393111d640 1 -- 172.21.15.40:0/2122602788 >> 172.21.15.40:0/2122602788 conn(0x7f392c0a18f0 msgr2=0x7f392c06ba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:45.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.875+0000 7f393111d640 1 -- 172.21.15.40:0/2122602788 shutdown_connections 2024-04-22T08:22:45.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.875+0000 7f393111d640 1 -- 172.21.15.40:0/2122602788 wait complete. 2024-04-22T08:22:45.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.876+0000 7f393111d640 10 monclient: build_initial_monmap 2024-04-22T08:22:45.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.876+0000 7f393111d640 10 monclient: monmap: 2024-04-22T08:22:45.882 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:45.882 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:22:45.882 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:22:45.882 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:22:45.883 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:45.883 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:45.883 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:22:45.883 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:22:45.883 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:22:45.883 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:45.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.877+0000 7f393111d640 1 Processor -- start 2024-04-22T08:22:45.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.877+0000 7f393111d640 1 -- start start 2024-04-22T08:22:45.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.877+0000 7f393111d640 10 monclient: init 2024-04-22T08:22:45.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.877+0000 7f393111d640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:45.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.877+0000 7f393111d640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-22T08:22:45.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.877+0000 7f393111d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f392c070e00 0x7f392c1ca850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:45.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.877+0000 7f393111d640 10 monclient(hunting): picked mon.noname-a con 0x7f392c070e00 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:22:45.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.877+0000 7f393111d640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f392c0a0690 0x7f392c12e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:45.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.877+0000 7f393111d640 10 monclient(hunting): picked mon.noname-c con 0x7f392c0a0690 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:22:45.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.877+0000 7f393111d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f392c13b260 0x7f392c12e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:45.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.877+0000 7f393111d640 10 monclient(hunting): picked mon.noname-b con 0x7f392c13b260 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:22:45.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.877+0000 7f393111d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:45.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.877+0000 7f393111d640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f392c1cad90 con 0x7f392c070e00 2024-04-22T08:22:45.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.877+0000 7f393111d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:45.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.877+0000 7f393111d640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f392c09e9e0 con 0x7f392c13b260 2024-04-22T08:22:45.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.877+0000 7f393111d640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:45.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.877+0000 7f393111d640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f392c130aa0 con 0x7f392c0a0690 2024-04-22T08:22:45.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.877+0000 7f393111d640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:45.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.877+0000 7f393111d640 10 monclient(hunting): authenticate will time out at 1439.871368s 2024-04-22T08:22:45.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.877+0000 7f392ad9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f392c070e00 0x7f392c1ca850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:45.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.877+0000 7f392ad9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f392c070e00 0x7f392c1ca850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44632/0 (socket says 172.21.15.40:44632) 2024-04-22T08:22:45.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.877+0000 7f392ad9d640 1 -- 172.21.15.40:0/3772608300 learned_addr learned my addr 172.21.15.40:0/3772608300 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:22:45.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.877+0000 7f392ad9d640 10 monclient(hunting): get_auth_request con 0x7f392c070e00 auth_method 0 2024-04-22T08:22:45.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.877+0000 7f392ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:45.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.877+0000 7f392ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:45.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.877+0000 7f392ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:45.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.877+0000 7f392a59c640 1 --2- 172.21.15.40:0/3772608300 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f392c0a0690 0x7f392c12e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:45.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.877+0000 7f392ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:45.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.877+0000 7f392ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:45.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.877+0000 7f392ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:45.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.877+0000 7f392a59c640 10 monclient(hunting): get_auth_request con 0x7f392c0a0690 auth_method 0 2024-04-22T08:22:45.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.877+0000 7f392a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:45.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.877+0000 7f392a59c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:45.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.877+0000 7f392a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:45.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.877+0000 7f392a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:45.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.877+0000 7f392a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:45.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.877+0000 7f392a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:45.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.877+0000 7f392ad9d640 10 monclient(hunting): handle_auth_done global_id 114327 payload 824 2024-04-22T08:22:45.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.877+0000 7f392ad9d640 1 -- 172.21.15.40:0/3772608300 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f392c0a0690 msgr2=0x7f392c12e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:45.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.877+0000 7f392ad9d640 1 --2- 172.21.15.40:0/3772608300 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f392c0a0690 0x7f392c12e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:45.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.877+0000 7f392b59e640 1 --2- 172.21.15.40:0/3772608300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f392c13b260 0x7f392c12e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:45.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.877+0000 7f392ad9d640 1 -- 172.21.15.40:0/3772608300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f392c13b260 msgr2=0x7f392c12e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:45.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.877+0000 7f392ad9d640 1 --2- 172.21.15.40:0/3772608300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f392c13b260 0x7f392c12e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:45.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.877+0000 7f392ad9d640 10 monclient: _finish_hunting 0 2024-04-22T08:22:45.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.877+0000 7f392ad9d640 1 monclient: found mon.noname-a 2024-04-22T08:22:45.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.878+0000 7f392ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:45.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.878+0000 7f392ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:22:45.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.878+0000 7f392ad9d640 1 -- 172.21.15.40:0/3772608300 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3920009400 con 0x7f392c070e00 2024-04-22T08:22:45.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.878+0000 7f392b59e640 10 monclient: get_auth_request con 0x7f392c13b260 auth_method 0 2024-04-22T08:22:45.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.878+0000 7f392b59e640 1 --2- 172.21.15.40:0/3772608300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f392c13b260 0x7f392c12e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T08:22:45.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.880+0000 7f392ad9d640 1 -- 172.21.15.40:0/3772608300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f392c070e00 msgr2=0x7f392c1ca850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:45.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.880+0000 7f392ad9d640 1 --2- 172.21.15.40:0/3772608300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f392c070e00 0x7f392c1ca850 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f391c00cc10 tx=0x7f391c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:45.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.880+0000 7f39137fe640 1 -- 172.21.15.40:0/3772608300 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f391c012400 con 0x7f392c070e00 2024-04-22T08:22:45.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.880+0000 7f39137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:45.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.880+0000 7f39137fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:22:45.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.880+0000 7f39137fe640 10 monclient: dump: 2024-04-22T08:22:45.889 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:45.889 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:45.889 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:45.889 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:45.889 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:45.889 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:45.889 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:45.889 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:45.890 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:45.890 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:45.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.880+0000 7f39137fe640 10 monclient: _finish_auth 0 2024-04-22T08:22:45.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.880+0000 7f39137fe640 10 monclient: _check_auth_tickets 2024-04-22T08:22:45.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.880+0000 7f39137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:45.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.880+0000 7f39137fe640 1 -- 172.21.15.40:0/3772608300 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f391c0125a0 con 0x7f392c070e00 2024-04-22T08:22:45.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.880+0000 7f39137fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:45.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.880+0000 7f39137fe640 1 -- 172.21.15.40:0/3772608300 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f391c012950 con 0x7f392c070e00 2024-04-22T08:22:45.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.880+0000 7f393111d640 5 monclient: authenticate success, global_id 114327 2024-04-22T08:22:45.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.880+0000 7f393111d640 10 monclient: _renew_subs 2024-04-22T08:22:45.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.880+0000 7f393111d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:45.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.880+0000 7f393111d640 1 -- 172.21.15.40:0/3772608300 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f392c139490 con 0x7f392c070e00 2024-04-22T08:22:45.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.880+0000 7f39137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:45.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.880+0000 7f39137fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:22:45.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.880+0000 7f39137fe640 10 monclient: dump: 2024-04-22T08:22:45.891 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:45.891 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:45.891 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:45.891 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:45.891 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:45.891 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:45.891 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:45.891 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:45.891 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:45.891 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:45.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.880+0000 7f392959a640 4 set_mon_vals no callback set 2024-04-22T08:22:45.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.880+0000 7f393111d640 10 monclient: _renew_subs 2024-04-22T08:22:45.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.880+0000 7f393111d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:45.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.880+0000 7f393111d640 1 -- 172.21.15.40:0/3772608300 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f392c138480 con 0x7f392c070e00 2024-04-22T08:22:45.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.881+0000 7f39137fe640 1 -- 172.21.15.40:0/3772608300 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f391c012af0 con 0x7f392c070e00 2024-04-22T08:22:45.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.881+0000 7f393111d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:45.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.881+0000 7f393111d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:45.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.881+0000 7f393111d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:45.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.881+0000 7f393111d640 1 -- 172.21.15.40:0/3772608300 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f392c04e9b0 con 0x7f392c070e00 2024-04-22T08:22:45.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.881+0000 7f39137fe640 1 --2- 172.21.15.40:0/3772608300 >> v2:172.21.15.189:6800/30168 conn(0x7f39180377d0 0x7f3918039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:45.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.881+0000 7f39137fe640 1 -- 172.21.15.40:0/3772608300 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(317..317 src has 239..317) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f391c04a8b0 con 0x7f392c070e00 2024-04-22T08:22:45.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.882+0000 7f392a59c640 1 --2- 172.21.15.40:0/3772608300 >> v2:172.21.15.189:6800/30168 conn(0x7f39180377d0 0x7f3918039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:45.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.882+0000 7f392a59c640 10 monclient: get_auth_request con 0x7f39180377d0 auth_method 0 2024-04-22T08:22:45.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.884+0000 7f392a59c640 1 --2- 172.21.15.40:0/3772608300 >> v2:172.21.15.189:6800/30168 conn(0x7f39180377d0 0x7f3918039c90 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f3920032000 tx=0x7f392002f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:45.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.886+0000 7f39137fe640 1 -- 172.21.15.40:0/3772608300 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f391c0055b0 con 0x7f392c070e00 2024-04-22T08:22:45.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.886+0000 7f39137fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:45.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:45.886+0000 7f39137fe640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:22:46.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.064+0000 7f393111d640 1 -- 172.21.15.40:0/3772608300 --> v2:172.21.15.189:6800/30168 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f392c09e230 con 0x7f39180377d0 2024-04-22T08:22:46.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.064+0000 7f39137fe640 1 -- 172.21.15.40:0/3772608300 <== mgr.4102 v2:172.21.15.189:6800/30168 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f3920030bc0 con 0x7f39180377d0 2024-04-22T08:22:46.064 INFO:teuthology.orchestra.run.smithi040.stderr:instructed osd(s) 2 to deep-scrub 2024-04-22T08:22:46.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.066+0000 7f393111d640 1 -- 172.21.15.40:0/3772608300 >> v2:172.21.15.189:6800/30168 conn(0x7f39180377d0 msgr2=0x7f3918039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:46.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.066+0000 7f393111d640 1 --2- 172.21.15.40:0/3772608300 >> v2:172.21.15.189:6800/30168 conn(0x7f39180377d0 0x7f3918039c90 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f3920032000 tx=0x7f392002f000 comp rx=0 tx=0).stop 2024-04-22T08:22:46.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.066+0000 7f393111d640 10 monclient: shutdown 2024-04-22T08:22:46.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.066+0000 7f393111d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:46.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.066+0000 7f393111d640 1 -- 172.21.15.40:0/3772608300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f392c070e00 msgr2=0x7f392c1ca850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:46.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.066+0000 7f393111d640 1 --2- 172.21.15.40:0/3772608300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f392c070e00 0x7f392c1ca850 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f391c00cc10 tx=0x7f391c00a430 comp rx=0 tx=0).stop 2024-04-22T08:22:46.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.066+0000 7f393111d640 1 -- 172.21.15.40:0/3772608300 shutdown_connections 2024-04-22T08:22:46.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.066+0000 7f393111d640 1 --2- 172.21.15.40:0/3772608300 >> v2:172.21.15.189:6800/30168 conn(0x7f39180377d0 0x7f3918039c90 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:46.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.067+0000 7f393111d640 1 --2- 172.21.15.40:0/3772608300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f392c13b260 0x7f392c12e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:46.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.067+0000 7f393111d640 1 --2- 172.21.15.40:0/3772608300 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f392c0a0690 0x7f392c12e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:46.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.067+0000 7f393111d640 1 --2- 172.21.15.40:0/3772608300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f392c070e00 0x7f392c1ca850 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:46.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.067+0000 7f393111d640 1 -- 172.21.15.40:0/3772608300 >> 172.21.15.40:0/3772608300 conn(0x7f392c0a18f0 msgr2=0x7f392c132630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:46.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.067+0000 7f393111d640 1 -- 172.21.15.40:0/3772608300 shutdown_connections 2024-04-22T08:22:46.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.067+0000 7f393111d640 1 -- 172.21.15.40:0/3772608300 wait complete. 2024-04-22T08:22:46.076 INFO:tasks.ceph:Scrubbing osd.3 2024-04-22T08:22:46.077 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-22T08:22:46.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.179+0000 7f3186a8f640 10 monclient: get_monmap_and_config 2024-04-22T08:22:46.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.179+0000 7f3186a8f640 10 monclient: build_initial_monmap 2024-04-22T08:22:46.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.179+0000 7f3186a8f640 10 monclient: monmap: 2024-04-22T08:22:46.179 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:46.179 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:46.179 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:22:46.180603+0000 2024-04-22T08:22:46.180 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:22:46.180603+0000 2024-04-22T08:22:46.180 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:46.180 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:46.180 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:22:46.180 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:22:46.180 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:22:46.180 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:22:46.180 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:22:46.180 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:46.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.180+0000 7f3186a8f640 1 Processor -- start 2024-04-22T08:22:46.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.180+0000 7f3186a8f640 1 -- start start 2024-04-22T08:22:46.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.180+0000 7f3186a8f640 10 monclient: init 2024-04-22T08:22:46.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.181+0000 7f3186a8f640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:46.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.181+0000 7f3186a8f640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-04-22T08:22:46.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.181+0000 7f3186a8f640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f31800a1c20 0x7f31800a2000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:46.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.181+0000 7f3186a8f640 10 monclient(hunting): picked mon.noname-c con 0x7f31800a1c20 addr v2:172.21.15.189:3300/0 2024-04-22T08:22:46.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.181+0000 7f3186a8f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f31800a2540 addr v1:172.21.15.189:6789/0 2024-04-22T08:22:46.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.181+0000 7f3186a8f640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f31801364d0 0x7f318013e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:46.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.181+0000 7f3186a8f640 10 monclient(hunting): picked mon.noname-a con 0x7f31801364d0 addr v2:172.21.15.40:3300/0 2024-04-22T08:22:46.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.181+0000 7f3186a8f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f318009f600 con 0x7f31800a2540 2024-04-22T08:22:46.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.181+0000 7f3186a8f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:46.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.181+0000 7f3186a8f640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f318009f8e0 con 0x7f31801364d0 2024-04-22T08:22:46.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.181+0000 7f3186a8f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:46.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.181+0000 7f3186a8f640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31800a29b0 con 0x7f31800a1c20 2024-04-22T08:22:46.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.181+0000 7f3186a8f640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:46.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.181+0000 7f3186a8f640 10 monclient(hunting): authenticate will time out at 1440.174861s 2024-04-22T08:22:46.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.181+0000 7f318502c640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f31801364d0 0x7f318013e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:46.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.181+0000 7f318502c640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f31801364d0 0x7f318013e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44634/0 (socket says 172.21.15.40:44634) 2024-04-22T08:22:46.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.181+0000 7f318502c640 1 -- 172.21.15.40:0/2543528207 learned_addr learned my addr 172.21.15.40:0/2543528207 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:22:46.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.181+0000 7f318502c640 10 monclient(hunting): get_auth_request con 0x7f31801364d0 auth_method 0 2024-04-22T08:22:46.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.181+0000 7f318502c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:46.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.181+0000 7f318502c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:46.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.181+0000 7f318502c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:46.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.181+0000 7f318502c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:46.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.181+0000 7f318502c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:46.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.181+0000 7f318482b640 1 --2- 172.21.15.40:0/2543528207 >> v2:172.21.15.189:3300/0 conn(0x7f31800a1c20 0x7f31800a2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:46.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.181+0000 7f318502c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:46.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.181+0000 7f318482b640 10 monclient(hunting): get_auth_request con 0x7f31800a1c20 auth_method 0 2024-04-22T08:22:46.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.181+0000 7f318482b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:46.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.181+0000 7f318482b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:46.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.181+0000 7f318482b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:46.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.181+0000 7f318482b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:46.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.181+0000 7f318482b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:46.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.181+0000 7f318482b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:46.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.181+0000 7f318502c640 10 monclient(hunting): handle_auth_done global_id 114330 payload 274 2024-04-22T08:22:46.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.181+0000 7f318502c640 1 -- 172.21.15.40:0/2543528207 >> v2:172.21.15.189:3300/0 conn(0x7f31800a1c20 msgr2=0x7f31800a2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:46.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.181+0000 7f318502c640 1 --2- 172.21.15.40:0/2543528207 >> v2:172.21.15.189:3300/0 conn(0x7f31800a1c20 0x7f31800a2000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:46.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.181+0000 7f318502c640 1 -- 172.21.15.40:0/2543528207 >> v1:172.21.15.189:6789/0 conn(0x7f31800a2540 legacy=0x7f3180135dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:46.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.181+0000 7f318502c640 10 monclient: _finish_hunting 0 2024-04-22T08:22:46.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.181+0000 7f318502c640 1 monclient: found mon.noname-a 2024-04-22T08:22:46.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.181+0000 7f318502c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:46.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.181+0000 7f318502c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:22:46.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.181+0000 7f318502c640 1 -- 172.21.15.40:0/2543528207 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f318009d160 con 0x7f31801364d0 2024-04-22T08:22:46.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.184+0000 7f318502c640 1 -- 172.21.15.40:0/2543528207 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31801364d0 msgr2=0x7f318013e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:46.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.184+0000 7f318502c640 1 --2- 172.21.15.40:0/2543528207 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31801364d0 0x7f318013e020 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f318009e500 tx=0x7f317802bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72fd407955348dd1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:46.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.184+0000 7f317f7fe640 1 -- 172.21.15.40:0/2543528207 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f317802cb70 con 0x7f31801364d0 2024-04-22T08:22:46.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.184+0000 7f317f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:46.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.184+0000 7f317f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:22:46.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.184+0000 7f317f7fe640 10 monclient: dump: 2024-04-22T08:22:46.193 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:46.193 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:46.193 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:46.193 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:46.193 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:46.193 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:46.193 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:46.193 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:46.193 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:46.193 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:46.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.184+0000 7f317f7fe640 10 monclient: _finish_auth 0 2024-04-22T08:22:46.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.184+0000 7f317f7fe640 10 monclient: _check_auth_tickets 2024-04-22T08:22:46.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.184+0000 7f317f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:46.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.184+0000 7f3186a8f640 5 monclient: authenticate success, global_id 114330 2024-04-22T08:22:46.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.184+0000 7f3186a8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:22:46.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.184+0000 7f317f7fe640 1 -- 172.21.15.40:0/2543528207 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f317802ccd0 con 0x7f31801364d0 2024-04-22T08:22:46.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.184+0000 7f317f7fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:46.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.184+0000 7f317f7fe640 1 -- 172.21.15.40:0/2543528207 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3178034610 con 0x7f31801364d0 2024-04-22T08:22:46.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.184+0000 7f3186a8f640 10 monclient: get_monmap_and_config success 2024-04-22T08:22:46.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.184+0000 7f3186a8f640 4 set_mon_vals no callback set 2024-04-22T08:22:46.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.184+0000 7f317f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:46.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.184+0000 7f317f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:22:46.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.184+0000 7f317f7fe640 10 monclient: dump: 2024-04-22T08:22:46.194 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:46.194 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:46.194 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:46.194 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:46.194 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:46.195 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:46.195 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:46.195 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:46.195 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:46.195 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:46.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.185+0000 7f3186a8f640 10 monclient: shutdown 2024-04-22T08:22:46.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.185+0000 7f3186a8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:46.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.185+0000 7f3186a8f640 1 -- 172.21.15.40:0/2543528207 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31801364d0 msgr2=0x7f318013e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:46.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.185+0000 7f3186a8f640 1 --2- 172.21.15.40:0/2543528207 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31801364d0 0x7f318013e020 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f318009e500 tx=0x7f317802bdc0 comp rx=0 tx=0).stop 2024-04-22T08:22:46.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.185+0000 7f3186a8f640 1 -- 172.21.15.40:0/2543528207 shutdown_connections 2024-04-22T08:22:46.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.185+0000 7f3186a8f640 1 --2- 172.21.15.40:0/2543528207 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31801364d0 0x7f318013e020 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:46.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.185+0000 7f3186a8f640 1 --2- 172.21.15.40:0/2543528207 >> v2:172.21.15.189:3300/0 conn(0x7f31800a1c20 0x7f31800a2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:46.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.185+0000 7f3186a8f640 1 -- 172.21.15.40:0/2543528207 >> 172.21.15.40:0/2543528207 conn(0x7f31800a6840 msgr2=0x7f31800a2d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:46.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.185+0000 7f3186a8f640 1 -- 172.21.15.40:0/2543528207 shutdown_connections 2024-04-22T08:22:46.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.185+0000 7f3186a8f640 1 -- 172.21.15.40:0/2543528207 wait complete. 2024-04-22T08:22:46.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.185+0000 7f3186a8f640 10 monclient: build_initial_monmap 2024-04-22T08:22:46.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.185+0000 7f3186a8f640 10 monclient: monmap: 2024-04-22T08:22:46.196 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:46.196 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:22:46.196 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:22:46.196 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:22:46.196 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:46.196 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:46.196 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:22:46.196 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:22:46.196 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:22:46.196 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:46.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.186+0000 7f3186a8f640 1 Processor -- start 2024-04-22T08:22:46.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.186+0000 7f3186a8f640 1 -- start start 2024-04-22T08:22:46.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.186+0000 7f3186a8f640 10 monclient: init 2024-04-22T08:22:46.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.186+0000 7f3186a8f640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:46.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.186+0000 7f3186a8f640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-22T08:22:46.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.186+0000 7f3186a8f640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31800a1c20 0x7f31801cad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:46.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.186+0000 7f3186a8f640 10 monclient(hunting): picked mon.noname-c con 0x7f31800a1c20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:22:46.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.186+0000 7f3186a8f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31800a2540 0x7f31801226d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:46.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.186+0000 7f3186a8f640 10 monclient(hunting): picked mon.noname-b con 0x7f31800a2540 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:22:46.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.186+0000 7f3186a8f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31801364d0 0x7f3180122c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:46.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.186+0000 7f3186a8f640 10 monclient(hunting): picked mon.noname-a con 0x7f31801364d0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:22:46.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.186+0000 7f3186a8f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:46.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.186+0000 7f3186a8f640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3180125c70 con 0x7f31801364d0 2024-04-22T08:22:46.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.186+0000 7f3186a8f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:46.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.186+0000 7f3186a8f640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3180124d10 con 0x7f31800a2540 2024-04-22T08:22:46.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.186+0000 7f3186a8f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:46.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.186+0000 7f3186a8f640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3180135dc0 con 0x7f31800a1c20 2024-04-22T08:22:46.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.186+0000 7f3186a8f640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:46.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.186+0000 7f3186a8f640 10 monclient(hunting): authenticate will time out at 1440.180654s 2024-04-22T08:22:46.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.186+0000 7f318482b640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31800a1c20 0x7f31801cad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:46.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.186+0000 7f318482b640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31800a1c20 0x7f31801cad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49922/0 (socket says 172.21.15.40:49922) 2024-04-22T08:22:46.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.187+0000 7f318482b640 1 -- 172.21.15.40:0/2208279645 learned_addr learned my addr 172.21.15.40:0/2208279645 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:22:46.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.186+0000 7f318502c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31801364d0 0x7f3180122c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:46.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.187+0000 7f318482b640 10 monclient(hunting): get_auth_request con 0x7f31800a1c20 auth_method 0 2024-04-22T08:22:46.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.187+0000 7f318482b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:46.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.187+0000 7f318482b640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:46.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.187+0000 7f318482b640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:46.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.187+0000 7f318502c640 10 monclient(hunting): get_auth_request con 0x7f31801364d0 auth_method 0 2024-04-22T08:22:46.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.187+0000 7f318502c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:46.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.187+0000 7f318502c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:46.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.187+0000 7f318502c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:46.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.187+0000 7f318482b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:46.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.187+0000 7f318482b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:46.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.187+0000 7f318482b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:46.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.187+0000 7f318502c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:46.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.187+0000 7f318502c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:46.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.187+0000 7f318502c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:46.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.187+0000 7f317ffff640 1 --2- 172.21.15.40:0/2208279645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31800a2540 0x7f31801226d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:46.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.187+0000 7f317ffff640 10 monclient(hunting): get_auth_request con 0x7f31800a2540 auth_method 0 2024-04-22T08:22:46.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.187+0000 7f317ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:46.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.187+0000 7f317ffff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:46.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.187+0000 7f317ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:46.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.187+0000 7f317ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:46.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.187+0000 7f317ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:46.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.187+0000 7f317ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:46.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.187+0000 7f318502c640 10 monclient(hunting): handle_auth_done global_id 114333 payload 824 2024-04-22T08:22:46.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.187+0000 7f318502c640 1 -- 172.21.15.40:0/2208279645 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31800a1c20 msgr2=0x7f31801cad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:46.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.187+0000 7f318502c640 1 --2- 172.21.15.40:0/2208279645 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31800a1c20 0x7f31801cad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:46.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.187+0000 7f318502c640 1 -- 172.21.15.40:0/2208279645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31800a2540 msgr2=0x7f31801226d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:46.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.187+0000 7f318502c640 1 --2- 172.21.15.40:0/2208279645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31800a2540 0x7f31801226d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:46.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.187+0000 7f318502c640 10 monclient: _finish_hunting 0 2024-04-22T08:22:46.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.187+0000 7f318502c640 1 monclient: found mon.noname-a 2024-04-22T08:22:46.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.187+0000 7f318502c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:46.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.187+0000 7f318502c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:22:46.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.187+0000 7f318502c640 1 -- 172.21.15.40:0/2208279645 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3178009b30 con 0x7f31801364d0 2024-04-22T08:22:46.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.187+0000 7f318482b640 1 --2- 172.21.15.40:0/2208279645 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31800a1c20 0x7f31801cad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:22:46.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.189+0000 7f318502c640 1 -- 172.21.15.40:0/2208279645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31801364d0 msgr2=0x7f3180122c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:46.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.189+0000 7f318502c640 1 --2- 172.21.15.40:0/2208279645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31801364d0 0x7f3180122c10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3178032000 tx=0x7f3178004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:46.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.189+0000 7f317d7fa640 1 -- 172.21.15.40:0/2208279645 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3178034ca0 con 0x7f31801364d0 2024-04-22T08:22:46.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.189+0000 7f317d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:46.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.189+0000 7f317d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:22:46.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.189+0000 7f317d7fa640 10 monclient: dump: 2024-04-22T08:22:46.201 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:46.201 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:46.201 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:46.202 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:46.202 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:46.202 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:46.202 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:46.202 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:46.202 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:46.202 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:46.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.189+0000 7f317d7fa640 10 monclient: _finish_auth 0 2024-04-22T08:22:46.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.189+0000 7f317d7fa640 10 monclient: _check_auth_tickets 2024-04-22T08:22:46.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.189+0000 7f317d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:46.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.189+0000 7f317d7fa640 1 -- 172.21.15.40:0/2208279645 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3178034e00 con 0x7f31801364d0 2024-04-22T08:22:46.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.189+0000 7f3186a8f640 5 monclient: authenticate success, global_id 114333 2024-04-22T08:22:46.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.189+0000 7f3186a8f640 10 monclient: _renew_subs 2024-04-22T08:22:46.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.189+0000 7f3186a8f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:46.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.189+0000 7f3186a8f640 1 -- 172.21.15.40:0/2208279645 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3180071db0 con 0x7f31801364d0 2024-04-22T08:22:46.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.189+0000 7f317d7fa640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:46.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.189+0000 7f317d7fa640 1 -- 172.21.15.40:0/2208279645 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f317803e690 con 0x7f31801364d0 2024-04-22T08:22:46.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.189+0000 7f317d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:46.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.189+0000 7f317d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:22:46.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.189+0000 7f317d7fa640 10 monclient: dump: 2024-04-22T08:22:46.203 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:46.203 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:46.203 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:46.203 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:46.203 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:46.203 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:46.203 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:46.204 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:46.204 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:46.204 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:46.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.189+0000 7f317effd640 4 set_mon_vals no callback set 2024-04-22T08:22:46.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.189+0000 7f3186a8f640 10 monclient: _renew_subs 2024-04-22T08:22:46.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.189+0000 7f3186a8f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:46.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.189+0000 7f3186a8f640 1 -- 172.21.15.40:0/2208279645 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3180124140 con 0x7f31801364d0 2024-04-22T08:22:46.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.190+0000 7f3186a8f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:46.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.190+0000 7f3186a8f640 1 -- 172.21.15.40:0/2208279645 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f318009df70 con 0x7f31801364d0 2024-04-22T08:22:46.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.190+0000 7f317d7fa640 1 -- 172.21.15.40:0/2208279645 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f317803e870 con 0x7f31801364d0 2024-04-22T08:22:46.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.190+0000 7f317d7fa640 1 --2- 172.21.15.40:0/2208279645 >> v2:172.21.15.189:6800/30168 conn(0x7f316c0377d0 0x7f316c039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:46.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.191+0000 7f317d7fa640 1 -- 172.21.15.40:0/2208279645 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(317..317 src has 239..317) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f317802ce40 con 0x7f31801364d0 2024-04-22T08:22:46.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.191+0000 7f317d7fa640 1 --2- 172.21.15.40:0/2208279645 >> v2:172.21.15.189:6805/30673 conn(0x7f316c03d280 0x7f316c03f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:46.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.191+0000 7f317d7fa640 1 -- 172.21.15.40:0/2208279645 --> v2:172.21.15.189:6805/30673 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f316c03fd10 con 0x7f316c03d280 2024-04-22T08:22:46.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.191+0000 7f317d7fa640 1 -- 172.21.15.40:0/2208279645 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=317) v2 ==== 24+0+0 (secure 0 0 0) 0x7f31780712a0 con 0x7f31801364d0 2024-04-22T08:22:46.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.191+0000 7f317d7fa640 10 monclient: handle_get_version_reply finishing 1 version 317 2024-04-22T08:22:46.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.191+0000 7f318482b640 1 --2- 172.21.15.40:0/2208279645 >> v2:172.21.15.189:6800/30168 conn(0x7f316c0377d0 0x7f316c039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:46.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.191+0000 7f318482b640 10 monclient: get_auth_request con 0x7f316c0377d0 auth_method 0 2024-04-22T08:22:46.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.191+0000 7f317ffff640 1 --2- 172.21.15.40:0/2208279645 >> v2:172.21.15.189:6805/30673 conn(0x7f316c03d280 0x7f316c03f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:46.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.191+0000 7f317ffff640 10 monclient: get_auth_request con 0x7f316c03d280 auth_method 0 2024-04-22T08:22:46.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.193+0000 7f318482b640 1 --2- 172.21.15.40:0/2208279645 >> v2:172.21.15.189:6800/30168 conn(0x7f316c0377d0 0x7f316c039c90 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f31700112c0 tx=0x7f3170009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:46.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.196+0000 7f317ffff640 1 --2- 172.21.15.40:0/2208279645 >> v2:172.21.15.189:6805/30673 conn(0x7f316c03d280 0x7f316c03f6a0 crc :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:46.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.197+0000 7f317d7fa640 1 -- 172.21.15.40:0/2208279645 <== osd.3 v2:172.21.15.189:6805/30673 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f316c03fd10 con 0x7f316c03d280 2024-04-22T08:22:46.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.210+0000 7f3186a8f640 1 -- 172.21.15.40:0/2208279645 --> v2:172.21.15.189:6805/30673 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f318004e9b0 con 0x7f316c03d280 2024-04-22T08:22:46.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.210+0000 7f317d7fa640 1 -- 172.21.15.40:0/2208279645 <== osd.3 v2:172.21.15.189:6805/30673 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f318004e9b0 con 0x7f316c03d280 2024-04-22T08:22:46.211 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:22:46.211 INFO:teuthology.orchestra.run.smithi040.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-22T08:22:46.211 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:22:46.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.211+0000 7f3186a8f640 1 -- 172.21.15.40:0/2208279645 >> v2:172.21.15.189:6805/30673 conn(0x7f316c03d280 msgr2=0x7f316c03f6a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:46.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.211+0000 7f3186a8f640 1 --2- 172.21.15.40:0/2208279645 >> v2:172.21.15.189:6805/30673 conn(0x7f316c03d280 0x7f316c03f6a0 crc :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:46.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.211+0000 7f3186a8f640 1 -- 172.21.15.40:0/2208279645 >> v2:172.21.15.189:6800/30168 conn(0x7f316c0377d0 msgr2=0x7f316c039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:46.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.211+0000 7f3186a8f640 1 --2- 172.21.15.40:0/2208279645 >> v2:172.21.15.189:6800/30168 conn(0x7f316c0377d0 0x7f316c039c90 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f31700112c0 tx=0x7f3170009190 comp rx=0 tx=0).stop 2024-04-22T08:22:46.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.211+0000 7f3186a8f640 10 monclient: shutdown 2024-04-22T08:22:46.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.211+0000 7f3186a8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:46.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.211+0000 7f3186a8f640 1 -- 172.21.15.40:0/2208279645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31801364d0 msgr2=0x7f3180122c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:46.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.211+0000 7f3186a8f640 1 --2- 172.21.15.40:0/2208279645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31801364d0 0x7f3180122c10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3178032000 tx=0x7f3178004a40 comp rx=0 tx=0).stop 2024-04-22T08:22:46.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.211+0000 7f318502c640 1 -- 172.21.15.40:0/2208279645 reap_dead start 2024-04-22T08:22:46.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.211+0000 7f3186a8f640 1 -- 172.21.15.40:0/2208279645 shutdown_connections 2024-04-22T08:22:46.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.211+0000 7f3186a8f640 1 -- 172.21.15.40:0/2208279645 >> 172.21.15.40:0/2208279645 conn(0x7f31800a6840 msgr2=0x7f318009b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:46.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.211+0000 7f3186a8f640 1 -- 172.21.15.40:0/2208279645 shutdown_connections 2024-04-22T08:22:46.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.211+0000 7f3186a8f640 1 -- 172.21.15.40:0/2208279645 wait complete. 2024-04-22T08:22:46.221 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-22T08:22:46.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.321+0000 7fa0f1470640 10 monclient: get_monmap_and_config 2024-04-22T08:22:46.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.321+0000 7fa0f1470640 10 monclient: build_initial_monmap 2024-04-22T08:22:46.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.321+0000 7fa0f1470640 10 monclient: monmap: 2024-04-22T08:22:46.321 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:46.321 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:46.322 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:22:46.322483+0000 2024-04-22T08:22:46.322 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:22:46.322483+0000 2024-04-22T08:22:46.322 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:46.322 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:46.322 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:22:46.322 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:22:46.322 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:22:46.322 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:22:46.322 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:22:46.322 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:46.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.322+0000 7fa0f1470640 1 Processor -- start 2024-04-22T08:22:46.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.322+0000 7fa0f1470640 1 -- start start 2024-04-22T08:22:46.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.322+0000 7fa0f1470640 10 monclient: init 2024-04-22T08:22:46.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.322+0000 7fa0f1470640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:46.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.322+0000 7fa0f1470640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-04-22T08:22:46.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.322+0000 7fa0f1470640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa0ec0a20d0 0x7fa0ec0a24b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:46.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.322+0000 7fa0f1470640 10 monclient(hunting): picked mon.noname-c con 0x7fa0ec0a20d0 addr v2:172.21.15.189:3300/0 2024-04-22T08:22:46.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.322+0000 7fa0f1470640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa0ec0a2d10 addr v1:172.21.15.40:6789/0 2024-04-22T08:22:46.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.322+0000 7fa0f1470640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa0ec0a38d0 addr v1:172.21.15.189:6789/0 2024-04-22T08:22:46.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.322+0000 7fa0f1470640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0ec0a45f0 con 0x7fa0ec0a2d10 2024-04-22T08:22:46.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.322+0000 7fa0f1470640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0ec0a0360 con 0x7fa0ec0a38d0 2024-04-22T08:22:46.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.322+0000 7fa0f1470640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:46.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.322+0000 7fa0f1470640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0ec0a0640 con 0x7fa0ec0a20d0 2024-04-22T08:22:46.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.322+0000 7fa0f1470640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:46.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.322+0000 7fa0f1470640 10 monclient(hunting): authenticate will time out at 1440.316557s 2024-04-22T08:22:46.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.322+0000 7fa0ea7fc640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa0ec0a2d10 0x7fa0ec0a31c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:49372/0 (socket says 172.21.15.40:49372) 2024-04-22T08:22:46.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.322+0000 7fa0ea7fc640 1 -- 172.21.15.40:0/287961374 learned_addr learned my addr 172.21.15.40:0/287961374 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:22:46.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.323+0000 7fa0eaffd640 1 --2- 172.21.15.40:0/287961374 >> v2:172.21.15.189:3300/0 conn(0x7fa0ec0a20d0 0x7fa0ec0a24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:46.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.323+0000 7fa0eaffd640 10 monclient(hunting): get_auth_request con 0x7fa0ec0a20d0 auth_method 0 2024-04-22T08:22:46.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.323+0000 7fa0eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:46.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.323+0000 7fa0eaffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:46.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.323+0000 7fa0eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:46.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.323+0000 7fa0eaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:46.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.323+0000 7fa0eaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:46.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.323+0000 7fa0eaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:46.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.323+0000 7fa0eaffd640 10 monclient(hunting): handle_auth_done global_id 65296 payload 274 2024-04-22T08:22:46.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.323+0000 7fa0eaffd640 1 -- 172.21.15.40:0/287961374 >> v1:172.21.15.189:6789/0 conn(0x7fa0ec0a38d0 legacy=0x7fa0ec13db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:46.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.323+0000 7fa0eaffd640 1 -- 172.21.15.40:0/287961374 >> v1:172.21.15.40:6789/0 conn(0x7fa0ec0a2d10 legacy=0x7fa0ec0a31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:46.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.323+0000 7fa0eaffd640 10 monclient: _finish_hunting 0 2024-04-22T08:22:46.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.323+0000 7fa0eaffd640 1 monclient: found mon.noname-c 2024-04-22T08:22:46.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.323+0000 7fa0eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:46.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.323+0000 7fa0eaffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.189:3300/0 2024-04-22T08:22:46.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.323+0000 7fa0eaffd640 1 -- 172.21.15.40:0/287961374 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0ec0a0920 con 0x7fa0ec0a20d0 2024-04-22T08:22:46.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.326+0000 7fa0eaffd640 1 -- 172.21.15.40:0/287961374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0ec0a20d0 msgr2=0x7fa0ec0a24b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:46.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.326+0000 7fa0eaffd640 1 --2- 172.21.15.40:0/287961374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0ec0a20d0 0x7fa0ec0a24b0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc0097a0 tx=0x7fa0dc02b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=151ff688f665b5ec server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:46.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.326+0000 7fa0e9ffb640 1 -- 172.21.15.40:0/287961374 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0dc030a00 con 0x7fa0ec0a20d0 2024-04-22T08:22:46.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.326+0000 7fa0e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:46.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.326+0000 7fa0e9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:22:46.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.326+0000 7fa0e9ffb640 10 monclient: dump: 2024-04-22T08:22:46.329 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:46.329 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:46.329 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:46.329 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:46.329 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:46.329 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:46.329 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:46.329 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:46.329 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:46.329 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:46.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.326+0000 7fa0e9ffb640 10 monclient: _finish_auth 0 2024-04-22T08:22:46.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.326+0000 7fa0e9ffb640 10 monclient: _check_auth_tickets 2024-04-22T08:22:46.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.326+0000 7fa0e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:46.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.326+0000 7fa0e9ffb640 1 -- 172.21.15.40:0/287961374 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0dc02e050 con 0x7fa0ec0a20d0 2024-04-22T08:22:46.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.326+0000 7fa0e9ffb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:46.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.326+0000 7fa0e9ffb640 1 -- 172.21.15.40:0/287961374 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0dc030d80 con 0x7fa0ec0a20d0 2024-04-22T08:22:46.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.326+0000 7fa0e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:46.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.326+0000 7fa0e9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:22:46.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.326+0000 7fa0e9ffb640 10 monclient: dump: 2024-04-22T08:22:46.330 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:46.330 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:46.330 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:46.330 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:46.330 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:46.330 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:46.331 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:46.331 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:46.331 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:46.331 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:46.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.326+0000 7fa0f1470640 5 monclient: authenticate success, global_id 65296 2024-04-22T08:22:46.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.326+0000 7fa0f1470640 10 monclient: get_monmap_and_config success 2024-04-22T08:22:46.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.326+0000 7fa0f1470640 4 set_mon_vals no callback set 2024-04-22T08:22:46.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.327+0000 7fa0f1470640 10 monclient: shutdown 2024-04-22T08:22:46.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.327+0000 7fa0f1470640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:46.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.327+0000 7fa0f1470640 1 -- 172.21.15.40:0/287961374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0ec0a20d0 msgr2=0x7fa0ec0a24b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:46.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.327+0000 7fa0f1470640 1 --2- 172.21.15.40:0/287961374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0ec0a20d0 0x7fa0ec0a24b0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc0097a0 tx=0x7fa0dc02b710 comp rx=0 tx=0).stop 2024-04-22T08:22:46.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.327+0000 7fa0f1470640 1 -- 172.21.15.40:0/287961374 shutdown_connections 2024-04-22T08:22:46.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.327+0000 7fa0f1470640 1 --2- 172.21.15.40:0/287961374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0ec0a20d0 0x7fa0ec0a24b0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:46.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.327+0000 7fa0f1470640 1 -- 172.21.15.40:0/287961374 >> 172.21.15.40:0/287961374 conn(0x7fa0ec0a8e60 msgr2=0x7fa0ec0a4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:46.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.327+0000 7fa0f1470640 1 -- 172.21.15.40:0/287961374 shutdown_connections 2024-04-22T08:22:46.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.327+0000 7fa0f1470640 1 -- 172.21.15.40:0/287961374 wait complete. 2024-04-22T08:22:46.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.327+0000 7fa0f1470640 10 monclient: build_initial_monmap 2024-04-22T08:22:46.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.327+0000 7fa0f1470640 10 monclient: monmap: 2024-04-22T08:22:46.340 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:46.340 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:22:46.341 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:22:46.341 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:22:46.341 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:46.341 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:46.341 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:22:46.341 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:22:46.341 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:22:46.341 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:46.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.328+0000 7fa0f1470640 1 Processor -- start 2024-04-22T08:22:46.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.328+0000 7fa0f1470640 1 -- start start 2024-04-22T08:22:46.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.328+0000 7fa0f1470640 10 monclient: init 2024-04-22T08:22:46.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.328+0000 7fa0f1470640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:46.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.328+0000 7fa0f1470640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-22T08:22:46.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.328+0000 7fa0f1470640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0ec0a20d0 0x7fa0ec1c4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:46.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.328+0000 7fa0f1470640 10 monclient(hunting): picked mon.noname-a con 0x7fa0ec0a20d0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:22:46.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.328+0000 7fa0f1470640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0ec0a2d10 0x7fa0ec1c73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:46.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.328+0000 7fa0f1470640 10 monclient(hunting): picked mon.noname-c con 0x7fa0ec0a2d10 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:22:46.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.328+0000 7fa0f1470640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0ec0a38d0 0x7fa0ec1c7930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:46.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.328+0000 7fa0f1470640 10 monclient(hunting): picked mon.noname-b con 0x7fa0ec0a38d0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:22:46.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.328+0000 7fa0f1470640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:46.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.328+0000 7fa0f1470640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0ec070ef0 con 0x7fa0ec0a20d0 2024-04-22T08:22:46.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.328+0000 7fa0f1470640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:46.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.328+0000 7fa0f1470640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0ec071750 con 0x7fa0ec0a38d0 2024-04-22T08:22:46.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.328+0000 7fa0f1470640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:46.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.328+0000 7fa0f1470640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0ec1a9640 con 0x7fa0ec0a2d10 2024-04-22T08:22:46.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.328+0000 7fa0f1470640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:46.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.328+0000 7fa0f1470640 10 monclient(hunting): authenticate will time out at 1440.322489s 2024-04-22T08:22:46.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.328+0000 7fa0eaffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0ec0a20d0 0x7fa0ec1c4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:46.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.328+0000 7fa0eaffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0ec0a20d0 0x7fa0ec1c4770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44652/0 (socket says 172.21.15.40:44652) 2024-04-22T08:22:46.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.328+0000 7fa0eaffd640 1 -- 172.21.15.40:0/494644674 learned_addr learned my addr 172.21.15.40:0/494644674 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:22:46.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.328+0000 7fa0eaffd640 10 monclient(hunting): get_auth_request con 0x7fa0ec0a20d0 auth_method 0 2024-04-22T08:22:46.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.328+0000 7fa0eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:46.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.328+0000 7fa0eaffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:46.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.328+0000 7fa0eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:46.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.328+0000 7fa0ea7fc640 1 --2- 172.21.15.40:0/494644674 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0ec0a2d10 0x7fa0ec1c73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:46.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.328+0000 7fa0ea7fc640 10 monclient(hunting): get_auth_request con 0x7fa0ec0a2d10 auth_method 0 2024-04-22T08:22:46.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.328+0000 7fa0ea7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:46.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.328+0000 7fa0ea7fc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:46.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.328+0000 7fa0ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:46.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.328+0000 7fa0eaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:46.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.328+0000 7fa0eaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:46.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.328+0000 7fa0eaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:46.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.328+0000 7fa0ea7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:46.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.328+0000 7fa0ea7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:46.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.328+0000 7fa0ea7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:46.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.329+0000 7fa0eb7fe640 1 --2- 172.21.15.40:0/494644674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0ec0a38d0 0x7fa0ec1c7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:46.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.329+0000 7fa0ea7fc640 10 monclient(hunting): handle_auth_done global_id 124169 payload 824 2024-04-22T08:22:46.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.329+0000 7fa0ea7fc640 1 -- 172.21.15.40:0/494644674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0ec0a38d0 msgr2=0x7fa0ec1c7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:46.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.329+0000 7fa0ea7fc640 1 --2- 172.21.15.40:0/494644674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0ec0a38d0 0x7fa0ec1c7930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:46.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.329+0000 7fa0ea7fc640 1 -- 172.21.15.40:0/494644674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0ec0a20d0 msgr2=0x7fa0ec1c4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:46.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.329+0000 7fa0ea7fc640 1 --2- 172.21.15.40:0/494644674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0ec0a20d0 0x7fa0ec1c4770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:46.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.329+0000 7fa0ea7fc640 10 monclient: _finish_hunting 0 2024-04-22T08:22:46.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.329+0000 7fa0ea7fc640 1 monclient: found mon.noname-c 2024-04-22T08:22:46.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.329+0000 7fa0ea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:46.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.329+0000 7fa0ea7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:22:46.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.329+0000 7fa0ea7fc640 1 -- 172.21.15.40:0/494644674 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0dc009400 con 0x7fa0ec0a2d10 2024-04-22T08:22:46.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.329+0000 7fa0eb7fe640 10 monclient: get_auth_request con 0x7fa0ec0a38d0 auth_method 0 2024-04-22T08:22:46.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.329+0000 7fa0eaffd640 1 --2- 172.21.15.40:0/494644674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0ec0a20d0 0x7fa0ec1c4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:22:46.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.329+0000 7fa0eb7fe640 1 --2- 172.21.15.40:0/494644674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0ec0a38d0 0x7fa0ec1c7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T08:22:46.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.331+0000 7fa0ea7fc640 1 -- 172.21.15.40:0/494644674 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0ec0a2d10 msgr2=0x7fa0ec1c73f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:46.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.331+0000 7fa0ea7fc640 1 --2- 172.21.15.40:0/494644674 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0ec0a2d10 0x7fa0ec1c73f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa0e000c3e0 tx=0x7fa0e000c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:46.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.331+0000 7fa0d37fe640 1 -- 172.21.15.40:0/494644674 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e00070d0 con 0x7fa0ec0a2d10 2024-04-22T08:22:46.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.331+0000 7fa0d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:46.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.331+0000 7fa0d37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:22:46.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.331+0000 7fa0d37fe640 10 monclient: dump: 2024-04-22T08:22:46.347 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:46.347 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:46.347 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:46.347 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:46.347 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:46.347 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:46.347 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:46.347 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:46.348 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:46.348 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:46.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.331+0000 7fa0d37fe640 10 monclient: _finish_auth 0 2024-04-22T08:22:46.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.331+0000 7fa0d37fe640 10 monclient: _check_auth_tickets 2024-04-22T08:22:46.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.331+0000 7fa0d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:46.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.331+0000 7fa0f1470640 5 monclient: authenticate success, global_id 124169 2024-04-22T08:22:46.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.331+0000 7fa0f1470640 10 monclient: _renew_subs 2024-04-22T08:22:46.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.331+0000 7fa0d37fe640 1 -- 172.21.15.40:0/494644674 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0e0007230 con 0x7fa0ec0a2d10 2024-04-22T08:22:46.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.331+0000 7fa0f1470640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:22:46.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.331+0000 7fa0f1470640 1 -- 172.21.15.40:0/494644674 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0ec1a9d90 con 0x7fa0ec0a2d10 2024-04-22T08:22:46.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.331+0000 7fa0d37fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:46.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.331+0000 7fa0d37fe640 1 -- 172.21.15.40:0/494644674 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e00075e0 con 0x7fa0ec0a2d10 2024-04-22T08:22:46.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.331+0000 7fa0d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:46.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.331+0000 7fa0d37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:22:46.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.331+0000 7fa0d37fe640 10 monclient: dump: 2024-04-22T08:22:46.349 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:46.349 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:46.349 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:46.349 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:46.349 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:46.349 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:46.349 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:46.349 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:46.349 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:46.349 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:46.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.331+0000 7fa0f1470640 10 monclient: _renew_subs 2024-04-22T08:22:46.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.331+0000 7fa0f1470640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:22:46.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.331+0000 7fa0f1470640 1 -- 172.21.15.40:0/494644674 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0ec1abb10 con 0x7fa0ec0a2d10 2024-04-22T08:22:46.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.331+0000 7fa0e97fa640 4 set_mon_vals no callback set 2024-04-22T08:22:46.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.332+0000 7fa0d37fe640 1 -- 172.21.15.40:0/494644674 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fa0e001b3b0 con 0x7fa0ec0a2d10 2024-04-22T08:22:46.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.332+0000 7fa0f1470640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:46.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.332+0000 7fa0f1470640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:46.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.332+0000 7fa0f1470640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:22:46.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.332+0000 7fa0f1470640 1 -- 172.21.15.40:0/494644674 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0ec04e9b0 con 0x7fa0ec0a2d10 2024-04-22T08:22:46.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.332+0000 7fa0d37fe640 1 --2- 172.21.15.40:0/494644674 >> v2:172.21.15.189:6800/30168 conn(0x7fa0d80377d0 0x7fa0d8039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:46.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.332+0000 7fa0d37fe640 1 -- 172.21.15.40:0/494644674 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(317..317 src has 239..317) v4 ==== 3506+0+0 (secure 0 0 0) 0x7fa0e00498e0 con 0x7fa0ec0a2d10 2024-04-22T08:22:46.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.332+0000 7fa0ea7fc640 1 -- 172.21.15.40:0/494644674 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0ec0a2d10 msgr2=0x7fa0ec1c73f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-22T08:22:46.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.332+0000 7fa0ea7fc640 1 -- 172.21.15.40:0/494644674 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0ec0a2d10 msgr2=0x7fa0ec1c73f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:22:46.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.332+0000 7fa0ea7fc640 1 --2- 172.21.15.40:0/494644674 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0ec0a2d10 0x7fa0ec1c73f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa0e000c3e0 tx=0x7fa0e000c8a0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:22:46.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.332+0000 7fa0ea7fc640 1 --2- 172.21.15.40:0/494644674 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0ec0a2d10 0x7fa0ec1c73f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa0e000c3e0 tx=0x7fa0e000c8a0 comp rx=0 tx=0).stop 2024-04-22T08:22:46.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.332+0000 7fa0d37fe640 10 monclient: ms_handle_reset current mon [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:22:46.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.332+0000 7fa0d37fe640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:46.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.332+0000 7fa0d37fe640 1 -- 172.21.15.40:0/494644674 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0ec0a2d10 msgr2=0x7fa0ec1c73f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-22T08:22:46.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.332+0000 7fa0d37fe640 1 --2- 172.21.15.40:0/494644674 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0ec0a2d10 0x7fa0ec1c73f0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:46.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.332+0000 7fa0d37fe640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-22T08:22:46.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.332+0000 7fa0d37fe640 1 --2- 172.21.15.40:0/494644674 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0d803cc60 0x7fa0d803f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:46.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.332+0000 7fa0d37fe640 10 monclient(hunting): picked mon.c con 0x7fa0d803cc60 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:22:46.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.332+0000 7fa0d37fe640 1 --2- 172.21.15.40:0/494644674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0ec0a20d0 0x7fa0d8041a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:46.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.332+0000 7fa0d37fe640 10 monclient(hunting): picked mon.a con 0x7fa0ec0a20d0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:22:46.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.332+0000 7fa0d37fe640 1 --2- 172.21.15.40:0/494644674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0ec0a38d0 0x7fa0d80444b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:46.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.332+0000 7fa0d37fe640 10 monclient(hunting): picked mon.b con 0x7fa0ec0a38d0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:22:46.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.332+0000 7fa0d37fe640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:46.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.332+0000 7fa0d37fe640 1 -- 172.21.15.40:0/494644674 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d8044e30 con 0x7fa0ec0a20d0 2024-04-22T08:22:46.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.332+0000 7fa0d37fe640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:46.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.332+0000 7fa0d37fe640 1 -- 172.21.15.40:0/494644674 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d8044fa0 con 0x7fa0ec0a38d0 2024-04-22T08:22:46.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.332+0000 7fa0d37fe640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:46.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.332+0000 7fa0d37fe640 1 -- 172.21.15.40:0/494644674 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d8045160 con 0x7fa0d803cc60 2024-04-22T08:22:46.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.332+0000 7fa0d37fe640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:46.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.332+0000 7fa0eaffd640 1 --2- 172.21.15.40:0/494644674 >> v2:172.21.15.189:6800/30168 conn(0x7fa0d80377d0 0x7fa0d8039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:46.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.332+0000 7fa0ea7fc640 1 --2- 172.21.15.40:0/494644674 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0d803cc60 0x7fa0d803f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:46.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.333+0000 7fa0eaffd640 10 monclient(hunting): get_auth_request con 0x7fa0d80377d0 auth_method 0 2024-04-22T08:22:46.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.333+0000 7fa0ea7fc640 10 monclient(hunting): get_auth_request con 0x7fa0d803cc60 auth_method 0 2024-04-22T08:22:46.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.333+0000 7fa0eb7fe640 1 --2- 172.21.15.40:0/494644674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0ec0a20d0 0x7fa0d8041a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:46.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.333+0000 7fa0ea7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:46.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.333+0000 7fa0ea7fc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:46.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.333+0000 7fa0ea7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:22:46.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.333+0000 7fa0eb7fe640 10 monclient(hunting): get_auth_request con 0x7fa0ec0a20d0 auth_method 0 2024-04-22T08:22:46.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.333+0000 7fa0eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:46.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.333+0000 7fa0eb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:46.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.333+0000 7fa0eb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:22:46.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.333+0000 7fa0ea7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:46.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.333+0000 7fa0ea7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:46.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.333+0000 7fa0ea7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-22T08:22:46.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.333+0000 7fa0eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:46.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.333+0000 7fa0eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:46.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.333+0000 7fa0eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-22T08:22:46.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.333+0000 7fa0eaffd640 1 --2- 172.21.15.40:0/494644674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0ec0a38d0 0x7fa0d80444b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:46.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.333+0000 7fa0eaffd640 10 monclient(hunting): get_auth_request con 0x7fa0ec0a38d0 auth_method 0 2024-04-22T08:22:46.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.333+0000 7fa0eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:46.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.333+0000 7fa0eaffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:46.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.333+0000 7fa0eaffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-22T08:22:46.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.333+0000 7fa0ea7fc640 10 monclient(hunting): handle_auth_done global_id 124169 payload 293 2024-04-22T08:22:46.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.333+0000 7fa0ea7fc640 1 -- 172.21.15.40:0/494644674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0ec0a38d0 msgr2=0x7fa0d80444b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:46.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.333+0000 7fa0ea7fc640 1 --2- 172.21.15.40:0/494644674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0ec0a38d0 0x7fa0d80444b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:46.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.333+0000 7fa0ea7fc640 1 -- 172.21.15.40:0/494644674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0ec0a20d0 msgr2=0x7fa0d8041a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:46.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.333+0000 7fa0ea7fc640 1 --2- 172.21.15.40:0/494644674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0ec0a20d0 0x7fa0d8041a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:46.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.333+0000 7fa0ea7fc640 10 monclient: _finish_hunting 0 2024-04-22T08:22:46.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.333+0000 7fa0ea7fc640 1 monclient: found mon.c 2024-04-22T08:22:46.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.333+0000 7fa0ea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:46.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.333+0000 7fa0ea7fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:22:46.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.333+0000 7fa0ea7fc640 1 -- 172.21.15.40:0/494644674 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7fa0d8045320 con 0x7fa0d803cc60 2024-04-22T08:22:46.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.333+0000 7fa0ea7fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:46.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.333+0000 7fa0ea7fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:22:46.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.333+0000 7fa0ea7fc640 1 -- 172.21.15.40:0/494644674 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0ec04e9b0 con 0x7fa0d803cc60 2024-04-22T08:22:46.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.333+0000 7fa0ea7fc640 10 monclient: _finish_auth 0 2024-04-22T08:22:46.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.333+0000 7fa0ea7fc640 10 monclient: _check_auth_tickets 2024-04-22T08:22:46.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.333+0000 7fa0ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:46.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.333+0000 7fa0eb7fe640 1 --2- 172.21.15.40:0/494644674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0ec0a20d0 0x7fa0d8041a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:22:46.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.335+0000 7fa0ea7fc640 1 -- 172.21.15.40:0/494644674 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0d803cc60 msgr2=0x7fa0d803f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:46.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.335+0000 7fa0ea7fc640 1 --2- 172.21.15.40:0/494644674 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0d803cc60 0x7fa0d803f050 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa0e00078a0 tx=0x7fa0e0007f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:46.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.335+0000 7fa0eaffd640 1 --2- 172.21.15.40:0/494644674 >> v2:172.21.15.189:6800/30168 conn(0x7fa0d80377d0 0x7fa0d8039c90 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc0097a0 tx=0x7fa0dc02d000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:46.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.337+0000 7fa0d37fe640 1 -- 172.21.15.40:0/494644674 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e003e020 con 0x7fa0d803cc60 2024-04-22T08:22:46.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.337+0000 7fa0d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:46.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.337+0000 7fa0d37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:22:46.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.337+0000 7fa0d37fe640 10 monclient: dump: 2024-04-22T08:22:46.355 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:46.355 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:46.355 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:46.355 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:46.355 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:46.355 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:46.356 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:46.356 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:46.356 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:46.356 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:46.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.337+0000 7fa0d37fe640 1 -- 172.21.15.40:0/494644674 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0e0039070 con 0x7fa0d803cc60 2024-04-22T08:22:46.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.337+0000 7fa0d37fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:46.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.337+0000 7fa0e8ff9640 4 set_mon_vals no callback set 2024-04-22T08:22:46.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.338+0000 7fa0d37fe640 1 -- 172.21.15.40:0/494644674 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fa0e001b8f0 con 0x7fa0d803cc60 2024-04-22T08:22:46.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.342+0000 7fa0d37fe640 1 -- 172.21.15.40:0/494644674 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7fa0e000e460 con 0x7fa0d803cc60 2024-04-22T08:22:46.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.342+0000 7fa0d37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:46.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.342+0000 7fa0d37fe640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:22:46.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.522+0000 7fa0f1470640 1 -- 172.21.15.40:0/494644674 --> v2:172.21.15.189:6800/30168 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7fa0ec09f810 con 0x7fa0d80377d0 2024-04-22T08:22:46.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.522+0000 7fa0d37fe640 1 -- 172.21.15.40:0/494644674 <== mgr.4102 v2:172.21.15.189:6800/30168 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fa0dc0325e0 con 0x7fa0d80377d0 2024-04-22T08:22:46.522 INFO:teuthology.orchestra.run.smithi040.stderr:instructed osd(s) 3 to deep-scrub 2024-04-22T08:22:46.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.524+0000 7fa0f1470640 1 -- 172.21.15.40:0/494644674 >> v2:172.21.15.189:6800/30168 conn(0x7fa0d80377d0 msgr2=0x7fa0d8039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:46.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.524+0000 7fa0f1470640 1 --2- 172.21.15.40:0/494644674 >> v2:172.21.15.189:6800/30168 conn(0x7fa0d80377d0 0x7fa0d8039c90 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc0097a0 tx=0x7fa0dc02d000 comp rx=0 tx=0).stop 2024-04-22T08:22:46.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.524+0000 7fa0f1470640 10 monclient: shutdown 2024-04-22T08:22:46.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.524+0000 7fa0f1470640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:46.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.524+0000 7fa0f1470640 1 -- 172.21.15.40:0/494644674 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0d803cc60 msgr2=0x7fa0d803f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:46.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.524+0000 7fa0f1470640 1 --2- 172.21.15.40:0/494644674 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0d803cc60 0x7fa0d803f050 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa0e00078a0 tx=0x7fa0e0007f20 comp rx=0 tx=0).stop 2024-04-22T08:22:46.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.524+0000 7fa0f1470640 1 -- 172.21.15.40:0/494644674 shutdown_connections 2024-04-22T08:22:46.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.524+0000 7fa0f1470640 1 --2- 172.21.15.40:0/494644674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0ec0a38d0 0x7fa0d80444b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:46.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.524+0000 7fa0f1470640 1 --2- 172.21.15.40:0/494644674 >> v2:172.21.15.189:6800/30168 conn(0x7fa0d80377d0 0x7fa0d8039c90 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:46.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.524+0000 7fa0f1470640 1 --2- 172.21.15.40:0/494644674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0ec0a20d0 0x7fa0d8041a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:46.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.524+0000 7fa0f1470640 1 --2- 172.21.15.40:0/494644674 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0d803cc60 0x7fa0d803f050 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:46.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.524+0000 7fa0f1470640 1 -- 172.21.15.40:0/494644674 >> 172.21.15.40:0/494644674 conn(0x7fa0ec0a8e60 msgr2=0x7fa0ec06f030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:46.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.524+0000 7fa0f1470640 1 -- 172.21.15.40:0/494644674 shutdown_connections 2024-04-22T08:22:46.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.524+0000 7fa0f1470640 1 -- 172.21.15.40:0/494644674 wait complete. 2024-04-22T08:22:46.534 INFO:tasks.ceph:Scrubbing osd.4 2024-04-22T08:22:46.534 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-22T08:22:46.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.638+0000 7fcb89c62640 10 monclient: get_monmap_and_config 2024-04-22T08:22:46.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.638+0000 7fcb89c62640 10 monclient: build_initial_monmap 2024-04-22T08:22:46.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.638+0000 7fcb89c62640 10 monclient: monmap: 2024-04-22T08:22:46.638 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:46.639 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:46.639 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:22:46.639116+0000 2024-04-22T08:22:46.639 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:22:46.639116+0000 2024-04-22T08:22:46.639 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:46.639 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:46.639 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:22:46.640 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:22:46.640 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:22:46.640 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:22:46.640 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:22:46.640 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:46.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.639+0000 7fcb89c62640 1 Processor -- start 2024-04-22T08:22:46.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.639+0000 7fcb89c62640 1 -- start start 2024-04-22T08:22:46.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.639+0000 7fcb89c62640 10 monclient: init 2024-04-22T08:22:46.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.639+0000 7fcb89c62640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:46.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.639+0000 7fcb89c62640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-04-22T08:22:46.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.639+0000 7fcb89c62640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb84133f00 0x7fcb841342e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:46.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.639+0000 7fcb89c62640 10 monclient(hunting): picked mon.noname-b con 0x7fcb84133f00 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:22:46.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.639+0000 7fcb89c62640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcb8409dfa0 0x7fcb8409ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:46.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.639+0000 7fcb89c62640 10 monclient(hunting): picked mon.noname-a con 0x7fcb8409dfa0 addr v2:172.21.15.40:3300/0 2024-04-22T08:22:46.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.639+0000 7fcb89c62640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fcb84131a20 0x7fcb8409ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:46.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.639+0000 7fcb89c62640 10 monclient(hunting): picked mon.noname-c con 0x7fcb84131a20 addr v2:172.21.15.189:3300/0 2024-04-22T08:22:46.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.639+0000 7fcb89c62640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:46.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.639+0000 7fcb89c62640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb8409e3e0 con 0x7fcb8409dfa0 2024-04-22T08:22:46.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.639+0000 7fcb89c62640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:46.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.639+0000 7fcb89c62640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb84071770 con 0x7fcb84131a20 2024-04-22T08:22:46.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.639+0000 7fcb89c62640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:46.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.639+0000 7fcb89c62640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8409fff0 con 0x7fcb84133f00 2024-04-22T08:22:46.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.639+0000 7fcb89c62640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:46.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.639+0000 7fcb89c62640 10 monclient(hunting): authenticate will time out at 1440.633445s 2024-04-22T08:22:46.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.639+0000 7fcb82ffd640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcb8409dfa0 0x7fcb8409ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:46.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.639+0000 7fcb837fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb84133f00 0x7fcb841342e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:46.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.639+0000 7fcb82ffd640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcb8409dfa0 0x7fcb8409ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44666/0 (socket says 172.21.15.40:44666) 2024-04-22T08:22:46.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.639+0000 7fcb82ffd640 1 -- 172.21.15.40:0/2390680030 learned_addr learned my addr 172.21.15.40:0/2390680030 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:22:46.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.639+0000 7fcb837fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb84133f00 0x7fcb841342e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49956/0 (socket says 172.21.15.40:49956) 2024-04-22T08:22:46.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.639+0000 7fcb82ffd640 10 monclient(hunting): get_auth_request con 0x7fcb8409dfa0 auth_method 0 2024-04-22T08:22:46.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.639+0000 7fcb82ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:46.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.639+0000 7fcb82ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:46.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.639+0000 7fcb82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:46.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.639+0000 7fcb837fe640 10 monclient(hunting): get_auth_request con 0x7fcb84133f00 auth_method 0 2024-04-22T08:22:46.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.639+0000 7fcb837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:46.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.639+0000 7fcb837fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:46.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.639+0000 7fcb837fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:46.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.639+0000 7fcb82ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:46.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.639+0000 7fcb82ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:46.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.639+0000 7fcb82ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:46.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.639+0000 7fcb837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:46.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.639+0000 7fcb837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:46.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.639+0000 7fcb837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:46.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.640+0000 7fcb83fff640 1 --2- 172.21.15.40:0/2390680030 >> v2:172.21.15.189:3300/0 conn(0x7fcb84131a20 0x7fcb8409ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:46.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.640+0000 7fcb83fff640 10 monclient(hunting): get_auth_request con 0x7fcb84131a20 auth_method 0 2024-04-22T08:22:46.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.640+0000 7fcb83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:46.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.640+0000 7fcb83fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:46.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.640+0000 7fcb83fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:46.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.640+0000 7fcb82ffd640 10 monclient(hunting): handle_auth_done global_id 114339 payload 274 2024-04-22T08:22:46.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.640+0000 7fcb82ffd640 1 -- 172.21.15.40:0/2390680030 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb84133f00 msgr2=0x7fcb841342e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:46.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.640+0000 7fcb82ffd640 1 --2- 172.21.15.40:0/2390680030 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb84133f00 0x7fcb841342e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:46.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.640+0000 7fcb82ffd640 1 -- 172.21.15.40:0/2390680030 >> v2:172.21.15.189:3300/0 conn(0x7fcb84131a20 msgr2=0x7fcb8409ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:46.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.640+0000 7fcb82ffd640 1 --2- 172.21.15.40:0/2390680030 >> v2:172.21.15.189:3300/0 conn(0x7fcb84131a20 0x7fcb8409ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:46.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.640+0000 7fcb82ffd640 10 monclient: _finish_hunting 0 2024-04-22T08:22:46.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.640+0000 7fcb82ffd640 1 monclient: found mon.noname-a 2024-04-22T08:22:46.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.640+0000 7fcb82ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:46.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.640+0000 7fcb82ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:22:46.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.640+0000 7fcb82ffd640 1 -- 172.21.15.40:0/2390680030 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb840a02d0 con 0x7fcb8409dfa0 2024-04-22T08:22:46.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.640+0000 7fcb837fe640 1 --2- 172.21.15.40:0/2390680030 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb84133f00 0x7fcb841342e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:22:46.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.642+0000 7fcb82ffd640 1 -- 172.21.15.40:0/2390680030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb8409dfa0 msgr2=0x7fcb8409ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:46.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.642+0000 7fcb82ffd640 1 --2- 172.21.15.40:0/2390680030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb8409dfa0 0x7fcb8409ea50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fcb84065c60 tx=0x7fcb7802b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b10da4bf29668a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:46.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.643+0000 7fcb827fc640 1 -- 172.21.15.40:0/2390680030 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb78031960 con 0x7fcb8409dfa0 2024-04-22T08:22:46.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.643+0000 7fcb827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:46.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.643+0000 7fcb827fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:22:46.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.643+0000 7fcb827fc640 10 monclient: dump: 2024-04-22T08:22:46.658 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:46.659 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:46.659 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:46.659 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:46.659 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:46.659 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:46.659 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:46.659 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:46.659 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:46.659 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:46.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.643+0000 7fcb827fc640 10 monclient: _finish_auth 0 2024-04-22T08:22:46.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.643+0000 7fcb827fc640 10 monclient: _check_auth_tickets 2024-04-22T08:22:46.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.643+0000 7fcb827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:46.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.643+0000 7fcb827fc640 1 -- 172.21.15.40:0/2390680030 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb78031ac0 con 0x7fcb8409dfa0 2024-04-22T08:22:46.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.643+0000 7fcb89c62640 5 monclient: authenticate success, global_id 114339 2024-04-22T08:22:46.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.643+0000 7fcb89c62640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:22:46.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.643+0000 7fcb827fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:46.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.643+0000 7fcb827fc640 1 -- 172.21.15.40:0/2390680030 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb78031e50 con 0x7fcb8409dfa0 2024-04-22T08:22:46.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.643+0000 7fcb827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:46.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.643+0000 7fcb827fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:22:46.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.643+0000 7fcb827fc640 10 monclient: dump: 2024-04-22T08:22:46.660 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:46.660 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:46.660 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:46.660 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:46.660 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:46.660 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:46.660 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:46.660 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:46.661 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:46.661 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:46.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.643+0000 7fcb89c62640 10 monclient: get_monmap_and_config success 2024-04-22T08:22:46.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.643+0000 7fcb89c62640 4 set_mon_vals no callback set 2024-04-22T08:22:46.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.643+0000 7fcb89c62640 10 monclient: shutdown 2024-04-22T08:22:46.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.643+0000 7fcb89c62640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:46.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.643+0000 7fcb89c62640 1 -- 172.21.15.40:0/2390680030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb8409dfa0 msgr2=0x7fcb8409ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:46.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.643+0000 7fcb89c62640 1 --2- 172.21.15.40:0/2390680030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb8409dfa0 0x7fcb8409ea50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fcb84065c60 tx=0x7fcb7802b7f0 comp rx=0 tx=0).stop 2024-04-22T08:22:46.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.643+0000 7fcb89c62640 1 -- 172.21.15.40:0/2390680030 shutdown_connections 2024-04-22T08:22:46.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.643+0000 7fcb89c62640 1 --2- 172.21.15.40:0/2390680030 >> v2:172.21.15.189:3300/0 conn(0x7fcb84131a20 0x7fcb8409ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:46.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.643+0000 7fcb89c62640 1 --2- 172.21.15.40:0/2390680030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb8409dfa0 0x7fcb8409ea50 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:46.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.643+0000 7fcb89c62640 1 --2- 172.21.15.40:0/2390680030 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb84133f00 0x7fcb841342e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:46.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.643+0000 7fcb89c62640 1 -- 172.21.15.40:0/2390680030 >> 172.21.15.40:0/2390680030 conn(0x7fcb840a0b50 msgr2=0x7fcb840a0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:46.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.643+0000 7fcb89c62640 1 -- 172.21.15.40:0/2390680030 shutdown_connections 2024-04-22T08:22:46.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.643+0000 7fcb89c62640 1 -- 172.21.15.40:0/2390680030 wait complete. 2024-04-22T08:22:46.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.644+0000 7fcb89c62640 10 monclient: build_initial_monmap 2024-04-22T08:22:46.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.644+0000 7fcb89c62640 10 monclient: monmap: 2024-04-22T08:22:46.662 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:46.662 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:22:46.662 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:22:46.662 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:22:46.662 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:46.662 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:46.662 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:22:46.662 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:22:46.663 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:22:46.663 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:46.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.644+0000 7fcb89c62640 1 Processor -- start 2024-04-22T08:22:46.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.644+0000 7fcb89c62640 1 -- start start 2024-04-22T08:22:46.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.644+0000 7fcb89c62640 10 monclient: init 2024-04-22T08:22:46.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb89c62640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:46.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb89c62640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-22T08:22:46.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb89c62640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb8409dfa0 0x7fcb841c8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:46.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb89c62640 10 monclient(hunting): picked mon.noname-c con 0x7fcb8409dfa0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:22:46.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb89c62640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb84131a20 0x7fcb841c90e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:46.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb89c62640 10 monclient(hunting): picked mon.noname-a con 0x7fcb84131a20 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:22:46.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb89c62640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb84133f00 0x7fcb8411cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:46.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb89c62640 10 monclient(hunting): picked mon.noname-b con 0x7fcb84133f00 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:22:46.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb89c62640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:46.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb89c62640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb84120e60 con 0x7fcb84131a20 2024-04-22T08:22:46.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb89c62640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:46.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb89c62640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8411f660 con 0x7fcb84133f00 2024-04-22T08:22:46.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb89c62640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:46.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb89c62640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb84071960 con 0x7fcb8409dfa0 2024-04-22T08:22:46.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb89c62640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:46.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb89c62640 10 monclient(hunting): authenticate will time out at 1440.639203s 2024-04-22T08:22:46.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb837fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb8409dfa0 0x7fcb841c8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:46.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb82ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb84131a20 0x7fcb841c90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:46.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb837fe640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb8409dfa0 0x7fcb841c8ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49968/0 (socket says 172.21.15.40:49968) 2024-04-22T08:22:46.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb837fe640 1 -- 172.21.15.40:0/1119354645 learned_addr learned my addr 172.21.15.40:0/1119354645 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:22:46.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb837fe640 10 monclient(hunting): get_auth_request con 0x7fcb8409dfa0 auth_method 0 2024-04-22T08:22:46.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:46.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb837fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:46.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb837fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:46.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb82ffd640 10 monclient(hunting): get_auth_request con 0x7fcb84131a20 auth_method 0 2024-04-22T08:22:46.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb82ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:46.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb82ffd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:46.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:46.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:46.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:46.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:46.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb82ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:46.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb82ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:46.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb82ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:46.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb83fff640 1 --2- 172.21.15.40:0/1119354645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb84133f00 0x7fcb8411cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:46.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb837fe640 10 monclient(hunting): handle_auth_done global_id 124175 payload 824 2024-04-22T08:22:46.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb837fe640 1 -- 172.21.15.40:0/1119354645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb84133f00 msgr2=0x7fcb8411cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:46.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb837fe640 1 --2- 172.21.15.40:0/1119354645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb84133f00 0x7fcb8411cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:46.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb837fe640 1 -- 172.21.15.40:0/1119354645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb84131a20 msgr2=0x7fcb841c90e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:46.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb837fe640 1 --2- 172.21.15.40:0/1119354645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb84131a20 0x7fcb841c90e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:46.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb837fe640 10 monclient: _finish_hunting 0 2024-04-22T08:22:46.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb837fe640 1 monclient: found mon.noname-c 2024-04-22T08:22:46.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:46.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb837fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:22:46.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb837fe640 1 -- 172.21.15.40:0/1119354645 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb74009400 con 0x7fcb8409dfa0 2024-04-22T08:22:46.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb83fff640 10 monclient: get_auth_request con 0x7fcb84133f00 auth_method 0 2024-04-22T08:22:46.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb83fff640 1 --2- 172.21.15.40:0/1119354645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb84133f00 0x7fcb8411cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T08:22:46.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.645+0000 7fcb82ffd640 1 --2- 172.21.15.40:0/1119354645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb84131a20 0x7fcb841c90e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:22:46.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.648+0000 7fcb837fe640 1 -- 172.21.15.40:0/1119354645 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb8409dfa0 msgr2=0x7fcb841c8ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:46.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.648+0000 7fcb837fe640 1 --2- 172.21.15.40:0/1119354645 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb8409dfa0 0x7fcb841c8ba0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fcb7400ebe0 tx=0x7fcb7400c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:46.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.648+0000 7fcb6bfff640 1 -- 172.21.15.40:0/1119354645 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb74009760 con 0x7fcb8409dfa0 2024-04-22T08:22:46.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.648+0000 7fcb6bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:46.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.648+0000 7fcb6bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:22:46.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.648+0000 7fcb6bfff640 10 monclient: dump: 2024-04-22T08:22:46.667 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:46.667 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:46.667 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:46.667 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:46.668 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:46.668 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:46.668 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:46.668 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:46.668 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:46.668 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:46.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.648+0000 7fcb6bfff640 10 monclient: _finish_auth 0 2024-04-22T08:22:46.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.648+0000 7fcb6bfff640 10 monclient: _check_auth_tickets 2024-04-22T08:22:46.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.648+0000 7fcb6bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:46.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.648+0000 7fcb6bfff640 1 -- 172.21.15.40:0/1119354645 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb7400ce40 con 0x7fcb8409dfa0 2024-04-22T08:22:46.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.648+0000 7fcb89c62640 5 monclient: authenticate success, global_id 124175 2024-04-22T08:22:46.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.648+0000 7fcb89c62640 10 monclient: _renew_subs 2024-04-22T08:22:46.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.648+0000 7fcb89c62640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:22:46.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.648+0000 7fcb89c62640 1 -- 172.21.15.40:0/1119354645 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb780094d0 con 0x7fcb8409dfa0 2024-04-22T08:22:46.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.648+0000 7fcb6bfff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:46.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.648+0000 7fcb6bfff640 1 -- 172.21.15.40:0/1119354645 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb7400eee0 con 0x7fcb8409dfa0 2024-04-22T08:22:46.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.648+0000 7fcb6bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:46.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.648+0000 7fcb6bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:22:46.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.648+0000 7fcb6bfff640 10 monclient: dump: 2024-04-22T08:22:46.669 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:46.669 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:46.669 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:46.669 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:46.669 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:46.669 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:46.669 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:46.669 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:46.669 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:46.669 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:46.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.648+0000 7fcb89c62640 10 monclient: _renew_subs 2024-04-22T08:22:46.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.648+0000 7fcb81ffb640 4 set_mon_vals no callback set 2024-04-22T08:22:46.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.648+0000 7fcb89c62640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:22:46.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.648+0000 7fcb89c62640 1 -- 172.21.15.40:0/1119354645 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb841349a0 con 0x7fcb8409dfa0 2024-04-22T08:22:46.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.649+0000 7fcb6bfff640 1 -- 172.21.15.40:0/1119354645 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fcb74013690 con 0x7fcb8409dfa0 2024-04-22T08:22:46.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.649+0000 7fcb89c62640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:22:46.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.649+0000 7fcb89c62640 1 -- 172.21.15.40:0/1119354645 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcb84120fd0 con 0x7fcb8409dfa0 2024-04-22T08:22:46.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.649+0000 7fcb6bfff640 1 --2- 172.21.15.40:0/1119354645 >> v2:172.21.15.189:6800/30168 conn(0x7fcb700377d0 0x7fcb70039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:46.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.649+0000 7fcb6bfff640 1 -- 172.21.15.40:0/1119354645 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(317..317 src has 239..317) v4 ==== 3506+0+0 (secure 0 0 0) 0x7fcb7404a570 con 0x7fcb8409dfa0 2024-04-22T08:22:46.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.649+0000 7fcb6bfff640 1 --2- 172.21.15.40:0/1119354645 >> v2:172.21.15.189:6801/30672 conn(0x7fcb7003d280 0x7fcb7003f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:46.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.649+0000 7fcb6bfff640 1 -- 172.21.15.40:0/1119354645 --> v2:172.21.15.189:6801/30672 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcb7003fd10 con 0x7fcb7003d280 2024-04-22T08:22:46.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.649+0000 7fcb6bfff640 1 -- 172.21.15.40:0/1119354645 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_get_version_reply(handle=1 version=317) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcb7404ca90 con 0x7fcb8409dfa0 2024-04-22T08:22:46.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.649+0000 7fcb6bfff640 10 monclient: handle_get_version_reply finishing 1 version 317 2024-04-22T08:22:46.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.649+0000 7fcb82ffd640 1 --2- 172.21.15.40:0/1119354645 >> v2:172.21.15.189:6800/30168 conn(0x7fcb700377d0 0x7fcb70039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:46.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.649+0000 7fcb82ffd640 10 monclient: get_auth_request con 0x7fcb700377d0 auth_method 0 2024-04-22T08:22:46.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.649+0000 7fcb83fff640 1 --2- 172.21.15.40:0/1119354645 >> v2:172.21.15.189:6801/30672 conn(0x7fcb7003d280 0x7fcb7003f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:46.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.649+0000 7fcb83fff640 10 monclient: get_auth_request con 0x7fcb7003d280 auth_method 0 2024-04-22T08:22:46.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.652+0000 7fcb82ffd640 1 --2- 172.21.15.40:0/1119354645 >> v2:172.21.15.189:6800/30168 conn(0x7fcb700377d0 0x7fcb70039c90 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fcb8406cbf0 tx=0x7fcb78008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:46.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.654+0000 7fcb83fff640 1 --2- 172.21.15.40:0/1119354645 >> v2:172.21.15.189:6801/30672 conn(0x7fcb7003d280 0x7fcb7003f6a0 crc :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:46.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.655+0000 7fcb6bfff640 1 -- 172.21.15.40:0/1119354645 <== osd.4 v2:172.21.15.189:6801/30672 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fcb7003fd10 con 0x7fcb7003d280 2024-04-22T08:22:46.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.667+0000 7fcb89c62640 1 -- 172.21.15.40:0/1119354645 --> v2:172.21.15.189:6801/30672 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fcb84134820 con 0x7fcb7003d280 2024-04-22T08:22:46.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.668+0000 7fcb6bfff640 1 -- 172.21.15.40:0/1119354645 <== osd.4 v2:172.21.15.189:6801/30672 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fcb7c009b80 con 0x7fcb7003d280 2024-04-22T08:22:46.672 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:22:46.672 INFO:teuthology.orchestra.run.smithi040.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-22T08:22:46.672 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:22:46.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.668+0000 7fcb89c62640 1 -- 172.21.15.40:0/1119354645 >> v2:172.21.15.189:6801/30672 conn(0x7fcb7003d280 msgr2=0x7fcb7003f6a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:46.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.668+0000 7fcb89c62640 1 --2- 172.21.15.40:0/1119354645 >> v2:172.21.15.189:6801/30672 conn(0x7fcb7003d280 0x7fcb7003f6a0 crc :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:46.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.668+0000 7fcb89c62640 1 -- 172.21.15.40:0/1119354645 >> v2:172.21.15.189:6800/30168 conn(0x7fcb700377d0 msgr2=0x7fcb70039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:46.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.668+0000 7fcb89c62640 1 --2- 172.21.15.40:0/1119354645 >> v2:172.21.15.189:6800/30168 conn(0x7fcb700377d0 0x7fcb70039c90 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fcb8406cbf0 tx=0x7fcb78008030 comp rx=0 tx=0).stop 2024-04-22T08:22:46.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.668+0000 7fcb89c62640 10 monclient: shutdown 2024-04-22T08:22:46.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.668+0000 7fcb89c62640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:46.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.668+0000 7fcb89c62640 1 -- 172.21.15.40:0/1119354645 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb8409dfa0 msgr2=0x7fcb841c8ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:46.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.668+0000 7fcb89c62640 1 --2- 172.21.15.40:0/1119354645 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb8409dfa0 0x7fcb841c8ba0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fcb7400ebe0 tx=0x7fcb7400c520 comp rx=0 tx=0).stop 2024-04-22T08:22:46.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.668+0000 7fcb83fff640 1 -- 172.21.15.40:0/1119354645 reap_dead start 2024-04-22T08:22:46.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.668+0000 7fcb89c62640 1 -- 172.21.15.40:0/1119354645 shutdown_connections 2024-04-22T08:22:46.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.668+0000 7fcb89c62640 1 -- 172.21.15.40:0/1119354645 >> 172.21.15.40:0/1119354645 conn(0x7fcb840a0b50 msgr2=0x7fcb840a3740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:46.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.669+0000 7fcb89c62640 1 -- 172.21.15.40:0/1119354645 shutdown_connections 2024-04-22T08:22:46.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.669+0000 7fcb89c62640 1 -- 172.21.15.40:0/1119354645 wait complete. 2024-04-22T08:22:46.678 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-22T08:22:46.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.778+0000 7f0e9f877640 10 monclient: get_monmap_and_config 2024-04-22T08:22:46.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.778+0000 7f0e9f877640 10 monclient: build_initial_monmap 2024-04-22T08:22:46.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.778+0000 7f0e9f877640 10 monclient: monmap: 2024-04-22T08:22:46.778 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:46.778 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:46.778 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:22:46.778963+0000 2024-04-22T08:22:46.778 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:22:46.778963+0000 2024-04-22T08:22:46.778 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:46.778 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:46.778 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:22:46.778 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:22:46.779 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:22:46.779 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:22:46.779 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:22:46.779 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:46.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.778+0000 7f0e9f877640 1 Processor -- start 2024-04-22T08:22:46.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.778+0000 7f0e9f877640 1 -- start start 2024-04-22T08:22:46.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.778+0000 7f0e9f877640 10 monclient: init 2024-04-22T08:22:46.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.779+0000 7f0e9f877640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:46.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.779+0000 7f0e9f877640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-04-22T08:22:46.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.779+0000 7f0e9f877640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0e9809ee10 0x7f0e980a32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:46.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.779+0000 7f0e9f877640 10 monclient(hunting): picked mon.noname-c con 0x7f0e9809ee10 addr v2:172.21.15.189:3300/0 2024-04-22T08:22:46.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.779+0000 7f0e9f877640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0e980a1d30 addr v1:172.21.15.189:6789/0 2024-04-22T08:22:46.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.779+0000 7f0e9f877640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0e980a2180 0x7f0e981364a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:46.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.779+0000 7f0e9f877640 10 monclient(hunting): picked mon.noname-a con 0x7f0e980a2180 addr v2:172.21.15.40:3300/0 2024-04-22T08:22:46.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.779+0000 7f0e9f877640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e980a2590 con 0x7f0e980a1d30 2024-04-22T08:22:46.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.779+0000 7f0e9f877640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:46.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.779+0000 7f0e9f877640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e9809f1f0 con 0x7f0e980a2180 2024-04-22T08:22:46.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.779+0000 7f0e9f877640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:46.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.779+0000 7f0e9f877640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e980a39c0 con 0x7f0e9809ee10 2024-04-22T08:22:46.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.779+0000 7f0e9f877640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:46.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.779+0000 7f0e9f877640 10 monclient(hunting): authenticate will time out at 1440.773276s 2024-04-22T08:22:46.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.779+0000 7f0e9de14640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0e980a2180 0x7f0e981364a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:46.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.779+0000 7f0e9de14640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0e980a2180 0x7f0e981364a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44684/0 (socket says 172.21.15.40:44684) 2024-04-22T08:22:46.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.779+0000 7f0e9de14640 1 -- 172.21.15.40:0/2402941894 learned_addr learned my addr 172.21.15.40:0/2402941894 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:22:46.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.779+0000 7f0e9de14640 10 monclient(hunting): get_auth_request con 0x7f0e980a2180 auth_method 0 2024-04-22T08:22:46.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.779+0000 7f0e9de14640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:46.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.779+0000 7f0e9de14640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:46.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.779+0000 7f0e9de14640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:46.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.779+0000 7f0e9de14640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:46.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.779+0000 7f0e9de14640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:46.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.779+0000 7f0e9de14640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:46.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.779+0000 7f0e9d613640 1 --2- 172.21.15.40:0/2402941894 >> v2:172.21.15.189:3300/0 conn(0x7f0e9809ee10 0x7f0e980a32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:46.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.779+0000 7f0e9d613640 10 monclient(hunting): get_auth_request con 0x7f0e9809ee10 auth_method 0 2024-04-22T08:22:46.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.779+0000 7f0e9d613640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:46.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.779+0000 7f0e9d613640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:46.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.779+0000 7f0e9d613640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:46.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.779+0000 7f0e9de14640 10 monclient(hunting): handle_auth_done global_id 114345 payload 274 2024-04-22T08:22:46.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.779+0000 7f0e9de14640 1 -- 172.21.15.40:0/2402941894 >> v2:172.21.15.189:3300/0 conn(0x7f0e9809ee10 msgr2=0x7f0e980a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:46.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.779+0000 7f0e9de14640 1 --2- 172.21.15.40:0/2402941894 >> v2:172.21.15.189:3300/0 conn(0x7f0e9809ee10 0x7f0e980a32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:46.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.779+0000 7f0e9de14640 1 -- 172.21.15.40:0/2402941894 >> v1:172.21.15.189:6789/0 conn(0x7f0e980a1d30 legacy=0x7f0e98135d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:46.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.779+0000 7f0e9de14640 10 monclient: _finish_hunting 0 2024-04-22T08:22:46.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.779+0000 7f0e9de14640 1 monclient: found mon.noname-a 2024-04-22T08:22:46.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.779+0000 7f0e9de14640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:46.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.779+0000 7f0e9de14640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:22:46.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.779+0000 7f0e9de14640 1 -- 172.21.15.40:0/2402941894 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e9809fd30 con 0x7f0e980a2180 2024-04-22T08:22:46.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.782+0000 7f0e9de14640 1 -- 172.21.15.40:0/2402941894 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e980a2180 msgr2=0x7f0e981364a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:46.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.782+0000 7f0e9de14640 1 --2- 172.21.15.40:0/2402941894 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e980a2180 0x7f0e981364a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f0e980a43e0 tx=0x7f0e9002bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efa1384f446ebd6b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:46.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.783+0000 7f0e8ffff640 1 -- 172.21.15.40:0/2402941894 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e9002cb70 con 0x7f0e980a2180 2024-04-22T08:22:46.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.783+0000 7f0e8ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:46.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.783+0000 7f0e8ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:22:46.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.783+0000 7f0e8ffff640 10 monclient: dump: 2024-04-22T08:22:46.787 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:46.787 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:46.787 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:46.787 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:46.787 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:46.787 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:46.787 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:46.787 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:46.787 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:46.787 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:46.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.783+0000 7f0e8ffff640 10 monclient: _finish_auth 0 2024-04-22T08:22:46.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.783+0000 7f0e8ffff640 10 monclient: _check_auth_tickets 2024-04-22T08:22:46.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.783+0000 7f0e8ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:46.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.783+0000 7f0e9f877640 5 monclient: authenticate success, global_id 114345 2024-04-22T08:22:46.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.783+0000 7f0e9f877640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:22:46.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.783+0000 7f0e8ffff640 1 -- 172.21.15.40:0/2402941894 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e9002ccd0 con 0x7f0e980a2180 2024-04-22T08:22:46.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.783+0000 7f0e8ffff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:46.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.783+0000 7f0e9f877640 10 monclient: get_monmap_and_config success 2024-04-22T08:22:46.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.783+0000 7f0e9f877640 4 set_mon_vals no callback set 2024-04-22T08:22:46.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.783+0000 7f0e8ffff640 1 -- 172.21.15.40:0/2402941894 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e90034610 con 0x7f0e980a2180 2024-04-22T08:22:46.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.783+0000 7f0e8ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:46.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.783+0000 7f0e8ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:22:46.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.783+0000 7f0e8ffff640 10 monclient: dump: 2024-04-22T08:22:46.789 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:46.789 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:46.789 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:46.790 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:46.790 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:46.790 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:46.790 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:46.790 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:46.790 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:46.790 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:46.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.783+0000 7f0e9f877640 10 monclient: shutdown 2024-04-22T08:22:46.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.783+0000 7f0e9f877640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:46.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.783+0000 7f0e9f877640 1 -- 172.21.15.40:0/2402941894 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e980a2180 msgr2=0x7f0e981364a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:46.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.783+0000 7f0e9f877640 1 --2- 172.21.15.40:0/2402941894 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e980a2180 0x7f0e981364a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f0e980a43e0 tx=0x7f0e9002bdc0 comp rx=0 tx=0).stop 2024-04-22T08:22:46.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.783+0000 7f0e9f877640 1 -- 172.21.15.40:0/2402941894 shutdown_connections 2024-04-22T08:22:46.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.783+0000 7f0e9f877640 1 --2- 172.21.15.40:0/2402941894 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e980a2180 0x7f0e981364a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:46.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.783+0000 7f0e9f877640 1 --2- 172.21.15.40:0/2402941894 >> v2:172.21.15.189:3300/0 conn(0x7f0e9809ee10 0x7f0e980a32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:46.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.783+0000 7f0e9f877640 1 -- 172.21.15.40:0/2402941894 >> 172.21.15.40:0/2402941894 conn(0x7f0e980a4ef0 msgr2=0x7f0e980a5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:46.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.783+0000 7f0e9f877640 1 -- 172.21.15.40:0/2402941894 shutdown_connections 2024-04-22T08:22:46.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.783+0000 7f0e9f877640 1 -- 172.21.15.40:0/2402941894 wait complete. 2024-04-22T08:22:46.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.784+0000 7f0e9f877640 10 monclient: build_initial_monmap 2024-04-22T08:22:46.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.784+0000 7f0e9f877640 10 monclient: monmap: 2024-04-22T08:22:46.796 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:46.797 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:22:46.797 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:22:46.797 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:22:46.797 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:46.797 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:46.797 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:22:46.797 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:22:46.797 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:22:46.797 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:46.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.784+0000 7f0e9f877640 1 Processor -- start 2024-04-22T08:22:46.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.784+0000 7f0e9f877640 1 -- start start 2024-04-22T08:22:46.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.784+0000 7f0e9f877640 10 monclient: init 2024-04-22T08:22:46.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9f877640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:46.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9f877640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:22:46.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9f877640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e9809ee10 0x7f0e981cade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:46.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9f877640 10 monclient(hunting): picked mon.noname-a con 0x7f0e9809ee10 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:22:46.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9f877640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e980a1d30 0x7f0e981cb320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:46.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9f877640 10 monclient(hunting): picked mon.noname-b con 0x7f0e980a1d30 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:22:46.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9f877640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e980a2180 0x7f0e98122220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:46.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9f877640 10 monclient(hunting): picked mon.noname-c con 0x7f0e980a2180 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:22:46.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9f877640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:46.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9f877640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e98126210 con 0x7f0e9809ee10 2024-04-22T08:22:46.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9f877640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:46.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9f877640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e981249f0 con 0x7f0e980a1d30 2024-04-22T08:22:46.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9f877640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:46.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9f877640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e980a2730 con 0x7f0e980a2180 2024-04-22T08:22:46.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9f877640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:46.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9f877640 10 monclient(hunting): authenticate will time out at 1440.778882s 2024-04-22T08:22:46.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9d613640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e9809ee10 0x7f0e981cade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:46.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9d613640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e9809ee10 0x7f0e981cade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44694/0 (socket says 172.21.15.40:44694) 2024-04-22T08:22:46.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9d613640 1 -- 172.21.15.40:0/1715785543 learned_addr learned my addr 172.21.15.40:0/1715785543 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:22:46.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9d613640 10 monclient(hunting): get_auth_request con 0x7f0e9809ee10 auth_method 0 2024-04-22T08:22:46.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9d613640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:46.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9d613640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:46.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9d613640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:46.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9de14640 1 --2- 172.21.15.40:0/1715785543 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e980a2180 0x7f0e98122220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:46.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9de14640 10 monclient(hunting): get_auth_request con 0x7f0e980a2180 auth_method 0 2024-04-22T08:22:46.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9de14640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:46.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9de14640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:46.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9de14640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:46.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9d613640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:46.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9d613640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:46.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9d613640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:46.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9de14640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:46.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9de14640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:46.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9de14640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:46.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9ce12640 1 --2- 172.21.15.40:0/1715785543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e980a1d30 0x7f0e981cb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:46.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9ce12640 10 monclient(hunting): get_auth_request con 0x7f0e980a1d30 auth_method 0 2024-04-22T08:22:46.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9ce12640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:46.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9ce12640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:46.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9ce12640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:46.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9ce12640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:46.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9ce12640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:46.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9ce12640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:46.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9de14640 10 monclient(hunting): handle_auth_done global_id 124178 payload 824 2024-04-22T08:22:46.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9de14640 1 -- 172.21.15.40:0/1715785543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e980a1d30 msgr2=0x7f0e981cb320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:46.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9de14640 1 --2- 172.21.15.40:0/1715785543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e980a1d30 0x7f0e981cb320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:46.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9de14640 1 -- 172.21.15.40:0/1715785543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e9809ee10 msgr2=0x7f0e981cade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:46.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9de14640 1 --2- 172.21.15.40:0/1715785543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e9809ee10 0x7f0e981cade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:46.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9de14640 10 monclient: _finish_hunting 0 2024-04-22T08:22:46.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9de14640 1 monclient: found mon.noname-c 2024-04-22T08:22:46.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9de14640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:46.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9de14640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:22:46.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9de14640 1 -- 172.21.15.40:0/1715785543 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e90009b30 con 0x7f0e980a2180 2024-04-22T08:22:46.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.785+0000 7f0e9d613640 1 --2- 172.21.15.40:0/1715785543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e9809ee10 0x7f0e981cade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:22:46.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.787+0000 7f0e9de14640 1 -- 172.21.15.40:0/1715785543 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e980a2180 msgr2=0x7f0e98122220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:46.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.787+0000 7f0e9de14640 1 --2- 172.21.15.40:0/1715785543 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e980a2180 0x7f0e98122220 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0e9806c0a0 tx=0x7f0e90004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:46.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.787+0000 7f0e8dffb640 1 -- 172.21.15.40:0/1715785543 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e90034da0 con 0x7f0e980a2180 2024-04-22T08:22:46.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.787+0000 7f0e8dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:46.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.787+0000 7f0e8dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:22:46.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.787+0000 7f0e8dffb640 10 monclient: dump: 2024-04-22T08:22:46.802 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:46.802 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:46.802 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:46.802 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:46.803 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:46.803 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:46.803 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:46.803 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:46.803 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:46.803 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:46.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.787+0000 7f0e8dffb640 10 monclient: _finish_auth 0 2024-04-22T08:22:46.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.787+0000 7f0e8dffb640 10 monclient: _check_auth_tickets 2024-04-22T08:22:46.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.787+0000 7f0e8dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:46.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.788+0000 7f0e9f877640 5 monclient: authenticate success, global_id 124178 2024-04-22T08:22:46.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.788+0000 7f0e8dffb640 1 -- 172.21.15.40:0/1715785543 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e9003e450 con 0x7f0e980a2180 2024-04-22T08:22:46.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.788+0000 7f0e9f877640 10 monclient: _renew_subs 2024-04-22T08:22:46.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.788+0000 7f0e9f877640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:22:46.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.788+0000 7f0e9f877640 1 -- 172.21.15.40:0/1715785543 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e98136000 con 0x7f0e980a2180 2024-04-22T08:22:46.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.788+0000 7f0e8dffb640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:46.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.788+0000 7f0e8dffb640 1 -- 172.21.15.40:0/1715785543 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e9003e800 con 0x7f0e980a2180 2024-04-22T08:22:46.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.788+0000 7f0e8dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:46.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.788+0000 7f0e8dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:22:46.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.788+0000 7f0e8dffb640 10 monclient: dump: 2024-04-22T08:22:46.804 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:46.804 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:46.804 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:46.804 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:46.804 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:46.804 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:46.804 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:46.804 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:46.804 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:46.804 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:46.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.788+0000 7f0e9f877640 10 monclient: _renew_subs 2024-04-22T08:22:46.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.788+0000 7f0e9f877640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:22:46.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.788+0000 7f0e8f7fe640 4 set_mon_vals no callback set 2024-04-22T08:22:46.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.788+0000 7f0e9f877640 1 -- 172.21.15.40:0/1715785543 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e98125990 con 0x7f0e980a2180 2024-04-22T08:22:46.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.788+0000 7f0e8dffb640 1 -- 172.21.15.40:0/1715785543 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f0e9003e960 con 0x7f0e980a2180 2024-04-22T08:22:46.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.789+0000 7f0e9f877640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:46.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.789+0000 7f0e9f877640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:46.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.789+0000 7f0e9f877640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:22:46.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.789+0000 7f0e9f877640 1 -- 172.21.15.40:0/1715785543 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e98123ee0 con 0x7f0e980a2180 2024-04-22T08:22:46.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.789+0000 7f0e8dffb640 1 --2- 172.21.15.40:0/1715785543 >> v2:172.21.15.189:6800/30168 conn(0x7f0e7c0377d0 0x7f0e7c039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:46.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.789+0000 7f0e8dffb640 1 -- 172.21.15.40:0/1715785543 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(317..317 src has 239..317) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f0e9002ce40 con 0x7f0e980a2180 2024-04-22T08:22:46.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.789+0000 7f0e9d613640 1 --2- 172.21.15.40:0/1715785543 >> v2:172.21.15.189:6800/30168 conn(0x7f0e7c0377d0 0x7f0e7c039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:46.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.789+0000 7f0e9d613640 10 monclient: get_auth_request con 0x7f0e7c0377d0 auth_method 0 2024-04-22T08:22:46.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.791+0000 7f0e9d613640 1 --2- 172.21.15.40:0/1715785543 >> v2:172.21.15.189:6800/30168 conn(0x7f0e7c0377d0 0x7f0e7c039c90 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f0e94007e50 tx=0x7f0e94008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:46.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.793+0000 7f0e8dffb640 1 -- 172.21.15.40:0/1715785543 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f0e9003c940 con 0x7f0e980a2180 2024-04-22T08:22:46.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.793+0000 7f0e8dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:46.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.793+0000 7f0e8dffb640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:22:46.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.972+0000 7f0e9f877640 1 -- 172.21.15.40:0/1715785543 --> v2:172.21.15.189:6800/30168 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f0e980a3f00 con 0x7f0e7c0377d0 2024-04-22T08:22:46.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.972+0000 7f0e8dffb640 1 -- 172.21.15.40:0/1715785543 <== mgr.4102 v2:172.21.15.189:6800/30168 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f0e94010d60 con 0x7f0e7c0377d0 2024-04-22T08:22:46.972 INFO:teuthology.orchestra.run.smithi040.stderr:instructed osd(s) 4 to deep-scrub 2024-04-22T08:22:46.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.974+0000 7f0e9f877640 1 -- 172.21.15.40:0/1715785543 >> v2:172.21.15.189:6800/30168 conn(0x7f0e7c0377d0 msgr2=0x7f0e7c039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:46.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.974+0000 7f0e9f877640 1 --2- 172.21.15.40:0/1715785543 >> v2:172.21.15.189:6800/30168 conn(0x7f0e7c0377d0 0x7f0e7c039c90 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f0e94007e50 tx=0x7f0e94008040 comp rx=0 tx=0).stop 2024-04-22T08:22:46.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.974+0000 7f0e9f877640 10 monclient: shutdown 2024-04-22T08:22:46.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.974+0000 7f0e9f877640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:46.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.974+0000 7f0e9f877640 1 -- 172.21.15.40:0/1715785543 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e980a2180 msgr2=0x7f0e98122220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:46.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.974+0000 7f0e9f877640 1 --2- 172.21.15.40:0/1715785543 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e980a2180 0x7f0e98122220 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0e9806c0a0 tx=0x7f0e90004a40 comp rx=0 tx=0).stop 2024-04-22T08:22:46.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.974+0000 7f0e9f877640 1 -- 172.21.15.40:0/1715785543 shutdown_connections 2024-04-22T08:22:46.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.974+0000 7f0e9f877640 1 --2- 172.21.15.40:0/1715785543 >> v2:172.21.15.189:6800/30168 conn(0x7f0e7c0377d0 0x7f0e7c039c90 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:46.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.974+0000 7f0e9f877640 1 --2- 172.21.15.40:0/1715785543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e980a1d30 0x7f0e981cb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:46.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.974+0000 7f0e9f877640 1 --2- 172.21.15.40:0/1715785543 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e980a2180 0x7f0e98122220 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:46.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.974+0000 7f0e9f877640 1 --2- 172.21.15.40:0/1715785543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e9809ee10 0x7f0e981cade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:46.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.974+0000 7f0e9f877640 1 -- 172.21.15.40:0/1715785543 >> 172.21.15.40:0/1715785543 conn(0x7f0e980a4ef0 msgr2=0x7f0e9806f660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:46.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.974+0000 7f0e9f877640 1 -- 172.21.15.40:0/1715785543 shutdown_connections 2024-04-22T08:22:46.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:46.975+0000 7f0e9f877640 1 -- 172.21.15.40:0/1715785543 wait complete. 2024-04-22T08:22:46.984 INFO:tasks.ceph:Scrubbing osd.5 2024-04-22T08:22:46.985 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-22T08:22:47.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.087+0000 7fafcae5f640 10 monclient: get_monmap_and_config 2024-04-22T08:22:47.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.087+0000 7fafcae5f640 10 monclient: build_initial_monmap 2024-04-22T08:22:47.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.088+0000 7fafcae5f640 10 monclient: monmap: 2024-04-22T08:22:47.088 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:47.088 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:47.088 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:22:47.088784+0000 2024-04-22T08:22:47.088 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:22:47.088784+0000 2024-04-22T08:22:47.088 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:47.088 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:47.088 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:22:47.088 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:22:47.088 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:22:47.088 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:22:47.088 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:22:47.088 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:47.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.088+0000 7fafcae5f640 1 Processor -- start 2024-04-22T08:22:47.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.088+0000 7fafcae5f640 1 -- start start 2024-04-22T08:22:47.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.088+0000 7fafcae5f640 10 monclient: init 2024-04-22T08:22:47.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.089+0000 7fafcae5f640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:47.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.089+0000 7fafcae5f640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-04-22T08:22:47.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.089+0000 7fafcae5f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fafc409f2e0 addr v1:172.21.15.40:6789/0 2024-04-22T08:22:47.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.089+0000 7fafcae5f640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fafc409f6c0 0x7fafc409fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:47.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.089+0000 7fafcae5f640 10 monclient(hunting): picked mon.noname-b con 0x7fafc409f6c0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:22:47.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.089+0000 7fafcae5f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fafc4137850 addr v1:172.21.15.189:6789/0 2024-04-22T08:22:47.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.089+0000 7fafcae5f640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafc409c7d0 con 0x7fafc409f2e0 2024-04-22T08:22:47.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.089+0000 7fafcae5f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafc409ca20 con 0x7fafc4137850 2024-04-22T08:22:47.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.089+0000 7fafcae5f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:47.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.089+0000 7fafcae5f640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafc409db10 con 0x7fafc409f6c0 2024-04-22T08:22:47.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.089+0000 7fafcae5f640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:47.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.089+0000 7fafcae5f640 10 monclient(hunting): authenticate will time out at 1441.083098s 2024-04-22T08:22:47.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.089+0000 7fafc8bfb640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fafc409f2e0 0x7fafc409d400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:49386/0 (socket says 172.21.15.40:49386) 2024-04-22T08:22:47.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.089+0000 7fafc8bfb640 1 -- 172.21.15.40:0/3680265692 learned_addr learned my addr 172.21.15.40:0/3680265692 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:22:47.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.089+0000 7fafc3fff640 1 --2- 172.21.15.40:0/3680265692 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fafc409f6c0 0x7fafc409fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:47.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.089+0000 7fafc3fff640 10 monclient(hunting): get_auth_request con 0x7fafc409f6c0 auth_method 0 2024-04-22T08:22:47.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.089+0000 7fafc3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:47.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.089+0000 7fafc3fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:47.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.089+0000 7fafc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:47.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.089+0000 7fafc3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:47.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.089+0000 7fafc3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:47.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.089+0000 7fafc3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:47.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.089+0000 7fafc3fff640 10 monclient(hunting): handle_auth_done global_id 124181 payload 274 2024-04-22T08:22:47.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.089+0000 7fafc3fff640 1 -- 172.21.15.40:0/3680265692 >> v1:172.21.15.189:6789/0 conn(0x7fafc4137850 legacy=0x7fafc4139c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:47.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.089+0000 7fafc3fff640 1 -- 172.21.15.40:0/3680265692 >> v1:172.21.15.40:6789/0 conn(0x7fafc409f2e0 legacy=0x7fafc409d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:47.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.089+0000 7fafc3fff640 10 monclient: _finish_hunting 0 2024-04-22T08:22:47.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.089+0000 7fafc3fff640 1 monclient: found mon.noname-b 2024-04-22T08:22:47.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.089+0000 7fafc3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:47.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.089+0000 7fafc3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:22:47.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.089+0000 7fafc3fff640 1 -- 172.21.15.40:0/3680265692 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc409e570 con 0x7fafc409f6c0 2024-04-22T08:22:47.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.092+0000 7fafc3fff640 1 -- 172.21.15.40:0/3680265692 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fafc409f6c0 msgr2=0x7fafc409fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:47.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.092+0000 7fafc3fff640 1 --2- 172.21.15.40:0/3680265692 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fafc409f6c0 0x7fafc409fae0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fafb80097d0 tx=0x7fafb802b6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca87678cd5243458 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:47.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.092+0000 7fafc37fe640 1 -- 172.21.15.40:0/3680265692 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb8030a00 con 0x7fafc409f6c0 2024-04-22T08:22:47.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.092+0000 7fafc37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:47.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.092+0000 7fafc37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:22:47.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.092+0000 7fafc37fe640 10 monclient: dump: 2024-04-22T08:22:47.101 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:47.101 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:47.101 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:47.101 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:47.101 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:47.101 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:47.101 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:47.101 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:47.101 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:47.101 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:47.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.092+0000 7fafc37fe640 10 monclient: _finish_auth 0 2024-04-22T08:22:47.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.092+0000 7fafc37fe640 10 monclient: _check_auth_tickets 2024-04-22T08:22:47.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.092+0000 7fafc37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:47.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.092+0000 7fafc37fe640 1 -- 172.21.15.40:0/3680265692 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafb802e050 con 0x7fafc409f6c0 2024-04-22T08:22:47.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.092+0000 7fafc37fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:47.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.092+0000 7fafc37fe640 1 -- 172.21.15.40:0/3680265692 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb8030d80 con 0x7fafc409f6c0 2024-04-22T08:22:47.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.092+0000 7fafcae5f640 5 monclient: authenticate success, global_id 124181 2024-04-22T08:22:47.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.092+0000 7fafcae5f640 10 monclient: get_monmap_and_config success 2024-04-22T08:22:47.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.092+0000 7fafcae5f640 4 set_mon_vals no callback set 2024-04-22T08:22:47.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.092+0000 7fafc37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:47.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.092+0000 7fafc37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:22:47.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.092+0000 7fafc37fe640 10 monclient: dump: 2024-04-22T08:22:47.102 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:47.102 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:47.102 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:47.103 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:47.103 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:47.103 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:47.103 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:47.103 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:47.103 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:47.103 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:47.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.093+0000 7fafcae5f640 10 monclient: shutdown 2024-04-22T08:22:47.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.093+0000 7fafcae5f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:47.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.093+0000 7fafcae5f640 1 -- 172.21.15.40:0/3680265692 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fafc409f6c0 msgr2=0x7fafc409fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:47.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.093+0000 7fafcae5f640 1 --2- 172.21.15.40:0/3680265692 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fafc409f6c0 0x7fafc409fae0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fafb80097d0 tx=0x7fafb802b6f0 comp rx=0 tx=0).stop 2024-04-22T08:22:47.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.093+0000 7fafcae5f640 1 -- 172.21.15.40:0/3680265692 shutdown_connections 2024-04-22T08:22:47.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.093+0000 7fafcae5f640 1 --2- 172.21.15.40:0/3680265692 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fafc409f6c0 0x7fafc409fae0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:47.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.093+0000 7fafcae5f640 1 -- 172.21.15.40:0/3680265692 >> 172.21.15.40:0/3680265692 conn(0x7fafc40a0d30 msgr2=0x7fafc40a1140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:47.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.093+0000 7fafcae5f640 1 -- 172.21.15.40:0/3680265692 shutdown_connections 2024-04-22T08:22:47.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.093+0000 7fafcae5f640 1 -- 172.21.15.40:0/3680265692 wait complete. 2024-04-22T08:22:47.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.094+0000 7fafcae5f640 10 monclient: build_initial_monmap 2024-04-22T08:22:47.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.094+0000 7fafcae5f640 10 monclient: monmap: 2024-04-22T08:22:47.104 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:47.104 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:22:47.105 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:22:47.105 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:22:47.105 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:47.105 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:47.105 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:22:47.105 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:22:47.105 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:22:47.105 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:47.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.094+0000 7fafcae5f640 1 Processor -- start 2024-04-22T08:22:47.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.094+0000 7fafcae5f640 1 -- start start 2024-04-22T08:22:47.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.094+0000 7fafcae5f640 10 monclient: init 2024-04-22T08:22:47.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafcae5f640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:47.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafcae5f640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:22:47.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafcae5f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafc409f2e0 0x7fafc41c8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:47.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafcae5f640 10 monclient(hunting): picked mon.noname-a con 0x7fafc409f2e0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:22:47.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafcae5f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafc409f6c0 0x7fafc41c9220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:47.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafcae5f640 10 monclient(hunting): picked mon.noname-b con 0x7fafc409f6c0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:22:47.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafcae5f640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fafc4137850 0x7fafc411cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:47.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafcae5f640 10 monclient(hunting): picked mon.noname-c con 0x7fafc4137850 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:22:47.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafcae5f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:47.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafcae5f640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc411f3d0 con 0x7fafc409f2e0 2024-04-22T08:22:47.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafcae5f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:47.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafcae5f640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc4120500 con 0x7fafc409f6c0 2024-04-22T08:22:47.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafcae5f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:47.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafcae5f640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafc409bbf0 con 0x7fafc4137850 2024-04-22T08:22:47.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafcae5f640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:47.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafcae5f640 10 monclient(hunting): authenticate will time out at 1441.088945s 2024-04-22T08:22:47.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafc8bfb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafc409f2e0 0x7fafc41c8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:47.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafc8bfb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafc409f2e0 0x7fafc41c8ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44706/0 (socket says 172.21.15.40:44706) 2024-04-22T08:22:47.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafc93fc640 1 --2- 172.21.15.40:0/2300649775 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fafc4137850 0x7fafc411cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:47.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafc8bfb640 1 -- 172.21.15.40:0/2300649775 learned_addr learned my addr 172.21.15.40:0/2300649775 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:22:47.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafc8bfb640 10 monclient(hunting): get_auth_request con 0x7fafc409f2e0 auth_method 0 2024-04-22T08:22:47.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafc8bfb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:47.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafc8bfb640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:47.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafc8bfb640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:47.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafc93fc640 10 monclient(hunting): get_auth_request con 0x7fafc4137850 auth_method 0 2024-04-22T08:22:47.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafc93fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:47.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafc93fc640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:47.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafc93fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:47.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafc8bfb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:47.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafc8bfb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:47.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafc8bfb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:47.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafc93fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:47.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafc93fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:47.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafc93fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:47.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafc3fff640 1 --2- 172.21.15.40:0/2300649775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafc409f6c0 0x7fafc41c9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:47.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafc3fff640 10 monclient(hunting): get_auth_request con 0x7fafc409f6c0 auth_method 0 2024-04-22T08:22:47.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafc3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:47.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafc3fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:47.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:47.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafc3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:47.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafc3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:47.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafc3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:47.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafc8bfb640 10 monclient(hunting): handle_auth_done global_id 114351 payload 824 2024-04-22T08:22:47.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafc8bfb640 1 -- 172.21.15.40:0/2300649775 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fafc4137850 msgr2=0x7fafc411cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:47.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafc8bfb640 1 --2- 172.21.15.40:0/2300649775 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fafc4137850 0x7fafc411cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:47.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafc8bfb640 1 -- 172.21.15.40:0/2300649775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafc409f6c0 msgr2=0x7fafc41c9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:47.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafc8bfb640 1 --2- 172.21.15.40:0/2300649775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafc409f6c0 0x7fafc41c9220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:47.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafc8bfb640 10 monclient: _finish_hunting 0 2024-04-22T08:22:47.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafc8bfb640 1 monclient: found mon.noname-a 2024-04-22T08:22:47.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafc8bfb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:47.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafc8bfb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:22:47.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafc8bfb640 1 -- 172.21.15.40:0/2300649775 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb8009400 con 0x7fafc409f2e0 2024-04-22T08:22:47.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.095+0000 7fafc93fc640 1 --2- 172.21.15.40:0/2300649775 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fafc4137850 0x7fafc411cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:22:47.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.097+0000 7fafc8bfb640 1 -- 172.21.15.40:0/2300649775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafc409f2e0 msgr2=0x7fafc41c8ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:47.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.097+0000 7fafc8bfb640 1 --2- 172.21.15.40:0/2300649775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafc409f2e0 0x7fafc41c8ce0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fafb400c460 tx=0x7fafb400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:47.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.098+0000 7fafc17fa640 1 -- 172.21.15.40:0/2300649775 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb4007070 con 0x7fafc409f2e0 2024-04-22T08:22:47.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.098+0000 7fafc17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:47.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.098+0000 7fafc17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:22:47.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.098+0000 7fafc17fa640 10 monclient: dump: 2024-04-22T08:22:47.111 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:47.111 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:47.111 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:47.111 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:47.111 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:47.111 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:47.111 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:47.111 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:47.111 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:47.111 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:47.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.098+0000 7fafc17fa640 10 monclient: _finish_auth 0 2024-04-22T08:22:47.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.098+0000 7fafc17fa640 10 monclient: _check_auth_tickets 2024-04-22T08:22:47.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.098+0000 7fafc17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:47.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.098+0000 7fafc17fa640 1 -- 172.21.15.40:0/2300649775 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafb40071f0 con 0x7fafc409f2e0 2024-04-22T08:22:47.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.098+0000 7fafc17fa640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:47.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.098+0000 7fafc17fa640 1 -- 172.21.15.40:0/2300649775 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb40075a0 con 0x7fafc409f2e0 2024-04-22T08:22:47.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.098+0000 7fafc17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:47.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.098+0000 7fafc17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:22:47.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.098+0000 7fafc17fa640 10 monclient: dump: 2024-04-22T08:22:47.112 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:47.112 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:47.112 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:47.112 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:47.112 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:47.112 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:47.113 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:47.113 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:47.113 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:47.113 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:47.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.098+0000 7fafcae5f640 5 monclient: authenticate success, global_id 114351 2024-04-22T08:22:47.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.098+0000 7fafcae5f640 10 monclient: _renew_subs 2024-04-22T08:22:47.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.098+0000 7fafcae5f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:47.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.098+0000 7fafcae5f640 1 -- 172.21.15.40:0/2300649775 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafc4136c40 con 0x7fafc409f2e0 2024-04-22T08:22:47.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.098+0000 7fafc2ffd640 4 set_mon_vals no callback set 2024-04-22T08:22:47.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.098+0000 7fafcae5f640 10 monclient: _renew_subs 2024-04-22T08:22:47.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.098+0000 7fafcae5f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:47.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.098+0000 7fafcae5f640 1 -- 172.21.15.40:0/2300649775 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafc411e9d0 con 0x7fafc409f2e0 2024-04-22T08:22:47.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.098+0000 7fafc17fa640 1 -- 172.21.15.40:0/2300649775 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fafb401b400 con 0x7fafc409f2e0 2024-04-22T08:22:47.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.098+0000 7fafcae5f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2024-04-22T08:22:47.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.098+0000 7fafcae5f640 1 -- 172.21.15.40:0/2300649775 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fafc40a0020 con 0x7fafc409f2e0 2024-04-22T08:22:47.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.099+0000 7fafc17fa640 1 --2- 172.21.15.40:0/2300649775 >> v2:172.21.15.189:6800/30168 conn(0x7fafb00377d0 0x7fafb0039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:47.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.099+0000 7fafc17fa640 1 -- 172.21.15.40:0/2300649775 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(317..317 src has 239..317) v4 ==== 3506+0+0 (secure 0 0 0) 0x7fafb4049e30 con 0x7fafc409f2e0 2024-04-22T08:22:47.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.099+0000 7fafc17fa640 1 --2- 172.21.15.40:0/2300649775 >> v2:172.21.15.189:6806/30674 conn(0x7fafb003d280 0x7fafb003f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:47.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.099+0000 7fafc17fa640 1 -- 172.21.15.40:0/2300649775 --> v2:172.21.15.189:6806/30674 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fafb003fd10 con 0x7fafb003d280 2024-04-22T08:22:47.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.099+0000 7fafc17fa640 1 -- 172.21.15.40:0/2300649775 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=317) v2 ==== 24+0+0 (secure 0 0 0) 0x7fafb404f380 con 0x7fafc409f2e0 2024-04-22T08:22:47.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.099+0000 7fafc17fa640 10 monclient: handle_get_version_reply finishing 1 version 317 2024-04-22T08:22:47.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.099+0000 7fafc3fff640 1 --2- 172.21.15.40:0/2300649775 >> v2:172.21.15.189:6800/30168 conn(0x7fafb00377d0 0x7fafb0039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:47.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.099+0000 7fafc3fff640 10 monclient: get_auth_request con 0x7fafb00377d0 auth_method 0 2024-04-22T08:22:47.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.099+0000 7fafc93fc640 1 --2- 172.21.15.40:0/2300649775 >> v2:172.21.15.189:6806/30674 conn(0x7fafb003d280 0x7fafb003f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:47.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.099+0000 7fafc93fc640 10 monclient: get_auth_request con 0x7fafb003d280 auth_method 0 2024-04-22T08:22:47.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.101+0000 7fafc3fff640 1 --2- 172.21.15.40:0/2300649775 >> v2:172.21.15.189:6800/30168 conn(0x7fafb00377d0 0x7fafb0039c90 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fafc409b080 tx=0x7fafb8008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:47.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.104+0000 7fafc93fc640 1 --2- 172.21.15.40:0/2300649775 >> v2:172.21.15.189:6806/30674 conn(0x7fafb003d280 0x7fafb003f6a0 crc :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:47.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.105+0000 7fafc17fa640 1 -- 172.21.15.40:0/2300649775 <== osd.5 v2:172.21.15.189:6806/30674 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fafb003fd10 con 0x7fafb003d280 2024-04-22T08:22:47.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.117+0000 7fafcae5f640 1 -- 172.21.15.40:0/2300649775 --> v2:172.21.15.189:6806/30674 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fafc4120670 con 0x7fafb003d280 2024-04-22T08:22:47.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.118+0000 7fafc17fa640 1 -- 172.21.15.40:0/2300649775 <== osd.5 v2:172.21.15.189:6806/30674 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fafc4120670 con 0x7fafb003d280 2024-04-22T08:22:47.117 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-22T08:22:47.117 INFO:teuthology.orchestra.run.smithi040.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-22T08:22:47.117 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-22T08:22:47.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.118+0000 7fafcae5f640 1 -- 172.21.15.40:0/2300649775 >> v2:172.21.15.189:6806/30674 conn(0x7fafb003d280 msgr2=0x7fafb003f6a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:47.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.118+0000 7fafcae5f640 1 --2- 172.21.15.40:0/2300649775 >> v2:172.21.15.189:6806/30674 conn(0x7fafb003d280 0x7fafb003f6a0 crc :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:47.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.118+0000 7fafcae5f640 1 -- 172.21.15.40:0/2300649775 >> v2:172.21.15.189:6800/30168 conn(0x7fafb00377d0 msgr2=0x7fafb0039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:47.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.118+0000 7fafcae5f640 1 --2- 172.21.15.40:0/2300649775 >> v2:172.21.15.189:6800/30168 conn(0x7fafb00377d0 0x7fafb0039c90 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fafc409b080 tx=0x7fafb8008030 comp rx=0 tx=0).stop 2024-04-22T08:22:47.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.118+0000 7fafcae5f640 10 monclient: shutdown 2024-04-22T08:22:47.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.118+0000 7fafcae5f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:47.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.118+0000 7fafcae5f640 1 -- 172.21.15.40:0/2300649775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafc409f2e0 msgr2=0x7fafc41c8ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:47.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.118+0000 7fafcae5f640 1 --2- 172.21.15.40:0/2300649775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafc409f2e0 0x7fafc41c8ce0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fafb400c460 tx=0x7fafb400c920 comp rx=0 tx=0).stop 2024-04-22T08:22:47.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.118+0000 7fafc93fc640 1 -- 172.21.15.40:0/2300649775 reap_dead start 2024-04-22T08:22:47.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.118+0000 7fafcae5f640 1 -- 172.21.15.40:0/2300649775 shutdown_connections 2024-04-22T08:22:47.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.118+0000 7fafcae5f640 1 -- 172.21.15.40:0/2300649775 >> 172.21.15.40:0/2300649775 conn(0x7fafc40a0d30 msgr2=0x7fafc40a1e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:47.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.118+0000 7fafcae5f640 1 -- 172.21.15.40:0/2300649775 shutdown_connections 2024-04-22T08:22:47.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.118+0000 7fafcae5f640 1 -- 172.21.15.40:0/2300649775 wait complete. 2024-04-22T08:22:47.128 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-22T08:22:47.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.227+0000 7f1cc8155640 10 monclient: get_monmap_and_config 2024-04-22T08:22:47.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.227+0000 7f1cc8155640 10 monclient: build_initial_monmap 2024-04-22T08:22:47.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.227+0000 7f1cc8155640 10 monclient: monmap: 2024-04-22T08:22:47.228 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:47.228 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:47.228 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:22:47.228746+0000 2024-04-22T08:22:47.228 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:22:47.228746+0000 2024-04-22T08:22:47.228 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:47.228 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:47.228 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:22:47.228 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:22:47.228 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:22:47.228 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:22:47.228 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:22:47.229 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:47.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.228+0000 7f1cc8155640 1 Processor -- start 2024-04-22T08:22:47.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.228+0000 7f1cc8155640 1 -- start start 2024-04-22T08:22:47.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.228+0000 7f1cc8155640 10 monclient: init 2024-04-22T08:22:47.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.229+0000 7f1cc8155640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:47.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.229+0000 7f1cc8155640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-04-22T08:22:47.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.229+0000 7f1cc8155640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1cc012fc40 0x7f1cc0130020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:47.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.229+0000 7f1cc8155640 10 monclient(hunting): picked mon.noname-a con 0x7f1cc012fc40 addr v2:172.21.15.40:3300/0 2024-04-22T08:22:47.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.229+0000 7f1cc8155640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1cc009e210 addr v1:172.21.15.40:6789/0 2024-04-22T08:22:47.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.229+0000 7f1cc8155640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1cc01330a0 addr v1:172.21.15.189:6789/0 2024-04-22T08:22:47.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.229+0000 7f1cc8155640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cc009ea50 con 0x7f1cc009e210 2024-04-22T08:22:47.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.229+0000 7f1cc8155640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cc009cf90 con 0x7f1cc01330a0 2024-04-22T08:22:47.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.229+0000 7f1cc8155640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:47.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.229+0000 7f1cc8155640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cc009d270 con 0x7f1cc012fc40 2024-04-22T08:22:47.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.229+0000 7f1cc8155640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:47.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.229+0000 7f1cc8155640 10 monclient(hunting): authenticate will time out at 1441.223025s 2024-04-22T08:22:47.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.229+0000 7f1cc5ef1640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1cc012fc40 0x7f1cc0130020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:47.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.229+0000 7f1cc56f0640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f1cc009e210 0x7f1cc0132990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:49388/0 (socket says 172.21.15.40:49388) 2024-04-22T08:22:47.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.229+0000 7f1cc56f0640 1 -- 172.21.15.40:0/2020554172 learned_addr learned my addr 172.21.15.40:0/2020554172 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:22:47.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.229+0000 7f1cc5ef1640 10 monclient(hunting): get_auth_request con 0x7f1cc012fc40 auth_method 0 2024-04-22T08:22:47.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.229+0000 7f1cc5ef1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:47.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.229+0000 7f1cc5ef1640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:47.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.229+0000 7f1cc5ef1640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:47.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.229+0000 7f1cc5ef1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:47.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.229+0000 7f1cc5ef1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:47.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.229+0000 7f1cc5ef1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:47.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.229+0000 7f1cc5ef1640 10 monclient(hunting): handle_auth_done global_id 114354 payload 274 2024-04-22T08:22:47.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.229+0000 7f1cc5ef1640 1 -- 172.21.15.40:0/2020554172 >> v1:172.21.15.189:6789/0 conn(0x7f1cc01330a0 legacy=0x7f1cc013acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:47.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.229+0000 7f1cc5ef1640 1 -- 172.21.15.40:0/2020554172 >> v1:172.21.15.40:6789/0 conn(0x7f1cc009e210 legacy=0x7f1cc0132990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:47.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.229+0000 7f1cc5ef1640 10 monclient: _finish_hunting 0 2024-04-22T08:22:47.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.229+0000 7f1cc5ef1640 1 monclient: found mon.noname-a 2024-04-22T08:22:47.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.229+0000 7f1cc5ef1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:47.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.229+0000 7f1cc5ef1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2024-04-22T08:22:47.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.229+0000 7f1cc5ef1640 1 -- 172.21.15.40:0/2020554172 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc009d550 con 0x7f1cc012fc40 2024-04-22T08:22:47.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.232+0000 7f1cc5ef1640 1 -- 172.21.15.40:0/2020554172 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1cc012fc40 msgr2=0x7f1cc0130020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:47.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.232+0000 7f1cc5ef1640 1 --2- 172.21.15.40:0/2020554172 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1cc012fc40 0x7f1cc0130020 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1cb80097a0 tx=0x7f1cb802b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1534cc6825e07e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:47.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.232+0000 7f1cc4eef640 1 -- 172.21.15.40:0/2020554172 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb8030a00 con 0x7f1cc012fc40 2024-04-22T08:22:47.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.232+0000 7f1cc4eef640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:47.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.232+0000 7f1cc4eef640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-22T08:22:47.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.232+0000 7f1cc4eef640 10 monclient: dump: 2024-04-22T08:22:47.240 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:47.240 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:47.240 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:47.240 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:47.240 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:47.240 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:47.240 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:47.240 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:47.240 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:47.240 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:47.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.232+0000 7f1cc4eef640 10 monclient: _finish_auth 0 2024-04-22T08:22:47.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.232+0000 7f1cc4eef640 10 monclient: _check_auth_tickets 2024-04-22T08:22:47.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.232+0000 7f1cc4eef640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:47.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.232+0000 7f1cc4eef640 1 -- 172.21.15.40:0/2020554172 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cb802e050 con 0x7f1cc012fc40 2024-04-22T08:22:47.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.232+0000 7f1cc8155640 5 monclient: authenticate success, global_id 114354 2024-04-22T08:22:47.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.232+0000 7f1cc8155640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:22:47.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.232+0000 7f1cc4eef640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:47.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.232+0000 7f1cc8155640 10 monclient: get_monmap_and_config success 2024-04-22T08:22:47.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.232+0000 7f1cc4eef640 1 -- 172.21.15.40:0/2020554172 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb8030d80 con 0x7f1cc012fc40 2024-04-22T08:22:47.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.232+0000 7f1cc8155640 4 set_mon_vals no callback set 2024-04-22T08:22:47.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.232+0000 7f1cc4eef640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:47.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.232+0000 7f1cc4eef640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-22T08:22:47.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.232+0000 7f1cc4eef640 10 monclient: dump: 2024-04-22T08:22:47.241 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:47.242 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:47.242 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:47.242 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:47.242 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:47.242 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:47.242 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:47.243 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:47.243 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:47.243 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:47.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.233+0000 7f1cc8155640 10 monclient: shutdown 2024-04-22T08:22:47.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.233+0000 7f1cc8155640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:47.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.233+0000 7f1cc8155640 1 -- 172.21.15.40:0/2020554172 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1cc012fc40 msgr2=0x7f1cc0130020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:47.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.233+0000 7f1cc8155640 1 --2- 172.21.15.40:0/2020554172 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1cc012fc40 0x7f1cc0130020 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1cb80097a0 tx=0x7f1cb802b710 comp rx=0 tx=0).stop 2024-04-22T08:22:47.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.233+0000 7f1cc8155640 1 -- 172.21.15.40:0/2020554172 shutdown_connections 2024-04-22T08:22:47.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.233+0000 7f1cc8155640 1 --2- 172.21.15.40:0/2020554172 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1cc012fc40 0x7f1cc0130020 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:47.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.233+0000 7f1cc8155640 1 -- 172.21.15.40:0/2020554172 >> 172.21.15.40:0/2020554172 conn(0x7f1cc00a0b50 msgr2=0x7f1cc00a0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:47.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.233+0000 7f1cc8155640 1 -- 172.21.15.40:0/2020554172 shutdown_connections 2024-04-22T08:22:47.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.233+0000 7f1cc8155640 1 -- 172.21.15.40:0/2020554172 wait complete. 2024-04-22T08:22:47.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.233+0000 7f1cc8155640 10 monclient: build_initial_monmap 2024-04-22T08:22:47.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.233+0000 7f1cc8155640 10 monclient: monmap: 2024-04-22T08:22:47.244 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:47.244 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:22:47.244 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:22:47.244 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:22:47.244 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:47.244 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:47.244 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:22:47.244 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:22:47.244 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:22:47.244 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:47.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.234+0000 7f1cc8155640 1 Processor -- start 2024-04-22T08:22:47.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.234+0000 7f1cc8155640 1 -- start start 2024-04-22T08:22:47.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.234+0000 7f1cc8155640 10 monclient: init 2024-04-22T08:22:47.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.234+0000 7f1cc8155640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:47.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.234+0000 7f1cc8155640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-22T08:22:47.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.234+0000 7f1cc8155640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1cc009e210 0x7f1cc01cab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:47.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.234+0000 7f1cc8155640 10 monclient(hunting): picked mon.noname-c con 0x7f1cc009e210 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:22:47.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.234+0000 7f1cc8155640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1cc012fc40 0x7f1cc012ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:47.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.234+0000 7f1cc8155640 10 monclient(hunting): picked mon.noname-b con 0x7f1cc012fc40 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:22:47.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.234+0000 7f1cc8155640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1cc01330a0 0x7f1cc012f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:47.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.234+0000 7f1cc8155640 10 monclient(hunting): picked mon.noname-a con 0x7f1cc01330a0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:22:47.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.234+0000 7f1cc8155640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:47.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.234+0000 7f1cc8155640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc0121d00 con 0x7f1cc01330a0 2024-04-22T08:22:47.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.234+0000 7f1cc8155640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:47.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.234+0000 7f1cc8155640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc0071800 con 0x7f1cc012fc40 2024-04-22T08:22:47.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.234+0000 7f1cc8155640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:47.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.234+0000 7f1cc8155640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc01322a0 con 0x7f1cc009e210 2024-04-22T08:22:47.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.234+0000 7f1cc8155640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:47.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.234+0000 7f1cc8155640 10 monclient(hunting): authenticate will time out at 1441.228432s 2024-04-22T08:22:47.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.234+0000 7f1cc5ef1640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1cc009e210 0x7f1cc01cab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:47.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.234+0000 7f1cc5ef1640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1cc009e210 0x7f1cc01cab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:50000/0 (socket says 172.21.15.40:50000) 2024-04-22T08:22:47.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.234+0000 7f1cc5ef1640 1 -- 172.21.15.40:0/3940460789 learned_addr learned my addr 172.21.15.40:0/3940460789 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:22:47.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.234+0000 7f1cc5ef1640 10 monclient(hunting): get_auth_request con 0x7f1cc009e210 auth_method 0 2024-04-22T08:22:47.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.234+0000 7f1cc5ef1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:47.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.234+0000 7f1cc5ef1640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:47.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.234+0000 7f1cc5ef1640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:47.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.234+0000 7f1cc66f2640 1 --2- 172.21.15.40:0/3940460789 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1cc01330a0 0x7f1cc012f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:47.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.234+0000 7f1cc66f2640 10 monclient(hunting): get_auth_request con 0x7f1cc01330a0 auth_method 0 2024-04-22T08:22:47.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.234+0000 7f1cc66f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:47.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.234+0000 7f1cc66f2640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:47.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.234+0000 7f1cc66f2640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:47.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.234+0000 7f1cc5ef1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:47.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.234+0000 7f1cc5ef1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:47.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.234+0000 7f1cc5ef1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:47.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.234+0000 7f1cc66f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:47.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.234+0000 7f1cc66f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:47.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.234+0000 7f1cc66f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:47.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.234+0000 7f1cc56f0640 1 --2- 172.21.15.40:0/3940460789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1cc012fc40 0x7f1cc012ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:47.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.234+0000 7f1cc56f0640 10 monclient(hunting): get_auth_request con 0x7f1cc012fc40 auth_method 0 2024-04-22T08:22:47.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.234+0000 7f1cc56f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:47.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.234+0000 7f1cc56f0640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:47.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.234+0000 7f1cc56f0640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:47.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.235+0000 7f1cc56f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:47.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.235+0000 7f1cc56f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:47.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.235+0000 7f1cc56f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:47.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.235+0000 7f1cc5ef1640 10 monclient(hunting): handle_auth_done global_id 124187 payload 824 2024-04-22T08:22:47.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.235+0000 7f1cc5ef1640 1 -- 172.21.15.40:0/3940460789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1cc012fc40 msgr2=0x7f1cc012ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:47.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.235+0000 7f1cc5ef1640 1 --2- 172.21.15.40:0/3940460789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1cc012fc40 0x7f1cc012ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:47.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.235+0000 7f1cc5ef1640 1 -- 172.21.15.40:0/3940460789 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1cc01330a0 msgr2=0x7f1cc012f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:47.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.235+0000 7f1cc5ef1640 1 --2- 172.21.15.40:0/3940460789 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1cc01330a0 0x7f1cc012f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:47.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.235+0000 7f1cc5ef1640 10 monclient: _finish_hunting 0 2024-04-22T08:22:47.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.235+0000 7f1cc5ef1640 1 monclient: found mon.noname-c 2024-04-22T08:22:47.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.235+0000 7f1cc5ef1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:47.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.235+0000 7f1cc5ef1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:22:47.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.235+0000 7f1cc5ef1640 1 -- 172.21.15.40:0/3940460789 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cb8009400 con 0x7f1cc009e210 2024-04-22T08:22:47.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.235+0000 7f1cc66f2640 1 --2- 172.21.15.40:0/3940460789 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1cc01330a0 0x7f1cc012f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:22:47.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.237+0000 7f1cc5ef1640 1 -- 172.21.15.40:0/3940460789 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1cc009e210 msgr2=0x7f1cc01cab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:47.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.237+0000 7f1cc5ef1640 1 --2- 172.21.15.40:0/3940460789 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1cc009e210 0x7f1cc01cab50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1cb80098c0 tx=0x7f1cb80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:47.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.237+0000 7f1cbe7fc640 1 -- 172.21.15.40:0/3940460789 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb8032be0 con 0x7f1cc009e210 2024-04-22T08:22:47.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.237+0000 7f1cbe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:47.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.237+0000 7f1cbe7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:22:47.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.237+0000 7f1cbe7fc640 10 monclient: dump: 2024-04-22T08:22:47.249 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:47.249 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:47.249 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:47.249 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:47.249 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:47.249 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:47.249 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:47.249 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:47.249 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:47.249 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:47.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.237+0000 7f1cbe7fc640 10 monclient: _finish_auth 0 2024-04-22T08:22:47.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.237+0000 7f1cbe7fc640 10 monclient: _check_auth_tickets 2024-04-22T08:22:47.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.237+0000 7f1cbe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:47.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.237+0000 7f1cbe7fc640 1 -- 172.21.15.40:0/3940460789 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cb8032d40 con 0x7f1cc009e210 2024-04-22T08:22:47.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.237+0000 7f1cc8155640 5 monclient: authenticate success, global_id 124187 2024-04-22T08:22:47.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.237+0000 7f1cc8155640 10 monclient: _renew_subs 2024-04-22T08:22:47.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.237+0000 7f1cc8155640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:22:47.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.237+0000 7f1cc8155640 1 -- 172.21.15.40:0/3940460789 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cc0121a80 con 0x7f1cc009e210 2024-04-22T08:22:47.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.237+0000 7f1cbe7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:47.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.237+0000 7f1cbe7fc640 1 -- 172.21.15.40:0/3940460789 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb803d6a0 con 0x7f1cc009e210 2024-04-22T08:22:47.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.237+0000 7f1cbe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:47.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.237+0000 7f1cbe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:22:47.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.237+0000 7f1cbe7fc640 10 monclient: dump: 2024-04-22T08:22:47.250 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:47.251 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:47.251 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:47.251 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:47.251 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:47.251 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:47.251 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:47.251 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:47.251 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:47.251 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:47.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.237+0000 7f1cbffff640 4 set_mon_vals no callback set 2024-04-22T08:22:47.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.237+0000 7f1cc8155640 10 monclient: _renew_subs 2024-04-22T08:22:47.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.237+0000 7f1cc8155640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:22:47.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.237+0000 7f1cc8155640 1 -- 172.21.15.40:0/3940460789 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cc012f550 con 0x7f1cc009e210 2024-04-22T08:22:47.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.238+0000 7f1cbe7fc640 1 -- 172.21.15.40:0/3940460789 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f1cb8039040 con 0x7f1cc009e210 2024-04-22T08:22:47.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.238+0000 7f1cc8155640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:47.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.238+0000 7f1cc8155640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:47.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.238+0000 7f1cc8155640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:22:47.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.238+0000 7f1cc8155640 1 -- 172.21.15.40:0/3940460789 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cc004e9b0 con 0x7f1cc009e210 2024-04-22T08:22:47.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.238+0000 7f1cbe7fc640 1 --2- 172.21.15.40:0/3940460789 >> v2:172.21.15.189:6800/30168 conn(0x7f1cac0377d0 0x7f1cac039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:47.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.238+0000 7f1cbe7fc640 1 -- 172.21.15.40:0/3940460789 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(317..317 src has 239..317) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f1cb806c620 con 0x7f1cc009e210 2024-04-22T08:22:47.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.239+0000 7f1cc56f0640 1 --2- 172.21.15.40:0/3940460789 >> v2:172.21.15.189:6800/30168 conn(0x7f1cac0377d0 0x7f1cac039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:47.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.239+0000 7f1cc56f0640 10 monclient: get_auth_request con 0x7f1cac0377d0 auth_method 0 2024-04-22T08:22:47.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.241+0000 7f1cc56f0640 1 --2- 172.21.15.40:0/3940460789 >> v2:172.21.15.189:6800/30168 conn(0x7f1cac0377d0 0x7f1cac039c90 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f1cb4007c90 tx=0x7f1cb4007be0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:47.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.243+0000 7f1cbe7fc640 1 -- 172.21.15.40:0/3940460789 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f1cb803d800 con 0x7f1cc009e210 2024-04-22T08:22:47.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.243+0000 7f1cbe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:47.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.243+0000 7f1cbe7fc640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:22:47.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.422+0000 7f1cc8155640 1 -- 172.21.15.40:0/3940460789 --> v2:172.21.15.189:6800/30168 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f1cc0122c60 con 0x7f1cac0377d0 2024-04-22T08:22:47.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.423+0000 7f1cbe7fc640 1 -- 172.21.15.40:0/3940460789 <== mgr.4102 v2:172.21.15.189:6800/30168 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f1cc0122c60 con 0x7f1cac0377d0 2024-04-22T08:22:47.422 INFO:teuthology.orchestra.run.smithi040.stderr:instructed osd(s) 5 to deep-scrub 2024-04-22T08:22:47.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.425+0000 7f1cc8155640 1 -- 172.21.15.40:0/3940460789 >> v2:172.21.15.189:6800/30168 conn(0x7f1cac0377d0 msgr2=0x7f1cac039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:47.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.425+0000 7f1cc8155640 1 --2- 172.21.15.40:0/3940460789 >> v2:172.21.15.189:6800/30168 conn(0x7f1cac0377d0 0x7f1cac039c90 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f1cb4007c90 tx=0x7f1cb4007be0 comp rx=0 tx=0).stop 2024-04-22T08:22:47.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.425+0000 7f1cc8155640 10 monclient: shutdown 2024-04-22T08:22:47.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.425+0000 7f1cc8155640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:47.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.425+0000 7f1cc8155640 1 -- 172.21.15.40:0/3940460789 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1cc009e210 msgr2=0x7f1cc01cab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:47.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.425+0000 7f1cc8155640 1 --2- 172.21.15.40:0/3940460789 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1cc009e210 0x7f1cc01cab50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1cb80098c0 tx=0x7f1cb80046c0 comp rx=0 tx=0).stop 2024-04-22T08:22:47.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.425+0000 7f1cc8155640 1 -- 172.21.15.40:0/3940460789 shutdown_connections 2024-04-22T08:22:47.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.425+0000 7f1cc8155640 1 --2- 172.21.15.40:0/3940460789 >> v2:172.21.15.189:6800/30168 conn(0x7f1cac0377d0 0x7f1cac039c90 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:47.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.425+0000 7f1cc8155640 1 --2- 172.21.15.40:0/3940460789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1cc012fc40 0x7f1cc012ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:47.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.425+0000 7f1cc8155640 1 --2- 172.21.15.40:0/3940460789 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1cc01330a0 0x7f1cc012f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:47.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.425+0000 7f1cc8155640 1 --2- 172.21.15.40:0/3940460789 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1cc009e210 0x7f1cc01cab50 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:47.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.425+0000 7f1cc8155640 1 -- 172.21.15.40:0/3940460789 >> 172.21.15.40:0/3940460789 conn(0x7f1cc00a0b50 msgr2=0x7f1cc009f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:47.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.425+0000 7f1cc8155640 1 -- 172.21.15.40:0/3940460789 shutdown_connections 2024-04-22T08:22:47.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.425+0000 7f1cc8155640 1 -- 172.21.15.40:0/3940460789 wait complete. 2024-04-22T08:22:47.435 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:22:47.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.533+0000 7f4f0acab640 10 monclient: get_monmap_and_config 2024-04-22T08:22:47.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.533+0000 7f4f0acab640 10 monclient: build_initial_monmap 2024-04-22T08:22:47.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.533+0000 7f4f0acab640 10 monclient: monmap: 2024-04-22T08:22:47.533 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:47.533 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:47.534 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:22:47.534649+0000 2024-04-22T08:22:47.534 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:22:47.534649+0000 2024-04-22T08:22:47.534 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:47.534 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:47.534 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:22:47.534 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:22:47.534 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:22:47.534 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:22:47.534 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:22:47.534 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:47.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.534+0000 7f4f0acab640 1 Processor -- start 2024-04-22T08:22:47.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.534+0000 7f4f0acab640 1 -- start start 2024-04-22T08:22:47.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.534+0000 7f4f0acab640 10 monclient: init 2024-04-22T08:22:47.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.535+0000 7f4f0acab640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:47.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.535+0000 7f4f0acab640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-04-22T08:22:47.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.535+0000 7f4f0acab640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4f0409f300 addr v1:172.21.15.40:6789/0 2024-04-22T08:22:47.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.535+0000 7f4f0acab640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4f04136f30 addr v1:172.21.15.189:6789/0 2024-04-22T08:22:47.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.535+0000 7f4f0acab640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4f0413aa40 0x7f4f0413ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:47.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.535+0000 7f4f0acab640 10 monclient(hunting): picked mon.noname-c con 0x7f4f0413aa40 addr v2:172.21.15.189:3300/0 2024-04-22T08:22:47.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.535+0000 7f4f0acab640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f04071400 con 0x7f4f0409f300 2024-04-22T08:22:47.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.535+0000 7f4f0acab640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f040a0180 con 0x7f4f04136f30 2024-04-22T08:22:47.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.535+0000 7f4f0acab640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:47.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.535+0000 7f4f0acab640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f0409e0f0 con 0x7f4f0413aa40 2024-04-22T08:22:47.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.535+0000 7f4f0acab640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:47.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.535+0000 7f4f0acab640 10 monclient(hunting): authenticate will time out at 1441.529020s 2024-04-22T08:22:47.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.535+0000 7f4f08a47640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f4f0409f300 0x7f4f0409f6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:49390/0 (socket says 172.21.15.40:49390) 2024-04-22T08:22:47.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.535+0000 7f4f08a47640 1 -- 172.21.15.40:0/1509998488 learned_addr learned my addr 172.21.15.40:0/1509998488 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-04-22T08:22:47.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.535+0000 7f4f09248640 1 --2- 172.21.15.40:0/1509998488 >> v2:172.21.15.189:3300/0 conn(0x7f4f0413aa40 0x7f4f0413ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:47.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.535+0000 7f4f09248640 10 monclient(hunting): get_auth_request con 0x7f4f0413aa40 auth_method 0 2024-04-22T08:22:47.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.535+0000 7f4f09248640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:47.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.535+0000 7f4f09248640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:47.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.535+0000 7f4f09248640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:47.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.535+0000 7f4f09248640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:47.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.535+0000 7f4f09248640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:47.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.535+0000 7f4f09248640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:47.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.535+0000 7f4f09248640 10 monclient(hunting): handle_auth_done global_id 65314 payload 274 2024-04-22T08:22:47.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.535+0000 7f4f09248640 1 -- 172.21.15.40:0/1509998488 >> v1:172.21.15.189:6789/0 conn(0x7f4f04136f30 legacy=0x7f4f04139320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:47.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.535+0000 7f4f09248640 1 -- 172.21.15.40:0/1509998488 >> v1:172.21.15.40:6789/0 conn(0x7f4f0409f300 legacy=0x7f4f0409f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:22:47.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.535+0000 7f4f09248640 10 monclient: _finish_hunting 0 2024-04-22T08:22:47.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.535+0000 7f4f09248640 1 monclient: found mon.noname-c 2024-04-22T08:22:47.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.535+0000 7f4f09248640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:47.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.535+0000 7f4f09248640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.189:3300/0 2024-04-22T08:22:47.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.535+0000 7f4f09248640 1 -- 172.21.15.40:0/1509998488 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f040a0460 con 0x7f4f0413aa40 2024-04-22T08:22:47.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.538+0000 7f4f09248640 1 -- 172.21.15.40:0/1509998488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f0413aa40 msgr2=0x7f4f0413ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:47.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.538+0000 7f4f09248640 1 --2- 172.21.15.40:0/1509998488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f0413aa40 0x7f4f0413ce30 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f4efc009d90 tx=0x7f4efc02bd90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b817d4216874888 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:47.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.539+0000 7f4f037fe640 1 -- 172.21.15.40:0/1509998488 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4efc02ce30 con 0x7f4f0413aa40 2024-04-22T08:22:47.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.539+0000 7f4f037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:47.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.539+0000 7f4f037fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:22:47.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.539+0000 7f4f037fe640 10 monclient: dump: 2024-04-22T08:22:47.546 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:47.546 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:47.546 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:47.546 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:47.546 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:47.546 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:47.546 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:47.546 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:47.546 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:47.546 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:47.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.539+0000 7f4f037fe640 10 monclient: _finish_auth 0 2024-04-22T08:22:47.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.539+0000 7f4f037fe640 10 monclient: _check_auth_tickets 2024-04-22T08:22:47.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.539+0000 7f4f037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:47.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.539+0000 7f4f037fe640 1 -- 172.21.15.40:0/1509998488 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4efc034440 con 0x7f4f0413aa40 2024-04-22T08:22:47.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.539+0000 7f4f037fe640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:47.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.539+0000 7f4f037fe640 1 -- 172.21.15.40:0/1509998488 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4efc0347d0 con 0x7f4f0413aa40 2024-04-22T08:22:47.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.539+0000 7f4f0acab640 5 monclient: authenticate success, global_id 65314 2024-04-22T08:22:47.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.539+0000 7f4f0acab640 10 monclient: get_monmap_and_config success 2024-04-22T08:22:47.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.539+0000 7f4f0acab640 4 set_mon_vals no callback set 2024-04-22T08:22:47.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.539+0000 7f4f037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:47.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.539+0000 7f4f037fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-22T08:22:47.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.539+0000 7f4f037fe640 10 monclient: dump: 2024-04-22T08:22:47.548 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:47.548 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:47.548 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:47.548 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:47.548 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:47.548 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:47.548 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:47.548 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:47.548 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:47.548 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:47.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.539+0000 7f4f0acab640 10 monclient: shutdown 2024-04-22T08:22:47.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.539+0000 7f4f0acab640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:47.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.539+0000 7f4f0acab640 1 -- 172.21.15.40:0/1509998488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f0413aa40 msgr2=0x7f4f0413ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:47.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.539+0000 7f4f0acab640 1 --2- 172.21.15.40:0/1509998488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f0413aa40 0x7f4f0413ce30 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f4efc009d90 tx=0x7f4efc02bd90 comp rx=0 tx=0).stop 2024-04-22T08:22:47.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.539+0000 7f4f0acab640 1 -- 172.21.15.40:0/1509998488 shutdown_connections 2024-04-22T08:22:47.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.539+0000 7f4f0acab640 1 --2- 172.21.15.40:0/1509998488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f0413aa40 0x7f4f0413ce30 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:47.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.539+0000 7f4f0acab640 1 -- 172.21.15.40:0/1509998488 >> 172.21.15.40:0/1509998488 conn(0x7f4f040a0d30 msgr2=0x7f4f040a1140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:47.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.539+0000 7f4f0acab640 1 -- 172.21.15.40:0/1509998488 shutdown_connections 2024-04-22T08:22:47.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.539+0000 7f4f0acab640 1 -- 172.21.15.40:0/1509998488 wait complete. 2024-04-22T08:22:47.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.540+0000 7f4f0acab640 10 monclient: build_initial_monmap 2024-04-22T08:22:47.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.540+0000 7f4f0acab640 10 monclient: monmap: 2024-04-22T08:22:47.549 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:22:47.550 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:22:47.550 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:22:47.550 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:22:47.550 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:22:47.550 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:47.550 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:22:47.550 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:22:47.550 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:22:47.550 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:47.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.540+0000 7f4f0acab640 1 Processor -- start 2024-04-22T08:22:47.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.540+0000 7f4f0acab640 1 -- start start 2024-04-22T08:22:47.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.540+0000 7f4f0acab640 10 monclient: init 2024-04-22T08:22:47.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f0acab640 10 monclient: _reopen_session rank -1 2024-04-22T08:22:47.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f0acab640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-22T08:22:47.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f0acab640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4f0409f300 0x7f4f041c9220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:47.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f0acab640 10 monclient(hunting): picked mon.noname-c con 0x7f4f0409f300 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:22:47.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f0acab640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f04136f30 0x7f4f0406ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:47.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f0acab640 10 monclient(hunting): picked mon.noname-b con 0x7f4f04136f30 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:22:47.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f0acab640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f0413aa40 0x7f4f0406ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:47.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f0acab640 10 monclient(hunting): picked mon.noname-a con 0x7f4f0413aa40 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:22:47.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f0acab640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:47.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f0acab640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0411c0e0 con 0x7f4f0413aa40 2024-04-22T08:22:47.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f0acab640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:47.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f0acab640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f040715f0 con 0x7f4f04136f30 2024-04-22T08:22:47.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f0acab640 10 monclient(hunting): start opening mon connection 2024-04-22T08:22:47.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f0acab640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0411be70 con 0x7f4f0409f300 2024-04-22T08:22:47.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f0acab640 10 monclient(hunting): _renew_subs 2024-04-22T08:22:47.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f0acab640 10 monclient(hunting): authenticate will time out at 1441.535290s 2024-04-22T08:22:47.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f08a47640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4f0409f300 0x7f4f041c9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:47.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f09248640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f0413aa40 0x7f4f0406ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:47.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f08a47640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4f0409f300 0x7f4f041c9220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:50008/0 (socket says 172.21.15.40:50008) 2024-04-22T08:22:47.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f09248640 10 monclient(hunting): get_auth_request con 0x7f4f0413aa40 auth_method 0 2024-04-22T08:22:47.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f08a47640 1 -- 172.21.15.40:0/3709980887 learned_addr learned my addr 172.21.15.40:0/3709980887 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:22:47.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f09248640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:47.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f09248640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:47.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f09248640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:47.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f08a47640 10 monclient(hunting): get_auth_request con 0x7f4f0409f300 auth_method 0 2024-04-22T08:22:47.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f08a47640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:47.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f08a47640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:47.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f08a47640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:47.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f09248640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:47.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f09248640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:47.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f09248640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:47.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f08a47640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:22:47.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f08a47640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:22:47.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f08a47640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:22:47.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f03fff640 1 --2- 172.21.15.40:0/3709980887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f04136f30 0x7f4f0406ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:47.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f03fff640 10 monclient(hunting): get_auth_request con 0x7f4f04136f30 auth_method 0 2024-04-22T08:22:47.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:22:47.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f03fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:22:47.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f03fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:22:47.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f08a47640 10 monclient(hunting): handle_auth_done global_id 124190 payload 824 2024-04-22T08:22:47.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f08a47640 1 -- 172.21.15.40:0/3709980887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f04136f30 msgr2=0x7f4f0406ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:47.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f08a47640 1 --2- 172.21.15.40:0/3709980887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f04136f30 0x7f4f0406ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:47.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f08a47640 1 -- 172.21.15.40:0/3709980887 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f0413aa40 msgr2=0x7f4f0406ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:47.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f08a47640 1 --2- 172.21.15.40:0/3709980887 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f0413aa40 0x7f4f0406ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:47.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f08a47640 10 monclient: _finish_hunting 0 2024-04-22T08:22:47.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f08a47640 1 monclient: found mon.noname-c 2024-04-22T08:22:47.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f08a47640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:22:47.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f08a47640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:22:47.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f08a47640 1 -- 172.21.15.40:0/3709980887 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4efc009a70 con 0x7f4f0409f300 2024-04-22T08:22:47.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.541+0000 7f4f09248640 1 --2- 172.21.15.40:0/3709980887 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f0413aa40 0x7f4f0406ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:22:47.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.544+0000 7f4f08a47640 1 -- 172.21.15.40:0/3709980887 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4f0409f300 msgr2=0x7f4f041c9220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:22:47.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.544+0000 7f4f08a47640 1 --2- 172.21.15.40:0/3709980887 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4f0409f300 0x7f4f041c9220 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4ef400b460 tx=0x7f4ef400b920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:47.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.544+0000 7f4f017fa640 1 -- 172.21.15.40:0/3709980887 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef4011aa0 con 0x7f4f0409f300 2024-04-22T08:22:47.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.544+0000 7f4f017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:47.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.544+0000 7f4f017fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:22:47.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.544+0000 7f4f017fa640 10 monclient: dump: 2024-04-22T08:22:47.555 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:47.555 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:47.555 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:47.555 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:47.555 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:47.555 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:47.555 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:47.555 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:47.555 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:47.555 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:47.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.544+0000 7f4f017fa640 10 monclient: _finish_auth 0 2024-04-22T08:22:47.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.544+0000 7f4f017fa640 10 monclient: _check_auth_tickets 2024-04-22T08:22:47.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.544+0000 7f4f017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:22:47.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.544+0000 7f4f017fa640 1 -- 172.21.15.40:0/3709980887 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ef4011c20 con 0x7f4f0409f300 2024-04-22T08:22:47.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.544+0000 7f4f017fa640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:22:47.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.544+0000 7f4f017fa640 1 -- 172.21.15.40:0/3709980887 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef40103e0 con 0x7f4f0409f300 2024-04-22T08:22:47.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.544+0000 7f4f017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:22:47.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.544+0000 7f4f017fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:22:47.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.544+0000 7f4f017fa640 10 monclient: dump: 2024-04-22T08:22:47.556 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:22:47.556 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:22:47.556 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:47.556 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:22:47.556 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:22:47.556 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:22:47.557 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:22:47.557 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:22:47.557 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:22:47.557 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:22:47.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.544+0000 7f4f0acab640 5 monclient: authenticate success, global_id 124190 2024-04-22T08:22:47.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.544+0000 7f4f0acab640 10 monclient: _renew_subs 2024-04-22T08:22:47.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.544+0000 7f4f0acab640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:22:47.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.544+0000 7f4f02ffd640 4 set_mon_vals no callback set 2024-04-22T08:22:47.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.544+0000 7f4f0acab640 1 -- 172.21.15.40:0/3709980887 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f04135010 con 0x7f4f0409f300 2024-04-22T08:22:47.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.544+0000 7f4f0acab640 10 monclient: _renew_subs 2024-04-22T08:22:47.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.544+0000 7f4f0acab640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:22:47.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.544+0000 7f4f0acab640 1 -- 172.21.15.40:0/3709980887 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f04070460 con 0x7f4f0409f300 2024-04-22T08:22:47.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.545+0000 7f4f017fa640 1 -- 172.21.15.40:0/3709980887 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f4ef4010580 con 0x7f4f0409f300 2024-04-22T08:22:47.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.545+0000 7f4f0acab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:47.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.545+0000 7f4f0acab640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:47.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.545+0000 7f4f0acab640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:22:47.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.545+0000 7f4f017fa640 1 --2- 172.21.15.40:0/3709980887 >> v2:172.21.15.189:6800/30168 conn(0x7f4ef00377d0 0x7f4ef0039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:22:47.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.545+0000 7f4f0acab640 1 -- 172.21.15.40:0/3709980887 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f0409ef10 con 0x7f4f0409f300 2024-04-22T08:22:47.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.545+0000 7f4f017fa640 1 -- 172.21.15.40:0/3709980887 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(317..317 src has 239..317) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f4ef404a5d0 con 0x7f4f0409f300 2024-04-22T08:22:47.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.545+0000 7f4f03fff640 1 --2- 172.21.15.40:0/3709980887 >> v2:172.21.15.189:6800/30168 conn(0x7f4ef00377d0 0x7f4ef0039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:47.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.545+0000 7f4f03fff640 10 monclient: get_auth_request con 0x7f4ef00377d0 auth_method 0 2024-04-22T08:22:47.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.545+0000 7f4f03fff640 1 -- 172.21.15.40:0/3709980887 >> v2:172.21.15.189:6800/30168 conn(0x7f4ef00377d0 msgr2=0x7f4ef0039c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-22T08:22:47.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.545+0000 7f4f03fff640 1 -- 172.21.15.40:0/3709980887 >> v2:172.21.15.189:6800/30168 conn(0x7f4ef00377d0 msgr2=0x7f4ef0039c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T08:22:47.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.545+0000 7f4f03fff640 1 --2- 172.21.15.40:0/3709980887 >> v2:172.21.15.189:6800/30168 conn(0x7f4ef00377d0 0x7f4ef0039c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T08:22:47.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.545+0000 7f4f03fff640 1 --2- 172.21.15.40:0/3709980887 >> v2:172.21.15.189:6800/30168 conn(0x7f4ef00377d0 0x7f4ef0039c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T08:22:47.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.549+0000 7f4f017fa640 1 -- 172.21.15.40:0/3709980887 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f4f0409ef10 con 0x7f4f0409f300 2024-04-22T08:22:47.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.550+0000 7f4f017fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:22:47.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.550+0000 7f4f017fa640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:22:47.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.724+0000 7f4f0acab640 1 -- 172.21.15.40:0/3709980887 --> v2:172.21.15.189:6800/30168 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4f0409ef10 con 0x7f4ef00377d0 2024-04-22T08:22:47.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.746+0000 7f4f03fff640 1 --2- 172.21.15.40:0/3709980887 >> v2:172.21.15.189:6800/30168 conn(0x7f4ef00377d0 0x7f4ef0039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:22:47.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.746+0000 7f4f03fff640 10 monclient: get_auth_request con 0x7f4ef00377d0 auth_method 0 2024-04-22T08:22:47.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.749+0000 7f4f03fff640 1 --2- 172.21.15.40:0/3709980887 >> v2:172.21.15.189:6800/30168 conn(0x7f4ef00377d0 0x7f4ef0039c90 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f4f040717d0 tx=0x7f4ef8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:22:47.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.751+0000 7f4f017fa640 1 -- 172.21.15.40:0/3709980887 <== mgr.4102 v2:172.21.15.189:6800/30168 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+53878 (secure 0 0 0) 0x7f4f0409ef10 con 0x7f4ef00377d0 2024-04-22T08:22:47.751 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:22:47.751 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-22T08:22:47.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.753+0000 7f4f0acab640 1 -- 172.21.15.40:0/3709980887 >> v2:172.21.15.189:6800/30168 conn(0x7f4ef00377d0 msgr2=0x7f4ef0039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:47.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.753+0000 7f4f0acab640 1 --2- 172.21.15.40:0/3709980887 >> v2:172.21.15.189:6800/30168 conn(0x7f4ef00377d0 0x7f4ef0039c90 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f4f040717d0 tx=0x7f4ef8009040 comp rx=0 tx=0).stop 2024-04-22T08:22:47.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.753+0000 7f4f0acab640 10 monclient: shutdown 2024-04-22T08:22:47.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.753+0000 7f4f0acab640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:22:47.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.753+0000 7f4f0acab640 1 -- 172.21.15.40:0/3709980887 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4f0409f300 msgr2=0x7f4f041c9220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:22:47.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.753+0000 7f4f0acab640 1 --2- 172.21.15.40:0/3709980887 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4f0409f300 0x7f4f041c9220 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4ef400b460 tx=0x7f4ef400b920 comp rx=0 tx=0).stop 2024-04-22T08:22:47.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.753+0000 7f4f0acab640 1 -- 172.21.15.40:0/3709980887 shutdown_connections 2024-04-22T08:22:47.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.753+0000 7f4f0acab640 1 --2- 172.21.15.40:0/3709980887 >> v2:172.21.15.189:6800/30168 conn(0x7f4ef00377d0 0x7f4ef0039c90 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:47.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.753+0000 7f4f0acab640 1 --2- 172.21.15.40:0/3709980887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f04136f30 0x7f4f0406ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:47.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.753+0000 7f4f0acab640 1 --2- 172.21.15.40:0/3709980887 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f0413aa40 0x7f4f0406ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:47.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.753+0000 7f4f0acab640 1 --2- 172.21.15.40:0/3709980887 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4f0409f300 0x7f4f041c9220 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:22:47.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.753+0000 7f4f0acab640 1 -- 172.21.15.40:0/3709980887 >> 172.21.15.40:0/3709980887 conn(0x7f4f040a0d30 msgr2=0x7f4f04133b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:22:47.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.753+0000 7f4f0acab640 1 -- 172.21.15.40:0/3709980887 shutdown_connections 2024-04-22T08:22:47.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:22:47.753+0000 7f4f0acab640 1 -- 172.21.15.40:0/3709980887 wait complete. 2024-04-22T08:22:47.763 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":614,"stamp":"2024-04-22T08:22:46.206891+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":185760,"kb_used_data":7232,"kb_used_omap":9,"kb_used_meta":178422,"kb_avail":566045280,"statfs":{"total":579820584960,"available":579630366720,"internally_reserved":0,"allocated":7405568,"data_stored":3236636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9560,"internal_metadata":182704808},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002582"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":304,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-04-22T08:22:17.159438+0000","last_change":"2024-04-22T08:12:51.443086+0000","last_active":"2024-04-22T08:22:17.159438+0000","last_peered":"2024-04-22T08:22:17.159438+0000","last_clean":"2024-04-22T08:22:17.159438+0000","last_became_active":"2024-04-22T08:12:51.442730+0000","last_became_peered":"2024-04-22T08:12:51.442730+0000","last_unstale":"2024-04-22T08:22:17.159438+0000","last_undegraded":"2024-04-22T08:22:17.159438+0000","last_fullsized":"2024-04-22T08:22:17.159438+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:06:13.481863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":293,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-04-22T08:22:16.957788+0000","last_change":"2024-04-22T08:12:50.841276+0000","last_active":"2024-04-22T08:22:16.957788+0000","last_peered":"2024-04-22T08:22:16.957788+0000","last_clean":"2024-04-22T08:22:16.957788+0000","last_became_active":"2024-04-22T08:12:50.841008+0000","last_became_peered":"2024-04-22T08:12:50.841008+0000","last_unstale":"2024-04-22T08:22:16.957788+0000","last_undegraded":"2024-04-22T08:22:16.957788+0000","last_fullsized":"2024-04-22T08:22:16.957788+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:47:00.224321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":304,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-04-22T08:22:17.160920+0000","last_change":"2024-04-22T08:12:51.403419+0000","last_active":"2024-04-22T08:22:17.160920+0000","last_peered":"2024-04-22T08:22:17.160920+0000","last_clean":"2024-04-22T08:22:17.160920+0000","last_became_active":"2024-04-22T08:12:51.403182+0000","last_became_peered":"2024-04-22T08:12:51.403182+0000","last_unstale":"2024-04-22T08:22:17.160920+0000","last_undegraded":"2024-04-22T08:22:17.160920+0000","last_fullsized":"2024-04-22T08:22:17.160920+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:52:39.730597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":293,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-04-22T08:22:16.957078+0000","last_change":"2024-04-22T08:12:51.403260+0000","last_active":"2024-04-22T08:22:16.957078+0000","last_peered":"2024-04-22T08:22:16.957078+0000","last_clean":"2024-04-22T08:22:16.957078+0000","last_became_active":"2024-04-22T08:12:51.402980+0000","last_became_peered":"2024-04-22T08:12:51.402980+0000","last_unstale":"2024-04-22T08:22:16.957078+0000","last_undegraded":"2024-04-22T08:22:16.957078+0000","last_fullsized":"2024-04-22T08:22:16.957078+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:08:04.625870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":295,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-04-22T08:22:17.162182+0000","last_change":"2024-04-22T08:12:50.836566+0000","last_active":"2024-04-22T08:22:17.162182+0000","last_peered":"2024-04-22T08:22:17.162182+0000","last_clean":"2024-04-22T08:22:17.162182+0000","last_became_active":"2024-04-22T08:12:50.836250+0000","last_became_peered":"2024-04-22T08:12:50.836250+0000","last_unstale":"2024-04-22T08:22:17.162182+0000","last_undegraded":"2024-04-22T08:22:17.162182+0000","last_fullsized":"2024-04-22T08:22:17.162182+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:42:13.693209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":302,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-04-22T08:22:16.950081+0000","last_change":"2024-04-22T08:12:50.836508+0000","last_active":"2024-04-22T08:22:16.950081+0000","last_peered":"2024-04-22T08:22:16.950081+0000","last_clean":"2024-04-22T08:22:16.950081+0000","last_became_active":"2024-04-22T08:12:50.836210+0000","last_became_peered":"2024-04-22T08:12:50.836210+0000","last_unstale":"2024-04-22T08:22:16.950081+0000","last_undegraded":"2024-04-22T08:22:16.950081+0000","last_fullsized":"2024-04-22T08:22:16.950081+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:24:53.345661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":304,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-04-22T08:22:17.158781+0000","last_change":"2024-04-22T08:12:50.842633+0000","last_active":"2024-04-22T08:22:17.158781+0000","last_peered":"2024-04-22T08:22:17.158781+0000","last_clean":"2024-04-22T08:22:17.158781+0000","last_became_active":"2024-04-22T08:12:50.842392+0000","last_became_peered":"2024-04-22T08:12:50.842392+0000","last_unstale":"2024-04-22T08:22:17.158781+0000","last_undegraded":"2024-04-22T08:22:17.158781+0000","last_fullsized":"2024-04-22T08:22:17.158781+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:53:14.717539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":295,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-04-22T08:22:17.162502+0000","last_change":"2024-04-22T08:12:50.835820+0000","last_active":"2024-04-22T08:22:17.162502+0000","last_peered":"2024-04-22T08:22:17.162502+0000","last_clean":"2024-04-22T08:22:17.162502+0000","last_became_active":"2024-04-22T08:12:50.835541+0000","last_became_peered":"2024-04-22T08:12:50.835541+0000","last_unstale":"2024-04-22T08:22:17.162502+0000","last_undegraded":"2024-04-22T08:22:17.162502+0000","last_fullsized":"2024-04-22T08:22:17.162502+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:52:21.798509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":454,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-04-22T08:22:17.158877+0000","last_change":"2024-04-22T08:12:51.404305+0000","last_active":"2024-04-22T08:22:17.158877+0000","last_peered":"2024-04-22T08:22:17.158877+0000","last_clean":"2024-04-22T08:22:17.158877+0000","last_became_active":"2024-04-22T08:12:51.403208+0000","last_became_peered":"2024-04-22T08:12:51.403208+0000","last_unstale":"2024-04-22T08:22:17.158877+0000","last_undegraded":"2024-04-22T08:22:17.158877+0000","last_fullsized":"2024-04-22T08:22:17.158877+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:12:49.822456+0000","last_clean_scrub_stamp":"2024-04-22T08:12:49.822456+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T20:04:43.253802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640380,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28056,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343784,"statfs":{"total":96636764160,"available":96608034816,"internally_reserved":0,"allocated":1056768,"data_stored":363130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.35199999999999998,"15min":0.371},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":0.39800000000000002,"5min":2.5920000000000001,"15min":2.5920000000000001},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.40899999999999997,"15min":0.38700000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.21299999999999999},"max":{"1min":0.51400000000000001,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.32000000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.34300000000000003,"15min":0.34399999999999997},"min":{"1min":0.249,"5min":0.23400000000000001,"15min":0.17100000000000001},"max":{"1min":0.443,"5min":0.499,"15min":0.55700000000000005},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.38100000000000001,"15min":0.34499999999999997},"min":{"1min":0.254,"5min":0.24299999999999999,"15min":0.20699999999999999},"max":{"1min":0.48499999999999999,"5min":0.55600000000000005,"15min":0.70099999999999996},"last":0.30199999999999999}]},{"osd":3,"last update":"Mon Apr 22 08:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.59699999999999998,"15min":0.56499999999999995},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.22500000000000001},"max":{"1min":0.82999999999999996,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.58799999999999997,"15min":0.60199999999999998},"min":{"1min":0.33400000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.749,"5min":0.84799999999999998,"15min":1.8440000000000001},"last":0.46300000000000002}]},{"osd":4,"last update":"Mon Apr 22 08:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58399999999999996,"15min":0.68600000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.30499999999999999},"max":{"1min":0.78800000000000003,"5min":2.7879999999999998,"15min":6.2949999999999999},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.55100000000000005,"15min":0.68400000000000005},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.69499999999999995,"5min":0.91200000000000003,"15min":6.2149999999999999},"last":0.56299999999999994}]},{"osd":5,"last update":"Mon Apr 22 08:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.56899999999999995,"15min":0.53400000000000003},"min":{"1min":0.40000000000000002,"5min":0.35799999999999998,"15min":0.27800000000000002},"max":{"1min":0.72499999999999998,"5min":0.84299999999999997,"15min":0.85199999999999998},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56399999999999995,"15min":0.53500000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.29699999999999999},"max":{"1min":0.72999999999999998,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.47499999999999998}]}]},{"osd":0,"up_from":11,"seq":47244640379,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28580,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343260,"statfs":{"total":96636764160,"available":96607498240,"internally_reserved":0,"allocated":1589248,"data_stored":892058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.32000000000000001,"15min":0.34599999999999997},"min":{"1min":0.23699999999999999,"5min":0.189,"15min":0.18099999999999999},"max":{"1min":0.46500000000000002,"5min":0.47599999999999998,"15min":1.7190000000000001},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.32600000000000001,"15min":0.33000000000000002},"min":{"1min":0.22,"5min":0.21199999999999999,"15min":0.19800000000000001},"max":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.56999999999999995},"last":0.251}]},{"osd":2,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.373,"15min":0.36099999999999999},"min":{"1min":0.29099999999999998,"5min":0.26300000000000001,"15min":0.222},"max":{"1min":0.499,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.38,"15min":0.39200000000000002},"min":{"1min":0.24299999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.51400000000000001,"5min":0.60399999999999998,"15min":0.626},"last":0.22800000000000001}]},{"osd":3,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55300000000000005,"15min":0.54900000000000004},"min":{"1min":0.37,"5min":0.35699999999999998,"15min":0.32200000000000001},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.432},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.54500000000000004,"15min":0.53300000000000003},"min":{"1min":0.42999999999999999,"5min":0.32300000000000001,"15min":0.29499999999999998},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.92000000000000004},"last":0.52400000000000002}]},{"osd":4,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.55000000000000004,"15min":0.55100000000000005},"min":{"1min":0.42599999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.72099999999999997,"5min":0.81799999999999995,"15min":0.85999999999999999},"last":0.497},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.54700000000000004,"15min":0.54200000000000004},"min":{"1min":0.39700000000000002,"5min":0.36299999999999999,"15min":0.33900000000000002},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.55300000000000005}]},{"osd":5,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55000000000000004,"15min":0.54600000000000004},"min":{"1min":0.41399999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.71999999999999997,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54900000000000004,"15min":0.55500000000000005},"min":{"1min":0.38500000000000001,"5min":0.36799999999999999,"15min":0.35899999999999999},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.57399999999999995}]}]},{"osd":5,"up_from":11,"seq":47244640380,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44440,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":43390,"kb_avail":94327400,"statfs":{"total":96636764160,"available":96591257600,"internally_reserved":0,"allocated":1056768,"data_stored":363130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":44431814},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:22:04 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.49099999999999999,"15min":0.49199999999999999},"min":{"1min":0.314,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.60699999999999998,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.622},{"interface":"front","average":{"1min":0.496,"5min":0.49399999999999999,"15min":0.51700000000000002},"min":{"1min":0.32700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":1.484},"last":0.60299999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:22:04 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.49399999999999999,"15min":0.504},"min":{"1min":0.36099999999999999,"5min":0.247,"15min":0.247},"max":{"1min":0.64200000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52600000000000002,"15min":0.52700000000000002},"min":{"1min":0.36899999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.65500000000000003,"5min":0.75700000000000001,"15min":0.81499999999999995},"last":0.65100000000000002}]},{"osd":2,"last update":"Mon Apr 22 08:22:04 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.5,"15min":0.51300000000000001},"min":{"1min":0.40500000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.67500000000000004,"5min":0.68600000000000005,"15min":0.76800000000000002},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.499,"5min":0.504,"15min":0.58099999999999996},"min":{"1min":0.38400000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.57599999999999996,"5min":0.74099999999999999,"15min":3.6970000000000001},"last":0.63400000000000001}]},{"osd":3,"last update":"Mon Apr 22 08:22:04 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.39900000000000002,"15min":0.39400000000000002},"min":{"1min":0.28699999999999998,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.52000000000000002,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41599999999999998,"15min":0.40899999999999997},"min":{"1min":0.29299999999999998,"5min":0.25900000000000001,"15min":0.255},"max":{"1min":0.60099999999999998,"5min":0.62,"15min":0.621},"last":0.49199999999999999}]},{"osd":4,"last update":"Mon Apr 22 08:22:04 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.38100000000000001,"15min":0.41699999999999998},"min":{"1min":0.252,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.47199999999999998,"5min":1.0820000000000001,"15min":1.528},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.379,"5min":0.40400000000000003,"15min":0.42999999999999999},"min":{"1min":0.27800000000000002,"5min":0.24099999999999999,"15min":0.23599999999999999},"max":{"1min":0.61299999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.44800000000000001}]}]},{"osd":4,"up_from":11,"seq":47244640379,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28060,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343780,"statfs":{"total":96636764160,"available":96608030720,"internally_reserved":0,"allocated":1056768,"data_stored":363130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.48099999999999998,"15min":0.47899999999999998},"min":{"1min":0.317,"5min":0.215,"15min":0.215},"max":{"1min":0.66000000000000003,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.47999999999999998,"15min":0.48399999999999999},"min":{"1min":0.29399999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.66000000000000003,"5min":0.67300000000000004,"15min":0.69299999999999995},"last":0.40899999999999997}]},{"osd":1,"last update":"Mon Apr 22 08:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.501,"15min":0.49399999999999999},"min":{"1min":0.36899999999999999,"5min":0.27600000000000002,"15min":0.26900000000000002},"max":{"1min":0.71499999999999997,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.432},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.497,"15min":0.5},"min":{"1min":0.28399999999999997,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.68899999999999995,"5min":0.71299999999999997,"15min":0.78800000000000003},"last":0.47699999999999998}]},{"osd":2,"last update":"Mon Apr 22 08:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.52900000000000003,"15min":0.55700000000000005},"min":{"1min":0.35699999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.54100000000000004,"15min":0.53500000000000003},"min":{"1min":0.35199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.441}]},{"osd":3,"last update":"Mon Apr 22 08:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.373,"15min":0.39000000000000001},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.57799999999999996,"5min":1.204,"15min":1.204},"last":0.312},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.39100000000000001,"15min":0.40000000000000002},"min":{"1min":0.24199999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.54700000000000004,"5min":0.62,"15min":0.89100000000000001},"last":0.32900000000000001}]},{"osd":5,"last update":"Mon Apr 22 08:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.38600000000000001,"15min":0.38200000000000001},"min":{"1min":0.26200000000000001,"5min":0.251,"15min":0.23100000000000001},"max":{"1min":0.57499999999999996,"5min":0.625,"15min":0.65800000000000003},"last":0.372},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45300000000000001,"15min":0.432},"min":{"1min":0.30299999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.70499999999999996,"5min":2.2360000000000002,"15min":2.2360000000000002},"last":0.46400000000000002}]}]},{"osd":3,"up_from":11,"seq":47244640378,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28568,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343272,"statfs":{"total":96636764160,"available":96607510528,"internally_reserved":0,"allocated":1589248,"data_stored":892058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:22:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.53000000000000003,"15min":0.50800000000000001},"min":{"1min":0.34200000000000003,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.64400000000000002,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.52600000000000002,"15min":0.51000000000000001},"min":{"1min":0.38,"5min":0.32000000000000001,"15min":0.309},"max":{"1min":0.63400000000000001,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.60399999999999998}]},{"osd":1,"last update":"Mon Apr 22 08:22:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.505,"15min":0.51700000000000002},"min":{"1min":0.39100000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.505,"5min":0.502,"15min":0.502},"min":{"1min":0.34999999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.84599999999999997},"last":0.63}]},{"osd":2,"last update":"Mon Apr 22 08:22:03 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.496,"15min":0.52700000000000002},"min":{"1min":0.32200000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.86899999999999999},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.50700000000000001,"15min":0.53000000000000003},"min":{"1min":0.36299999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.63400000000000001,"5min":0.69099999999999995,"15min":1.0289999999999999},"last":0.61599999999999999}]},{"osd":4,"last update":"Mon Apr 22 08:22:03 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.35799999999999998,"15min":0.38500000000000001},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.53400000000000003,"5min":0.70499999999999996,"15min":0.877},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.379,"15min":0.40500000000000003},"min":{"1min":0.27500000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":2.7000000000000002},"last":0.46700000000000003}]},{"osd":5,"last update":"Mon Apr 22 08:22:03 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.39100000000000001,"15min":0.40600000000000003},"min":{"1min":0.23899999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.98399999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39900000000000002,"15min":0.436},"min":{"1min":0.26200000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.61699999999999999,"5min":0.63300000000000001,"15min":0.81399999999999995},"last":0.499}]}]},{"osd":1,"up_from":11,"seq":47244640378,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28056,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343784,"statfs":{"total":96636764160,"available":96608034816,"internally_reserved":0,"allocated":1056768,"data_stored":363130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.35599999999999998,"15min":0.33900000000000002},"min":{"1min":0.24399999999999999,"5min":0.182,"15min":0.182},"max":{"1min":0.47099999999999997,"5min":1.629,"15min":2.1850000000000001},"last":0.189},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.36899999999999999,"15min":0.36299999999999999},"min":{"1min":0.23899999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.52300000000000002,"5min":1.8220000000000001,"15min":3.1320000000000001},"last":0.25}]},{"osd":2,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39800000000000002,"15min":0.38100000000000001},"min":{"1min":0.252,"5min":0.22900000000000001,"15min":0.22},"max":{"1min":0.48899999999999999,"5min":2.0219999999999998,"15min":2.0219999999999998},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.436,"15min":0.42099999999999999},"min":{"1min":0.309,"5min":0.27400000000000002,"15min":0.255},"max":{"1min":0.66300000000000003,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":0.28000000000000003}]},{"osd":3,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.55600000000000005,"15min":0.58199999999999996},"min":{"1min":0.45300000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.77800000000000002,"5min":0.80500000000000005,"15min":3.504},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56299999999999994,"15min":0.56699999999999995},"min":{"1min":0.41099999999999998,"5min":0.36599999999999999,"15min":0.35799999999999998},"max":{"1min":0.68500000000000005,"5min":1.774,"15min":1.774},"last":0.45600000000000002}]},{"osd":4,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.59399999999999997,"15min":0.59699999999999998},"min":{"1min":0.40799999999999997,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.72499999999999998,"5min":3.5579999999999998,"15min":3.5579999999999998},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.56899999999999995,"15min":0.58299999999999996},"min":{"1min":0.46700000000000003,"5min":0.34599999999999997,"15min":0.316},"max":{"1min":0.73199999999999998,"5min":0.85299999999999998,"15min":3.085},"last":0.48099999999999998}]},{"osd":5,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.57399999999999995,"15min":0.57899999999999996},"min":{"1min":0.439,"5min":0.372,"15min":0.372},"max":{"1min":0.76000000000000001,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.56799999999999995,"15min":0.59499999999999997},"min":{"1min":0.42399999999999999,"5min":0.38900000000000001,"15min":0.35999999999999999},"max":{"1min":0.747,"5min":0.80200000000000005,"15min":2.6520000000000001},"last":0.435}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:22:47.766 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-04-22T08:12:49.822456+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=12, tm_sec=49, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=22, tm_sec=43, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T08:22:47.767 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-04-22T08:12:49.822456+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=12, tm_sec=49, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=22, tm_sec=43, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T08:22:47.767 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-04-22T08:12:49.822456+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=12, tm_sec=49, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=22, tm_sec=43, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T08:22:47.767 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-04-22T08:12:49.822456+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=12, tm_sec=49, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=22, tm_sec=43, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T08:22:47.767 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-04-22T08:12:49.822456+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=12, tm_sec=49, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=22, tm_sec=43, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T08:22:47.767 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-04-22T08:12:49.822456+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=12, tm_sec=49, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=22, tm_sec=43, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T08:22:47.767 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-22T08:12:49.822456+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=12, tm_sec=49, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=22, tm_sec=43, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T08:22:47.767 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-04-22T08:12:49.822456+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=12, tm_sec=49, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=22, tm_sec=43, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T08:22:47.767 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-22T08:12:49.822456+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=12, tm_sec=49, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=8, tm_min=22, tm_sec=43, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T08:22:47.768 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-22T08:23:07.769 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T08:23:07.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.881+0000 7f90c0cf4640 10 monclient: get_monmap_and_config 2024-04-22T08:23:07.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.881+0000 7f90c0cf4640 10 monclient: build_initial_monmap 2024-04-22T08:23:07.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.881+0000 7f90c0cf4640 10 monclient: monmap: 2024-04-22T08:23:07.881 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:23:07.881 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:23:07.881 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:23:07.882350+0000 2024-04-22T08:23:07.881 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:23:07.882350+0000 2024-04-22T08:23:07.881 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:23:07.881 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:23:07.881 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:23:07.882 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:23:07.882 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:23:07.882 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:23:07.882 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:23:07.882 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:23:07.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.882+0000 7f90c0cf4640 1 Processor -- start 2024-04-22T08:23:07.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.882+0000 7f90c0cf4640 1 -- start start 2024-04-22T08:23:07.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.882+0000 7f90c0cf4640 10 monclient: init 2024-04-22T08:23:07.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.882+0000 7f90c0cf4640 10 monclient: _reopen_session rank -1 2024-04-22T08:23:07.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.882+0000 7f90c0cf4640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-04-22T08:23:07.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.882+0000 7f90c0cf4640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f90bc132ba0 0x7f90bc09da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:23:07.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.882+0000 7f90c0cf4640 10 monclient(hunting): picked mon.noname-b con 0x7f90bc132ba0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:23:07.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.882+0000 7f90c0cf4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f90bc132f80 addr v1:172.21.15.189:6789/0 2024-04-22T08:23:07.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.882+0000 7f90c0cf4640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f90bc1380d0 0x7f90bc13a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:23:07.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.882+0000 7f90c0cf4640 10 monclient(hunting): picked mon.noname-a con 0x7f90bc1380d0 addr v2:172.21.15.40:3300/0 2024-04-22T08:23:07.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.882+0000 7f90c0cf4640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90bc071900 con 0x7f90bc132f80 2024-04-22T08:23:07.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.883+0000 7f90c0cf4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:23:07.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.883+0000 7f90c0cf4640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90bc09d590 con 0x7f90bc1380d0 2024-04-22T08:23:07.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.883+0000 7f90c0cf4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:23:07.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.883+0000 7f90c0cf4640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90bc09df60 con 0x7f90bc132ba0 2024-04-22T08:23:07.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.883+0000 7f90c0cf4640 10 monclient(hunting): _renew_subs 2024-04-22T08:23:07.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.883+0000 7f90c0cf4640 10 monclient(hunting): authenticate will time out at 1461.876806s 2024-04-22T08:23:07.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.883+0000 7f90ba59c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f90bc132ba0 0x7f90bc09da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:23:07.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.883+0000 7f90bad9d640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f90bc1380d0 0x7f90bc13a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:23:07.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.883+0000 7f90bad9d640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f90bc1380d0 0x7f90bc13a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38860/0 (socket says 172.21.15.40:38860) 2024-04-22T08:23:07.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.883+0000 7f90ba59c640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f90bc132ba0 0x7f90bc09da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:60060/0 (socket says 172.21.15.40:60060) 2024-04-22T08:23:07.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.883+0000 7f90bad9d640 1 -- 172.21.15.40:0/2714392120 learned_addr learned my addr 172.21.15.40:0/2714392120 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:23:07.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.883+0000 7f90bad9d640 10 monclient(hunting): get_auth_request con 0x7f90bc1380d0 auth_method 0 2024-04-22T08:23:07.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.883+0000 7f90bad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:23:07.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.883+0000 7f90bad9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:23:07.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.883+0000 7f90bad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:23:07.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.883+0000 7f90ba59c640 10 monclient(hunting): get_auth_request con 0x7f90bc132ba0 auth_method 0 2024-04-22T08:23:07.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.883+0000 7f90ba59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:23:07.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.883+0000 7f90ba59c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:23:07.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.883+0000 7f90ba59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:23:07.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.883+0000 7f90bad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:23:07.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.883+0000 7f90bad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:23:07.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.883+0000 7f90bad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:23:07.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.883+0000 7f90ba59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:23:07.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.883+0000 7f90ba59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:23:07.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.883+0000 7f90ba59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:23:07.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.883+0000 7f90ba59c640 10 monclient(hunting): handle_auth_done global_id 124193 payload 274 2024-04-22T08:23:07.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.883+0000 7f90ba59c640 1 -- 172.21.15.40:0/2714392120 >> v2:172.21.15.40:3300/0 conn(0x7f90bc1380d0 msgr2=0x7f90bc13a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:23:07.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.883+0000 7f90ba59c640 1 --2- 172.21.15.40:0/2714392120 >> v2:172.21.15.40:3300/0 conn(0x7f90bc1380d0 0x7f90bc13a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:23:07.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.883+0000 7f90ba59c640 1 -- 172.21.15.40:0/2714392120 >> v1:172.21.15.189:6789/0 conn(0x7f90bc132f80 legacy=0x7f90bc1333a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:23:07.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.883+0000 7f90ba59c640 10 monclient: _finish_hunting 0 2024-04-22T08:23:07.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.883+0000 7f90ba59c640 1 monclient: found mon.noname-b 2024-04-22T08:23:07.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.883+0000 7f90ba59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:23:07.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.883+0000 7f90ba59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:23:07.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.883+0000 7f90ba59c640 1 -- 172.21.15.40:0/2714392120 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90bc09cc80 con 0x7f90bc132ba0 2024-04-22T08:23:07.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.883+0000 7f90bad9d640 1 --2- 172.21.15.40:0/2714392120 >> v2:172.21.15.40:3300/0 conn(0x7f90bc1380d0 0x7f90bc13a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:23:07.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.886+0000 7f90ba59c640 1 -- 172.21.15.40:0/2714392120 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f90bc132ba0 msgr2=0x7f90bc09da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:23:07.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.886+0000 7f90ba59c640 1 --2- 172.21.15.40:0/2714392120 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f90bc132ba0 0x7f90bc09da20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f90ac0097a0 tx=0x7f90ac02b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33e6b9ab11b4e4f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:23:07.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.886+0000 7f90b9d9b640 1 -- 172.21.15.40:0/2714392120 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ac030920 con 0x7f90bc132ba0 2024-04-22T08:23:07.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.886+0000 7f90b9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:23:07.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.886+0000 7f90b9d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:23:07.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.886+0000 7f90b9d9b640 10 monclient: dump: 2024-04-22T08:23:07.893 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:23:07.893 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:23:07.893 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:23:07.894 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:23:07.894 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:23:07.894 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:23:07.894 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:23:07.894 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:23:07.894 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:23:07.894 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:23:07.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.886+0000 7f90b9d9b640 10 monclient: _finish_auth 0 2024-04-22T08:23:07.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.886+0000 7f90b9d9b640 10 monclient: _check_auth_tickets 2024-04-22T08:23:07.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.886+0000 7f90b9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:23:07.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.886+0000 7f90b9d9b640 1 -- 172.21.15.40:0/2714392120 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90ac02e050 con 0x7f90bc132ba0 2024-04-22T08:23:07.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.886+0000 7f90c0cf4640 5 monclient: authenticate success, global_id 124193 2024-04-22T08:23:07.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.886+0000 7f90c0cf4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:23:07.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.886+0000 7f90b9d9b640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:23:07.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.886+0000 7f90c0cf4640 10 monclient: get_monmap_and_config success 2024-04-22T08:23:07.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.886+0000 7f90c0cf4640 4 set_mon_vals no callback set 2024-04-22T08:23:07.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.886+0000 7f90b9d9b640 1 -- 172.21.15.40:0/2714392120 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ac030ca0 con 0x7f90bc132ba0 2024-04-22T08:23:07.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.886+0000 7f90b9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:23:07.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.886+0000 7f90b9d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:23:07.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.886+0000 7f90b9d9b640 10 monclient: dump: 2024-04-22T08:23:07.895 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:23:07.895 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:23:07.895 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:23:07.895 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:23:07.895 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:23:07.895 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:23:07.895 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:23:07.896 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:23:07.896 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:23:07.896 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:23:07.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.887+0000 7f90c0cf4640 10 monclient: shutdown 2024-04-22T08:23:07.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.887+0000 7f90c0cf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:23:07.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.887+0000 7f90c0cf4640 1 -- 172.21.15.40:0/2714392120 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f90bc132ba0 msgr2=0x7f90bc09da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:23:07.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.887+0000 7f90c0cf4640 1 --2- 172.21.15.40:0/2714392120 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f90bc132ba0 0x7f90bc09da20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f90ac0097a0 tx=0x7f90ac02b710 comp rx=0 tx=0).stop 2024-04-22T08:23:07.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.887+0000 7f90c0cf4640 1 -- 172.21.15.40:0/2714392120 shutdown_connections 2024-04-22T08:23:07.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.887+0000 7f90c0cf4640 1 --2- 172.21.15.40:0/2714392120 >> v2:172.21.15.40:3300/0 conn(0x7f90bc1380d0 0x7f90bc13a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:23:07.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.887+0000 7f90c0cf4640 1 --2- 172.21.15.40:0/2714392120 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f90bc132ba0 0x7f90bc09da20 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:23:07.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.887+0000 7f90c0cf4640 1 -- 172.21.15.40:0/2714392120 >> 172.21.15.40:0/2714392120 conn(0x7f90bc0a0d30 msgr2=0x7f90bc0a1140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:23:07.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.887+0000 7f90c0cf4640 1 -- 172.21.15.40:0/2714392120 shutdown_connections 2024-04-22T08:23:07.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.887+0000 7f90c0cf4640 1 -- 172.21.15.40:0/2714392120 wait complete. 2024-04-22T08:23:07.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.887+0000 7f90c0cf4640 10 monclient: build_initial_monmap 2024-04-22T08:23:07.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.887+0000 7f90c0cf4640 10 monclient: monmap: 2024-04-22T08:23:07.897 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:23:07.897 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:23:07.897 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:23:07.897 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:23:07.897 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:23:07.897 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:23:07.897 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:23:07.897 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:23:07.897 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:23:07.897 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:23:07.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.888+0000 7f90c0cf4640 1 Processor -- start 2024-04-22T08:23:07.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.888+0000 7f90c0cf4640 1 -- start start 2024-04-22T08:23:07.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.888+0000 7f90c0cf4640 10 monclient: init 2024-04-22T08:23:07.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.888+0000 7f90c0cf4640 10 monclient: _reopen_session rank -1 2024-04-22T08:23:07.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.888+0000 7f90c0cf4640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-22T08:23:07.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.888+0000 7f90c0cf4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90bc132ba0 0x7f90bc12fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:23:07.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.888+0000 7f90c0cf4640 10 monclient(hunting): picked mon.noname-b con 0x7f90bc132ba0 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:23:07.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.888+0000 7f90c0cf4640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f90bc132f80 0x7f90bc130310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:23:07.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.888+0000 7f90c0cf4640 10 monclient(hunting): picked mon.noname-c con 0x7f90bc132f80 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:23:07.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.888+0000 7f90c0cf4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90bc1380d0 0x7f90bc133c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:23:07.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.888+0000 7f90c0cf4640 10 monclient(hunting): picked mon.noname-a con 0x7f90bc1380d0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:23:07.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.888+0000 7f90c0cf4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:23:07.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.888+0000 7f90c0cf4640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90bc130850 con 0x7f90bc1380d0 2024-04-22T08:23:07.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.888+0000 7f90c0cf4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:23:07.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.888+0000 7f90c0cf4640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90bc070380 con 0x7f90bc132ba0 2024-04-22T08:23:07.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.888+0000 7f90c0cf4640 10 monclient(hunting): start opening mon connection 2024-04-22T08:23:07.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.888+0000 7f90c0cf4640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90bc06fc80 con 0x7f90bc132f80 2024-04-22T08:23:07.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.888+0000 7f90c0cf4640 10 monclient(hunting): _renew_subs 2024-04-22T08:23:07.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.888+0000 7f90c0cf4640 10 monclient(hunting): authenticate will time out at 1461.882552s 2024-04-22T08:23:07.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.888+0000 7f90b3fff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f90bc132f80 0x7f90bc130310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:23:07.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.888+0000 7f90bad9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90bc1380d0 0x7f90bc133c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:23:07.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.888+0000 7f90b3fff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f90bc132f80 0x7f90bc130310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:60062/0 (socket says 172.21.15.40:60062) 2024-04-22T08:23:07.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.888+0000 7f90b3fff640 1 -- 172.21.15.40:0/3646618268 learned_addr learned my addr 172.21.15.40:0/3646618268 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:23:07.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.888+0000 7f90b3fff640 10 monclient(hunting): get_auth_request con 0x7f90bc132f80 auth_method 0 2024-04-22T08:23:07.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.888+0000 7f90b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:23:07.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.888+0000 7f90b3fff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:23:07.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.888+0000 7f90b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:23:07.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.888+0000 7f90bad9d640 10 monclient(hunting): get_auth_request con 0x7f90bc1380d0 auth_method 0 2024-04-22T08:23:07.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.888+0000 7f90bad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:23:07.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.888+0000 7f90bad9d640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:23:07.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.888+0000 7f90bad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:23:07.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.888+0000 7f90b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:23:07.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.889+0000 7f90b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:23:07.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.889+0000 7f90b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:23:07.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.889+0000 7f90bad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:23:07.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.889+0000 7f90bad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:23:07.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.889+0000 7f90bad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:23:07.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.889+0000 7f90ba59c640 1 --2- 172.21.15.40:0/3646618268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90bc132ba0 0x7f90bc12fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:23:07.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.889+0000 7f90ba59c640 10 monclient(hunting): get_auth_request con 0x7f90bc132ba0 auth_method 0 2024-04-22T08:23:07.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.889+0000 7f90ba59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:23:07.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.889+0000 7f90ba59c640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:23:07.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.889+0000 7f90ba59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:23:07.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.889+0000 7f90b3fff640 10 monclient(hunting): handle_auth_done global_id 124196 payload 824 2024-04-22T08:23:07.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.889+0000 7f90b3fff640 1 -- 172.21.15.40:0/3646618268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90bc132ba0 msgr2=0x7f90bc12fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:23:07.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.889+0000 7f90b3fff640 1 --2- 172.21.15.40:0/3646618268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90bc132ba0 0x7f90bc12fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:23:07.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.889+0000 7f90b3fff640 1 -- 172.21.15.40:0/3646618268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90bc1380d0 msgr2=0x7f90bc133c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:23:07.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.889+0000 7f90b3fff640 1 --2- 172.21.15.40:0/3646618268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90bc1380d0 0x7f90bc133c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:23:07.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.889+0000 7f90b3fff640 10 monclient: _finish_hunting 0 2024-04-22T08:23:07.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.889+0000 7f90b3fff640 1 monclient: found mon.noname-c 2024-04-22T08:23:07.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.889+0000 7f90b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:23:07.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.889+0000 7f90b3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:23:07.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.889+0000 7f90b3fff640 1 -- 172.21.15.40:0/3646618268 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b4009b30 con 0x7f90bc132f80 2024-04-22T08:23:07.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.891+0000 7f90b3fff640 1 -- 172.21.15.40:0/3646618268 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f90bc132f80 msgr2=0x7f90bc130310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:23:07.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.891+0000 7f90b3fff640 1 --2- 172.21.15.40:0/3646618268 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f90bc132f80 0x7f90bc130310 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f90a400c430 tx=0x7f90a400c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:23:07.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.891+0000 7f90b2ffd640 1 -- 172.21.15.40:0/3646618268 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a40119a0 con 0x7f90bc132f80 2024-04-22T08:23:07.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.891+0000 7f90b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:23:07.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.891+0000 7f90b2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:23:07.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.891+0000 7f90b2ffd640 10 monclient: dump: 2024-04-22T08:23:07.901 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:23:07.901 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:23:07.902 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:23:07.902 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:23:07.902 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:23:07.902 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:23:07.902 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:23:07.902 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:23:07.902 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:23:07.902 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:23:07.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.891+0000 7f90b2ffd640 10 monclient: _finish_auth 0 2024-04-22T08:23:07.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.891+0000 7f90b2ffd640 10 monclient: _check_auth_tickets 2024-04-22T08:23:07.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.891+0000 7f90b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:23:07.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.891+0000 7f90b2ffd640 1 -- 172.21.15.40:0/3646618268 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90a4011b00 con 0x7f90bc132f80 2024-04-22T08:23:07.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.891+0000 7f90c0cf4640 5 monclient: authenticate success, global_id 124196 2024-04-22T08:23:07.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.891+0000 7f90c0cf4640 10 monclient: _renew_subs 2024-04-22T08:23:07.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.891+0000 7f90c0cf4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:23:07.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.891+0000 7f90c0cf4640 1 -- 172.21.15.40:0/3646618268 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90ac009400 con 0x7f90bc132f80 2024-04-22T08:23:07.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.891+0000 7f90b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:23:07.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.891+0000 7f90b2ffd640 1 -- 172.21.15.40:0/3646618268 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a4012420 con 0x7f90bc132f80 2024-04-22T08:23:07.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.891+0000 7f90b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:23:07.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.891+0000 7f90b2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:23:07.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.891+0000 7f90b2ffd640 10 monclient: dump: 2024-04-22T08:23:07.903 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:23:07.903 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:23:07.903 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:23:07.903 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:23:07.903 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:23:07.903 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:23:07.903 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:23:07.903 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:23:07.903 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:23:07.904 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:23:07.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.891+0000 7f90c0cf4640 10 monclient: _renew_subs 2024-04-22T08:23:07.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.891+0000 7f90b8d99640 4 set_mon_vals no callback set 2024-04-22T08:23:07.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.891+0000 7f90c0cf4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:23:07.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.891+0000 7f90c0cf4640 1 -- 172.21.15.40:0/3646618268 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90bc09a710 con 0x7f90bc132f80 2024-04-22T08:23:07.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.892+0000 7f90b2ffd640 1 -- 172.21.15.40:0/3646618268 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7f90a4012640 con 0x7f90bc132f80 2024-04-22T08:23:07.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.892+0000 7f90c0cf4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:23:07.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.892+0000 7f90c0cf4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:23:07.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.892+0000 7f90c0cf4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:23:07.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.892+0000 7f90c0cf4640 1 -- 172.21.15.40:0/3646618268 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90bc04e9b0 con 0x7f90bc132f80 2024-04-22T08:23:07.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.892+0000 7f90b2ffd640 1 --2- 172.21.15.40:0/3646618268 >> v2:172.21.15.189:6800/30168 conn(0x7f909c037820 0x7f909c039ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:23:07.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.892+0000 7f90b2ffd640 1 -- 172.21.15.40:0/3646618268 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(317..317 src has 239..317) v4 ==== 3506+0+0 (secure 0 0 0) 0x7f90a404a140 con 0x7f90bc132f80 2024-04-22T08:23:07.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.893+0000 7f90ba59c640 1 --2- 172.21.15.40:0/3646618268 >> v2:172.21.15.189:6800/30168 conn(0x7f909c037820 0x7f909c039ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:23:07.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.893+0000 7f90ba59c640 10 monclient: get_auth_request con 0x7f909c037820 auth_method 0 2024-04-22T08:23:07.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.895+0000 7f90ba59c640 1 --2- 172.21.15.40:0/3646618268 >> v2:172.21.15.189:6800/30168 conn(0x7f909c037820 0x7f909c039ce0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f90ac032000 tx=0x7f90ac02f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:23:07.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.897+0000 7f90b2ffd640 1 -- 172.21.15.40:0/3646618268 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7f90a4010d40 con 0x7f90bc132f80 2024-04-22T08:23:07.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.897+0000 7f90b2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:23:07.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:07.897+0000 7f90b2ffd640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:23:08.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.077+0000 7f90c0cf4640 1 -- 172.21.15.40:0/3646618268 --> v2:172.21.15.189:6800/30168 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f90bc09cf90 con 0x7f909c037820 2024-04-22T08:23:08.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.080+0000 7f90b2ffd640 1 -- 172.21.15.40:0/3646618268 <== mgr.4102 v2:172.21.15.189:6800/30168 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+54131 (secure 0 0 0) 0x7f90bc09cf90 con 0x7f909c037820 2024-04-22T08:23:08.080 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-22T08:23:08.080 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-22T08:23:08.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.082+0000 7f90c0cf4640 1 -- 172.21.15.40:0/3646618268 >> v2:172.21.15.189:6800/30168 conn(0x7f909c037820 msgr2=0x7f909c039ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:23:08.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.082+0000 7f90c0cf4640 1 --2- 172.21.15.40:0/3646618268 >> v2:172.21.15.189:6800/30168 conn(0x7f909c037820 0x7f909c039ce0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f90ac032000 tx=0x7f90ac02f000 comp rx=0 tx=0).stop 2024-04-22T08:23:08.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.082+0000 7f90c0cf4640 10 monclient: shutdown 2024-04-22T08:23:08.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.082+0000 7f90c0cf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:23:08.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.082+0000 7f90c0cf4640 1 -- 172.21.15.40:0/3646618268 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f90bc132f80 msgr2=0x7f90bc130310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:23:08.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.082+0000 7f90c0cf4640 1 --2- 172.21.15.40:0/3646618268 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f90bc132f80 0x7f90bc130310 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f90a400c430 tx=0x7f90a400c8f0 comp rx=0 tx=0).stop 2024-04-22T08:23:08.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.082+0000 7f90c0cf4640 1 -- 172.21.15.40:0/3646618268 shutdown_connections 2024-04-22T08:23:08.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.082+0000 7f90c0cf4640 1 --2- 172.21.15.40:0/3646618268 >> v2:172.21.15.189:6800/30168 conn(0x7f909c037820 0x7f909c039ce0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:23:08.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.082+0000 7f90c0cf4640 1 --2- 172.21.15.40:0/3646618268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90bc1380d0 0x7f90bc133c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:23:08.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.082+0000 7f90c0cf4640 1 --2- 172.21.15.40:0/3646618268 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f90bc132f80 0x7f90bc130310 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:23:08.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.082+0000 7f90c0cf4640 1 --2- 172.21.15.40:0/3646618268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90bc132ba0 0x7f90bc12fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:23:08.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.082+0000 7f90c0cf4640 1 -- 172.21.15.40:0/3646618268 >> 172.21.15.40:0/3646618268 conn(0x7f90bc0a0d30 msgr2=0x7f90bc0a3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:23:08.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.082+0000 7f90c0cf4640 1 -- 172.21.15.40:0/3646618268 shutdown_connections 2024-04-22T08:23:08.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.082+0000 7f90c0cf4640 1 -- 172.21.15.40:0/3646618268 wait complete. 2024-04-22T08:23:08.092 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":624,"stamp":"2024-04-22T08:23:06.211050+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":185724,"kb_used_data":7232,"kb_used_omap":9,"kb_used_meta":178422,"kb_avail":566045316,"statfs":{"total":579820584960,"available":579630403584,"internally_reserved":0,"allocated":7405568,"data_stored":3236636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9560,"internal_metadata":182704808},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002466"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":316,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-04-22T08:22:46.898073+0000","last_change":"2024-04-22T08:22:46.898175+0000","last_active":"2024-04-22T08:22:46.898073+0000","last_peered":"2024-04-22T08:22:46.898073+0000","last_clean":"2024-04-22T08:22:46.898073+0000","last_became_active":"2024-04-22T08:12:51.442730+0000","last_became_peered":"2024-04-22T08:12:51.442730+0000","last_unstale":"2024-04-22T08:22:46.898073+0000","last_undegraded":"2024-04-22T08:22:46.898073+0000","last_fullsized":"2024-04-22T08:22:46.898073+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:22:46.898052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:22:46.898052+0000","last_clean_scrub_stamp":"2024-04-22T08:22:46.898052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:20:11.891367+0000","scrub_duration":0.0026156700000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":305,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-04-22T08:22:46.587812+0000","last_change":"2024-04-22T08:22:46.587891+0000","last_active":"2024-04-22T08:22:46.587812+0000","last_peered":"2024-04-22T08:22:46.587812+0000","last_clean":"2024-04-22T08:22:46.587812+0000","last_became_active":"2024-04-22T08:12:50.841008+0000","last_became_peered":"2024-04-22T08:12:50.841008+0000","last_unstale":"2024-04-22T08:22:46.587812+0000","last_undegraded":"2024-04-22T08:22:46.587812+0000","last_fullsized":"2024-04-22T08:22:46.587812+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:22:46.587793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:22:46.587793+0000","last_clean_scrub_stamp":"2024-04-22T08:22:46.587793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:59:32.530798+0000","scrub_duration":0.0025187899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":316,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-04-22T08:22:47.914443+0000","last_change":"2024-04-22T08:22:47.914546+0000","last_active":"2024-04-22T08:22:47.914443+0000","last_peered":"2024-04-22T08:22:47.914443+0000","last_clean":"2024-04-22T08:22:47.914443+0000","last_became_active":"2024-04-22T08:12:51.403182+0000","last_became_peered":"2024-04-22T08:12:51.403182+0000","last_unstale":"2024-04-22T08:22:47.914443+0000","last_undegraded":"2024-04-22T08:22:47.914443+0000","last_fullsized":"2024-04-22T08:22:47.914443+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:22:47.914420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:22:47.914420+0000","last_clean_scrub_stamp":"2024-04-22T08:22:47.914420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:41:52.888486+0000","scrub_duration":0.0028351449999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":305,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-04-22T08:22:48.544008+0000","last_change":"2024-04-22T08:22:48.544121+0000","last_active":"2024-04-22T08:22:48.544008+0000","last_peered":"2024-04-22T08:22:48.544008+0000","last_clean":"2024-04-22T08:22:48.544008+0000","last_became_active":"2024-04-22T08:12:51.402980+0000","last_became_peered":"2024-04-22T08:12:51.402980+0000","last_unstale":"2024-04-22T08:22:48.544008+0000","last_undegraded":"2024-04-22T08:22:48.544008+0000","last_fullsized":"2024-04-22T08:22:48.544008+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:22:48.543987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:22:48.543987+0000","last_clean_scrub_stamp":"2024-04-22T08:22:48.543987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:04:14.635346+0000","scrub_duration":0.0020103030000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":307,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-04-22T08:22:55.066905+0000","last_change":"2024-04-22T08:22:55.067026+0000","last_active":"2024-04-22T08:22:55.066905+0000","last_peered":"2024-04-22T08:22:55.066905+0000","last_clean":"2024-04-22T08:22:55.066905+0000","last_became_active":"2024-04-22T08:12:50.836250+0000","last_became_peered":"2024-04-22T08:12:50.836250+0000","last_unstale":"2024-04-22T08:22:55.066905+0000","last_undegraded":"2024-04-22T08:22:55.066905+0000","last_fullsized":"2024-04-22T08:22:55.066905+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:22:55.066882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:22:55.066882+0000","last_clean_scrub_stamp":"2024-04-22T08:22:55.066882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:05:59.727986+0000","scrub_duration":0.0031178880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":314,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-04-22T08:22:48.395676+0000","last_change":"2024-04-22T08:22:48.395803+0000","last_active":"2024-04-22T08:22:48.395676+0000","last_peered":"2024-04-22T08:22:48.395676+0000","last_clean":"2024-04-22T08:22:48.395676+0000","last_became_active":"2024-04-22T08:12:50.836210+0000","last_became_peered":"2024-04-22T08:12:50.836210+0000","last_unstale":"2024-04-22T08:22:48.395676+0000","last_undegraded":"2024-04-22T08:22:48.395676+0000","last_fullsized":"2024-04-22T08:22:48.395676+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:22:48.395654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:22:48.395654+0000","last_clean_scrub_stamp":"2024-04-22T08:22:48.395654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T19:41:49.909008+0000","scrub_duration":0.0028549180000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":317,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-04-22T08:22:50.823051+0000","last_change":"2024-04-22T08:22:50.823051+0000","last_active":"2024-04-22T08:22:50.823051+0000","last_peered":"2024-04-22T08:22:50.823051+0000","last_clean":"2024-04-22T08:22:50.823051+0000","last_became_active":"2024-04-22T08:12:50.842392+0000","last_became_peered":"2024-04-22T08:12:50.842392+0000","last_unstale":"2024-04-22T08:22:50.823051+0000","last_undegraded":"2024-04-22T08:22:50.823051+0000","last_fullsized":"2024-04-22T08:22:50.823051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:22:50.822872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:22:50.822872+0000","last_clean_scrub_stamp":"2024-04-22T08:22:50.822872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:26:55.650251+0000","scrub_duration":0.0023543629999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":307,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-04-22T08:22:52.045099+0000","last_change":"2024-04-22T08:22:52.045199+0000","last_active":"2024-04-22T08:22:52.045099+0000","last_peered":"2024-04-22T08:22:52.045099+0000","last_clean":"2024-04-22T08:22:52.045099+0000","last_became_active":"2024-04-22T08:12:50.835541+0000","last_became_peered":"2024-04-22T08:12:50.835541+0000","last_unstale":"2024-04-22T08:22:52.045099+0000","last_undegraded":"2024-04-22T08:22:52.045099+0000","last_fullsized":"2024-04-22T08:22:52.045099+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T08:22:52.045076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T08:22:52.045076+0000","last_clean_scrub_stamp":"2024-04-22T08:22:52.045076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:09:59.571619+0000","scrub_duration":0.0031079509999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"317'119","reported_seq":469,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-04-22T08:22:48.887657+0000","last_change":"2024-04-22T08:22:48.887657+0000","last_active":"2024-04-22T08:22:48.887657+0000","last_peered":"2024-04-22T08:22:48.887657+0000","last_clean":"2024-04-22T08:22:48.887657+0000","last_became_active":"2024-04-22T08:12:51.403208+0000","last_became_peered":"2024-04-22T08:12:51.403208+0000","last_unstale":"2024-04-22T08:22:48.887657+0000","last_undegraded":"2024-04-22T08:22:48.887657+0000","last_fullsized":"2024-04-22T08:22:48.887657+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'119","last_scrub_stamp":"2024-04-22T08:22:48.887535+0000","last_deep_scrub":"317'119","last_deep_scrub_stamp":"2024-04-22T08:22:48.887535+0000","last_clean_scrub_stamp":"2024-04-22T08:22:48.887535+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:15:27.429440+0000","scrub_duration":0.0057381940000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640384,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343792,"statfs":{"total":96636764160,"available":96608043008,"internally_reserved":0,"allocated":1056768,"data_stored":363130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:23:04 2024","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.32600000000000001,"15min":0.36599999999999999},"min":{"1min":0.23599999999999999,"5min":0.187,"15min":0.187},"max":{"1min":0.38100000000000001,"5min":0.47899999999999998,"15min":2.5920000000000001},"last":0.379},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.38,"15min":0.38400000000000001},"min":{"1min":0.27600000000000002,"5min":0.249,"15min":0.21299999999999999},"max":{"1min":0.438,"5min":0.60799999999999998,"15min":2.7349999999999999},"last":0.438}]},{"osd":1,"last update":"Mon Apr 22 08:23:04 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.33900000000000002,"15min":0.34300000000000003},"min":{"1min":0.25700000000000001,"5min":0.23899999999999999,"15min":0.17100000000000001},"max":{"1min":0.40999999999999998,"5min":0.47199999999999998,"15min":0.55700000000000005},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.38200000000000001,"15min":0.34699999999999998},"min":{"1min":0.27100000000000002,"5min":0.251,"15min":0.20699999999999999},"max":{"1min":0.47699999999999998,"5min":0.55600000000000005,"15min":0.70099999999999996},"last":0.46700000000000003}]},{"osd":3,"last update":"Mon Apr 22 08:23:04 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.59099999999999997,"15min":0.56699999999999995},"min":{"1min":0.48799999999999999,"5min":0.26700000000000002,"15min":0.22500000000000001},"max":{"1min":0.70099999999999996,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.57099999999999995,"15min":0.59599999999999997},"min":{"1min":0.41699999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.73399999999999999,"5min":0.84799999999999998,"15min":1.8440000000000001},"last":0.73399999999999999}]},{"osd":4,"last update":"Mon Apr 22 08:23:04 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.56599999999999995,"15min":0.66900000000000004},"min":{"1min":0.44600000000000001,"5min":0.33800000000000002,"15min":0.30499999999999999},"max":{"1min":0.75600000000000001,"5min":0.83299999999999996,"15min":6.2949999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.55200000000000005,"15min":0.66500000000000004},"min":{"1min":0.44700000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.77400000000000002,"5min":0.91200000000000003,"15min":6.2149999999999999},"last":0.77400000000000002}]},{"osd":5,"last update":"Mon Apr 22 08:23:04 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56899999999999995,"15min":0.53700000000000003},"min":{"1min":0.441,"5min":0.38900000000000001,"15min":0.27800000000000002},"max":{"1min":0.73699999999999999,"5min":0.84299999999999997,"15min":0.85199999999999998},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.56100000000000005,"15min":0.53800000000000003},"min":{"1min":0.42599999999999999,"5min":0.35299999999999998,"15min":0.29699999999999999},"max":{"1min":0.71899999999999997,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.71899999999999997}]}]},{"osd":0,"up_from":11,"seq":47244640383,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28568,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343272,"statfs":{"total":96636764160,"available":96607510528,"internally_reserved":0,"allocated":1589248,"data_stored":892058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.32000000000000001,"15min":0.34599999999999997},"min":{"1min":0.23699999999999999,"5min":0.189,"15min":0.18099999999999999},"max":{"1min":0.46500000000000002,"5min":0.47599999999999998,"15min":1.7190000000000001},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.32600000000000001,"15min":0.33000000000000002},"min":{"1min":0.22,"5min":0.21199999999999999,"15min":0.19800000000000001},"max":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.56999999999999995},"last":0.312}]},{"osd":2,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.373,"15min":0.36099999999999999},"min":{"1min":0.29099999999999998,"5min":0.26300000000000001,"15min":0.222},"max":{"1min":0.499,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.38,"15min":0.39200000000000002},"min":{"1min":0.24299999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.51400000000000001,"5min":0.60399999999999998,"15min":0.626},"last":0.27800000000000002}]},{"osd":3,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55300000000000005,"15min":0.54900000000000004},"min":{"1min":0.37,"5min":0.35699999999999998,"15min":0.32200000000000001},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.505},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.54500000000000004,"15min":0.53300000000000003},"min":{"1min":0.42999999999999999,"5min":0.32300000000000001,"15min":0.29499999999999998},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.92000000000000004},"last":0.41499999999999998}]},{"osd":4,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.55000000000000004,"15min":0.55100000000000005},"min":{"1min":0.42599999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.72099999999999997,"5min":0.81799999999999995,"15min":0.85999999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.54700000000000004,"15min":0.54200000000000004},"min":{"1min":0.39700000000000002,"5min":0.36299999999999999,"15min":0.33900000000000002},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.45600000000000002}]},{"osd":5,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55000000000000004,"15min":0.54600000000000004},"min":{"1min":0.41399999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.71999999999999997,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54900000000000004,"15min":0.55500000000000005},"min":{"1min":0.38500000000000001,"5min":0.36799999999999999,"15min":0.35899999999999999},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.53400000000000003}]}]},{"osd":5,"up_from":11,"seq":47244640384,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44432,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":43390,"kb_avail":94327408,"statfs":{"total":96636764160,"available":96591265792,"internally_reserved":0,"allocated":1056768,"data_stored":363130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":44431814},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:22:04 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.49099999999999999,"15min":0.49199999999999999},"min":{"1min":0.314,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.60699999999999998,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.496,"5min":0.49399999999999999,"15min":0.51700000000000002},"min":{"1min":0.32700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":1.484},"last":0.51100000000000001}]},{"osd":1,"last update":"Mon Apr 22 08:22:04 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.49399999999999999,"15min":0.504},"min":{"1min":0.36099999999999999,"5min":0.247,"15min":0.247},"max":{"1min":0.64200000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52600000000000002,"15min":0.52700000000000002},"min":{"1min":0.36899999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.65500000000000003,"5min":0.75700000000000001,"15min":0.81499999999999995},"last":0.54200000000000004}]},{"osd":2,"last update":"Mon Apr 22 08:22:04 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.5,"15min":0.51300000000000001},"min":{"1min":0.40500000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.67500000000000004,"5min":0.68600000000000005,"15min":0.76800000000000002},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.499,"5min":0.504,"15min":0.58099999999999996},"min":{"1min":0.38400000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.57599999999999996,"5min":0.74099999999999999,"15min":3.6970000000000001},"last":0.44800000000000001}]},{"osd":3,"last update":"Mon Apr 22 08:22:04 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.39900000000000002,"15min":0.39400000000000002},"min":{"1min":0.28699999999999998,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.52000000000000002,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41599999999999998,"15min":0.40899999999999997},"min":{"1min":0.29299999999999998,"5min":0.25900000000000001,"15min":0.255},"max":{"1min":0.60099999999999998,"5min":0.62,"15min":0.621},"last":0.32700000000000001}]},{"osd":4,"last update":"Mon Apr 22 08:22:04 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.38100000000000001,"15min":0.41699999999999998},"min":{"1min":0.252,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.47199999999999998,"5min":1.0820000000000001,"15min":1.528},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.379,"5min":0.40400000000000003,"15min":0.42999999999999999},"min":{"1min":0.27800000000000002,"5min":0.24099999999999999,"15min":0.23599999999999999},"max":{"1min":0.61299999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.35499999999999998}]}]},{"osd":4,"up_from":11,"seq":47244640383,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28060,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343780,"statfs":{"total":96636764160,"available":96608030720,"internally_reserved":0,"allocated":1056768,"data_stored":363130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:23:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.496,"15min":0.47999999999999998},"min":{"1min":0.254,"5min":0.215,"15min":0.215},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.501,"5min":0.49199999999999999,"15min":0.48599999999999999},"min":{"1min":0.32000000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.622,"5min":0.67300000000000004,"15min":0.69299999999999995},"last":0.60999999999999999}]},{"osd":1,"last update":"Mon Apr 22 08:23:02 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.503,"15min":0.495},"min":{"1min":0.34499999999999997,"5min":0.27600000000000002,"15min":0.26900000000000002},"max":{"1min":0.66300000000000003,"5min":0.71499999999999997,"15min":1.2270000000000001},"last":0.621},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.50700000000000001,"15min":0.502},"min":{"1min":0.371,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.78800000000000003},"last":0.58099999999999996}]},{"osd":2,"last update":"Mon Apr 22 08:23:02 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.53300000000000003,"15min":0.55400000000000005},"min":{"1min":0.307,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.71099999999999997,"5min":0.997,"15min":0.997},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.56000000000000005,"15min":0.53700000000000003},"min":{"1min":0.41899999999999998,"5min":0.35199999999999998,"15min":0.29199999999999998},"max":{"1min":0.69399999999999995,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.64300000000000002}]},{"osd":3,"last update":"Mon Apr 22 08:23:02 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.37,"15min":0.38700000000000001},"min":{"1min":0.28199999999999997,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.54000000000000004,"5min":0.58399999999999996,"15min":1.204},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.29299999999999998,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.55700000000000005,"5min":0.62,"15min":0.89100000000000001},"last":0.51500000000000001}]},{"osd":5,"last update":"Mon Apr 22 08:23:02 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.39400000000000002,"15min":0.38400000000000001},"min":{"1min":0.29199999999999998,"5min":0.251,"15min":0.23100000000000001},"max":{"1min":0.58599999999999997,"5min":0.625,"15min":0.65800000000000003},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.433,"15min":0.433},"min":{"1min":0.32600000000000001,"5min":0.30299999999999999,"15min":0.30199999999999999},"max":{"1min":0.56100000000000005,"5min":0.70499999999999996,"15min":2.2360000000000002},"last":0.56100000000000005}]}]},{"osd":3,"up_from":11,"seq":47244640382,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28568,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343272,"statfs":{"total":96636764160,"available":96607510528,"internally_reserved":0,"allocated":1589248,"data_stored":892058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:22:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.53000000000000003,"15min":0.50800000000000001},"min":{"1min":0.34200000000000003,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.64400000000000002,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.52600000000000002,"15min":0.51000000000000001},"min":{"1min":0.38,"5min":0.32000000000000001,"15min":0.309},"max":{"1min":0.63400000000000001,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.66500000000000004}]},{"osd":1,"last update":"Mon Apr 22 08:22:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.505,"15min":0.51700000000000002},"min":{"1min":0.39100000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.502,"15min":0.502},"min":{"1min":0.34999999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.84599999999999997},"last":0.629}]},{"osd":2,"last update":"Mon Apr 22 08:22:03 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.496,"15min":0.52700000000000002},"min":{"1min":0.32200000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.86899999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.50700000000000001,"15min":0.53000000000000003},"min":{"1min":0.36299999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.63400000000000001,"5min":0.69099999999999995,"15min":1.0289999999999999},"last":0.503}]},{"osd":4,"last update":"Mon Apr 22 08:22:03 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.35799999999999998,"15min":0.38500000000000001},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.53400000000000003,"5min":0.70499999999999996,"15min":0.877},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.379,"15min":0.40500000000000003},"min":{"1min":0.27500000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":2.7000000000000002},"last":0.58099999999999996}]},{"osd":5,"last update":"Mon Apr 22 08:22:03 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.39100000000000001,"15min":0.40600000000000003},"min":{"1min":0.23899999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.98399999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39900000000000002,"15min":0.436},"min":{"1min":0.26200000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.61699999999999999,"5min":0.63300000000000001,"15min":0.81399999999999995},"last":0.56100000000000005}]}]},{"osd":1,"up_from":11,"seq":47244640382,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343792,"statfs":{"total":96636764160,"available":96608043008,"internally_reserved":0,"allocated":1056768,"data_stored":363130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.35599999999999998,"15min":0.33900000000000002},"min":{"1min":0.24399999999999999,"5min":0.182,"15min":0.182},"max":{"1min":0.47099999999999997,"5min":1.629,"15min":2.1850000000000001},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.36899999999999999,"15min":0.36299999999999999},"min":{"1min":0.23899999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.52300000000000002,"5min":1.8220000000000001,"15min":3.1320000000000001},"last":0.378}]},{"osd":2,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39800000000000002,"15min":0.38100000000000001},"min":{"1min":0.252,"5min":0.22900000000000001,"15min":0.22},"max":{"1min":0.48899999999999999,"5min":2.0219999999999998,"15min":2.0219999999999998},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.436,"15min":0.42099999999999999},"min":{"1min":0.309,"5min":0.27400000000000002,"15min":0.255},"max":{"1min":0.66300000000000003,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":0.47099999999999997}]},{"osd":3,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.55600000000000005,"15min":0.58199999999999996},"min":{"1min":0.45300000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.77800000000000002,"5min":0.80500000000000005,"15min":3.504},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56299999999999994,"15min":0.56699999999999995},"min":{"1min":0.41099999999999998,"5min":0.36599999999999999,"15min":0.35799999999999998},"max":{"1min":0.68500000000000005,"5min":1.774,"15min":1.774},"last":0.66300000000000003}]},{"osd":4,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.59399999999999997,"15min":0.59699999999999998},"min":{"1min":0.40799999999999997,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.72499999999999998,"5min":3.5579999999999998,"15min":3.5579999999999998},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.56899999999999995,"15min":0.58299999999999996},"min":{"1min":0.46700000000000003,"5min":0.34599999999999997,"15min":0.316},"max":{"1min":0.73199999999999998,"5min":0.85299999999999998,"15min":3.085},"last":0.59099999999999997}]},{"osd":5,"last update":"Mon Apr 22 08:22:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.57399999999999995,"15min":0.57899999999999996},"min":{"1min":0.439,"5min":0.372,"15min":0.372},"max":{"1min":0.76000000000000001,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.56799999999999995,"15min":0.59499999999999997},"min":{"1min":0.42399999999999999,"5min":0.38900000000000001,"15min":0.35999999999999999},"max":{"1min":0.747,"5min":0.80200000000000005,"15min":2.6520000000000001},"last":0.61799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T08:23:08.093 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-22T08:23:08.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.192+0000 7fe441b1f640 10 monclient: get_monmap_and_config 2024-04-22T08:23:08.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.192+0000 7fe441b1f640 10 monclient: build_initial_monmap 2024-04-22T08:23:08.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.192+0000 7fe441b1f640 10 monclient: monmap: 2024-04-22T08:23:08.192 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:23:08.192 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:23:08.192 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:23:08.193236+0000 2024-04-22T08:23:08.192 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:23:08.193236+0000 2024-04-22T08:23:08.192 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:23:08.192 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:23:08.192 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2024-04-22T08:23:08.192 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c-legacy 2024-04-22T08:23:08.192 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2024-04-22T08:23:08.193 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2024-04-22T08:23:08.193 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.189:3300/0 mon.noname-c 2024-04-22T08:23:08.193 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:23:08.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.193+0000 7fe441b1f640 1 Processor -- start 2024-04-22T08:23:08.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.193+0000 7fe441b1f640 1 -- start start 2024-04-22T08:23:08.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.193+0000 7fe441b1f640 10 monclient: init 2024-04-22T08:23:08.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.193+0000 7fe441b1f640 10 monclient: _reopen_session rank -1 2024-04-22T08:23:08.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.193+0000 7fe441b1f640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-04-22T08:23:08.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.193+0000 7fe441b1f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe43c131e00 addr v1:172.21.15.189:6789/0 2024-04-22T08:23:08.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.193+0000 7fe441b1f640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe43c09de60 0x7fe43c1321e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:23:08.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.193+0000 7fe441b1f640 10 monclient(hunting): picked mon.noname-b con 0x7fe43c09de60 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:23:08.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.193+0000 7fe441b1f640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe43c133010 0x7fe43c13ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:23:08.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.193+0000 7fe441b1f640 10 monclient(hunting): picked mon.noname-a con 0x7fe43c133010 addr v2:172.21.15.40:3300/0 2024-04-22T08:23:08.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.193+0000 7fe441b1f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe43c09b550 con 0x7fe43c131e00 2024-04-22T08:23:08.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.193+0000 7fe441b1f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:23:08.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.193+0000 7fe441b1f640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe43c09cc10 con 0x7fe43c133010 2024-04-22T08:23:08.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.193+0000 7fe441b1f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:23:08.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.193+0000 7fe441b1f640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe43c09cef0 con 0x7fe43c09de60 2024-04-22T08:23:08.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.193+0000 7fe441b1f640 10 monclient(hunting): _renew_subs 2024-04-22T08:23:08.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.193+0000 7fe441b1f640 10 monclient(hunting): authenticate will time out at 1462.187525s 2024-04-22T08:23:08.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.193+0000 7fe43bfff640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe43c133010 0x7fe43c13ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:23:08.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.193+0000 7fe43affd640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe43c09de60 0x7fe43c1321e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:23:08.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.193+0000 7fe43bfff640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe43c133010 0x7fe43c13ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38866/0 (socket says 172.21.15.40:38866) 2024-04-22T08:23:08.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.193+0000 7fe43affd640 10 monclient(hunting): get_auth_request con 0x7fe43c09de60 auth_method 0 2024-04-22T08:23:08.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.193+0000 7fe43bfff640 1 -- 172.21.15.40:0/2716597424 learned_addr learned my addr 172.21.15.40:0/2716597424 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:23:08.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.193+0000 7fe43affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:23:08.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.193+0000 7fe43affd640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:23:08.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.193+0000 7fe43affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:23:08.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.193+0000 7fe43bfff640 10 monclient(hunting): get_auth_request con 0x7fe43c133010 auth_method 0 2024-04-22T08:23:08.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.193+0000 7fe43bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:23:08.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.193+0000 7fe43bfff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:23:08.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.193+0000 7fe43bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:23:08.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.194+0000 7fe43bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:23:08.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.194+0000 7fe43bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:23:08.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.194+0000 7fe43bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:23:08.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.194+0000 7fe43affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:23:08.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.194+0000 7fe43affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:23:08.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.194+0000 7fe43affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:23:08.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.194+0000 7fe43affd640 10 monclient(hunting): handle_auth_done global_id 124199 payload 274 2024-04-22T08:23:08.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.194+0000 7fe43affd640 1 -- 172.21.15.40:0/2716597424 >> v2:172.21.15.40:3300/0 conn(0x7fe43c133010 msgr2=0x7fe43c13ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:23:08.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.194+0000 7fe43affd640 1 --2- 172.21.15.40:0/2716597424 >> v2:172.21.15.40:3300/0 conn(0x7fe43c133010 0x7fe43c13ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:23:08.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.194+0000 7fe43affd640 1 -- 172.21.15.40:0/2716597424 >> v1:172.21.15.189:6789/0 conn(0x7fe43c131e00 legacy=0x7fe43c132900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T08:23:08.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.194+0000 7fe43affd640 10 monclient: _finish_hunting 0 2024-04-22T08:23:08.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.194+0000 7fe43affd640 1 monclient: found mon.noname-b 2024-04-22T08:23:08.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.194+0000 7fe43affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:23:08.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.194+0000 7fe43affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2024-04-22T08:23:08.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.194+0000 7fe43affd640 1 -- 172.21.15.40:0/2716597424 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe43c09d1d0 con 0x7fe43c09de60 2024-04-22T08:23:08.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.194+0000 7fe43bfff640 1 --2- 172.21.15.40:0/2716597424 >> v2:172.21.15.40:3300/0 conn(0x7fe43c133010 0x7fe43c13ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:23:08.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.197+0000 7fe43affd640 1 -- 172.21.15.40:0/2716597424 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe43c09de60 msgr2=0x7fe43c1321e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:23:08.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.197+0000 7fe43affd640 1 --2- 172.21.15.40:0/2716597424 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe43c09de60 0x7fe43c1321e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe43c09b050 tx=0x7fe43002bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=993edc4ff6eb07dc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:23:08.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.197+0000 7fe43a7fc640 1 -- 172.21.15.40:0/2716597424 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe43002cc40 con 0x7fe43c09de60 2024-04-22T08:23:08.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.197+0000 7fe43a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:23:08.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.197+0000 7fe43a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-22T08:23:08.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.197+0000 7fe43a7fc640 10 monclient: dump: 2024-04-22T08:23:08.204 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:23:08.204 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:23:08.204 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:23:08.204 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:23:08.204 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:23:08.204 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:23:08.204 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:23:08.204 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:23:08.204 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:23:08.205 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:23:08.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.197+0000 7fe43a7fc640 10 monclient: _finish_auth 0 2024-04-22T08:23:08.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.197+0000 7fe43a7fc640 10 monclient: _check_auth_tickets 2024-04-22T08:23:08.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.197+0000 7fe43a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:23:08.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.197+0000 7fe43a7fc640 1 -- 172.21.15.40:0/2716597424 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe43002cda0 con 0x7fe43c09de60 2024-04-22T08:23:08.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.197+0000 7fe441b1f640 5 monclient: authenticate success, global_id 124199 2024-04-22T08:23:08.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.197+0000 7fe441b1f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-22T08:23:08.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.197+0000 7fe43a7fc640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:23:08.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.197+0000 7fe43a7fc640 1 -- 172.21.15.40:0/2716597424 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe430034660 con 0x7fe43c09de60 2024-04-22T08:23:08.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.197+0000 7fe441b1f640 10 monclient: get_monmap_and_config success 2024-04-22T08:23:08.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.197+0000 7fe441b1f640 4 set_mon_vals no callback set 2024-04-22T08:23:08.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.197+0000 7fe43a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:23:08.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.197+0000 7fe43a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:23:08.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.197+0000 7fe43a7fc640 10 monclient: dump: 2024-04-22T08:23:08.206 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:23:08.206 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:23:08.206 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:23:08.206 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:23:08.206 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:23:08.206 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:23:08.206 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:23:08.206 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:23:08.206 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:23:08.206 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:23:08.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.197+0000 7fe441b1f640 10 monclient: shutdown 2024-04-22T08:23:08.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.197+0000 7fe441b1f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:23:08.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.197+0000 7fe441b1f640 1 -- 172.21.15.40:0/2716597424 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe43c09de60 msgr2=0x7fe43c1321e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:23:08.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.197+0000 7fe441b1f640 1 --2- 172.21.15.40:0/2716597424 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe43c09de60 0x7fe43c1321e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe43c09b050 tx=0x7fe43002bc80 comp rx=0 tx=0).stop 2024-04-22T08:23:08.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.197+0000 7fe441b1f640 1 -- 172.21.15.40:0/2716597424 shutdown_connections 2024-04-22T08:23:08.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.197+0000 7fe441b1f640 1 --2- 172.21.15.40:0/2716597424 >> v2:172.21.15.40:3300/0 conn(0x7fe43c133010 0x7fe43c13ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:23:08.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.197+0000 7fe441b1f640 1 --2- 172.21.15.40:0/2716597424 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe43c09de60 0x7fe43c1321e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:23:08.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.197+0000 7fe441b1f640 1 -- 172.21.15.40:0/2716597424 >> 172.21.15.40:0/2716597424 conn(0x7fe43c0a0b50 msgr2=0x7fe43c0a0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:23:08.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.197+0000 7fe441b1f640 1 -- 172.21.15.40:0/2716597424 shutdown_connections 2024-04-22T08:23:08.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.197+0000 7fe441b1f640 1 -- 172.21.15.40:0/2716597424 wait complete. 2024-04-22T08:23:08.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.198+0000 7fe441b1f640 10 monclient: build_initial_monmap 2024-04-22T08:23:08.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.198+0000 7fe441b1f640 10 monclient: monmap: 2024-04-22T08:23:08.207 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2024-04-22T08:23:08.207 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-22T08:23:08.207 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2024-04-22T08:23:08.207 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2024-04-22T08:23:08.207 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2024-04-22T08:23:08.207 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:23:08.208 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2024-04-22T08:23:08.208 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.noname-b 2024-04-22T08:23:08.208 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2024-04-22T08:23:08.208 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:23:08.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.198+0000 7fe441b1f640 1 Processor -- start 2024-04-22T08:23:08.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.198+0000 7fe441b1f640 1 -- start start 2024-04-22T08:23:08.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.198+0000 7fe441b1f640 10 monclient: init 2024-04-22T08:23:08.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.199+0000 7fe441b1f640 10 monclient: _reopen_session rank -1 2024-04-22T08:23:08.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.199+0000 7fe441b1f640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-22T08:23:08.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.199+0000 7fe441b1f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe43c09de60 0x7fe43c1308f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:23:08.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.199+0000 7fe441b1f640 10 monclient(hunting): picked mon.noname-a con 0x7fe43c09de60 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2024-04-22T08:23:08.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.199+0000 7fe441b1f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe43c131e00 0x7fe43c130e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:23:08.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.199+0000 7fe441b1f640 10 monclient(hunting): picked mon.noname-b con 0x7fe43c131e00 addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-22T08:23:08.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.199+0000 7fe441b1f640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe43c133010 0x7fe43c11a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:23:08.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.199+0000 7fe441b1f640 10 monclient(hunting): picked mon.noname-c con 0x7fe43c133010 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2024-04-22T08:23:08.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.199+0000 7fe441b1f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:23:08.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.199+0000 7fe441b1f640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe43c1c7630 con 0x7fe43c09de60 2024-04-22T08:23:08.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.199+0000 7fe441b1f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:23:08.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.199+0000 7fe441b1f640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe43c09ff00 con 0x7fe43c131e00 2024-04-22T08:23:08.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.199+0000 7fe441b1f640 10 monclient(hunting): start opening mon connection 2024-04-22T08:23:08.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.199+0000 7fe441b1f640 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe43c09ed10 con 0x7fe43c133010 2024-04-22T08:23:08.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.199+0000 7fe441b1f640 10 monclient(hunting): _renew_subs 2024-04-22T08:23:08.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.199+0000 7fe441b1f640 10 monclient(hunting): authenticate will time out at 1462.193217s 2024-04-22T08:23:08.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.199+0000 7fe43bfff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe43c133010 0x7fe43c11a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:23:08.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.199+0000 7fe43b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe43c09de60 0x7fe43c1308f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:23:08.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.199+0000 7fe43bfff640 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe43c133010 0x7fe43c11a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:60080/0 (socket says 172.21.15.40:60080) 2024-04-22T08:23:08.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.199+0000 7fe43bfff640 1 -- 172.21.15.40:0/1077258247 learned_addr learned my addr 172.21.15.40:0/1077258247 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-22T08:23:08.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.199+0000 7fe43bfff640 10 monclient(hunting): get_auth_request con 0x7fe43c133010 auth_method 0 2024-04-22T08:23:08.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.199+0000 7fe43bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:23:08.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.199+0000 7fe43bfff640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:23:08.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.199+0000 7fe43bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:23:08.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.199+0000 7fe43b7fe640 10 monclient(hunting): get_auth_request con 0x7fe43c09de60 auth_method 0 2024-04-22T08:23:08.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.199+0000 7fe43b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-22T08:23:08.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.199+0000 7fe43b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-22T08:23:08.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.199+0000 7fe43b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-22T08:23:08.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.199+0000 7fe43bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:23:08.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.199+0000 7fe43bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:23:08.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.199+0000 7fe43bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:23:08.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.199+0000 7fe43b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-22T08:23:08.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.199+0000 7fe43b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-22T08:23:08.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.199+0000 7fe43b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-22T08:23:08.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.199+0000 7fe43affd640 1 --2- 172.21.15.40:0/1077258247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe43c131e00 0x7fe43c130e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:23:08.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.199+0000 7fe43bfff640 10 monclient(hunting): handle_auth_done global_id 124202 payload 824 2024-04-22T08:23:08.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.199+0000 7fe43bfff640 1 -- 172.21.15.40:0/1077258247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe43c131e00 msgr2=0x7fe43c130e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:23:08.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.199+0000 7fe43bfff640 1 --2- 172.21.15.40:0/1077258247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe43c131e00 0x7fe43c130e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:23:08.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.199+0000 7fe43bfff640 1 -- 172.21.15.40:0/1077258247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe43c09de60 msgr2=0x7fe43c1308f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:23:08.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.199+0000 7fe43bfff640 1 --2- 172.21.15.40:0/1077258247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe43c09de60 0x7fe43c1308f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:23:08.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.199+0000 7fe43bfff640 10 monclient: _finish_hunting 0 2024-04-22T08:23:08.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.199+0000 7fe43bfff640 1 monclient: found mon.noname-c 2024-04-22T08:23:08.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.199+0000 7fe43bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-22T08:23:08.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.199+0000 7fe43bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2024-04-22T08:23:08.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.199+0000 7fe43bfff640 1 -- 172.21.15.40:0/1077258247 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe43400f050 con 0x7fe43c133010 2024-04-22T08:23:08.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.199+0000 7fe43b7fe640 1 --2- 172.21.15.40:0/1077258247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe43c09de60 0x7fe43c1308f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T08:23:08.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.202+0000 7fe43bfff640 1 -- 172.21.15.40:0/1077258247 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe43c133010 msgr2=0x7fe43c11a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-22T08:23:08.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.202+0000 7fe43bfff640 1 --2- 172.21.15.40:0/1077258247 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe43c133010 0x7fe43c11a6a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe43400f500 tx=0x7fe43400f9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:23:08.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.202+0000 7fe423fff640 1 -- 172.21.15.40:0/1077258247 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe434012050 con 0x7fe43c133010 2024-04-22T08:23:08.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.202+0000 7fe423fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:23:08.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.202+0000 7fe423fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-22T08:23:08.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.202+0000 7fe423fff640 10 monclient: dump: 2024-04-22T08:23:08.212 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:23:08.212 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:23:08.212 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:23:08.212 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:23:08.212 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:23:08.212 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:23:08.212 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:23:08.212 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:23:08.212 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:23:08.212 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:23:08.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.202+0000 7fe423fff640 10 monclient: _finish_auth 0 2024-04-22T08:23:08.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.202+0000 7fe423fff640 10 monclient: _check_auth_tickets 2024-04-22T08:23:08.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.202+0000 7fe423fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-22T08:23:08.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.202+0000 7fe441b1f640 5 monclient: authenticate success, global_id 124202 2024-04-22T08:23:08.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.202+0000 7fe423fff640 1 -- 172.21.15.40:0/1077258247 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe434013040 con 0x7fe43c133010 2024-04-22T08:23:08.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.202+0000 7fe441b1f640 10 monclient: _renew_subs 2024-04-22T08:23:08.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.202+0000 7fe441b1f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:23:08.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.202+0000 7fe441b1f640 1 -- 172.21.15.40:0/1077258247 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe43000b4b0 con 0x7fe43c133010 2024-04-22T08:23:08.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.202+0000 7fe423fff640 10 monclient: handle_config config(0 keys) v1 2024-04-22T08:23:08.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.202+0000 7fe423fff640 1 -- 172.21.15.40:0/1077258247 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe434003df0 con 0x7fe43c133010 2024-04-22T08:23:08.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.202+0000 7fe423fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-04-22T08:23:08.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.202+0000 7fe423fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-22T08:23:08.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.202+0000 7fe423fff640 10 monclient: dump: 2024-04-22T08:23:08.213 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2024-04-22T08:23:08.213 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 987d842f-7205-47b9-86e0-2e2623f3c370 2024-04-22T08:23:08.213 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2024-04-22T08:12:05.320036+0000 2024-04-22T08:23:08.213 INFO:teuthology.orchestra.run.smithi040.stderr:created 2024-04-22T08:12:05.320036+0000 2024-04-22T08:23:08.213 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 18 (reef) 2024-04-22T08:23:08.214 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2024-04-22T08:23:08.214 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-22T08:23:08.214 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-04-22T08:23:08.214 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2024-04-22T08:23:08.214 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-22T08:23:08.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.202+0000 7fe441b1f640 10 monclient: _renew_subs 2024-04-22T08:23:08.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.202+0000 7fe441b1f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:23:08.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.202+0000 7fe441b1f640 1 -- 172.21.15.40:0/1077258247 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe43c11e040 con 0x7fe43c133010 2024-04-22T08:23:08.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.202+0000 7fe439ffb640 4 set_mon_vals no callback set 2024-04-22T08:23:08.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.203+0000 7fe423fff640 1 -- 172.21.15.40:0/1077258247 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 44782+0+0 (secure 0 0 0) 0x7fe4340177e0 con 0x7fe43c133010 2024-04-22T08:23:08.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.203+0000 7fe441b1f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-22T08:23:08.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.203+0000 7fe441b1f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:23:08.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.203+0000 7fe441b1f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:23:08.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.203+0000 7fe441b1f640 1 -- 172.21.15.40:0/1077258247 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe43c04e9b0 con 0x7fe43c133010 2024-04-22T08:23:08.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.203+0000 7fe423fff640 1 --2- 172.21.15.40:0/1077258247 >> v2:172.21.15.189:6800/30168 conn(0x7fe4280377d0 0x7fe428039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T08:23:08.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.203+0000 7fe423fff640 1 -- 172.21.15.40:0/1077258247 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(317..317 src has 239..317) v4 ==== 3506+0+0 (secure 0 0 0) 0x7fe43404ecb0 con 0x7fe43c133010 2024-04-22T08:23:08.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.203+0000 7fe43b7fe640 1 --2- 172.21.15.40:0/1077258247 >> v2:172.21.15.189:6800/30168 conn(0x7fe4280377d0 0x7fe428039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T08:23:08.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.203+0000 7fe43b7fe640 10 monclient: get_auth_request con 0x7fe4280377d0 auth_method 0 2024-04-22T08:23:08.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.206+0000 7fe43b7fe640 1 --2- 172.21.15.40:0/1077258247 >> v2:172.21.15.189:6800/30168 conn(0x7fe4280377d0 0x7fe428039c90 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fe42c004320 tx=0x7fe42c007e30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T08:23:08.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.207+0000 7fe423fff640 1 -- 172.21.15.40:0/1077258247 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182227 (secure 0 0 0) 0x7fe434004030 con 0x7fe43c133010 2024-04-22T08:23:08.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.207+0000 7fe423fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-22T08:23:08.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.207+0000 7fe423fff640 10 monclient: _finish_command 1 = system:0 2024-04-22T08:23:08.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.389+0000 7fe441b1f640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-04-22T08:23:08.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.389+0000 7fe441b1f640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-04-22T08:23:08.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.389+0000 7fe441b1f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2024-04-22T08:23:08.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.389+0000 7fe441b1f640 1 -- 172.21.15.40:0/1077258247 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7fe43c11c5f0 con 0x7fe43c133010 2024-04-22T08:23:08.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.395+0000 7fe423fff640 1 -- 172.21.15.40:0/1077258247 <== mon.2 v2:172.21.15.40:3301/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fe434054020 con 0x7fe43c133010 2024-04-22T08:23:08.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.395+0000 7fe423fff640 10 monclient: handle_config config(1 keys) v1 2024-04-22T08:23:08.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.395+0000 7fe439ffb640 4 set_mon_vals no callback set 2024-04-22T08:23:08.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.395+0000 7fe439ffb640 10 set_mon_vals mon_health_to_clog = false 2024-04-22T08:23:08.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.395+0000 7fe423fff640 1 -- 172.21.15.40:0/1077258247 <== mon.2 v2:172.21.15.40:3301/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v3)=0 v3) v1 ==== 123+0+0 (secure 0 0 0) 0x7fe43401e030 con 0x7fe43c133010 2024-04-22T08:23:08.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.396+0000 7fe423fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-04-22T08:23:08.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.396+0000 7fe423fff640 10 monclient: _finish_command 2 = system:0 2024-04-22T08:23:08.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.398+0000 7fe441b1f640 1 -- 172.21.15.40:0/1077258247 >> v2:172.21.15.189:6800/30168 conn(0x7fe4280377d0 msgr2=0x7fe428039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:23:08.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.398+0000 7fe441b1f640 1 --2- 172.21.15.40:0/1077258247 >> v2:172.21.15.189:6800/30168 conn(0x7fe4280377d0 0x7fe428039c90 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fe42c004320 tx=0x7fe42c007e30 comp rx=0 tx=0).stop 2024-04-22T08:23:08.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.398+0000 7fe441b1f640 10 monclient: shutdown 2024-04-22T08:23:08.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.398+0000 7fe441b1f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-22T08:23:08.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.398+0000 7fe441b1f640 1 -- 172.21.15.40:0/1077258247 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe43c133010 msgr2=0x7fe43c11a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T08:23:08.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.398+0000 7fe441b1f640 1 --2- 172.21.15.40:0/1077258247 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe43c133010 0x7fe43c11a6a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe43400f500 tx=0x7fe43400f9c0 comp rx=0 tx=0).stop 2024-04-22T08:23:08.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.398+0000 7fe441b1f640 1 -- 172.21.15.40:0/1077258247 shutdown_connections 2024-04-22T08:23:08.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.398+0000 7fe441b1f640 1 --2- 172.21.15.40:0/1077258247 >> v2:172.21.15.189:6800/30168 conn(0x7fe4280377d0 0x7fe428039c90 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:23:08.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.398+0000 7fe441b1f640 1 --2- 172.21.15.40:0/1077258247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe43c131e00 0x7fe43c130e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:23:08.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.398+0000 7fe441b1f640 1 --2- 172.21.15.40:0/1077258247 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe43c133010 0x7fe43c11a6a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:23:08.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.398+0000 7fe441b1f640 1 --2- 172.21.15.40:0/1077258247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe43c09de60 0x7fe43c1308f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T08:23:08.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.398+0000 7fe441b1f640 1 -- 172.21.15.40:0/1077258247 >> 172.21.15.40:0/1077258247 conn(0x7fe43c0a0b50 msgr2=0x7fe43c09b6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T08:23:08.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.398+0000 7fe441b1f640 1 -- 172.21.15.40:0/1077258247 shutdown_connections 2024-04-22T08:23:08.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-22T08:23:08.399+0000 7fe441b1f640 1 -- 172.21.15.40:0/1077258247 wait complete. 2024-04-22T08:23:08.408 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-22T08:23:08.408 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-22T08:23:08.409 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-22T08:23:08.409 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T08:23:08.448 INFO:tasks.ceph.osd.0:Stopped 2024-04-22T08:23:08.448 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-22T08:23:08.448 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T08:23:08.518 INFO:tasks.ceph.osd.1:Stopped 2024-04-22T08:23:08.518 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-22T08:23:08.518 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T08:23:08.585 INFO:tasks.ceph.osd.2:Stopped 2024-04-22T08:23:08.585 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-22T08:23:08.585 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T08:23:08.625 INFO:tasks.ceph.osd.3:Stopped 2024-04-22T08:23:08.625 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-22T08:23:08.625 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T08:23:08.703 INFO:tasks.ceph.osd.4:Stopped 2024-04-22T08:23:08.703 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-22T08:23:08.703 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T08:23:08.775 INFO:tasks.ceph.osd.5:Stopped 2024-04-22T08:23:08.775 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-22T08:23:08.775 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-22T08:23:08.775 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T08:23:08.844 INFO:tasks.ceph.mgr.x:Stopped 2024-04-22T08:23:08.844 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-22T08:23:08.844 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-22T08:23:08.844 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T08:23:08.855 INFO:tasks.ceph.mon.a:Stopped 2024-04-22T08:23:08.855 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-22T08:23:08.855 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T08:23:08.906 INFO:tasks.ceph.mon.c:Stopped 2024-04-22T08:23:08.906 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-22T08:23:08.906 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T08:23:08.920 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2024-04-22T08:23:08.921 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2024-04-22T08:23:08.921 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2024-04-22T08:23:08.921 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-04-22T08:23:08.921 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-04-22T08:23:08.921 INFO:tasks.ceph.mon.b:Stopped 2024-04-22T08:23:08.921 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-22T08:23:08.921 DEBUG:teuthology.orchestra.run.smithi040:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | head -n 1 2024-04-22T08:23:08.976 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi040.front.sepia.ceph.com 2024-04-22T08:23:08.976 DEBUG:teuthology.orchestra.run.smithi040:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-22T08:23:09.143 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi040.front.sepia.ceph.com 2024-04-22T08:23:09.144 DEBUG:teuthology.orchestra.run.smithi040:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-22T08:23:09.249 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi040.front.sepia.ceph.com 2024-04-22T08:23:09.249 DEBUG:teuthology.orchestra.run.smithi040:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-22T08:23:09.318 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi189.front.sepia.ceph.com 2024-04-22T08:23:09.318 DEBUG:teuthology.orchestra.run.smithi189:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-22T08:23:09.456 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi189.front.sepia.ceph.com 2024-04-22T08:23:09.456 DEBUG:teuthology.orchestra.run.smithi189:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-22T08:23:09.567 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi189.front.sepia.ceph.com 2024-04-22T08:23:09.567 DEBUG:teuthology.orchestra.run.smithi189:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-22T08:23:09.626 INFO:tasks.ceph:Archiving mon data... 2024-04-22T08:23:09.628 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-04-21_22:00:02-rados-reef-distro-default-smithi/7667865/data/mon.a.tgz 2024-04-22T08:23:09.628 DEBUG:teuthology.orchestra.run.smithi040:> mktemp 2024-04-22T08:23:09.643 INFO:teuthology.orchestra.run.smithi040.stdout:/tmp/tmp.2BeoprfstZ 2024-04-22T08:23:09.644 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.2BeoprfstZ 2024-04-22T08:23:09.803 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0666 /tmp/tmp.2BeoprfstZ 2024-04-22T08:23:09.886 DEBUG:teuthology.orchestra.remote:smithi040:/tmp/tmp.2BeoprfstZ is 447KB 2024-04-22T08:23:09.983 DEBUG:teuthology.orchestra.run.smithi040:> rm -fr /tmp/tmp.2BeoprfstZ 2024-04-22T08:23:09.998 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-04-21_22:00:02-rados-reef-distro-default-smithi/7667865/data/mon.c.tgz 2024-04-22T08:23:09.998 DEBUG:teuthology.orchestra.run.smithi040:> mktemp 2024-04-22T08:23:10.052 INFO:teuthology.orchestra.run.smithi040.stdout:/tmp/tmp.h7KfM0dXTj 2024-04-22T08:23:10.052 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.h7KfM0dXTj 2024-04-22T08:23:10.200 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0666 /tmp/tmp.h7KfM0dXTj 2024-04-22T08:23:10.282 DEBUG:teuthology.orchestra.remote:smithi040:/tmp/tmp.h7KfM0dXTj is 444KB 2024-04-22T08:23:10.381 DEBUG:teuthology.orchestra.run.smithi040:> rm -fr /tmp/tmp.h7KfM0dXTj 2024-04-22T08:23:10.396 DEBUG:teuthology.misc:Transferring archived files from smithi189:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-04-21_22:00:02-rados-reef-distro-default-smithi/7667865/data/mon.b.tgz 2024-04-22T08:23:10.396 DEBUG:teuthology.orchestra.run.smithi189:> mktemp 2024-04-22T08:23:10.414 INFO:teuthology.orchestra.run.smithi189.stdout:/tmp/tmp.MjdJNT90pE 2024-04-22T08:23:10.415 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.MjdJNT90pE 2024-04-22T08:23:10.580 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 0666 /tmp/tmp.MjdJNT90pE 2024-04-22T08:23:10.662 DEBUG:teuthology.orchestra.remote:smithi189:/tmp/tmp.MjdJNT90pE is 445KB 2024-04-22T08:23:10.760 DEBUG:teuthology.orchestra.run.smithi189:> rm -fr /tmp/tmp.MjdJNT90pE 2024-04-22T08:23:10.775 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-22T08:23:10.775 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-22T08:23:10.777 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-22T08:23:10.838 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-04-22T08:23:10.838 DEBUG:teuthology.orchestra.run.smithi040:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-04-22T08:23:10.842 DEBUG:teuthology.orchestra.run.smithi189:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-04-22T08:23:10.867 INFO:teuthology.orchestra.run.smithi040.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-04-22T08:23:10.904 INFO:teuthology.orchestra.run.smithi189.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-04-22T08:23:10.908 INFO:tasks.ceph:Archiving crash dumps... 2024-04-22T08:23:10.910 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-21_22:00:02-rados-reef-distro-default-smithi/7667865/remote/smithi040/crash 2024-04-22T08:23:10.910 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-22T08:23:10.935 DEBUG:teuthology.misc:Transferring archived files from smithi189:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-21_22:00:02-rados-reef-distro-default-smithi/7667865/remote/smithi189/crash 2024-04-22T08:23:10.936 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-22T08:23:10.971 INFO:tasks.ceph:Compressing logs... 2024-04-22T08:23:10.971 DEBUG:teuthology.orchestra.run.smithi040:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T08:23:10.979 DEBUG:teuthology.orchestra.run.smithi189:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T08:23:13.921 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2024-04-22T08:23:13.922 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2024-04-22T08:23:13.922 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2024-04-22T08:23:13.922 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-04-22T08:23:13.922 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2024-04-22T08:23:20.211 INFO:tasks.ceph:Archiving logs... 2024-04-22T08:23:20.212 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-21_22:00:02-rados-reef-distro-default-smithi/7667865/remote/smithi040/log 2024-04-22T08:23:20.213 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-22T08:23:21.557 DEBUG:teuthology.misc:Transferring archived files from smithi189:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-21_22:00:02-rados-reef-distro-default-smithi/7667865/remote/smithi189/log 2024-04-22T08:23:21.558 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-22T08:23:22.913 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-22T08:23:22.925 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-22T08:23:22.925 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-22T08:23:22.928 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-22T08:23:23.015 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-22T08:23:23.025 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-22T08:23:23.025 DEBUG:teuthology.orchestra.run.smithi040:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T08:23:23.030 DEBUG:teuthology.orchestra.run.smithi189:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T08:23:23.042 INFO:teuthology.orchestra.run.smithi040.stderr:bash: line 1: ntpq: command not found 2024-04-22T08:23:23.046 INFO:teuthology.orchestra.run.smithi189.stderr:bash: line 1: ntpq: command not found 2024-04-22T08:23:23.047 INFO:teuthology.orchestra.run.smithi040.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T08:23:23.047 INFO:teuthology.orchestra.run.smithi040.stdout:=============================================================================== 2024-04-22T08:23:23.048 INFO:teuthology.orchestra.run.smithi040.stdout:^- hv01.front.sepia.ceph.com 2 6 377 63 +1260us[+1267us] +/- 651ms 2024-04-22T08:23:23.048 INFO:teuthology.orchestra.run.smithi040.stdout:^* hv02.front.sepia.ceph.com 3 6 377 62 +74us[ +80us] +/- 77ms 2024-04-22T08:23:23.048 INFO:teuthology.orchestra.run.smithi040.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 61 -36us[ -36us] +/- 67ms 2024-04-22T08:23:23.048 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T08:23:23.051 INFO:teuthology.orchestra.run.smithi189.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T08:23:23.051 INFO:teuthology.orchestra.run.smithi189.stdout:=============================================================================== 2024-04-22T08:23:23.051 INFO:teuthology.orchestra.run.smithi189.stdout:^- hv01.front.sepia.ceph.com 2 6 377 60 +1259us[+1259us] +/- 651ms 2024-04-22T08:23:23.051 INFO:teuthology.orchestra.run.smithi189.stdout:^* hv02.front.sepia.ceph.com 3 6 377 62 +72us[ +90us] +/- 77ms 2024-04-22T08:23:23.051 INFO:teuthology.orchestra.run.smithi189.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 62 -44us[ -44us] +/- 67ms 2024-04-22T08:23:23.051 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T08:23:23.052 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-22T08:23:23.061 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-22T08:23:23.062 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-22T08:23:23.088 DEBUG:teuthology.orchestra.run.smithi040:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-22T08:23:23.159 DEBUG:teuthology.orchestra.run.smithi189:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-22T08:23:23.223 DEBUG:teuthology.orchestra.run.smithi040:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-22T08:23:23.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T08:23:23.253 DEBUG:teuthology.orchestra.run.smithi189:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-22T08:23:23.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T08:23:23.282 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-22T08:23:23.291 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-22T08:23:23.362 INFO:teuthology.task.internal:Duration was 1057.144029 seconds 2024-04-22T08:23:23.363 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-22T08:23:23.395 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-22T08:23:23.395 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-22T08:23:23.397 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-22T08:23:23.435 INFO:teuthology.orchestra.run.smithi040.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T08:23:23.436 INFO:teuthology.orchestra.run.smithi189.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T08:23:23.698 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-22T08:23:23.698 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi040.front.sepia.ceph.com 2024-04-22T08:23:23.699 DEBUG:teuthology.orchestra.run.smithi040:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-22T08:23:23.760 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi189.front.sepia.ceph.com 2024-04-22T08:23:23.761 DEBUG:teuthology.orchestra.run.smithi189:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-22T08:23:23.783 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-22T08:23:23.783 DEBUG:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T08:23:23.803 DEBUG:teuthology.orchestra.run.smithi189:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T08:23:23.848 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-22T08:23:23.848 DEBUG:teuthology.orchestra.run.smithi040:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-22T08:23:23.870 DEBUG:teuthology.orchestra.run.smithi189:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-22T08:23:23.986 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-22T08:23:24.000 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-22T08:23:24.001 DEBUG:teuthology.orchestra.run.smithi040:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-22T08:23:24.025 DEBUG:teuthology.orchestra.run.smithi189:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-22T08:23:24.049 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-22T08:23:24.060 DEBUG:teuthology.orchestra.run.smithi040:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-22T08:23:24.068 DEBUG:teuthology.orchestra.run.smithi189:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-22T08:23:24.088 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern = core 2024-04-22T08:23:24.110 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern = core 2024-04-22T08:23:24.132 DEBUG:teuthology.orchestra.run.smithi040:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-22T08:23:24.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T08:23:24.171 DEBUG:teuthology.orchestra.run.smithi189:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-22T08:23:24.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T08:23:24.186 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-22T08:23:24.206 INFO:teuthology.task.internal:Transferring archived files... 2024-04-22T08:23:24.206 DEBUG:teuthology.misc:Transferring archived files from smithi040:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-21_22:00:02-rados-reef-distro-default-smithi/7667865/remote/smithi040 2024-04-22T08:23:24.206 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-22T08:23:24.243 DEBUG:teuthology.misc:Transferring archived files from smithi189:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-21_22:00:02-rados-reef-distro-default-smithi/7667865/remote/smithi189 2024-04-22T08:23:24.243 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-22T08:23:24.274 INFO:teuthology.task.internal:Removing archive directory... 2024-04-22T08:23:24.274 DEBUG:teuthology.orchestra.run.smithi040:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-22T08:23:24.281 DEBUG:teuthology.orchestra.run.smithi189:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-22T08:23:24.324 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-22T08:23:24.335 INFO:teuthology.task.internal:Not uploading archives. 2024-04-22T08:23:24.335 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-22T08:23:24.363 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-22T08:23:24.363 DEBUG:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-22T08:23:24.365 DEBUG:teuthology.orchestra.run.smithi189:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-22T08:23:24.378 INFO:teuthology.orchestra.run.smithi040.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 22 08:23 /home/ubuntu/cephtest 2024-04-22T08:23:24.379 INFO:teuthology.orchestra.run.smithi189.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 22 08:23 /home/ubuntu/cephtest 2024-04-22T08:23:24.380 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-22T08:23:24.395 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-22T08:23:24.464 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/one workloads/pool-create-delete} duration: 1057.1440291404724 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-04-22T08:23:24.464 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-22T08:23:24.551 INFO:teuthology.run:pass